* Committed my KB article on SLURM accounting fields.
Last updated: 2019-07-22.
This commit is contained in:
232
slurm/20190411.Slurm-accounting.md
Normal file
232
slurm/20190411.Slurm-accounting.md
Normal file
@@ -0,0 +1,232 @@
|
||||
SLURM ACCOUNTING (sacct)
|
||||
========================
|
||||
|
||||
CAVEAT:
|
||||
This document was originally developed by referencing SLURM
|
||||
18.08.1 used on Turing.
|
||||
I also tried to consult the newer version (master branch
|
||||
around July 2019).
|
||||
Newer version may introduce additional features, or features
|
||||
incompatible with this version.
|
||||
Please use a grain of salt when reading, and always consult with
|
||||
manual pages, source code, etc in case of doubt.
|
||||
|
||||
|
||||
UNDERSTANDING SLURM ACCOUNTING FIELDS
|
||||
-------------------------------------
|
||||
|
||||
SLURM accounting can produce very many fields.
|
||||
|
||||
`JobID`:
|
||||
The "cooked" job ID. Please see the discussion below.
|
||||
|
||||
`JobIDRaw`:
|
||||
The "raw" job ID. Please see the discussion below.
|
||||
|
||||
`TimelimitRaw`:
|
||||
The raw value of time limit, in minutes.
|
||||
|
||||
|
||||
|
||||
### About SLURM Job IDs
|
||||
|
||||
SLURM produces one or more records in the accounting database for every job.
|
||||
When a user submits a job to SLURM, SLURM assigns that job a unique job number,
|
||||
like this:
|
||||
|
||||
$ sbatch calculation.job
|
||||
Submitted batch job 8918299
|
||||
|
||||
However, internally within SLURM, there can be one or more "job steps" created
|
||||
and executed while this job is being launched and executed.
|
||||
(Things get more even complicated with newer "heterogenous job" feature,
|
||||
in which various parts of a job can require very different resources.
|
||||
See [this documentation](https://slurm.schedmd.com/heterogeneous_jobs.html)
|
||||
for more information.)
|
||||
|
||||
Several regex patterns have observed in the JobID field (from Turing
|
||||
accounting):
|
||||
|
||||
* `[0-9]+`
|
||||
|
||||
* `[0-9]+_[0-9]+` (for job arrays)
|
||||
|
||||
* `[0-9]+\.[0-9]+`
|
||||
|
||||
* `[0-9]+\.batch`
|
||||
|
||||
For all cases, the `JobIDRaw` is the same as `JobID` except in the case of
|
||||
`/[0-9]+_[0-9]+/`, where the `JobIDRaw` is a running number `[0-9]+`.
|
||||
This is the case where the submitter specifies an array of jobs.
|
||||
|
||||
From the slurm's sacct source code (`src/sacct/print.c`) one can find that there
|
||||
are other patterns too (look for string `case PRINT_JOBIDRAW:`).
|
||||
The key function in `print.c` is `print_fields`.
|
||||
In particular look at the lengthy `case` statement where it tackles
|
||||
`PRINT_JOBID` and `PRINT_JOBIDRAW` cases.
|
||||
|
||||
A job can be of different types:
|
||||
|
||||
* `JOB`
|
||||
* `JOBSTEP`
|
||||
* `JOBCOMP`
|
||||
|
||||
A `JOBSTEP` can have several subtypes:
|
||||
|
||||
* `SLURM_BATCH_SCRIPT`, in which case JobIDRaw will obtain the `.batch` suffix.
|
||||
* `SLURM_EXTERN_CONT`, in which case JobIDRaw will obtain the `.extern` suffix.
|
||||
Apparently, this is meant to indicate "external" type of job steps,
|
||||
including.
|
||||
* many others; but in this case, it will print JobIDRaw in `[0-9]+\.[0-9]+`
|
||||
pattern
|
||||
* Other types (usually it will have index numbers like 0, 1, 2, ...)
|
||||
|
||||
|
||||
#### Vanilla Job
|
||||
|
||||
A "vanilla" job entry corresponds to a single job submitted by a user to SLURM.
|
||||
This will not be a job array.
|
||||
|
||||
* Characteristics : `JobID ~ /^[0-9]+$/`.
|
||||
|
||||
|
||||
#### Array Job
|
||||
|
||||
An "array" job entry corresponds to a single job as part of a job
|
||||
array submitted by a user to SLURM.
|
||||
|
||||
* Characteristics : `JobID ~ /^[0-9]+_[0-9]+$/`.
|
||||
|
||||
The Job ID contains two numbers separated by an underscore.
|
||||
The number before the underscore refers to the job ID as reported by
|
||||
sbatch upon the submission of the job.
|
||||
|
||||
NOTE: Newer version of SLURM will allow textual word instead of
|
||||
numbers to identify one job in an array.
|
||||
Those text-based job label (instead of integer) will be marked by
|
||||
square brackets around the job suffix:
|
||||
|
||||
* Characteristics (textual array label): `JobID ~ /^[0-9]+_\[.*\]+$/`.
|
||||
|
||||
|
||||
#### Heterogenous Job
|
||||
|
||||
A heterogenous job entry corresponds to a part of a heterogenous job
|
||||
submitted by a user to SLURM.
|
||||
|
||||
* Characteristics : `JobID ~ /^[0-9]+\+[0-9]+$/`.
|
||||
|
||||
The Job ID contains two numbers separated by a plus sign.
|
||||
The number before the underscore refers to the job ID as reported by
|
||||
sbatch upon the submission of the job.
|
||||
|
||||
This will not be a job array.
|
||||
|
||||
|
||||
#### Job Step: Batch script
|
||||
|
||||
This corresponds to the execution of the batch script (submitted to
|
||||
sbatch) when more than one CPU cores were requested by the job.
|
||||
|
||||
Characteristics of SLURM_BATCH_SCRIPT accounting records:
|
||||
|
||||
* JobIDRaw =~ /^[0-9]+\.batch$/
|
||||
|
||||
* The record does NOT have user ID (field `User`)
|
||||
|
||||
* `JobName` is always `batch`
|
||||
|
||||
|
||||
#### Job Step: External
|
||||
|
||||
SLURM_EXTERN_CONT apparently is a way to account for "external processes".
|
||||
It is still not 100% obvious what this means, but from reading the
|
||||
source code, there are two types of stuff that will fall under this
|
||||
category:
|
||||
|
||||
* Job prologue
|
||||
|
||||
* Direct SSH access into an allocated compute node: in this case, the
|
||||
`pam_adopt_slurm` module will make the determination as to which
|
||||
SLURM job launches the ssh (if any) and attribute the portion of
|
||||
this computation to the calling job.
|
||||
|
||||
There were some other steps observed, whose JobIDRaw becomes `NNNNN.N`.
|
||||
I wonder if these "job steps" are due to the calls of "srun" within
|
||||
the batch script, because the job names are indicative: `pw.x`,
|
||||
`pmi_proxy`, etc..
|
||||
(Example job: 5947279 , Nov 2018.)
|
||||
|
||||
|
||||
#### Job Step: All the others
|
||||
|
||||
These correspond to job steps that were launched by `srun` or other
|
||||
similar mechanism instead the job script.
|
||||
A prime example is the `mpirun` launch, which will record a new job step.
|
||||
|
||||
|
||||
|
||||
|
||||
#### Job Completion
|
||||
|
||||
`JOBCOMP` appears to mark a job completion.
|
||||
Not sure if this kind of record appears on Turing accounting;
|
||||
that may be only when a specific "job completion" task is specified.
|
||||
|
||||
|
||||
#### Questions & (Possible) Answers
|
||||
|
||||
* Why there is a separate "NNNNN.batch" record?
|
||||
It is perhaps when the job is multi-node.
|
||||
It appears to me that the ".batch" record is for accounting the batch script
|
||||
itself (which will run only on node #0 of the allocated resources).
|
||||
|
||||
|
||||
#### The Takeaway
|
||||
|
||||
Why all this complicated explanation?
|
||||
My original goal was to find the accounting records which covers the
|
||||
whole-job statistics without getting bogged down by the minute details
|
||||
of each job.
|
||||
This is what I found after this exploration:
|
||||
|
||||
> We only need to include accounting records where the `JobIDRaw` field
|
||||
> contains only whole integers (i.e. matching regex `^[0-9]+$`).
|
||||
|
||||
|
||||
|
||||
## References
|
||||
|
||||
- `sacct` manual page:
|
||||
<https://slurm.schedmd.com/sacct.html>
|
||||
|
||||
SLURM administrator's documentation contains helpful bits and pieces
|
||||
to decipher the accounting records; unfortunately in themselves they
|
||||
are not sufficient.
|
||||
|
||||
- Accounting:
|
||||
<https://slurm.schedmd.com/accounting.html> .
|
||||
|
||||
- Job Launch design guide:
|
||||
<https://slurm.schedmd.com/job_launch.html> .
|
||||
|
||||
> This guide describes at a high level the processes which occur in
|
||||
> order to initiate a job including the daemons and plugins involved
|
||||
> in the process. It describes the process of job allocation, step
|
||||
> allocation, task launch and job termination.
|
||||
|
||||
In SLURM, launching a job is a multistep process.
|
||||
Various "job steps" described this guide eventually make their own
|
||||
entries in the SLURM accounting database.
|
||||
|
||||
|
||||
|
||||
### Working Notes
|
||||
|
||||
These are my private working notes:
|
||||
|
||||
- daily-notes/2019/20190326.slurm-acct.txt
|
||||
- daily-notes/2019/20190411.slurm-acct.txt
|
||||
- daily-notes/2019/20190430.slurm-acct-201811.txt
|
||||
- docs/kb/turing-slurm/20180106.SLURM-accounting.txt
|
||||
|
||||
Reference in New Issue
Block a user