srun(1) - Run parallel jobs
-A, --account=<account>
       Charge  resources  used by this job to specified account.  The account is an arbitrary string. The
       account name may be changed after job submission using the scontrol command.
--acctg-freq=<seconds>
       Define  the  job  accounting  sampling   interval.    This   can   be   used   to   override   the
       JobAcctGatherFrequency  parameter  in  SLURM's  configuration  file,  slurm.conf.  A value of zero
       disables real  the  periodic  job  sampling  and  provides  accounting  information  only  on  job
       termination (reducing SLURM interference with the job).
-B --extra-node-info=<sockets[:cores[:threads]]>
       Request a specific allocation of resources with details as to the number and type of computational
       resources within a cluster: number of sockets (or physical processors) per node, cores per socket,
       and  threads per core.  The total amount of resources being requested is the product of all of the
       terms.  Each value specified is  considered  a  minimum.   An  asterisk  (*)  can  be  used  as  a
       placeholder  indicating  that  all  available  resources of that type are to be utilized.  As with
       nodes, the individual levels can also be specified in separate options if desired:
           --sockets-per-node=<sockets>
           --cores-per-socket=<cores>
           --threads-per-core=<threads>
       If task/affinity plugin is enabled, then specifying an allocation  in  this  manner  also  sets  a
       default  --cpu_bind  option  of  threads  if  the -B option specifies a thread count, otherwise an
       option of cores if a core count is specified, otherwise an option of sockets.   If  SelectType  is
       configured  to select/cons_res, it must have a parameter of CR_Core, CR_Core_Memory, CR_Socket, or
       CR_Socket_Memory for this option to be honored.  This option is not supported on BlueGene  systems
       (select/bluegene  plugin  is  configured).   If  not specified, the scontrol show job will display
       'ReqS:C:T=*:*:*'.
--begin=<time>
       Defer initiation of this job until the specified time.  It accepts times of the form  HH:MM:SS  to
       run  a  job  at a specific time of day (seconds are optional).  (If that time is already past, the
       next day is assumed.)  You may also specify midnight, noon, or teatime (4pm) and you  can  have  a
       time-of-day  suffixed  with  AM or PM for running in the morning or the evening.  You can also say
       what day the job will be run, by specifying a date of the  form  MMDDYY  or  MM/DD/YY  YYYY-MM-DD.
       Combine  date and time using the following format YYYY-MM-DD[THH:MM[:SS]]. You can also give times
       like now + count time-units, where the time-units can be seconds (default), minutes, hours,  days,
       or  weeks  and  you  can tell SLURM to run the job today with the keyword today and to run the job
       tomorrow with the keyword tomorrow.  The value may be  changed  after  job  submission  using  the
       scontrol command.  For example:
          --begin=16:00
          --begin=now+1hour
          --begin=now+60           (seconds by default)
          --begin=2010-01-20T12:34:00

       Notes on date/time specifications:
        -  Although  the  'seconds' field of the HH:MM:SS time specification is allowed by the code, note
       that the poll time of the SLURM scheduler is not precise enough to guarantee dispatch of  the  job
       on  the  exact second.  The job will be eligible to start on the next poll following the specified
       time. The exact poll interval depends on the SLURM scheduler (e.g., 60 seconds  with  the  default
       sched/builtin).
        - If no time (HH:MM:SS) is specified, the default is (00:00:00).
        -  If  a  date is specified without a year (e.g., MM/DD) then the current year is assumed, unless
       the combination of MM/DD and HH:MM:SS has already passed for that year, in  which  case  the  next
       year is used.
--checkpoint=<time>
       Specifies  the  interval  between  creating checkpoints of the job step.  By default, the job step
       will have no checkpoints created.  Acceptable time formats include  "minutes",  "minutes:seconds",
       "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds".
--checkpoint-dir=<directory>
       Specifies the directory into which the job or job step's checkpoint should be written (used by the
       checkpoint/blcr and checkpoint/xlch plugins only).  The  default  value  is  the  current  working
       directory.    Checkpoint   files   will   be   of   the   form   "<job_id>.ckpt"   for   jobs  and
       "<job_id>.<step_id>.ckpt" for job steps.
--comment=<string>
       An arbitrary comment.
-C, --constraint=<list>
       Specify a list of constraints.  The constraints are features that have been assigned to the  nodes
       by  the  slurm  administrator.  The list of constraints may include multiple features separated by
       ampersand (AND) and/or vertical bar (OR) operators.  For example: --constraint="opteron&video"  or
       --constraint="fast|faster".   In  the  first example, only nodes having both the feature "opteron"
       AND the feature "video" will be used.  There is no mechanism to specify that  you  want  one  node
       with  feature  "opteron"  and another node with feature "video" in case no node has both features.
       If only one of a set of possible options should be used for all allocated nodes, then use  the  OR
       operator     and    enclose    the    options    within    square    brackets.     For    example:
       "--constraint=[rack1|rack2|rack3|rack4]" might be used to specify that all nodes must be allocated
       on  a  single  rack  of  the cluster, but any of those four racks can be used.  A request can also
       specify the number of nodes needed with some feature by appending an asterisk and count after  the
       feature  name.   For example "srun --nodes=16 --constraint=graphics*4 ..."  indicates that the job
       requires 16 nodes at that at  least  four  of  those  nodes  must  have  the  feature  "graphics."
       Constraints  with  node  counts  may  only  be  combined with AND operators.  If no nodes have the
       requested features, then the job will be rejected by the slurm job manager. This  option  is  used
       for job allocations, but ignored for job step allocations.
--contiguous
       If  set,  then the allocated nodes must form a contiguous set.  Not honored with the topology/tree
       or topology/3d_torus plugins, both of which can modify the node ordering.  Not honored for  a  job
       step's allocation.
--cores-per-socket=<cores>
       Restrict  node  selection  to  nodes  with at least the specified number of cores per socket.  See
       additional information under -B option above when task/affinity plugin is enabled.
-c, --cpus-per-task=<ncpus>
       Request  that  ncpus  be allocated per process. This may be useful if the job is multithreaded and
       requires more than one CPU per task for optimal performance. The default is one CPU  per  process.
       If  -c  is  specified  without  -n,  as  many  tasks  will be allocated per node as possible while
       satisfying the -c restriction. For instance on a cluster with 8 CPUs per node, a job request for 4
       nodes  and 3 CPUs per task may be allocated 3 or 6 CPUs per node (1 or 2 tasks per node) depending
       upon resource consumption by other jobs. Such a job may be unable to execute more than a total  of
       4  tasks.   This  option may also be useful to spawn tasks without allocating resources to the job
       step from the job's allocation when running multiple job steps with the --exclusive option.

       WARNING: There are configurations and options interpreted differently by job and job step requests
       which  can  result  in inconsistencies for this option.  For example srun -c2 --threads-per-core=1
       prog may allocate two cores for the job, but if each of those cores contains two threads, the  job
       allocation  will  include  four CPUs. The job step allocation will then launch two threads per CPU
       for a total of two tasks.

       WARNING: When srun is executed from within salloc or sbatch, there are configurations and  options
       which  can  result  in  inconsistent  allocations when -c has a value greater than -c on salloc or
       sbatch.
-d, --dependency=<dependency_list>
       Defer the start of this job until  the  specified  dependencies  have  been  satisfied  completed.
       <dependency_list>  is  of  the  form <type:job_id[:job_id][,type:job_id[:job_id]]>.  Many jobs can
       share the same dependency and these jobs may even belong to different  users. The   value  may  be
       changed after job submission using the scontrol command.
-D, --chdir=<path>
       have the remote processes do a chdir to path before beginning execution. The default is  to  chdir
       to the current working directory of the srun process.
-e, --error=<mode>
       Specify  how  stderr is to be redirected. By default in interactive mode, srun redirects stderr to
       the same file as stdout, if one is specified. The --error option is provided to allow  stdout  and
       stderr  to  be  redirected to different locations.  See IO Redirection below for more options.  If
       the specified file already exists, it will be overwritten.
-E, --preserve-env
       Pass the current values of environment variables SLURM_NNODES  and  SLURM_NTASKS  through  to  the
       executable, rather than computing them from commandline parameters.
--epilog=<executable>
       srun  will  run  executable  just  after  the  job step completes.  The command line arguments for
       executable will be the command and arguments of the job step.  If executable is  "none",  then  no
       epilog will be run.  This parameter overrides the SrunEpilog parameter in slurm.conf.
--exclusive
       When  used to initiate a job, the job allocation cannot share nodes with other running jobs.  This
       is the opposite of --share, whichever option is seen last  on  the  command  line  will  win.  The
       default  shared/exclusive  behavior  depends  on  system  configuration and the partition's Shared
       option takes precedence over the job's option.

       This option can also be used when initiating more than one job step within  an  existing  resource
       allocation,  where  you  want  separate processors to be dedicated to each job step. If sufficient
       processors are not available to initiate the job step, it will be deferred. This can be thought of
       as  providing resource management for the job within it's allocation. Note that all CPUs allocated
       to a job are available to each job step unless the --exclusive option is used plus  task  affinity
       is  configured.  Since  resource  management is provided by processor, the --ntasks option must be
       specified, but the following options should NOT be specified --relative, --distribution=arbitrary.
       See EXAMPLE below.
--gid=<group>
       If  srun  is  run  as root, and the --gid option is used, submit the job with group's group access
       permissions.  group may be the group name or the numerical group ID.
--gres=<list>
       Specifies a comma delimited list of generic consumable resources.  The format of each entry on the
       list  is  "name[:count[*cpu]]".   The  name  is that of the consumable resource.  The count is the
       number of those resources with a default value of 1.  The specified resources will be allocated to
       the  job  on  each  node  allocated unless "*cpu" is appended, in which case the resources will be
       allocated on a per cpu basis.  The available generic consumable resources is configurable  by  the
       system  administrator.   A  list of available generic consumable resources will be printed and the
       command  will  exit   if   the   option   argument   is   "help".    Examples   of   use   include
       "--gres=gpu:2*cpu,disk=40G"  and  "--gres=help".  NOTE: By default, a job step is allocated all of
       the generic resources that have allocated to the job. To change the behavior so that each job step
       is  allocated  no generic resources, explicitly set the value of --gres to specify zero counts for
       each generic resource OR set "--gres=none" OR set  the  SLURM_STEP_GRES  environment  variable  to
       "none".
-H, --hold
       Specify  the  job  is  to  be submitted in a held state (priority of zero).  A held job can now be
       released using scontrol to reset its priority (e.g. "scontrol release <job_id>").
-h, --help
       Display help information and exit.
--hint=<type>
       Bind tasks according to application hints

       compute_bound
              Select settings for compute bound applications: use all cores in each  socket,  one  thread
              per core

       memory_bound
              Select settings for memory bound applications: use only one core in each socket, one thread
              per core

       [no]multithread
              [don't] use extra threads with in-core  multi-threading  which  can  benefit  communication
              intensive applications

       help   show this help message
-I, --immediate[=<seconds>]
       exit  if  resources  are not available within the time period specified.  If no argument is given,
       resources must be available immediately for the request to succeed.  By  default,  --immediate  is
       off, and the command will block until resources become available.
-i, --input=<mode>
       Specify  how stdin is to redirected. By default, srun redirects stdin from the terminal all tasks.
       See IO Redirection below for more options.  For OS X, the poll() function does not support  stdin,
       so input from a terminal is not possible.
-J, --job-name=<jobname>
       Specify  a  name  for  the  job.  The specified name will appear along with the job id number when
       querying running jobs on the system. The default is the supplied executable program's name.  NOTE:
       This  information may be written to the slurm_jobacct.log file. This file is space delimited so if
       a space is used in the jobname name it will cause problems in properly displaying the contents  of
       the slurm_jobacct.log file when the sacct command is used.
--jobid=<jobid>
       Initiate  a  job step under an already allocated job with job id id.  Using this option will cause
       srun to behave exactly as if the SLURM_JOB_ID environment variable was set.
-K, --kill-on-bad-exit[=0|1]
       Controls whether or not to terminate a job if any task exits with a non-zero exit  code.  If  this
       option  is  not specified, the default action will be based upon the SLURM configuration parameter
       of KillOnBadExit. If this option is specified, it will  take  precedence  over  KillOnBadExit.  An
       option  argument  of  zero  will  not  terminate  the job. A non-zero argument or no argument will
       terminate the job.  Note: This option takes precedence over the -W, --wait option to terminate the
       job immediately if a task exits with a non-zero exit code.
-k, --no-kill
       Do not automatically terminate a job of one of the nodes it has been allocated fails.  This option
       is only recognized on a job allocation, not for the submission of individual job steps.   The  job
       will  assume all responsibilities for fault-tolerance.  Tasks launch using this option will not be
       considered terminated (e.g. -K, --kill-on-bad-exit and -W, --wait options will have no effect upon
       the job step).  The active job step (MPI job) will likely suffer a fatal error, but subsequent job
       steps may be run if this option is specified.  The default action is to  terminate  the  job  upon
       node failure.
-l, --label
       prepend  task  number  to  lines  of  stdout/err. Normally, stdout and stderr from remote tasks is
       line-buffered directly to the stdout and stderr of srun.  The --label option will prepend lines of
       output with the remote task id.
-L, --licenses=<license>
       Specification of licenses (or other resources available on all nodes of the cluster) which must be
       allocated to this job.  License names can be followed by an asterisk and count (the default  count
       is one).  Multiple license names should be comma separated (e.g.  "--licenses=foo*4,bar").
-m, --distribution=
       <block|cyclic|arbitrary|plane=<options>[:block|cyclic]>
--mail-type=<type>
       Notify  user  by  email  when  certain event types occur.  Valid type values are BEGIN, END, FAIL,
       REQUEUE, and ALL (any state change). The user to be notified is indicated with --mail-user.
--mail-user=<user>
       User to receive email notification of state changes as defined by --mail-type.  The default  value
       is the submitting user.
--mem=<MB>
       Specify  the  real  memory required per node in MegaBytes.  Default value is DefMemPerNode and the
       maximum value is MaxMemPerNode. If configured, both of parameters can be seen using  the  scontrol
       show  config command.  This parameter would generally be used if whole nodes are allocated to jobs
       (SelectType=select/linear).   Also  see  --mem-per-cpu.   --mem  and  --mem-per-cpu  are  mutually
       exclusive.
--mem-per-cpu=<MB>
       Mimimum  memory  required  per  allocated CPU in MegaBytes.  Default value is DefMemPerCPU and the
       maximum value is MaxMemPerCPU (see exception below). If configured, both of parameters can be seen
       using  the  scontrol  show config command.  Note that if the job's --mem-per-cpu value exceeds the
       configured MaxMemPerCPU, then the user's limit will  be  treated  as  a  memory  limit  per  task;
       --mem-per-cpu  will be reduced to a value no larger than MaxMemPerCPU; --cpus-per-task will be set
       and value of --cpus-per-task multiplied by the new --mem-per-cpu value  will  equal  the  original
       --mem-per-cpu  value  specified by the user.  This parameter would generally be used if individual
       processors are allocated  to  jobs  (SelectType=select/cons_res).   Also  see  --mem.   --mem  and
       --mem-per-cpu are mutually exclusive.
--mincpus=<n>
       Specify a minimum number of logical cpus/processors per node.
--msg-timeout=<seconds>
       Modify  the  job  launch  message  timeout.   The  default  value  is  MessageTimeout in the SLURM
       configuration file slurm.conf.  Changes to this are typically not recommended, but could be useful
       to diagnose problems.
--mpi=<mpi_type>
       Identify the type of MPI to be used. May result in unique initiation procedures.
--multi-prog
       Run  a  job  with  different  programs  and  different  arguments for each task. In this case, the
       executable program specified is actually  a  configuration  file  specifying  the  executable  and
       arguments for each task. See MULTIPLE PROGRAM CONFIGURATION below for details on the configuration
       file contents.
-N, --nodes=<minnodes[-maxnodes]>
       Request that a minimum of minnodes nodes be allocated to this job.  A maximum node count may  also
       be specified with maxnodes.  If only one number is specified, this is used as both the minimum and
       maximum node count.  The partition's node limits supersede those of the  job.   If  a  job's  node
       limits  are outside of the range permitted for its associated partition, the job will be left in a
       PENDING state.  This permits possible execution at a later  time,  when  the  partition  limit  is
       changed.   If  a  job  node limit exceeds the number of nodes configured in the partition, the job
       will be rejected.  Note that the environment variable SLURM_NNODES will be set  to  the  count  of
       nodes  actually  allocated to the job. See the ENVIRONMENT VARIABLES section for more information.
       If -N is not specified,  the  default  behavior  is  to  allocate  enough  nodes  to  satisfy  the
       requirements of the -n and -c options.  The job will be allocated as many nodes as possible within
       the range specified and without delaying the initiation of the job.  The node count  specification
       may include a numeric value followed by a suffix of "k" (multiplies numeric value by 1,024) or "m"
       (multiplies numeric value by 1,048,576).
-n, --ntasks=<number>
       Specify the number of tasks to run. Request that srun allocate resources for  ntasks  tasks.   The
       default is one task per node, but note that the --cpus-per-task option will change this default.
--network=<type>
       Specify  the  communication  protocol to be used.  This option is supported on AIX systems.  Since
       POE is used to launch tasks,  this  option  is  not  normally  used  or  is  specified  using  the
       SLURM_NETWORK  environment variable.  The interpretation of type is system dependent.  For systems
       with an IBM Federation switch, the  following  comma-separated  and  case  insensitive  types  are
       recognized:  IP (the default is user-space), SN_ALL, SN_SINGLE, BULK_XFER and adapter names  (e.g.
       SNI0 and SNI1).  For more information, on IBM systems see poe  documentation  on  the  environment
       variables MP_EUIDEVICE and MP_USE_BULK_XFER.  Note that only four jobs steps may be active at once
       on a node with the BULK_XFER option due to limitations in the Federation switch driver.
--nice[=adjustment]
       Run the job with an adjusted scheduling priority within  SLURM.   With  no  adjustment  value  the
       scheduling priority is decreased by 100. The adjustment range is from -10000 (highest priority) to
       10000 (lowest priority). Only privileged users can  specify  a  negative  adjustment.  NOTE:  This
       option is presently ignored if SchedulerType=sched/wiki or SchedulerType=sched/wiki2.
--ntasks-per-core=<ntasks>
       Request  the  maximum  ntasks be invoked on each core.  Meant to be used with the --ntasks option.
       Related to --ntasks-per-node except at the core level instead  of  the  node  level.   Masks  will
       automatically be generated to bind the tasks to specific core unless --cpu_bind=none is specified.
       NOTE:    This    option    is    not    supported    unless    SelectTypeParameters=CR_Core     or
       SelectTypeParameters=CR_Core_Memory is configured.
--ntasks-per-node=<ntasks>
       Request  the  maximum  ntasks  be invoked on each node.  Meant to be used with the --nodes option.
       This is related to --cpus-per-task=ncpus, but does not require knowledge of the actual  number  of
       cpus on each node.  In some cases, it is more convenient to be able to request that no more than a
       specific number of tasks be invoked on each node.  Examples of this include  submitting  a  hybrid
       MPI/OpenMP  app  where only one MPI "task/rank" should be assigned to each node while allowing the
       OpenMP portion to utilize all of the parallelism present in  the  node,  or  submitting  a  single
       setup/cleanup/monitoring job to each node of a pre-existing allocation as one step in a larger job
       script.
--ntasks-per-socket=<ntasks>
       Request the maximum ntasks be invoked on each socket.  Meant to be used with the --ntasks  option.
       Related  to  --ntasks-per-node  except  at the socket level instead of the node level.  Masks will
       automatically be generated to bind  the  tasks  to  specific  sockets  unless  --cpu_bind=none  is
       specified.    NOTE:   This  option  is  not  supported  unless  SelectTypeParameters=CR_Socket  or
       SelectTypeParameters=CR_Socket_Memory is configured.
-O, --overcommit
       Overcommit resources. Normally, srun  will  not  allocate  more  than  one  process  per  CPU.  By
       specifying --overcommit you are explicitly allowing more than one process per CPU. However no more
       than MAX_TASKS_PER_NODE tasks are permitted to execute  per  node.   NOTE:  MAX_TASKS_PER_NODE  is
       defined in the file slurm.h and is not a variable, it is set at SLURM build time.
-o, --output=<mode>
       Specify the mode for stdout redirection. By default in interactive mode, srun collects stdout from
       all tasks and line buffers this output to the attached  terminal.  With  --output  stdout  may  be
       redirected  to a file, to one file per task, or to /dev/null. See section IO Redirection below for
       the various forms of mode.  If the specified file already exists, it will be overwritten.

       If --error is not also specified on the command line, both stdout and stderr will directed to  the
       file specified by --output.
--open-mode=<append|truncate>
       Open  the output and error files using append or truncate mode as specified.  The default value is
       specified by the system configuration parameter JobFileAppend.
-p, --partition=<partition_names>
       Request a specific partition for the resource allocation.  If not specified, the default  behavior
       is  to  allow  the  slurm  controller  to select the default partition as designated by the system
       administrator. If the job can use more than one partition, specify their names in a comma separate
       list and the one offering earliest initiation will be used.
--prolog=<executable>
       srun  will  run  executable  just  before  launching the job step.  The command line arguments for
       executable will be the command and arguments of the job step.  If executable is  "none",  then  no
       prolog will be run.  This parameter overrides the SrunProlog parameter in slurm.conf.
--propagate[=rlimits]
       Allows users to specify which of the modifiable (soft) resource limits to propagate to the compute
       nodes and apply to their jobs.  If rlimits is not specified, then  all  resource  limits  will  be
       propagated.   The  following rlimit names are supported by Slurm (although some options may not be
       supported on some systems):
--pty  Execute task zero in pseudo terminal mode.  Implicitly sets --unbuffered.  Implicitly sets --error
       and  --output  to  /dev/null  for  all tasks except task zero, which may cause those tasks to exit
       immediately (e.g. shells will typically  exit  immediately  in  that  situation).   Not  currently
       supported on AIX platforms.
-Q, --quiet
       Suppress informational messages from srun. Errors will still be displayed.
-q, --quit-on-interrupt
       Quit  immediately  on  single  SIGINT  (Ctrl-C).  Use  of  this option disables the status feature
       normally available when srun receives a single Ctrl-C  and  causes  srun  to  instead  immediately
       terminate the running job.
--qos=<qos>
       Request a quality of service for the job.  QOS values can be defined for each user/cluster/account
       association in the SLURM database.  Users will be limited to their association's  defined  set  of
       qos's  when  the  SLURM  configuration parameter, AccountingStorageEnforce, includes "qos" in it's
       definition.
-r, --relative=<n>
       Run a job step relative to node n of the current allocation.  This option may be  used  to  spread
       several job steps out among the nodes of the current job. If -r is used, the current job step will
       begin at node n of the allocated nodelist, where the first node is  considered  node  0.   The  -r
       option  is  not  permitted  with -w or -x option and will result in a fatal error when not running
       within a prior allocation (i.e. when SLURM_JOB_ID is not set). The default for  n  is  0.  If  the
       value  of  --nodes  exceeds  the  number of nodes identified with the --relative option, a warning
       message will be printed and the --relative option will take precedence.
--resv-ports
       Reserve communication ports for this job.  Used for OpenMPI.
--reservation=<name>
       Allocate resources for the job from the named reservation.
--restart-dir=<directory>
       Specifies the directory from which the job or job step's checkpoint should be read  (used  by  the
       checkpoint/blcrm and checkpoint/xlch plugins only).
-s, --share
       The  job allocation can share nodes with other running jobs.  This is the opposite of --exclusive,
       whichever option is seen last on the command line  will  be  used.  The  default  shared/exclusive
       behavior  depends  on system configuration and the partition's Shared option takes precedence over
       the job's option.  This option may result the allocation being granted sooner than if the  --share
       option  was  not  set and allow higher system utilization, but application performance will likely
       suffer due to competition for resources within a node.
--slurmd-debug=<level>
       Specify a debug level for slurmd(8). level may be an integer value between 0 [quiet,  only  errors
       are  displayed] and 4 [verbose operation].  The slurmd debug information is copied onto the stderr
       of the job. By default only errors are displayed.
--sockets-per-node=<sockets>
       Restrict node selection to nodes with at least the specified number of  sockets.   See  additional
       information under -B option above when task/affinity plugin is enabled.
-T, --threads=<nthreads>
       Allows  limiting  the  number  of  concurrent  threads  used to send the job request from the srun
       process to the slurmd processes on the allocated nodes. Default is to use one thread per allocated
       node  up  to  a  maximum  of  60  concurrent  threads. Specifying this option limits the number of
       concurrent threads to nthreads (less than or equal to 60).  This should only be used to set a  low
       thread count for testing on very small memory computers.
-t, --time=<time>
       Set  a  limit on the total run time of the job or job step.  If the requested time limit for a job
       exceeds the  partition's  time  limit,  the  job  will  be  left  in  a  PENDING  state  (possibly
       indefinitely).  If the requested time limit for a job step exceeds the partition's time limit, the
       job step will not be initiated.  The default time limit is the partition's time limit.   When  the
       time  limit is reached, the job's tasks are sent SIGTERM followed by SIGKILL. If the time limit is
       for the job, all job steps are signaled. If the time limit is for a  single  job  step  within  an
       existing  job allocation, only that job step will be affected. A job time limit supercedes all job
       step time limits. The interval between SIGTERM and SIGKILL is specified by the SLURM configuration
       parameter KillWait.  A time limit of zero requests that no time limit be imposed.  Acceptable time
       formats   include    "minutes",    "minutes:seconds",    "hours:minutes:seconds",    "days-hours",
       "days-hours:minutes" and "days-hours:minutes:seconds".
--task-epilog=<executable>
       The  slurmstepd  daemon will run executable just after each task terminates. This will be executed
       before any TaskEpilog parameter in slurm.conf is executed. This is meant to be a very  short-lived
       program.  If  it  fails  to  terminate  within  a  few  seconds,  it will be killed along with any
       descendant processes.
--task-prolog=<executable>
       The slurmstepd daemon will run executable just before launching each task. This will  be  executed
       after  any  TaskProlog  parameter  in  slurm.conf  is  executed.   Besides  the normal environment
       variables, this has SLURM_TASK_PID available to identify the process ID of the task being started.
       Standard  output from this program of the form "export NAME=value" will be used to set environment
       variables for the task being spawned.
--test-only
       Returns an estimate of when a job would be scheduled to run given the current job  queue  and  all
       the  other  srun  arguments  specifying  the  job.   This  limits  srun's  behavior to just return
       information; no job is actually submitted.  EXCEPTION:  On  Bluegene/Q  systems  on  when  running
       within  an existing job allocation, this disables the use of "runjob" to launch tasks. The program
       will be executed directly by the slurmd dameon.
--threads-per-core=<threads>
       Restrict node selection to nodes with at least the specified number  of  threads  per  core.   See
       additional information under -B option above when task/affinity plugin is enabled.
--time-min=<time>
       Set  a minimum time limit on the job allocation.  If specified, the job may have it's --time limit
       lowered to a value no lower than --time-min if doing so permits the job to begin execution earlier
       than  otherwise  possible.   The  job's  time limit will not be changed after the job is allocated
       resources.  This is performed by a backfill scheduling algorithm to allocate  resources  otherwise
       reserved  for higher priority jobs.  Acceptable time formats include "minutes", "minutes:seconds",
       "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds".
--tmp=<MB>
       Specify a minimum amount of temporary disk space.
-u, --unbuffered
       Do not line buffer stdout from remote tasks. This option cannot be used with --label.
--usage
       Display brief help message and exit.
--uid=<user>
       Attempt to submit and/or run a job as user instead of the invoking user id.  The  invoking  user's
       credentials  will  be used to check access permissions for the target partition. User root may use
       this option to run jobs as a normal user in a RootOnly partition for example. If run as root, srun
       will  drop  its  permissions to the uid specified after node allocation is successful. user may be
       the user name or numerical user ID.
-V, --version
       Display version information and exit.
-v, --verbose
       Increase the verbosity of srun's informational messages.   Multiple  -v's  will  further  increase
       srun's verbosity.  By default only errors will be displayed.
-W, --wait=<seconds>
       Specify how long to wait after the first task terminates before terminating all remaining tasks. A
       value of 0 indicates an unlimited wait (a warning will be issued after 60  seconds).  The  default
       value  is  set by the WaitTime parameter in the slurm configuration file (see slurm.conf(5)). This
       option can be useful to insure that a job is terminated in a timely fashion in the event that  one
       or  more  tasks  terminate  prematurely.  Note: The -K, --kill-on-bad-exit option takes precedence
       over -W, --wait to terminate the job immediately if a task exits with a non-zero exit code.
-w, --nodelist=<host1,host2,... or filename>
       Request a specific list of hosts. The job will contain at least  these  hosts.  The  list  may  be
       specified as a comma-separated list of hosts, a range of hosts (host[1-5,7,...] for example), or a
       filename.  The host list will be assumed to be a filename if it contains a "/" character.  If  you
       specify a max node count (-N1-2) if there are more than 2 hosts in the file only the first 2 nodes
       will be used in the request list.
--wckey=<wckey>
       Specify wckey to be used with job.  If TrackWCKey=no (default) in the  slurm.conf  this  value  is
       ignored.
-X, --disable-status
       Disable  the  display  of  task  status  when  srun  receives  a  single  SIGINT (Ctrl-C). Instead
       immediately forward the SIGINT to the running job.  Without this option a  second  Ctrl-C  in  one
       second  is  required to forcibly terminate the job and srun will immediately exit. May also be set
       via the environment variable SLURM_DISABLE_STATUS.
-x, --exclude=<host1,host2,... or filename>
       Request that a specific list of hosts not be included in the resources allocated to this job.  The
       host list will be assumed to be a filename if it contains a "/"character.
-Z, --no-allocate
       Run  the  specified  tasks  on  a  set  of nodes without creating a SLURM "job" in the SLURM queue
       structure, bypassing the normal resource allocation step.  The list of  nodes  must  be  specified
       with  the  -w,  --nodelist  option.   This  is  a  privileged  option only available for the users
       "SlurmUser" and "root".

The following options support Blue Gene systems, but may be applicable to other systems as well.
--blrts-image=<path>
       Path to blrts image for bluegene block.  BGL only.  Default from blugene.conf if not set.
--cnload-image=<path>
       Path to compute node image for bluegene block.  BGP only.  Default from blugene.conf if not set.
--conn-type=<type>
       Require the partition connection type to be of a certain type.  On Blue  Gene  the  acceptable  of
       type  are  MESH,  TORUS  and  NAV.  If NAV, or if not set, then SLURM will try to fit a TORUS else
       MESH.  You should not normally set this option.  SLURM will normally allocate a TORUS if  possible
       for  a  given  geometry.   If  running  on a BGP system and wanting to run in HTC mode (only for 1
       midplane and below).  You can use HTC_S for SMP, HTC_D for Dual, HTC_V for virtual node mode,  and
       HTC_L  for Linux mode.  A comma separated lists of connection types may be specified, one for each
       dimension.
-g, --geometry=<XxYxZ>
       Specify the geometry requirements for the job. The three numbers represent the  required  geometry
       giving  dimensions in the X, Y and Z directions. For example "--geometry=2x3x4", specifies a block
       of nodes having 2 x 3 x 4 = 24 nodes (actually base partitions on Blue Gene).
--ioload-image=<path>
       Path to io image for bluegene block.  BGP only.  Default from blugene.conf if not set.
--linux-image=<path>
       Path to linux image for bluegene block.  BGL only.  Default from blugene.conf if not set.
--mloader-image=<path>
       Path to mloader image for bluegene block.  Default from blugene.conf if not set.
-R, --no-rotate
       Disables rotation of the job's requested geometry in  order  to  fit  an  appropriate  block.   By
       default the specified geometry can rotate in three dimensions.
--ramdisk-image=<path>
       Path to ramdisk image for bluegene block.  BGL only.  Default from blugene.conf if not set.
--reboot
       Force the allocated nodes to reboot before starting the job.

srun  will  submit the job request to the slurm job controller, then initiate all processes on the remote
nodes. If the request cannot be met immediately, srun will block until the resources are free to run  the
job.  If  the  -I  (--immediate) option is specified srun will terminate if resources are not immediately
available.