Phoenix Partitions and QoSs

Bertha 。 2022-05-09 05:46 37阅读 0赞

The Phoenix cluster nodes are grouped into different sets (partitions) each fulfilling different purposes. These partitions can be considered to be separate "job queues", each of which can have a variety of sets of constraints on certain resources (nodes, processors, memory, time, etc.), so-called quality of service (QoS) constraints, associated with it. The Phoenix cluster runs the SLURM scheduler to manage all partitions.

The following will help you select the right partition for your job.

## Contents ##

\[[hide][]\]

*  [1How to submit to a partition (queue)][1How to submit to a partition _queue]
 *  [2Partitions][]
    
     *  [2.1The batch partition][]
     *  [2.2The cpu, skylake and skypool partitions][2.2The cpu_ skylake and skypool partitions]
        
         *  [2.2.1cpu partition limits][]
         *  [2.2.2skylake partition limits][]
         *  [2.2.3skypool partition limits][]
         *  [2.2.4Long QoS][]
     *  [2.3The gpu and volta partitions][]
        
         *  [2.3.1gpu partition limits][]
         *  [2.3.2volta partition limits][]
     *  [2.4The test and skytest partitions][]
        
         *  [2.4.1test partition limits][]
         *  [2.4.2skytest partition limits][]
     *  [2.5The highmem partition][]
        
         *  [2.5.1highmem partition limits][]

# How to submit to a partition (queue)\[[edit][] | [edit source][]\] #

There are two ways to tell SLURM which partition your job should run on. The first is through a direct command when submitting a job, i.e.:

sbatch -p <partition> jobscript

The second, more common, way is to define it in your jobscript via:

#SBATCH -p <partition>

# Partitions\[[edit][edit 1] | [edit source][edit source 1]\] #

The Phoenix cluster currently supports a number of different major SLURM partitions:

*  **batch**
 *  **cpu**
 *  **skylake**
 *  **skypool**
 *  **gpu**
 *  **volta**
 *  **test**
 *  **skytest**
 *  **highmem**

## The `batch` partition\[[edit][edit 2] | [edit source][edit source 2]\] ##

This is the default partition, and in the majority of cases you should submit your job to the `batch` partition. When the `batch` partition is specified, the scheduler will use an internal algorithm will determine which partition your job is eligible for, and it will be re-routed to all eligible partitions. The following partitions are considered for job eligibility:

*  **cpu, skylake, skypool** for CPU-only jobs
 *  **gpu, volta** for GPU jobs

The majority of jobs will be eligible for multiple partitions, and will run in the first partition with available resources that meet the job requirements.

## The `cpu`, `skylake` and `skypool` partitions\[[edit][edit 3] | [edit source][edit source 3]\] ##

These partitions are meant for general computational jobs that do not require any GPU accelerators and a moderate amount of RAM. Most of the jobs submitted to Phoenix will run in these partitions.

### cpu partition limits\[[edit][edit 4] | [edit source][edit source 4]\] ###

The `cpu` partition targets Phoenix nodes that have the Haswell Intel CPU architecture. It is also the partition with most resources (i.e. nodes) associated with it.

<table> 
 <tbody> 
  <tr> 
   <td colspan="2"><strong>cpu job constraints</strong></td> 
  </tr> 
  <tr> 
   <td>Max Job time</td> 
   <td>3-00:00:00</td> 
  </tr> 
  <tr> 
   <td>Max cpus/node</td> 
   <td>32</td> 
  </tr> 
  <tr> 
   <td>Max RAM per cpu</td> 
   <td>4 GB</td> 
  </tr> 
  <tr> 
   <td>Max RAM per node</td> 
   <td>125 GB</td> 
  </tr> 
  <tr> 
   <td>Cost</td> 
   <td>1 SU per CPU hour</td> 
  </tr> 
  <tr> 
   <td>Can use GPUs</td> 
   <td><strong>No</strong></td> 
  </tr> 
 </tbody> 
</table>

### skylake partition limits\[[edit][edit 5] | [edit source][edit source 5]\] ###

The `skylake` partition targets Phoenix nodes that have the Skylake Intel CPU architecture. This partition is suitable for general computational jobs that do not require any GPU accelerators, and a moderate to large amount of RAM. The Skylake nodes have up to 384 GB of memory available, which is triple the amount available to the Haswell nodes.

<table> 
 <tbody> 
  <tr> 
   <td colspan="2"><strong>skylake job constraints</strong></td> 
  </tr> 
  <tr> 
   <td>Max Job time</td> 
   <td>3-00:00:00</td> 
  </tr> 
  <tr> 
   <td>Max cpus/node</td> 
   <td>40</td> 
  </tr> 
  <tr> 
   <td>Max RAM per cpu</td> 
   <td>9665 MB</td> 
  </tr> 
  <tr> 
   <td>Max RAM per node</td> 
   <td>377 GB</td> 
  </tr> 
  <tr> 
   <td>Cost</td> 
   <td>1.25 SU per CPU hour</td> 
  </tr> 
  <tr> 
   <td>Can use GPUs</td> 
   <td><strong>No</strong></td> 
  </tr> 
 </tbody> 
</table>

### skypool partition limits\[[edit][edit 6] | [edit source][edit source 6]\] ###

Jobs that are eligible to run on the `skylake` partition will also be considered for the `skypool` partition. The purpose of the `skypool` partition is to maximise CPU utilisation across the Skylake nodes by allowing CPU-only jobs that have a sufficiently small core-per-node requirement to run alongside GPU jobs in the `volta` partition.

<table> 
 <tbody> 
  <tr> 
   <td colspan="2"><strong>skypool job constraints</strong></td> 
  </tr> 
  <tr> 
   <td>Max Job time</td> 
   <td>3-00:00:00</td> 
  </tr> 
  <tr> 
   <td>Max cpus/node</td> 
   <td>32</td> 
  </tr> 
  <tr> 
   <td>Max RAM per cpu</td> 
   <td>9665 MB</td> 
  </tr> 
  <tr> 
   <td>Max RAM per node</td> 
   <td>301 GB</td> 
  </tr> 
  <tr> 
   <td>Cost</td> 
   <td>1.25 SU per CPU hour</td> 
  </tr> 
  <tr> 
   <td>Can use GPUs</td> 
   <td><strong>No</strong></td> 
  </tr> 
 </tbody> 
</table>

### Long QoS\[[edit][edit 7] | [edit source][edit source 7]\] ###

If for some reason it is not possible to run your job within three days by e.g. increasing the number of CPUs/RAM or by check-pointing (saving the current state and restarting it), we offer a long QoS upon **special request**. Each request will be assessed on an individual basis. Accessing the long QoS requires a fair-share factor of greater than 0.25.

If you have been granted access to the long QoS, you can select it by adding the following to your job submission script:

#SBATCH --qos=long

The long QoS features are the following:

<table> 
 <tbody> 
  <tr> 
   <td colspan="2"><strong>long QoS constraints</strong></td> 
  </tr> 
  <tr> 
   <td>Max Job time</td> 
   <td>7-00:00:00</td> 
  </tr> 
 </tbody> 
</table>

## The `gpu` and `volta` partitions\[[edit][edit 8] | [edit source][edit source 8]\] ##

These partitions are suitable for computational jobs which require GPU accelerators. Programmes that can run efficiently on GPUs can see speed-ups of 10X or better. (1 CPU hour vs. 1 GPU hour).

### gpu partition limits\[[edit][edit 9] | [edit source][edit source 9]\] ###

The `gpu` partition targets the Phoenix nodes that have Nvidia K80 accelerators. For the Nvidia K80s, the cost for one GPU-hour is 8 service units (SU). The minimum number of cores per GPU is 2, and the maximum is 4. The K80 nodes have 2 x K80 cards, and as each K80 is a dual-GPU device, there is a maximum of 4 GPUs that can be used per node.

<table> 
 <tbody> 
  <tr> 
   <td colspan="2"><strong>gpu job constraints</strong></td> 
  </tr> 
  <tr> 
   <td>Max Job time</td> 
   <td>2-00:00:00</td> 
  </tr> 
  <tr> 
   <td>Max cpus/node</td> 
   <td>16</td> 
  </tr> 
  <tr> 
   <td>Max RAM per cpu</td> 
   <td>4 GB</td> 
  </tr> 
  <tr> 
   <td>Max RAM per node</td> 
   <td>64 GB</td> 
  </tr> 
  <tr> 
   <td>Cost</td> 
   <td>8 SU per GPU hour</td> 
  </tr> 
  <tr> 
   <td>Can use GPUs</td> 
   <td><strong>Yes</strong></td> 
  </tr> 
 </tbody> 
</table>

The resources available for jobs that are assigned to the `gpu` partition depend on the number of GPUs requested. The scheduler controls these limits by **automatically** assigning each job one of three QoS flags, named `gxs, gxm, gxl`. These flags are designed to optimise the overall system utilisation, with each providing different core and memory limits based on the number of GPUs required. The following table summarises the constrains.

<table> 
 <tbody> 
  <tr> 
   <td><strong>QoS</strong></td> 
   <td><strong>gxs</strong></td> 
   <td><strong>gxm</strong></td> 
   <td><strong>gxl</strong></td> 
  </tr> 
  <tr> 
   <td>GPUs/node</td> 
   <td>1</td> 
   <td>2</td> 
   <td>3 or 4</td> 
  </tr> 
  <tr> 
   <td>Max cpus/node requestable</td> 
   <td>4</td> 
   <td>8</td> 
   <td>16</td> 
  </tr> 
  <tr> 
   <td>Max RAM/node requestable</td> 
   <td>16 GB</td> 
   <td>32 GB</td> 
   <td>64 GB</td> 
  </tr> 
  <tr> 
   <td>Max RAM per cpu</td> 
   <td>4 GB</td> 
   <td>4 GB</td> 
   <td>4 GB</td> 
  </tr> 
  <tr> 
   <td>Max Job time</td> 
   <td colspan="3">2-00:00:00</td> 
  </tr> 
  <tr> 
   <td>Cost</td> 
   <td colspan="3">8 SU per GPU hour</td> 
  </tr> 
 </tbody> 
</table>

Jobs submitted to the `gpu` partition will also be considered for routing to the `volta` partition, if the resource requirements allow for it.

If you wish to specifically request K80 GPUs for your job, you can do so using the **kepler** gres subtype. For example, to request 2 **kepler** GPUs, place the following line in your job script:

#SBATCH --gres=gpu:kepler:2

### volta partition limits\[[edit][edit 10] | [edit source][edit source 10]\] ###

The `volta` partition targets the Phoenix nodes that have Nvidia V100 accelerators. This partition is suitable for all computational jobs which require up to 2 GPU accelerators per node. In particular, the Nvidia V100 cards have a number of tensor cores that are optimised for deep learning workloads.

For the Nvidia V100s, the cost for one GPU-hour is 32 service units (SU). The minimum number of cores per GPU is 2, and the maximum is 8. There is a maximum of 2 V100 GPUs that can be used per node.

<table> 
 <tbody> 
  <tr> 
   <td colspan="2"><strong>volta job constraints</strong></td> 
  </tr> 
  <tr> 
   <td>Max Job time</td> 
   <td>2-00:00:00</td> 
  </tr> 
  <tr> 
   <td>Max cpus/node</td> 
   <td>16</td> 
  </tr> 
  <tr> 
   <td>Max RAM per cpu</td> 
   <td>9665 MB</td> 
  </tr> 
  <tr> 
   <td>Max RAM per node</td> 
   <td>151 GB</td> 
  </tr> 
  <tr> 
   <td>Cost</td> 
   <td>32 SU per GPU hour</td> 
  </tr> 
  <tr> 
   <td>Can use GPUs</td> 
   <td><strong>Yes</strong></td> 
  </tr> 
 </tbody> 
</table>

The resources available for jobs that are assigned to the `volta` partition depend on the number of GPUs requested. The scheduler controls these limits by **automatically** assigning each job one of two QoS flags, named `vxs, vxm`. These flags are designed to optimise the overall system utilisation, with each providing different core and memory limits based on the number of GPUs required. The following table summarises the constrains.

<table> 
 <tbody> 
  <tr> 
   <td><strong>QoS</strong></td> 
   <td><strong>vxs</strong></td> 
   <td><strong>vxm</strong></td> 
  </tr> 
  <tr> 
   <td>GPUs/node</td> 
   <td>1</td> 
   <td>2</td> 
  </tr> 
  <tr> 
   <td>Max cpus/node requestable</td> 
   <td>8</td> 
   <td>16</td> 
  </tr> 
  <tr> 
   <td>Max RAM/node requestable</td> 
   <td>76 GB</td> 
   <td>152 GB</td> 
  </tr> 
  <tr> 
   <td>Max RAM per cpu</td> 
   <td>9.5 GB</td> 
   <td>9.5 GB</td> 
  </tr> 
  <tr> 
   <td>Max Job time</td> 
   <td colspan="2">2-00:00:00</td> 
  </tr> 
  <tr> 
   <td>Cost</td> 
   <td colspan="2">32 SU per GPU hour</td> 
  </tr> 
 </tbody> 
</table>

Jobs submitted to the `gpu` partition will also be considered for routing to the `volta` partition, if the resource requirements allow for it.

If you wish to specifically request V100 GPUs for your job, you can do so using the **volta** gres subtype. For example, to request 2 **volta** GPUs, place the following line in your job script:

#SBATCH --gres=gpu:volta:2

## The `test` and `skytest` partitions\[[edit][edit 11] | [edit source][edit source 11]\] ##

These partitions are meant for job testing. Jobs submitted to these partitions have elevated priority to ensure a quick start time. There are strict limits on the wall time and the number of cpus a test job can run on. It is also **three times** more expensive to run through the job testing partitions. Furthermore, users with a fair share factor of less than 0.25 cannot run using these queues until their fair share rises above the threshold.

### test partition limits\[[edit][edit 12] | [edit source][edit source 12]\] ###

The `test` partition targets Phoenix nodes which have Haswell cores and K80 GPUs.

<table> 
 <tbody> 
  <tr> 
   <td colspan="2"><strong>test job constraints</strong></td> 
  </tr> 
  <tr> 
   <td>Max Job time</td> 
   <td>02:00:00</td> 
  </tr> 
  <tr> 
   <td>Max RAM per cpu</td> 
   <td>4 GB</td> 
  </tr> 
  <tr> 
   <td>Max RAM per node</td> 
   <td>64 GB</td> 
  </tr> 
  <tr> 
   <td>Cost</td> 
   <td>3 SU per CPU hour, 24 SU per GPU hour</td> 
  </tr> 
  <tr> 
   <td>Max number of cpus</td> 
   <td>16</td> 
  </tr> 
  <tr> 
   <td>Max number of GPUs</td> 
   <td>4</td> 
  </tr> 
  <tr> 
   <td>Can use GPUs</td> 
   <td><strong>Yes</strong></td> 
  </tr> 
 </tbody> 
</table>

### skytest partition limits\[[edit][edit 13] | [edit source][edit source 13]\] ###

The `skytest` partition targets Phoenix nodes which have Skylake cores and V100 GPUs.

<table> 
 <tbody> 
  <tr> 
   <td colspan="2"><strong>skytest job constraints</strong></td> 
  </tr> 
  <tr> 
   <td>Max Job time</td> 
   <td>02:00:00</td> 
  </tr> 
  <tr> 
   <td>Max RAM per cpu</td> 
   <td>9665 MB</td> 
  </tr> 
  <tr> 
   <td>Max RAM per node</td> 
   <td>190 GB</td> 
  </tr> 
  <tr> 
   <td>Cost</td> 
   <td>3.75 SU per CPU hour, 96 SU per GPU hour</td> 
  </tr> 
  <tr> 
   <td>Max number of cpus</td> 
   <td>20</td> 
  </tr> 
  <tr> 
   <td>Max number of GPUs</td> 
   <td>2</td> 
  </tr> 
  <tr> 
   <td>Can use GPUs</td> 
   <td><strong>Yes</strong></td> 
  </tr> 
 </tbody> 
</table>

## The `highmem` partition\[[edit][edit 14] | [edit source][edit source 14]\] ##

This partition is meant for jobs that have high memory requirements. The Phoenix cluster currently holds 3 nodes with 512 GB of RAM and 3 nodes with 1.5 TB of RAM. Demand for these nodes can fluctuate. Make sure to check your memory requirements thoroughly before submitting to the highmem partition to avoid unnecessary queuing times.

### highmem partition limits\[[edit][edit 15] | [edit source][edit source 15]\] ###

The `highmem` partition targets Phoenix nodes which have Haswell cores and a minimum of 512 GB of RAM.

<table> 
 <tbody> 
  <tr> 
   <td colspan="2"><strong>highmem job constraints</strong></td> 
  </tr> 
  <tr> 
   <td>Max Job time</td> 
   <td>3-00:00:00</td> 
  </tr> 
  <tr> 
   <td>Max RAM per cpu</td> 
   <td>16 GB or 48 GB</td> 
  </tr> 
  <tr> 
   <td>Max RAM per node</td> 
   <td>503 GB or 1511 GB</td> 
  </tr> 
  <tr> 
   <td>Cost</td> 
   <td>1 SU per CPU hour</td> 
  </tr> 
  <tr> 
   <td>Can use GPUs</td> 
   <td><strong>No</strong></td> 
  </tr> 
 </tbody> 
</table>

[hide]: https://wiki.adelaide.edu.au/hpc/Partitions_and_QoSs#
[1How to submit to a partition _queue]: https://wiki.adelaide.edu.au/hpc/Partitions_and_QoSs#How_to_submit_to_a_partition_.28queue.29
[2Partitions]: https://wiki.adelaide.edu.au/hpc/Partitions_and_QoSs#Partitions
[2.1The batch partition]: https://wiki.adelaide.edu.au/hpc/Partitions_and_QoSs#The_batch_partition
[2.2The cpu_ skylake and skypool partitions]: https://wiki.adelaide.edu.au/hpc/Partitions_and_QoSs#The_cpu.2C_skylake_and_skypool_partitions
[2.2.1cpu partition limits]: https://wiki.adelaide.edu.au/hpc/Partitions_and_QoSs#cpu_partition_limits
[2.2.2skylake partition limits]: https://wiki.adelaide.edu.au/hpc/Partitions_and_QoSs#skylake_partition_limits
[2.2.3skypool partition limits]: https://wiki.adelaide.edu.au/hpc/Partitions_and_QoSs#skypool_partition_limits
[2.2.4Long QoS]: https://wiki.adelaide.edu.au/hpc/Partitions_and_QoSs#Long_QoS
[2.3The gpu and volta partitions]: https://wiki.adelaide.edu.au/hpc/Partitions_and_QoSs#The_gpu_and_volta_partitions
[2.3.1gpu partition limits]: https://wiki.adelaide.edu.au/hpc/Partitions_and_QoSs#gpu_partition_limits
[2.3.2volta partition limits]: https://wiki.adelaide.edu.au/hpc/Partitions_and_QoSs#volta_partition_limits
[2.4The test and skytest partitions]: https://wiki.adelaide.edu.au/hpc/Partitions_and_QoSs#The_test_and_skytest_partitions
[2.4.1test partition limits]: https://wiki.adelaide.edu.au/hpc/Partitions_and_QoSs#test_partition_limits
[2.4.2skytest partition limits]: https://wiki.adelaide.edu.au/hpc/Partitions_and_QoSs#skytest_partition_limits
[2.5The highmem partition]: https://wiki.adelaide.edu.au/hpc/Partitions_and_QoSs#The_highmem_partition
[2.5.1highmem partition limits]: https://wiki.adelaide.edu.au/hpc/Partitions_and_QoSs#highmem_partition_limits
[edit]: https://wiki.adelaide.edu.au/hpc/index.php?title=Partitions_and_QoSs&veaction=edit&vesection=1
[edit source]: https://wiki.adelaide.edu.au/hpc/index.php?title=Partitions_and_QoSs&action=edit&section=1
[edit 1]: https://wiki.adelaide.edu.au/hpc/index.php?title=Partitions_and_QoSs&veaction=edit&vesection=2
[edit source 1]: https://wiki.adelaide.edu.au/hpc/index.php?title=Partitions_and_QoSs&action=edit&section=2
[edit 2]: https://wiki.adelaide.edu.au/hpc/index.php?title=Partitions_and_QoSs&veaction=edit&vesection=3
[edit source 2]: https://wiki.adelaide.edu.au/hpc/index.php?title=Partitions_and_QoSs&action=edit&section=3
[edit 3]: https://wiki.adelaide.edu.au/hpc/index.php?title=Partitions_and_QoSs&veaction=edit&vesection=4
[edit source 3]: https://wiki.adelaide.edu.au/hpc/index.php?title=Partitions_and_QoSs&action=edit&section=4
[edit 4]: https://wiki.adelaide.edu.au/hpc/index.php?title=Partitions_and_QoSs&veaction=edit&vesection=5
[edit source 4]: https://wiki.adelaide.edu.au/hpc/index.php?title=Partitions_and_QoSs&action=edit&section=5
[edit 5]: https://wiki.adelaide.edu.au/hpc/index.php?title=Partitions_and_QoSs&veaction=edit&vesection=6
[edit source 5]: https://wiki.adelaide.edu.au/hpc/index.php?title=Partitions_and_QoSs&action=edit&section=6
[edit 6]: https://wiki.adelaide.edu.au/hpc/index.php?title=Partitions_and_QoSs&veaction=edit&vesection=7
[edit source 6]: https://wiki.adelaide.edu.au/hpc/index.php?title=Partitions_and_QoSs&action=edit&section=7
[edit 7]: https://wiki.adelaide.edu.au/hpc/index.php?title=Partitions_and_QoSs&veaction=edit&vesection=8
[edit source 7]: https://wiki.adelaide.edu.au/hpc/index.php?title=Partitions_and_QoSs&action=edit&section=8
[edit 8]: https://wiki.adelaide.edu.au/hpc/index.php?title=Partitions_and_QoSs&veaction=edit&vesection=9
[edit source 8]: https://wiki.adelaide.edu.au/hpc/index.php?title=Partitions_and_QoSs&action=edit&section=9
[edit 9]: https://wiki.adelaide.edu.au/hpc/index.php?title=Partitions_and_QoSs&veaction=edit&vesection=10
[edit source 9]: https://wiki.adelaide.edu.au/hpc/index.php?title=Partitions_and_QoSs&action=edit&section=10
[edit 10]: https://wiki.adelaide.edu.au/hpc/index.php?title=Partitions_and_QoSs&veaction=edit&vesection=11
[edit source 10]: https://wiki.adelaide.edu.au/hpc/index.php?title=Partitions_and_QoSs&action=edit&section=11
[edit 11]: https://wiki.adelaide.edu.au/hpc/index.php?title=Partitions_and_QoSs&veaction=edit&vesection=12
[edit source 11]: https://wiki.adelaide.edu.au/hpc/index.php?title=Partitions_and_QoSs&action=edit&section=12
[edit 12]: https://wiki.adelaide.edu.au/hpc/index.php?title=Partitions_and_QoSs&veaction=edit&vesection=13
[edit source 12]: https://wiki.adelaide.edu.au/hpc/index.php?title=Partitions_and_QoSs&action=edit&section=13
[edit 13]: https://wiki.adelaide.edu.au/hpc/index.php?title=Partitions_and_QoSs&veaction=edit&vesection=14
[edit source 13]: https://wiki.adelaide.edu.au/hpc/index.php?title=Partitions_and_QoSs&action=edit&section=14
[edit 14]: https://wiki.adelaide.edu.au/hpc/index.php?title=Partitions_and_QoSs&veaction=edit&vesection=15
[edit source 14]: https://wiki.adelaide.edu.au/hpc/index.php?title=Partitions_and_QoSs&action=edit&section=15
[edit 15]: https://wiki.adelaide.edu.au/hpc/index.php?title=Partitions_and_QoSs&veaction=edit&vesection=16
[edit source 15]: https://wiki.adelaide.edu.au/hpc/index.php?title=Partitions_and_QoSs&action=edit&section=16