Phoenix Getting Started Guide
This guide should help you getting acquainted with the Adelaide high performance computing (HPC) facility Phoenix. The workflow you will find here is slightly different to what you are probably used to on your personal computer. In summary, the workflow can be separated into a couple of steps:
- connecting to a login node (sometimes called head node, a computer whose sole purpose is to handle user traffic from and to phoenix);
- uploading/editing an input file, preparing a 'compute job';
- submitting a job to the job queue
- retrieving/downloading the results
Contents
[hide]
Connecting to Phoenix[edit | edit source]
Please follow this guide to connect to Phoenix.
After logging in, it is usually advisable to change the default tcsh to bash. To change the shell environment permanently, use the following command to assign /bin/bash to $SHELL.
rcshell -s /bin/bash
Transferring files to Phoenix[edit | edit source]
Before you run your application, you will need to upload your data and/or program code from your computer to your $FAST directory on the Phoenix system. Fortunately, this step is very easy and there are a number of ways to do this.
Interacting with the Phoenix System: Mastering Linux Basics[edit | edit source]
Like most HPC systems, Phoenix uses Linux as its operating system. Interacting with the system is easier if you are familiar with the basics of the Linux command line interface. As a starting point to learn about Linux basics and discover the most useful commands for using Phoenix, you can refer to our Linux for HPC guide.
Loading software packages[edit | edit source]
Software packages on Phoenix are organised in modules, some of the currently available packages and documentations can be found here.
In most cases, your required software is not loaded by default on the Phoenix system. After logging in, you will need to load your required software before you can perform any calculations. Phoenix uses the module system to manage the software environment. To see a list of the available software, use the module avail command as in the example below. If you can not find your required software under this list, there is a good chance we can make it available to you, contact us via email to make a software installation request.
$ module avail ----------------------------------------------------- /usr/share/Modules/modulefiles ------------------------------------------------------ dot module-git module-info modules null use.own ------------------------------------------------------------ /etc/modulefiles ------------------------------------------------------------- cuda/6.0 cuda/7.0 gnu-parallel/20150322 intelmpi/5.0.3.048 openmpi/gnu/1.8.4 subread/1.4.6-p2 cuda/6.5 gcc/4.8.4 intel/13.1.3.174 matlab/2014a openmpi/intel/1.8.1
To load a software package, use the module load command:
$ module load cuda/6.5
To unload a software package, use the module unload command:
$ module unload gcc/4.8.4
To swap between software packages, use the module swap command:
$ module swap cuda/6.5 cuda/7.0
Preparing to submit a job[edit | edit source]
There are two components required to submit a job for analysis in the Phoenix system.
- The software you wish to run (and any associated data files)
- A job script that requests system resources
To ensure a fair and optimized system for all Phoenix users, we use a resource management tool, SLURM, for job scheduling. In order to submit a job to Phoenix, you must create a SLURM job script and save this along with the program files, into your directory folder on Phoenix. Below are examples of sample job scripts called <my_job.sh> for each of the two common job types, namely simple jobs and parallel (MPI) jobs. For each job type, a downloadable version is provided for you to use. Please configure your job script according to one of following that best suits your requirements.
Creating a simple job script[edit | edit source]
A job script is a text file that specifies the resources your code needs to run. The job scheduler then uses these resources to determine when to to run your job. Let's have a look at a simple job script example for some sequential code (that only runs on 1 CPU core):
#!/bin/bash #SBATCH -p batch # partition (this is the queue your job will be added to) #SBATCH -N 1 # number of nodes (use a single node) #SBATCH -n 1 # number of cores (sequential job uses 1 core) #SBATCH --time=01:00:00 # time allocation, which has the format (D-HH:MM:SS), here set to 1 hour #SBATCH --mem=4GB # specify memory required per node (here set to 4 GB) # Executing script (Example here is sequential script) ./my_sequential_program # your software with any arguments
We'll begin by explaining the purpose of each line of the script example:
- The header line
#!/bin/bashsimply tells the scheduler which shell language is going to be used to interpret the script. The default shell on Phoenix is bash. - The next set of lines all begin with the prefix
#SBATCH. This prefix is used to indicate that we are specifying a resource request for the scheduler.- The scheduler divides the cluster workload into partitions (work queues). Different partitions are used for different types of compute job. Each compute job must select a partition with the -p option. To learn more about the different partitions available on Phoenix, see <reference>.
- The Phoenix cluster is a collection of compute nodes, where each node has multiple CPU cores. Each job must specify the CPU resources required by using the -N option to request nodes and the -n to request the number of cores per node required. See <reference>
- Each compute job needs to specify an estimate of the amount of time it needs to complete. This is commonly referred to as the walltime, specified with the --time=HH:MM:SS option. The estimated walltime needs to be larger than the actual time needed by the job, otherwise the scheduler will terminate the job for exceeding its requested time.
- Dedicated memory (RAM) is allocated for each job when it runs, and the amount of memory required per node must be specified with the --mem option.
A simple job is one in which the computational process is sequential and is carried out by a single node. (Note: If your program file does not use MPI or MPI enabled libraries, your job belongs to this category.) Depending on your computational needs, you may need to use either CPU or GPU-accelerated computing nodes. This post provides some insights about the differences between CPU and GPU-accelerated computing. If you need further support with this, please contact the team to discuss.
Below is a sample job script for simple CPU jobs. You will need to create an .sh file in your directory on Phoenix, and can copy and paste the script below into that file. Please remember you must then configure the job script to your needs. The most common fields that need modification are the number of nodes and cores you wish to use, the duration of time for which you wish to run the job, and the email address to which notifications should be sent (i.e. your email address).
#!/bin/bash #SBATCH -p batch # partition (this is the queue your job will be added to) #SBATCH -N 1 # number of nodes (due to the nature of sequential processing, here uses single node) #SBATCH -n 2 # number of cores (here uses 2) #SBATCH --time=01:00:00 # time allocation, which has the format (D-HH:MM), here set to 1 hour #SBATCH --mem=4GB # specify memory required per node (here set to 4 GB) # Notification configuration #SBATCH --mail-type=END # Type of email notifications will be sent (here set to END, which means an email will be sent when the job is done) #SBATCH --mail-type=FAIL # Type of email notifications will be sent (here set to FAIL, which means an email will be sent when the job is fail to complete) #SBATCH [email protected] # Email to which notification will be sent # Executing script (Example here is sequential script and you have to select suitable compiler for your case.) bash ./my_program.sh # bash script used here for demonstration purpose, you should select proper compiler for your needs
For simple GPU jobs, the following example job script can be copied and pasted into a new .sh file in your Phoenix directory:
#!/bin/bash # Configure the resources required #SBATCH -p batch # partition (this is the queue your job will be added to) #SBATCH -n 1 # number of tasks (sequential job starts 1 task) (check this if your job unexpectedly uses 2 nodes) #SBATCH -c 8 # number of cores (sequential job calls a multi-thread program that uses 8 cores) #SBATCH --time=01:00:00 # time allocation, which has the format (D-HH:MM), here set to 1 hour #SBATCH --gres=gpu:4 # generic resource required (here requires 4 GPUs) #SBATCH --mem=16GB # specify memory required per node (here set to 16 GB) # Configure notifications #SBATCH --mail-type=END # Type of email notifications will be sent (here set to END, which means an email will be sent when the job is done) #SBATCH --mail-type=FAIL # Type of email notifications will be sent (here set to FAIL, which means an email will be sent when the job is fail to complete) #SBATCH [email protected] # Email to which notification will be sent # Execute your script (due to sequential nature, please select proper compiler as your script corresponds to) bash ./my_program.sh # bash script used here for demonstration purpose, you should select proper compiler for your needs
Creating a MPI Job Script[edit | edit source]
A parallel (MPI) job is one that harnesses the computational power of multiple nodes, which are networked together and will perform related calculations or processes simultaneously. This can allow highly complex computational processes to be performed in much shorter time-frames. To enable parallel computing, the program you use will need to be MPI enabled or incorporate MPI enabled library. If you do need to run a parallel job on CPUs, following job script is an example:
#!/bin/bash #SBATCH -p batch # partition (this is the queue your job will be added to) #SBATCH -N 2 # number of nodes (here uses 2) #SBATCH -n 64 # number of cores (here 64 cores requested) #SBATCH --time=01:00:00 # time allocation, which has the format (D-HH:MM), here set to 1 hour #SBATCH --mem=32GB # specify memory required per node (here set to 32 GB) mpirun -np 64 ./my_program
For jobs that use GPU accelerators, <my_job.sh> will look like something like the example below:
#!/bin/bash #SBATCH -p batch # partition (this is the queue your job will be added to) #SBATCH -n 2 # number of cores (here 2 cores requested) #SBATCH --time=01:00:00 # time allocation, which has the format (D-HH:MM), here set to 1 hour #SBATCH --gres=gpu:1 # generic resource required (here requires 1 GPU) #SBATCH --mem=8GB # specify memory required per node (here set to 8 GB) mpirun -np 8 ./my_program
More commonly used options, which can be added to #SBATCH lines includes
#SBATCH --mail-type=END # Type of email notifications will be sent (here set to END, which means an email will be sent when the job is done) #SBATCH --mail-type=FAIL # Type of email notifications will be sent (here set to FAIL, which means an email will be sent when the job is fail to complete) #SBATCH [email protected] # Email to which notification will be sent
SLURM is very powerful and allows detailed tailoring to fit your specific needs. If you want to explore all available SLURM parameters available, simply type following in the command line:
man sbatch
Debiting compute time: Multiple associations[edit | edit source]
The -A association argument specifies the association from which you wish the compute time to be be debited. Note that you normally only need to specify an association if you have access to multiple allocations, otherwise the scheduler will debit the resources used from your default association.
HPC user guides - Selecting a Job Queue[edit | edit source]
Standard compute jobs should select the batch queue. For further details and examples, please refer to the job requirement guide.
Submitting A Job[edit | edit source]
To submit a job script to the queue use the sbatch command:
$ sbatch my_job.sh
If your job script requires additional variables you can define these with the --export option to sbatch:
$ sbatch --export=ALL,var1=val1,var2=val2 my_job.sh
Be sure to include the ALL option to --export to ensure your job runs correctly.
Monitoring The Queue[edit | edit source]
You can view your job's progress through the queue with the squeue command:
$ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 2916 batch my_job. a1234567 PD 0:00 1 (Resources) 2915 batch my_job. a1234567 R 01:03 2 phoenixcs[1-2] 2914 batch my_job. a1234567 R 00:21 1 phoenixg1
The fifth column gives the status of the job as follows: R - running, PD - Pending, F - Failed, ST - Stopped, TO - Timeout
Running squeue without arguments will list all currently running jobs. However if the list displayed is too long for you to easily locate your job, you can limit the search to your own jobs by using -u argument like this
squeue -u aXXXXXXX
where aXXXXXXX is your UofA ID number
Canceling a Job[edit | edit source]
To cancel a job you own, use the sbatch command followed by the slurm job ID:
$ scancel 2914
To cancel all jobs you own in a particular queue, use the -p argument:
$ scancel -p batch