HeX2

HX2 (or Hex 2) is the new high throughput Cluster at Imperial designed for the running a large number of jobs or smaller scale AI workflows.

Cluster Specification

HX2 has 4 racks of CPU compute giving a total of 27,648 CPUs and 1 rack of GPU for a total of 96 NVidia H200 GPUs.

CPU node specs

2x Intel(R) Xeon(R) 6960P 2 TB RAM 1x 200GbE

GPU node spec

4x NVIDIA H200 with NVlink 2x INTEL(R) XEON(R) PLATINUM 8562Y+ 1.5 TB RAM 2x 200GbE

Please note that CPUs 1,2 on CPU nodes and CPUs 1,2,62,63 on GPU nodes are assigned to run the WEKA storage and can not be used for jobs.

Access

Access Control Settings

Access to HeX2 is handled on ReCAP, where RCS grants users cluster compute allocations to research groups once requested. To request a HeX2 compute allocation, contact your supervisor or research group manager.

Connecting to HX2

There are three login nodes for HX2 which are accessible over ssh. These can be accessed using the login.hx2.hpc.ic.ac.uk hostname. If you see a message asking you to confirm the host key, the fingerprint given in the message should match one of those given below:

• SHA256:TrZpzQanCpxfLfnnLMJJw5TATsYK3kZq//NUrNXEg6w (ED25519)

Do note that due to security concerns, key-based authentication is disabled for the login-nodes. Users will need to login using their college username and password.

Storage

Weka

HX2 has its own dedicated high performance file system utilising the Weka running on Lenovo hardware. This dedicated storage ensures the high loads generated by HX2 do not adversely affect the RDS or similar systems and, problems with the RDS do not cause job failures on HX2. This means that you must move data to your HX2 home directory before running jobs (sftp/rsync and other similar tools can be used to transfer data from the RDS to HX1 during this pilot phase). Please read the following sections for more information.

Home Directory

When you login to HX2, you will have a home directory automatically created for you on the file system local to HX2. User home directories on HX2 are intended to provide working space for current jobs only and it is the expectation that users will move their data to other systems once the data is no longer needed on the cluster. Accordingly, there are NO BACKUP, DISASTER RECOVERY OR SNAPSHOTS for HX2, and in the event of major hardware failure, accidental data deletion, file system corruption, etc. the data will be lost. It is therefore imperitive that you copy any important files to another storage system such as RDF Active once they have been generated.

Quota

Your home directory on HX2 is subject to a default quota of 1 TB and 2 million files/directories (inodes).

Quota increases are possible on request if justified. We ask that users make efforts to:

• Minimise the amount of data that needs to be stored for live projects.
• Avoid having large numbers of files within their home directory.

Users who request a quota increase but are storing many unused files in their home directory will be asked to move or remove these files first before a quota increase is granted. Any quota increases above the default values will be re-assessed every 6 months and users may be asked to provide an updated justification for their quota level.

Retention of Data

The HX2 file system is meant for live data only and any important files should be copied elsewhere (including to the RDS) after being generated. Minimising the amount of data stored on the HX2 filesystem ensures that it maintains the high performance we need it for. For these reasons, RCS staff will be undertaking the following steps to ensure that the file system is only used for live data:

• If RCS staff believe that a user is storing unused data in their home directory, then they may be contacted and asked to move the data to another storage space if justification cannot be provided.
• If RCS staff believe an account on HX2 has been unused for 6+ months, then we will contact the user and ask them to clean up their home directory. If we have not received a response from the user or registered supervisor within 6 weeks, then the RCS staff reserve the right to remove the data from the home directory.
• If a user leaves the university and their account becomes deactivated, then the data in that home account will be removed promptly after the user has left. It is the responsibility of the user to ensure that a copy of important data exists elsewhere before they leave the university.

Shared Project Areas

We are still working on a solution for shared project areas, however we welcome requests for these from either HPC or RDS Project Admins. There will be no charge for these shared spaces but they are for live data sharing only and any quota must be justified (with a review every 6 months). As with home directories, data within these shared areas that have not been accessed for some time are at risk of being deleted.

Software

This section will explain how to access software that has been centrally installed on HX2.

Please make sure any software you run on HX2 has been optimised for the hardware. Ideally use software provided by us (which has already been optimised) or otherwise please make sure to use relevant optimisation flags when compiling your own software. Please avoid simply copying binaries from other systems such as CX3, unless the software is commercial and/or only the binary is available.

Loading Applications

Loading modules/applications on HX2 is similar to that on CX3 except it is not necessary to load the "production" modules (tools/prod). Please refer to our main Loading Applications page for advice on how to load modules on HX1.

EasyBuild

Most of the software installed on HX2 is done so using the EasyBuild software installation system and will have been optimised for the hardware. Please see our EasyBuild page for more information.

Python and Conda Environments

Installing miniforge

Miniforge is a minimal installer for conda that is tailored to use the conda-forge channel by default. It installs both the conda and mamba tools for managing your environments.

The following instructions are based on those found on the Miniforge repository by have been adapted for use on HX1.

[username@hx2a04login01 ~]$ curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 89.3M  100 89.3M    0     0  93.1M      0 --:--:-- --:--:-- --:--:--  194M

You should end up with a file called Miniforge3-Linux-x86_64.sh. You can then run the installer with:

[username@hx2a04login01 ~]$ bash Miniforge3-Linux-x86_64.sh

Welcome to Miniforge3 25.3.0-3

In order to continue the installation process, please review the license
agreement.
Please, press ENTER to continue
>>>

After accepting the license, you will be asked to confirm where to install Miniforge3. The default location of miniforge3 in your home directory is fine for most circumstances.

Once the files have finished unpacking, you will be asked:

Do you wish to update your shell profile to automatically initialize conda?
This will activate conda on startup and change the command prompt when activated.
If you'd prefer that conda's base environment not be activated on startup,
   run the following command when conda is activated:

conda config --set auto_activate_base false

You can undo this by running `conda init --reverse $SHELL`? [yes|no]
[no] >>>

We strongly advise that you do not update your shell profile i.e. respond with "no".

You can now enable miniforge in your environment by running the shell hook:

[username@hx2a04login01 ~]$ eval "$(/hx2-weka/home/username/miniforge3/bin/conda shell.bash hook)"
(base) [username@hx2a04login01 ~]$

You can then use the conda commands as usual. Please see our Conda page for more information on using conda.

Job Submission

Job submission on HX2 differs slightly, as it uses the SLURM Workload Manager as its batch scheduler instead of PBS Pro. Slurm is already widely used at many central HPC facilities across UK universities.

Key commands

Slurm Command	PBS Pro equivalent	Description
sbatch	qsub	Submit a job script to the queue
squeue	qstat	Report the state of jobs in the queue
scancel	qdel	Cancel a job in the queue

Basic job script

The following shows an example of a basic job script for slurm. It will ask for 1 cpu (based on 1 task x 1 cpus-per-task) for 5 minutes, with a total of 1 GB of RAM. When the job starts it will run ./my_program.

#!/bin/bash
#SBATCH --job-name=test_job
#SBATCH --time=00:05:00
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=1G

./my_program

Submitting Jobs

A submission script can be submitted to the queue using the sbatch command. As an example:

$ sbatch run_my_program.slurm
Submitted batch job 12345

When the submission script has been successfully accepted by the queue, the job number will be displayed as an output to sbatch.

Job Sizing Guidance

The final partition/queue layout is still being developed. Note that on standard compute nodes, jobs are limited to 142 out of the 144 available cores; 2 cores are set aside to support the Weka file system and operating system. This is increased to 4 cores on GPU nodes, so a maximum of 60 cores can be requested.

single core jobs

A single core submission script would look like the following:

#!/bin/bash
#SBATCH --job-name=test_job
#SBATCH --time=00:05:00
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=1

./my_program

Multi-threaded jobs (OpenMP/shared memory)

If the program uses threads on a single node:

#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=8

What this means:

• --nodes=1 -> one node
• --ntasks-per-node=1 -> one process per node
• --cpus-per-task=8 -> that process can use 8 CPU cores (threads)

You may also need to set:

export OMP_NUM_THREADS=8

or

export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}

To control threading within the OpenMP program.

MPI or distributed tasks

If your job supports multiple independent process (such as MPI):

#SBATCH --nodes=1
#SBATCH --ntasks-per-node=8
#SBATCH --cpus-per-task=1

What this means:

• Slurm launches 8 processes
• Each process gets 1 CPU core

You would launch your processes with:

srun ./my_mpi_program

or

mpirun ./my_mpi_program

In both cases, the mpi distribution must be suitably configured to work with Slurm (the centrally provided MPI distributions are already configured with this integration).

Hybrid MPI and threading

If each MPI rank uses multiple threads:

#SBATCH --nodes=1
#SBATCH --ntasks-per-node=4
#SBATCH --cpus-per-task=4

This means:

• 4 MPI processes are started
• Each MPI process gets 4 CPU cores (threads)
• a total of 16 CPU cores are allocated to the job

As with the multi-threaded job, you may need to set:

export OMP_NUM_THREADS=4

or

export OMP_NUM_THREADS=${SLURM_CPUS_PER_TASK}

gpu/h200

There is an additional limit of 12 GPU's total per user on the a100 queue to allow for fair usage of the GPUs.

Example Slurm Jobs

MPI Jobs

Hybrid OpenMP/MPI Jobs

MPI Distribution Specific Information

Intel MPI

OpenMPI

GPU Jobs

GPU Specification

Multi-node GPU Jobs

Example GPU Jobs

MPI GPU Jobs (single node)

Known Issues