RunAI
For this cluster, with the oversight of the RCS Technical Advisory Group, we have chosen to use RunAI for GPU orchestration and scheduling. This is a Kubernetes based platform designed specifically for organisations running lots of AI workloads.
Below are some examples on getting started with Run:AI that we have based on the documentation from NVidia at https://run-ai-docs.nvidia.com/self-hosted/getting-started/quick-starts/ai-practitioner-quick-start.
Weka Storage
HX3 has its own dedicated high performance file system utilising the Weka file system running on Lenovo hardware. This dedicated storage ensures the high loads generated by HX3 do not adversely affect the RDS, RDF-Active or similar systems and, problems with the RDS do not cause job failures on HX3. This means data must be copied to the cluster before it can be processed, and any important results should be copied off and saved on a dedicated storage platform like the RDF-Active.
Create persistent storage
Data on HeX3 should be stored in a persistent volume or PVC. These can be mounted inside different containers, allowing data to be moved around.
To create a new PVC:
- go to Workload manager → Data sources.
- Click +NEW DATA SOURCE and select PVC from the dropdown menu.
- Within the new form, set the scope. This should be your department or lower.
Warning
The scope of this sets the access level and should be set to most specific as possible.
- Enter a name for the data source. The name must be unique.
- For the data options:
- select New PVC
- storage class is "storageclass-wekafs-dir-api
- To allow all nodes to read and write from/to the PVC, select Read-write by many nodes for the access mode.
- Input how much storage is needed, for example 1 TB
- Select Filesystem (default) as the volume mode. The volume will be mounted as a filesystem, enabling the usage of directories and files.
- Set the Container path, this should make sense to you and maybe related to the name of the PVC. For example, /myusername-home which is where the PVC will be mounted inside containers.
- Click CREATE DATA SOURCE
Create a workspace job
A workspace job is best used for interactive work were you need to test and develop your code
To create a new Jupyterhub workspace:
- Go to the Workload manager → Workloads
- Click +NEW WORKLOAD and select Workspace
- The hx3 cluster should be auto-selected
- Select the project in which your workspace will run. This should have been provided by your RCS Associate or your project lead. Only create a new project if told to do so
- If you have not already created a template select "Start from scratch", otherwise select your template.
- Give the workspace a useful name, if the name already exists in the project, you will be requested to submit a different name.
- Click CONTINUE
If you need to create the template you can do it now:
- Under Environment, click the load icon.
- A side pane appears, displaying a list of available environments. Select the ‘jupyter-scipy’ environment for your workspace (Image URL: jupyter/scipy-notebook)
- Under Compute resources, click the load icon. A side pane appears, displaying a list of available compute resources. Select the ‘one-gpu’ compute resource for your workload.
- If you created a PVC as above you can mount it now:
- Expand "Data & storage"
- Click the load icon
- The PVCs you have access to should appear in a side panel, select the required and clock Load.
- Scroll back up to "Security"
- Click Custom
- Under "Group ID (GID)" enter 1000
- You can now click CREATE WORKSPACE
You should be returned to the Workloads overview page and your workload should be in the "starting" or "initializing" state. Once this changes to "Running": * Click on the URL under "Connections(s)" * In the pop up window scroll to the right and click the connect icon.
You should now have a Jupyterhub session with 1 GPU and access to your mounted PVC.