1 How do I prepare for the deployment? #

When GPU resources are available to Kubernetes, applications can request them by defining resource requests and limits. For NVIDIA GPUs, the GPU Operator typically makes the GPU resource available as nvidia.com/gpu.

For example, add the following configuration to the application’s values.yaml file:

resources:
  requests:
    nvidia.com/gpu: 1
  limits:
    nvidia.com/gpu: 1

Specify the number of GPUs according to the requirements of the application and the available cluster resources.

When the workload is scheduled, Kubernetes schedules it on a node where the requested GPU resources are available and allocates the requested resources to the workload.

In some environments, you might want to restrict an application to specific GPU nodes. For example, this can be useful when a cluster contains nodes with different GPU models or when GPU nodes are reserved for particular workloads.

You can use Kubernetes labels and node selectors to control where the workload is scheduled.

For example, label a node:

> kubectl label node <GPU_NODE_NAME> accelerator=nvidia-gpu

Then configure the application to select nodes with that label:

nodeSelector:
  accelerator: nvidia-gpu

Node labels and selectors are optional. They are not required for Kubernetes to allocate GPU resources.

The method used to verify GPU usage depends on the application.

First, verify that Kubernetes scheduled the workload successfully on a node that provides the requested GPU resources. You can inspect the workload and its assigned node by using kubectl. For example:

> kubectl describe pod <POD_NAME>

After the workload is running, use application-specific information to verify that the application detected and is using the assigned GPU. Depending on the application, this information might be available through logs, metrics, or the user interface.

For example, Ollama logs information about the available compute libraries and detected GPUs when GPU acceleration is enabled. The following example shows an Ollama log indicating that a compatible GPU was detected:

[...] source=gpu.go:204 msg="looking for compatible GPUs"
[...] source=types.go:105 msg="inference compute" id=GPU-... library=cuda compute=...

Refer to the documentation of the deployed application for application-specific instructions on verifying GPU detection and usage.