8 vLLM #

vLLM is an open-source, high-throughput engine designed for fast large language model (LLM) serving and inference. It uses PagedAttention to manage attention keys and values efficiently, reducing memory waste and increasing serving capacity. This local deployment enables high-performance model execution with an OpenAI-compatible API interface.

8.1 How do I get the values list for vLLM Helm chart? #

Before deploying vLLM, it is important to know more about the supported configurations.

(Optional) To get the values for a specific vLLM chart version, log in to SUSE Application Collection and identify the version. To prevent the password from being stored in the shell history, save it to the password.txt file.
```
> skopeo login dp.apps.rancher.io \
  --username <APPCO-USERNAME> \
  --password-stdin < password.txt
> skopeo list-tags docker://dp.apps.rancher.io/charts/vllm
```
Log in to SUSE Application Collection where vLLM chart is stored. To prevent the password from being stored in the shell history, save it to the password.txt file.
```
> helm registry login dp.apps.rancher.io \
  --username <APPCO-USERNAME> \
  --password-stdin < appco-password.txt
```
Display all values of the vLLM Helm chart. Omit --version to display the latest chart.
```
> helm show values oci://dp.apps.rancher.io/charts/vllm \
  --version <VERSION>
```

8.2 How do I install vLLM? #

You can install vLLM either using Rancher Prime or on the command line with Helm.

8.2.1 Installing with Rancher Prime #

Refer to AI application installation for a general installation procedure.

8.2.2 Installing with Helm #

Log in to SUSE Application Collection. To prevent the password from being stored in the shell history, save it to the password.txt file.
```
> helm registry login dp.apps.rancher.io \
  --username=<APPCO-USERNAME> \
  --password-stdin < password.txt
```

Create a namespace for vLLM.

> kubectl create namespace vllm-system \
  --dry-run=client \
  -o yaml | kubectl apply -f -

(Optional) If you have not created image pull secrets for SUSE Application Collection, create them now. To prevent the password from being stored in the shell history, save it to the password.txt file.

> kubectl create secret docker-registry application-collection \
  --docker-server=dp.apps.rancher.io \
  --docker-username=<APPCO-USERNAME> \
  --docker-password="$(< password.txt)" \
  -n vllm-system \
  --dry-run=client -o yaml | kubectl apply -f -

(Optional) To override default installation values, list them in a values.yaml file. For a complete list of vLLM chart values, refer to Section 8.1, “How do I get the values list for vLLM Helm chart?”.

Install vLLM. To keep the default installation values, omit -f values.yaml.

> helm install vllm \
  oci://dp.apps.rancher.io/charts/vllm \
  -n vllm-system \
  -f values.yaml

8.3 What custom values examples are available for vLLM? #

vLLM is an open-source high-performance inference and serving engine for large language models (LLMs). It is designed to maximize throughput and reduce latency by using an efficient memory management system that handles dynamic batching and streaming outputs.

8.3.1 Integrating vLLM with Open WebUI #

You can integrate vLLM in Open WebUI either using the Open WebUI Web user interface, or updating Open WebUI custom values file during Open WebUI deployment (see Example 5.4, “Open WebUI custom values file with a connection to vLLM”).

Integrating vLLM with Open WebUI via the Web user interface.

Requirements #

You must have Open WebUI administrator privileges to access configuration screens or settings mentioned in this section.

In the bottom left of the Open WebUI window, click your avatar icon to open the user menu and select btn:[Admin Panel].
Click the btn:[Settings] tab and select btn:[Connections] from the left menu.
In the btn:[Manage OpenAI API Connections] section, add a new connection URL to the vLLM router service, for example:
```
http://vllm-router-service.<SAIF_NAMESPACE>.svc.cluster.local:80/v1
```
Confirm with btn:[Save].
Figure 8.1: Adding a vLLM connection to Open WebUI #

To override the default values during the Helm chart installation or update, you can create a YAML file with custom values. Then, apply these values by specifying the path to the custom values file with the -f option of the helm command.

Example 8.1: Minimal configuration #

The following custom values file installs vLLM using a model that is publicly available.

global:
  imagePullSecrets:
  - application-collection
  
servingEngineSpec:
  modelSpec:
  - name: "phi3-mini-4k"
    registry: "dp.apps.rancher.io"
    repository: "containers/vllm-openai"
    tag: "0.19.0"
    imagePullPolicy: "IfNotPresent"
    modelURL: "microsoft/Phi-3-mini-4k-instruct"
    replicaCount: 1
    requestCPU: 6
    requestMemory: "16Gi"
    requestGPU: 1

Example 8.2: Validating the installation #

Pulling the images can take a long time. You can monitor the status of the vLLM installation by running the following command:

> kubectl get pods -n <SAIF_NAMESPACE>

NAME                                           READY   STATUS    RESTARTS   AGE
[...]
vllm-deployment-router-7588bf995c-5jbkf        1/1     Running   0          8m9s
vllm-phi3-mini-4k-deployment-vllm-79d6fdc-tx7  1/1     Running   0          8m9s

Pods for the vLLM deployment should transition to the states Ready and Running.

Validating the stack #

Expose the vllm-router-service port to the host machine:

> kubectl port-forward svc/vllm-router-service \
  -n <SAIF_NAMESPACE> 30080:80

Query the OpenAI-compatible API to list the available models:
```
> curl -o- http://localhost:30080/v1/models
```

Send a query to the OpenAI /completion endpoint to generate a completion for a prompt:

> curl -X POST http://localhost:30080/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "microsoft/Phi-3-mini-4k-instruct",
    "prompt": "Once upon a time,",
    "max_tokens": 10
  }'

# example output of generated completions
{
    "id": "cmpl-3dd11a3624654629a3828c37bac3edd2",
    "object": "text_completion",
    "created": 1757530703,
    "model": "microsoft/Phi-3-mini-4k-instruct",
    "choices": [
        {
            "index": 0,
            "text": " in a bustling city full of concrete and",
            "logprobs": null,
            "finish_reason": "length",
            "stop_reason": null,
            "prompt_logprobs": null
        }
    ],
    "usage": {
        "prompt_tokens": 5,
        "total_tokens": 15,
        "completion_tokens": 10,
        "prompt_tokens_details": null
    },
    "kv_transfer_params": null
}

Example 8.3: Basic configuration #

The following vLLM custom values file includes basic configuration options.

Prerequisites #

Access to a Hugging Face token (HF_TOKEN).
The model meta-llama/Llama-3.1-8B-Instruct from this example is a gated model that requires you to accept the agreement to access it. For more information, see https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct.
Update the storageClass: entry for each modelSpec.

# vllm_custom_overrides.yaml
global:
  imagePullSecrets:
  - application-collection
  
servingEngineSpec:
  modelSpec:
  - name: "llama3" 1
    registry: "dp.apps.rancher.io" 2
    repository: "containers/vllm-openai" 3
    tag: "0.19.0" 4
    imagePullPolicy: "IfNotPresent"
    modelURL: "meta-llama/Llama-3.1-8B-Instruct" 5
    replicaCount: 1 6
    requestCPU: 10 7
    requestMemory: "16Gi" 8
    requestGPU: 1 9
    storageClass: <STORAGE_CLASS>
    pvcStorage: "50Gi" 10
    pvcAccessMode:
      - ReadWriteOnce

    vllmConfig:
      enableChunkedPrefill: false 11
      enablePrefixCaching: false 12
      maxModelLen: 4096 13
      dtype: "bfloat16" 14
      extraArgs: ["--disable-log-requests", "--gpu-memory-utilization", "0.8"] 15

    hf_token: <HF_TOKEN> 16

1	The unique identifier for your model deployment.
2	The Docker image registry containing the model’s serving engine image.
3	The Docker image repository containing the model’s serving engine image.
4	The version of the model image to use.
5	The URL pointing to the model on Hugging Face or another hosting service.
6	The number of replicas for the deployment, which allows scaling for load.
7	The amount of CPU resources requested per replica.
8	Memory allocation for the deployment. Sufficient memory is required to load the model.
9	The number of GPUs to allocate for the deployment.
10	The Persistent Volume Claim (PVC) size for model storage.
11	Optimizes performance by prefetching model chunks.
12	Enables caching of prompt prefixes to speed up inference for repeated prompts.
13	The maximum sequence length the model can handle.
14	The data type for model weights, such as `bfloat16` for mixed-precision inference and faster performance on modern GPUs.
15	Additional command-line arguments for vLLM, such as disabling request logging or setting GPU memory utilization.
16	Your Hugging Face token for accessing gated models. Replace `HF_TOKEN` with your actual token.

Example 8.4: Loading prefetched models from persistent storage #

Prefetching models to a Persistent Volume Claim (PVC) prevents repeated downloads from Hugging Face during pod startup. The process involves creating a PVC and a job to fetch the model. This PVC is mounted at /models, where the prefetch job stores the model weights. Subsequently, the vLLM modelURL is set to this path, which ensures that the model is loaded locally instead of being downloaded when the pod starts.

Define a PVC for model weights using the following YAML specification.

# pvc-models.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: models-pvc
  namespace: <SAIF_NAMESPACE>
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 50Gi # Adjust size based on your model
  storageClassName: <STORAGE_CLASS>

Save it as pvc-models.yaml and apply with kubectl apply -f pvc-models.yaml.

Create a secret resource for the Hugging Face token.

> kubectl create secret -n <SAIF_NAMESPACE> \
  generic huggingface-credentials \
  --from-literal=HUGGING_FACE_HUB_TOKEN=<HF_TOKEN>

Create a YAML specification for prefetching the model and save it as job-prefetch-llama3.1-8b.yaml.

# job-prefetch-llama3.1-8b.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: prefetch-llama3.1-8b
  namespace: <SAIF_NAMESPACE>
spec:
  template:
    spec:
      restartPolicy: OnFailure
      containers:
      - name: hf-download
        image: python:3.10-slim
        env:
        - name: HF_TOKEN
          valueFrom: { secretKeyRef: { name: huggingface-credentials, key: <HUGGING_FACE_HUB_TOKEN> } }
        - name: HF_HUB_ENABLE_HF_TRANSFER
          value: "1"
        - name: HF_HUB_DOWNLOAD_TIMEOUT
          value: "60"
        command: ["bash","-lc"]
        args:
        - |
          set -e
          echo "Logging in..."
          echo "Installing Hugging Face CLI..."
          pip install "huggingface_hub[cli]"
          pip install "hf_transfer"
          hf auth login --token "${HF_TOKEN}"
          echo "Downloading Llama 3.1 8B Instruct to /models/llama-3.1-8b-it ..."
          hf download meta-llama/Llama-3.1-8B-Instruct --local-dir /models/llama-3.1-8b-it
        volumeMounts:
        - name: models
          mountPath: /models
      volumes:
      - name: models
        persistentVolumeClaim:
          claimName: models-pvc

Apply the specification with the following commands:

> kubectl apply -f job-prefetch-llama3.1-8b.yaml
> kubectl -n <SAIF_NAMESPACE> \
  wait --for=condition=complete job/prefetch-llama3.1-8b

Update the vLLM custom values file with support for PVC.

# vllm_custom_overrides.yaml
global:
  imagePullSecrets:
  - application-collection
  
servingEngineSpec:
  modelSpec:
  - name: "llama3"
    registry: "dp.apps.rancher.io"
    repository: "containers/vllm-openai"
    tag: "0.19.0"
    imagePullPolicy: "IfNotPresent"
    modelURL: "/models/llama-3.1-8b-it"
    replicaCount: 1

    requestCPU: 10
    requestMemory: "16Gi"
    requestGPU: 1

    extraVolumes:
      - name: models-pvc
        persistentVolumeClaim:
          claimName: models-pvc 1

    extraVolumeMounts:
      - name: models-pvc
        mountPath: /models 2

    vllmConfig:
      maxModelLen: 4096

    hf_token: <HF_TOKEN>

Specify your PVC name.

The mount path must match the base directory of the servingEngineSpec.modelSpec.modeURL value specified above.

Save it as vllm_custom_overrides.yaml and apply with kubectl apply -f vllm_custom_overrides.yaml.

The following example lists mounted PVCs for a pod.

> kubectl exec -it vllm-llama3-deployment-vllm-858bd967bd-w26f7 \
  -n <SAIF_NAMESPACE> -- ls -l /models
drwxr-xr-x 1 root root 608 Aug 22 16:29 llama-3.1-8b-it

Example 8.5: Configuration with multiple models #

This example shows how to configure multiple models to run on different GPUs. Remember to update the entries hf_token and storageClass.

Note: Ray is not supported

Ray is currently not supported. Therefore, sharding a single large model across multiple GPUs is not supported.

# vllm_custom_overrides.yaml
global:
  imagePullSecrets:
  - application-collection
  
servingEngineSpec:
  modelSpec:
  - name: "llama3"
    registry: "dp.apps.rancher.io"
    repository: "containers/vllm-openai"
    tag: "0.19.0"
    imagePullPolicy: "IfNotPresent"
    modelURL: "meta-llama/Llama-3.1-8B-Instruct"
    replicaCount: 1
    requestCPU: 10
    requestMemory: "16Gi"
    requestGPU: 1
    pvcStorage: "50Gi"
    storageClass: <STORAGE_CLASS>
    vllmConfig:
      maxModelLen: 4096
    hf_token: <HF_TOKEN_FOR_LLAMA_31>

  - name: "mistral"
    registry: "dp.apps.rancher.io"
    repository: "containers/vllm-openai"
    tag: "0.19.0"
    imagePullPolicy: "IfNotPresent"
    modelURL: "mistralai/Mistral-7B-Instruct-v0.2"
    replicaCount: 1
    requestCPU: 10
    requestMemory: "16Gi"
    requestGPU: 1
    pvcStorage: "50Gi"
    storageClass: <STORAGE_CLASS>
    vllmConfig:
      maxModelLen: 4096
    hf_token: <HF_TOKEN_FOR_MISTRAL>

Example 8.6: CPU offloading #

This example demonstrates how to enable KV cache offloading to the CPU using LMCache in a vLLM deployment. You can enable LMCache and set the CPU offloading buffer size using the lmcacheConfig field. In the following example, the buffer is set to 20 GB, but you can adjust this value based on your workload. Remember to update the entries hf_token and storageClass.

Warning: Experimental Features

Setting lmcacheConfig.enabled to true implicitly enables the LMCACHE_USE_EXPERIMENTAL flag for LMCache. These experimental features are only supported on newer GPU generations. It is not recommended to enable them without a compelling reason.

# vllm_custom_overrides.yaml
global:
  imagePullSecrets:
  - application-collection
  }
servingEngineSpec:
  modelSpec:
  - name: "mistral"
    registry: "dp.apps.rancher.io"
    repository: "containers/lmcache-vllm-openai"
    tag: "0.3.9"
    imagePullPolicy: "IfNotPresent"
    modelURL: "mistralai/Mistral-7B-Instruct-v0.2"
    replicaCount: 1
    requestCPU: 10
    requestMemory: "40Gi"
    requestGPU: 1
    pvcStorage: "50Gi"
    storageClass: <STORAGE_CLASS>
    pvcAccessMode:
      - ReadWriteOnce
    vllmConfig:
      maxModelLen: 32000

    lmcacheConfig:
      enabled: false
      cpuOffloadingBufferSize: "20"

    hf_token: <HF_TOKEN>

Example 8.7: Shared remote KV cache storage with LMCache #

This example shows how to enable remote KV cache storage using LMCache in a vLLM deployment. The configuration defines a cacheserverSpec and uses two replicas. Remember to replace the placeholder values for hf_token and storageClass before applying the configuration.

Warning: Experimental features

# vllm_custom_overrides.yaml
global:
  imagePullSecrets:
  - application-collection
  
servingEngineSpec:
  modelSpec:
  - name: "mistral"
    registry: "dp.apps.rancher.io"
    repository: "containers/lmcache-vllm-openai"
    tag: "0.3.9"
    imagePullPolicy: "IfNotPresent"
    modelURL: "mistralai/Mistral-7B-Instruct-v0.2"
    replicaCount: 2
    requestCPU: 10
    requestMemory: "40Gi"
    requestGPU: 1
    pvcStorage: "50Gi"
    storageClass: <STORAGE_CLASS>
    vllmConfig:
      enablePrefixCaching: true
      maxModelLen: 16384
    lmcacheConfig:
      enabled: false
      cpuOffloadingBufferSize: "20"
    hf_token: <HF_TOKEN>
    initContainer:
      name: "wait-for-cache-server"
      image: "dp.apps.rancher.io/containers/lmcache-vllm-openai:0.3.9"
      command: ["/bin/sh", "-c"]
      args:
        - |
          timeout 60 bash -c '
          while true; do
            /opt/venv/bin/python3 /workspace/LMCache/examples/kubernetes/health_probe.py $(RELEASE_NAME)-cache-server-service $(LMCACHE_SERVER_SERVICE_PORT) && exit 0
            echo "Waiting for LMCache server..."
            sleep 2
          done'
cacheserverSpec:
  replicaCount: 1
  containerPort: 8080
  servicePort: 81
  serde: "naive"
  registry: "dp.apps.rancher.io"
  repository: "containers/lmcache-vllm-openai"
  tag: "0.3.9"
  resources:
    requests:
      cpu: "4"
      memory: "8G"
    limits:
      cpu: "4"
      memory: "10G"
  labels:
    environment: "cacheserver"
    release: "cacheserver"
routerSpec:
  resources:
    requests:
      cpu: "1"
      memory: "2G"
    limits:
      cpu: "1"
      memory: "2G"
  routingLogic: "session"
  sessionKey: "x-user-id"

8.4 How do I upgrade vLLM? #

You can upgrade vLLM either using Rancher Prime or on the command line with Helm.

8.4.1 Upgrading with Rancher Prime #

Refer to AI application upgrade for a general upgrade procedure.

8.4.2 Upgrading with Helm #

Log in to SUSE Application Collection. To prevent the password from being stored in the shell history, save it to the password.txt file.
```
> helm registry login dp.apps.rancher.io \
  --username=<APPCO-USERNAME> \
  --password-stdin < password.txt
```

Upgrade vLLM.

> helm upgrade vllm \
  oci://dp.apps.rancher.io/charts/vllm \
  -n vllm-system \
  --version <VERSION_NUMBER> \ 1
  --wait --timeout 15m \
  --force-conflicts

1	If you omit `--version`, vLLM gets upgraded to the latest available version.

8.5 How do I remove vLLM? #

You can remove vLLM either using Rancher Prime or on the command line with Helm.

8.5.1 Removal using Rancher Prime #

Refer to AI application removal for a general removal procedure.

8.5.2 Removal using Helm #

To remove vLLM using Helm, use the following command:

> helm uninstall vllm -n vllm-system