6 How do I create a blueprint CR for GPU inference with vLLM? #
To deploy GPU-accelerated inference with vLLM, adapt an existing CPU blueprint by creating a separate hardware variant with dedicated GPU configurations.
Hardware variants are organized in separate directories. Create a GPU variant from your CPU base configuration:
cp -r blueprints/my-demo-ollama blueprints/my-demo-vllm
cd blueprints/my-demo-vllm
mv my-demo-ollama-1-0-0.yaml my-demo-vllm-1-0-0.yamlUpdate the resource fields following the renaming checklist in Chapter 3, Quick start: copy an existing blueprint:
Set
spec.displayNametoMy Demo (vLLM, GPU).Update the
spec.descriptiontext to specify NVIDIA GPU requirements.Cross-reference the sibling CPU variant in
README.md.
6.1 Replacing Ollama with vLLM #
Replace the Ollama component definition with the vLLM component:
- chartName: vllm
chartRepo: application-collection
chartVersion: 0.1.10
values:
servingEngineSpec:
runtimeClassName: nvidia
modelSpec:
- name: qwen25-vl-3b
registry: dp.apps.rancher.io
repository: containers/vllm-openai
tag: "0.19.0-5.29"
modelURL: Qwen/Qwen2.5-VL-3B-Instruct
replicaCount: 1
requestCPU: 6
requestMemory: "16Gi"
requestGPU: 1
pvcStorage: "50Gi"
vllmConfig:
maxModelLen: 32768
dtype: bfloat16
gpuMemoryUtilization: 0.9
# Vision flags: enable remote code execution and limit images per prompt
extraArgs:
- "--trust-remote-code"
- "--limit-mm-per-prompt.image=2"
routerSpec:
enableRouter: true
serviceType: ClusterIP
containerPort: 8000
servicePort: 80Key configuration notes:
nameandmodelURL: Thenameparameter provides a resource label compliant with Kubernetes naming standards. ThemodelURLparameter identifies the Hugging Face model repository to serve.GPU allocation: Each entry under
modelSpecrequests dedicated GPU resources (such asrequestGPU: 1). Multiple models run behind a unified OpenAI-compatible routing endpoint (vllm-router-service:80), allowing clients to select models by name.Memory management: Tune
maxModelLento fit the model within GPU memory before adjustinggpuMemoryUtilization.Common arguments: Use
--trust-remote-codeand--limit-mm-per-prompt.image=Nfor vision models. Use--enable-auto-tool-choice --tool-call-parser=hermesfor function calling.
6.2 Accessing the vLLM service #
Test the model endpoint by forwarding the service port locally:
> kubectl -n <NAMESPACE> port-forward svc/vllm-router-service 8000:80
> curl -s localhost:8000/v1/models | python3 -m json.tool===
If the Helm release uses a custom name, the service follows the naming convention <RELEASE_NAME>-router-service.
===
Because vLLM provides an OpenAI-compatible API, point your client applications to /v1/chat/completions on the router service port.
6.3 Managing gated models #
Certain models require accepting terms of use on Hugging Face before download. To provide credentials securely:
Configure the
hf_tokenkey inmodelSpec.Avoid hardcoding secret tokens directly into the blueprint CR or git repositories.
Reference pre-created Kubernetes secrets containing the token in production environments.
6.4 Configuring hybrid architectures #
You can combine GPU and CPU inference components in the same blueprint. For example, deploy vLLM with GPU acceleration for large language and vision models, alongside an Ollama CPU instance dedicated to lightweight text embedding models (such as nomic-embed-text). This preserves GPU resources while maintaining high-throughput text embeddings.