|Index|Creating SUSE AI Factory Blueprints|How do I create a blueprint CR for GPU inference with vLLM?
Applies to SUSE AI Factory

6 How do I create a blueprint CR for GPU inference with vLLM?

To deploy GPU-accelerated inference with vLLM, adapt an existing CPU blueprint by creating a separate hardware variant with dedicated GPU configurations.

Hardware variants are organized in separate directories. Create a GPU variant from your CPU base configuration:

cp -r blueprints/my-demo-ollama blueprints/my-demo-vllm
cd blueprints/my-demo-vllm
mv my-demo-ollama-1-0-0.yaml my-demo-vllm-1-0-0.yaml

Update the resource fields following the renaming checklist in Chapter 3, Quick start: copy an existing blueprint:

  1. Set spec.displayName to My Demo (vLLM, GPU).

  2. Update the spec.description text to specify NVIDIA GPU requirements.

  3. Cross-reference the sibling CPU variant in README.md.

6.1 Replacing Ollama with vLLM

Replace the Ollama component definition with the vLLM component:

  - chartName: vllm
    chartRepo: application-collection
    chartVersion: 0.1.10
    values:
      servingEngineSpec:
        runtimeClassName: nvidia
        modelSpec:
          - name: qwen25-vl-3b
            registry: dp.apps.rancher.io
            repository: containers/vllm-openai
            tag: "0.19.0-5.29"
            modelURL: Qwen/Qwen2.5-VL-3B-Instruct
            replicaCount: 1
            requestCPU: 6
            requestMemory: "16Gi"
            requestGPU: 1
            pvcStorage: "50Gi"
            vllmConfig:
              maxModelLen: 32768
              dtype: bfloat16
              gpuMemoryUtilization: 0.9
              # Vision flags: enable remote code execution and limit images per prompt
              extraArgs:
              - "--trust-remote-code"
              - "--limit-mm-per-prompt.image=2"
      routerSpec:
        enableRouter: true
        serviceType: ClusterIP
        containerPort: 8000
        servicePort: 80

Key configuration notes:

  • name and modelURL: The name parameter provides a resource label compliant with Kubernetes naming standards. The modelURL parameter identifies the Hugging Face model repository to serve.

  • GPU allocation: Each entry under modelSpec requests dedicated GPU resources (such as requestGPU: 1). Multiple models run behind a unified OpenAI-compatible routing endpoint (vllm-router-service:80), allowing clients to select models by name.

  • Memory management: Tune maxModelLen to fit the model within GPU memory before adjusting gpuMemoryUtilization.

  • Common arguments: Use --trust-remote-code and --limit-mm-per-prompt.image=N for vision models. Use --enable-auto-tool-choice --tool-call-parser=hermes for function calling.

6.2 Accessing the vLLM service

Test the model endpoint by forwarding the service port locally:

> kubectl -n <NAMESPACE> port-forward svc/vllm-router-service 8000:80
> curl -s localhost:8000/v1/models | python3 -m json.tool
Note
Note

=== If the Helm release uses a custom name, the service follows the naming convention <RELEASE_NAME>-router-service. ===

Because vLLM provides an OpenAI-compatible API, point your client applications to /v1/chat/completions on the router service port.

6.3 Managing gated models

Certain models require accepting terms of use on Hugging Face before download. To provide credentials securely:

  1. Configure the hf_token key in modelSpec.

  2. Avoid hardcoding secret tokens directly into the blueprint CR or git repositories.

  3. Reference pre-created Kubernetes secrets containing the token in production environments.

6.4 Configuring hybrid architectures

You can combine GPU and CPU inference components in the same blueprint. For example, deploy vLLM with GPU acceleration for large language and vision models, alongside an Ollama CPU instance dedicated to lightweight text embedding models (such as nomic-embed-text). This preserves GPU resources while maintaining high-throughput text embeddings.