2 LiteLLM #

LiteLLM (https://docs.litellm.ai) is an open source large language model (LLM) proxy and abstraction layer. The service translates diverse provider application programming interfaces (APIs) into a unified, OpenAI-compatible format. Using LiteLLM simplifies multi-model integration so that you can switch between proprietary cloud models and local open source models with minimal code changes. The proxy also provides centralized key management, detailed usage tracking and rate-limiting across multiple teams.

2.1 How do I get the values list for LiteLLM Helm chart? #

Before deploying LiteLLM, it is important to know more about the supported configurations.

(Optional) To get the values for a specific LiteLLM chart version, log in to SUSE Registry and identify the version. To prevent the password from being stored in the shell history, save it to the password.txt file.
```
> skopeo login registry.suse.com \
  --username regcode \
  --password-stdin < password.txt
> skopeo list-tags docker://registry.suse.com/ai/charts/litellm
```
Log in to SUSE Registry where LiteLLM chart is stored. To prevent the password from being stored in the shell history, save it to the password.txt file.
```
> helm registry login registry.suse.com \
  --username regcode \
  --password-stdin < password.txt
```
Display all values of the LiteLLM Helm chart. Omit --version to display the latest chart.
```
> helm show values oci://registry.suse.com/ai/charts/litellm \
  --version <VERSION>
```

2.2 How do I install LiteLLM? #

You can install LiteLLM either using Rancher Prime or on the command line with Helm.

2.2.1 Installing with Rancher Prime #

Refer to AI application installation for a general installation procedure.

2.2.2 Installing with Helm #

Log in to SUSE Registry. To prevent the password from being stored in the shell history, save it to the password.txt file.
```
> helm registry login registry.suse.com \
  --username=regcode \
  --password-stdin < password.txt
```

Create a namespace for LiteLLM.

> kubectl create namespace litellm-system \
  --dry-run=client \
  -o yaml | kubectl apply -f -

(Optional) If you have not created image pull secrets for SUSE Registry, create them now. To prevent the password from being stored in the shell history, save it to the password.txt file.

> kubectl create secret docker-registry suse-ai-registry \
  --docker-server=registry.suse.com \
  --docker-username=regcode \
  --docker-password="$(< password.txt)" \
  -n litellm-system \
  --dry-run=client -o yaml | kubectl apply -f -

(Optional) To override default installation values, list them in a values.yaml file. For a complete list of LiteLLM chart values, refer to Section 2.1, “How do I get the values list for LiteLLM Helm chart?”.

Install LiteLLM. To keep the default installation values, omit -f values.yaml.

> helm install litellm \
  oci://registry.suse.com/ai/charts/litellm \
  -n litellm-system \
  -f values.yaml

2.3 What custom values examples are available for LiteLLM? #

To override the default values during the Helm chart installation or update, you can create a YAML file with custom values. Then, apply these values by specifying the path to the custom values file with the -f option of the helm command.

2.3.1 Basic custom values file with PostgreSQL deployment and an automatically generated master key #

# litellm_custom_overrides.yaml
global:
  imagePullSecrets:
    - application-collection 1
    - suse-ai-registry 2
postgresql:
  persistence:
    storageClassName: "local-path" 3

1	Specify the secret containing the credentials for the SUSE Application Collection in `global.imagePullSecrets`.
2	Specify the secret containing the credentials for the SUSE Registry in `global.imagePullSecrets`.
3	Use `local-path` storage only for testing purposes. For production use, we recommend using a storage solution suitable for persistent storage, such as SUSE Storage.

The LiteLLM chart from SUSE Registry makes use of subcharts such as Redis and PostgreSQL from the SUSE Application Collection registry. Therefore, both registry secrets must be configured.

2.3.2 Validating the installation #

You can monitor the status of the LiteLLM installation by running the following command:

> kubectl get pods -n <SAIF_NAMESPACE>

NAME                         READY   STATUS    RESTARTS   AGE
[...]
litellm-5d8c4c864f-9v7q5      1/1     Running   1          2m33s
litellm-postgresql-0          1/1     Running   0          2m33s

Pods for the LiteLLM deployment should transition to the Ready and Running states.

2.3.3 Validating local access #

By default, the configuration does not enable Ingress. To expose the service on localhost, run:

> kubectl \
  -n litellm-system port-forward service/litellm 4000:4000

Your LiteLLM Proxy Server is now running at http://127.0.0.1:4000.

2.3.4 Admin UI access #

You can access the administration page at http://127.0.0.1:4000/ui and log in using the administrator account. Retrieve the password for the administrator user from the secret resource:

> kubectl get secret litellm-masterkey \
  -n litellm-system -o jsonpath='{.data.masterkey}' | base64 --decode

2.4 How do I upgrade LiteLLM? #

You can upgrade LiteLLM either using Rancher Prime or on the command line with Helm.

2.4.1 Upgrading with Rancher Prime #

Refer to AI application upgrade for a general upgrade procedure.

2.4.2 Upgrading with Helm #

Log in to SUSE Registry. To prevent the password from being stored in the shell history, save it to the password.txt file.
```
> helm registry login registry.suse.com \
  --username=regcode \
  --password-stdin < password.txt
```

Upgrade LiteLLM.

> helm upgrade litellm \
  oci://registry.suse.com/ai/charts/litellm \
  -n litellm-system \
  --version <VERSION_NUMBER> \ 1
  --wait --timeout 15m \
  --force-conflicts

1	If you omit `--version`, LiteLLM gets upgraded to the latest available version.

2.5 How do I remove LiteLLM? #

You can remove LiteLLM either using Rancher Prime or on the command line with Helm.

2.5.1 Removal using Rancher Prime #

Refer to AI application removal for a general removal procedure.

2.5.2 Removal using Helm #

To remove LiteLLM using Helm, use the following command:

> helm uninstall litellm -n litellm-system