Prometheus Federator

Prometheus Federator, also referred to as Project Monitoring v2, deploys a Helm Project Operator (based on the rancher/helm-project-operator), an operator that manages deploying Helm charts each containing a Project Monitoring Stack, where each stack contains:

Important:

Prometheus Federator is designed to be deployed alongside an existing Prometheus Operator deployment in a cluster that has already installed the Prometheus Operator CRDs.

How does the operator work?

  1. On deploying this chart, users can create ProjectHelmCharts CRs with spec.helmApiVersion set to monitoring.cattle.io/v1alpha1 (also known as "Project Monitors" in the Rancher UI) in a Project Registration Namespace (cattle-project-<id>).

  2. On seeing each ProjectHelmChartCR, the operator will automatically deploy a Project Prometheus stack on the Project Owner’s behalf in the Project Release Namespace (cattle-project-<id>-monitoring) based on a HelmChart CR and a HelmRelease CR automatically created by the ProjectHelmChart controller in the Operator / System Namespace.

  3. RBAC will automatically be assigned in the Project Release Namespace to allow users to view the Prometheus, Alertmanager, and Grafana UIs of the Project Monitoring Stack deployed; this will be based on RBAC defined on the Project Registration Namespace against the default Kubernetes user-facing roles. For more information, see the section on configuring RBAC.

What is a Project?

In Prometheus Federator, a Project is a group of namespaces that can be identified by a metav1.LabelSelector. By default, the label used to identify projects is field.cattle.io/projectId, the label used to identify namespaces that are contained within a given Rancher Project.

Configuring the Helm release created by a ProjectHelmChart

The spec.values of this ProjectHelmChart’s resources will correspond to the values.yaml override to be supplied to the underlying Helm chart deployed by the operator on the user’s behalf; to see the underlying chart’s values.yaml spec, either:

  • View the chart’s definition located at rancher/prometheus-federator under charts/rancher-project-monitoring (where the chart version will be tied to the version of this operator).

  • Look for the ConfigMap named monitoring.cattle.io.v1alpha1 that is automatically created in each Project Registration Namespace, which will contain both the values.yaml and questions.yaml that was used to configure the chart (which was embedded directly into the prometheus-federator binary).

Namespaces

As a Project Operator based on rancher/helm-project-operator, Prometheus Federator has three different classifications of namespaces that the operator looks out for:

  1. Operator / System Namespace: The namespace that the operator is deployed into (e.g., cattle-monitoring-system). This namespace will contain all HelmCharts and HelmReleases for all ProjectHelmCharts watched by this operator. Only Cluster Admins should have access to this namespace.

  2. Project Registration Namespace (cattle-project-<id>): The set of namespaces that the operator watches for ProjectHelmCharts within. The RoleBindings and ClusterRoleBindings that apply to this namespace will also be the source of truth for the auto-assigned RBAC created in the Project Release Namespace. For details, refer to the RBAC page. Project Owners (admin), Project Members (edit), and Read-Only Members (view) should have access to this namespace.

    Notes:

    • Project Registration Namespaces will be auto-generated by the operator and imported into the Project it is tied to if .Values.global.cattle.projectLabel is provided, which is set to field.cattle.io/projectId by default. This indicates that a Project Registration Namespace should be created by the operator if at least one namespace is observed with that label. The operator will not let these namespaces be deleted unless either all namespaces with that label are gone (e.g., this is the last namespace in that project, in which case the namespace will be marked with the label "helm.cattle.io/helm-project-operator-orphaned": "true", which signals that it can be deleted), or it is no longer watching that project because the project ID was provided under .Values.helmProjectOperator.otherSystemProjectLabelValues, which serves as a denylist for Projects. These namespaces will also never be auto-deleted to avoid destroying user data; it is recommended that users clean up these namespaces manually if desired on creating or deleting a project.

    • If .Values.global.cattle.projectLabel is not provided, the Operator / System Namespace will also be the Project Registration Namespace.

  3. Project Release Namespace (cattle-project-<id>-monitoring): The set of namespaces that the operator deploys Project Monitoring Stacks within on behalf of a ProjectHelmChart; the operator will also automatically assign RBAC to Roles created in this namespace by the Project Monitoring Stack based on bindings found in the Project Registration Namespace. Only Cluster Admins should have access to this namespace; Project Owners (admin), Project Members (edit), and Read-Only Members (view) will be assigned limited access to this namespace by the deployed Helm Chart and Prometheus Federator.

    Notes:

    • Project Release Namespaces are automatically deployed and imported into the project whose ID is specified under .Values.helmProjectOperator.projectReleaseNamespaces.labelValue, which defaults to the value of .Values.global.cattle.systemProjectId if not specified, whenever a ProjectHelmChart is specified in a Project Registration Namespace.

    • Project Release Namespaces follow the same orphaning conventions as Project Registration Namespaces (see note above).

    • If .Values.projectReleaseNamespaces.enabled is false, the Project Release Namespace will be the same as the Project Registration Namespace.

Helm Resources (HelmChart, HelmRelease)

On deploying a ProjectHelmChart, the Prometheus Federator will automatically create and manage two child custom resources that manage the underlying Helm resources in turn:

  • A HelmChart CR (managed via an embedded k3s-io/helm-contoller in the operator): This custom resource automatically creates a Job in the same namespace that triggers a helm install, helm upgrade, or helm uninstall depending on the change applied to the HelmChart CR. This CR is automatically updated on changes to the ProjectHelmChart (e.g., modifying the values.yaml) or changes to the underlying Project definition (e.g., adding or removing namespaces from a project).

Important:

If a ProjectHelmChart is not deploying or updating the underlying Project Monitoring Stack for some reason, the Job created by this resource in the Operator / System namespace should be the first place you check to see if there’s something wrong with the Helm operation. However, this is generally only accessible by a Cluster Admin.

  • A HelmRelease CR (managed via an embedded rancher/helm-locker in the operator): This custom resource automatically locks a deployed Helm release in place and automatically overwrites updates to underlying resources unless the change happens via a Helm operation (helm install, helm upgrade, or helm uninstall performed by the HelmChart CR).

HelmRelease CRs emit Kubernetes Events that detect when an underlying Helm release is being modified and locks it back to place. To view these events, you can use kubectl describe helmrelease <helm-release-name> -n <operator/system-namespace>; you can also view the logs on this operator to see when changes are detected and which resources modifications were attempted on.

Both of these resources are created for all Helm charts in the Operator / System namespaces to avoid escalation of privileges to underprivileged users.

Advanced Helm Project Operator Configuration

For more information on advanced configurations, refer to this page.

Prometheus Federator on the Local Cluster

Prometheus Federator is a resource intensive application. Installing it to the local cluster is possible, but not recommended.