Prometheus Integration

Purpose

Prometheus server can monitor various metrics and provide an observation of the Antrea Controller and Agent components. The doc provides general guidelines to the configuration of Prometheus server to operate with the Antrea components.

About Prometheus

Prometheus is an open source monitoring and alerting server. Prometheus is capable of collecting metrics from various Kubernetes components, storing and providing alerts. Prometheus can provide visibility by integrating with other products such as Grafana.

One of Prometheus capabilities is self-discovery of Kubernetes services which expose their metrics. So Prometheus can scrape the metrics of any additional components which are added to the cluster without further configuration changes.

Antrea Configuration

Enable Prometheus metrics listener by setting enablePrometheusMetrics parameter to true in the Controller and the Agent configurations.

Prometheus Configuration

Prometheus version

Prometheus integration with Antrea is validated as part of CI using Prometheus v2.19.3.

Prometheus RBAC

Prometheus requires access to Kubernetes API resources for the service discovery capability. Reading metrics also requires access to the "/metrics" API endpoints. yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: prometheus rules: - apiGroups: [""] resources: - nodes - nodes/proxy - services - endpoints - pods verbs: ["get", "list", "watch"] - apiGroups: - networking.k8s.io resources: - ingresses verbs: ["get", "list", "watch"] - nonResourceURLs: ["/metrics"] verbs: ["get"]

Antrea Metrics Listener Access

To scrape the metrics from Antrea Controller and Agent, Prometheus needs the following permissions yaml kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: prometheus-antrea rules: - nonResourceURLs: - /metrics verbs: - get

Antrea Components Scraping configuration

Add the following jobs to Prometheus scraping configuration to enable metrics collection from Antrea components. Antrea Agent metrics endpoint is exposed through Antrea apiserver on apiport config parameter given in antrea-agent.conf (default value is 10350). Antrea Controller metrics endpoint is exposed through Antrea apiserver on apiport config parameter given in antrea-controller.conf (default value is 10349).

Controller Scraping

- job_name: 'antrea-controllers'
kubernetes_sd_configs:
- role: endpoints
scheme: https
tls_config:
  ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_container_name]
  action: keep
  regex: kube-system;antrea-controller
- source_labels: [__meta_kubernetes_pod_node_name, __meta_kubernetes_pod_name]
  target_label: instance

Agent Scraping

- job_name: 'antrea-agents'
kubernetes_sd_configs:
- role: pod
scheme: https
tls_config:
  ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_container_name]
  action: keep
  regex: kube-system;antrea-agent
- source_labels: [__meta_kubernetes_pod_node_name, __meta_kubernetes_pod_name]
  target_label: instance

For further reference see the enclosed configuration file.

The configuration file above can be used to deploy Prometheus Server with scraping configuration for Antrea services. To deploy this configuration use kubectl apply -f build/yamls/antrea-prometheus.yml