Network Flow Visibility in Antrea

Table of Contents

• Overview
• Flow Exporter
  • Configuration
    • Configuration pre Antrea v1.13
  • IPFIX Information Elements (IEs) in a Flow Record
    • IEs from IANA-assigned IE Registry
    • IEs from Reverse IANA-assigned IE Registry
    • IEs from Antrea IE Registry
  • Supported Capabilities
    • Types of Flows and Associated Information
    • Connection Metrics
• Flow Aggregator
  • Deciding which mode to use
  • Aggregate Mode
    • Installation
      • Configuring secure connections to the ClickHouse database
      • Example of flow-aggregator.conf
    • IPFIX Information Elements (IEs) in an Aggregated Flow Record
      • IEs from Antrea IE Registry
    • Supported Capabilities
      • Storage of Flow Records
      • Correlation of Flow Records
      • Aggregation of Flow Records
    • Antctl Support
  • Proxy Mode (v2.3 and above)
    • Installation
    • IPFIX Information Elements (IEs) in a Proxied Flow Record
  • FlowStreamService (alpha)
    • Authenticating to the FlowStreamService
    • Authorizing flow visibility
    • What a client sees of each endpoint
    • What administrators should know
  • Version skew between Flow Aggregator and Antrea Agent
• Quick Deployment
  • Image-building Steps
  • Deployment Steps
• Flow Collectors
  • Go-ipfix Collector
    • Deployment Steps
    • Output Flow Records
  • Grafana Flow Collector (migrated)
  • ELK Flow Collector (removed)

Overview

Antrea is a Kubernetes network plugin that provides network connectivity and security features for Pod workloads. Considering the scale and dynamism of Kubernetes workloads in a cluster, Network Flow Visibility helps in the management and configuration of Kubernetes resources such as Network Policy, Services, Pods etc., and thereby provides opportunities to enhance the performance and security aspects of Pod workloads.

For visualizing the network flows, Antrea monitors the flows in Linux conntrack module. These flows are converted to flow records, and then flow records are post-processed before they are sent to the configured external flow collector. High-level design is given below:

Flow Exporter

In Antrea, the basic building block for the Network Flow Visibility is the Flow Exporter. Flow Exporter operates within Antrea Agent; it builds and maintains a connection store by polling and dumping flows from conntrack module periodically. Connections from the connection store are exported to the Flow Aggregator Service using the IPFIX protocol, and for this purpose we use the IPFIX exporter process from the go-ipfix library.

Configuration

In addition to enabling the Flow Exporter feature gate (if needed), you need to ensure that the flowExporter.enable flag is set to true in the Antrea Agent configuration.

Your antrea-agent ConfigMap should look like this:

  antrea-agent.conf: |
    # FeatureGates is a map of feature names to bools that enable or disable experimental features.
    featureGates:
    # Enable flowexporter which exports polled conntrack connections as IPFIX flow records from each agent to a configured collector.
      FlowExporter: true

    flowExporter:
      # Enable FlowExporter, a feature used to export polled conntrack connections as
      # IPFIX flow records from each agent to a configured collector. To enable this
      # feature, you need to set "enable" to true, and ensure that the FlowExporter
      # feature gate is also enabled.
      enable: true
      # Provide the IPFIX collector address as a string with format <HOST>:[<PORT>][:<PROTO>].
      # HOST can either be the DNS name, IP, or Service name of the Flow Collector. If
      # using an IP, it can be either IPv4 or IPv6. However, IPv6 address should be
      # wrapped with []. When the collector is running in-cluster as a Service, set
      # <HOST> to <Service namespace>/<Service name>. For example,
      # "flow-aggregator/flow-aggregator" can be provided to connect to the Antrea
      # Flow Aggregator Service.
      # If PORT is empty, we default to 4739. If no PROTO is given, we consider "tls" as default.
      # We support "grpc", "tls", "tcp" and "udp" protocols. "tls" is used for securing
      # communication between flow exporter and flow aggregator. When "grpc" is configured,
      # mTLS will always be used.
      flowCollectorAddr: "flow-aggregator/flow-aggregator:4739:tls"

      # Provide flow poll interval as a duration string. This determines how often the
      # flow exporter dumps connections from the conntrack module. Flow poll interval
      # should be greater than or equal to 1s (one second).
      # Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
      flowPollInterval: "5s"

      # Provide the active flow export timeout, which is the timeout after which a flow
      # record is sent to the collector for active flows. Thus, for flows with a continuous
      # stream of packets, a flow record will be exported to the collector once the elapsed
      # time since the last export event is equal to the value of this timeout.
      # Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
      activeFlowExportTimeout: "5s"

      # Provide the idle flow export timeout, which is the timeout after which a flow
      # record is sent to the collector for idle flows. A flow is considered idle if no
      # packet matching this flow has been observed since the last export event.
      # Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
      idleFlowExportTimeout: "15s"

      # Provide the set of protocols to filter specific flows that will be
      # exported. Invalid protocols will not error and instead a warning is
      # logged on the antrea agent. By default the full set of supported
      # protocols are exported which are:
      # "tcp", "udp", "sctp"
      protocolFilter: nil    

Please note that the default value for flowExporter.flowCollectorAddr is "flow-aggregator/flow-aggregator:4739:tls", which enables the Flow Exporter to connect the Flow Aggregator Service, assuming it is running in the same K8 cluster with the Name and Namespace set to flow-aggregator. If you deploy the Flow Aggregator Service with a different Name and Namespace, then set flowExporter.flowCollectorAddr appropriately.

Please note that the default values for flowExporter.flowPollInterval, flowExporter.activeFlowExportTimeout, and flowExporter.idleFlowExportTimeout parameters are set to 5s, 5s, and 15s, respectively. TLS communication between the Flow Exporter and the Flow Aggregator is enabled by default. Please modify them as per your requirements.

Configuration pre Antrea v1.13

Prior to the Antrea v1.13 release, the flowExporter option group in the Antrea Agent configuration did not exist. To enable the Flow Exporter feature, one simply needed to enable the feature gate, and the Flow Exporter related configuration could be configured using the (now deprecated) flowCollectorAddr, flowPollInterval, activeFlowExportTimeout, idleFlowExportTimeout parameters.

IPFIX Information Elements (IEs) in a Flow Record

There are 43 IPFIX IEs in each exported flow record, which are defined in the IANA-assigned IE registry, the Reverse IANA-assigned IE registry and the Antrea IE registry. The reverse IEs are used to provide bi-directional information about the flow. The Enterprise ID is 0 for IANA-assigned IE registry, 29305 for reverse IANA IE registry, 56506 for Antrea IE registry. All the IEs used by the Antrea Flow Exporter are listed below:

IEs from IANA-assigned IE Registry

IEs from Reverse IANA-assigned IE Registry

IEs from Antrea IE Registry

Supported Capabilities

Types of Flows and Associated Information

Currently, the Flow Exporter feature provides visibility for Pod-to-Pod, Pod-to-Service and Pod-to-External network flows along with the associated statistics such as data throughput (bits per second), packet throughput (packets per second), cumulative byte count and cumulative packet count. Pod-To-Service flow visibility is supported only when Antrea Proxy enabled, which is the case by default starting with Antrea v0.11. In the future, we will enable the support for External-To-Service flows.

Kubernetes information such as Node name, Pod name, Pod Namespace, Service name, NetworkPolicy name and NetworkPolicy Namespace, is added to the flow records. Network Policy Rule Action (Allow, Reject, Drop) is also supported for both Antrea-native NetworkPolicies and K8s NetworkPolicies. For K8s NetworkPolicies, connections dropped due to isolated Pod behavior will be assigned the Drop action. For flow records that are exported from any given Antrea Agent, the Flow Exporter only provides the information of Kubernetes entities that are local to the Antrea Agent. In other words, flow records are only complete for intra-Node flows, but incomplete for inter-Node flows. It is the responsibility of the Flow Aggregator to correlate flows from the source and destination Nodes and produce complete flow records.

Both Flow Exporter and Flow Aggregator are supported in IPv4 clusters, IPv6 clusters and dual-stack clusters.

Connection Metrics

We support following connection metrics as Prometheus metrics that are exposed through Antrea Agent apiserver endpoint: antrea_agent_conntrack_total_connection_count, antrea_agent_conntrack_antrea_connection_count, antrea_agent_denied_connection_count, antrea_agent_conntrack_max_connection_count, and antrea_agent_flow_collector_reconnection_count

Flow Aggregator

The Flow Aggregator consists of a K8s Deployment and Service. It has 2 main modes of operation:

• Aggregate (default): in this mode, the Flow Aggregator stores, correlates and aggregates the flow records received from the Flow Exporter of Antrea Agents. For more information about this mode, including installation instructions, refer to the Aggregate Mode section.
• Proxy (new in Antrea v2.3): in this mode, the Flow Aggregator operates statelessly, and the flow records received from the Flow Exporter of Antrea Agents are sent directly to an IPFIX collector, without buffering or correlation / aggregation. For more information about this mode, including installation instructions, refer to the Proxy Mode section.

The Flow Aggregator is implemented as an IPFIX mediator. It consists of an IPFIX Collector Process, an IPFIX Intermediate Process, and an IPFIX Exporter Process. We use the go-ipfix library to implement the Flow Aggregator. The “Intermediate Process” differs greatly based on whether the Flow Aggregator operates in Aggregate or Proxy mode.

Deciding which mode to use

If you are looking to export “raw” IPFIX records to an external IPFIX collector, and want to minimize the footprint of the Flow Aggregator, you may be interested in the Proxy mode. Note that the external IPFIX collector will be responsible for correlating flow records generated by the source and destination K8s Nodes, based on the 5-tuple (if the source and destination Nodes are the same, flow records will be sent by a single Flow Exporter). Note that the same information will not be present based on whether the record was exported from the source or destination. For example, for Pod-to-Service traffic, destination Service information will only be included in the source records. See #6773 for more background on why Proxy mode was introduced. Note that Proxy mode cannot be used with a non-IPFIX destination collector, and is only available starting with Antrea v2.3.

Otherwise, the default Aggregate mode is probably right for you. It supports all available destination collectors (e.g., ClickHouse), and its output is easier to consume as the source and destination records have already been correlated / aggregated.

Aggregate Mode

Installation

To deploy a released version of the Flow Aggregator, pick a deployment manifest from the list of releases. For any given release <TAG> (e.g. v2.3.0), you can deploy Flow Aggregator as follows:

kubectl apply -f https://github.com/antrea-io/antrea/releases/download/<TAG>/flow-aggregator.yml

We recommend updating the manifest with your desired configuration before applying it.

The following configuration parameters have to be provided through the Flow Aggregator ConfigMap. flowCollector is mandatory for go-ipfix collector, and clickHouse is mandatory for Grafana Flow Collector. We provide an example value for this parameter in the following snippet.

• If you have deployed the go-ipfix collector, then please set flowCollector.enable to true and use the address for flowCollector.address: <Ipfix-Collector Cluster IP>:<port>:<tcp|udp>
• If you have deployed the Grafana Flow Collector, then please enable the collector by setting clickHouse.enable to true. If it is deployed following the deployment steps, the ClickHouse server is already exposed via a K8s Service, and no further configuration is required. If a different FQDN or IP is desired, please use the URL for clickHouse.databaseURL in the following format: <protocol>://<ClickHouse server FQDN or IP>:<ClickHouse port>.

Configuring secure connections to the ClickHouse database

Starting with Antrea v1.13, you can enable TLS when connecting to the ClickHouse Server by setting clickHouse.databaseURL with protocol tls or https. You can also change the value of clickHouse.tls.insecureSkipVerify to determine whether to skip the verification of the server’s certificate. If you want to provide a custom CA certificate, you can set clickHouse.tls.caCert to true and the flow Aggregator will read the certificate key pair from theclickhouse-ca Secret.

Make sure to follow the following form when creating the clickhouse-ca Secret with the custom CA certificate:

apiVersion: v1
kind: Secret
metadata:
  name: clickhouse-ca
  namespace: flow-aggregator
data:
  ca.crt: <BASE64 ENCODED CA CERTIFICATE>

You can use kubectl apply -f <PATH TO SECRET YAML> to create the above Secret, or use kubectl create secret:

kubectl create secret generic clickhouse-ca -n flow-aggregator --from-file=ca.crt=<PATH TO CA CERTIFICATE>

Prior to Antrea v1.13, secure connections to ClickHouse are not supported, and TCP is the only supported protocol when connecting to the ClickHouse server from the Flow Aggregator.

Example of flow-aggregator.conf

flow-aggregator.conf: |  
  # Provide the active flow record timeout as a duration string. This determines
  # how often the flow aggregator exports the active flow records to the flow
  # collector. Thus, for flows with a continuous stream of packets, a flow record
  # will be exported to the collector once the elapsed time since the last export
  # event in the flow aggregator is equal to the value of this timeout.
  # Note that a record is only exported if a matching flow record has been
  # received from an Antrea Agent since the last export, hence this timeout is
  # only honored as long as it is not smaller than the Agent's
  # activeFlowExportTimeout ("5s" by default), which is the case with the
  # default configuration.
  # Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
  activeFlowRecordTimeout: 60s

  # Provide the inactive flow record timeout as a duration string. This determines
  # how long the flow aggregator retains the aggregated record for a flow after the
  # last matching record was received: once this timeout expires, the aggregated
  # record is deleted, and a subsequent record for the same flow starts a new
  # aggregation. When this timeout is greater than activeFlowRecordTimeout (which is
  # the case by default), the deletion never causes a record to be exported, as any
  # new data has already been exported when the active flow record timeout last
  # expired.
  # Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
  inactiveFlowRecordTimeout: 90s

  # Provide the transport protocol for the flow aggregator collecting process, which is tls, tcp or udp.
  aggregatorTransportProtocol: "tls"

  # Provide an extra DNS name or IP address of flow aggregator for generating TLS certificate.
  flowAggregatorAddress: ""

  # recordContents enables configuring some fields in the flow records. Fields can
  # be excluded to reduce record size, but some features or external tooling may
  # depend on these fields.
  recordContents:
    # Determine whether source and destination Pod labels will be included in the flow records.
    podLabels: false

  # apiServer contains APIServer related configuration options.
  apiServer:
    # The port for the flow-aggregator APIServer to serve on.
    apiPort: 10348

    # Comma-separated list of Cipher Suites. If omitted, the default Go Cipher Suites will be used.
    # https://golang.org/pkg/crypto/tls/#pkg-constants
    # Note that TLS1.3 Cipher Suites cannot be added to the list. But the apiserver will always
    # prefer TLS1.3 Cipher Suites whenever possible.
    tlsCipherSuites: ""

    # TLS min version from: VersionTLS10, VersionTLS11, VersionTLS12, VersionTLS13.
    tlsMinVersion: ""

  # flowCollector contains external IPFIX or JSON collector related configuration options.
  flowCollector:
    # Enable is the switch to enable exporting flow records to external flow collector.
    enable: false

    # Provide the flow collector address as string with format <IP>:<port>[:<proto>], where proto is tcp or udp.
    # If no L4 transport proto is given, we consider tcp as default.
    address: ""

    # Provide the 32-bit Observation Domain ID which will uniquely identify this instance of the flow
    # aggregator to an external flow collector. If omitted, an Observation Domain ID will be generated
    # from the persistent cluster UUID generated by Antrea. Failing that (e.g. because the cluster UUID
    # is not available), a value will be randomly generated, which may vary across restarts of the flow
    # aggregator.
    #observationDomainID:

    # Provide format for records sent to the configured flow collector.
    # Supported formats are IPFIX and JSON.
    recordFormat: "IPFIX"

  # clickHouse contains ClickHouse related configuration options.
  clickHouse:
    # Enable is the switch to enable exporting flow records to ClickHouse.
    enable: false

    # Database is the name of database where Antrea "flows" table is created.
    database: "default"

    # DatabaseURL is the url to the database. Provide the database URL as a string with format
    # <Protocol>://<ClickHouse server FQDN or IP>:<ClickHouse port>. The protocol has to be
    # one of the following: "tcp", "tls", "http", "https". When "tls" or "https" is used, tls
    # will be enabled.
    databaseURL: "tcp://clickhouse-clickhouse.flow-visibility.svc:9000"

    # TLS configuration options, when using TLS to connect to the ClickHouse service.
    tls:
      # InsecureSkipVerify determines whether to skip the verification of the server's certificate chain and host name.
      # Default is false.
      insecureSkipVerify: false

      # CACert indicates whether to use custom CA certificate. Default root CAs will be used if this field is false.
      # If true, a Secret named "clickhouse-ca" must be provided with the following keys:
      # ca.crt: <CA certificate>
      caCert: false

    # Debug enables debug logs from ClickHouse sql driver.
    debug: false

    # Compress enables lz4 compression when committing flow records.
    compress: true

    # CommitInterval is the periodical interval between batch commit of flow records to DB.
    # Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
    # The minimum interval is 1s based on ClickHouse documentation for best performance.
    commitInterval: "8s"

Please note that the default values for activeFlowRecordTimeout, inactiveFlowRecordTimeout, aggregatorTransportProtocol parameters are set to 60s, 90s and tls respectively. Please make sure that aggregatorTransportProtocol and protocol of flowCollectorAddr in agent-agent.conf are set to tls to guarantee secure communication works properly. Protocol of flowCollectorAddr and aggregatorTransportProtocol must always match, so TLS must either be enabled for both sides or disabled for both sides. Please modify the parameters as per your requirements.

Please note that the default value for recordContents.podLabels is false, which indicates source and destination Pod labels will not be included in the flow records exported to flowCollector and clickHouse. If you would like to include them, you can modify the value to true.

Please note that the default value for apiServer.apiPort is 10348, which is the port used to expose the Flow Aggregator’s APIServer. Please modify the parameters as per your requirements.

Please note that the default value for clickHouse.commitInterval is 8s, which is based on experiment results to achieve best ClickHouse write performance and data retention. Based on ClickHouse recommendation for best performance, this interval is required be no shorter than 1s. Also note that flow aggregator has a cache limit of ~500k records for ClickHouse-Grafana collector. If clickHouse.commitInterval is set to a value too large, there’s a risk of losing records.

IPFIX Information Elements (IEs) in an Aggregated Flow Record

In addition to IPFIX information elements provided in the above section, the Flow Aggregator adds the following fields to the flow records.

IEs from Antrea IE Registry

Supported Capabilities

Storage of Flow Records

Flow Aggregator stores the received flow records from Antrea Agents in a hash map, where the flow key is 5-tuple of a network connection. 5-tuple consists of Source IP, Destination IP, Source Port, Destination Port and Transport protocol. Therefore, Flow Aggregator maintains one flow record for any given connection, and this flow record gets updated till the connection in the Kubernetes cluster becomes invalid.

Correlation of Flow Records

In the case of inter-Node flows, there are two flow records, one from the source Node, where the flow originates from, and another one from the destination Node, where the destination Pod resides. Both the flow records contain incomplete information as mentioned here. Flow Aggregator provides support for the correlation of the flow records from the source Node and the destination Node, and it exports a single flow record with complete information for both inter-Node and intra-Node flows.

Aggregation of Flow Records

Flow Aggregator aggregates the flow records that belong to a single connection. As part of aggregation, fields such as flow timestamps, flow statistics etc. are updated. For the purpose of updating flow statistics fields, Flow Aggregator introduces the new fields in Antrea Enterprise IPFIX registry corresponding to the Source Node and Destination Node, so that flow statistics from different Nodes can be preserved.

Antctl Support

antctl can access the Flow Aggregator API to dump flow records and print metrics about flow record processing. Refer to the antctl documentation for more information.

Proxy Mode (v2.3 and above)

Installation

To deploy a released version of the Flow Aggregator, pick a deployment manifest from the list of releases. For any given release <TAG> (e.g. v2.3.0), you can deploy Flow Aggregator as follows:

kubectl apply -f https://github.com/antrea-io/antrea/releases/download/<TAG>/flow-aggregator.yml

We recommend updating the manifest with your desired configuration before applying it. In order to proxy records to an external collector, you will need to populate the flowCollector section of the Flow Aggregator ConfigMap.

Alternatively, you can use Helm to easily install the latest released version of the Flow Aggregator:

# If you are installing an Antrea Helm chart for the first time:
helm repo add antrea https://charts.antrea.io
# To update available charts:
helm repo update

helm install flow-aggregator antrea/flow-aggregator --set mode=Proxy,flowCollector.enable=true,flowCollector.address="<addr:port:proto>" -n flow-aggregator --create-namespace

In Proxy mode, not all configuration parameters are applicable. flowCollector is the only supported collector, and activeFlowRecordTimeout / inactiveFlowRecordTimeout are not applicable. The FlowStreamService is not supported either: it is disabled by default in Proxy mode, and the Flow Aggregator refuses to start if flowStreamService.enable is set to true. If you start from the release manifest, remove the flowstream-grpc port and the observability.antrea.io/flow-stream-service label from the Service.

IPFIX Information Elements (IEs) in a Proxied Flow Record

In addition to IPFIX information elements provided in the above section, the Flow Aggregator adds the following fields to the flow records before forwarding them to the external collector:

FlowStreamService (alpha)

The Flow Aggregator can expose a gRPC server-streaming API, FlowStreamService, on port 14740, which streams flow records out of its in-memory record buffer to consumers such as antrea-ui: historical records first, then live ones if the client asked to follow. It is alpha, and enabled by default in Aggregate mode. It is not supported in Proxy mode: records are not correlated in that mode, so a record only carries the policy information of the Node which exported it, while the redaction described in What a client sees of each endpoint relies on records describing the whole connection. It can be disabled with:

flowStreamService:
  enable: false

The connection uses server-side TLS with the same self-signed certificate as the gRPC collector.

When the FlowStreamService is enabled, the Flow Aggregator Service carries the label observability.antrea.io/flow-stream-service: "true"; it does not carry it otherwise. As a cluster can run several Flow Aggregators, and only some of them may expose the FlowStreamService (for example, a Flow Aggregator running in Proxy mode for IPFIX export does not), clients can use this label to discover the FlowStreamService to connect to, instead of hardcoding its Namespace and name:

kubectl get services -A -l observability.antrea.io/flow-stream-service=true

Authenticating to the FlowStreamService

Clients must present a Kubernetes credential, as one of:

• a bearer token in gRPC metadata, as authorization: Bearer <token>, which the Flow Aggregator validates with a TokenReview. kubectl create token <serviceaccount> and, in a cluster whose kube-apiserver trusts an OIDC issuer, an OIDC ID token both work.
• an X.509 client certificate, presented as the TLS client credential of the gRPC connection, which the Flow Aggregator verifies against the cluster’s client CA bundle — the one kube-apiserver publishes in the extension-apiserver-authentication ConfigMap. This covers the clients whose only credential is a short-lived client certificate, such as one issued by a Pinniped Concierge. Note that the Flow Aggregator’s own client certificate is signed by its self-signed CA rather than the cluster’s, so it cannot be used here.

A call carrying both is identified by its certificate, which is the order kube-apiserver itself tries them in; if that certificate does not verify, the bearer token is tried next.

A stream whose credential is missing, malformed or invalid is rejected with UNAUTHENTICATED before any record is read.

Authorizing flow visibility

What a client then receives is decided by ordinary Kubernetes RBAC on a virtual resource, flows in API group observability.antrea.io. Nothing serves that resource: it exists to be named in a Role or ClusterRole, the same way kubelet authorizes its own endpoints through nodes/proxy. The Flow Aggregator resolves each stream with SubjectAccessReviews, for which its ServiceAccount is already bound to system:auth-delegator.

Two permissions exist:

flows in a Namespace implies flows/identity there: an endpoint in that Namespace is disclosed in full without a separate identity check, provided the flows verbs held there include the one the stream was opened with (watch for a following stream, list otherwise). Granting flows/identity on its own is for the remaining case — being recognizable inside records someone receives through a different Namespace.

list authorizes a stream that drains the record buffer and closes; watch authorizes one that then follows live records. Granting only list therefore yields history-only access. watch is what the gRPC API’s follow flag maps onto.

A client must name the scope it wants to observe. There are two ways to consume flows:

• a subject with cluster-wide visibility sets the cluster-wide flag and streams the flows of every Namespace, including flows with no Namespace at all, over a single stream. This is a single cluster-wide check, which only a ClusterRoleBinding can satisfy;
• a subject with visibility in individual Namespaces names one Namespace per request, and opens one stream per Namespace it wants to observe.

A request naming more than one Namespace is rejected with INVALID_ARGUMENT, and a request naming a Namespace the client may not observe is rejected with PERMISSION_DENIED. Kubernetes offers no reverse lookup from a subject to the Namespaces it may access, so “show me everything I am allowed to see” is not something the Flow Aggregator can answer, which is why the scope is always explicit. The same distinction surfaces in the Antrea UI: selecting a Namespace is mandatory on the flow visibility tab unless the user has cluster-wide visibility.

A record is streamed if either of its endpoints is in the Namespaces the client was authorized for. Requiring both would hide exactly the cross-Namespace flows users are looking for — “my egress to a service I cannot see was dropped” is the common case.

Two ClusterRoles are shipped by default: antrea-flow-viewer, which grants flows, and antrea-flow-identity-viewer, which grants flows/identity. Each can be bound directly, in a Namespace or cluster-wide, and each is also aggregated into a built-in role, as the paragraphs after these examples describe. Examples:

# Let the "network-ops" group stream flows in the "frontend" Namespace.
kubectl create rolebinding flow-viewer -n frontend \
  --clusterrole antrea-flow-viewer --group network-ops

# Let it stream every flow in the cluster, including flows with no Namespace at all.
kubectl create clusterrolebinding flow-viewer \
  --clusterrole antrea-flow-viewer --group network-ops

# Let a group with no read access at all recognize kube-system's workloads inside
# records they already receive.
kubectl create rolebinding flow-identity -n kube-system \
  --clusterrole antrea-flow-identity-viewer --group some-group-without-read-access

antrea-flow-viewer carries rbac.authorization.k8s.io/aggregate-to-admin, so a Namespace administrator observes their own Namespace’s flows without any of the bindings above as long as they hold the “admin” role to that Namespace: a Namespace’s flows are treated as part of what owning that Namespace already entails. A true cluster administrator, holding * on *, has cluster-wide visibility for the same reason.

antrea-flow-identity-viewer carries rbac.authorization.k8s.io/aggregate-to-view instead, so who can already read the Namespace’s objects automatically gets flows/identity in that namespace, with no separate binding.

Both roles are cluster singletons: their names are fixed, they name no ServiceAccount and no release, and one copy of each serves every Flow Aggregator in the cluster. A second Helm release in the same cluster must instead set flowStreamService.installClusterRoles=false and bind against the roles the first release installed, since Helm will not adopt objects another release owns. A release in which the FlowStreamService is disabled, as is always the case in Proxy mode, does not install the roles.

Ownership follows the release that installed them, so uninstalling that release deletes both roles even while other Flow Aggregator releases are still running. To hand them to a surviving release, set installClusterRoles=true there and helm upgrade it: the roles are recreated and that release becomes their owner. Do this only once the previous owner is gone, since Helm will not let one release adopt what another still owns.

Existing bindings fail closed during this period. RBAC resolves a binding’s roleRef when it authorizes a request rather than when the binding is created, so a binding whose role is missing grants nothing in the meanwhile, and resumes working once the role is back.

Who currently holds flow visibility can be audited with:

kubectl get rolebindings,clusterrolebindings -A -o json | \
  jq '.items[] | select(.roleRef.name | startswith("antrea-flow-"))'

What a client sees of each endpoint

Each endpoint of a record — source and destination — is disclosed independently, according to the client’s permissions in that endpoint’s Namespace. A single record could carry one endpoint in full and the other redacted.

A destination Service’s IP sits at the Identity tier rather than the Flow tier, because it maps back to the Service it belongs to, so granting flows/identity in a Namespace discloses the IPs of that Namespace’s Services along with their names. There is no field carrying a Service’s name on its own: the name rides inside destination_service_port_name, which the Flow Exporter sets to namespace/name:portName, so that one field carries the Service’s Namespace and name as well as the port’s name.

Those Service fields follow the tier of the destination endpoint, which is resolved from destination_pod_namespace — the Namespace of the Pod that received the connection, after DNAT. The Service’s own Namespace, the ns/ prefix of destination_service_port_name, is normally that same Namespace, but the two can differ, because a Service’s endpoints are not constrained to its own Namespace: a selector-less Service in ns-a can be pointed at a Pod IP in ns-b by a manually created EndpointSlice, which AntreaProxy accepts. If a Service has endpoints in another Namespace, its name, UID and ClusterIP are disclosed to whoever may observe flows in the Namespace its backends live in, whatever the Service’s own Namespace has granted — the Service fields were tiered against ns-b’s consent, not ns-a’s. This is a documented caveat rather than something the tiering tries to undo: creating such a Service takes EndpointSlice write access in ns-a, and a Service’s name and ClusterIP are discoverable through cluster DNS by any Pod anyway.

The converse also holds. A destination endpoint with no Namespace at all, e.g. a hostNetwork backend, or a Multicluster endpoint in another cluster — sits at the Flow tier on every non-cluster-wide stream, so the Service fields are withheld even on a stream opened for the Service’s own Namespace. Nothing is over-disclosed there; the client simply cannot see which of its own Services such a flow went through.

The flow type is deliberately not redacted, so FLOW_TYPE_INTRA_NODE still reports that the two endpoints share a Node. Paired with an endpoint the client can place — its own Pods, whose Nodes it can read from the downward API with no Antrea grant at all — that discloses co-tenancy with the client’s own workloads even where the peer’s Node name was withheld, and the flow_types filter yields the same bit without reading the field. This is a deliberate trade: the internal-versus-external distinction the flow type carries is what lets a client tell a genuinely external endpoint from one whose Namespace was withheld, and collapsing INTRA_NODE to INTER_NODE would make the field mean something different on every stream. The peer’s Node is never named.

Each endpoint’s tier is reported on the record itself (source_disclosure / destination_disclosure), so that a client can tell a field that was withheld from one the Flow Aggregator never had — an empty policy Namespace, for instance, is what a cluster-scoped policy legitimately looks like. FULL is the zero value of both fields, so a record that was never redacted, and every record reaching a cluster-wide stream, reports both endpoints as fully disclosed without the Flow Aggregator having to modify anything. An unidentified endpoint keeps its Namespace only if the connection was allowed: withholding it for denied connections is what keeps a client from scanning the Pod CIDR and mapping IPs to Namespaces by reading back its own denied flows. Anything other than an explicit drop or reject counts as allowed, including “no policy applied at all”, so the mitigation only bites where the traffic really was denied by a policy: in a cluster without default-deny, scan probes not explicitly dropped or rejected read as allowed and disclose the peer Namespace. That is an accepted limit rather than an oversight. Namespace names may also be discoverable through CoreDNS reverse lookups for Service IPs and their endpoint IPs, though this does not cover every Pod.

A few fields belong to the record rather than to either endpoint. ipfix, which carries the IP of the Node that exported the record, is withheld unless both endpoints are disclosed in full: it is present on every record, including a Pod-to-Pod flow where neither Node is named. Because it is a record-level field, an endpoint can lose it from a record while its own source_disclosure / destination_disclosure still reports FULL: a marker describes an endpoint, not the record it arrives on.

proxy_snat_ip and proxy_snat_port are not redacted, even though they too hold a Node address. Only a from-external flow carries them — they are the address the ingress Node masqueraded the external client to, which is what the destination Pod actually observed as its peer. Such a flow has no source Namespace for a stream to be authorized for, so it only ever reaches a stream through its destination, a Namespace that stream may observe in full.

A rule action and the policy type are disclosed at every tier, even for a policy the client may not otherwise know about: losing them would lose “why did my connection fail”, which is most of the troubleshooting value, and knowing whether a cluster-scoped or a namespaced policy dropped the connection tells a client whether to escalate to the platform team or to the peer. The policy’s identity (its Namespace, name, UID and rule name) comes with the Identity tier, so that “which of your policies dropped my traffic” is answerable once the peer’s Namespace has granted flows/identity. Below that tier it stays hidden, which is what keeps a client from mapping the policy set of a Namespace that never consented to being identified.

Consequently, the same flow can look different depending on which Namespace a stream was opened for. Take a subject holding antrea-flow-viewer in ns-a and ns-b: it sees an ns-a-to-ns-b flow with both endpoints in full on either stream, since it could have opened the same stream for the peer’s Namespace anyway. A subject holding antrea-flow-viewer in ns-a only sees the ns-b endpoint at the Identity tier if it holds antrea-flow-identity-viewer in ns-b, and at the Flow tier otherwise. The verb matters too: a subject holding only list on flows in ns-b sees the ns-b endpoint in full on a non-following ns-a stream, but not on a following one, which falls back to what flows/identity there yields. A stream is always opened with exactly the scope the client named — a request is authorized in full or rejected outright, never narrowed — so there is nothing for the server to report back.

What administrators should know

• A grant on a shared infrastructure Namespace approximates a cluster-wide grant. kube-system, ingress, monitoring and service-mesh control planes have connections to nearly everything, so flow visibility there exposes flows involving nearly every workload.
• A wildcard Role confers flow visibility, and full disclosure with it. A namespaced Role granting apiGroups: ["*"], resources: ["*"], as tenants are sometimes given, includes flows in that Namespace. Its holder therefore sees that Namespace’s endpoints in full, Node and Egress details included, even in records received through some other Namespace, as long as the Role grants the verb that stream was opened with. Binding admin there has the same effect, because antrea-flow-viewer is aggregated to it. Binding view or edit does not: they stop at the Identity tier. RBAC cannot express “not reachable via a wildcard”, and there is no query for “who can do X”, so a cluster that hands out wildcard Roles should scan for inadvertent flows grants.
• A Service with endpoints in another Namespace discloses itself there. The Service fields of a record are tiered by the Namespace of the Pod that received the connection, so a Service whose EndpointSlices point outside its own Namespace is named to whoever may observe flows where its backends are. See the paragraph on the Service tier above.
• Revoking a grant takes up to 11 minutes to end an established stream. Authorization decisions, both allow and deny, are cached for 10 minutes — the shortest lifetime Kubernetes gives a projected ServiceAccount token — and an established stream re-checks its own scope once a minute, which is normally answered from that cache. The two add up in the worst case. Granting access is subject to the same delay. Opening a new stream is never served from a stale allow decision beyond the 10-minute window, and fails closed if the API server is unreachable; an established stream survives a control-plane blip. Peer disclosure that rests on a cluster-wide grant is kept the same way while its re-check cannot complete; disclosure that rests on a grant in the peer’s own Namespace falls back to what the subject holds cluster-wide instead, which is the Flow tier if it holds nothing there.
• Flow visibility is not derived from read access to the objects involved, but identity is. A subject granted watch flows in a Namespace sees Pod names and labels for it even without get pods there — flows and object read access are separate grants. flows/identity is the exception: antrea-flow-identity-viewer is aggregated to view, so anyone bound to the built-in view, edit or admin Role in a Namespace already holds it there, with no antrea-flow-* binding to show up in the audit command above.
• Identity changes mid-stream are invisible. The credential is resolved when the stream opens, so a token revoked or a group membership changed afterwards takes effect on the next connection, not on the current one.

Version skew between Flow Aggregator and Antrea Agent

As a rule, we recommend keeping the Flow Aggregator and the Antrea Agent at the same (minor) version. During upgrades, there will be a small time windows during which this is not possible. Prior to Antrea v2.3, the Flow Aggregator Pod may crash (and restart) in case of version mismatch, e.g., because of the introduction of new Information Elements. Starting with Antrea v2.3, the Flow Aggregator should be able to handle older or newer Agents gracefully. If possible, we do recommend upgrading the Flow Aggregator Deployment last (i.e., after all Antrea Agents have been upgraded). The version skew between the Flow Aggregator and Antrea Agents should be at most 4 minor versions, which also corresponds to the maximum supported version “delta” for Antrea upgrades, as per our versioning policy.

Quick Deployment

If you would like to quickly try Network Flow Visibility feature, you can deploy Antrea, the Flow Aggregator Service, the Grafana Flow Collector on the Vagrant setup.

Image-building Steps

Build required image under antrea by using make command:

make
make flow-aggregator-image

Deployment Steps

Given any external IPFIX flow collector, you can deploy Antrea and the Flow Aggregator Service on a default Vagrant setup by running the following commands:

./infra/vagrant/provision.sh
./infra/vagrant/push_antrea.sh --flow-collector <externalFlowCollectorAddress>

If you would like to deploy the Grafana Flow Collector, you can run the following command:

./infra/vagrant/provision.sh
./infra/vagrant/push_antrea.sh --flow-collector Grafana

Flow Collectors

Here we list two choices the external configured flow collector: go-ipfix collector and Grafana flow collector. For each collector, we introduce how to deploy it and how to output or visualize the collected flow records information.

Go-ipfix Collector

Deployment Steps

The go-ipfix collector can be built from go-ipfix library. It is used to collect, decode and log the IPFIX records.

• To deploy a released version of the go-ipfix collector, please choose one deployment manifest from the list of releases (supported after v0.5.2). For any given release (e.g. v0.5.2), you can deploy the collector as follows:

kubectl apply -f https://github.com/vmware/go-ipfix/releases/download/<TAG>/ipfix-collector.yaml

• To deploy the latest version of the go-ipfix collector (built from the main branch), use the checked-in deployment manifest:

kubectl apply -f https://raw.githubusercontent.com/vmware/go-ipfix/main/build/yamls/ipfix-collector.yaml

Go-ipfix collector also supports customization on its parameters: port and protocol. Please follow the go-ipfix documentation to configure those parameters if needed.

Output Flow Records

To output the flow records collected by the go-ipfix collector, use the command below:

kubectl logs <ipfix-collector-pod-name> -n ipfix

Grafana Flow Collector (migrated)

Starting with Antrea v1.8, support for the Grafana Flow Collector has been migrated to Theia.

The Grafana Flow Collector was added in Antrea v1.6.0. In Antrea v1.7.0, we start to move the network observability and analytics functionalities of Antrea to Project Theia, including the Grafana Flow Collector. Going forward, further development of the Grafana Flow Collector will be in the Theia repo. For the up-to-date version of Grafana Flow Collector and other Theia features, please refer to the Theia document.

ELK Flow Collector (removed)

Starting with Antrea v1.7, support for the ELK Flow Collector has been removed. Please consider using the Grafana Flow Collector instead, which is actively maintained.