Documentation
Introduction
- Overview
- Getting Started
- Support for K8s Installers
- Deploying on Kind
- Deploying on Minikube
- Configuration
Cloud Deployment
Reference
- Antrea Network Policy
- Antctl
- Architecture
- Traffic Encryption (Ipsec / WireGuard)
- Securing Control Plane
- Security considerations
- Troubleshooting
- OS-specific Known Issues
- OVS Pipeline
- Feature Gates
- Network Flow Visibility
- Traceflow Guide
- NoEncap and Hybrid Traffic Modes
- Egress Guide
- NodePortLocal Guide
- Antrea IPAM Guide
- Exposing Services of type LoadBalancer
- Versioning
- Antrea API Groups
- Antrea API Reference
Windows
Integrations
Cookbooks
Multicluster
Developer Guide
Project Information
Antrea Multi-cluster Architecture
Antrea Multi-cluster implements Multi-cluster Service API for Service communication across multiple Kubernetes clusters.
The below diagram depicts a basic multi-cluster topology in Antrea.
Given a set of Kubernetes clusters, there will be a leader cluster and several member clusters. By default, a leader cluster itself is also a member cluster of a ClusterSet. A cluster can also be configured as a dedicated leader cluster of multiple ClusterSets.
Terminology
ClusterSet is a placeholder name for a group of clusters with a high degree of mutual trust and shared ownership that share Services amongst themselves. Within a ClusterSet, Namespace sameness applies, which means all Namespaces with a given name are considered to be the same Namespace. The ClusterSet Custom Resource Definition(CRD) defines a ClusterSet including the leader and member clusters information.
The ClusterClaim CRD is used to claim a cluster itself as a member of a ClusterSet with unique cluster ID, to claim a ClusterSet with unique ClusterSet ID.
The MemberClusterAnnounce CRD declares a member cluster configuration to the leader cluster.
The Common Area is an abstraction in the Antrea Multi-cluster implementation provides a storage interface for resource export/import that can be read/written by all member and leader clusters in the ClusterSet. The Common Area is implemented with a Namespace in the leader cluster for a given ClusterSet.
Antrea Multi-cluster Controller
In a member cluster, Antrea Multi-cluster creates a Deployment that runs Antrea Multi-cluster Controller which is responsible for exporting resource to and importing resource from a leader cluster in a ClusterSet.
In a leader cluster, Antrea Multi-cluster creates a Deployment that runs Antrea Multi-cluster Controller which is responsible for converting resources from different member clusters into one encapsulated resource as long as these resources have the same kind and match Namespace sameness.
In ClusterSet initialization, Antrea Multi-cluster Controller in a member cluster watches ClusterSet, ClusterClaim, and creates a MemberClusterAnnounce in the leader cluster.
For exporting resources from the member cluster, it watches ServiceExport, Service and Endpoints
resources, encapsulates Services and Endpoints into ResourceExports according to ServiceExports,
and writes the ResourceExports to leader cluster. For resource importing, it watches ResourceImports
from leader cluster, and creates multi-cluster Services and Endpoints with a prefix antrea-mc-
plus exported Service name, and also ServiceImports which have the same name as the Service name.
In a leader cluster, for ClusterSet initialization, Antrea Multi-cluster controller watches and validates the ClusterSet and Clusterclaim. For resource export/import, it watches ResourceExports and encapsulates them into ResourceImports.
Service Export and Import
The current multi-cluster implementation supports Service discovery and Service export/import among member clusters. The above diagram depicts Antrea Multi-cluster resource export/import pipeline.
Given two Services in the member clusters - foo.ns.cluster1.local
and foo.ns.cluster2.local
,
multi-cluster Services may be generated by the following the resource export/import pipeline.
- Administrators create resource ServiceExports
foo
in Namespacens
in each of the clusters. - The Exporters in member clusters
cluster1
andcluster2
see ServiceExportfoo
, collect the associated Services and Endpoints, and create ResourceExports for them in the Common Area of the leader cluster. - The controller in the leader cluster sees ResourcesExports in the Common Area, computes multi-cluster
Service
cluster1-ns-foo-service
,cluster2-ns-foo-service
and associated Endpointscluster1-ns-foo-endpoints
,cluster2-ns-foo-endpoints
. - The controller creates ResourcesImport enclosing multi-cluster Service
ns-foo-service
and Endpointsns-foo-endpoints
in the Common Area. - The Importer in each member cluster watches ResourceImports; decapsulates them and gets Service
ns/antrea-mc-foo
, Endpointsns/antrea-mc-foo
, and creates the resources and as well as a ServiceImportns/foo
locally if they don’t exist or updates them if the resources have already been created by the Importer earlier.
Antrea Multi-cluster Service
Antrea Multi-cluster Controller only supports Service of type ClusterIP at this moment. In order to support multi-cluster Service access between member clusters, Antrea requires member clusters' Pod IPs are reachable and no overlapping between all member clusters.
When Antrea Multi-cluster Controller in member cluster watches ResourceImport’s creation event in leader cluster, it will create multi-cluster Service, Endpoints and ServiceImport locally. The Service Ports definition will be the same as exported Services, the Endpoints will be Pod IPs from all member clusters. The new created Antrea Multi-cluster Service is just like a regular Kubernetes Service, so Pods in a member cluster can access the multi-cluster Service as usual without any extra setting.
Antrea Multi-cluster policy enforcement
At this moment, Antrea does not support Pod-level policy enforcement for cross-cluster traffic. Access
towards Multi-cluster Services can be regulated with Antrea ClusterNetworkPolicy toService
rules. In
each member cluster, users can create an Antrea ClusterNetworkPolicy selecting Pods in that cluster, with
the imported Mutli-cluster Service name and Namespace in an egress toService
rule, and the Action to
take for traffic matching this rule. For more information regarding Antrea ClusterNetworkPolicy (ACNP),
refer to
this document.
Multi-cluster admins can also specify certain ClusterNetworkPolicies to be replicated across the entire ClusterSet. The ACNP to be replicated should be created as a ResourceExport in the leader cluster, and the resource export/import pipeline will ensure member clusters receive this ACNP spec to be replicated. Each member cluster’s Multi-cluster Controller will then create an ACNP in their respective clusters.