Documentation
Introduction
- Overview
- Getting Started
- Support for K8s Installers
- Deploying on Kind
- Deploying on Minikube
- Configuration
- Installing with Helm
Cloud Deployment
Reference
- Antrea Network Policy
- Antctl
- Architecture
- Traffic Encryption (Ipsec / WireGuard)
- Securing Control Plane
- Security considerations
- Troubleshooting
- OS-specific Known Issues
- OVS Pipeline
- Feature Gates
- Antrea Proxy
- Network Flow Visibility
- Traceflow Guide
- NoEncap and Hybrid Traffic Modes
- Egress Guide
- NodePortLocal Guide
- Antrea IPAM Guide
- Exposing Services of type LoadBalancer
- Traffic Control
- BGP Support
- Versioning
- Antrea API Groups
- Antrea API Reference
Windows
Integrations
Cookbooks
Multicluster
Developer Guide
Project Information
BGPPolicy
Table of Contents
- What is BGPPolicy?
- Prerequisites
- The BGPPolicy resource
- BGP router ID
- BGP Authentication
- Example Usage
- Using antctl
- Limitations
What is BGPPolicy?
BGPPolicy
is a custom resource that allows users to run a BGP process on selected Kubernetes Nodes and advertise
Service IPs, Pod IPs, and Egress IPs to remote BGP peers, facilitating the integration of Kubernetes workloads with an
external BGP-enabled network.
Prerequisites
BGPPolicy was introduced in Antrea v2.1 as an alpha feature. A feature gate, BGPPolicy
, must be enabled on antrea-agent
in the antrea-config
ConfigMap for the feature to work, like the following:
kind: ConfigMap
apiVersion: v1
metadata:
name: antrea-config
namespace: kube-system
data:
antrea-agent.conf: |
featureGates:
BGPPolicy: true
The BGPPolicy resource
A BGPPolicy in Kubernetes is a Custom Resource Definition (CRD) object.
The following manifest creates a BGPPolicy object. It will start a BGP process with ASN 64512
, listening on port 179
,
on Nodes labeled with bgp=enabled
. The process will advertise LoadBalancerIPs and ExternalIPs to a BGP peer at IP
address 192.168.77.200
, which has ASN 65001
and listens on port 179
:
apiVersion: crd.antrea.io/v1alpha1
kind: BGPPolicy
metadata:
name: example-bgp-policy
spec:
nodeSelector:
matchLabels:
bgp: enabled
localASN: 64512
listenPort: 179
advertisements:
service:
ipTypes: [LoadBalancerIP, ExternalIP]
bgpPeers:
- address: 192.168.77.200
asn: 65001
port: 179
NodeSelector
The nodeSelector
field selects which Kubernetes Nodes the BGPPolicy applies to based on the Node labels. The field is
mandatory.
Note: If multiple BGPPolicy objects select the same Node, the one with the earliest creation time will be chosen as the effective BGPPolicy.
LocalASN
The localASN
field defines the Autonomous System Number (ASN) that the local BGP process uses. The available private
ASN range is 64512-65535
. The field is mandatory.
ListenPort
The listenPort
field specifies the port on which the BGP process listens. The default value is 179. The valid port
range is 1-65535
.
Advertisements
The advertisements
field configures which IPs are advertised to BGP peers.
pod
: Specifies how to advertise Pod IPs. The Node IPAM Pod CIDRs will be advertised by settingpod:{}
. Note that IPs allocated by Antrea Flexible IPAM are not yet supported.egress
: Specifies how to advertise Egress IPs. All Egress IPs will be advertised by settingegress:{}
. A Node will only advertise Egress IPs which are local (i.e., assigned to the Node).service
: Specifies how to advertise Service IPs. TheipTypes
field lists the types of Service IPs to be advertised, which can includeClusterIP
,ExternalIP
, andLoadBalancerIP
.- All Nodes can advertise all ClusterIPs, respecting
internalTrafficPolicy
. IfinternalTrafficPolicy
is set toLocal
, a Node will only advertise ClusterIPs with at least one local Endpoint. - All Nodes can advertise all ExternalIPs and LoadBalancerIPs, respecting
externalTrafficPolicy
. IfexternalTrafficPolicy
is set toLocal
, a Node will only advertise IPs with at least one local Endpoint.
- All Nodes can advertise all ClusterIPs, respecting
BGPPeers
The bgpPeers
field lists the BGP peers to which the advertisements are sent.
address
: The IP address of the BGP peer.asn
: The Autonomous System Number of the BGP peer.port
: The port number on which the BGP peer listens. The default value is 179.multihopTTL
: The Time To Live (TTL) value used in BGP packets sent to the BGP peer, with a range of 1 to 255. The default value is 1.gracefulRestartTimeSeconds
: Specifies how long the BGP peer waits for the BGP session to re-establish after a restart before deleting stale routes, with a range of 1 to 3600 seconds. The default value is 120 seconds.
BGP router ID
The BGP router identifier (ID) is a 4-byte field that is usually represented as an IPv4 address. Antrea uses the following steps to choose the BGP router ID:
- If the
node.antrea.io/bgp-router-id
annotation is present on the Node and its value is a valid IPv4 address string, we will use the provided value. - Otherwise, for an IPv4-only or dual-stack Kubernetes cluster, the Node’s IPv4 address (assigned to the transport interface) is used.
- Otherwise, for IPv6-only clusters, a 32-bit integer will be generated by hashing the Node’s name, then converted to the string representation of an IPv4 address.
After this selection process, the node.antrea.io/bgp-router-id
annotation is added or updated as necessary to reflect
the selected BGP router ID.
The router ID is generated once and will not be updated if the Node configuration changes (e.g., if the Node’s IPv4 address is updated).
BGP Authentication
BGP authentication ensures that BGP sessions are established and maintained only with legitimate peers. Users can provide
authentication passwords for different BGP peering sessions by storing them in a Kubernetes Secret. The Secret must
be defined in the same Namespace as Antrea (kube-system
by default) and must be named antrea-bgp-passwords
.
By default, this Secret is not created, and BGP authentication is considered unconfigured for all BGP peers. If the
Secret is created like in the following example, each entry should have a key that is the concatenated string of the BGP
peer IP address and ASN (e.g., 192.168.77.100-65000
, 2001:db8::1-65000
), with the value being the password for that
BGP peer. If a given BGP peer does not have a corresponding key in the Secret data, then authentication is considered
disabled for that peer.
apiVersion: v1
kind: Secret
metadata:
name: antrea-bgp-passwords
namespace: kube-system
stringData:
192.168.77.100-65000: "password"
2001:db8::1-65000: "password"
type: Opaque
Example Usage
Combined Advertisements of Service, Pod, and Egress IPs
In this example, we will advertise Service IPs of types LoadBalancerIP and ExternalIPs, along with Pod CIDRs and Egress IPs from the selected Nodes to multiple remote BGP peers.
apiVersion: crd.antrea.io/v1alpha1
kind: BGPPolicy
metadata:
name: advertise-all-ips
spec:
nodeSelector:
matchLabels:
bgp: enabled
localASN: 64512
listenPort: 179
advertisements:
service:
ipTypes: [LoadBalancerIP, ExternalIP]
pod: {}
egress: {}
bgpPeers:
- address: 192.168.77.200
asn: 65001
port: 179
- address: 192.168.77.201
asn: 65001
port: 179
Advertise Egress IPs to external BGP peers with more than one hop
In this example, we configure the BGPPolicy to advertise Egress IPs from selected Nodes to a remote BGP peer located
multiple hops away from the cluster. It’s crucial to set the multihopTTL
to a value equal to or greater than the
number of hops, allowing BGP packets to traverse multiple hops to reach the peer.
apiVersion: crd.antrea.io/v1alpha1
kind: BGPPolicy
metadata:
name: advertise-all-egress-ips
spec:
nodeSelector:
matchLabels:
bgp: enabled
localASN: 64512
listenPort: 179
advertisements:
egress: {}
bgpPeers:
- address: 192.168.78.201
asn: 65001
port: 179
multihopTTL: 2
Using antctl
Please refer to the corresponding antctl page.
Limitations
- The routes received from remote BGP peers will not be installed. Therefore, you must ensure that the path from Nodes to the remote BGP network is properly configured and routable. This involves configuring your network infrastructure to handle the routing of traffic between your Kubernetes cluster and the remote BGP network.
- Only Linux Nodes are supported. The feature has not been validated on Windows Nodes, though theoretically it can work with Windows Nodes.
- Advanced BGP features such as BGP communities, route filtering, route reflection, confederations, and other BGP policy mechanisms defined in BGP RFCs are not supported.