Documentation
Introduction
- Overview
- Getting Started
- Support for K8s Installers
- Deploying on Kind
- Deploying on Minikube
- Configuration
- Installing with Helm
Cloud Deployment
Reference
- Antrea Network Policy
- Antctl
- Architecture
- Traffic Encryption (Ipsec / WireGuard)
- Securing Control Plane
- Security considerations
- Troubleshooting
- OS-specific Known Issues
- OVS Pipeline
- Feature Gates
- Antrea Proxy
- Network Flow Visibility
- Traceflow Guide
- NoEncap and Hybrid Traffic Modes
- Egress Guide
- NodePortLocal Guide
- Antrea IPAM Guide
- Exposing Services of type LoadBalancer
- Traffic Control
- Versioning
- Antrea API Groups
- Antrea API Reference
Windows
Integrations
Cookbooks
Multicluster
Developer Guide
Project Information
External Node
Table of Contents
- What is ExternalNode?
- Prerequisites
- The ExternalNode resource
- Install Antrea Agent on VM
- VM network configuration
- RBAC for antrea-agent
- Apply Antrea NetworkPolicy to ExternalNode
- OpenFlow pipeline
- Limitations
What is ExternalNode?
ExternalNode
is a CRD API that enables Antrea to manage the network connectivity
and security on a Non-Kubernetes Node (like a virtual machine or a bare-metal
server). It supports specifying which network interfaces on the external Node
are expected to be protected with Antrea NetworkPolicy rules. The virtual machine
or bare-metal server represented by an ExternalNode
resource can be either
Linux or Windows. “External Node” will be used to designate such a virtual
machine or bare-metal server in the rest of this document.
Antrea NetworkPolicies are applied to an external Node by leveraging the
ExternalEntity
resource. antrea-controller
creates an ExternalEntity
resource for each network interface specified in the ExternalNode
resource.
antrea-agent
is running on the external Node, and it controls network
connectivity and security by attaching the network interface(s) to an OVS bridge.
A
new OpenFlow pipeline has been implemented, dedicated to
the ExternalNode feature.
You may be interested in using this capability for the below scenarios:
- To apply Antrea NetworkPolicy to an external Node.
- You want the same security configurations on the external Node for all Operating Systems.
This guide demonstrates how to configure ExternalNode
to achieve the above
result.
Prerequisites
ExternalNode
is introduced in v1.8 as an alpha feature. The feature gate
ExternalNode
must be enabled in the antrea-controller
and antrea-agent
configuration. The configuration for antrea-controller
is modified in the
antrea-config
ConfigMap as follows for the feature to work:
apiVersion: v1
kind: ConfigMap
metadata:
name: antrea-config-dcfb6k2hkm
namespace: kube-system
data:
antrea-controller.conf: |
featureGates:
ExternalNode: true
The antrea-controller
implements the antrea
Service, which accepts
connections from each antrea-agent
and is an important part of the
NetworkPolicy implementation. By default, the antrea
Service has type
ClusterIP
. Because external Nodes run outside of the Kubernetes cluster, they
cannot directly access the ClusterIP
address. Therefore, the antrea
Service
needs to become externally-accessible, by changing its type to NodePort
or
LoadBalancer
.
Since antrea-agent
is running on an external Node which is not managed by
Kubernetes, a configuration file needs to be present on each machine where the
antrea-agent
is running, and the path to this file will be provided to the
antrea-agent
as a command-line argument. Refer to the
sample configuration
to learn theantrea-agent
configuration options when running on an external Node.
A further
section will provide detailed steps
for running the antrea-agent
on a VM.
The ExternalNode resource
An example ExternalNode
resource:
apiVersion: crd.antrea.io/v1alpha1
kind: ExternalNode
metadata:
name: vm1
namespace: vm-ns
labels:
role: db
spec:
interfaces:
- ips: [ "172.16.100.3" ]
name: ""
Note: Only one interface is supported for Antrea v1.8.
Name and Namespace
The name
field in an ExternalNode
uniquely identifies an external Node.
The ExternalNode
name is provided to antrea-agent
via an environment
variable NODE_NAME
, otherwise antrea-agent
will use the hostname to find
the ExternalNode
resource if NODE_NAME
is not set.
ExternalNode
resource is Namespace
scoped. The Namespace
is provided to
antrea-agent
with option externalNodeNamespace
in
antrea-agent.conf.
externalNodeNamespace: vm-ns
Interfaces
The interfaces
field specifies the list of the network interfaces expected to
be guarded by Antrea NetworkPolicy. At least one interface is required. Interface
name
or ips
is used to identify the target interface. The field ips
must be provided in the CRD, but name
is optional. Multiple IPs on a single
interface is supported. In the case that multiple interfaces
are configured,
name
must be specified for every interface
.
antrea-controller
creates an ExternalEntity
for each interface whenever an
ExternalNode
is created. The created ExternalEntity
has the following
characteristics:
- It is configured within the same Namespace as the
ExternalNode
. - The
name
is generated according to the following principles:- Use the
ExternalNode
name directly, if there is only one interface, and interface name is not specified. - Use the format
$ExternalNode.name-$hash($interface.name)[:5]
for other cases.
- Use the
- The
externalNode
field is set with theExternalNode
name. - The
owner
is referring to theExternalNode
resource. - All labels added on
ExternalNode
are copied to theExternalEntity
. - Each IP address of the interface is added as an endpoint in the
endpoints
list, and the interface name is used as the endpoint name if it is set.
The ExternalEntity
resource created for the above ExternalNode
interface
would look like this:
apiVersion: crd.antrea.io/v1alpha2
kind: ExternalEntity
metadata:
labels:
role: db
name: vm1
namespace: vm-ns
ownerReferences:
- apiVersion: v1alpha1
kind: ExternalNode
name: vm1
uid: 99b09671-72da-4c64-be93-17185e9781a5
resourceVersion: "5513"
uid: 5f360f32-7806-4d2d-9f36-80ce7db8de10
spec:
endpoints:
- ip: 172.16.100.3
externalNode: vm1
Install Antrea Agent on VM
Prerequisites on Kubernetes cluster
-
Enable
ExternalNode
feature on theantrea-controller
, and expose the antrea Service externally (e.g., as a NodePort Service). -
Create a Namespace for
antrea-agent
. This document will usevm-ns
as an example Namespace for illustration.kubectl create ns vm-ns
-
Create a ServiceAccount, ClusterRole and ClusterRoleBinding for
antrea-agent
as shown below. If you use a Namespace other thanvm-ns
, you need to update the VM RBAC manifest and changevm-ns
to the right Namespace.kubectl apply -f https://raw.githubusercontent.com/antrea-io/antrea/main/build/yamls/externalnode/vm-agent-rbac.yml
-
Create
antrea-agent.kubeconfig
file forantrea-agent
to access the K8S API server.CLUSTER_NAME="kubernetes" SERVICE_ACCOUNT="vm-agent" NAMESPACE="vm-ns" KUBECONFIG="antrea-agent.kubeconfig" APISERVER=$(kubectl config view -o jsonpath="{.clusters[?(@.name==\"$CLUSTER_NAME\")].cluster.server}") TOKEN=$(kubectl -n $NAMESPACE get secrets -o jsonpath="{.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='$SERVICE_ACCOUNT')].data.token}"|base64 --decode) kubectl config --kubeconfig=$KUBECONFIG set-cluster $CLUSTER_NAME --server=$APISERVER --insecure-skip-tls-verify=true kubectl config --kubeconfig=$KUBECONFIG set-credentials antrea-agent --token=$TOKEN kubectl config --kubeconfig=$KUBECONFIG set-context antrea-agent@$CLUSTER_NAME --cluster=$CLUSTER_NAME --user=antrea-agent kubectl config --kubeconfig=$KUBECONFIG use-context antrea-agent@$CLUSTER_NAME # Copy antrea-agent.kubeconfig to the VM
-
Create
antrea-agent.antrea.kubeconfig
file forantrea-agent
to access theantrea-controller
API server.# Specify the antrea-controller API server endpoint. Antrea-Controller needs # to be exposed via the Node IP or a public IP that is reachable from the VM ANTREA_API_SERVER="https://172.18.0.1:443" ANTREA_CLUSTER_NAME="antrea" NAMESPACE="vm-ns" KUBECONFIG="antrea-agent.antrea.kubeconfig" TOKEN=$(kubectl -n $NAMESPACE get secrets -o jsonpath="{.items[?(@.metadata.annotations['kubernetes\.io/service-account\.name']=='$SERVICE_ACCOUNT')].data.token}"|base64 --decode) kubectl config --kubeconfig=$KUBECONFIG set-cluster $ANTREA_CLUSTER_NAME --server=$ANTREA_API_SERVER --insecure-skip-tls-verify=true kubectl config --kubeconfig=$KUBECONFIG set-credentials antrea-agent --token=$TOKEN kubectl config --kubeconfig=$KUBECONFIG set-context antrea-agent@$ANTREA_CLUSTER_NAME --cluster=$ANTREA_CLUSTER_NAME --user=antrea-agent kubectl config --kubeconfig=$KUBECONFIG use-context antrea-agent@$ANTREA_CLUSTER_NAME # Copy antrea-agent.antrea.kubeconfig to the VM
-
Create an
ExternalNode
resource for the VM.After preparing the
ExternalNode
configuration yaml for the VM, we can apply it in the cluster.cat << EOF | kubectl apply -f - apiVersion: crd.antrea.io/v1alpha1 kind: ExternalNode metadata: name: vm1 namespace: vm-ns labels: role: db spec: interfaces: - ips: [ "172.16.100.3" ] name: "" EOF
Installation on Linux VM
Prerequisites on Linux VM
OVS needs to be installed on the VM. For more information about OVS installation please refer to the getting-started guide.
Installation steps on Linux VM
-
Build
antrea-agent
binary in the root of the antrea code tree and copy theantrea-agent
binary from thebin
directory to the Linux VM.make docker-bin
-
Copy configuration files to the VM, including antrea-agent.conf, which specifies agent configuration parameters;
antrea-agent.antrea.kubeconfig
andantrea-agent.kubeconfig
, which were generated in steps 4 and 5 of Prerequisites on Kubernetes cluster. -
Bootstrap
antrea-agent
using one of these 2 methods:-
Bootstrap
antrea-agent
using the installation script as shown below (Ubuntu 18.04 and 20.04 only)../install-vm.sh --ns vm-ns --bin ./antrea-agent --config ./antrea-agent.conf \ --kubeconfig ./antrea-agent.kubeconfig \ --antrea-kubeconfig ./antrea-agent.antrea.kubeconfig --nodename vm1
-
Bootstrap
antrea-agent
manually. First edit theantrea-agent.conf
file to setclientConnection
,antreaClientConnection
andexternalNodeNamespace
to the correct values.AGENT_NAMESPACE="vm-ns" AGENT_CONF_PATH="/etc/antrea" mkdir -p $AGENT_CONF_PATH # Copy antrea-agent kubeconfig files cp ./antrea-agent.kubeconfig $AGENT_CONF_PATH cp ./antrea-agent.antrea.kubeconfig $AGENT_CONF_PATH # Update clientConnection and antreaClientConnection sed -i "s|kubeconfig: |kubeconfig: $AGENT_CONF_PATH/|g" antrea-agent.conf sed -i "s|#externalNodeNamespace: default|externalNodeNamespace: $AGENT_NAMESPACE|g" antrea-agent.conf # Copy antrea-agent configuration file cp ./antrea-agent.conf $AGENT_CONF_PATH
Then create
antrea-agent
service. Below is a sample snippet to startantrea-agent
as a service on Ubuntu 18.04 or later:Note: Environment variable
NODE_NAME
needs to be set in the service configuration, if the VM’s hostname is different from the name defined in theExternalNode
resource.AGENT_BIN_PATH="/usr/sbin" AGENT_LOG_PATH="/var/log/antrea" mkdir -p $AGENT_BIN_PATH mkdir -p $AGENT_LOG_PATH cat << EOF > /etc/systemd/system/antrea-agent.service Description="antrea-agent as a systemd service" After=network.target [Service] Environment="NODE_NAME=vm1" ExecStart=$AGENT_BIN_PATH/antrea-agent \ --config=$AGENT_CONF_PATH/antrea-agent.conf \ --logtostderr=false \ --log_file=$AGENT_LOG_PATH/antrea-agent.log Restart=on-failure [Install] WantedBy=multi-user.target EOF sudo systemctl daemon-reload sudo systemctl enable antrea-agent sudo systemctl start antrea-agent
-
Installation on Windows VM
Prerequisites on Windows VM
-
Enable the Windows Hyper-V optional feature on Windows VM.
Install-WindowsFeature Hyper-V-Powershell Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V -All -NoRestart
-
OVS needs to be installed on the VM. For more information about OVS installation please refer to the Antrea Windows documentation.
-
Download nssm which will be used to create the Windows service for
antrea-agent
.
Note: Only Windows Server 2019 is supported in the first release at the moment.
Installation steps on Windows VM
-
Build
antrea-agent
binary in the root of the antrea code tree and copy theantrea-agent
binary from thebin
directory to the Windows VM.#! /bin/bash make docker-windows-bin
-
Copy antrea-agent.conf,
antrea-agent.kubeconfig
andantrea-agent.antrea.kubeconfig
files to the VM. Please refer to the step 2 of Installation on Linux VM section for more information.$WIN_AGENT_CONF_PATH="C:\antrea-agent\conf" New-Item -ItemType Directory -Force -Path $WIN_AGENT_CONF_PATH # Copy antrea-agent kubeconfig files Copy-Item .\antrea-agent.kubeconfig $WIN_AGENT_CONF_PATH Copy-Item .\antrea-agent.antrea.kubeconfig $WIN_AGENT_CONF_PATH # Copy antrea-agent configuration file Copy-Item .\antrea-agent.conf $WIN_AGENT_CONF_PATH
-
Bootstrap
antrea-agent
using one of these 2 methods:-
Bootstrap
antrea-agent
using the installation script as shown below (only Windows Server 2019 is tested and supported)..\Install-vm.ps1 -Namespace vm-ns -BinaryPath .\antrea-agent.exe ` -ConfigPath .\antrea-agent.conf -KubeConfigPath .\antrea-agent.kubeconfig ` -AntreaKubeConfigPath .\antrea-agent.antrea.kubeconfig ` -InstallDir C:\antrea-agent -NodeName vm1
-
Bootstrap
antrea-agent
manually. First edit theantrea-agent.conf
file to setclientConnection
,antreaClientConnection
andexternalNodeNamespace
to the correct values. Configure environment variableNODE_NAME
if the VM’s hostname is different from the name defined in theExternalNode
resource.[Environment]::SetEnvironmentVariable("NODE_NAME", "vm1") [Environment]::SetEnvironmentVariable("NODE_NAME", "vm1", [System.EnvironmentVariableTarget]::Machine)
Then create
antrea-agent
service using nssm. Below is a sample snippet to startantrea-agent
as a service:$WIN_AGENT_BIN_PATH="C:\antrea-agent" $WIN_AGENT_LOG_PATH="C:\antrea-agent\logs" New-Item -ItemType Directory -Force -Path $WIN_AGENT_BIN_PATH New-Item -ItemType Directory -Force -Path $WIN_AGENT_LOG_PATH Copy-Item .\antrea-agent.exe $WIN_AGENT_BIN_PATH nssm.exe install antrea-agent $WIN_AGENT_BIN_PATH\antrea-agent.exe --config $WIN_AGENT_CONF_PATH\antrea-agent.conf --log_file $WIN_AGENT_LOG_PATH\antrea-agent.log --logtostderr=false nssm.exe start antrea-agent
-
VM network configuration
antrea-agent
uses the interface IPs or name to find the network interface on
the external Node, and then attaches it to the OVS bridge. The network interface
is attached to OVS as uplink, and a new OVS internal Port is created to take over
the uplink interface’s IP/MAC and routing configurations. On Windows, the DNS
configurations are also moved to the OVS internal port from uplink. Before
attaching the uplink to OVS, the network interface is renamed with a suffix
“~”, and OVS internal port is configured with the original name of the uplink.
As a result, IP/MAC/routing entries are seen on a network interface configuring
with the same name on the external Node.
The outbound traffic sent from the external Node enters OVS from the internal port, and finally output from the uplink, and the inbound traffic enters OVS from the uplink and output to the internal port. The IP packet is processed by the OpenFlow pipeline, and the non-IP packet is forwarded directly.
The following diagram depicts the OVS bridge and traffic forwarding on an external Node:
RBAC for antrea-agent
An external Node is regarded as an untrusted entity on the network. To follow
the least privilege principle, the RBAC configuration for antrea-agent
running on an external Node is as follows:
- Only
get
,list
andwatch
permissions are given on resourceExternalNode
- Only
update
permission is given on resourceantreaagentinfos
, andcreate
permission is moved toantrea-controller
For more details please refer to vm-agent-rbac.yml
antrea-agent
reports its status by updating the antreaagentinfo
resource
which is created with the same name as the ExternalNode
. antrea-controller
creates an antreaagentinfo
resource for each new ExternalNode
, and then
antrea-agent
updates it every minute with its latest status. antreaagentinfo
is deleted by antrea-controller
when the ExternalNode
is deleted.
Apply Antrea NetworkPolicy to ExternalNode
Antrea NetworkPolicy configuration
An Antrea NetworkPolicy is applied to an ExternalNode
by providing an
externalEntitySelector
in the appliedTo
field. The ExternalEntity
resource is automatically created for each interface of an ExternalNode
.
ExternalEntity
resources are used by antrea-controller
to process the
NetworkPolicies, and each antrea-agent
(including those running on external
Nodes) receives the appropriate internal AntreaNetworkPolicy objects.
Following types of (from/to) network peers are supported in an Antrea NetworkPolicy applied to an external Node:
- ExternalEntities selected by an
externalEntitySelector
- An
ipBlock
- A FQDN address in an egress rule
Following actions are supported in an Antrea NetworkPolicy applied to an external Node:
- Allow
- Drop
- Reject
Below is an example of applying an Antrea NetworkPolicy to the external Nodes
labeled with role=db
to reject SSH connections from IP “172.16.100.5” or from
other external Nodes labeled with role=front
:
kind: NetworkPolicy
metadata:
name: anp1
namespace: vm-ns
spec:
priority: 9000.0
appliedTo:
- externalEntitySelector:
matchLabels:
role: db
ingress:
- action: Reject
ports:
- protocol: TCP
port: 22
from:
- externalEntitySelector:
matchLabels:
role: front
- ipBlock:
cidr: 172.16.100.5/32
Bypass Antrea NetworkPolicy
In some cases, users may want some particular traffic to bypass Antrea
NetworkPolicy rules on an external Node, e.g.,the SSH connection from a special
host to the external Node. policyBypassRules
can be added in the agent
configuration to define traffic that needs to bypass NetworkPolicy enforcement.
Below is a configuration example:
policyBypassRules:
- direction: ingress
protocol: tcp
cidr: 1.1.1.1/32
port: 22
The direction
can be ingress
or egress
. The supported protocols include:
tcp
,udp
, icmp
and ip
. The cidr
gives the peer address, which is the
destination in an egress
rule, and the source in an ingress
rule. For tcp
and udp
protocols, the port
is required to specify the destination port.
OpenFlow pipeline
A new OpenFlow pipeline is implemented by antrea-agent
dedicated for
ExternalNode
feature.
Non-IP packet
NonIPTable
is a new OpenFlow table introduced only on external Nodes,
which is dedicated to all non-IP packets. A non-IP packet is forwarded between
the pair ports directly, e.g., a non-IP packet entering OVS from the uplink
interface is output to the paired internal port, and a packet from the internal
port is output to the uplink.
IP packet
A new OpenFlow pipeline is set up on external Nodes to process IP packets. Antrea NetworkPolicy enforcement is the major function in this new pipeline, and the OpenFlow tables used are similar to the Pod pipeline. No L3 routing is provided on an external Node, and a simple L2 forwarding policy is implemented. OVS connection tracking is used to assist the NetworkPolicy function; as a result only the first packet is validated by the OpenFlow entries, and the subsequent packets in an accepted connection are allowed directly.
- Egress/Ingress Tables
Table XgressSecurityClassifierTable
is installed in both stageEgressSecurity
and stageIngressSecurity
, which is used to install the OpenFlow entries for
the
policyBypassRules
in the agent configuration.
This is an example of the OpenFlow entry for the above configuration:
table=IngressSecurityClassifier, priority=200,ct_state=+new+trk,tcp,nw_src=1.1.1.1,tp_dst=22 actions=resubmit(,IngressMetric)
Other OpenFlow tables in stageEgressSecurity
and stageIngressSecurity
are
the same as those installed on a Kubernetes worker Node. For more details about
these tables, please refer to the general
introduction
of Antrea OVS pipeline.
- L2 Forwarding Tables
L2ForwardingCalcTable
is used to calculate the expected output port of an IP
packet. As the pair ports with the internal port and uplink always exist on the
OVS bridge, and both interfaces are configured with the same MAC address, the
match condition of an OpenFlow entry in L2ForwardingCalcTable
uses the input
port number but not the MAC address of the packet. The flow actions are:
- set flag
OFPortFoundRegMark
, and - set the peer port as the
TargetOFPortField
, and - enforce the packet to go to stageIngressSecurity.
Below is an example OpenFlow entry in L2ForwardingCalcTable
table=L2ForwardingCalc, priority=200,ip,in_port=ens224 actions=load:0x1->NXM_NX_REG0[8],load:0x7->NXM_NX_REG1[],resubmit(,IngressSecurityClassifier)
table=L2ForwardingCalc, priority=200,ip,in_port="ens224~" actions=load:0x1->NXM_NX_REG0[8],load:0x8->NXM_NX_REG1[],resubmit(,IngressSecurityClassifier)
Limitations
This feature currently supports only one interface per ExternalNode
object,
and ips
must be set in the interface. The support for multiple network
interfaces will be added in the future.
ExternalNode
name must be unique in the cluster
scope even though it is
itself a Namespaced resource.