Docker Multi-host with OVS

network

Docker integration with OpenVSwitch:
In this article I’m going to show you how to setup a multi-host docker cluster where containers on different hosts can talk to each other without NATing (Network Address Translation). I’m going to use OpenVSwitch using GRE tunnels, GRE tunnels are Linux kernel technologies that are true and tested (they are used in production for years).

The Open Virtual Switch (OVS) is an L2 switch just like physical switch, it has ports where you plug ethernet plugs and it works at MAC address / ARP level to connect them.

In this article we will keep SELinux enabled.

I used two VM’s (host1 and host2) each with Fedora 23 as the operating system.

Steps:

You need to run all those steps on all nodes. Let’s install some dependencies,

yum update -y
yum install -y  docker nc

Networking:
You can install Open vSwitch on Fedora by running this command:

yum install -y openvswitch bridge-utils
systemctl start openvswitch
systemctl enable openvswitch

Network configuration:

# The 'other' host
REMOTE_IP=192.168.122.189
# Name of the bridge
BRIDGE_NAME=docker0
# Bridge address
BRIDGE_ADDRESS=172.16.42.2/24

# Deactivate the docker0 bridge
ip link set $BRIDGE_NAME down
# Remove the docker0 bridge
brctl delbr $BRIDGE_NAME
# Delete the Open vSwitch bridge
ovs-vsctl del-br br0
# Add the docker0 bridge
brctl addbr $BRIDGE_NAME
# Set up the IP for the docker0 bridge
ip a add $BRIDGE_ADDRESS dev $BRIDGE_NAME
# Activate the bridge
ip link set $BRIDGE_NAME up
# Add the br0 Open vSwitch bridge
ovs-vsctl add-br br0
# Create the tunnel to the other host and attach it to the
# br0 bridge
ovs-vsctl add-port br0 gre0 -- set interface gre0 type=gre options:remote_ip=$REMOTE_IP
# Add the br0 bridge to docker0 bridge
brctl addif $BRIDGE_NAME br0

# Some useful commands to confirm the settings:
# ip a s
# ip r s
# ovs-vsctl show
# brctl show

After executing these commands on both hosts you should be able to ping the docker0 bridge addresses from both hosts.

Here is an example from host2 (ip 192.168.122.189):

ping 172.16.42.1

Networking explained:
The above script has some useful comments that help to understand what it’s doing, but here’s a high level view on the networking part.
1- Every container run with Docker is attached to docker0 bridge. This is a regular bridge you can create on every Linux system, without the need for Open vSwitch.
2- The docker0 bridge is attached to another bridge: br0. This time it’s an Open vSwitch bridge. This means that all traffic between containers is routed through br0 too. You can think about two switches connected to each other.
3- Additionally we need to connect together the networks from both hosts in which the containers are running. A GRE tunnel is used for this purpose. This tunnel is attached to the br0 Open vSwitch bridge and as a result to docker0 too.

Finally:
Now you can create containers on Host1 then go to Host2 and try to ping any container in Host1 and see the result, you should can ping.

 

Leave a Reply

Your email address will not be published.