借助Flannel，可以使得docker实例具备跨宿主机的网络通讯能力，并能使容器在Flannel内网的IP地址固定。
在每台主机上准备Go编译平台以及编译etcd和flannel
| #wget https://storage.googleapis.com/golang/go1.6.2.linux-amd64.tar.gz --2016-06-22 14:53:31-- https://storage.googleapis.com/golang/go1.6.2.linux-amd64.tar.gz Resolving storage.googleapis.com (storage.googleapis.com)... 74.125.23.128, 2404:6800:4005:80a::2010 Connecting to storage.googleapis.com (storage.googleapis.com)|74.125.23.128|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 84840658 (81M) [application/x-gzip] Saving to: go1.6.2.linux-amd64.tar.gz100%[===============================================================>] 84,840,658 429KB/s in 2m 46s2016-06-22 14:56:18 (499 KB/s) - go1.6.2.linux-amd64.tar.gz saved [84840658/84840658] #tar -zxvf go1.6.2.linux-amd64.tar.gz && cd go #export GOBIN=`pwd`/bin #export PATH=$GOROOT/bin:$PATH #export GOOS=linux #export GOARCH=amd64 #cd .. #git clone https://github.com/coreos/etcd.git Cloning into 'etcd'... blog.nvacg.org版权所有 remote: Counting objects: 48104, done. remote: Compressing objects: 100% (18/18), done. remote: Total 48104 (delta 2), reused 0 (delta 0), pack-reused 48084 Receiving objects: 100% (48104/48104), 25.42 MiB | 1.29 MiB/s, done. Resolving deltas: 100% (30114/30114), done. Checking connectivity... done. |
| #git clone https://github.com/coreos/flannel.git Cloning into 'flannel'... remote: Counting objects: 5686, done. remote: Compressing objects: 100% (4/4), done. remote: Total 5686 (delta 0), reused 0 (delta 0), pack-reused 5682 Receiving objects: 100% (5686/5686), 10.95 MiB | 1.40 MiB/s, done. Resolving deltas: 100% (1930/1930), done. Checking connectivity... done. #cd etcd && ./build && cd .. ./build #cd flannel && ./build && cd .. Building flanneld... blog.nvacg.org版权所有 |
随即可以在etcd/bin和flannel/bin目录中找到二进制执行文件，将他们拷贝到系统执行目录下
| #ls etcd/bin && ls flannel/bin etcd etcdctl flannel #cp -a etcd/bin/. /usr/sbin #cp -a flannel/bin/. /usr/sbin #cp flannel/dist/mk-docker-opts.sh /usr/sbin blog.nvacg.org版权所有 |
在主机1上启动etcd，宿主机1的实IP是172.16.71.33，参数中各端口可随意指定，为防止混淆，所有宿主机均使用同样的端口供etcd监听，当后续启动时需要将 --initial-cluster-state new 改为 --initial-cluster-state existing
| #etcd --name node1 --initial-advertise-peer-urls http://172.16.71.33:9030 --listen-peer-urls http://172.16.71.33:9030 --advertise-client-urls http://172.16.71.33:9020 --listen-client-urls http://0.0.0.0:9020 --initial-cluster-token nvacg-docker-cluster-webserver --initial-cluster node1=http://172.16.71.33:9030,node2=http://172.16.71.34:9030 --initial-cluster-state new > /dev/null 2>&1 & |
在主机2上启动etcd，宿主机2的实IP是172.16.71.34 ，当后续启动时需要将 --initial-cluster-state new 改为 --initial-cluster-state existing
| #etcd --name node2 --initial-advertise-peer-urls http://172.16.71.34:9030 --listen-peer-urls http://172.16.71.34:9030 --advertise-client-urls http://172.16.71.34:9020 --listen-client-urls http://0.0.0.0:9020 --initial-cluster-token nvacg-docker-cluster-webserver --initial-cluster node1=http://172.16.71.33:9030,node2=http://172.16.71.34:9030 --initial-cluster-state new > /dev/null 2>&1 & |
接下来设置内部网段，在任意节点上执行 (当etcd的启动参数使用默认端口2379时，可不加 —endpoints 参数)，子网掩码至少是/16
| #etcdctl --endpoints 127.0.0.1:9020 set /coreos.com/network/config '{"Network":"10.128.0.0/16"}' |
查看下节点的存活情况
| #etcdctl --endpoints 127.0.0.1:9020 cluster-health member 8113781b1480a5bd is healthy: got healthy result from http://172.16.71.33:9020 member b61040edf801d4e9 is healthy: got healthy result from http://172.16.71.34:9020 cluster is healthy blog.nvacg.org版权所有 |
在主机1上启动flanneld服务
| #flanneld -etcd-endpoints http://127.0.0.1:9020 > /dev/null 2>&1 & [2] 2393 I0624 11:09:16.273288 02393 main.go:126] Installing signal handlers I0624 11:09:16.273572 02393 manager.go:133] Determining IP address of default interface I0624 11:09:16.273987 02393 manager.go:163] Using 172.16.71.33 as external interface I0624 11:09:16.274054 02393 manager.go:164] Using 172.16.71.33 as external endpoint I0624 11:09:16.276052 02393 local_manager.go:150] Found lease (192.168.30.128/25) for current IP (172.16.71.33) but not compatible with current config, deleting I0624 11:09:16.281494 02393 local_manager.go:179] Picking subnet in range 10.128.1.0 ... 10.128.255.0 I0624 11:09:16.284777 02393 manager.go:246] Lease acquired: 10.128.12.0/24 I0624 11:09:16.284939 02393 network.go:98] Watching for new subnet leases I0624 11:09:39.700273 02393 network.go:191] Subnet added: 10.128.51.0/24 |
在主机2上启动flanneld服务
| #flanneld -etcd-endpoints http://127.0.0.1:9020 > /dev/null 2>&1 & [2] 9840 root@dockertest:/home/smm/flannel/dist# I0624 11:09:42.665807 09840 main.go:126] Installing signal handlers I0624 11:09:42.666043 09840 manager.go:133] Determining IP address of default interface I0624 11:09:42.666483 09840 manager.go:163] Using 172.16.71.34 as external interface I0624 11:09:42.666521 09840 manager.go:164] Using 172.16.71.34 as external endpoint I0624 11:09:42.668116 09840 local_manager.go:179] Picking subnet in range 10.128.1.0 ... 10.128.255.0 I0624 11:09:42.697901 09840 manager.go:246] Lease acquired: 10.128.51.0/24 I0624 11:09:42.698173 09840 network.go:98] Watching for new subnet leases I0624 11:09:42.699231 09840 network.go:191] Subnet added: 10.128.12.0/24 |
重新配置docker网卡，在各节点上执行
| #mk-docker-opts.sh -i #. /run/flannel/subnet.env #ifconfig docker0 $FLANNEL_SUBNET #service docker restart blog.nvacg.org版权所有 |
如果做了上述更改后，实例无法连接到外网，可以先停止docker服务，然后执行
iptables -t nat -F
然后重新启动docker服务即可
然后直接将参数写入到docker的环境变量/etc/default/docker中去，这需要我们对/run/flannel/subnet.env进行一下翻译:
--bip = $FLANNEL_SUBNET
--ip-masq = $FLANNEL_IPMASQ
--mtu = $FLANNEL_MTU
这样当docker启动（或重新启动）的时侯会自动将docker0网桥的设置改为flannel相同的设置，建议是直接重启一下宿主机，如果直接重启docker，可能服务会无法启动，这是因为先前系统中已经存在docker0网桥，并处于活动状态，docker尝试更改它的参数但是失败了。此时需要手动停止网桥，删除它，再启动(或重启)docker服务应该就可以了。
以主机1为例，此时该主机上docker0网卡和flannel0网卡的状态如下
| #ifconfig docker0 Link encap:Ethernet HWaddr 7e:41:01:f7:2f:bc inet addr:10.128.12.1 Bcast:10.128.12.255 Mask:255.255.255.0 inet6 addr: fe80::7c41:1ff:fef7:2fbc/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:107 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:16271 (16.2 KB) eth0 Link encap:Ethernet HWaddr 00:0c:29:22:0c:3a inet addr:172.16.71.33 Bcast:172.16.71.255 Mask:255.255.255.0 inet6 addr: fe80::20c:29ff:fe22:c3a/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:117369 errors:0 dropped:0 overruns:0 frame:0 TX packets:101022 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:11207503 (11.2 MB) TX bytes:9848109 (9.8 MB) flannel0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 inet addr:10.128.12.0 P-t-P:10.128.12.0 Mask:255.255.0.0 UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1472 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:500 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) blog.nvacg.org版权所有 |
看下flannel节点信息
| #etcdctl --endpoints 127.0.0.1:9020 ls /coreos.com/network/subnets /coreos.com/network/subnets/10.128.12.0-24 /coreos.com/network/subnets/10.128.51.0-24 blog.nvacg.org版权所有 |
现在，主机1上承载的实例可以ping到主机2上承载的实例，也可以连通主机2上承载的实例中监听的端口。
目前，flannel方式组网仍然是由docker来分配地址，即使解决了跨主机联网的问题，各实例分配到的地址仍然是动态的，仅供测试使用，若需要在无人值守环境使用，需要配合skydns使用，通过主机名方式解析其他实例的地址。对于需要设置固定IP的场合，可以先启动容器或镜像后经由pipework分配静态IP的方式。对于上例中宿主机1上docker的网络是10.128.12.0/24，则其网关默认分配为10.128.12.1，则pipework参数为
pipework docker0 容器名称或容器ID 10.128.12.XXX/24@10.128.12.1
另外需要声明一点，由于采用flannel时，对外映射端口(不是指实例到实例，而是指对外提供的服务)的方式仍然是通过iptables曝光端口到宿主机后由宿主机对外提供该端口的方式，因此启动容器时仍然采用-p参数来映射端口，由于需要使用该参数来映射端口，故不能指定--net=none参数，否则端口映射无效。实际经过测试，即使是常规方式启动的实例仍然可以使用pipework来为实例设置第二IP。
如果你是采用--net=none参数启动+pipework设置网络参数的，若要暴露端口，请直接修改iptables来达到暴露端口的目的。
请注意，本站的所有文章均要求阁下在转载时注明出处和原作者，阁下转载本站文章即表示阁下同意并遵守此规程，除非特别注明转载出处，否则文章即为其发布者所著。本站及文章作者保留文章的著作权并有权在阁下违反上述规程时予以追究。