I am creating a small educational (not production!) Kubernetes cluster on DO droplets. The droplets have both public and private networks. After running kubeadm on the controlplane server and adding Weave as a CNI, I ran the kubeadm join command on the worker and the weave pod was crashing.

The worker node was not ready:

$ kubectl get nodes
NAME                         STATUS     ROLES    AGE     VERSION
workernode      NotReady   <none>   17s     v1.18.2
controlplane    Ready      master   4m21s   v1.18.2

Logs from the crashing weave pod:

$ kubectl logs -f weave-net-x694n -n kube-system -c weave-npc
INFO: 2020/05/02 22:12:45.787801 Starting Weaveworks NPC 2.6.2; node name "workernode"
INFO: 2020/05/02 22:12:45.802439 Serving /metrics on :6781
Sat May  2 22:12:45 2020 <5> ulogd.c:408 registering plugin `NFLOG'
Sat May  2 22:12:45 2020 <5> ulogd.c:408 registering plugin `BASE'
Sat May  2 22:12:45 2020 <5> ulogd.c:408 registering plugin `PCAP'
Sat May  2 22:12:45 2020 <5> ulogd.c:981 building new pluginstance stack: 'log1:NFLOG,base1:BASE,pcap1:PCAP'
WARNING: scheduler configuration failed: Function not implemented
DEBU: 2020/05/02 22:12:45.842014 Got list of ipsets: []
ERROR: logging before flag.Parse: E0502 22:13:15.868118   12496 reflector.go:205] github.com/weaveworks/weave/prog/weave-npc/main.go:320: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
ERROR: logging before flag.Parse: E0502 22:13:15.868265   12496 reflector.go:205] github.com/weaveworks/weave/prog/weave-npc/main.go:321: Failed to list *v1.Pod: Get https://10.96.0.1:443/api/v1/pods?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
ERROR: logging before flag.Parse: E0502 22:13:15.868398   12496 reflector.go:205] github.com/weaveworks/weave/prog/weave-npc/main.go:322: Failed to list *v1.NetworkPolicy: Get https://10.96.0.1:443/apis/networking.k8s.io/v1/networkpolicies?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
ERROR: logging before flag.Parse: E0502 22:13:46.875635   12496 reflector.go:205] github.com/weaveworks/weave/prog/weave-npc/main.go:320: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
ERROR: logging before flag.Parse: E0502 22:13:46.875735   12496 reflector.go:205] github.com/weaveworks/weave/prog/weave-npc/main.go:321: Failed to list *v1.Pod: Get https://10.96.0.1:443/api/v1/pods?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
ERROR: logging before flag.Parse: E0502 22:13:46.875798   12496 reflector.go:205] github.com/weaveworks/weave/prog/weave-npc/main.go:322: Failed to list *v1.NetworkPolicy: Get https://10.96.0.1:443/apis/networking.k8s.io/v1/networkpolicies?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
ERROR: logging before flag.Parse: E0502 22:14:17.876355   12496 reflector.go:205] github.com/weaveworks/weave/prog/weave-npc/main.go:320: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
ERROR: logging before flag.Parse: E0502 22:14:17.878053   12496 reflector.go:205] github.com/weaveworks/weave/prog/weave-npc/main.go:321: Failed to list *v1.Pod: Get https://10.96.0.1:443/api/v1/pods?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
ERROR: logging before flag.Parse: E0502 22:14:17.883057   12496 reflector.go:205] github.com/weaveworks/weave/prog/weave-npc/main.go:322: Failed to list *v1.NetworkPolicy: Get https://10.96.0.1:443/apis/networking.k8s.io/v1/networkpolicies?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
ERROR: logging before flag.Parse: E0502 22:14:48.877166   12496 reflector.go:205] github.com/weaveworks/weave/prog/weave-npc/main.go:320: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
ERROR: logging before flag.Parse: E0502 22:14:48.881690   12496 reflector.go:205] github.com/weaveworks/weave/prog/weave-npc/main.go:321: Failed to list *v1.Pod: Get https://10.96.0.1:443/api/v1/pods?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
ERROR: logging before flag.Parse: E0502 22:14:48.885439   12496 reflector.go:205] github.com/weaveworks/weave/prog/weave-npc/main.go:322: Failed to list *v1.NetworkPolicy: Get https://10.96.0.1:443/apis/networking.k8s.io/v1/networkpolicies?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
ERROR: logging before flag.Parse: E0502 22:15:19.878893   12496 reflector.go:205] github.com/weaveworks/weave/prog/weave-npc/main.go:320: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
ERROR: logging before flag.Parse: E0502 22:15:19.882762   12496 reflector.go:205] github.com/weaveworks/weave/prog/weave-npc/main.go:321: Failed to list *v1.Pod: Get https://10.96.0.1:443/api/v1/pods?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
ERROR: logging before flag.Parse: E0502 22:15:19.886479   12496 reflector.go:205] github.com/weaveworks/weave/prog/weave-npc/main.go:322: Failed to list *v1.NetworkPolicy: Get https://10.96.0.1:443/apis/networking.k8s.io/v1/networkpolicies?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
ERROR: logging before flag.Parse: E0502 22:15:50.879833   12496 reflector.go:205] github.com/weaveworks/weave/prog/weave-npc/main.go:320: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
ERROR: logging before flag.Parse: E0502 22:15:50.884554   12496 reflector.go:205] github.com/weaveworks/weave/prog/weave-npc/main.go:321: Failed to list *v1.Pod: Get https://10.96.0.1:443/api/v1/pods?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
ERROR: logging before flag.Parse: E0502 22:15:50.888274   12496 reflector.go:205] github.com/weaveworks/weave/prog/weave-npc/main.go:322: Failed to list *v1.NetworkPolicy: Get https://10.96.0.1:443/apis/networking.k8s.io/v1/networkpolicies?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
ERROR: logging before flag.Parse: E0502 22:16:21.880613   12496 reflector.go:205] github.com/weaveworks/weave/prog/weave-npc/main.go:320: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
ERROR: logging before flag.Parse: E0502 22:16:21.885087   12496 reflector.go:205] github.com/weaveworks/weave/prog/weave-npc/main.go:321: Failed to list *v1.Pod: Get https://10.96.0.1:443/api/v1/pods?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout

kubelet log on the worker:

May 02 21:55:03 workernode kubelet[9208]: W0502 21:55:03.962619    9208 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
May 02 21:55:04 workernode kubelet[9208]: E0502 21:55:04.912597    9208 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
May 02 21:55:08 workernode kubelet[9208]: W0502 21:55:08.962840    9208 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
May 02 21:55:09 workernode kubelet[9208]: E0502 21:55:09.924452    9208 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
May 02 21:55:13 workernode kubelet[9208]: W0502 21:55:13.963023    9208 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
May 02 21:55:14 workernode kubelet[9208]: E0502 21:55:14.936887    9208 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
May 02 21:55:18 workernode kubelet[9208]: W0502 21:55:18.963251    9208 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
May 02 21:55:19 workernode kubelet[9208]: E0502 21:55:19.951526    9208 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
May 02 21:55:23 workernode kubelet[9208]: I0502 21:55:23.383651    9208 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: 77d29024194227e55b230ca79264fc820ab6db74a7f1cbf7e85605e017f27479
May 02 21:55:23 workernode kubelet[9208]: I0502 21:55:23.384033    9208 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: ac7bf25deaf97788933e98d205768549267e6a823dbcf51589518b5755f7b9fd
May 02 21:55:23 workernode kubelet[9208]: E0502 21:55:23.384658    9208 pod_workers.go:191] Error syncing pod 4f9fded3-b8e3-47c5-86f2-c4b73ddac668 ("weave-net-mpxf8_kube-system(4f9fded3-b8e3-47c5-86f2-c4b73ddac668)"), skipping: failed to "StartContainer" for "weave" with CrashLoopBackOff: "back-off 40s restarting failed container=weave pod=weave-net-mpxf8_kube-system(4f9fded3-b8e3-47c5-86f2-c4b73ddac668)"
May 02 21:55:23 workernode kubelet[9208]: W0502 21:55:23.963541    9208 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
May 02 21:55:24 workernode kubelet[9208]: E0502 21:55:24.964955    9208 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
May 02 21:55:28 workernode kubelet[9208]: W0502 21:55:28.963834    9208 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
May 02 21:55:29 workernode kubelet[9208]: E0502 21:55:29.976776    9208 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
May 02 21:55:33 workernode kubelet[9208]: W0502 21:55:33.964124    9208 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
May 02 21:55:34 workernode kubelet[9208]: E0502 21:55:34.987196    9208 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
May 02 21:55:35 workernode kubelet[9208]: I0502 21:55:35.346939    9208 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: ac7bf25deaf97788933e98d205768549267e6a823dbcf51589518b5755f7b9fd
May 02 21:55:35 workernode kubelet[9208]: E0502 21:55:35.347481    9208 pod_workers.go:191] Error syncing pod 4f9fded3-b8e3-47c5-86f2-c4b73ddac668 ("weave-net-mpxf8_kube-system(4f9fded3-b8e3-47c5-86f2-c4b73ddac668)"), skipping: failed to "StartContainer" for "weave" with CrashLoopBackOff: "back-off 40s restarting failed container=weave pod=weave-net-mpxf8_kube-system(4f9fded3-b8e3-47c5-86f2-c4b73ddac668)"
May 02 21:55:38 workernode kubelet[9208]: W0502 21:55:38.964458    9208 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
May 02 21:55:40 workernode kubelet[9208]: E0502 21:55:39.999710    9208 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
May 02 21:55:43 workernode kubelet[9208]: W0502 21:55:43.964705    9208 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
May 02 21:55:45 workernode kubelet[9208]: E0502 21:55:45.011433    9208 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
May 02 21:55:48 workernode kubelet[9208]: W0502 21:55:48.964964    9208 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
May 02 21:55:49 workernode kubelet[9208]: I0502 21:55:49.346795    9208 topology_manager.go:219] [topologymanager] RemoveContainer - Container ID: ac7bf25deaf97788933e98d205768549267e6a823dbcf51589518b5755f7b9fd
May 02 21:55:49 workernode kubelet[9208]: E0502 21:55:49.347293    9208 pod_workers.go:191] Error syncing pod 4f9fded3-b8e3-47c5-86f2-c4b73ddac668 ("weave-net-mpxf8_kube-system(4f9fded3-b8e3-47c5-86f2-c4b73ddac668)"), skipping: failed to "StartContainer" for "weave" with CrashLoopBackOff: "back-off 40s restarting failed container=weave pod=weave-net-mpxf8_kube-system(4f9fded3-b8e3-47c5-86f2-c4b73ddac668)"
May 02 21:55:50 workernode kubelet[9208]: E0502 21:55:50.022462    9208 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
May 02 21:55:53 workernode kubelet[9208]: W0502 21:55:53.965185    9208 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
May 02 21:55:55 workernode kubelet[9208]: E0502 21:55:55.034992    9208 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

Incidentally:

$ kubectl get services --all-namespaces
NAMESPACE     NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
default       kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP                  15m
kube-system   kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   15m

I suspected the issue was reaching 10.96.0.1 from the weaver container running on the worker, but I didn’t know exactly what to do. My iptables skills (and my understanding of Kubernetes networking) did not stretch this far.. :-(

Until I found this page: https://github.com/weaveworks/weave/issues/3420 and in particular the comment marcosnils commented on Oct 19, 2018. I executed the command as is in the comment:

iptables -t nat -I KUBE-SERVICES -d 10.96.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-MARK-MASQ

and I immediately saw that the log of the weave pod on the worker changed:

ERROR: logging before flag.Parse: E0502 22:16:21.888804   12496 reflector.go:205] github.com/weaveworks/weave/prog/weave-npc/main.go:322: Failed to list *v1.NetworkPolicy: Get https://10.96.0.1:443/apis/networking.k8s.io/v1/networkpolicies?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout
DEBU: 2020/05/02 22:16:22.919951 EVENT AddPod {"metadata":{"creationTimestamp":"2020-05-02T22:12:44Z","generateName":"kube-proxy-","labels":{"controller-revision-hash":"5f7b7d4f89","k8s-app":"kube-proxy","pod-template-generation":"1"},"name":"kube-proxy-nfkbz","namespace":"kube-system","resourceVersion":"623","selfLink":"/api/v1/namespaces/kube-system/pods/kube-proxy-nfkbz","uid":"c4bbc665-40d8-4730-adfd-2b1920e4bc2c"},"spec":{"affinity":{"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchFields":[{"key":"metadata.name","operator":"In","values":["centos-s-1vcpu-1gb-fra1-02"]}]}]}}},"containers":[{"image":"k8s.gcr.io/kube-proxy:v1.18.2","imagePullPolicy":"IfNotPresent","name":"kube-proxy","terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File"}],"dnsPolicy":"ClusterFirst","hostNetwork":true,"nodeName":"centos-s-1vcpu-1gb-fra1-02","nodeSelector":{"kubernetes.io/os":"linux"},"priority":2000001000,"priorityClassName":"system-node-critical","restartPolicy":"Always","schedulerName":"default-scheduler","securityContext":{},"serviceAccount":"kube-proxy","serviceAccountName":"kube-proxy","terminationGracePeriodSeconds":30},"status":{"conditions":[{"lastProbeTime":null,"lastTransitionTime":"2020-05-02T22:12:44Z","status":"True","type":"Initialized"},{"lastProbeTime":null,"lastTransitionTime":"2020-05-02T22:12:46Z","status":"True","type":"Ready"},{"lastProbeTime":null,"lastTransitionTime":"2020-05-02T22:12:46Z","status":"True","type":"ContainersReady"},{"lastProbeTime":null,"lastTransitionTime":"2020-05-02T22:12:44Z","status":"True","type":"PodScheduled"}],"hostIP":"10.114.0.3","phase":"Running","podIP":"10.114.0.3","qosClass":"BestEffort","startTime":"2020-05-02T22:12:44Z"}}
INFO: 2020/05/02 22:16:22.929680 creating ipset: &npc.selectorSpec{key:"", podSelector:labels.internalSelector{}, namespaceSelector:labels.Selector(nil), policyTypes:[]npc.policyType(nil), ipsetType:"hash:ip", ipsetName:"weave-iuZcey(5DeXbzgRFs8Szo]+@p", nsName:"kube-system"}

and I also saw the weaver pod on the worker node go from crashing to Running and passing the readiness probe (2/2).

Thanks to https://github.com/marcosnils for the resolution.