We are using a DHCP CNI plugin to setup networking for Kubernetes pods. On a node (CentOS) that looked just like other nodes, we noticed that the pods were not getting their default gateways set (they don’t have a default route).
After spending a lot of time checking configurations of DHCP server, switches, and the way the nodes had been connected, we concluded that the problem was with the node itself.
In fact, the plugin uses /usr/sbin/dhclient to setup networking for the virtual interface inside a network namespace of the pod. dhclient further calls /usr/sbin/dhclient-script to do the configurations after it gets information from a DHCP server. The virtual interface was named “eth0”, so dhclient decided that it should also read /etc/sysconfig/network-scripts/ifcfg-eth0 and pass the variables along with DHCP lease information to dhclient-script.
On this specific node, eth0 was connected to a different subnet so it has a different GATEWAY in ifcfg-eth0, which dhclient-script sees as invalid and skipped adding the gateway. (Of course, if DEFROUTE is set to a variable other than “yes”, dhclient-script also skips this part).
Actually I didn’t look into the implementation of /usr/bin/dhclient, but I guess that’s how it happened. In our case, the link on eth0 was no longer used so we just turned it down and commented out the GATEWAY line in the file to fix it.