The `docker compose` command that killed my SSH session

DNS is not responsible for every network error. Sometimes a connection timeout hides a configuration error that you only see at the first docker compose up.

In the extremely hot August, I discovered UpCloud, a new European cloud provider. So I decided to give it a spin (link with referral code). In their offering, each server comes connected with a private utility network, the ability to attach other software-defined networks (SDN), and the obvious public IP.

I was on a mobile connection and started creating the server and installing Docker to give it a try by running some services. The connection was incurring timeouts. I initially considered it a network problem due to the 4G connection. Switching to another connection provider seemed to improve the situation, so I moved over. In fact, this was only happening on one server; creating a new one seemed to work better. I even contacted support to investigate the network issue, and they promptly investigated the routes from my IP/provider to the datacenter, finding that everything was good.

It was working once the obvious update check revealed that a new kernel version was available, as well as a new Docker version. As I was playing around, upgrades were not a big deal, so I gave it a go and restarted. The SSH connection timeouts appeared again. Every docker compose command either hung for minutes or dropped my SSH connection outright. Pulls stalled. up -d froze halfway. Sometimes the session came back; sometimes I had to reconnect.

It looked exactly like a Docker networking regression. It wasn’t. It was (spoiler) a race condition that had been sitting in the configuration, waiting for a reboot or a network cache refresh to happen.

The symptoms

The combination of session timeouts and hangs with docker compose invocations that touched containers was a curious one, and I didn’t correlate them until the restart after the upgrade.

I pulled kern.log, dmesg, syslog, and the Docker journal (journalctl -xu docker.service) for the relevant window and went looking for a failure. There wasn’t one. No OOM kill. No hung task, no soft lockup, no call trace. No NIC reset, no conntrack overflow, no neighbour: table full. sshd started at boot and was never touched again. Docker’s daemon log was clean (Daemon has completed initialization); containers started, veths attached, no errors.

Why was it working from the kernel’s viewpoint? With no clear errors, I gave the logs to Claude Opus 5, and it suggested that I check some network configurations:

The logs don't capture the failure, so reproduce it while watching the route table. In tmux, pane 1:

ip monitor route

Pane 2:

docker compose up -d

If the default route flips to eth2 at the moment a veth appears, that's your answer. Alongside, capture the current state:

ip -4 route show default
ip route get 8.8.8.8
networkctl status eth0 eth2
lsmod | grep br_netfilter
sudo nft list ruleset
sudo ufw status verbose

The actual clue

ip monitor route showed nothing but IPv6 link-local churn on the veths during a compose stop && up. Instead ip -4 route show defaultand ip route get 8.8.8.8 were surprisingly informative.

$ ip -4 route show default

default via 10.0.0.1      dev eth2 proto dhcp src 10.0.0.6      metric 100

default via 203.0.113.10  dev eth0 proto dhcp src 203.0.113.42  metric 100

Two default routes with the same metric: eth0 is the public interface, eth2 is UpCloud default utility network with no path to the internet.

$ ip route get 8.8.8.8

8.8.8.8 via 10.0.0.1 dev eth2 src 10.0.0.6

The tie was resolving to the interface that goes nowhere. Every outbound connection from the host was being handed to a gateway that black-holes it. That’s the hang: not an error, just TCP timeouts all the way down. DNS resolution, on the other hand, worked perfectly as the DHCP pushed two explicit routes for the resolvers via the public gateway.

A metric is a priority number on a route. Lower wins. It exists to rank routes that would otherwise be equally valid.

The mechanism behind the SSH drops

Linux maintains cached destination/route state for traffic. A veth lifecycle event, that Docker issue while handling network between containers, can invalidate cached routing state, causing subsequent traffic to perform route lookup again. With two equal-metric default routes, that lookup can select eth2, sending traffic toward the utility network instead of eth0. The reply then leaves through the wrong network and is dropped upstream.

You might expect the socket’s bound source address to constrain the choice of output device. It doesn’t work that way for an ordinary unicast source address. The IPv4 route lookup can carry the source address as part of the flow information without treating it as a requirement to use the interface that owns that address. The selected route determines the output device.

So the important sequence is:

docker compose down
        ↓
veth lifecycle event
        ↓
cached IPv4 routing state can become stale
        ↓
subsequent traffic performs route lookup
        ↓
equal-metric default route selects eth2
        ↓
traffic leaves through the utility network
        ↓
SSH session stalls

One thing worth flagging is that ip monitor route will not necessarily show you this. I ran it during a compose stop && up and saw nothing but IPv6 link-local churn on the veths. The routing table did not change as the relevant one was cached routing state being invalidated.

The fix

The upgrade rebooted the machine, and on this boot the available default routes ended up with eth2 being selected. For the previous uptime the machine had presumably been using the preferred path for the entire life.

The netplan config, generated by cloud-init (/etc/netplan/50-cloud-init.yaml), had no route metrics in it at all:

network:
  version: 2
  ethernets:
    eth0:
      match: {macaddress: "..."}
      dhcp4: true
      set-name: "eth0"
    eth1: {...}
    eth2: {...}

Three interfaces, plain dhcp4: true. For DHCP routes Netplan uses a default route metric of 100. When multiple equal-metric default routes are available, the selected path can depend on the state and ordering of the routes and interfaces. In this case, the behavior was sensitive to how the interfaces were initialized.

To break the tie we can set route-metric explicitly. As 50-cloud-init.yaml might be regenerated I created another netplan yaml file as 99-route-metrics.yaml with:

# /etc/netplan/99-route-metrics.yaml

network:
  version: 2
  ethernets:
    eth0:
      dhcp4-overrides:
        route-metric: 100
    eth1:
      dhcp4-overrides:
        route-metric: 3000
    eth2:
      dhcp4-overrides:
        route-metric: 2000

eth0 is pinned explicitly rather than left implicit as it is assigned a lower route-metric.

You can confirm your file is read by executing:

netplan get ethernets.eth2.dhcp4-overrides

# route-metric: 2000

Then apply it with rollback protection, because you are changing the routing of the link you’re connected over:

netplan try

Run that inside tmux. If applying the change drops your session the tmux pane survives on the server. Reconnect, reattach, and press ENTER to confirm. If you can’t get back within 120 seconds, netplan try rolls back on its own. Have your provider’s web console open as a fallback either way.

You can then reboot and confirm the change is persisted:

$ ip -4 route show default

default via 203.0.113.10 dev eth0 proto dhcp src 203.0.113.42 metric 100

default via 10.0.0.1      dev eth2 proto dhcp src 10.0.0.6     metric 2000

$ ip route get 8.8.8.8

8.8.8.8 via 203.0.113.10 dev eth0 src 203.0.113.42

Takeaways

Check ip route get before you check anything else. On any multi-homed box, one command tells you where traffic actually goes, and it’s the first thing to verify when “the network is weird.” I went through kernel logs, firewall rules, and memory pressure before running it.

A clean log during a failure is information. It rules out crashes, kills, and resource exhaustion in one pass, which is most of the space. If nothing failed, nothing failed — look at routing and policy instead.

Partial connectivity is more confusing than none. DNS surviving on /32 routes turned a routing failure into what looked like a slow remote service. When some things work and some hang, look for what’s different about their route selection, not what’s different about the services.

Equal-metric default routes are a latent bug, not a broken state. Nothing is misconfigured in a way any validator will flag. The config is accepted, the routes install, and the machine works — until it reboots and the race goes the other way. If you have a multi-NIC cloud VM where more than one interface takes DHCP, go look at your metrics now, while it’s working.