Builds stuck in running when using Matrix

Sometimes after a job created by Matrix fails because of Docker reasons (no IPs available or so), that job gets stuck in running and doesn’t stop despite be stopping all jobs using the CLI and removing all containers/networks.

Any idea of how to debug this? We do a special thing by starting a systemd container and execing into it.