Thomas Bach - Software Architect

Using epdb to debug distributed services

Posted on January 11, 2024

As I am working a lot on distributed systems, I find myself often in a situation where a bug occurs in a program and

  1. I cannot really pinpoint the executing service in the layer or
  2. (more frequently) I cannot easily attach myself to a debugging session using pdb.

Examples for the latter might be, a service which spawns off several processes to distribute load, a system service (daemon) which forks after the setup routine or a deeply nested service in a larger message passing stack.

This is just the right moment to install epdb. Epdb lets you debug a Python program over a network socket. A tool which seems to fly below the radar. Of course, it is also available on PyPI.

Now, the usage is really simple: install it in the environment where your program is running and add

import epdb; epdb.serve()

at the location you want to start debugging. Note that you can also pass in a port to epdb.serve.

Run your program and ensure that the program executes the code path where the above statement can be found. In a shell execute:

python -c 'import epdb; epdb.connect()'

and you will be dropped into a Python debugging interface, just like pdb, but over the network.

I usually start a loop around the command above:

until python3 -c 'import epdb; epdb.connect()'; do sleep 1s; done