Image
How to use the Python debugger (pdb)
Pdb is a powerful tool for finding syntax errors, spelling mistakes, missing code, and other issues in your Python code.
We all know things can go wrong when writing programs. Syntax errors, spelling mistakes, even forgotten sections of code; it's all possible. Sometimes, issues like these are easy to detect. Here's an example:
$ python3
Python 3.9.9 (main, Nov 19 2021, 00:00:00)
[GCC 10.3.1 20210422 (Red Hat 10.3.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> DEBUG: bool = True
>>>
>>> def print_var_function(val: str):
... print(f"This is a simple dummy function that prints val={val}")
...
>>> if __name__ == "__main__":
... print(f"What is your name?")
... name = input()
... if DEBUG:
... print_var_function(name)
...
What is your name?
jose
This is a simple dummy function that prints val=jose
Even if you write small Python programs, you will find out soon enough that tricks like this are not enough to debug a program. Instead, you can take advantage of the Python debugger (pdb) and get a better insight into how your application is behaving.
Getting started
Before starting this tutorial, you should have:
- Working knowledge of Python (objects, workflows, data structures)
- Curiosity about how you can troubleshoot your scripts in real time
- A machine that can run Python 3 (for example, I'm using Fedora Linux)
Note: I will use a modern version of Python (3.7+) in this tutorial, but you can find the older syntax for some of the operations in the official pdb documentation.
Case study: A simple script to generate a network diagram
A friend of yours gave you a small Python script to test. He said he wrote it in a rush, and it may contain bugs (in fact, he admitted he tried to run it, but he is pretty sure the proof of concept is good). He also said the script depends on the module Diagrams. It's time for you to try his script.
First, create a virtual environment and install some dependencies:
python3 -m venv ~/virtualenv/pythondebugger
. ~/virtualenv/pythondebugger/bin/activate
pip install --upgrade pip diagrams
Next, download and install the following script:
$ pushd $HOME
$ git clone git@github.com:josevnz/tutorials.git
$ pushd tutorials/PythonDebugger
Unfortunately, when you run the script, it crashes:
(pythondebugger) $ ./simple_diagram.py --help
Traceback (most recent call last):
File "/home/josevnz/tutorials/PythonDebugger/./simple_diagram.py", line 8, in <module>
from diagrams.onprem.queue import Celeri
ImportError: cannot import name 'Celeri' from 'diagrams.onprem.queue' (/home/josevnz/virtualenv/pythondebugger/lib64/python3.9/site-packages/diagrams/onprem/queue.py)
So at least one import is wrong. You suspect it is Celery (which your friend spelled Celeri), so you launch the script again with the Python debugger mode enabled:
(pythondebugger) $ python3 -m pdb simple_diagram.py --help
> /home/josevnz/tutorials/PythonDebugger/simple_diagram.py(2)<module>()
-> """
(Pdb)
No crashes, and the prompt (Pdb) tells you that you are currently on line 2 of the program:
(pythondebugger) $ python3 -m pdb simple_diagram.py --help
> /home/josevnz/tutorials/PythonDebugger/simple_diagram.py(2)<module>()
-> """
(Pdb) l
1 #!/usr/bin/env python
2 -> """
3 Script that show a basic Airflow + Celery Topology
4 """
5 import argparse
6 from diagrams import Cluster, Diagram
7 from diagrams.onprem.workflow import Airflow
8 from diagrams.onprem.queue import Celeri
9
10
11 def generate_diagram(diagram_file: str, workers_n: int):
You can press c
(continue) until there is a breakpoint:
(Pdb) c
Traceback (most recent call last):
File "/usr/lib64/python3.9/pdb.py", line 1723, in main
pdb._runscript(mainpyfile)
File "/usr/lib64/python3.9/pdb.py", line 1583, in _runscript
self.run(statement)
File "/usr/lib64/python3.9/bdb.py", line 580, in run
exec(cmd, globals, locals)
File "<string>", line 1, in <module>
File "/home/josevnz/tutorials/PythonDebugger/simple_diagram.py", line 2, in <module>
"""
ImportError: cannot import name 'Celeri' from 'diagrams.onprem.queue' (/home/josevnz/virtualenv/pythondebugger/lib64/python3.9/site-packages/diagrams/onprem/queue.py)
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> /home/josevnz/tutorials/PythonDebugger/simple_diagram.py(2)<module>()
-> """
There's the problem: The bad import won't let the script continue. So, start with n
(next) and move line by line:
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(2)<module>()
-> """
(Pdb) step
Post mortem debugger finished. The /home/josevnz/tutorials/PythonDebugger//simple_diagram.py will be restarted
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(2)<module>()
-> """
(Pdb) n
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(5)<module>()
-> import argparse
(Pdb) n
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(6)<module>()
-> from diagrams import Cluster, Diagram
(Pdb) n
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(7)<module>()
-> from diagrams.onprem.workflow import Airflow
(Pdb) n
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(8)<module>()
-> from diagrams.onprem.queue import Celeri
(Pdb) n
ImportError: cannot import name 'Celeri' from 'diagrams.onprem.queue' (/home/josevnz/virtualenv/pythondebugger/lib64/python3.9/site-packages/diagrams/onprem/queue.py)
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(8)<module>()
-> from diagrams.onprem.queue import Celeri
You won't be able to proceed unless you fix the broken instruction in line 8. You need to replace import Celeri
with import Celery
:
(Pdb) from diagrams.onprem.queue import Celery
(Pdb) exit()
For the sake of argument, maybe you want the program to enter the debug mode if a module is missing (in fact, you insist there is a Celeri module). The change in the code is simple; just capture the ImportError
and call the breakpoint()
function (before you show the stack trace using traceback):
#!/usr/bin/env python
"""
Script that show a basic Airflow + Celery Topology
"""
try:
import argparse
from diagrams import Cluster, Diagram
from diagrams.onprem.workflow import Airflow
from diagrams.onprem.queue import Celeri
except ImportError:
breakpoint()
If you call the script normally and one of the imports is missing, the debugger will start for you after printing the stack trace:
(pythondebugger) $ python3 simple_diagram.py --help
Exception while importing modules:
------------------------------------------------------------
Traceback (most recent call last):
File "/home/josevnz/tutorials/PythonDebugger//simple_diagram.py", line 11, in <module>
from diagrams.onprem.queue import Celeri
ImportError: cannot import name 'Celeri' from 'diagrams.onprem.queue' (/home/josevnz/virtualenv/pythondebugger/lib64/python3.9/site-packages/diagrams/onprem/queue.py)
------------------------------------------------------------
Starting the debugger
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(21)<module>()
-> def generate_diagram(diagram_file: str, workers_n: int):
(Pdb)
Fix the bad import and move on to see what the script can do:
(pythondebugger) $ python3 simple_diagram.py --help
usage: /home/josevnz/tutorials/PythonDebugger//simple_diagram.py [-h] [--workers WORKERS] diagram
Generate network diagrams for examples used on this tutorial
positional arguments:
diagram Name of the network diagram to generate
optional arguments:
-h, --help show this help message and exit
--workers WORKERS Number of workers
# Generate a diagram with 3 workers
(pythondebugger) [josevnz@dmaf5 ]$ python3 simple_diagram.py --workers 3 my_airflow.png
The resulting diagram looks like this:
Take a closer look at the code with step, continue, args, and breakpoints
Run the script again but using a negative number of workers:
$ python3 simple_diagram.py --workers -3 my_airflow2.png
This produces a weird image with no Celery workers:
This is unexpected. Use the debugger to understand what happened and also come up with a way to prevent this; start by asking to see the full source code with ll
:
(pythondebugger) $ python3 -m pdb simple_diagram.py --workers -3 my_airflow2.png
> /home/josevnz/tutorials/PythonDebugger/simple_diagram.py(2)<module>()
-> """
(Pdb) ll
1 #!/usr/bin/env python
2 -> """
3 Script that show a basic Airflow + Celery Topology
4 """
5 try:
6 import sys
7 import argparse
8 import traceback
9 from diagrams import Cluster, Diagram
10 from diagrams.onprem.workflow import Airflow
11 from diagrams.onprem.queue import Celery
12 except ImportError:
13 print("Exception while importing modules:")
14 print("-"*60)
15 traceback.print_exc(file=sys.stderr)
16 print("-"*60)
17 print("Starting the debugger", file=sys.stderr)
18 breakpoint()
19
20
21 def generate_diagram(diagram_file: str, workers_n: int):
22 """
23 Generate the network diagram for the given number of workers
24 @param diagram_file: Where to save the diagram
25 @param workers_n: Number of workers
26 """
27 with Diagram("Airflow topology", filename=diagram_file, show=False):
28 with Cluster("Airflow"):
29 airflow = Airflow("Airflow")
30
31 with Cluster("Celery workers"):
32 workers = []
33 for i in range(workers_n):
34 workers.append(Celery(f"Worker {i}"))
35 airflow - workers
36
37
38 if __name__ == "__main__":
39 PARSER = argparse.ArgumentParser(
40 description="Generate network diagrams for examples used on this tutorial",
41 prog=__file__
42 )
43 PARSER.add_argument(
44 '--workers',
45 action='store',
46 type=int,
47 default=1,
48 help="Number of workers"
49 )
50 PARSER.add_argument(
51 'diagram',
52 action='store',
53 help="Name of the network diagram to generate"
54 )
55 ARGS = PARSER.parse_args()
56
57 generate_diagram(ARGS.diagram, ARGS.workers)
(Pdb)
Going step by step here is not very efficient, so go just deep enough in the code, to line 57, to see the effect of passing the number of workers == -1
(to pretty print the ARGS variable):
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(57)<module>()
-> generate_diagram(ARGS.diagram, ARGS.workers)
(Pdb) pp ARGS
Namespace(workers=-3, diagram='my_airflow2.png')
Also, it is very useful to know the object type. Check to see the ARGS type:
(Pdb) whatis ARGS
<class 'argparse.Namespace'>
It looks like the next logical step is to dive into the function that generates the diagram with l
(list) to confirm where you are:
(Pdb) l
52 action='store',
53 help="Name of the network diagram to generate"
54 )
55 ARGS = PARSER.parse_args()
56
57 -> generate_diagram(ARGS.diagram, ARGS.workers)
[EOF]
Dive one s
(step) inside and then move n
(next) one instruction:
(Pdb) s
--Call--
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(21)generate_diagram()
-> def generate_diagram(diagram_file: str, workers_n: int):
(Pdb) n
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(27)generate_diagram()
-> with Diagram("Airflow topology", filename=diagram_file, show=False):
(Pdb) l
22 """
23 Generate the network diagram for the given number of workers
24 @param diagram_file: Where to save the diagram
25 @param workers_n: Number of workers
26 """
27 -> with Diagram("Airflow topology", filename=diagram_file, show=False):
28 with Cluster("Airflow"):
29 airflow = Airflow("Airflow")
30
31 with Cluster("Celery workers"):
32 workers = []
The prompt tells you that you are inside the generate_diagram
function. Confirm which (arguments) were passed:
(Pdb) a generate_diagram
diagram_file = 'my_airflow2.png'
workers_n = -3
(Pdb)
Inspect the code again and study where it iterates workers_n
for the number of times to add Celery workers to the diagram:
(Pdb) l
33 for i in range(workers_n):
34 workers.append(Celery(f"Worker {i}"))
35 airflow - workers
36
37
38 if __name__ == "__main__":
39 PARSER = argparse.ArgumentParser(
40 description="Generate network diagrams for examples used on this tutorial",
41 prog=__file__
42 )
43 PARSER.add_argument(
Lines 33-34 populate the workers
list with Celery objects. You can see what happens to the range()
function when you pass a negative number when trying to create an iterator:
(Pdb) p range(workers_n)
range(0, -3)
The theory says this will generate an empty iterator, meaning the loop will never run. Confirm the before and after state of the workers
variable. To do that, set up two b
(breakpoints):
- After the
workers
variable is initialized, on line 33. - After the loop that populates the
workers
exits, on line 35.
You don't want to execute code one step at a time, so c
(continue) through the b
(breakpoints) instead:
(Pdb) b simple_diagram.py:33
Breakpoint 1 at /home/josevnz/tutorials/PythonDebugger//simple_diagram.py:33
(Pdb) b simple_diagram.py:35
Breakpoint 2 at /home/josevnz/tutorials/PythonDebugger//simple_diagram.py:35
(Pdb) b
Num Type Disp Enb Where
1 breakpoint keep yes at /home/josevnz/tutorials/PythonDebugger//simple_diagram.py:33
2 breakpoint keep yes at /home/josevnz/tutorials/PythonDebugger//simple_diagram.py:35
Now it's time to print the contents of workers
and continue as promised:
(Pdb) c
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(33)generate_diagram()
-> for i in range(workers_n):
(Pdb) p workers
[]
(Pdb) c
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(35)generate_diagram()
-> airflow - workers
(Pdb) workers
[]
If you press c
(continue) the program will exit, or r
(return) will get you back to main. Use return:
(Pdb) r
--Return--
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(35)generate_diagram()->None
-> airflow - workers
(Pdb) c
The program finished and will be restarted
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(2)<module>()
-> """
(Pdb) exit()
You can fix the application by adding some defensive coding to the --workers
argument to accept values between 1 and 10 (a diagram with more than 10 workers probably won't look very good). Next, it's time to change the code to add a validation function:
def valid_range(value: str, upper: int = 10):
try:
int_val = int(value)
if 1 <= int_val <= upper:
return int_val
raise ArgumentTypeError(f"Not true: 1<= {value} <= {upper}")
except ValueError:
raise ArgumentTypeError(f"'{value}' is not an Integer")
if __name__ == "__main__":
PARSER = argparse.ArgumentParser(
description="Generate network diagrams for examples used on this tutorial",
prog=__file__
)
PARSER.add_argument(
'--workers',
action='store',
type=valid_range,
default=1,
help="Number of workers"
)
Of course, you want to learn if this works or not, so relaunch the debugger and set a breakpoint on line 39:
(Pdb) b simple_diagram.py:39
Breakpoint 1 at /home/josevnz/tutorials/PythonDebugger//simple_diagram.py:39
(Pdb) c
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(39)valid_range()
-> try:
(Pdb) l
34 for i in range(workers_n):
35 workers.append(Celery(f"Worker {i}"))
36 airflow - workers
37
38 def valid_range(value: str, upper: int = 10):
39 B-> try:
40 int_val = int(value)
41 if 1 <= int_val <= upper:
42 return int_val
43 raise ArgumentTypeError(f"Not true: 1<= {value} <= {upper}")
44 except ValueError:
(Pdb) a valid_range
value = '-3'
upper = 10
(Pdb) p int(value)
-3
(Pdb) n
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(40)valid_range()
-> int_val = int(value)
(Pdb) n
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(41)valid_range()
-> if 1 <= int_val <= upper:
(Pdb) p 1 <= int_val <= upper
False
(Pdb) n
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(43)valid_range()
-> raise ArgumentTypeError(f"Not true: 1<= {value} <= {upper}")
(Pdb) n
argparse.ArgumentTypeError: Not true: 1<= -3 <= 10
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(43)valid_range()
-> raise ArgumentTypeError(f"Not true: 1<= {value} <= {upper}")
(Pdb) n
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(44)valid_range()
-> except ValueError:
(Pdb) n
--Return--
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(44)valid_range()->None
-> except ValueError:
(Pdb) n
--Call--
> /usr/lib64/python3.9/argparse.py(744)__init__()
-> def __init__(self, argument, message):
(Pdb) c
usage: /home/josevnz/tutorials/PythonDebugger//simple_diagram.py [-h] [--workers WORKERS] diagram
/home/josevnz/tutorials/PythonDebugger//simple_diagram.py: error: argument --workers: Not true: 1<= -3 <= 10
The program exited via sys.exit(). Exit status: 2
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(2)<module>()
-> """
A lot happened here as you moved through using n
and c
:
- The function was called as expected, and the breakpoint on line 39 was called.
- You printed the arguments for the validation function.
- You evaluated (
p
) the sanity checks and confirmed that-3
did not pass the range check, and an exception was raised as a result. - The program exited with an error.
[ Learn how to modernize your IT with managed cloud services. ]
Without the debugger you can confirm what you inspected before:
pythondebugger) $ python3 simple_diagram.py --workers -3 my_airflow2.png
usage: /home/josevnz/tutorials/PythonDebugger//simple_diagram.py [-h] [--workers WORKERS] diagram
/home/josevnz/tutorials/PythonDebugger//simple_diagram.py: error: argument --workers: Not true: 1<= -3 <= 10
# A good call
(pythondebugger) [josevnz@dmaf5 ]$ python3 simple_diagram.py --workers 7 my_airflow2.png && echo "OK"
OK
Learn how to jump before running
You can skip through code by jumping with the debugger. Be aware that depending on where you jump, you may disable the execution of code (but this is useful to understand the workflow of your program). For example, set up a breakpoint on line 32, just before you create the workers
list, and then j
(jump) to line 36 and print the value of workers:
(pythondebugger) $ python3 -m pdb simple_diagram.py --workers 2 my_airflow4.png
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(2)<module>()
-> """
(Pdb) b simple_diagram.py:32
Breakpoint 1 at /home/josevnz/tutorials/PythonDebugger//simple_diagram.py:32
(Pdb) c
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(32)generate_diagram()
-> with Cluster("Celery workers"):
(Pdb) j 36
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(36)generate_diagram()
-> airflow - workers
(Pdb) workers
*** NameError: name 'workers' is not defined
Oh! workers
never initialized. Jump back to line 32, set a breakpoint to line 36, and continue to see what happens:
(Pdb) j 32
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(32)generate_diagram()
-> with Cluster("Celery workers"):
(Pdb) b simple_diagram.py:36
Breakpoint 2 at /home/josevnz/tutorials/PythonDebugger//simple_diagram.py:36
(Pdb) c
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(36)generate_diagram()
-> airflow - workers
(Pdb) p workers
[<onprem.queue.Celery>, <onprem.queue.Celery>]
(Pdb)
Yes, time traveling is also confusing!
Try code with interact
If you were paying close attention, you might have noticed that the label on the Celery workers ranges from Worker 0
to Worker N-1
. Having a Worker 0
is not intuitive. The fix is trivial, so see if you can "hot-fix" the code:
(pythondebugger) $ python3 -m pdb simple_diagram.py --workers 2 my_airflow4.png
> /home/josevnz/tutorials/PythonDebugger//simple_diagram.py(2)<module>()
-> """
(Pdb) b simple_diagram.py:35
Breakpoint 1 at /home/josevnz/tutorials/PythonDebugger//simple_diagram.py:35
(Pdb) c
> /home/josevnz/tutorials/PythonDebugger/simple_diagram.py(35)generate_diagram()
-> workers.append(Celery(f"Worker {i}"))
(Pdb) p i
0
(Pdb) p f"Worker {i}"
'Worker 0'
(Pdb) interact
*interactive*
>>> f"Worker {i}"
'Worker 0'
>>> f"Worker {i+1}"
'Worker 1'
>>> workers.append(Celery(f"Worker {i+1}"))
>>> workers
[<onprem.queue.Celery>]
Here's what happens:
- Set a breakpoint on line 35, which is where you create Celery object instances.
- Print the value of
i
. You see it is0
and will go throughworkers_n - 1
. - Evaluate the expression of
f"Worker {i}"
. - Start an interactive session. This session inherits all the variables and context to the moment of the breakpoint. This means you have access to
i
and theworkers
list. - Test a new expression by adding
i+1
and confirming if the fix works.
This is less expensive than restarting the program with the fix. Also, imagine the value of doing this if your function were much more complex and getting data from remote resources like a database. Pure gold!
Going back to the last fix, replace line 35 with this:
for i in range(workers_n):
workers.append(Celery(f"Worker {i+1}"))
What did you learn?
A lot is possible with the Python debugger. In this tutorial, you:
- Ran an application in debug mode
- Executed a script step by step
- Inspected the content of variables and learned about members of a module
- Used breakpoints and jumps
- Incorporated the debugger in the application
- Applied a hot-fix in the running code
You now know that you don't need an integrated development environment (IDE) to debug and troubleshoot a Python application, and that pdb is a very powerful tool you should have in your toolset.
Image
Write a script in Python that fetches hosts using Nmap to generate dynamic inventories.
Image
Learn to use functions, classes, loops, and more in your Python scripts to simplify common sysadmin tasks.
Image
Learn how to choose the right graphical user interface library for writing user-friendly apps.
Jose Vicente Nunez
Proud dad and husband, software developer and sysadmin. Recreational runner and geek. More about me