|
Size: 4642
Comment:
|
Size: 7563
Comment: rewrite debuggin with GDB
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| Some types of bugs can be difficult to debug from within Python. Some include: | There are types of bugs that are difficult to debug from within Python: |
| Line 7: | Line 7: |
| In these cases, C level debugging with {{{gdb}}} can be helpful (it may be the only way to find out what is going on in some cases). To gather the information, the following steps need to be performed: | In these cases, you can try {{{gdb}}}. |
| Line 9: | Line 9: |
| 1. get a Python interpreter with debugging symbols 1. install Python specific GDB macros (*) 1. run the program under GDB / attach to already running process. 1. obtain backtrace. |
== Prerequisites == |
| Line 14: | Line 11: |
| (*) If you have {{{gdb}}} 7 it includes the ability to debug Python without the need for python gdb macros. See [[https://fedoraproject.org/wiki/Features/EasierPythonDebugging|EasierPythonDebugging]] and [[http://bugs.python.org/issue8032|issue8032]] for instructions. | You need to have `gdb` on your system and Python debugging extensions. Extensions package includes debugging symbols and adds Python-specific commands into `gdb`. On a modern Linux system, you can easily install these with: |
| Line 16: | Line 13: |
| Even if the information obtained doesn't make sense to you, it may be able to help someone else track down the problem. If you are trying to track down an intermittent problem, perform steps 1 and 2 right away and the last steps when the problem occurs. | Fedora: . {{{sudo yum install gdb python-debuginfo}}} |
| Line 18: | Line 16: |
| = Debugging Interpreter = Ubuntu 12.04 provides detached debugging symbols in the {{{python2.7-dbg}}} package: |
Ubuntu: . {{{sudo apt-get install gdb python2.7-dbg}}} |
| Line 21: | Line 19: |
| . {{{sudo apt-get install python2.7-dbg}}} | For `gdb` support on legacy systems, look at the end of this page. |
| Line 23: | Line 21: |
| = GDB Macros = A set of GDB macros are distributed with Python that aid in debugging the Python process. You can install them by adding the contents of `Misc/gdbinit` in the Python sources to `~/.gdbinit` -- or copy it [[http://svn.python.org/view/python/branches/release27-maint/Misc/gdbinit?view=log|from Subversion]]. Be sure to use the correct version for your version of Python or some features will not work. |
== Running with `gdb` == |
| Line 26: | Line 23: |
| Note that the new GDB commands this file adds will only work correctly if debugging symbols are available. | There are two possible ways: |
| Line 28: | Line 25: |
| Also, with gcc 4.5.2 on Ubuntu (at least) the macros fail because the '''call_function''' routine appears to be between '''PyEval_EvalFrameEx''' and '''PyEval_EvalCodeEx''' so the macro fails with '''No symbol "co" in current context.'''. Recompiling python with '''make "CFLAGS=-g -fno-inline -fno-strict-aliasing"''' solves this problem. | 1. run python under `gdb` from the start 2. attach to already running python process |
| Line 30: | Line 28: |
| = Attaching GDB To Python = There are two ways to attach {{{gdb}}} to a Python process: |
To run python under {{{gdb}}} there are also two ways. |
| Line 33: | Line 30: |
| 1. run the program under gdb from the start, wait for the problem 1. attach to the running Python process. To run under {{{gdb}}} from the start, run the following commands: . {{{ |
Interactive: {{{ |
| Line 44: | Line 37: |
| This will run the program til it exits, segfaults or you manually stop execution (using ctrl+C). | Automatic: {{{ $ gdb -ex r --args python <programname>.py <arguments> }}} This will run the program til it exits, segfaults or you manually stop execution (using Ctrl+C). |
| Line 48: | Line 45: |
| . {{{ | {{{ |
| Line 52: | Line 49: |
| Attaching to a running process like this will cause it to stop. You can tell it to continue running | Attaching to a running process like this will cause it to stop. You can tell it to continue running with `c` command. |
| Line 54: | Line 51: |
| == Getting a Stack Trace == | == Debugging process == If your program segfaulted, `gdb` will automatically pause the program, so you can switch into `gdb` console to inspect its state. You can also manually interrupt program execution by pressing Ctrl+C in the console. See the page [[https://fedoraproject.org/wiki/Features/EasierPythonDebugging|EasierPythonDebugging]] for the list of Python helper commands for `gdb`. === Getting a C Stack Trace === |
| Line 76: | Line 80: |
| == Working With Hung Processes == | === Getting a Python Stack Trace === If you have Python extensions installed, you can enter: {{{ (gdb) py-bt }}} to get stack trace with familiar Python source code. === Working With Hung Processes === |
| Line 79: | Line 94: |
| If the process is in a busy loop, you may want to continue execution for a bit (using the {{{cont}}} command), then break (ctrl+C) again and bring up a stack trace. | If the process is in a busy loop, you may want to continue execution for a bit (using the {{{cont}}} command), then break (Ctrl+C) again and bring up a stack trace. |
| Line 81: | Line 96: |
| == Getting Python Stack Traces From GDB == | If the hang occurs in some thread, the following commands may be handy: {{{ (gdb) info threads Id Target Id Frame 37 Thread 0xa29feb40 (LWP 17914) "NotificationThr" 0xb7fdd424 in __kernel_vsyscall () 36 Thread 0xa03fcb40 (LWP 17913) "python2.7" 0xb7fdd424 in __kernel_vsyscall () 35 Thread 0xa0bfdb40 (LWP 17911) "QProcessManager" 0xb7fdd424 in __kernel_vsyscall () 34 Thread 0xa13feb40 (LWP 17910) "python2.7" 0xb7fdd424 in __kernel_vsyscall () 33 Thread 0xa1bffb40 (LWP 17909) "python2.7" 0xb7fdd424 in __kernel_vsyscall () 31 Thread 0xa31ffb40 (LWP 17907) "QFileInfoGather" 0xb7fdd424 in __kernel_vsyscall () 30 Thread 0xa3fdfb40 (LWP 17906) "QInotifyFileSys" 0xb7fdd424 in __kernel_vsyscall () 29 Thread 0xa481cb40 (LWP 17905) "QFileInfoGather" 0xb7fdd424 in __kernel_vsyscall () 7 Thread 0xa508db40 (LWP 17883) "QThread" 0xb7fdd424 in __kernel_vsyscall () 6 Thread 0xa5cebb40 (LWP 17882) "python2.7" 0xb7fdd424 in __kernel_vsyscall () 5 Thread 0xa660cb40 (LWP 17881) "python2.7" 0xb7fdd424 in __kernel_vsyscall () 3 Thread 0xabdffb40 (LWP 17876) "gdbus" 0xb7fdd424 in __kernel_vsyscall () 2 Thread 0xac7b7b40 (LWP 17875) "dconf worker" 0xb7fdd424 in __kernel_vsyscall () * 1 Thread 0xb7d876c0 (LWP 17863) "python2.7" 0xb7fdd424 in __kernel_vsyscall () }}} Current thread is marked with `*`. To see where it is in Python code, use `py-list`: {{{ (gdb) py-list 2025 # Open external files with our Mac app 2026 if sys.platform == "darwin" and 'Spyder.app' in __file__: 2027 main.connect(app, SIGNAL('open_external_file(QString)'), 2028 lambda fname: main.open_external_file(fname)) 2029 >2030 app.exec_() 2031 return main 2032 2033 2034 def __remove_temp_session(): 2035 if osp.isfile(TEMP_SESSION_PATH): }}} To see Python code positions for all threads, use: {{{ (gdb) thread apply all py-list ... 200 201 def accept(self): >202 sock, addr = self._sock.accept() 203 return _socketobject(_sock=sock), addr 204 accept.__doc__ = _realsocket.accept.__doc__ 205 206 def dup(self): 207 """dup() -> socket object Thread 35 (Thread 0xa0bfdb40 (LWP 17911)): Unable to locate python frame Thread 34 (Thread 0xa13feb40 (LWP 17910)): 197 for method in _delegate_methods: 198 setattr(self, method, dummy) 199 close.__doc__ = _realsocket.close.__doc__ 200 201 def accept(self): >202 sock, addr = self._sock.accept() 203 return _socketobject(_sock=sock), addr ... }}} == Legacy systems == It may happen that you need to use `gdb` on a legacy system without advanced Python support. In this case you may find the following information useful. === GDB Macros === A set of GDB macros are distributed with Python that aid in debugging the Python process. You can install them by adding the contents of `Misc/gdbinit` in the Python sources to `~/.gdbinit` -- or copy it [[http://svn.python.org/view/python/branches/release27-maint/Misc/gdbinit?view=log|from Subversion]]. Be sure to use the correct version for your version of Python or some features will not work. Note that the new GDB commands this file adds will only work correctly if debugging symbols are available. Also, with gcc 4.5.2 on Ubuntu (at least) the macros fail because the '''call_function''' routine appears to be between '''PyEval_EvalFrameEx''' and '''PyEval_EvalCodeEx''' so the macro fails with '''No symbol "co" in current context.'''. Recompiling python with '''make "CFLAGS=-g -fno-inline -fno-strict-aliasing"''' solves this problem. === Getting Python Stack Traces With GDB Macros === |
| Line 84: | Line 174: |
| . {{{ | {{{ |
| Line 90: | Line 180: |
| . {{{ | {{{ |
| Line 94: | Line 184: |
| == More useful macros not in python's gdbinit file == | ==== More useful macros not in python's gdbinit file ==== |
There are types of bugs that are difficult to debug from within Python:
- segfaults (not uncaught Python exceptions)
hung processes (in cases where you can't get a Python traceback or debug with pdb)
- out of control daemon processes
In these cases, you can try gdb.
Prerequisites
You need to have gdb on your system and Python debugging extensions. Extensions package includes debugging symbols and adds Python-specific commands into gdb. On a modern Linux system, you can easily install these with:
Fedora:
sudo yum install gdb python-debuginfo
Ubuntu:
sudo apt-get install gdb python2.7-dbg
For gdb support on legacy systems, look at the end of this page.
Running with `gdb`
There are two possible ways:
run python under gdb from the start
- attach to already running python process
To run python under gdb there are also two ways.
Interactive:
$ gdb python ... (gdb) run <programname>.py <arguments>
Automatic:
$ gdb -ex r --args python <programname>.py <arguments>
This will run the program til it exits, segfaults or you manually stop execution (using Ctrl+C).
If the process is already running, you can attach to it provided you know the process ID.
$ gdb python <pid of running process>
Attaching to a running process like this will cause it to stop. You can tell it to continue running with c command.
Debugging process
If your program segfaulted, gdb will automatically pause the program, so you can switch into gdb console to inspect its state. You can also manually interrupt program execution by pressing Ctrl+C in the console.
See the page EasierPythonDebugging for the list of Python helper commands for gdb.
Getting a C Stack Trace
If you are debugging a segfault, this is probably the first thing you want to do.
At the (gdb) prompt, just run the following command:
(gdb) bt #0 0x0000002a95b3b705 in raise () from /lib/libc.so.6 #1 0x0000002a95b3ce8e in abort () from /lib/libc.so.6 #2 0x00000000004c164f in posix_abort (self=0x0, noargs=0x0) at ../Modules/posixmodule.c:7158 #3 0x0000000000489fac in call_function (pp_stack=0x7fbffff110, oparg=0) at ../Python/ceval.c:3531 #4 0x0000000000485fc2 in PyEval_EvalFrame (f=0x66ccd8) at ../Python/ceval.c:2163 ...
With luck, this will give some idea of where the problem is occurring and if it doesn't help you fix the problem, it can help someone else track down the problem.
The quality of the results will depend greatly on the amount of debug information available.
Getting a Python Stack Trace
If you have Python extensions installed, you can enter:
(gdb) py-bt
to get stack trace with familiar Python source code.
Working With Hung Processes
If a process appears hung, it will either be waiting on something (a lock, IO, etc), or be in a busy loop somewhere. In either case, attaching to the process and getting a back trace can help.
If the process is in a busy loop, you may want to continue execution for a bit (using the cont command), then break (Ctrl+C) again and bring up a stack trace.
If the hang occurs in some thread, the following commands may be handy:
(gdb) info threads Id Target Id Frame 37 Thread 0xa29feb40 (LWP 17914) "NotificationThr" 0xb7fdd424 in __kernel_vsyscall () 36 Thread 0xa03fcb40 (LWP 17913) "python2.7" 0xb7fdd424 in __kernel_vsyscall () 35 Thread 0xa0bfdb40 (LWP 17911) "QProcessManager" 0xb7fdd424 in __kernel_vsyscall () 34 Thread 0xa13feb40 (LWP 17910) "python2.7" 0xb7fdd424 in __kernel_vsyscall () 33 Thread 0xa1bffb40 (LWP 17909) "python2.7" 0xb7fdd424 in __kernel_vsyscall () 31 Thread 0xa31ffb40 (LWP 17907) "QFileInfoGather" 0xb7fdd424 in __kernel_vsyscall () 30 Thread 0xa3fdfb40 (LWP 17906) "QInotifyFileSys" 0xb7fdd424 in __kernel_vsyscall () 29 Thread 0xa481cb40 (LWP 17905) "QFileInfoGather" 0xb7fdd424 in __kernel_vsyscall () 7 Thread 0xa508db40 (LWP 17883) "QThread" 0xb7fdd424 in __kernel_vsyscall () 6 Thread 0xa5cebb40 (LWP 17882) "python2.7" 0xb7fdd424 in __kernel_vsyscall () 5 Thread 0xa660cb40 (LWP 17881) "python2.7" 0xb7fdd424 in __kernel_vsyscall () 3 Thread 0xabdffb40 (LWP 17876) "gdbus" 0xb7fdd424 in __kernel_vsyscall () 2 Thread 0xac7b7b40 (LWP 17875) "dconf worker" 0xb7fdd424 in __kernel_vsyscall () * 1 Thread 0xb7d876c0 (LWP 17863) "python2.7" 0xb7fdd424 in __kernel_vsyscall ()
Current thread is marked with *. To see where it is in Python code, use py-list:
(gdb) py-list 2025 # Open external files with our Mac app 2026 if sys.platform == "darwin" and 'Spyder.app' in __file__: 2027 main.connect(app, SIGNAL('open_external_file(QString)'), 2028 lambda fname: main.open_external_file(fname)) 2029 >2030 app.exec_() 2031 return main 2032 2033 2034 def __remove_temp_session(): 2035 if osp.isfile(TEMP_SESSION_PATH):
To see Python code positions for all threads, use:
(gdb) thread apply all py-list ... 200 201 def accept(self): >202 sock, addr = self._sock.accept() 203 return _socketobject(_sock=sock), addr 204 accept.__doc__ = _realsocket.accept.__doc__ 205 206 def dup(self): 207 """dup() -> socket object Thread 35 (Thread 0xa0bfdb40 (LWP 17911)): Unable to locate python frame Thread 34 (Thread 0xa13feb40 (LWP 17910)): 197 for method in _delegate_methods: 198 setattr(self, method, dummy) 199 close.__doc__ = _realsocket.close.__doc__ 200 201 def accept(self): >202 sock, addr = self._sock.accept() 203 return _socketobject(_sock=sock), addr ...
Legacy systems
It may happen that you need to use gdb on a legacy system without advanced Python support. In this case you may find the following information useful.
GDB Macros
A set of GDB macros are distributed with Python that aid in debugging the Python process. You can install them by adding the contents of Misc/gdbinit in the Python sources to ~/.gdbinit -- or copy it from Subversion. Be sure to use the correct version for your version of Python or some features will not work.
Note that the new GDB commands this file adds will only work correctly if debugging symbols are available.
Also, with gcc 4.5.2 on Ubuntu (at least) the macros fail because the call_function routine appears to be between PyEval_EvalFrameEx and PyEval_EvalCodeEx so the macro fails with No symbol "co" in current context.. Recompiling python with make "CFLAGS=-g -fno-inline -fno-strict-aliasing" solves this problem.
Getting Python Stack Traces With GDB Macros
At the gdb prompt, you can get a Python stack trace:
(gdb) pystack
Alternatively, you can get a list of the Python locals along with each stack frame:
(gdb) pystackv
More useful macros not in python's gdbinit file
See http://web.archive.org/web/20070915134837/http://www.mashebali.com/?Python_GDB_macros:The_Macros for some more handy python gdb macros.