Why your Python program can't start when using python-dbg...

Recently, I installed python-dbg and started noticing errors when trying to load C-extension modules such as the PyCrypto library. There are errors such as "undefined symbol" called by Py_InitModule4_64:

>>> import Crypto.Cipher.AES
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/local/lib/python2.7/dist-packages/Crypto/Cipher/AES.py", line 50, in 
    from Crypto.Cipher import _AES
ImportError: /usr/local/lib/python2.7/dist-packages/Crypto/Cipher/_AES.so: undefined symbol: Py_InitModule4_64
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/apport_python_hook.py", line 66, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python2.7/dist-packages/apport/__init__.py", line 1, in 
    from apport.report import Report
  File "/usr/lib/python2.7/dist-packages/apport/report.py", line 20, in 
    import apport.fileutils
  File "/usr/lib/python2.7/dist-packages/apport/fileutils.py", line 22, in 
    from apport.packaging_impl import impl as packaging
  File "/usr/lib/python2.7/dist-packages/apport/packaging_impl.py", line 20, in 
    import apt
  File "/usr/lib/python2.7/dist-packages/apt/__init__.py", line 21, in 
    import apt_pkg
ImportError: /usr/lib/python2.7/dist-packages/apt_pkg.so: undefined symbol: Py_InitModule4_64

Original exception was:
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/local/lib/python2.7/dist-packages/Crypto/Cipher/AES.py", line 50, in 
    from Crypto.Cipher import _AES
ImportError: /usr/local/lib/python2.7/dist-packages/Crypto/Cipher/_AES.so: undefined symbol: Py_InitModule4_64

What's going on? Apparently the python2.7-dbg binary is configured with the --with-pydebug flag, which turns on the Py_DEBUG ifdef directive, which turns on the Py_TRACE_REFS directive:

From the /usr/share/doc/python2.7-dbg/README.debug file:

python2.7-dbg contains two sets of packages:

 - debugging symbols for the standard python2.7 build. When this package
   is installed, gdb will automatically load up the debugging symbols
   from it when debugging python2.7 or one of the included extension
   modules.

 - a separate python2.7-dbg binary, configured --with-pydebug, enabling the
   additional debugging code to help debug memory management problems.
.
.
   Py_DEBUG implies LLTRACE, Py_REF_DEBUG,
   Py_TRACE_REFS, and PYMALLOC_DEBUG (if WITH_PYMALLOC is enabled).
   In addition, C assert()s are enabled (via the C way: by not defining
   NDEBUG), and some routines do additional sanity checks inside
   "#ifdef Py_DEBUG" blocks.

We can find this definition declared inside the Python source code under Include/modsupport.h file. Normally under 64-bit platforms,
the Py_InitModule4 is renamed to Py_InitModule4_64 as defined in PEP0353.

#ifdef Py_TRACE_REFS
 /* When we are tracing reference counts, rename Py_InitModule4 so
    modules compiled with incompatible settings will generate a
    link-time error. */
 #if SIZEOF_SIZE_T != SIZEOF_INT
 #undef Py_InitModule4
 #define Py_InitModule4 Py_InitModule4TraceRefs_64
 #else
 #define Py_InitModule4 Py_InitModule4TraceRefs
 #endif
#endif

Once the python2.7-dbg package is installed, you can use gdb with the Python debugging symbols but it apparently doesn't provide nearly the rich annotated information supplied with the Py_TRACE_REF flag. The problem though is the python-dbg binary however is incompatible with modules not compiled with them as mentioned in /usr/share/doc/python2.7/SpecialBuilds.txt.gz:

Py_TRACE_REFS
-------------

Turn on heavy reference debugging.  This is major surgery.  Every PyObject grows
two more pointers, to maintain a doubly-linked list of all live heap-allocated
objects.  Most built-in type objects are not in this list, as they're statically
allocated.  Starting in Python 2.3, if COUNT_ALLOCS (see below) is also defined,
a static type object T does appear in this list if at least one object of type T
has been created.

Note that because the fundamental PyObject layout changes, Python modules
compiled with Py_TRACE_REFS are incompatible with modules compiled without it.

Py_TRACE_REFS implies Py_REF_DEBUG.

The solution seems to be to use gdb with the standard debugging symbols and the default Python binary (which appear to have compiler optimizations that prevent gdb from reading Python frame information), or recompiling all your C-extension modules with the --with-pydebug flag.  Ubuntu comes with -dbg packages for many Python libraries, but if you have ones normally installed via pip installs, these may need to be recompiled manually (i.e. via python-dbg setup.py --debug).