Building your own Python version for an easier debugging experience.

One of the major issues in using the stock Python distribution (i.e. Ubuntu) is that it's compiled with a bunch of optimizations in the final binary. You can install the debug symbols (via the python-dbg package), but if you want to be able to troubleshoot any C-based extensions, it's likely you're going to want to be able to examine what lines actually triggered segmentation dumps by using the core file.   In addition, if you can also break into a running Python process or want to inspect the memory structures to see what's actually being allocated, you unfortunately can't do this type of debugging with the standard build.

What if you simply wanted to have a version of Python that could be used for debugging? What if you still wanted to keep the standard version too? Here's the steps that I took to accomplish this goal.  To make sure the bzip2 and sqlite3 modules were installed in the Python distribution, I had to install the development packages for them.  I also had to override the default C compile flags by using OPT="" and CFLAGS="-O1 -g" (the -g flag produces debugging symbols)

cd /somewhere/you/want/source/code/
sudo apt-get source python
sudo apt-get install libbz2-dev
sudo apt-get install libsqlite3-dev
OPT="" CFLAGS="-O1 -g" ./configure --prefix=/home/rhu/.virtualenvs/pydev --with-pydebug
make install

The next step is to install the virtualenv package from source. Normally you don't need to do this part but because we are using a custom Python binary, we need to use it for the compile step.  You can download the latest version from http://pypi.python.org/packages/source/v/virtualenv.  I decided to store it in the .virtualenvs directory that is normally created by the virtualenvwrapper package, which provides shortcuts to activate and deactivate Python virtual environments.

tar -zxvf virtualenv-1.x.x.tar.gz
cd virtualenv-1.x.x/
~//home/rhu/.virtualenvs/pydev/bin/python setup.py install

You'll need to activate the virtual environment by running the activate script. If you have the virtualenvwrapper package setup, you can also just type "workon pydev".

Note that all your Python packages will need to be recompiled.   They should inherit the same compiler flags used for building your Python package, so you should see "-O1 -g" compiler optimizations being used.  Note that I used the --with-pydebug flag, which will do some reference debugging and make your existing libraries incompatible.  By doing all of this work inside a virtual environment, you can avoid library conflicts with other installations.

(Note: using -O0 caused some code such as the librabbitmq to fail to compile for some reason, so I opted to keep the minimal amount of optimizations enabled.  If someone figures out what the minimal compiler optimizations to use that would allow the -O0 flag to be used, please let me know!  )