Installing PIP packages, and using them from IDA on a 64-bit machine
Recently, one of our customers came to us asking how he should proceed to be able to install python packages, using PIP, and use those from IDA.
The issue he was facing is that his system is a 64-bit Ubuntu 12.04 VM.
Therefore using the Ubuntu-bundled PIP will just result in installing the desired package (let’s say
ssdeep) for the system Python runtime, which is a 64-bit runtime and therefore not compatible with IDA.
The best (as in: cleanest) solution I have found is to:
- build a 32-bits python on the system.
pip-install packages in that 32-bits python’s sub-directories.
PYTHONPATHto point to the 32-bits python’s sub-directories.
We figured we’d write it down here just in case it might help others.
- Install autoconf
- Install ia32-libs
Building & installing a 32-bits python
..$ export LD_LIBRARY_PATH=/lib/i386-linux-gnu/:/usr/lib32:$LD_LIBRARY_PATH
- Download Python2.7.4
- Note:You should make sure that the MD5 checksum and the size of the file you downloaded match those that are advertised on the page. That would prevent a man-in-the-middle attacker from providing you with a malicious Python bundle.
- Build it. Note that you’ll probably have to sudo-create a few symlinks. I had to do this, on the Ubuntu 12.04 64-bit VM I tested this on:
- For the sake of completeness, here are my build commands (don’t forget the flags, of course):
..$ CFLAGS=-m32 LDFLAGS=-m32 ./configure --prefix=/opt/Python2.7.4-32bits
..$ CFLAGS=-m32 LDFLAGS=-m32 make -j 8
Once the build completes
Here’s what I have as last lines of the build:
INFO: Can't locate Tcl/Tk libs and/or headers Python build finished, but the necessary bits to build these modules were not found: _bsddb _curses _curses_panel _sqlite3 _tkinter bsddb185 bz2 dbm gdbm readline sunaudiodev To find the necessary bits, look in setup.py in detect_modules() for the module's name.
If you see, below that, that it failed to build, say
'binascii', then something went wrong.
Make sure you run
make -j 1 to check out what went wrong (i.e., what library it claims not being able to find)
Once you have succesfully built your 32-bits Python, it’s time to install it:
sudo make install
Trying your freshly-built python
..$ /opt/Python2.7.4-32bits/bin/python2.7 Python 2.7.4 (default, Apr 26 2013, 16:03:38) [GCC 4.6.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import binascii >>>
No complaint so far. Good.
pkg_resources is available.
pkg_resources. If it fails, you’ll probably have to do the following:
..$ cd /tmp ..$ curl -O http://python-distribute.org/distribute_setup.py ..$ less distribute_setup.py # (*) ..$ sudo /opt/Python2.7.4-32bits/bin/python2.7 distribute_setup.py
That will print out quite a fair amount of info, and should succeed.
(*) Note: A careful reader has pointed out that it would be fairly easy to intercept (man-in-the-middle) such an HTTP request, and serve malicious content that would then be piped (as root) to Python.
That’s why I think it’s important to mention, as a third step (i.e.,
less ...), that the code that was downloaded should ideally be checked. Hopefully, http://python-distribute.org will soon provide HTTPS support, which will limit such MITM attack risks.
Trying your freshly-built python, again
We want to make sure
pkg_resources can be imported.
..$ /opt/Python2.7.4-32bits/bin/python2.7 Python 2.7.4 (default, Apr 26 2013, 16:03:38) [GCC 4.6.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import pkg_resources >>>
Still no complaint. Good.
If yours complains, you’ll have to first make sure you fix whatever is causing it to fail, because the next will not work without that.
Installing PIP for your new Python build
Since using your system’s PIP will probably not work (as it would build & install things in a 64-bits python sub-directory), you’ll have to install a PIP package specifically for your freshly-built Python.
Here’s how I proceeded:
..$ cd /tmp; ..$ curl -O https://raw.github.com/pypa/pip/master/contrib/get-pip.py; ..$ sudo /opt/Python2.7.4-32bits/bin/python2.7 get-pip.py
PIP is now installed.
PIP-installing a package (i.e.,
To download/build/install the
ssdeep package I ran, as root (either that, or you’ll have to give your user the rights to write in /opt/Python2.7.4-32bits):
..$ su Password: root ..$ export CFLAGS=-m32 root ..$ export LDFLAGS=-m32 root ..$ export LD_LIBRARY_PATH=/lib/i386-linux-gnu/:/usr/lib32:$LD_LIBRARY_PATH root ..$ /opt/Python2.7.4-32bits/bin/python2.7 /opt/Python2.7.4-32bits/bin/pip install ssdeep
Notice how I use my freshly-built python, with my fresly-installed PIP (and not the system one.)
Note: Don’t forget the
export lines, or PIP will partially build stuff for x64, and partially for x86. That, as you can guess, won’t quite work.
If you forgot the
export lines and started building anyway (and the build failed because of the mixed architecture issue I just wrote about), make sure you delete whatever is in
/tmp/pip-build-*, so that there won’t be stale object files of inappropriate architecture in there.
Check out the PIP-installed package works
..$ /opt/Python2.7.4-32bits/bin/python2.7 Python 2.7.4 (default, Apr 26 2013, 16:03:38) [GCC 4.6.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import ssdeep >>> ssdeep <module 'ssdeep' from '/opt/Python2.7.4-32bits/lib/python2.7/site-packages/ssdeep.so'> >>> dir(ssdeep) ['Error', '__all__', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__test__', '__version__', 'compare', 'hash', 'hash_from_file', 'sys'] >>>
So far so good.
Testing the PIP-installed package in IDA
Since that’s still the goal (though you might have forgotten by now, given the amount of directions above.. 😉 ),
we’ll now try and make use of that PIP-installed package in IDA.
..$ export PYTHONPATH=/opt/Python2.7.4-32bits/lib/python2.7/site-packages:/opt/Python2.7.4-32bits/:$PYTHONPATH
If all went well, typing
import ssdeep in the Python input line should properly, silently, nicely import the package.