Installing PIP packages, and using them from IDA on a 64-bit machine
Recently, one of our customers came to us asking how he should proceed to be able to install python packages, using PIP, and use those from IDA. The issue he was facing is that his system is a 64-bit Ubuntu 12.04 VM. Therefore using the Ubuntu-bundled PIP will just result in installing the desired package (let’s say
PIP-installing a package (i.e.,
To download/build/install the
ssdeep) for the system Python runtime, which is a 64-bit runtime and therefore not compatible with IDA. The best (as in: cleanest) solution I have found is to:
- build a 32-bits python on the system.
pip-install packages in that 32-bits python’s sub-directories.
PYTHONPATHto point to the 32-bits python’s sub-directories.
- Install autoconf
- Install ia32-libs
Building & installing a 32-bits python
..$ export LD_LIBRARY_PATH=/lib/i386-linux-gnu/:/usr/lib32:$LD_LIBRARY_PATH
- Download Python2.7.4
- Note:You should make sure that the MD5 checksum and the size of the file you downloaded match those that are advertised on the page. That would prevent a man-in-the-middle attacker from providing you with a malicious Python bundle.
- Build it. Note that you’ll probably have to sudo-create a few symlinks. I had to do this, on the Ubuntu 12.04 64-bit VM I tested this on:
- For the sake of completeness, here are my build commands (don’t forget the flags, of course):
..$ CFLAGS=-m32 LDFLAGS=-m32 ./configure --prefix=/opt/Python2.7.4-32bits
..$ CFLAGS=-m32 LDFLAGS=-m32 make -j 8
Once the build completesHere’s what I have as last lines of the build:
INFO: Can't locate Tcl/Tk libs and/or headers Python build finished, but the necessary bits to build these modules were not found: _bsddb _curses _curses_panel _sqlite3 _tkinter bsddb185 bz2 dbm gdbm readline sunaudiodev To find the necessary bits, look in setup.py in detect_modules() for the module's name.If you see, below that, that it failed to build, say
'binascii', then something went wrong. Make sure you run
make -j 1to check out what went wrong (i.e., what library it claims not being able to find) Once you have succesfully built your 32-bits Python, it’s time to install it:
sudo make install
Trying your freshly-built python
..$ /opt/Python2.7.4-32bits/bin/python2.7 Python 2.7.4 (default, Apr 26 2013, 16:03:38) [GCC 4.6.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import binascii >>>No complaint so far. Good.
pkg_resources is available.
pkg_resources. If it fails, you’ll probably have to do the following:
..$ cd /tmp ..$ curl -O http://python-distribute.org/distribute_setup.py ..$ less distribute_setup.py # (*) ..$ sudo /opt/Python2.7.4-32bits/bin/python2.7 distribute_setup.pyThat will print out quite a fair amount of info, and should succeed. (*) Note: A careful reader has pointed out that it would be fairly easy to intercept (man-in-the-middle) such an HTTP request, and serve malicious content that would then be piped (as root) to Python. That’s why I think it’s important to mention, as a third step (i.e.,
less ...), that the code that was downloaded should ideally be checked. Hopefully, http://python-distribute.org will soon provide HTTPS support, which will limit such MITM attack risks.
Trying your freshly-built python, againWe want to make sure
pkg_resourcescan be imported.
..$ /opt/Python2.7.4-32bits/bin/python2.7 Python 2.7.4 (default, Apr 26 2013, 16:03:38) [GCC 4.6.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import pkg_resources >>>Still no complaint. Good. If yours complains, you’ll have to first make sure you fix whatever is causing it to fail, because the next will not work without that.
Installing PIP for your new Python buildSince using your system’s PIP will probably not work (as it would build & install things in a 64-bits python sub-directory), you’ll have to install a PIP package specifically for your freshly-built Python. Here’s how I proceeded:
..$ cd /tmp; ..$ curl -O https://raw.github.com/pypa/pip/master/contrib/get-pip.py; ..$ sudo /opt/Python2.7.4-32bits/bin/python2.7 get-pip.pyPIP is now installed.
PIP-installing a package (i.e.,
To download/build/install the
ssdeeppackage I ran, as root (either that, or you’ll have to give your user the rights to write in /opt/Python2.7.4-32bits):
..$ su Password: root ..$ export CFLAGS=-m32 root ..$ export LDFLAGS=-m32 root ..$ export LD_LIBRARY_PATH=/lib/i386-linux-gnu/:/usr/lib32:$LD_LIBRARY_PATH root ..$ /opt/Python2.7.4-32bits/bin/python2.7 /opt/Python2.7.4-32bits/bin/pip install ssdeepNotice how I use my freshly-built python, with my fresly-installed PIP (and not the system one.) Note: Don’t forget the
exportlines, or PIP will partially build stuff for x64, and partially for x86. That, as you can guess, won’t quite work. If you forgot the
exportlines and started building anyway (and the build failed because of the mixed architecture issue I just wrote about), make sure you delete whatever is in
/tmp/pip-build-*, so that there won’t be stale object files of inappropriate architecture in there.
Check out the PIP-installed package works
..$ /opt/Python2.7.4-32bits/bin/python2.7 Python 2.7.4 (default, Apr 26 2013, 16:03:38) [GCC 4.6.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import ssdeep >>> ssdeep <module 'ssdeep' from '/opt/Python2.7.4-32bits/lib/python2.7/site-packages/ssdeep.so'> >>> dir(ssdeep) ['Error', '__all__', '__builtins__', '__doc__', '__file__', '__name__', '__package__', '__test__', '__version__', 'compare', 'hash', 'hash_from_file', 'sys'] >>>So far so good.
Testing the PIP-installed package in IDASince that’s still the goal (though you might have forgotten by now, given the amount of directions above.. 😉 ), we’ll now try and make use of that PIP-installed package in IDA.
..$ export PYTHONPATH=/opt/Python2.7.4-32bits/lib/python2.7/site-packages:/opt/Python2.7.4-32bits/:$PYTHONPATH
import ssdeepin the Python input line should properly, silently, nicely import the package.