Adventures with Heroku: python-saml, libxmlsec1, and SWIG

Posted by: jag 3 years, 2 months ago
Posted in: Coding

(Comments)

A client uses OneLogin for their single-sign on services. We're building their website hosted on Heroku. We needed to build that SSO integration, and fortunately the folks over at OneLogin have released a very adaptable and well demonstrated toolkit for writing SAML SSO integration in Python. Heroku was the hard part.

The python-saml library requires several binary libraries: libxml2, libxmlsec1, and SWIG (a library for writing thin wrappers around C libraries in scripting languages). Out of those, only libxml2 is installed in the standard Heroku Cedar-14 stack. The latter two have to be installed by a buildpack.

Ex-Herokuling David Dollar, builder of several of the canonical buildpacks, authored a buildpack to install additional packages from the Ubuntu Trusty apt repositories. When combined with the Heroku multi-buildpack, a meta-buildpack that runs multiple buildpack compiles for a single project (also originally authored by Dollar), this buildpack will install the required packages to $BUILD_DIR/.apt and set appropriate build environment variables (like INCLUDE_PATH and LD_LIBRARY_PATH) for later buildpacks to use. All we have to do is make two files:

# .buildpacks
https://github.com/ddollar/heroku-buildpack-apt#master
https://github.com/heroku/heroku-buildpack-python#v57

# Aptfile
libxmlsec1
libxmlsec1-dev
swig

Sounds great.

But it only sounds that way. We run into a few problems right off the bat.

First, the standard Heroku buildpack for Python unsets and stomps on the environment variables that would allow the compiler to find our apt-installed headers and for the linker to find our apt-installed libraries. Second, even if it didn't, the environment variables set by the apt buildpack are invalid for the Python buildpack. The Python buildpack moves your $BUILD_DIR to /app during the build process and moves it back afterward - so the paths that the apt buildpack set referecing the original $BUILD_DIR are invalid and don't exist at compile-time. Third, the dm.xmlsec.binding library uses xmlsec1-config to determine its cflags and libs flags. The xmlsec1-config utility answers based on the path that the package was configured to be installed to, using /usr as its base path; we need it to report /app/.apt/usr instead. Finally, the M2Crypto library uses SWIG and swig -swiglib reports the path for its configuration files to be where it was built to be installed to, which is /usr/share/swig2.0 - correspondingly we need it to report /app/.apt/usr/share/swig2.0 instead.

Oof.

To finally get our Heroku app through the build process, we had to make the following changes:

  1. We had to export SWIG_LIB=/app/.apt/usr/share/swig2.0 into the environment - this changes the base path that swig -swiglib reports.
  2. We had to make our own archive for dm.xmlsec.binding and include it specifically in our requirements.txt - in it, we merely modified the setup.py to acknowledge an environment variable XMLSEC_CONFIG to which we set to --base /app/.apt/usr and passed that variable value along to setup.py's execution of xmlsec1-config.
  3. We had to make our own fork of the stock Python buildpack, in which we made a symlink at $ORIG_BUILD_DIR/.apt to /app/.apt before compile, removed the symlink after compile, and preserved the existing environment variable values for compiler and linker directives like INCLUDE_PATH and LD_LIBRARY_PATH.

Once all of that was done, it built like a charm. But I'm really not thrilled with the custom buildpack and the custom setup script. But if you're likewise beating your head against the wall trying to get python-saml installed on Heroku and can't because of compile errors in building dm.xmlsec.binding, hope this helps.

Current rating: 4.5

Comments