Just another WordPress.com weblog

Archive for June, 2012

Bundling VM with thirdparty libraries

Hi, there.. (or here? 😉 )

You can stop reading if you using Linux or any other flavor of unix with decent package management.

However, if you using MacOS , or Windows.. you may continue reading. 🙂

Problem: suppose you want to add a plugin to VM, which using some well-known open-source library.
On linux, this is piece of cake.. you just install that library using package management tool , like:

apt-get install somelibrary

and tell plugin to look the installed library at well known place in system. Done.

On Mac and Windows, things are different: you are on you own.. There is no well-known place in system (unless it is of course Apple/Microsoft product), defined where all 3rd-party/shared libraries installed. And often, there’s even no binary distribution for such library, and even if it exists,
if you install it, you never know where it is located, because again there is no standard place.
So, the only solution is to bundle these libs with own app. In most cases you have to download sources from project site and build it by own.

So, the first thing which rang bells in my head is the way how we deal with it today.
For instance, Freetype: If you using Pharo , you know that it provides a freetype support by default,
as well as VMs which we build on Jenkins CI server including corresponding freetype plugin.

From user’s perspective, everything is nice: he can use nice fonts in images.. but if we open a cover of our shiny car, things are not so nice. And i will explain why:

On windows and macs, we were forced to include a whole freetype library as a static component of plugin.. i.e. it just statically linked to FT2Plugin.
What is wrong with such approach? Suppose i want to add another plugin, which using very same library. Now, if i do the same, it will result in having two separate (autonomous) instances of same library used by two different plugins..

And this is exactly what happening with Cairo (which i use in Athens via FFI) because cairo also using freetype (what a coincidence!).
So, in Pharo, we already have freetype. Now Cairo also using freetype. But since cairo is loaded using FFI, there’s no way how it can communicate with statically linked freetype library resided inside FT2Plugin. As result, when Athens using Cairo you have two instances of same library loaded..
But even worse: i cannot reuse existing code from Pharo freetype support code and cope it with cairo. Because if i load a font face using freetype plugin, i cannot pass that face to cairo, because the face object instantiated by one library, and cairo using another one, which know absolutely nothing about each other.

So, obviously an appropriate solution would be to force both FT2Plugin and Cairo library to use same shared, dynamically loadable, library and abandon idea of statically linking a library to single plugin.
Also, since we cannot expect that it will be installed on system somewhere, we should bundle it with our application (which in our case a VM).

Btw, a freetype is not the sole example of statically embedding 3rd-party libraries into out beloved VM. We have couple more under cover:
– FloatMathPlugin includes whole fdlibm
– JPEGReadWriter2Plugin includes a copy of JPEG library
– RePlugin includes PCRE library (perl-compatible regular expressions)

and i think there’s more. And since those libraries are private to those plugins, if some another plugin want to use same functionality, there is no way how they can share same library -> so, you got code duplication and memory waste.

So, i decided to extend the CMakeVMMaker to have a notion of 3rd-party library, and our automated build system can:
– download library from official site
– configure & build it
– bundle it alongside with VM

Here the brief description of a new interface.

First, i introduced a class, named CMThirdpartyLibrary to describe the configuration of thirdparty library as well as a steps to build it and bundle with VM.

This is an abstract class, which shapes a most common attributes of 3rd-party package as well as a more or less standard way to build & include it to VM. A subclass of it can describe the concrete library, as well as customize the way it is built.

So, let’s take a concrete example – CMFreetype2 class, which defines a freetype2 library.

First thing: a library name.

CMFreetype2 class>>canonicalName
^ 'freetype2'

This is a name of a library, used to identify it. For obvious reason, there should be no two classes with same canonical name.
To include a library into cmake loop, use #addThirdpartyLibrary: <libname> message, sent to configuration.

For example:

NBCogCocoaIOSConfig new
addThirdpartyLibrary: 'freetype2';

This will add necessary instructions to generated CMakeLists.txt file to include library & bundle it with VM.

This is how, FT2Plugin using it now:

MacOSConfig>>configureFT2Plugin: maker
“extra rules for Freetype plugin”
| lib |

maker isExternal ifFalse: [
self error: ‘building internal FT2Plugin is not supported yet’

“add freetype library into loop”
lib := self addThirdpartyLibrary: ‘freetype2’.

“link plugin with freetype lib”
maker addExternalLibrary: lib targetForLinking.
maker includeDirectories: lib includeDir.

A canonical name also serves as a build target. So, you can do in command-line:

build#  cmake .
build#  make freetype2

this will build a freetype2 library.

The class structure breaks a build process on common stages, which should be more or less similar to all libs:

gen message: ‘Configuring thirdparty package: ‘, self canonicalName.


Setting variables

The #setVariables method serves mainly to set-up a most common variables in cmake, like paths, names , options and so on, so other parts of generated cmake script can reuse them without duplicating definitions over and over again.

Here variables set by CMThirdpartyLibrary class:


set: #libName toString: self canonicalName;
set: #workDir toString: ‘${thirdpartyDir}/${libName}’;
set: #unpackedDirName toString: self unpackedDirName;
set: #libSourcesDir toString: ‘${workDir}/${unpackedDirName}’;
set: #url toString: self downloadURL;
set: #md5sum toString: self archiveMD5Sum;
set: #installPrefix toString: ‘${thirdpartyDir}/out’.

we add few more in freetype:

super setVariables.

“add include path”
set: #freetype2_includeDir toString: ‘${installPrefix}/include’;
set: #libraryFileName to: self libraryFileName;
set: #freetype2_location toString: ‘${externalModulesDir}/${libraryFileName}’;
set: #ft2config toString: ‘${libSourcesDir}/builds/unix/config.status’;
set: #ft2libInstalled toString: ‘${installPrefix}/lib/${libraryFileName}’

Downloading and unpacking sources.


^ ‘http://ftp.igh.cnrs.fr/pub/nongnu/freetype/freetype-2.4.9.tar.gz&#8217;

^ ‘c15f6dc8ed190d67b89ae09aaf7896b4’

^ ‘freetype-2.4.9’

(the #unpack method is provided by CMThirdpartyLibrary class, which using tar to unpack the downloaded file. Of course if library sources using different archive format (like zip), you will need to override that method to emit an appropriate shell commands to unpack it.

By default, all thirdparty libraries using a ‘thirdparty’ subdirectory in ‘build’ directory:

[~/projects/cog/sig-cog/build/thirdParty]: ls
cairo      freetype2  libpng     out        pixman     pkg-config


Here, each subdirectory , except ‘out’ created automatically by concrete library configuration which contains downloaded source code,
as well as CMakeLists.txt generated by CMakeVMMaker.

Configuring and building library

The #build method is nothing more than invoking configure, make , make install in freetype source subdirectory..
Which is 3 “lines of code” in shell command line, but a bit of boilerplate in cmake 😉


add_custom_command(OUTPUT “${ft2config}”
COMMAND ./configure –prefix=”${installPrefix}” CFLAGS=”-arch i386” LDFLAGS=”-arch i386”
WORKING_DIRECTORY “${libSourcesDir}”
DEPENDS “${unpackTarget}”
add_custom_command(OUTPUT “${ft2libInstalled}”
COMMAND make install
WORKING_DIRECTORY “${libSourcesDir}”
DEPENDS “${ft2config}”
COMMENT “Building ${libName}”

Since most of the libraries usually having own unique idiosyncrasic way how they are built, this method is one which you will need to change when adopting new library.
Here, as you can see i use custom prefix ${installPrefix}, passed to configure command. So `make install` will copy the built artifacts into place, defined by me, not to some ‘default’ location of your file system. You may ask, why i doing `make install` step, while i can just copy artifacts produced by `make` command. It is because i need to deal with dependencies (see below), and because it is easier to pick all artifacts from a single place, in contrast to searching among numerous and project-specific subdirectories to figure out what exactly you need. Also, if project developers may decide to change the directory structure of their sources, `make install` step makes your configuration agnostic to these changes.

Copying artefacts

The last stage is quite simple. We just copying the dynamic library to our VM bundle:


gen puts:
OUTPUT “${externalModulesDir}/${libraryFileName}”
COMMAND cp ${installPrefix}/lib/${libraryFileName} ${externalModulesDir}/${libraryFileName}
DEPENDS “${ft2libInstalled}”

Usually, you need to copy just a single file (so in fact this method can be even put into a superclass in future, after some refactoring).
What this step doing is just invoking:

cp build/thirdparty/out/lib/freetype.6.dylib  build/../results/CogVM.app/Contents/Plugins/freetype.6.dylib


So, we’re almost done.


In case of Cairo, it has dependencies from 3 other libraries (actually 5, but others are ‘standard’ on Mac – libz, libbz2)

– pixman

– libpng

– freetype

To define dependencies, just implement #dependencies method in library class:

^ #( ‘pkg-config’ ‘pixman’ ‘libpng’ ‘freetype2’)

All those names is another third-party libs, which should be defined in corresponding classes. CMake config deals with order of dependencies, (so the dependent target are built after one which it depending on). This is where `make install` is very useful: cairo configure script locates the required libraries in my directory (thirdparty/out/…), so i can be sure that it built using libraries which i built, not libraries which may or may not be installed in your system, like using `port` or `homebrew` etc)

The last thing about bundling (and it is Mac-specific) is to change the references in produced .dylib files to correct ones. I tried to use BundleUtils provided by cmake.. but it is prone to be buggy, so i had to write own. All it does is replacing an absolute paths in all .dylib-s (including external plugin libs) to path relative to .app bundle.

P.S. I spent a whole week doing this.. Ohh.. it is really pain in the ass dealing with all those details. But i tried to make an interface which can be reused and it should be simpler to include another library in future. Because without this, i could stop just in a first day.. i had a shell script which just builds cairo.. that all i was needed 🙂

P.P.S. Screw autoconf, screw cmake, screw C, screw GCC, ld, libtool, pkg-config, command-line. Pff.. i am really fed up with this!! I hate this stuff!!!

P.P.P.S Screw wordpress wysiwyg formatting