Just another WordPress.com weblog

Hi, there.. (or here? 😉 )

You can stop reading if you using Linux or any other flavor of unix with decent package management.

However, if you using MacOS , or Windows.. you may continue reading. 🙂

Problem: suppose you want to add a plugin to VM, which using some well-known open-source library.
On linux, this is piece of cake.. you just install that library using package management tool , like:

apt-get install somelibrary

and tell plugin to look the installed library at well known place in system. Done.

On Mac and Windows, things are different: you are on you own.. There is no well-known place in system (unless it is of course Apple/Microsoft product), defined where all 3rd-party/shared libraries installed. And often, there’s even no binary distribution for such library, and even if it exists,
if you install it, you never know where it is located, because again there is no standard place.
So, the only solution is to bundle these libs with own app. In most cases you have to download sources from project site and build it by own.

So, the first thing which rang bells in my head is the way how we deal with it today.
For instance, Freetype: If you using Pharo , you know that it provides a freetype support by default,
as well as VMs which we build on Jenkins CI server including corresponding freetype plugin.

From user’s perspective, everything is nice: he can use nice fonts in images.. but if we open a cover of our shiny car, things are not so nice. And i will explain why:

On windows and macs, we were forced to include a whole freetype library as a static component of plugin.. i.e. it just statically linked to FT2Plugin.
What is wrong with such approach? Suppose i want to add another plugin, which using very same library. Now, if i do the same, it will result in having two separate (autonomous) instances of same library used by two different plugins..

And this is exactly what happening with Cairo (which i use in Athens via FFI) because cairo also using freetype (what a coincidence!).
So, in Pharo, we already have freetype. Now Cairo also using freetype. But since cairo is loaded using FFI, there’s no way how it can communicate with statically linked freetype library resided inside FT2Plugin. As result, when Athens using Cairo you have two instances of same library loaded..
But even worse: i cannot reuse existing code from Pharo freetype support code and cope it with cairo. Because if i load a font face using freetype plugin, i cannot pass that face to cairo, because the face object instantiated by one library, and cairo using another one, which know absolutely nothing about each other.

So, obviously an appropriate solution would be to force both FT2Plugin and Cairo library to use same shared, dynamically loadable, library and abandon idea of statically linking a library to single plugin.
Also, since we cannot expect that it will be installed on system somewhere, we should bundle it with our application (which in our case a VM).

Btw, a freetype is not the sole example of statically embedding 3rd-party libraries into out beloved VM. We have couple more under cover:
– FloatMathPlugin includes whole fdlibm
– JPEGReadWriter2Plugin includes a copy of JPEG library
– RePlugin includes PCRE library (perl-compatible regular expressions)

and i think there’s more. And since those libraries are private to those plugins, if some another plugin want to use same functionality, there is no way how they can share same library -> so, you got code duplication and memory waste.

So, i decided to extend the CMakeVMMaker to have a notion of 3rd-party library, and our automated build system can:
– download library from official site
– configure & build it
– bundle it alongside with VM

Here the brief description of a new interface.

First, i introduced a class, named CMThirdpartyLibrary to describe the configuration of thirdparty library as well as a steps to build it and bundle with VM.

This is an abstract class, which shapes a most common attributes of 3rd-party package as well as a more or less standard way to build & include it to VM. A subclass of it can describe the concrete library, as well as customize the way it is built.

So, let’s take a concrete example – CMFreetype2 class, which defines a freetype2 library.

First thing: a library name.

CMFreetype2 class>>canonicalName
^ 'freetype2'

This is a name of a library, used to identify it. For obvious reason, there should be no two classes with same canonical name.
To include a library into cmake loop, use #addThirdpartyLibrary: <libname> message, sent to configuration.

For example:

NBCogCocoaIOSConfig new
generateForRelease;
addThirdpartyLibrary: 'freetype2';
generate.

This will add necessary instructions to generated CMakeLists.txt file to include library & bundle it with VM.

This is how, FT2Plugin using it now:

MacOSConfig>>configureFT2Plugin: maker
“extra rules for Freetype plugin”
| lib |

maker isExternal ifFalse: [
self error: ‘building internal FT2Plugin is not supported yet’
].

“add freetype library into loop”
lib := self addThirdpartyLibrary: ‘freetype2’.

“link plugin with freetype lib”
maker addExternalLibrary: lib targetForLinking.
maker includeDirectories: lib includeDir.

A canonical name also serves as a build target. So, you can do in command-line:

build#  cmake .
build#  make freetype2

this will build a freetype2 library.

The class structure breaks a build process on common stages, which should be more or less similar to all libs:

CMThirdpartyLibrary>>generate
gen message: ‘Configuring thirdparty package: ‘, self canonicalName.

self
setVariables;
download;
unpack;
build;
copyArtefacts;
defineAsTarget.

Setting variables

The #setVariables method serves mainly to set-up a most common variables in cmake, like paths, names , options and so on, so other parts of generated cmake script can reuse them without duplicating definitions over and over again.

Here variables set by CMThirdpartyLibrary class:

CMThirdpartyLibrary>>setVariables

gen
set: #libName toString: self canonicalName;
set: #workDir toString: ‘${thirdpartyDir}/${libName}’;
set: #unpackedDirName toString: self unpackedDirName;
set: #libSourcesDir toString: ‘${workDir}/${unpackedDirName}’;
set: #url toString: self downloadURL;
set: #md5sum toString: self archiveMD5Sum;
set: #installPrefix toString: ‘${thirdpartyDir}/out’.

we add few more in freetype:

CMFreetype2>>setVariables
super setVariables.

“add include path”
gen
set: #freetype2_includeDir toString: ‘${installPrefix}/include’;
set: #libraryFileName to: self libraryFileName;
set: #freetype2_location toString: ‘${externalModulesDir}/${libraryFileName}’;
set: #ft2config toString: ‘${libSourcesDir}/builds/unix/config.status’;
set: #ft2libInstalled toString: ‘${installPrefix}/lib/${libraryFileName}’

Downloading and unpacking sources.

downloadURL

^ ‘http://ftp.igh.cnrs.fr/pub/nongnu/freetype/freetype-2.4.9.tar.gz&#8217;

archiveMD5Sum
^ ‘c15f6dc8ed190d67b89ae09aaf7896b4’

unpackedDirName
^ ‘freetype-2.4.9’

(the #unpack method is provided by CMThirdpartyLibrary class, which using tar to unpack the downloaded file. Of course if library sources using different archive format (like zip), you will need to override that method to emit an appropriate shell commands to unpack it.

By default, all thirdparty libraries using a ‘thirdparty’ subdirectory in ‘build’ directory:

[~/projects/cog/sig-cog/build/thirdParty]: ls
cairo      freetype2  libpng     out        pixman     pkg-config
[~/projects/cog/sig-cog/build/thirdParty]:

 

Here, each subdirectory , except ‘out’ created automatically by concrete library configuration which contains downloaded source code,
as well as CMakeLists.txt generated by CMakeVMMaker.

Configuring and building library

The #build method is nothing more than invoking configure, make , make install in freetype source subdirectory..
Which is 3 “lines of code” in shell command line, but a bit of boilerplate in cmake 😉

CMFreetype2>>build
gen
puts:

add_custom_command(OUTPUT “${ft2config}”
COMMAND ./configure –prefix=”${installPrefix}” CFLAGS=”-arch i386” LDFLAGS=”-arch i386”
WORKING_DIRECTORY “${libSourcesDir}”
DEPENDS “${unpackTarget}”
)
add_custom_command(OUTPUT “${ft2libInstalled}”
COMMAND make
COMMAND make install
WORKING_DIRECTORY “${libSourcesDir}”
DEPENDS “${ft2config}”
COMMENT “Building ${libName}”
)

Since most of the libraries usually having own unique idiosyncrasic way how they are built, this method is one which you will need to change when adopting new library.
Here, as you can see i use custom prefix ${installPrefix}, passed to configure command. So `make install` will copy the built artifacts into place, defined by me, not to some ‘default’ location of your file system. You may ask, why i doing `make install` step, while i can just copy artifacts produced by `make` command. It is because i need to deal with dependencies (see below), and because it is easier to pick all artifacts from a single place, in contrast to searching among numerous and project-specific subdirectories to figure out what exactly you need. Also, if project developers may decide to change the directory structure of their sources, `make install` step makes your configuration agnostic to these changes.

Copying artefacts

The last stage is quite simple. We just copying the dynamic library to our VM bundle:

CMFreetype2>>copyArtefacts

gen puts:
‘add_custom_command(
OUTPUT “${externalModulesDir}/${libraryFileName}”
COMMAND cp ${installPrefix}/lib/${libraryFileName} ${externalModulesDir}/${libraryFileName}
DEPENDS “${ft2libInstalled}”
)’

Usually, you need to copy just a single file (so in fact this method can be even put into a superclass in future, after some refactoring).
What this step doing is just invoking:

cp build/thirdparty/out/lib/freetype.6.dylib  build/../results/CogVM.app/Contents/Plugins/freetype.6.dylib

 

So, we’re almost done.

Dependencies

In case of Cairo, it has dependencies from 3 other libraries (actually 5, but others are ‘standard’ on Mac – libz, libbz2)

– pixman

– libpng

– freetype

To define dependencies, just implement #dependencies method in library class:

dependencies
^ #( ‘pkg-config’ ‘pixman’ ‘libpng’ ‘freetype2’)

All those names is another third-party libs, which should be defined in corresponding classes. CMake config deals with order of dependencies, (so the dependent target are built after one which it depending on). This is where `make install` is very useful: cairo configure script locates the required libraries in my directory (thirdparty/out/…), so i can be sure that it built using libraries which i built, not libraries which may or may not be installed in your system, like using `port` or `homebrew` etc)

The last thing about bundling (and it is Mac-specific) is to change the references in produced .dylib files to correct ones. I tried to use BundleUtils provided by cmake.. but it is prone to be buggy, so i had to write own. All it does is replacing an absolute paths in all .dylib-s (including external plugin libs) to path relative to .app bundle.

P.S. I spent a whole week doing this.. Ohh.. it is really pain in the ass dealing with all those details. But i tried to make an interface which can be reused and it should be simpler to include another library in future. Because without this, i could stop just in a first day.. i had a shell script which just builds cairo.. that all i was needed 🙂

P.P.S. Screw autoconf, screw cmake, screw C, screw GCC, ld, libtool, pkg-config, command-line. Pff.. i am really fed up with this!! I hate this stuff!!!

P.P.P.S Screw wordpress wysiwyg formatting

Advertisements

Comments on: "Bundling VM with thirdparty libraries" (5)

  1. Hey, thanks for taking the trouble doing all of that. Definitely useful for a ton of stuff. I do agree with your P.P.S. and P.P.P.S. – For WordPress, check out Mars Edit on OSX (commercial) or MS Live Writer on Windows. It allows you to write off-line and edit properly and then upload all of the stuff direct to WordPress. Provided XMLRPC is enabled.

    I may have overlooked it, but what if Plugin A requires version A and Plugin B requires version B? This is all the same as DLL Hell on Windows.

    Maybe you want to check this out: http://msdn.microsoft.com/en-us/library/windows/desktop/aa375142(v=vs.85).aspx (about DLL redirection).

    Keep up the good work!

    • Igor Stasenko said:

      If you need two versions of same lib, you can just define one with ‘versionA’ name
      another with ‘versionB’ name and build them separately.. it is only a question of using different name (and file names , of course).

      Thanks for the link about .local stuff. It looks like a cheap default extension of OS’es library search algorithm.. But i doubt that we will need it, because i am sure, you cannot find things freetype.dll in search path on windows 🙂 And never find it in system directory (such as windows/system or windows/system32).
      But i’m not there yet.. making this working on windows will take another round..
      But i expect it to be easier. First because i already having most of things shaped, and second there for sure no interference with anything installed on windows.. it simply not exists. While on mac, since it partially compatible with linux world, there are some interference between xcode/gcc/port & libs & search paths etc.

  2. Thank you, Sig!!!! Isn’t it ironic that we chose Smalltalk because it is so much nicer/better/more fun than C and shell scripts and here we are, right back to where we started?! Keep it up, brother… the computer revolution hasn’t happened yet, but we’re getting closer 🙂

    • Igor Stasenko said:

      yes, ironic, but straight to the point: before you can enjoy a beautiful and comfortable smalltalk environment, you have to work hard fighting with dinosaurs and monsters in a dark dungeon.. it’s like saving a princess from evil troll 🙂

  3. I’ve been building the Windows PharoVM and indeed, that’s a clean way to manage/build the third party things.

    I had a look at the code you wrote. Man, that’s very very nice. I’d love to be able to have an extracted tooling from all of the VMMaker just to make standard C programs written in Pharo and using your toolchain to build.

    Question: how is working? Where is the implementation of this pragma? The Finder will give me all calls using it. But where is this one done?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: