Compiling libmagic and libmagic python bindings natively on OS X for use in any directory

These instructions compile a native OS X version of libmagic and the libmagic python binding which use the system magic database, and adjust their internal paths to allow you to use them from any arbitrary working directory, which is normally restricted in versions of OS X with System Integrity Protection.

Assumes OS X with Xcode or Xcode command-line tools, not macports or homebrew

To install Xcode command-line tools in recent versions of OS X, open Terminal, type xcode-select --install and click "Install"

Older versions of OS X may need to manually download disk images of older versions of Xcode and/or the Xcode command-line tools, as various certificates may have expired (e.g. 10.7.4 requires a manual install from the Xcode 4.6.2 command-line tools image)

Download Apple's latest modified file 5.04 (used in 10.7+) from http://opensource.apple.com/tarballs/file/file-47.tar.gz

Create a directory to put the compiled stuff in, you can delete it later

$ mkdir ~/securitoys

Force libmagic to use the OS X system magic database:

$ cd ~/Downloads/
$ cd file-47/file/
$ mv configure configure.bak
$ sed "s/pkgdatadir='\$(datadir)\/misc'/pkgdatadir='\/usr\/share\/file'/" configure.bak > configure
$ chmod 755 configure

Compile file and libmagic as fat/universal binaries:

$ ./configure --prefix=$HOME/securitoys/ LDFLAGS="-arch i386 -arch x86_64" CFLAGS="-arch i386 -arch x86_64" ARCHFLAGS="-arch i386 -arch x86_64"
$ make
$ make install
install: /usr/share/file/magic.mgc: Operation not permitted
$ file ~/securitoys/lib/libmagic.1.0.0.dylib 
/Users/username/securitoys/lib/libmagic.1.0.0.dylib: Mach-O universal binary with 2 architectures
/Users/username/securitoys/lib/libmagic.1.0.0.dylib (for architecture i386):	Mach-O dynamically linked shared library i386
/Users/username/securitoys/lib/libmagic.1.0.0.dylib (for architecture x86_64):	Mach-O 64-bit dynamically linked shared library x86_64

It's okay that installing magic.mgc fails, because there's already the system one there.

Using an older version of OS X/Xcode and getting these warnings and errors after running make?

$ make
WARNING: `aclocal-1.10' is missing on your system.  You should only need it if
         you modified `acinclude.m4' or `configure.ac'.  You might want
         to install the `Automake' and `Perl' packages.  Grab them from
         any GNU archive site.
WARNING: `automake-1.10' is missing on your system.  You should only need it if
         you modified `Makefile.am', `acinclude.m4' or `configure.ac'.
         You might want to install the `Automake' and `Perl' packages.
         Grab them from any GNU archive site.
WARNING: `autoconf' is missing on your system.  You should only need it if
         you modified `configure.ac'.  You might want to install the
         `Autoconf' and `GNU m4' packages.  Grab them from any GNU
         archive site.
llvm-gcc-4.2: -E, -S, -save-temps and -M options are not allowed with multiple -arch flags

Just add --disable-dependency-tracking to your ./configure command like so:

$ make clean
$ ./configure --prefix=$HOME/securitoys/ LDFLAGS="-arch i386 -arch x86_64" CFLAGS="-arch i386 -arch x86_64" ARCHFLAGS="-arch i386 -arch x86_64" --disable-dependency-tracking
$ make
$ make install

Edit the Python setup file to point to your newly compiled libmagic folders:

$ cd python/

Open setup.py in a plain text editor and change:

    library_dirs = ['./','../','../src','/usr/lib/'],
    include_dirs = ['./','../','../src','/usr/include/'],

to:

    library_dirs = ['./','../','../src','/usr/lib/','/Users/username/securitoys/lib'],
    include_dirs = ['./','../','../src','/usr/include/','/Users/username/securitoys/include'],

Compile the Python binding:

$ python setup.py build
$ file build/lib.macosx-10.xx-yy/magic.so 
build/lib.macosx-10.xx-yy/magic.so: Mach-O universal binary with 2 architectures
build/lib.macosx-10.xx-yy/magic.so (for architecture i386):	Mach-O bundle i386
build/lib.macosx-10.xx-yy/magic.so (for architecture x86_64):	Mach-O 64-bit bundle x86_64

Rename the absolute paths in the Python binding:

$ cd build/lib.macosx-10.xx-yy/
$ otool -L magic.so 
magic.so:
	/Users/username/securitoys/lib/libmagic.1.dylib (compatibility version 2.0.0, current version 2.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)
$ install_name_tool -change /Users/username/securitoys/lib/libmagic.1.dylib @loader_path/libmagic.1.dylib magic.so 
$ otool -L magic.so 
magic.so:
	@loader_path/libmagic.1.dylib (compatibility version 2.0.0, current version 2.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)

Rename the absolute paths in libmagic:

$ cd ~/securitoys/lib/
$ otool -L libmagic.1.0.0.dylib 
libmagic.1.0.0.dylib:
	/Users/milianov/securitoys/lib/libmagic.1.dylib (compatibility version 2.0.0, current version 2.0.0)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)
$ install_name_tool -id libmagic.1.dylib libmagic.1.0.0.dylib 
$ mv libmagic.1.0.0.dylib libmagic.1.dylib 
$ otool -L libmagic.1.dylib 
libmagic.1.dylib:
	libmagic.1.dylib (compatibility version 2.0.0, current version 2.0.0)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)

Now, you can copy the binary Python module magic.so and its dependency, libmagic.1.dylib wherever you need them, even on OS X systems with System Integrity Protection (10.11 El Capitan and later). It will use the OS X system magic database, and will not require a separate magic.mgc database.

vitorio/osx-libmagic-movable.md