This tutorial describes how to build the Adobe DNG SDK on Linux.

DNG SDK build on linux

DNG 1.4 Parser
This tutorial describes how to build the Adobe DNG SDK on Linux.
It generates the dng_validate C++ program that can parse any DNG images, a bit like a "Hello world" for DNG image processing. 


Adobe DNG SDK 1.4
XMP SDK
DNG SDK
dcraw issue
Compatibility issue
Security Patch


Adobe DNG SDK 1.4

The goal of this blog entry is to execute the dng_validate command built from the source.
We show how to build the XMP libraries then suggest a Makefile configuration to build a DNG SDK sample app. 
All the patches were generated by the diff command. 

XMP SDK


In the current directory, download the XMP SDK: 
wget http://download.macromedia.com/pub/developer/xmp/sdk/XMP-Toolkit-SDK-CC-201306.zip
unzip XMP-Toolkit-SDK-CC-201306.zip
mv XMP-Toolkit-SDK-CC201306 xmp_sdk

Make sure you have at least the following packages installed: 

sudo apt-get install cmake libjpeg8-dev uuid-dev

The 2 next snippets download zlib and expat sources. Instructions are explained in the ReadMe.txt files in the third-party directories. 

zlib

cd xmp_sdk/third-party/zlib
wget http://zlib.net/zlib-1.2.8.tar.gz
tar xzf zlib-1.2.8.tar.gz
cp zlib-1.2.8/*.h zlib-1.2.8/*.c .

expat

cd xmp_sdk/third-party/expat
Download here the source at http://sourceforge.net/projects/expat/files/expat/2.1.0/expat-2.1.0.tar.gz/download.
tar xzf expat-2.1.0.tar.gz
cp -R expat-2.1.0/lib .

For cross-platform compatibility, to be able to support both Mac & Windows, cmake is the tool selected by the SDK authors to build the source. We go through all the build errors. 

cd xmp_sdk/build

Build error 1

make
...
CMake Error at shared/SharedConfig_Common.cmake:38 (if):
  if given arguments:

    "LESS" "413"

  Unknown arguments specified
...

I use gcc 4.8.2 for the build. There seems to be a XMP_VERSIONING_GCC_VERSION variable not properly set. We just delete the version check by removing line 37 till 42 in xmp_sdk/build/shared/SharedConfig_Common.cmake: 

xmp_sdk/build/shared/SharedConfig_Common.cmake
36a37,41
>               # workaround for visibility problem and gcc 4.1.x
>               if(${${COMPONENT}_VERSIONING_GCC_VERSION} LESS 413)
>                       # only remove inline hidden...
>                       string(REGEX REPLACE "-fvisibility-inlines-hidden" "" CMAKE_CXX_FLAGS ${CMAKE_CXX_FLAGS})
>               endif()

Build error 2

make
...
Linking CXX shared library /home/alexis/linux/dng2/xmp_sdk/public/libraries/i80386linux_x64/release/libXMPCore.so
g++: error: /usr/lib64/gcc/x86_64-redhat-linux/4.4.4//libssp.a: No such file or directory
...

The README.txt mentions a XMP_ENABLE_SECURE_SETTINGS: 

xmp_sdk/build/README.txt
... He may also want to change the parameter XMP_ENABLE_SECURE_SETTINGS as per the configured gcc.
 a) If the gcc is configured with --enable-libssp (can be checked by executing gcc -v), he has to set the variable XMP_GCC_LIBPATH inside of file /build/XMP_Linux.cmake to the path containing the static lib( libssp.a).In this case he can set the variable the XMP_ENABLE_SECURE_SETTINGS on.
b) If the gcc is configured with --disable-libssp, he has to set the variable XMP_ENABLE_SECURE_SETTINGS off.

I don't have any --enable-libssp in my gcc version: 

$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.8/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 4.8.2-21' --with-bugurl=file:///usr/share/doc/gcc-4.8/README.Bugs --enable-languages=c,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.8 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.8 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --disable-libmudflap --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.8-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.8-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.8-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --with-arch-32=i586 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.8.2 (Debian 4.8.2-21)

so I disable it in xmp_sdk/build/shared/ToolchainGCC.cmake line 37: 

xmp_sdk/build/shared/ToolchainGCC.cmake
37c37
< set(XMP_ENABLE_SECURE_SETTINGS "OFF")
---
> set(XMP_ENABLE_SECURE_SETTINGS "ON")

Build error 3

/home/alexis/linux/dng2/xmp_sdk/XMPFiles/build/../../XMPFiles/source/NativeMetadataSupport/ValueObject.h:111:60: error: 'memcmp' was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation [-fpermissive]
    doSet = ( memcmp( mArray, buffer, numElements*sizeof(T) ) != 0 );
                                                            ^
In file included from /home/alexis/linux/dng2/xmp_sdk/XMPFiles/build/../../XMPFiles/source/FormatSupport/TIFF_Support.hpp:17:0,
                 from /home/alexis/linux/dng2/xmp_sdk/XMPFiles/build/../../XMPFiles/source/FormatSupport/ReconcileLegacy.hpp:15,
                 from /home/alexis/linux/dng2/xmp_sdk/XMPFiles/build/../../XMPFiles/source/FormatSupport/Reconcile_Impl.hpp:15,
                 from /home/alexis/linux/dng2/xmp_sdk/XMPFiles/source/FormatSupport/WAVE/WAVEReconcile.cpp:24:
/usr/include/string.h:65:12: note: 'int memcmp(const void*, const void*, size_t)' declared here, later in the translation unit
 extern int memcmp (const void *__s1, const void *__s2, size_t __n)

This is an error relative to my new version of gcc, described in the Name lookup changes section.
We simply add the #include "string.h" in xmp_sdk/XMPFiles/source/NativeMetadataSupport/ValueObject.h, line 16:

xmp_sdk/XMPFiles/source/NativeMetadataSupport/ValueObject.h
16,17d15
< #include "string.h"

Validate the build

make
[100%] Built target XMPFilesStatic

Check that the shared and static libraries got properly generated: 

$ ls xmp_sdk/public/libraries/i80386linux_x64/release
libXMPCore.so*  libXMPFiles.so*  staticXMPCore.ar  staticXMPFiles.ar

DNG SDK


In the current directory, where the XMP SDK got downloaded, download the DNG SDK:

wget http://download.adobe.com/pub/adobe/dng/dng_sdk_1_4.zip
unzip dng_sdk_1_4.zip

There are multiple ways to come up with the binary. In this case, 
We seperate the compilation and linking steps in 2 separate targets.
We link with the shared libraries of the XMP Toolkit.
We did not install system wide in /usr/local/lib and /usr/local/include the required XMP files that were generated during the XMP build.

cd dng_sdk/source

Makefile

Create the "Makefile" build configuration file: 

# Binary name
APP=dng_validate

# A DNG image
DNG_IMAGE=~/job/image_samples/9436b5a2336f0a575a5b0ef3adf0b25171125081.dng

# The XMP SDK build directory if we don't want to install it system-wide.
XMP_PUB_DIR=/home/alexis/linux/dng/xmp_sdk/public

INCL=-I $(XMP_PUB_DIR)/include
XMP_RELEASE=$(XMP_PUB_DIR)/libraries/i80386linux_x64/release
LIB=-ljpeg -lz -lpthread -ldl -L $(XMP_RELEASE) -lXMPCore -lXMPFiles

# Execute the binary
all: $(APP)
 LD_LIBRARY_PATH=$(XMP_RELEASE) ./$(APP) $(DNG_IMAGE)

# Linking
$(APP): *.o
 g++ $^ $(LIB) -o $(APP)

# Compilation
*.o: *.cpp
 g++ -c -Wall $(INCL) $^

clean:
 rm $(APP) *.o

Build error 1

To avoid errors like 

dng_flags.h:36:28: fatal error: RawEnvironment.h: No such file or directory
 #include "RawEnvironment.h"

Create the RawEnvironment.h file containing build settings: 

#define qLinux 1
#define qDNGThreadSafe 1
#define UNIX_ENV 1

Build error 2

To avoid errors like 

dng_flags.h:40:2: error: #error Unable to figure out platform

A qLinux setting got created in RawEnvironment.h. The "!defined(qMacOS) || !defined(qWinOS)" kind of test line 39 in dng_flags.h does not seem effective on Linux platform even when qMacOS or qWinOS are defined. We replace it by something more standard, for example "#ifndef qLinux": 

39 #ifndef qLinux
40 #error Unable to figure out platform
41 #endif


chmod +w dng_flags.h
39c39
< #ifndef qLinux
---
> #if !defined(qMacOS) || !defined(qWinOS)

Build error 3

dng_string.cpp:1792:24: error: 'isdigit' was not declared in this scope
    if (isdigit ((int) c) || c == '.' || c == '-' || c == '+' || c == 'e' || c == 'E')

We simply need to include "ctype.h" in dng_string.cpp for linux platform at line 33: 

32 #if qiPhone || qAndroid || qLinux
33 #include <ctype.h> // for isdigit
34 #endif

At line 32 we add qLinux is the list of platforms on when to include ctype.h. 

chmod +x dng_string.cpp
32c32
< #if qiPhone || qAndroid || qLinux
---
> #if qiPhone || qAndroid

Build error 4

/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'

The main function declared in dng_validate.cpp was not found. Enable the command build setting in line 200 in dng_flags.h: 

200c200
< #define qDNGValidateTarget 1
---
> #define qDNGValidateTarget 0

Build error 5

make clean
make

We now have a linking error 

dng_xmp_sdk.cpp:(.text._ZN9TXMPFilesISsE8OpenFileEP6XMP_IOjj[_ZN9TXMPFilesISsE8OpenFileEP6XMP_IOjj]+0x40): undefined reference to `WXMPFiles_OpenFile_2'
collect2: error: ld returned 1 exit status

We want to link with the shared libraries. Delete the static build setting in dng_xmp_sdk.cpp line 48:

chmod +w dng_xmp_sdk.cpp
48a49,50
> #define XMP_StaticBuild 1
> 

Validate the build

The execution of the command looks like: 

make
...
LD_LIBRARY_PATH=/home/alexis/linux/dng/xmp_sdk/public/libraries/i80386linux_x64/release ./dng_validate ~/job/image_samples/9436b5a2336f0a575a5b0ef3adf0b25171125081.dng
Validating "/home/alexis/job/image_samples/9436b5a2336f0a575a5b0ef3adf0b25171125081.dng"...
Raw image read time: 0.074 sec
Linearization time: 0.051 sec
Interpolate time: 0.000 sec
Validation complete


dcraw issue

Compatibility Issue


The DNG specification PDF, line 101, mentions the lossy jpeg breaking change:
dng_sdk/documents/DNG\ Spec\ 1.4.0.0.pdf

Compatibility Issue 11: Lossy JPEG Compression

Support for compression code 34892 (Lossy JPEG) was added in DNG Version 1.4.0.0. DNG writers should set the DNGBackwardVersion tag to a minimum of 1.4.0.0 if the main image IFD uses compression code 34892.

Security Patch

Why are we looking at a custom DNG parser? 

The dcraw vanilla command gets stuck in an (almost) infinite loop, hence blocking your CPU unnecessarily, potentially for hours.

More precisely, for lossy DNG version 1.4 images, it keeps calling fseek on an invalid offset at line 2119: 

2102 void CLASS lossy_dng_load_raw()
2103 {
2104   struct jpeg_decompress_struct cinfo;
2105   struct jpeg_error_mgr jerr;
2106   JSAMPARRAY buf;
2107   JSAMPLE (*pixel)[3];
2108   unsigned sorder=order, ntags, opcode, deg, i, j, c;
2109   unsigned save=data_offset-4, trow=0, tcol=0, row, col;
2110   ushort curve[3][256];
2111   double coeff[9], tot;
2112 
2113   fseek (ifp, meta_offset, SEEK_SET);
2114   order = 0x4d4d;
2115   ntags = get4();
2116   while (ntags--) {
2117     opcode = get4(); get4(); get4();
2118     if (opcode != 8)
2119     { fseek (ifp, get4(), SEEK_CUR); continue; }

For now we could apply this "security" patch aka hack, although that does not fix much of the problem. 

2108c2108
<   unsigned sorder=order, ntags, opcode, should_seek, deg, i, j, c;
---
>   unsigned sorder=order, ntags, opcode, deg, i, j, c;
2119,2127c2119
<     {
<       should_seek = get4();
<       if (should_seek == -1) {
<         fprintf(stderr, "Invalid fseek\n");
<         exit(EXIT_FAILURE);
<       }
<       fseek (ifp, should_seek, SEEK_CUR);
<       continue;
<     }
---
>     { fseek (ifp, get4(), SEEK_CUR); continue; }

I have disabled enable_secure_settings in GCCToolChain.make but still libssp.a no such file or directory while building xmp-sdk