Generate a JSON Compilation Database from an Xcode project

generate-xcode-compilation-database.md

Introduction

A JSON compilation database is a very handy output format which is parsed and used by many development tools. Unfortunately for us Apple Developers, it is not straightforward to generate one from within Xcode, as it is (probably) not Apple's priority and therefore there is no toggle/switch/setting that can be easily enabled to get this information.

There is however a solution, thanks to Apple using Clang/LLVM as their main toolchain.

Implementation

The standard way to generate this with clang would be to use the -MJ flag and give it a file name that typically corresponds to the input file. Using this flag indirectly through Xcode is hard, given that we're not aware of all the other arguments when a compiler call is executed.

However, there is a second hidden/badly documented LLVM flag: -gen-cdb-fragment-path - it is implemented in terms of -MJ and has the same functionality, but it's argument in contrast is an output directory.

This allows us to collect all fragments from individual compiler executions in a central location, which greatly simplifies processing them.

xcconfig

OTHER_CFLAGS = $(inherited) -gen-cdb-fragment-path $(PROJECT_DIR)/CompilationDatabase

CMake

CMake has this functionality built in, represented by the variable CMAKE_EXPORT_COMPILE_COMMANDS, however this is only implemented for Makefile and Ninja generators and it is ignored for all others.

We can reuse this flag to set the required compiler flag for the generated Xcode project, which will in turn result in generated fragments. They will still need to be collected and processed separately (manually).

if(CMAKE_GENERATOR STREQUAL "Xcode")
	if(CMAKE_EXPORT_COMPILE_COMMANDS)
		set(CMAKE_XCODE_ATTRIBUTE_OTHER_CFLAGS  "$(inherited) -gen-cdb-fragment-path ${CMAKE_SOURCE_DIR}/CompilationDatabase")
	endif()
endif()

Processing

Once a build is executed with this flag, the output directory will contain a many JSON files, corresponding for each input source file that was compiled.

Each fragment on its own isn't valid JSON, but they can easily be combined into an array of objects which represents the actual compilation database.

Please check the script below for a reference implementation for combining the fragments into a compilation database.

References

1. CLion - Generating a Compilation Database
2. Clang - Generating a Compilation Database
3. How to generate a JSON Compilation Database?
4. JSON Compilation Database Format Specification
5. LLVM: Add a new option to emit a fragment of a compilation database for each compilation
6. LLVM: Add a new option to emit a fragment of a compilation database for each compilation 2

generate-xcode-compilation-database.sh

#!/usr/bin/env bash

# Global variables
readonly GXC_SCRIPT_PATH="$(test -L "${BASH_SOURCE[0]}" && readlink "${BASH_SOURCE[0]}" || echo "${BASH_SOURCE[0]}")"
readonly GXC_SCRIPT_DIR=$(cd "$(dirname "${GXC_SCRIPT_PATH}")"; pwd)

# brew install jq
: "${JQ:=$(command -v jq)}"
: "${OUTPUT_DIR:=$PWD}"

process_fragments() {
	local -a files=()
	for file in "$@"; do
		[[ -f "$file" ]] || continue

		# Fragments generated by clang have a comma before EOF
		# If a fragment is stil invalid after removing it, it should be skipped
		if sed -e '$s/,$//' "$file" | "$JQ" . > /dev/null; then
			echo "Processing: $file" >&2
			files+=("$file")
		else
			echo "Skipping: $file" >&2
		fi
	done

	if (( ${#files[@]} == 0 )); then
		echo "No input files found!" >&2
		return
	fi

	sed -e '1s/^/[\'$'\n''/' -e '$s/,$/\'$'\n'']/' "${files[@]}" > "${OUTPUT_DIR}/compile_commands.json"
}

generate_database() {
	echo "Running build command ..." >&2
	if grep -q 'OTHER_CFLAGS' <<< "$*"; then
		echo "OTHER_CLAGS detected in build command! This is unsupported as they will be overridden." >&2
		return 1
	fi

	# xcrun xcodebuild ...
	# https://reviews.llvm.org/D66555
	
	# Note: Some tools require extra cflags to properly parse the compilation database, e.g. Infer
	# OTHER_CFLAGS="\$(inherited) -DNS_FORMAT_ARGUMENT(A)= -D_Nullable_result=_Nullable -gen-cdb-fragment-path ${OUTPUT_DIR}/CompilationDatabase"
	"$@" COMPILER_INDEX_STORE_ENABLE=NO OTHER_CFLAGS="\$(inherited) -gen-cdb-fragment-path ${OUTPUT_DIR}/CompilationDatabase"
	process_fragments "${OUTPUT_DIR}/CompilationDatabase"/*.json
}

# main
if [[ "${BASH_SOURCE[0]}" != "$0" ]]; then
	echo "Script is being sourced: ${GXC_SCRIPT_PATH}"
else
	(( DEBUG == 0 )) || set -x
	set -euo pipefail

	if (( $# == 0 )); then
		cat <<-EOF
		Usage: $(basename "$0") <xcodebuild-command>

		Environment variables:
		JQ: Path to a jq binary
		OUTPUT_DIR: Writable directory for storing the result. Default: (\$PWD) - $PWD
		EOF

		exit 1
	fi

	generate_database "$@"
fi

# Example invocation
# generate-xcode-compilation-database.sh xcodebuild build -project TestProject.xcodeproj -target TestTarget -configuration Debug

Hey @nguyenvukhang,
Thanks for your feedback. I just tested it on macOS 13.3.1 with Xcode 14.3 (14E222b) and it was working as expected.
The result that you're describing can happen if e.g. you have a pure, Swift only target, meaning no c/cpp/objc files as part of that target.
As the functionality in the script is passed along using OTHER_CFLAGS, no compilation database will be generated if clang wasn't ever executed.

Thank you very much. I used xcpretty to generate the compile_commands.json file, but the file had bugs. However, using your method, the file is correct.

For me, it doesn't seem to generate anything for CPP files, only C files.

I'm using CMake 3.26.1, macOS 13.4.1, Xcode 14.3.1.

My problem seems to be that CMAKE_XCODE_ATTRIBUTE_OTHER_CFLAGS does not propagate through to the target's C++ flags in the Xcode project, only to the C flags.

This seems to be because the generated Xcode project is missing "${inherited}" inside the C++ flags section of the target settings.

If I add it manually, then things work, but that's no use, as I need CMake to generate it properly.

@oviano try setting CMAKE_XCODE_ATTRIBUTE_OTHER_CPLUSPLUSFLAGS with the same values as well.

@oviano try setting CMAKE_XCODE_ATTRIBUTE_OTHER_CPLUSPLUSFLAGS with the same values as well.

I tried this and surprisingly it did not work either because it still does not set $(inherited) in the Other C++ Flags section of the Xcode properties, so while the above sets it for the project, the target does not inherit it. I presume it is a CMake bug.

However, I worked around it by manually adding $(inherited) to the CMAKE_CXX_FLAGS.

I tried in a XCode cocoapods workspace, it seems only generate json files with main project.

Another way to get the compile_commands.json file is to use https://github.com/xcpretty/xcpretty — like this:

xcodebuild -project path/to/.xcodeproj | \
    xcpretty -r json-compilation-database --output path/for/compile_commands.json

Or else, if you’re running a build from the Xcode GUI, you can press the Export… button in the Build tab to export the log, and then do:

cat Build\ Foo_202n-nn-nnTnn-nn-nn.txt | \
    xcpretty -r json-compilation-database --output path/for/compile_commands.json

I know somebody here mentioned that the “I used xcpretty to generate the compile_commands.json file, but the file had bugs” but I personally have never run into any bugs with it.