Basics of generating a compile_commands.json file with Bazel

BUILD

py_binary(
  name = 'generate_compile_command',
  srcs = [
    'generate_compile_command.py',
  ],
  deps = [
    '//third_party/bazel:extra_actions_proto_py',
  ],
)

action_listener(
  name = 'generate_compile_commands_listener',
  visibility = ['//visibility:public'],
  mnemonics = [
    'CppCompile',
  ],
  extra_actions = [':generate_compile_commands_action'],
)

extra_action(
  name = 'generate_compile_commands_action',
  tools = [
    ':generate_compile_command',
  ],
  out_templates = [
    '$(ACTION_ID)_compile_command',
  ],
  cmd = '$(location :generate_compile_command) $(EXTRA_ACTION_FILE)' +
        ' $(output $(ACTION_ID)_compile_command)',
)

generate_compile_command.py

# This is the implementation of a Bazel extra_action which genenerates
# _compile_command files for generate_compile_commands.py to consume.

import sys

import third_party.bazel.protos.extra_actions_base_pb2 as extra_actions_base_pb2

def _get_cpp_command(cpp_compile_info):
  compiler = cpp_compile_info.tool
  options = ' '.join(cpp_compile_info.compiler_option)
  source = cpp_compile_info.source_file
  output = cpp_compile_info.output_file
  return '%s %s -c %s -o %s' % (compiler, options, source, output), source

def main(argv):
  action = extra_actions_base_pb2.ExtraActionInfo()
  with open(argv[1], 'rb') as f:
    action.MergeFromString(f.read())
  command, source_file = _get_cpp_command(
      action.Extensions[extra_actions_base_pb2.CppCompileInfo.cpp_compile_info])
  with open(argv[2], 'w') as f:
    f.write(command)
    f.write('\0')
    f.write(source_file)

if __name__ == '__main__':
  sys.exit(main(sys.argv))

generate_compile_commands_json.py

#!/usr/bin/python3

# This reads the _compile_command files :generate_compile_commands_action
# generates a outputs a compile_commands.json file at the top of the source
# tree for things like clang-tidy to read.

# Overall usage directions: run bazel with
# --experimental_action_listener=//tools/actions:generate_compile_commands_listener
# for all the files you want to use clang-tidy with and then run this script.
# Afer that, `clang-tidy build_tests/gflags.cc` should work.

import sys
import pathlib
import os.path
import subprocess

'''
Args:
  path: The pathlib.Path to _compile_command file.
  command_directory: The directory commands are run from.

Returns a string to stick in compile_commands.json.
'''
def _get_command(path, command_directory):
  with path.open('r') as f:
    contents = f.read().split('\0')
    if len(contents) != 2:
      # Old/incomplete file or something; silently ignore it.
      return None
    return '''{
        "directory": "%s",
        "command": "%s",
        "file": "%s",
      },''' % (command_directory, contents[0].replace('"', '\\"'), contents[1])

'''
Args:
  path: A directory pathlib.Path to look for _compile_command files under.
  command_directory: The directory commands are run from.

Yields strings to stick in compile_commands.json.
'''
def _get_compile_commands(path, command_directory):
  for f in path.iterdir():
    if f.is_dir():
      yield from _get_compile_commands(f, command_directory)
    elif f.name.endswith('_compile_command'):
      command = _get_command(f, command_directory)
      if command:
        yield command

def main(argv):
  source_path = os.path.join(os.path.dirname(__file__), '../..')
  action_outs = os.path.join(source_path,
                             'bazel-bin/../extra_actions',
                             'tools/actions/generate_compile_commands_action')
  command_directory = subprocess.check_output(
      ('bazel', 'info', 'execution_root'),
      cwd=source_path).decode('utf-8').rstrip()
  commands = _get_compile_commands(pathlib.Path(action_outs), command_directory)
  with open(os.path.join(source_path, 'compile_commands.json'), 'w') as f:
    f.write('[')
    for command in commands:
      f.write(command)
    f.write(']')

if __name__ == '__main__':
  sys.exit(main(sys.argv))

Thanks a lot for the clarification 😄

It's good to have this here for posterity since this gist is referenced in a lot of places and one of the first results in any search related to Bazel and compile commands.

Happy holidays!

For anyone landing here, we'd highly recommend using https://github.com/hedronvision/bazel-compile-commands-extractor, just released.

[It's fast and less brittle thanks to an aquery-based approach that directly asks Bazel what build commands it would run. The grail bio/bazel-compilation-database creator had some nice things to say about it and passing the torch. (See here for more)]

This works great with bazel 0.5.3 on Ubuntu 18.04.

Yesterday I tried Ubuntu 16.04, to install the Python 2.7's package protobuf, you need to do it as follows:

sudo apt install python-pip
sudo pip install protobuf=3.17.3

Otherwise the default installed protobuf with version 3.19+ is not working.

@bsilver8192 could you add a license to this gist? (preferably MIT)