Use Emacs with UE4 and Clangd

EamcsUe4Clang.org

#+TITLE: Use Emacs with UE4 and Clangd
#+AUTHOR: Xiaoduan Chang

* Abstract

  This article introduces how to setup a basically usable development
  environment using Emacs to develop Unreal Engine 4 project. The assumed
  working operating system is Windows, but it can be easily ported to Linux or
  MacOS. The UE4 project is still compiled using Microsoft Visual Studio tool
  chain, but Emacs is used as the main code editor.

  By "a basically usable development environment", I mean when using Emacs
  editing UE code, Emacs provides decent functions a programmer need in one's
  everyday development. At least, this include:

  + Jump to definition
  + Find references of symbol
  + Search symbol
  + Auto completion

  Those functions are provided by the language server protocol (LSP). Usually,
  it is quite simple to setup LSP server and client for a project. However, UE4
  is a little bit special since it is a huge, multi-platform targeted project
  with is own compilation tools.

* Prerequisite

  + Emacs, must be Windows native. Although we can run Emacs in Windows
    Subsystem for Linux (WSL), which is what I have always been done, this setup
    requires a Windows native Emacs, because UE4 project is compiled using
    Microsoft Visual Studio tool chain, and it is much more complicated to
    setup a Linux Emacs to use Windows Clangd, or Linux Clangd to compile
    Windows project.
  + A language server protocol (LSP) client for Emacs, personally I use
    lsp-mode.
  + LLVM, specifically Clangd, you may also want to exploit clang-format and
    clang-tidy. Make sure Clangd binary is visible in PATH.
  + UE4 source code. We need to modify the Unreal Build Tool a bit so the source
    code is required.

* Generate Compilation Database

  The simplest way to make Clangd work, is to provide it with a compilation
  database. The compilation database is just a JASON file which describes how to
  compile each file in the project to the compiler in detail. If the project
  uses CMake, then it is quite easy to generate it with CMake's
  ~GENERATE_COMPILE_DATABASE~ flag in Linux. However, UE4 uses Microsoft Visual
  Studio to compile its project. Fortunately, Unreal Build Tool provides a mode
  which could generate this compilation database since 4.24. The following
  command generate compilation database for project =MyProject=.

  #+begin_src
  UnrealBuildTool.exe -mode=GenerateClangDatabase -project="path\to\MyProject.uproject" -game -engine MyProjectEditor Development Win64
  #+end_src

  Since most of the time, programmer works using the =Development Editor= build
  configuration, the above command also specifies this build configuration by
  appending "Editor" after the project name. This is required to generate the
  correct including path, which is used by Clangd to find those generated header
  files. You can change these arguments if you are working with other build
  configuration. The final rule is to make sure the compilation database has the
  correct include path in it.

* Modify Unreal Build Tool Source Code

  Although Unreal Build Tool could generate the compilation database for us, it
  does not contain those paths where the generated header files are located.

  If you do not know what I am talking about, here is a simple explanation:
  Unreal Engine uses macros to mark its source code, and its Unreal Header Tool
  scans the source code before build starts, and generate the header file which
  is required for its reflection system to work, those file must be also visible
  to the LSP server, since they actually take part in the compilation process.

  To include those paths in the compilation database, we have to modify UBT's
  source code. In =GenerateClangDatabase.cs= find the following code:

  #+begin_src csharp
  foreach (DirectoryReference IncludePath in ModuleCompileEnvironment.SystemIncludePaths)
  {
    CommandBuilder.AppendFormat(" -I\"{0}\"", IncludePath);
  }
  #+end_src

  and add a line of code to it so it looks like this:

  #+begin_src csharp
  foreach (DirectoryReference IncludePath in ModuleCompileEnvironment.SystemIncludePaths)
  {
    CommandBuilder.AppendFormat(" -I\"{0}\"", IncludePath);
    CommandBuilder.AppendFormat(" -I\"{0}\"", Module.GeneratedCodeDirectory);
  }
  #+end_src

* Add =.clangd= File

  Now, the LSP server could work as long as you put the generated compilation
  database file to the root of your project. However, there is still another
  problem you have to deal with.

  In Unreal, the widely used macro ~UCLASS~ is defined as follows:

  #+begin_src c++
  #if UE_BUILD_DOCS || defined(__INTELLISENSE__ )
  #define UCLASS(...)
  #else
  #define UCLASS(...) BODY_MACRO_COMBINE(CURRENT_FILE_ID,_,__LINE__,_PROLOG)
  #endif
  #+end_src

  In Microsoft Visual Studio, the flag ~__INTELLISENSE__~ is defined, but it is
  not defined in the generated compilation database.

  Here, we can take advantage the =.clangd= file to solve this
  problem. =.clangd= is a configuration file used by Clangd in a per-project
  level. You can put a =.clangd= file in the root of your project to apply some
  project-specific compiler settings. We can use this configuration file to
  define the flag.

  Here is an example:

  #+begin_src
  CompileFlags:
    Add: [-D__INTELLISENSE__]
  #+end_src

  With this flag well defined, Clangd now should compile the whole UE project
  without reporting any error.

* Use Unreal Header Tool to Update Generated Header Files

  Basically, the setup could work really well. However, when you edit a header
  file and add or remove several lines so some of the macros are not in its
  original line, you will find LSP server is reporting compilation errors.

  This is due to those macros use ~__LINE__~ to generate line numbers to
  identify themselves. Those macros are then defined in the generated header
  files. If you move the macros in your source code, the definition of them will
  not match. That's where those compilation errors come from.

  Thus, to solve this problem, we can regenerate those header files after we
  finish editing our source code. This is usually done before each time the
  build process starts. However, we can solely execute the UHT to regenerate
  header files without start a building process since it might take too much
  time.

  #+begin_src
  Engine/Build/BatchFiles/Build.bat MyProjectEditor Development Win64 -SkipBuild -project=/Path/To/MyProject.uproject
  #+end_src

  The above command runs UE script to generate header files only without
  actually building the project. You can just execute this command after you
  finish editing the source code which lead to line number changes of those UE
  macros.

* Acknowledgement

  I could not figure all these issues alone. I am an Emacs user and I am working
  on Unreal projects. Naturally, I would like to use Emacs to edit the code of
  any project I am working on. Unreal is a complicated project, thanks to its
  widely use of macros to build its own reflections system. This makes most of
  the traditional Emacs code analyzer unable to work. Most of them are based on
  tag systems, which is totally broken when UE macros are involved.

  Thanks to LSP, it provides a way for Emacs to take advantage of a real
  compiler to parse the source code.

  Also, there are a lot of people same with me, trying to use Emacs with
  Unreal. I just collect all the information I found on the Internet and put
  them together, then it works. Thanks to all the guys who have contributed on
  thoses threads all over Internet.

* Reference

  + https://answers.unrealengine.com/questions/826106/get-compile-commandsjson.html
  + https://forums.unrealengine.com/t/linux-rdm-compile-commands-json/116025
  + https://forums.unrealengine.com/t/emacs-clangd-and-unreal-engine/246457/6

  Also, check lsp-mode and Clangd documents to find more detail about how to
  configure them.