This is a textual summary and list of notes of the podcast 'Artificial Intelligence Podcast' by Lex Fridman, from the episode Chris Lattner: Compilers, LLVM, Swift, TPU, and ML Accelerators.
Currently senior director at Google, working for projects covering CPU, GPU accelerators for Tensorflow, Google's library for Machine and Deep Learning, also involved in Swift related projects
Performed basic programming by read and copying code from books Started with simple GWBase, QBasic, then transitioned to Pascal, C, C++, and so and so forth.
- What is a compiler? It's a two sided problem where humans want to be able to write programs without having to care for every single piece of hardware, devices, and the binary representation of the actual code. Compilers allow humans to think on levels of abstractions that humans want to think, that can be either low or high level, or on the ARM architecture, or maybe programming for GPUs. Now, you have different languages, like Python, JavaScript. In the end, both sides are complicated - machines are complicated, and humans are complicated as well.
- On the front end, you have a parser, which dissects words from sentences, and in this case LLVM is a C parser. This is the front end. The middle part is often filled with optimizations, and the last part is very hardware specific related details. Usually, these three are very generalized groups which can be broken down into different groups.
- What LLVM is trying to do is that it is trying to generalize and standardize the middle part. Currently, it is working for Clang for C/C++, Rust, Julia, Objective C, Swift and that these are all very different languages which can all use the same optimization infrastructure and for hardware support as well. LLVM is something common that all these compilers for all these different languages can use.
- LLVM is an implementation, rather than a standard specification for some polices. But it demands on how you look at it. It is essentially a bunch of code. One of the interesting stuff is that many big companies, that are really just rivals, using the same shared kind of infrastructure technologies, not because of shared mutual interests, but because LLVM is such mature that different companies wish to use it as a base design and not have to rebuild all of the basic tooling services from scratch - that is an expensive proposition.
- Is it expensive or difficult? It demands on the level of skill sets, because those skill sets are hard to find. It often seems like the magic happens in a very small circle of talented people. As far as LLVM is concerned, it started of as a project in Illinois, The University Of Illinois.
- LLVM is open source, and using that knowledge, Google contributes a lot in C/C++ technologies and its ecosystem, similarly NVIDIA cares a lot as well since it is heavily invested in CUDA.
- Interaction with the subsystems is a lot, and one of the challenges that LLVM looked was that it first looked for other examples, and the most popular was GCC. But it had some certain flaws from a research perspective, for example, there were a lot of global variables that were used not necessarily relating to their original purpose, and that it was not easy to debug error messages.
- What LLVM pushed on was a better front-end, a better set of tools that helped users be friendly, and help it to be better keeper of error messages. Though, there are many phases / passes in the compiler. There are many structures involved, such as an abstract syntax tree, which keeps track of what symbols are 'called' where depending on their operation phase. There are branches, and there are nodes. For example, a for loop is often represented in a 'control flow graph' .
- The compiler does something similar to how neural networks work, in they way that it has 'neighbor nodes' that try to learn the representation of the data in each layer, and how it is transforming between the layers. These transformations are often applied iteratively. In the end, compilers tend to go for theorems that go for the higher level properties of the code and the text.
- In the 1980s, one of the biggest thing in the compiler world were registers, and it was important to keep track of how much information can be used at a single time in those registers, and the compiler has to choose what data or programs have to be put into those registers at a single time. Getting or writing code that gets this right is really important in terms of performance and effectiveness. There are many different kinds of techniques that can be used. It is basically trying to optimize against time the code it has to process.
- Later on, as CISC was getting replaced by RISC, which made things a bit more complicated for compilers, in the way that they had to care for pipelines now, and that many processors can take on the code and the need for registers at any given moment. This made considering the order of operations an important point in the design of LLVM. This meant scheduling, so that the pipelines could be prevented from getting stalling and getting blocked.
- Machine Learning is currently being trying to be used for the sake of trying to find quicker code generation for CPUs and GPUs. One problem statement would be to use brute force to generate the smallest matrix that could be fed to a GPU, though all of this is currently research material. Brute force approaches have been tried in the past, but currently attempts are being made using Reinforcement Learning.
- Have there been large or big changes / jumps in optimization? Yes, it has been largely driven by hardware and software. For example, Java introduced new ideas, though not novel, in a good way helped them to become industry standards, such as compilation, safe code, like memory safe code, portability etc. It helped to bring out research ideas into implementation. But also as well on the hardware set of instructions.
- Continuing on with Java, it's virtual machine, has a front end parses the code, in the second stage, there is optimizations and made portable, but before that, it is converted in byte code which is a set of standardized code.
- The interesting thing about LLVM, to Chris Lattner, is an important set of use of many important structures and design changes, for example, SONY has decided to use LLVM for graphical computation and compilation for its pipeline. It can be considered good pipeline design. GCC is similarly good, but not in terms of infrastructure software, and is basically a C / Fortran compiler, and not much more.
- As far as writing code is concerned, Chris does not write so much of the code these days, although he is involved many times patch changes and looks at changes from time to time, but he mostly spends time looking at if the project helps move away from technical changes in the direction, and making sure the project or projects are making some sort of progress.
- LLVM is really good at optimizations and compilations, and helped in the backend compilation and the transfer of many different changes in the architecture, like 64 bit change, the Intel change. Initially, it was Chris working on the weekends, trying to build the parser and the pre-processor on his own. Afterwards, he informed his manager who was excited enough to let someone else work with him on the same project. Slowly, the team got bigger, and they started approaching.
- Swift can be dynamically, as well as statically compiled. In notebooks, it is actually dynamically compiling the code that you execute in the notebooks. When you start typing, it creates a process, it compiles it through the Swift compiler, and then sends it to the JIT ( Just in Time compiler ), and then it puts the binary code in the source code's place.
- This was not obvious, but an important point was that it should be easy to use and test, so it could a single line that it is in Python, unlike in C/C++, Java which has to include headers, namespaces etc. This was a pretty big design point, for example, you can actually write firmware in Swift if you want to. Some want to use the low level headers, but some people just like use the higher level abstractions to make things work.
- Similarly, for dynamic languages like Python, which is dynamic, and where you can either say it has no type, or that it has either only one type, which is passed around implicitly.
Pretty raw - I have not proof read and refactored ( made text bold, italic ) for some important details and points. Hope it helps.