Author: Ivan Yashchuk
Organization: Chainer, as part of NumFocus
Mentor: Hiroyuki V Yamazaki
This document contains the overview of the work done for the Google Summer of Code 2019.
ChainerX is a C++ ndarray library with automatic differentiation capability.
Linear algebra operations form the core for most of the computation components for many machine learning methods and simulations of physical systems. This project was focusing on adding support for linear algebra routines to ChainerX and making them differentiable.
The progress is tracked under Issue #6764
| Routine | PR | Forward CUDA | Forward CPU | Backward | Merged? |
|---|---|---|---|---|---|
cholesky |
#7329 | ☑ | ☑ | ☑ | yes |
qr |
#7379 | ☑ | ☑ | ☑ | yes |
svd |
#7411 | ☑ | ☑ | ☑ | yes |
eigh |
#7503 | ☑ | ☑ | ☑ | yes |
solve |
#7474 | ☑ | ☑ | ☑ | yes |
inv |
#7474 | ☑ | ☑ | ☑ | yes |
pinv |
#7411 | ☑ | ☑ | ☑ | yes |
- Added
tril,triu,triroutines PR #7791 - Small fixes PR #7921, PR #7866, PR #7840
- Support zero-sized input PR #7980
- Support float16 precision input PR #7995
The project is to create a new ChainerX linalg namespace and fill it with the linear algebra methods from numpy.linalg. Also, the aim is to have the new linalg library to be plugin-based, as the rest of ChainerX, which will allow the developers to target other devices easily by adding neccessary kernels in the future.
The new ChainerX linalg library supports CPU and GPU backend linear algebra operations with LAPACK and cuSOLVER libraries. The backend to use is inferred internally by the location (CPU/GPU) the data is stored at and the call is dispatched to corresponding LAPACK or cuSOLVER based kernels.
Routines in ChainerX (chainerx_cc\chainerx\routines) are autograd operations on chainerx::Array. Routine defines forward (normal operation) and backward (computation of derivatives) logic and delegates the computation to device kernels (chainerx_cc\chainerx\kernels).
As these new routines are not implemented using available routines in ChainerX, they all need to have an algorithm for computing derivatives using reverse-mode automatic differentiation (aka backpropagation, backward mode) implemented. There are references to the formulas provided in the source code.
As a result of this GSoC project, cholesky, qr, solve, inv, svd, pinv, eigh, eigvalsh functions were implemented for ChainerX. Most of the changes are in master branch already and available both using C++ API and in chainerx.linalg namespace from Python interface.
ChainerX is very close to NumPy. Therefore if one is familiar with NumPy then it won't be difficult to start with ChainerX. To demonstrate this a demo for a few chainerx.linalg functions was made using Python interface to ChainerX.
Here is a notebook for simple Gaussian Process Regression (link).
- Add support for stacked matrices (Issue #6764 comment).
- Investigate and improve performance (Issue #7919)
- Add support for half-precision numbers
- Implement other functions from
numpy.linalg