Skip to content

Instantly share code, notes, and snippets.

@ronen-fr

ronen-fr/gsoc24_wip_1.md

Created March 5, 2024 14:26
Show Gist options
  • Save ronen-fr/de74b387e5715bf678a734d1e361414f to your computer and use it in GitHub Desktop.
Save ronen-fr/de74b387e5715bf678a734d1e361414f to your computer and use it in GitHub Desktop.
Download ZIP
GSOC24 - first edit of Yuval's
Raw
gsoc24_wip_1.md

Tidy Up Song

Background

Static analysis is a powerful tool when searching for code issues. It does, however, have its costs. Careful tuning is required so that the developers are not overwhelmed with "false positive" detections, while on the other hand - critical errors are not missed. It also requires simple integration into the development flow, so that new issues are continuously detected and addressed.

In this project we use clang-tidy for automated static analysis in the Ceph project.

clang-tidy - a component of the Clang toolchain (that is already supported by Ceph)- is a common open-source 'linter'.

clang-tidy is not a "static analyzer" per se, but close enough - and is actually wider in scope. It is highly tunable and easy to integrate.

(For a discussion of various tools that come to mind when analyzing C++ code, and what separates clang-tidy from Clang's static analyzer tool - https://www.linkedin.com/pulse/its-2020-time-touch-base-static-analysis-maurizio-martignano by Maurizio Martigano is an interesting read).

Evaluation Stage

Step 1 - Build Ceph and Run Basic Test

Start with would be to have access to a linux based development environment; a 4-CPU machine with 16G RAM and 100GB disk is a minimal requirement.

Unless you already have a linux distro you prefer, I would recommend choosing from:

  • Fedora (38/39) - my favorite!
  • Ubuntu (22.04 LTS)
  • WSL (Windows Subsystem for Linux)
  • Other Linux distros - try at your own risk :-)

Once you have that up and running, you should clone the Ceph repo from github (https://github.com/ceph/ceph). If you don’t know what github and git are, this is the right time to close these gaps :-) And yes, you should have a github account, so you can later share your work on the project.

Install any missing system dependencies using:

./install-deps.sh

Note that, the first build may take long time, so the following cmake parameter could be used to minimize the build time. With a fresh ceph clone use the following:

./do_cmake.sh -DWITH_SEASTAR=OFF -DCMAKE_C_COMPILER="clang" -DCMAKE_CXX_COMPILER="clang++" -DCMAKE_CXX_FLAGS:STRING="-std=gnu++20" -DALLOCATOR=tcmalloc -DWITH_LTTNG=OFF -DWITH_RDMA=OFF -DWITH_RADOSGW_LUA_PACKAGES=ON -DWITH_MANPAGE=OFF -DWITH_CCACHE=ON -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_EXPORT_COMPILE_COMMANDS:BOOL=ON -DALLOCATOR=tcmalloc -G Ninja -DBOOST_J=$(nproc) -DWITH_MGR_DASHBOARD_FRONTEND=OFF -DWITH_DPDK=OFF -DWITH_SPDK=OFF -DWITH_RBD=OFF -DWITH_KRBD=OFF -DWITH_RADOSGW_SELECT_PARQUET=OFF -DWITH_PYTHON3=3.12 -DWITH_SYSTEM_QATLIB=ON -DWITH_SYSTEM_QATZIP=ON

(For Fedora 38: use WITH_PYTHON3=3.11)

if the build directory already exists, you can rebuild the ninja files by using (from build):

cmake -DBOOST_J=$(nproc) ......

Once do_cmake.sh completes, move to the newly created 'build' directory, and initiate the build process:

ninja

Assuming the build completes successfully, you can run the unit tests (see: https://github.com/ceph/ceph#running-unit-tests).

Now you are ready to run the ceph processes, as explained here: https://github.com/ceph/ceph#running-a-test-cluster You probably would also like to check the developer guide (https://docs.ceph.com/docs/master/dev/developer_guide/) and learn more on how to build Ceph and run it locally (https://docs.ceph.com/docs/master/dev/quick_guide/).

For this qualification step: Run a specific standalone test, and copy the results. Use the following shell line:

SFS="/tmp/tst_`date +'%d_%T'`"; echo $SFS;  date; time ../qa/run-standalone.sh -x -v "osd-scrub-snaps.sh" 2>&1 | awk '{ print strftime(),$0 }' > $SFS; tail $SFS

The output of the 'tail' command includes a success/failure message for the test (which is expected to succeed). Share the contents of the $SFS file with us.

Step 2 - excercising problem detection & fixing using clang-tidy

Run a small subset of clang-tidy checks on a subset of Ceph source files.

  • a subset of the OSD source files may be a good first guess at a meaningful subset; Make sure there is a relevant compile_commands.json that describes how the files in the selected subset are to be compiled when building the project.

  • Select a very small set of checks. The checks performed should not be about style or specific style-guides: use a small set of bug-related or performance-related clang-tidy checks.

  • identify a few actual problems in the generated report.

  • issue a fix (including - create a PR that describes the problem and the fix).

Note: at this stage - do not report the problems detected to the Ceph bug list.

REACHED UP TO HERE FOR NOW

Step 2 - TODO

Run clang-tidy against yyy/xxx.cc and perform initial analysis of the output, select one issue which you find as "critical" and try to fix it by submitting a pull request (PR) to the ceph project.

Project Goals

  1. make sure that ceph compiles under clang
  2. tune up clang-tidy to find important issues that are common to Ceph (looking for a relatively small subset of critical issues)
  3. cleanup issues found in (2)
    • this could be by changing the code and submitting a PR
    • annotate the code so that clang-tidy ignores a false positive
    • tune clang-tidy so it won't emit false positives
    • remember that you don't have to cleanup all of them...
  4. add to jenkins/github actions (non blocking)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment