Skip to content

Instantly share code, notes, and snippets.

@adwait-godbole

adwait-godbole/GSoC-2024-final-report-bowtie-trend.md

Last active August 26, 2024 16:12
Show Gist options
  • Save adwait-godbole/14c58ac5db7d392de00bc74b828406ee to your computer and use it in GitHub Desktop.
Save adwait-godbole/14c58ac5db7d392de00bc74b828406ee to your computer and use it in GitHub Desktop.
Download ZIP
Raw
GSoC-2024-final-report-bowtie-trend.md

GSoC'24 @ JSON Schema (Final Report)

This page demonstrates the project work titled "bowtie-trend: Long-Term Reporting With Bowtie" during Google Summer of Code, 2024 under JSON Schema.

Overview:

  • Organization: JSON Schema
  • Project: bowtie-trend: Long-Term Reporting With Bowtie
  • Contributor: Adwait Godbole ([email protected])
  • Mentor: Julian Berman
  • Co-Mentors: Agnivesh Chaubey

About Project:

Bowtie is a meta-validator of the JSON Schema specification, providing universal access to JSON Schema implementations, giving users a way to use any implementation supported by Bowtie.

Till now, Bowtie supported validating instances against a given JSON Schema using only the latest release version of the specified implementation.

This project primarily focused on enabling multiple release versions support of the specified implementation for validating instances against a given JSON Schema. This enabled the scope for knowing and reporting on how the compliance numbers have changed over time as the implementations fixed bugs or as new tests were added to the test suite.

Contributions:

PR Link Description Status
#1250 Added backward compatibility for running ruby-json_schemer implementation for versions up to v0.2.17 Merged
#1292 Added backward compatibility for running js-hyperjump implementation for versions up to v0.18.0 Merged
#1428 Added backward compatibility for running python-jsonschema implementation for versions up to v4.0.0 Merged
#1470 Added backward compatibility for running java-json-schema implementation for versions up to v1.2.0 Merged
#1311 Modified the existing images.yml workflow for building and publishing versioned images of the specified "implementation" string input Merged
#1382 Taking a "version" string input in images.yml workflow for only building and publishing image of that specific version of the implementation input Merged
#1337 Added a new versioned-report.yml workflow for generating JSON Schema Test Suite reports for different versions of implementations Merged
#1462 Taking a "implementation" string input in versioned-report.yml workflow for only generating JSON Schema Test Suite reports for different versions of that specified implementation Merged
#1408 Publish an implementation metadata file suitable for use by json-schema.org Merged
#1429 Generating attestations for wheels and images generated in the GitHub workflows of Bowtie Merged
#1384 Add bowtie trend CLI command Open
#1401 Graphing the generated trend reports of the backward compatible implementations on the official Bowtie website at https://bowtie.report Open

Detailed Explanation:

Week 1-3:

In the first three weeks, I primarily focused on adding backward compatibility for some of the primarily used implementations at Bowtie.

  • Julian suggested me to start the backward compatibility journey by first adding support for ruby-json_schemer implementation as he had observed that it was historically failing multiple tests from the suite and later in the future versions had significant improvements in its compliance numbers.
  • Not being much familiar with the syntax of Ruby programming language, I devoted some time to learn Ruby from scratch to make myself comfortable enough to do small changes in the implementation's harness.
  • Finally modified the existing ruby-json_schemer's harness code in Bowtie to be backward compatible with its versions up to v0.2.17.
  • Later, modified the existing js-hyperjump implementation's harness code to add backward compatibility for versions up to v0.18.0. This task involved diving into the official source code and documentation of js-hyperjump.
  • Later, modified the existing python-jsonschema implementation's harness code to add backward compatibility for versions up to v4.0.0. This task also involved diving into the official source code and documentation of python-jsonschema library.
  • Later, took up the task for modifying the java-json-schema implementation's harness code for adding support for versions up to v1.2.0. Again not being much familiar with Java's syntax, had to learn a bit of it to get accustomed with it and finally made the changes in the harness code, again diving into the official source code and documentation of java-json-schema library.

By the end of these weeks, I had a decent number of implementations to test and generate the trend reports for.

Week 4-6:

Decided to dive into the GitHub workflows of Bowtie to modify and add some new ones to achieve the task of generating trend reports.

  • Not much familiar with the GitHub actions/workflows syntaxes, devoted some time to read a few blogs and official GitHub documentations to learn and understand about them.
  • Modified the existing images.yml workflow of Bowtie to take in an "implementation" name input and build Docker/Podman like container images, tagging them with their specific release versions and publishing them to the official ghcr.io packages repository of Bowtie.
  • Added a new versioned-report.yml workflow to Bowtie for generating versioned reports of all implementations that supported backward compatibility in their harnesses.
  • Modified the existing ui.yml workflow of Bowtie to download the latest/last generated versioned reports from versioned-report.yml workflow using the dawidd6/action-download-artifact@v6 GitHub action and deploying them along with the GitHub pages artifact of Bowtie website to serve those reports eventually to the trend graphs.
  • Modified the existing images.yml workflow again to take in a "version" input along with the already existing "implementation" name input and only build and publish images for the specified implementation and its version simplifying the workflow usage for future.
  • Meanwhile during this period, an urgent requirement arose from a few of the JSON Schema community members to Bowtie for publishing an implementations metadata file for official usage by json-schema.org. So halted my ongoing work for 2 days and worked on delivering them the file at the earliest via modifying the existing report.yml workflow.
  • Modified the already deployed versioned-report.yml workflow to take an "implementation" name input and only generate versioned reports for that implementation.
  • Later, modified the images.yml workflow and ci.yml workflow for generating attestations for wheels and images artifacts that were being produced by Bowtie, adhering to the supply chain security standards of software lifecycle as guided over here.

By the end of these weeks, I had all the workflows in place to trigger them and generate the versioned reports for implementations that supported backward compatibility.

Week 7-9:

  • Started writing out the bowtie trend CLI command code for taking in an implementation name using --implementation/-i flag and a dialect name using --dialect/-D flag for showing compliance numbers of that implementations' versions across that dialect's test suite.
  • Had to learn about Python's asyncio and rich library to streamline the process of downloading multiple versioned reports JSON files from the official Bowtie website while displaying progress for the same.
  • Raised the PR and was continuously working on the suggested changes by Julian making sure I was delivering up to the standards and agreed upon taking in a .tar input file in bowtie trend for the ease of writing out the integration tests and not making real HTTP calls in CI.
  • Worked on taking in the .tar input file and eventually was successful in integrating it bowtie trend and writing out the integration tests for the same.

Week 10-12:

  • Started writing out the code for generating trend line graph on the official Bowtie website by leveraging recharts library in TypeScript.
  • Iterated over a few design decisions of how the interactivity with the trend line graph should be structured making sure the user is able to select a dialect from a dropdown and able to see the compliance numbers over the test suite of that dialect across multiple versions of that implementation displaying the failedTests, erroredTests and the skippedTests.
  • Raised the PR and was continuously working on the suggested changes by Agnivesh and Julian again making sure I was delivering up to the standards and writing highly optimized code trying to reduce the time complexity as much as possible.
  • Refactored the code introducing a new Implementation class, encapsulating all the data and functions required by an implementation inside of it.
  • Also managed to leverage vitest and @testing-library/react for writing out the tests for testing the code's functionality.

End and Scope for More Work?

I managed to achieve the primary objectives of the project which were to do the below:

  • A new bowtie trend command which can aggregate together results from Bowtie, producing a trend report.
  • A page on the website which uses this trend report to produce graphs of failed tests over time.

Future work and additional enhancements involve the following:

  • Adding backward compatibility support for more implementations that are present in Bowtie.
  • Intoducing a way of querying the versioned reports to actually get to know deeper insights into exactly which tests were failing in earlier versions that are passing in the future versions and not just how the complaince numbers (totalUnsuccessfulTests) have changed in between.

GSoC and Its Importance

Google Summer of Code (GSoC) stands as a pivotal program that bridges the gap between aspiring open-source contributors and industry-expert developers. It provides an impactful platform for students to immerse themselves in real-world projects, all under the nurturing guidance of experienced mentors. The exposure that GSoC offers is truly invaluable, granting newcomers the opportunity to immerse themselves in the collaborative realm of open source and actively contribute to meaningful projects such as Bowtie. I am profoundly grateful for the chance I received to serve as a GSoC contributor, a journey that enriched knowledge and skills under the guidance of my mentors and fellow community members.

In conclusion, GSoC is a transformative journey that propelled me into the expansive world of open source, facilitating my growth both technically and professionally. It has cultivated within me the spirit of contribution and collaboration, virtues highly esteemed within the software industry. My commitment to this community remains unwavering; I intend to sustain my involvement well beyond GSoC, actively contributing to the project's evolution and success.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment