Skip to content

Instantly share code, notes, and snippets.

@casperdcl

casperdcl/gsod2019.md

Last active December 3, 2019 14:17
Show Gist options
  • Save casperdcl/2f67897cfb2d9280a92304bd69b415db to your computer and use it in GitHub Desktop.
Save casperdcl/2f67897cfb2d9280a92304bd69b415db to your computer and use it in GitHub Desktop.
Download ZIP
GSoD 2019 Wrap-up
Raw
gsod2019.md

GSoD 2019 Wrap-up

Meetings were held on Hangouts and/or Slack, and notes are posted in the Team Discussion page.

The GSoD19 project board contains issues and pull requests which have been completed, are currently in progress, and are still to do. As the board is dynamically updated, this is a static and concise summary of the current state:

Done

  • creation of machine user @openworm-bot for fine-grained permission control
    • including ssh & gpg keys, email address, API keys
    • handover of all credentials
  • creation of org-wide team & project board
  • creation of CircleCI projects/users
    • setup of org permissions
    • creating and linking a new project to the docs repo
    • posting notifications to org's slack team channel

Most work was done on the existing documentation, docs.openworm.org, where the source code may be found at openworm/openworm_docs.

Create a fully-controlled CI/build/deploy env for the docs subdomain

This was mostly openworm_docs#56 -> openworm_docs#57, openworm_docs#58

Notable external helper PRs were PyGithub#1270 and igraph/python-igraph#255.

  • add CI tests for documentation
    • includes all branches and PRs
  • automatically build documentation including pulling in metadata (from .openworm.yml files in the root of master branches) from all the organisation's other repos
  • automatically re-pull metadata every hour
  • automatically deploy master if changes are detected
  • add an interactive graph of relations for all the org's repos
  • configure GitHub pages to serve the site (circumventing readthedocs)
  • add badges and document how to maintain the documentation (meta! - see README at openworm/openworm_docs)
  • remove redundant old unused files (openworm_docs#59)
    • git's history can be used to restore these files if needed. Cluttering the production branch with backups is not needed
  • discuss & fix isolated docs pages (openworm_docs#63)
  • stop unneeded hourly docs updates (openworm_docs#73)
  • use Jinja templates to ease injecting calculated metadata into docs (openworm_docs#71 -> openworm_docs#72)

A screenshot of the newly-created interactive repository graph:

repo-graph

Screenshot of some of the metadata automatically pulled in from https://github.com/OpenWorm/*/master/.openworm.yml:

image

In Progress

ToDo

Most of these are organisation-wide tasks requiring multiple people

  • preview PR builds in a nicer way than CI artifacts (openworm_docs#66)
    • the build artifacts is the static html site folder produced by each CI job
    • Circle CI makes no guarantees about how long the artifacts will remain available
    • the artifacts are "fake" hosted (so hyperlinked pages will only render properly if index.html is appended and/or the CI artifact root is prefixed)
  • update outdated references (openworm_docs#65)
    • PyOpenWorm has been renamed OWMeta and the docs should be updated to reflect this
  • update all repos metadata (openworm_docs#70)
    • check that the metadata automatically pulled in by the new docs site is up-to-date
  • add missing repo metadata (openworm_docs#69)
    • add metadata to more of the org's repos so that they show up in the new docs site
  • replace heavyweight igraph with networkx (openworm_docs#74)
    • A graphing package is used to display the relationships/links between the org's repos. igraph -- while it may have better output algorithms -- can be difficult to install and is not as widespread as networkx
  • tidy up/increase accessibility of openworm.org, source at openworm/openworm.github.io
    • the root website could do with some redesigning, making useful links/resources more prominent and intuitively accessible

Conclusion

Challenges

  • coordinating across 3 timezones
  • coordinating & unifying documentation processing with a plethora of subprojects with separate leads
  • the hardest issue to debug ever (GitHub API rate of 5k/hr exceeded - required opening tickets with CircleCI, GitHub, TravisCI, conda-forge, affects ~9k open source repos (!) and still has not been fully resolved: conda-forge/conda-forge.github.io#908)

Learning

  • coordination, discussion and meta docs (documenting the documentation) can be time consuming!
  • trade-off between different free CI providers (e.g. CircleCI allows scheduling jobs every minute; TravisCI can only do daily at most)
  • fine-grained access permission management with machine users and restricted API keys
  • before the project began
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment