Mapknitter is a free and open-source software created and run by Public Lab. It lets people upload their own aerial images, position them in a web interface over some existing map data, share it, and export for print. It includes a bunch of sub-components - which contains a JavaScript UI- but the core application is written in Rails.
My GSoC project was about upgrading Mapknitter project from Rails 3.2 to Rails 6.0. Initially, the project was running on Rails 3.2. So, to get a smoother upgrade, some steps should be taken before.
I knew that this would involve a lot of work and challenges to overcome, but I got the chance to work with some stuff I've never imagined I would.
This initial upgrade was not only about the basics of an update of this kind. It involved a lot of preparation before it, this means writing a few more tests and refactoring the CI behavior.
Kaustubh Nair took charge of the testing part, and I took charge of the CI part, this has given me a chance to almost entirely change the way Travis CI works inside Mapknitter.
During this part of the program, I have mostly worked on setting a proper environment for us to work on top of. That's why I have focused not only on establishing a stable Travis CI but also on stable docker environments.
During the setups, I have realized that Travis had been using our production environment, which is not good at all. So we have successfully split the development/test/production environments and made some improvements to Travis integration by having parallel jobs (what caused the builds to be ~2x faster).
You can find more of my work here and some more improvements here.
Rails 5.2.3 was the one that took more time and had me involved in a lot of different stuff. Part of this was because we have decided to skip a few versions and went straight ahead with 5.2.3. Great decision by the way!
During this upgrade, I had the opportunity to work with Jenkins. I changed some of its workflows and refactored some of its scripts here.
Besides all of this, I got to refactor some of the Rails' Asset Pipeline workflow, this involved entirely using Yarn (totally deprecated Bower) and Sprockets. Also helped to set up some system tests here.
Skipping some Rails versions and going straight forward with Rails 5.2.3 came out to be a very smart decision because even though it took a quite long time to finish, actually got us pretty ahead of schedule when we finished it. This made our friend Vidit Chitkara able to start his work on top of our Rails 5 branch, then he started setting ActionCable.
Besides Yarn and ActionCable, with latest Rails we were able to remove some deprecated initializers and files, improve the database setting and its integration with Mapknitter core and Travis.
In this phase of the project, we were also able to properly use Rubocop linter and stylesheet, following the same standards used in Plots2 project. Which makes the PublicLab's ecosystem more cohesive and uniform. Now we have a whole single job to keep track of the asset pipeline in Travis, used to keep track of changes made across the libs used in Mapknitter and its changes influence on our codebase.
We have also set Rails' Bootsnap to improve the loading time of our app, since it caches some expensive operations, like HTTP requests.
There was a massive refactoring of our routes since this version of Rails doesn't allow abstract routing anymore, due to security issues. So by refactoring, we've made Mapknitter less sensitive to security breaches. Since we're talking security, we've enforced the usage of strong params to HTTP requests too, which wraps our "payloaded" objects better for requesting.
All of the described above has been majorly done at publiclab/mapknitter#685
To get Rails 6.0 running we had to upgrade our Ruby version. Complicated right?! Not really, just had to update our Gemfile and Dockerfile with the desired Ruby version - in our case 2.6.3 - and that's it. Also, a bunch of improvements have been gotten, such as deprecating old patterns in favor of newer ones, using the latest sintax and such.
This upgrade came out to be smoothest of the upgrade series I had been working on. Rudely I just had to run the update task provided by the Rails team - thank God for that task - and also kept on checking the RailsDiff website - a loyal friend to help in this kind of upgrade. Along with that, I removed a bunch of unwanted files that had been floating through the project.
Is worth to mention that when I finished the Rails 6.0.0 upgrade, I realized that would be super if we had Webpacker setup. This would involve many, many changes in our codebase. But I went straight ahead with this - motivated by our favorable schedule. In simple words, it would merely involve removing Sprockets and starting using Webpacker. Piece of cake, right?
Not really!
But in fact, I got pretty far with this. Actually, you can see everything I accomplished here. But this has taken more time than I expected, and after a friendly discussion with my mentors we decided to keep Webpacker implementation as a future work. But I took a lot of advantage of this actually. In fact, getting involved in this Rails 6 + Webpacker thing made me learn SO MUCH about the Rails Asset Pipeline y' all cannot imagine. Not only this, but I got the chance to get my good friend Arthur Diniz (Debian GSoC 2019 mentee) to collaborate with us of PublicLab, he gave us valuable info y' all can get here.
After a lot of negotiation, along with my mentors and colleagues, we got to use Rubocop to standardize more our coding and present to the newcomers some good coding practices. Y'all can get most of this here. One nice thing about this is that we used the same Rubocop settings as the ones used in Plots2 project, so we contribute across projects we can actually see patterns that make contributing easier.
Since I'm a good old DevOps enthusiast, solving this type of issue is my thing. With this MR I have helped to reduce our build time DRASTICALLY. See the comparative for yourself:
Before caching:
After caching:
One of the many wonderful experiences that GSoC 2019 have given to me is the opportunity to try tools I have only heard of. One of these tools was the automation server Jenkins.
As I said before, I'm a lover of DevOps, so nothing more pleasing to such creature than having the opportunity to get along with Jenkins. In fact, I got to make some improvements to the continuous delivery method provided by my coder friends of PublicLab. You can get some of the work I have made here, here and here.
If you guys didn't know my GSoC 2019 was entirely collaborative. Actually in all of the upgrades mentioned before I had the help of Kaustubh Nair and many others.
Anyway, setting Action Cable was not in my project scope at all, but Vidit Chitkara(another GSoC 2019 mentee) realized that for the Mapknitter Image Management and Synchronous Editing project we needed ActionCable up and running, and consequently Rails 5. This had us joining forces to get both projects rolling.
Visual representation of Vidit Chitkara and me joining forces
So, to get ActionCable we needed Rails 5 on MapKnitter asap and stable. So Kaustubh and I had it finished, and afterwards I hopped on publiclab/mapknitter#805 to get it on top of our Rails 5 Merge Request. For ActionCable we had to get a Puma server running, but MapKnitter already uses Passenger as its server. So we need to open another tab in our terminal and bundle exec puma -C config/puma.rb right?
NOPE!
We all know this doesn't work on production, so I had to setup Foreman, a powerful gem that allows you to describe a set of processes required to run your application. I had used it before in other projects, so I knew it would work. And it did!
I have accomplished a lot, thanks to many people that helped me. I will list below and quickly describe some more Merges I did with the help of good samaritans.
Believe it or not, I spent almost 2 weeks working on this issue. Came out to be 4-5 lines of code. This made me a little disappointed. xD
I am not the biggest fan of Javascript. But thank God for Sasha Boginsky (one of my mentors), she has helped me a lot during the project, and publiclab/mapknitter#885 was just one the merges she has braved to help me. Not only her but Jeffrey Warren, Gaurav Sachdeva, Sebastian Silva, and many others.
It's clear that my GSoC project was on top of Mapknitter, but during the program, I also got the chance to work on Plots2 project as well. I had the opportunity to fix a few issues there, among those, I could help to set ChromeDriver to system tests here and to fix a minor issue regarding the version of Google Chrome browser we also use for system tests here.
It's worth to mention that I could quickly fix those bugs in Plots2 thanks to a similar issue I got when working on the Noosfero project, right here and here.
This was a HUGE article, I know, but it just had to be. This was my first time being a GSoC mentee. Contributing to such a large organization as PublicLab was just a fantastic experience! I'll carry on the lessons I've learned for the rest of my life.
Well guys, that's it! GSoC 2019 has been a great experience to me, definitely want to participate again next year.
Special thanks to the entire PublicLab community for trusting me such hard task, but also the Noosfero team and LAPPIS team.
This is not a bug, it's a Feature - M. Richard