gh-review-apps.md

Review and Test PR's on Heroku Review Apps

As the result of exploring Heroku review apps we came up with an experiment to run this sprint:

• We'll spin up review apps manually from PR's that you can find on sapling's deploy dashboard on Heroku
• Heroku has this name convention for the apps https://hipcamp-sapling-pr-1920.herokuapp.com
• Review apps will be destroy automatically once the PR has been merged
• Please don't leave review apps alive if not necessary, will pay for them by the second
• We should stop using slack bot to deploy to sapling or sapling and run this experiment
• We should share the URL on Trello cards. I'm going to look into the bot and see if I can add them automatically
• We could potentially change the process, read bellow if interested

Things to consider:

• HC-TRAIL is our new Heroku pipeline
• Hipcamp-sapling is set as the staging app for the the staging stage on the pipeline
• Hipcamp-sapling can be set to get the new code when merged to master, but currently this process has to be run manually
• Review apps can take up to 10 minutes to build
• Review apps use sapling's postgres database, redis and elasticsearch instances

How to use

• Go to hipcamp-trail dashboard
• Enable github on your account. It will be prompt if you can't see any PR on the review apps section of the dashboard (make sure there are PR's open in the repo)
• From the PR's cards on the review apps sections you can create, destroy, rebuild, check logs

Notes:

• sapling needs to be updated manually
• some env vars are inherit from sapling other are overwritten via config/staging.rb
• apps have a life span of one day
• apps have to be created manually

Alternatives to manage data in review apps

1. Recommended by Heroku. Migrate every time and have seeds files to populate individual instances. We dont have seeds at the moment.
2. We could migrate, dump and restore on each review app. But we thought that it might be too time consuming, considering that just building the app takes ~10 minutes (measured only one time)
3. What we are doing now, sapling maintains data consistency by keeping its db up to date and migrating there first when necessary. Then the review apps connect to that instance.

Alternatives to manage env variables using app.json

1. Inherit form parent

  "env": {
    "ASSET_HOST": {
      "required": true
    }
  }

2. Hard code file

  "env": {
    "ASSET_HOST": {
      "required": true,
      "value": <hard code value>
    }
  }

3. Override env variables programatically using the Heroku Platform Api and inherit from parent (we could have a way to do this easily with some tools) PATCH /apps/{app_id_or_name}/config-vars

$ curl -n -X PATCH https://api.heroku.com/apps/$APP_ID_OR_NAME/config-vars \
  -d '{
  "FOO": "bar",
  "BAZ": "qux"
}' \
  -H "Content-Type: application/json" \
  -H "Accept: application/vnd.heroku+json; version=3"

Exploration

Definition

Review apps run the code in any GitHub pull request in a complete, disposable app on Heroku. Each review app has a unique URL you can share.

Cost, Dynos and add-ons used by review apps are charged in exactly the same way as for normal apps. Costs are pro-rated to the second and you’re only charged for the time that the review app exists (typically while the GitHub pull request is open). You can optionally try to specify free or low-cost add-on plans in your app.json if those plans are sufficient to run and test your app.

You can configure review apps to spin up automatically for each pull request, or create each manually at the push of a button from the Heroku Pipelines page of the connected repo.

Review apps can only be used in conjunction with Heroku Pipelines.

Heroku Pipelines

Recommended work flow for Heroku Pipelines:

1. A developer creates a pull request to make a change to the codebase.
2. Heroku automatically creates a review app for the pull request, allowing developers to test the change.
3. When the change is ready, it’s merged into the codebase’s master branch.
4. The master branch is automatically deployed to staging for further testing.
5. When it’s ready, the staging app is promoted to production, where the change is available to end users of the app.

Currently we go to production on step 4 not having a staging app to promote.

Possible steps

Changing current deployment process:

1. Add a pipeline to hipcamp-tinh
2. Add hipcamp-sapling as staging stage on the pipeline
3. Review apps will be created upon PR to master using sapling as base, manually from heroku deploy dashboard
4. On merges to master code gets deployed to hipcamp-sapling
5. Start promoting manually from staging to production or investigate if this step can be added to GitHub integration (I did not see that as an option)

Without changing current deployment process (currently set on heroku):

1. Create a pipeline only for hipcamp-sapling set as staging
2. Review apps gets created manually, with sapling as base from heroku deploy dashboard
3. DB_URL is inherited from sapling
4. Once PR are marged to master, hipcamp-tinh and hipcamp-sapling get the code

Warnings

• Copying full database contents from parent to Review apps (similar to heroku fork) is not currently supported. Copying production data to test apps means risk of data leaks or other programming mistakes operating on recent customer data. For those reasons, we instead recommend seeding databases comprehensively with non-production data using seed scripts run with the postdeploy command.
• Builds take around 10 minutes
• We are inheriting the DB url from the parent app in this case hc-sapling
• Check cost at the end of the sprint

This is an awesome audit, thank you Seba!

To me this process isn't adapted to us yet though, here's why:

1. We don't have a QA team / person yet. I don't think having a production-like environment ready (on sprout for example) after merging the PR for a second review is useful to us yet. Whoever reviews will do it using the preview app. We will only merge the PR when the reviewer is happy with the new feature. I'm just saying that I think we're not big enough (yet) to need to add an extra review step to our deployment process.
2. We will make our deployment process more structured but also more complicated, heavier and slower. How about hotfixes?
3. Maintaining database seeds is work—that we don't need to do with our current process
4. More expensive

Conclusion: awesome for bigger teams, not necessary for us right now.
If we're trying to solve the issue of not having enough staging apps for 6 engs, let's create a third staging app. Once we have a dedicated QA person or are a team of 10+ it will be time to consider preview apps.
Again, this is just my own opinion, happy to debate

Really nice work! In my experience, good seeds are time consuming and brittle to keep up. Though we do need to think about data custody/leaks and soon, it's not going to be viable to restore full backups of our app locally.

If we can autogenerate a subset of our production data ( and hopefully anonymize it ) we might be able to manage it programmatically, but this will also take maintenance and development.

I'm OK with seeing how it goes migrating against sprout, but feel the same way Pierre does that we might not quite be at the point where we need individual pipeline apps. I do think we need to have a conversation around developing and testing with live data though. Further, I think we should probably address this in a meeting to wrap our heads around it together.

:Pat

I disagree with @pierrea's points and I think there's some confusion around the difference between Review apps and the Pipeline process. The way I understand it is that they're related but disconnected for the most part.
I agree that we don't need the pipeline process right now, we're too small and it may slow us down. However, if we only use review apps and not the deployment pipeline then the only change to our process is that review apps can be created for each PR. With review apps we get the following advantages:

1. The review apps are independent of production so hotfixes and production deployments would stay as-is.
2. We can get rid of one our staging environments and will most likely end up saving us money (TBD). I'm almost positive it will be less expensive than creating a 3rd staging env.
3. The reality is that we are QA right now; so if we need to test something on a staging env we need to make sure that one is available. That goes away with review apps.

With respect to the database, I agree that we need a better solution than using a copy of production data, but with review apps, we won't need to use seeds. Each review app uses the connection specified by its parent, so each review app will be sharing a database but I don't see that as being a show stopper. Having multiple migrations are handled well by rails.

I see review apps as a good step forward.

I agree with @jon-eckstein. Leaving everything as it is, no formal pipeline, but using one staging app to use as parent can be convenient.