| docker pull meltano/meltano:latest-python3.8 |
This guide will walkthrough the basic construction of a singer tap that uses an api endpoint as its data source. Singer Taps can be built to pull from many different types of endpoints but most commonly used is an api.
Singer Resources: - Singer Docs
Mashey Required Setup: - Python 3.8 + - Singer-Python - Pytest
| def query_database( | |
| self, | |
| table, | |
| limit=1000, | |
| offset=0, | |
| limit_key='last_operation_time', | |
| limit_key_value='2000-01-11 00:00:00.000000' | |
| ): | |
| if table == 'procedure': | |
| self.sql.execute( |
This guide assumes:
The following process is to create a Meltano application that contains the desired extractors and loaders, and then deploy it on GCP Kubernetes Engine.
Environment Requirements:
This guide covers deploying a Docker image of a Rails and PostgreSQL application on Google Cloud Platform using Cloud Build, Container Registry, Cloud Key Management Service, Cloud Run, Cloud SQL, and Cloud Store. The following section provides links to a detailed guide created by a Google Developer Advocate which was the base for my build, but needed to be adjusted.
Google Cloud Run on Rails: a real life example Part 1
This guide is intended for a Rails application that uses PostgreSQL and is tested, developed, and deployed using Docker. The following link is a setup guide for a Rails + PostgreSQL application using Docker:
https://gist.github.com/iEv0lv3/17e33bc8d3ad4132d317ba5a1cf62d9b
The guide assumes the above Docker + Rails setup is complete. The approach in this guide creates a new Rails environment: staging, and the only responsibility of the staging environment is to be used with CircleCI for CI/CD testing. The environment name can be whatever you'd like, ci, cd, or circleci would all work as environment names as well. Why I prefer this approach is because it allows the use of standard CircleCI environments for the build and test jobs that utilize native CircleCI images for Ruby and PostgreSQL. It also allows for a high degree of customization for CircleCI without affecting Docker test, development, and production deployments.
Completing the ent
Docker desktop app is installed and running locally
if you are starting a new Rails project follow this guide
else you are working with an existing Rails project
Group Member Names:
Jordan Williams, Sebastian Sloan, David Holtkamp, Steven Anderson
Goals and Expectations for the Project (What does each group member hope to get out of this project? What do we want to achieve as a team? How will we know that we're successful?):
Group: as a team we hope to build on the foundation of our API skills and further develop new skills with new technologies. One of our primary goals is to make sure we collaborate effectively, help each other, and communicate well. Our gauge for success is if all group members have a good experience while working on this project, and also that we meet all criteria with minimum scores of 3.
Team strengths & collaboration styles (consider discussing your Pairin qualities here):