Terraform Deploy Prefect Server to GCP

README.md

Provisioning Prefect Server on GCP using Terraform

Requirements: Terraform, GCP credentials with permission to create a project Tested with: Terraform v0.12.23, google cloud provider v3.26

To run:

You can use an environment variable to set which service account key to use during provisioning export GOOGLE_APPLICATION_CREDENTIALS=/path/to/key The service account you use must belong to a GCP project that has the necessary APIs enabled (such as billing and resource manager)--if it does not, you may have to enable these APIs manually along the way in those projects In this directory, run terraform init then terraform apply -- you will need to provide a billing_id and organization_id for your GCP project

Once successful, be sure to add the following to your ~/.prefect/config.toml

[server]
endpoint = "YOUR_MACHINES_PUBLIC_IP:4200/graphql"

Also, you may need to run prefect backend server to interact with Prefect Server instead of Prefect Cloud

Note: Because of odd behaviour of the project services, you may need to run this twice before it passes (terraform sometimes does not wait for the compute engine api to be fully available) That error looks like this:

Error: Error creating Firewall: googleapi: Error 403: Compute Engine API has not been used in project ##### before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/compute.googleapis.com/overview?project=284203841876 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry., accessNotConfigured

  on gcloud.tf line 57, in resource "google_compute_firewall" "default":
  57: resource "google_compute_firewall" "default" {

main.tf

resource "random_id" "project_random" {
  prefix      = "prefect-"
  byte_length = "7"
}

resource "google_project" "project" {
  name            = random_id.project_random.hex
  project_id      = random_id.project_random.hex
  org_id          = var.organization_id
  billing_account = var.billing_id
  skip_delete     = "true"
}

resource "google_project_service" "service" {
  count   = length(var.project_services)
  project = google_project.project.number
  service = element(var.project_services, count.index)

  # Do not disable the service on destroy. On destroy, we are going to
  # destroy the project, but we need the APIs available to destroy the
  # underlying resources.
  disable_on_destroy = false
}

resource "google_compute_instance" "prefect" {
  depends_on   = [google_project_service.service]
  project      = google_project.project.name
  name         = "prefect-server"
  machine_type = var.machine_type
  zone         = var.zone

  tags = ["prefect-server"]

  boot_disk {
    initialize_params {
      image = "ubuntu-os-cloud/ubuntu-1804-bionic-v20200414"
      size  = "10"
    }
    device_name = "prefect-server"
  }

  network_interface {
    network = "default"
    access_config {
      network_tier = "PREMIUM"
    }
  }

  metadata_startup_script = file("startup.sh")

  service_account {
    scopes = ["logging-write","monitoring-write"]
  }
  allow_stopping_for_update = true
}

resource "google_compute_firewall" "default" {
  name        = "open-prefect-server-to-world"
  project     = google_project.project.number
  network     = "default"
  target_tags = ["prefect-server"]

  allow {
    protocol = "tcp"
    ports    = ["4200", "8080"]
  }
}

output server-ip {
    value = google_compute_instance.prefect.network_interface.0.access_config.0.nat_ip
}

output project-name {
    value = google_project.project.id
}

startup.sh

apt-get update -y
apt-get install -y python3 python3-pip
python3 -m pip install --upgrade pip wheel
apt-get install -y python3-venv
apt update -y
apt install -y docker.io
systemctl start docker
systemctl enable docker
curl -L "https://github.com/docker/compose/releases/download/1.25.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
git clone https://github.com/PrefectHQ/prefect.git
python3 -m venv prefect-env
source prefect-env/bin/activate
cd prefect && python3 -m pip install .
IP=`curl -H "Metadata-Flavor: Google" http://metadata/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip`
mkdir ~/.prefect
cat <<EOF >>~/.prefect/config.toml
[server]
  
  [server.ui]
    
    graphql_url = "http://$IP:4200/graphql"
EOF
prefect server start

variables.tf

variable organization_id {}
variable billing_id {}
variable project_services {
    default = ["compute.googleapis.com"]
}

variable machine_type {
    default = "n1-standard-1"
}

variable zone {
    default = "us-central1-a"
}

In order to access the server from outside, the last line of the startup.sh script should be:

prefect server start --expose

@RRRajput You made my day! 🙇‍♂️