GCP command cheatsheet

GCP Data flow

For VM instance, it's essential that:

Before a VM instance can be created, compute API should be enabled gcloud services enable compute.googleapis.com
zone and machine-type are required in order to create a VM instance
Metadata can be passed to the instance by using --metadata={{ attribute name }}={{ attribute value }} which then can be retrieved by curl -H "Metadata-Flavor: Google" http://http://metadata.google.internal/computeMetadata/v1/instance/attributes/{{ attribute name }} while inside the instance.
To set startup script of a VM instance from a file --metadata-from-file startup-script={{ Relative path to the startup script }}. But, do remember, startup-script is just an attribute so we can also do --metadata startup-script='...'. startup-script can be read by curl -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/attributes/startup-script
For the VM instance to be able to use any GCP services, it needs to have the correct scopes (https://cloud.google.com/sdk/gcloud/reference/alpha/compute/instances/set-scopes#--scopes). Set the scopes using --scopes=default,{{ extra scopes }}

About Google VPC Network

VPC Network API is already enabled when we enable GCE (Google Compute Engine)
gcloud commands for VPC Network is prefix by compute such as gcloud compute networks list
To create a new VPC: gcloud compute networks create exploration-network --bgp-routing-mode=global --description='just a test' --mtu=1500 --subnet-mode=auto
To delete a VPC: gcloud compute networks delete {{ name }} {{ name }} BUT all of their firewall-rules needs to be deleted first using gcloud compute firewall-rules delete {{ name }} {{ name }}; Get firewall-rules names from gcloud compute firewall-rules list
A VPC won't work without a firewall: gcloud compute firewall-rules create exploration-network-firewall --network exploration-network --allow tcp:22,tcp:3389,icmp

More examples:

# Create regional custom network
gcloud compute networks create sel-network --subnet-mode=custom --bgp-routing-mode=regional
# Create a subnet for the network
gcloud compute networks subnets create us-central1-subnet --range=172.168.0.0/24 --network=sel-network --region=us-central1
# Create firewall rule for the network
gcloud compute firewall-rules create sel-network-fw --network=sel-network --allow tcp:22,tcp:3389,icmp

GCP IAM

A permission is the lowest form of authorization, usually correspond to an API call to a service.
A role is a collection of permissions.
Service account is an identity that used by machine to access to Google Cloud Services. It can have roles. It acts like a resource means if a real user want to use a service account, he/she needs permission.
A real user (identified by your google account) needs to be granted roles or permissions to acccess GCP.
Using group is an effective way to manage your organization since group is a collection of users and service accounts. A group is identified by an email address, has an IAM policy. By adding an user to a group, you effectively granted all permissions that associated with that group. When the user is retired, by removing the user from his group, you remove all his permissions to the project.
There is a resource hierarchy, Organization -> Folder -> Project -> Resources. Policy (a JSON file that specified what role an user has on a scope of resources) is inherited from top to bottom

GCP SDK

# List all projects
gcloud projects list

# List current cloud shell confg
gcloud config list

# Create a new project
gcloud projects create services-exploration-labs-1211 --name="Services Exploration Labs"

# Attach project to a billing account. CAREFUL, this needs some time to propagate to the services
gcloud beta billing projects link services-exploration-labs-1211 --billing-acount={{ billing account id }}

# CAREFUL! Remember to set the current project since cloud shell is used for every projects that you have
gcloud config set project {{ project-id }}

# Get the current project
gcloud config get-value project

# Get all configurations
gcloud config configurations list

# Create a new configurations and switch to it
gcloud config configurations create {{ config name }}

# Delete a configuration
gcloud config configurations delete {{ config name }}

# Get configuration description
gcloud config configurations describle {{ config name }}

# List all services
gcloud services list 

# Enable a service
gcloud services enable compute.googleapis.com

# List all GC VMs
gcloud compute instances list

# Create an VM instance
gcloud compute instances create --machine-type={{ machine type }} test-vm

# Create an VM instance with scopes, https://cloud.google.com/sdk/gcloud/reference/alpha/compute/instances/set-scopes#--scopes
# Instance metadata will be written to http://metadata.google.internal/computeMetadata/v1/instance/attributes/{{ attr name }}
gcloud compute instances create --machine-type={{ machine type }} --scopes=default,storage-rw test-vm \
  --metadata-from-file startup-script={{ relative path }} \
  --metadata={{ attr name }}={{ attr value }}
  
# Inside the instance, retrieve metadata by
curl -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/attributes/{{ attr name }}

# Delete
gcloud compute instances delete test-vm

# List all compute machine types, zones, regions
gcloud compute {{ machine-types|zones|regions }} list
# e.g.
gcloud compute machine-types list --filter="NAME=f1-micro AND ZONE~asia-southeast"

# Set defauls for VM instance, CAREFUL! We can set an invalid zone that not in the list
gcloud config set compute/zone {{ zone }}
gcloud config set compute/region {{ region }}

# SSH to a compute VM
gcloud compute ssh {{ vm name }}

# Query for meta of the instance. CAREFUL! You need to be on the instance
curl -H "Metadata-Flavor:Google" http://metadata.google.internal/computeMetadata/v1/project/project-id

GCS bucket

# In Cloud Shell
# Add -m to gsutil to perform the command multi-thread

# A bucket can be created using cloud shell (https://cloud.google.com/storage/docs/locations)
gsutil mb -l {{ locations }} gs://{{ bucket-id }}

# Get labels of a bucket
gsutil label get gs://{{ bucket-id }}

# Set a json file as label for a bucket
gsutil label set label-file.json gs://{{ bucket-id }}

# Change a label 
gsutil label ch -l "label-key:label-val" gs://{{ bucket-id }}

# Delete a label
gsutil label ch -d "label-key" gs://{{ bucket-id }}

# Change labels like this works too! Label changes are executed from left to right
gsutil label ch -d "label-key" -l "label-key:label-val" gs://{{ bucket-id }}

# Check if bucket versioning is enabled
gsutil versioning get gs://{{ bucket-id }}

# Set bucket versioning status, turn off versioning doesn't delete versioned objects
gsutil versioning set {{ on|off }} gs://{{ bucket-id }}

# Use ls -a to see version of bucket objects. 'a' stands for archive
gsutil ls -a gs://{{ bucket-id }}/file-name.txt

# Restore a deleted object simply by copying the versioned name of that object to its original name. This create a brand-new object with different version number. Deleting the versioned old object won't delete the new object
gsutil cp gs://{{ bucket-id }}/file-name.txt#{{ v1 }} gs://{{ bucket-id }}/file-name.txt 

# Get current live version of an object.
# I don't know, tell me in the comment section if you know! Thanks!

# Get access control list (ACL) of an object
gsutil acl get gs://{{ bucket-id }}/file-name.txt

# Set ACL of an object (more using gsutil acl --help)
gsutil acl get gs://{{ bucket-id }}/file-name.txt > acl-file-name.json
# Make changes to acl-file-name.json
gsutil acl set acl-file-name.json gs://{{ bucket-id }}/file-name.txt

# Make an object publicly readable
gsutil acl ch -u AllUsers:R gs://{{ bucket-id }}/file-name.txt
# To Revert
gsutil acl set private gs://{{ bucket-id }}/file-name.txt

# List all buckets
gsutil list

# List all objects of a bucket
gsutil list gs://{{ bucket-id }}

# Move object inside a bucket
gsutil mv gs://{{ bucket-id }}/file-path.txt gs://{{ bucket-id }}/file-path-alt.txt

# Move object between bucket works too!
gsutil mv gs://{{ bucket-id }}/file-path.txt gs://{{ bucket-id-1 }}/file-path.txt

# Copy all objects from bucket-A to bucket-B in a "flat" way (not moving directory structure)
# If bucket-A has 2 objects with the same name belong to different directories, the outer most object wins
gsutil cp gs://bucket-A/** gs://bucket-B

# Use cp -r to also copy the directory structure
gsutil cp -r gs://bucket-A/* gs://bucket-B

# CAREFUL! This will create a directory named bucket-B in bucket-a
gsutil cp -r gs://bucket-A gs://bucket-B

# We can't create a directory but we can move an existing object into one that doesn't necessary exist
gsutil mv gs://{{ bucket-id }}/file-path.txt gs://{{ bucket-id }}/test-folder/file-path.txt 

# Delete all objects inside a bucket
gsutil rm -r gs://{{ bucket-id }}/*

# CAREFUL! Delete a whole bucket
gsutil rm -r gs;//{{ bucket-id }}

Tips & Summary:

Give your bucket a short name
GCS buckets work much like a file system with bucket list act as top level directories

vuongpd95/gcp-command-cheatsheet.md

GCP Data flow

GCP IAM

GCP SDK

GCS bucket