Custom Logs in GCP Logging

Logs can be sent to GCP logging very simple.

Basic Usage

You need to install the package google-cloud-logging.

Then the code will looks somehting like this:

import google.cloud.logging
import logging

def send_logs(request):
    # create logging handler
    client = google.cloud.logging.Client()
    client.setup_logging()

    # send logs in standard way
    logging.warning("This is text from CF")

Example of log entry

If it's used in Cloud Functions the code above will add an entry to logs as follows (some data is obfuscated):

{
  "textPayload": "This is text from CF",
  "insertId": "62f8c0e00002a59a54704f9d",
  "httpRequest": {
    "requestMethod": "POST",
    "requestUrl": "http://function-1-hvtjs6nopa-lm.a.run.app/",
    "userAgent": "PostmanRuntime/7.29.2",
    "protocol": "HTTP/1.1"
  },
  "resource": {
    "type": "cloud_run_revision",
    "labels": {
      "location": "europe-central2",
      "revision_name": "function-1-00001-wem",
      "project_id": "YOUR_GCP_PROJECT",
      "service_name": "function-1",
      "configuration_name": "function-1"
    }
  },
  "timestamp": "2022-08-14T09:31:12.173466Z",
  "severity": "WARNING",
  "labels": {
    "instanceId": "00c527f6d4b8e8f224d3815e6251f48b0fee12ebdcddfa55069759a04ffd8dd415b8d1dfe9d7950093e451d3ed30a706a602cbdc5498e7d3e1ec68d0d363e5be",
    "goog-managed-by": "cloudfunctions",
    "python_logger": "root"
  },
  "logName": "projects/YOUR_GCP_PROJECT/logs/run.googleapis.com%2Fstdout",
  "trace": "projects/YOUR_GCP_PROJECT/traces/25aac09f5526561fbc9c6c5183fccd3f",
  "sourceLocation": {
    "file": "/workspace/main.py",
    "line": "10",
    "function": "send_logs"
  },
  "receiveTimestamp": "2022-08-14T09:31:12.507411262Z",
  "spanId": "894b9c16ebc1ca02",
  "traceSampled": true
}

Custom Labels

In some cases you will need to add custom labels to indentify logs. It's can be done by means of labels parameter:

    client = google.cloud.logging.Client()
    client.setup_logging(labels={'app_name': 'myapp'})

It also could be ued from other code, such as Dataproc jobs.

Console Output

By default, logs become invisible in console output of a routine. For example, for Dataproc jobs you might have used print for debug logging. When you switch to standard logging, such logs will no longer be visible in job details. You need to add StreamHandler to see them again. But this will duplicate log entries in Cloud Logging, because GCP will be still capturing them and adding to logging. So that I'm not recommending to do this without reconsidering its purpose and usage of CLoud Logging.

    rootLogger = logging.getLogger()
    consoleHandler = logging.StreamHandler()
    rootLogger.addHandler(consoleHandler)

Custom Metrics

Based on the custom logs it's possible to create application metrics and use them in alerts.

Usage on Local

A Google Cloud project ID and appropriate service account credentials should be supplied directly to the Cloud Logging library for Python.

xtrmstep/send-logs-to-gcp-logging.md