polleyg

Exporting Stackdriver logs back to BigQuery for analysis is a very straightforward task (and a very good pattern to follow too BTW). However, when you are managing lots of projects you'll probably want to coalesce the logs from all your projects into one dedicated BigQuery project/dataset for easier querying and maintainability.

However, setting this up in the GCP console isn't that obvious to most users. This is because the configuration in the Stackdriver sink config doesn't actually allow you to select a different project/dataset to send the logs to. It only lets you select the currently selected/working project:

The tip/trick is to select Custom destination from the drop-down and then use the following format to specify the project/dataset where you want the logs to go. Plug in the project and dataset that you want to use:

`bigquery.googleapis.com/projects/[PRO

polleyg

Sometimes you need to troubleshoot and inspect the details of jobs (load, query etc.) in BigQuery. Inspecting the job history in the BigQuery web UI will only show the jobs that you have run. This is also true when you run run bq ls -j on the command line.

But, what if you need to get all jobs that have been run? An example would be auotmated jobs run by service accounts. A quick tip is to use the --all flag:

-a,--[no]all: Show all results. For jobs, will show jobs from all users. For datasets, will list hidden datasets. For transfer configs and runs, this flag is redundant and not necessary.

bq ls -j --all

(this tip originated from a question on Stack Overflow: https://stackoverflow.com/questions/47583485/bigquery-history-of-jobs-submitted-through-python-api)