stevehenderson

Sometimes you need to troubleshoot and inspect the details of jobs (load, query etc.) in BigQuery. Inspecting the job history in the BigQuery web UI will only show the jobs that you have run. This is also true when you run run bq ls -j on the command line.

But, what if you need to get all jobs that have been run? An example would be auotmated jobs run by service accounts. A quick tip is to use the --all flag:

-a,--[no]all: Show all results. For jobs, will show jobs from all users. For datasets, will list hidden datasets. For transfer configs and runs, this flag is redundant and not necessary.

bq ls -j --all

(this tip originated from a question on Stack Overflow: https://stackoverflow.com/questions/47583485/bigquery-history-of-jobs-submitted-through-python-api)

stevehenderson

gsutil ls gs://some-bucket/**

stevehenderson

#!/bin/bash
#
# A simple entropy walker.  
# Requirements:  apt install ent
#
echo 0,File-bytes,Entropy,Chi-square,Mean,Monte-Carlo-Pi,Serial-Correlation
for i in *; 
do echo $i $(ent "$i" -t | tail -n1); 
done

stevehenderson

gcloud sql import csv cloud-sql-instance-name gs://somebucket/folder/2/bigolecsv.gz -d somedatabase --table sometable

stevehenderson

How do you unmarshall nested json

Given dgraph res.Json:

"q": [
      {
        "city": {
 "uid": "0x51c7ebb",

stevehenderson

kubectl get pods --namespace somens -o wide | awk '{print $7,"   ", $1}' | sort

stevehenderson

So I'm playing around with Harmony @ cs.cornell.edu.

Install steps (Ubuntu 20) [Should work on WSL]

Install Prequisites

You need to have build-essential, clang, python3 + dev headers, and graphviz

I also always install anything python related in its own virtual environments, so I added the python3-venv package

stevehenderson

Sometimes you delete stuff on GKE Autopilot and it doesn't take:

kubectl delete pv troubledvolume --grace-period=0 --force
kubectl patch pv troubledvolume -p '{"metadata": {"finalizers": null}}'

stevehenderson

# The following sets the over all timeout
timeout: 7200s
- name: 'gcr.io/cloud-builders/gsutil'
  args: ['cp', 'gs://project-volumes/big-project/big-file.tar', '.']
  waitFor: ['-']  # The '-' indicates that this step begins immediately.
  timeout: 4600s  # Per step timeout

stevehenderson

- id: "Read Values"
  name: ubuntu
  entrypoint: bash
  args:
    - -c
    - |
      # Read from "/workspace"
      echo "Where am I " $(pwd)
      echo "Contents " $(ls)