inchoate

History

Show file at certain commit

git show <hash>:<file>

Show history of a file

git log -p <filename>

inchoate

Internet Scale Services Checklist

A checklist for designing and developing internet scale services, inspired by James Hamilton's 2007 paper "On Desgining and Deploying Internet-Scale Services."

• http://mvdirona.com/jrh/talksandpapers/jamesrh_lisa.pdf

Basic tenets

• Does the design expect failures to happen regularly and handle them gracefully?
• Have we kept things as simple as possible?

inchoate

The Problem

The documentation for how to deploy a pipeline with extra, non-PyPi, pure Python packages on GCP is missing some detail. This gist shows how to package and deploy an external pure-Python, non-PyPi dependency to a managed dataflow pipeline on GCP.

TL;DR: You external package needs to be a python (source/binary) distro properly packaged and shipped alongside your pipeline. It is not enough to only specify a tar file with a setup.py.

Preparing the External Package

Your external package must have a proper setup.py. What follow is an example setup.py for our ETL package. This is used to package version 1.1.1 of the etl library. The library requires 3 native PyPi packages to run. These are specified in the install_requires field. This package also ships with custom external JSON data, declared in the package_data section. Last, the setuptools.find_packages function searches for all available packages and returns that

inchoate

```
17:09 $ python -mzeep 'https://listings.colliershub.com/Services/API/Currencies.svc?wsdl'
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/vertrees/.virtualenvs/colliers_testing/lib/python2.7/site-packages/zeep/__main__.py", line 86, in <module>
    main(args)
  File "/Users/vertrees/.virtualenvs/colliers_testing/lib/python2.7/site-packages/zeep/__main__.py", line 75, in main

inchoate

In [7]: %paste
"""
Testing with Berkshire-Hathaway
"""

"""
Colliers API Testing
"""
import zeep

inchoate

RealMassive Simplified Data Ingestion Format

• Version: v0.0.1-pre. This is a proposal. We encourage feedback.
• Last Edited: 2017-02-28

Here we document our file format for importing CRE data into RealMassive. Third party data providers will be interested in this because, if they:

• export their data into this format;
• provide RealMassive with a publicly accessible file via URI;
• keep that file up to date

inchoate

"""Checks to see if media URIs listed in the given file are valid.

  Warning this is untested, shit code.

    "Valid" means:
        - the file is under 32 megs
        - the file exists at the URI

    Usage:
        # check all URLs in the `your-uri-file.txt` and output failures only.

inchoate

#
# This will build the memtier_benchmark tool from Redis labs and
# run it against a Redis server running on your Mac's localhost.
#
git clone https://github.com/RedisLabs/memtier_benchmark.git
cd memtier_benchmark
docker build -t memtier_benchmark .
docker run --rm -it  memtier_benchmark -s host.docker.internal

inchoate

# Service Graph:
kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=servicegraph -o jsonpath='{.items[0].metadata.name}') 8088:8088 &

# Kiali:
kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=kiali -o jsonpath='{.items[0].metadata.name}') 20001:20001 &

# Grafana:
kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=grafana -o jsonpath='{.items[0].metadata.name}') 3000:3000 &

# Prometheus:

inchoate

#!/usr/bin/env python3
# /// script
# requires-python = ">=3.12"
# dependencies = [
#     "typer",
#     "rich",
#     "requests",
#     "python-dotenv",
# ]
# ///