JamesOBenson

Transforming Code into Beautiful, Idiomatic Python

Notes from Raymond Hettinger's talk at pycon US 2013 video, slides.

The code examples and direct quotes are all from Raymond's talk. I've reproduced them here for my own edification and the hopes that others will find them as handy as I have!

Looping over a range of numbers

for i in [0, 1, 2, 3, 4, 5]:

JamesOBenson

#!/bin/bash
MYDNS="8.8.8.8"
DATE=$(date '+%Y-%m-%d')
UBUNTU1_URL="http://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-amd64-disk1.img"
UBUNTU2_URL="http://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.img"
CIRROS_URL="http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img"
CENTOS_URL="http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2c"
VOLUME_SIZE=10
NEW_VOLUME_SIZE=15
VOLUME_NAME=$(cat /proc/sys/kernel/random/uuid)

JamesOBenson

#!/bin/sh
SIZES="1 10 20 50"
MBGB="MB"   # Options are:  GB OR MB
SAMPLE_SIZE=5  # Number of time to run.
USER="admin"
PASSWORD="admin-password"
ENDPOINT="namenode2-1.utsaics.edu"

# Please also note that the port changed from 50070 to 9870 in Hadoop 3.0.
echo "Which version of Hadoop are you using?"

JamesOBenson

[defaults]
#callback_whitelist = timer, profile_tasks
fact_caching = jsonfile
fact_caching_connection = /tmp/.ansible_fact_cache
forks = 25                      # Default: 5
host_key_checking = False       # Default: True
internal_poll_interval = 0.01   # Must be greater than ansible 2.1, Default: 0.001 Greater is faster.
pipelining = True               # Default: False
poll_interval = 5               # Default: 15
#transport = paramiko           # May require some config changes

JamesOBenson

#!/bin/shell

# This will download all files in your ceph cluster under the 'vms' pool,
# Convert them from raw to qcow2, Create a virtualbox and mount that 
# disk to the new VM for local access.


array=()
while IFS='' read -r line; do array+=("$line"); done < <(rbd ls -p vms | grep disk)

JamesOBenson

Safest way to clean up boot partition - Ubuntu 14.04LTS-x64, Ubuntu 16.04LTS-x64

TL;DR:

dpkg --list linux-{headers,image,modules,tools}-\* | awk '{ if ($1=="ii") print $2}' | grep -v -e "$(uname -r | cut -d"-" -f1,2)" | xargs sudo dpkg --remove  # use dpkg to remove all but the currently running kernel version
sudo apt-get install -f  # It's common for apt to be in a broken state after running out of space on /boot
sudo apt-get autoremove  # It's really common for apt to be broken on a package that should have been removed
sudo apt-get install -y linux-generic linux-headers-generic linux-image-generic linux-tools-generic  # just in case latest kernel was mysteriously removed (e.g. when you are not running the latest kernel)
sudo update-grub  # just in case it was not done automatically by dpkg

JamesOBenson

# cat /var/log/manta-init.log
{"name":"manta-init","hostname":"b84470b6-13f8-447f-be15-7b524211ce56","pid":178453,"level":30,"msg":"UFDS: connected","time":"2020-04-20T17:59:04.623Z","v":0}
{"name":"manta-init","hostname":"b84470b6-13f8-447f-be15-7b524211ce56","pid":178453,"component":"ufds","clazz":"Client","ldap_id":"1__ldaps://ufds.ics.ics.utsa.edu","level":20,"msg":"connected after 1 attempt(s)","time":"2020-04-20T17:59:04.624Z","v":0}
{"name":"manta-init","hostname":"b84470b6-13f8-447f-be15-7b524211ce56","pid":178453,"level":30,"msg":"SAPI in full mode","time":"2020-04-20T17:59:04.648Z","v":0}
{"name":"manta-init","hostname":"b84470b6-13f8-447f-be15-7b524211ce56","pid":178453,"level":30,"msg":"getting or creating user poseidon","time":"2020-04-20T17:59:04.649Z","v":0}
{"name":"manta-init","hostname":"b84470b6-13f8-447f-be15-7b524211ce56","pid":178453,"level":30,"msg":"creating poseidon","time":"2020-04-20T17:59:04.672Z","v":0}
{"name":"manta-init","hostname":"b84470b6-13f8-447f-be15-7b524211ce56","pid":1

JamesOBenson

CN headnode   44454c4c-3500-1048-804d-b2c04f424d31 192.168.2.5
     SERVICE          SH ZONENAME
     buckets-api       1 70302e9b-7af0-4a67-b02b-fb11cd4714e8
     loadbalancer      1 61e5f2bf-d598-4324-9c75-dcb26953ff94
     moray             1 55617a89-75d6-494d-884e-465781785a8c
     webapi            1 c4be6c5e-7546-46e3-a957-2df0b739c562
CN R2-610-17  44454c4c-3900-1054-8054-b5c04f4e5131 192.168.2.89
     SERVICE          SH ZONENAME
     authcache         1 edfb241c-dfba-4549-8cf4-b9512d1eba69
     buckets-postgres  2 f2365b5c-3919-47c8-9436-d45deb2214c4

JamesOBenson

{
    "44454c4c-3000-1036-8056-b9c04f4e5131": {
        "buckets-mdapi": {
            "2": {
                "16d34111-14a4-4cb5-b2e9-35b25e576aae": 1
            }
        },
        "nameservice": {
            "b2c34e19-b352-4dcd-ba16-a31a6d6e0e22": 1
        },

JamesOBenson

sshlogin="[email protected]"
cephdir=vms
array=()
while IFS='' read -r line; do array+=("$line"); done < <(docker exec ceph_mon rbd ls -p "$cephdir")
for ID in "${array[@]}"; do
        echo "Do you want to skip this file: $ID"
        options=("yes" "no")
        select skipbackup in "${options[@]}"
        do