Skip to content

Instantly share code, notes, and snippets.

@pavgup
pavgup / gist:1deee218fa2bc078b117d4e51467384f
Created July 26, 2017 02:33
Random data science tricks
shuf -n 10 -e * | xargs -i mv {} path-to-new-folder <-- random file movement
@pavgup
pavgup / dogscats.py
Created July 28, 2017 22:09
fast.ai lesson 1 updated files for keras 2.0.2
from __future__ import division,print_function
import os, json
from glob import glob
import numpy as np
np.set_printoptions(precision=4, linewidth=100)
from matplotlib import pyplot as plt
import sys
sys.settrace
-----BEGIN PGP PUBLIC KEY BLOCK-----
mQINBFnX7LkBEADYNw+dXd/9Z7ZwQG1WJF27RYkBualuruarY7yDiXJoanYVkNzk
Cs+9AQ3rEnkYCdEBONdGSiKa2735BxQdojpTcSMeRNDEg8PJQJiAhQ5MkN5P2cTL
2m6HmOtw+gRKxVerD2e7WsMvt2j2YzpeCGqM72EKmS9coysreeL+7PZy32Oc9MTG
k/UhaP9Gu44AFlYg6t5W5aKYHoSujbaABGWKl74V77SQwYUE84lhl5eHKIGZvZ/P
HKV06NzyWWBgFv8MH3ISr+xiMzG+MNPTuwaFrnm9tePIMK+fB14mbn0t2mIXKwvx
9wB2QjWaXA9n7MCH4znK/eLsVcE1qkHT2S01XtldLCllUikwskHhEKTQOkEjKMH9
IAPYdJncR3ckwzCZo2mQQoVihnERcZ2gtrlwx2GV+900ntvBr8QHTLDUZkcO0bFF
o1ICHwwkpansWThgzkTKil6E/bCicPlh7FPmBiFKMqsz5IK8e6DGIAPjDGDJjkMy
@pavgup
pavgup / move-all-files.sh
Last active September 15, 2018 21:56
Moving all jpegs to a single directory
### The first line simply moves the files using their existing filenames, if there is a complex tree structure with the same filenames, you overrwrite fies
find . -type f -exec file --mime-type {} \; | awk '{if ($NF == "image/jpeg") {split($0,a,":"); print a[1]}}' | xargs -I{} cp '{}' /Users/pavan/Dental/all-images
### The second line adds complexity by grabbing random strings and dropping renaming the copied files randomly:
find . -type f -exec file --mime-type {} \; | awk '{if ($NF == "image/jpeg") {split($0,a,":"); print a[1]}}' | xargs -I{} sh -c 'cp "{}" /Users/pavan/Dental/all-images/$(cat /dev/urandom | env LC_CTYPE=C tr -cd 'a-f0-9' | head -c 64).jpg'
@pavgup
pavgup / aws-resource-count-ucsf.sh
Created November 20, 2018 23:29
Updated RedLock Count Resources (command line argument for credentials profile)
#!/bin/bash
aws_regions=(us-east-1 us-east-2 us-west-1 us-west-2 ap-south-1 ap-northeast-1 ap-northeast-2 ap-southeast-1 ap-southeast-2 eu-central-1 eu-west-1 sa-east-1 eu-west-2 ca-central-1)
echo "Total regions: "${#aws_regions[@]}
ec2_instance_count=0;
rds_instance_count=0;
elb_count=0;
elasticache_count=0;
redshift_count=0;
@pavgup
pavgup / nested-unzip.sh
Created September 16, 2019 13:55
Fusion Export Nested Unzipper
#!/bin/bash
# Nested Unzipping, Updated for MacOS Find Utility
# Pavan Gupta - September 16, 2019
# Drop this script into a directory with your zip file, execute, wait for nested directories
# TODO: Zips with zip files in them having the same name would
# cause unzip to drop files into the same directory causing collisions potentially
function extract(){
unzip $1 -d ${1/.zip/} && eval $2 && cd ${1/.zip/}
@pavgup
pavgup / gist:826289918d00fd041847d7cb5587854a
Created October 3, 2019 22:34
example of how to request network file storage and how to mount it
➜ ucsf-cdhi-pedfast cat ucsf-cdhi-pefast-cephfs.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ucsf-cdhi-pedfast-fs-pvc
namespace: ucsf-cdhi-pedfast
spec:
accessModes:
- ReadWriteMany
resources:
# This will use osd.5 as an example
# ceph commands are expected to be run in the rook-toolbox
1) disk fails
2) remove disk from node
3) mark out osd. `ceph osd out osd.5`
4) remove from crush map. `ceph osd crush remove osd.5`
5) delete caps. `ceph auth del osd.5`
6) remove osd. `ceph osd rm osd.5`
7) delete the deployment `kubectl delete deployment -n rook-ceph rook-ceph-osd-5`
8) delete osd data dir on node `rm -rf /var/lib/rook/osd5`
@pavgup
pavgup / gist:54211a6104fcf5fe63bf05d72dc1c95d
Last active January 30, 2020 15:33
recursive zip file extraction with dcm dropout? woo!
import zipfile
from pathlib import Path
# Pathlib Friendly (maybe required)
def extract(filename):
z = zipfile.ZipFile(filename)
for file in z.namelist():
# Ignore anything that's not a zip... and lets recurse
if ".zip" in file.lower():
# Create a subdirectory to hold our zip:
@pavgup
pavgup / config
Created January 30, 2020 19:51
alice config
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRFNU1UQXlOREl4TURVd05Gb1hEVEk1TVRBeU1USXhNRFV3TkZvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBT3hZCnlxdEdseklVd2lYWVBKTGh4MFhsdnNtRUVYS1Zwa3NZMS9ESWdZdUFYWjRBeXk3KzAzcUVmOElUa3BSRVI3ZnQKbEt4cXZ6dmsvd3Mwc2R3UlZRcEhjUUttYUo3Z3l2ZWh4Zkg2Z1lCbWR5TW4rZ0Nwczh1Z3ZSUGNsZi9zc3NDZwo2TmJncUJxQTRVNklQeW9oSWo2T0EwV2FPb01ZRk1XM0FjZ3g0N05tb21HYm9BMG5pRk5vWW43VkNJYTh6U2I5CkdwNTJXb3JXZitCMi9TZm9LQVIzVEhyeWNrK0IzMmVDWkExSDV6bGN4MENrNE1LeXpNS0VKUDR3dEplZkRRNVcKMGhUUSthWEVJMmJwRnBmT3JiSThCTW1vZjladktVTzRkR0NHTEd1MFBXcEFNc0MydnV1K0piOW1lVHhXQkpsSQphRVNRQjFtUTN2MURHZ2Q4ZmtFQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFLYXNzK0NYbUVZdEM4dXh6cEFvRG56Tjh5NTAKcUt4RDlHaVFNRU0vRXJhVFp0My9aY3JncURUQlNUUURoN2J1TlRi