Skip to content

Instantly share code, notes, and snippets.

@arm2arm
arm2arm / disks
Last active February 22, 2017 11:16
[root@clei36 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 1.8T 0 disk
├─sda1 8:1 0 500M 0 part /boot
└─sda2 8:2 0 1.8T 0 part
├─centos_clei36-swap 253:0 0 31.5G 0 lvm [SWAP]
├─centos_clei36-root 253:1 0 50G 0 lvm /
└─centos_clei36-home 253:2 0 1.8T 0 lvm /home
sdb 8:16 0 14T 0 disk
└─360014054d5aa7f25ddc40eba440f2275
@arm2arm
arm2arm / .block
Last active October 6, 2017 09:49
RAVE-demo
license: mit
@arm2arm
arm2arm / streamlit on cocalc.md
Last active July 21, 2021 20:48
Streamlit on cocalc

Install

conda create -y -n streamlit python=3.6
conda activate streamlit
pip install streamlit

Usage

in the x11 terminal:

streamlit hello 
@arm2arm
arm2arm / gist:533b108f7a23cde71a6f93624ec7d6e8
Last active February 22, 2021 20:38
python MPI on with conda
# HPC MPI enviroment for python jobs
```
conda create -n mpi
conda activate mpi
conda install openmpi openmpi-mpicc mpi4py matplotlib astropy scikit-learn pandas pytables
```
# testing
@arm2arm
arm2arm / rebalance_zfs.sh
Created April 6, 2021 15:59
rebalance zfs for the cold data
# If you change some settings in the zfs dataset you should rebalance it in order to store data with this current settings.
# Usage:
# zfs set recordsize=1M data/archive
# sh ./rebalance_zfs.sh /data/archive| parallel -P 16 --eta
mypath=$1
find $mypath -type f | xargs -d '\n' -I{} echo "rsync -a --inplace \"{}\" \"{}.abcdef\";unlink \"{}\";mv \"{}.abcdef\" \"{}\" "
@arm2arm
arm2arm / fs_hist.sh
Created April 6, 2021 16:27
getting histogram of the filesystem files sizes
# Usage:
# sh ./fs_hist.sh /data/archive
mypath=$1
find $mypath -type f | wc -l
find $mypath -type f -print0 | xargs -0 ls -l | awk '{ n=int(log($5)/log(2)); if (n<10) { n=10; } size[n]++ } END { for (i in size) printf("%d %d\n", 2^i, size[i]) }' | sort -n | awk 'function human(x) { x[1]/=1024; if (x[1]>=1024) { x[2]++; human(x) } } { a[1]=$1; a[2]=0; human(a); printf("%3d%s: %6d\n", a[1],substr("kMGTEPYZ",a[2]+1,1),$2) }'
@arm2arm
arm2arm / ipmi_get_power.sh
Last active August 18, 2022 13:11
simple ipmi power consumption collector for the node text exporter
#!/bin/bash
echo "
# HELP node_ipmi_get_power_watts Power sensor reading from ipmitool
# TYPE node_ipmi_get_power_watts gauge
" > /var/lib/node_exporter/textfile_collector/ipmitool_get_power.prom.$$
/usr/bin/ipmitool dcmi power reading| grep Instantaneous| awk '{printf"node_ipmi_get_power_watts{sensor=\"PW Consumption\"} %d\n",$4}' >> /var/lib/node_exporter/textfile_collector/ipmitool_power.prom.$$
/usr/bin/mv /var/lib/node_exporter/textfile_collector/ipmitool_power.prom.$$ /var/lib/node_exporter/textfile_collector/ipmitool_get_power.prom
@arm2arm
arm2arm / readme-gitlab-ci-error-139.md
Last active August 19, 2022 12:28
Gitlab CI ERROR: Job failed: exit code 139

Enviroment:

  • gtilab runner working on the debian 10.x VM
  • runner requre dind to compile docker images and push them to the local registry

Error:

  • All is working only
  • Containers with the CentOS 5,6 or debian 8 are failing
  • message: ERROR: Job failed: exit code 139
  • any operation with syscalls are stopping workers docker container
""" Sample TensorFlow XML-to-TFRecord converter
usage: generate_tfrecord.py [-h] [-x XML_DIR] [-l LABELS_PATH] [-o OUTPUT_PATH] [-i IMAGE_DIR] [-c CSV_PATH]
optional arguments:
-h, --help show this help message and exit
-x XML_DIR, --xml_dir XML_DIR
Path to the folder where the input .xml files are stored.
-l LABELS_PATH, --labels_path LABELS_PATH
Path to the labels (.pbtxt) file.
@arm2arm
arm2arm / kind-localhost-30443.yaml
Created September 9, 2022 12:52
reana kind config
apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
nodes:
- extraPortMappings:
- containerPort: 30443
hostPort: 30443
protocol: TCP
# START
- containerPort: 30080
hostPort: 30080