Skip to content

Instantly share code, notes, and snippets.

View mzpqnxow's full-sized avatar
🙈
Look at all this free stuff!

AG mzpqnxow

🙈
Look at all this free stuff!
View GitHub Profile
@mzpqnxow
mzpqnxow / bulkupsert.py
Created February 5, 2021 19:05 — forked from luke/bulkupsert.py
I needed to upsert (insert or update) bajillions of records into postgresql. After trying various libs including upsert (which was slow as hell) I ended up doing a bit of research and trying 3 different methods. This one won. While I'm manually building the sql string no user data is passed in. Its loaded via the copy from statement as CSV. Call…
import logging
import cStringIO
import csv
DEBUG = False
def data2csv(data):
si = cStringIO.StringIO()
cw = csv.writer(si, delimiter='\t',lineterminator="\n")
for row in data:
@mzpqnxow
mzpqnxow / sql.py
Created March 9, 2021 01:29 — forked from jorisvandenbossche/sql.py
Patched version of pandas.io.sql to support PostgreSQL
"""
Patched version to support PostgreSQL
(original version: https://github.com/pydata/pandas/blob/v0.13.1/pandas/io/sql.py)
Adapted functions are:
- added _write_postgresql
- updated table_exist
- updated get_sqltype
- updated get_schema
@mzpqnxow
mzpqnxow / description.md
Created March 13, 2021 14:52 — forked from mangecoeur/description.md
Pandas PostgresSQL support for loading to DB using fast COPY FROM method

This small subclass of the Pandas sqlalchemy-based SQL support for reading/storing tables uses the Postgres-specific "COPY FROM" method to insert large amounts of data to the database. It is much faster that using INSERT. To acheive this, the table is created in the normal way using sqlalchemy but no data is inserted. Instead the data is saved to a temporary CSV file (using Pandas' mature CSV support) then read back to Postgres using Psychopg2 support for COPY FROM STDIN.

@mzpqnxow
mzpqnxow / subprocess.py
Created April 15, 2021 18:40 — forked from thomasballinger/subprocess.py
Using a pseudo-terminal to interact with interactive Python in a subprocess
from subprocess import Popen, PIPE
import pty
import os
from select import select
import sys
import tty
master, slave = pty.openpty()
p = Popen(['python'], stdin=slave, stdout=PIPE, stderr=PIPE)
pin = os.fdopen(master, 'w')
@mzpqnxow
mzpqnxow / hardening_usbarmory.md
Created April 24, 2021 15:57 — forked from yann2192/hardening_usbarmory.md
Hardening USB Armory

Hardening the USB Armory

As a good crypto nerd, I usually use an entirely encrypted linux FS: / but also /boot using grub LUKS support. It's a good setup but it's not perfect, the BIOS and the bootloader are not protected.

I recently got a USBArmory and I wanted to apply the same (or a better) setup.

I found some useful links but no clear howto. So this is my setup.

@mzpqnxow
mzpqnxow / logging.py
Created August 8, 2021 15:36 — forked from kingspp/logging.py
Python Comprehensive Logging using YAML Configuration
import os
import yaml
import logging.config
import logging
import coloredlogs
def setup_logging(default_path='logging.yaml', default_level=logging.INFO, env_key='LOG_CFG'):
"""
| **@author:** Prathyush SP
| Logging Setup
@mzpqnxow
mzpqnxow / node.sh
Created August 22, 2021 15:24 — forked from yspb/node.sh
#make directories
mkdir /git
mkdir /db
mkdir /db/modules
mkdir /downloads
#update packages, install gcc and redis
add-apt-repository ppa:chris-lea/redis-server -y && \
apt-get update && \
apt-get upgrade -y && \
@mzpqnxow
mzpqnxow / redis.conf
Created August 22, 2021 15:25 — forked from yspb/redis.conf
################################## MODULES #####################################
loadmodule /db/modules/libredis-roaring.so
loadmodule /db/modules/rejson.so
loadmodule /db/modules/redisearch.so
################################## NETWORK #####################################
bind 127.0.0.1
protected-mode yes
[Unit]
Description=/etc/rc.local Compatibility
ConditionPathExists=/etc/rc.local
[Service]
Type=forking
ExecStart=/etc/rc.local start
TimeoutSec=0
StandardOutput=tty
RemainAfterExit=yes
@mzpqnxow
mzpqnxow / rc.local
Created August 22, 2021 15:26 — forked from yspb/rc.local
#!/bin/sh
sysctl vm.overcommit_memory=1
sysctl -w net.core.somaxconn=65535
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag
exit 0