This small subclass of the Pandas sqlalchemy-based SQL support for reading/storing tables uses the Postgres-specific "COPY FROM" method to insert large amounts of data to the database. It is much faster that using INSERT. To acheive this, the table is created in the normal way using sqlalchemy but no data is inserted. Instead the data is saved to a temporary CSV file (using Pandas' mature CSV support) then read back to Postgres using Psychopg2 support for COPY FROM STDIN.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import logging | |
import cStringIO | |
import csv | |
DEBUG = False | |
def data2csv(data): | |
si = cStringIO.StringIO() | |
cw = csv.writer(si, delimiter='\t',lineterminator="\n") | |
for row in data: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
""" | |
Patched version to support PostgreSQL | |
(original version: https://github.com/pydata/pandas/blob/v0.13.1/pandas/io/sql.py) | |
Adapted functions are: | |
- added _write_postgresql | |
- updated table_exist | |
- updated get_sqltype | |
- updated get_schema |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
from subprocess import Popen, PIPE | |
import pty | |
import os | |
from select import select | |
import sys | |
import tty | |
master, slave = pty.openpty() | |
p = Popen(['python'], stdin=slave, stdout=PIPE, stderr=PIPE) | |
pin = os.fdopen(master, 'w') |
As a good crypto nerd, I usually use an entirely encrypted linux FS: /
but also
/boot
using grub LUKS support. It's a good setup but it's not perfect, the BIOS and
the bootloader are not protected.
I recently got a USBArmory and I wanted to apply the same (or a better) setup.
I found some useful links but no clear howto. So this is my setup.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import os | |
import yaml | |
import logging.config | |
import logging | |
import coloredlogs | |
def setup_logging(default_path='logging.yaml', default_level=logging.INFO, env_key='LOG_CFG'): | |
""" | |
| **@author:** Prathyush SP | |
| Logging Setup |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#make directories | |
mkdir /git | |
mkdir /db | |
mkdir /db/modules | |
mkdir /downloads | |
#update packages, install gcc and redis | |
add-apt-repository ppa:chris-lea/redis-server -y && \ | |
apt-get update && \ | |
apt-get upgrade -y && \ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
################################## MODULES ##################################### | |
loadmodule /db/modules/libredis-roaring.so | |
loadmodule /db/modules/rejson.so | |
loadmodule /db/modules/redisearch.so | |
################################## NETWORK ##################################### | |
bind 127.0.0.1 | |
protected-mode yes |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[Unit] | |
Description=/etc/rc.local Compatibility | |
ConditionPathExists=/etc/rc.local | |
[Service] | |
Type=forking | |
ExecStart=/etc/rc.local start | |
TimeoutSec=0 | |
StandardOutput=tty | |
RemainAfterExit=yes |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/bin/sh | |
sysctl vm.overcommit_memory=1 | |
sysctl -w net.core.somaxconn=65535 | |
echo never > /sys/kernel/mm/transparent_hugepage/enabled | |
echo never > /sys/kernel/mm/transparent_hugepage/defrag | |
exit 0 |