CodeBear801

import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq

csv_file = 'id2ids.csv'
parquet_file = 'id2ids.parquet'
chunksize = 10_000_000

csv_stream = pd.read_csv(csv_file, sep='\t', chunksize=chunksize, low_memory=False)

rvaidya

import pandas as pd
import numpy as np
import fastparquet
from sqlalchemy import create_engine, schema, Table

# Copied from pandas with modifications


def __get_dtype(column, sqltype):
    import sqlalchemy.dialects as sqld

danabauer

General information about setting up a developer environment on a mac

• With a Homebrewed Python installation, tips from NPR news apps team: http://blog.apps.npr.org/2013/06/06/how-to-setup-a-developers-environment.html
• Hitchhiker's guide to Python (mac): http://docs.python-guide.org/en/latest/starting/install3/osx/#install3-osx
• https://hackercodex.com/guide/mac-development-configuration/
• https://hackercodex.com/guide/python-development-environment-on-mac-osx/
• python 2/3, virtuanlenv, virtualenvwrapper: http://www.marinamele.com/2014/07/install-python3-on-mac-os-x-and-use-virtualenv-and-virtualenvwrapper.html
• setting up virtualenvwrapper in you bash_profile: http://stackoverflow.com/questions/23997403/installed-virtualenv-and-virtualenvwrapper-python-says-no-module-named-virtuale
• KyngChaos installers for geo: http://www.kyngchaos.com/
• Anaconda for data science: https://www.continuum.io/downloads

devdrops

Mysqldump from Docker container

docker exec -i mysql_container mysqldump -uroot -proot --databases database_name --skip-comments > /path/to/my/dump.sql

OBS

• This will generate a dump.sql file in your host machine. Awesome, eh?
• Avoid using --compact on your dump. This will make MySQL check your constraints which will cause troubles when reading your file (damm you MySQL). And don't use --force to fix this scenario: recreate your dump without --compact ¯_(ツ)_/¯

amirziai

georgy7

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

# Modified.
# Original script source:
# http://blog.marcbelmont.com/2012/10/script-to-extract-email-attachments.html
# https://web.archive.org/web/20150312172727/http://blog.marcbelmont.com/2012/10/script-to-extract-email-attachments.html

# Usage:
# Run the script from a folder with file "all.mbox"

joshuaadickerson

mysqldump -uroot -ppass --tab="." --fields-enclosed-by=\" --fields-terminated-by="," myschema mytable | gzip > myschema.mytable.csv.gz

JonJanzen

#!/usr/bin/python
# Use this script to update a DNS override using the webfaction API
# be sure to set your username, password, dns override, and ethenet interface.
# Then add a crontab entry for the script, I use every 5 minutes
# */5 * * * * /path/to/ddns.py
# This is safe as the script exit(0)'s if the ip is the same as wehat is recorded in the file.
# Webfaction documentation on DNS overrides
# http://docs.webfaction.com/user-guide/domains.html#overriding-dns-records-with-the-control-panel

tecoholic

/**************************************************************************
 *                 OSM2GEO - OSM to GeoJSON converter
 * OSM to GeoJSON converter takes in a .osm XML file as input and produces
 * corresponding GeoJSON object.
 *
 * AUTHOR: P.Arunmozhi <[email protected]>
 * DATE  : 26 / Nov / 2011 
 * LICENSE : WTFPL - Do What The Fuck You Want To Public License
 * LICENSE URL: http://sam.zoy.org/wtfpl/
 *