szs8

Effective Engineer - Notes

• By Edmond Lau
• Highly Recommended 👍
• http://www.theeffectiveengineer.com/

What's an Effective Engineer?

• They are the people who get things done. Effective Engineers produce results.

Adopt the Right Mindsets

szs8

# see also https://github.com/wrobstory/pgshift

import gzip
from io import StringIO, BytesIO
from functools import wraps

import boto
from sqlalchemy import MetaData
from pandas import DataFrame
from pandas.io.sql import SQLTable, pandasSQL_builder

szs8

"""
DataFrameTable
==============

Quick and Dirty Qt app to view pandas DataFrames.  Includes sorting and
filterting.

Based on qtpandas in pandas sandbox module, by Jev Kuznetsov

Usage:

szs8

OS X ships with an ancient version of openssl and I ran into problems using starcluster with Anaconda python distribution (APD).

Every starcluster command failed with the following error:
SSLError: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

The continuum guys are working on fix. Meanwhile a quick workaround is to use homebrew to build python with a newer openssl version and then substitute the _ssl.so in APD with the homebrew _ssl.so


1) brew install python --with-brewed-openssl

szs8

#!/usr/bin/env python

import dateutil
from datetime import date, datetime
import numpy as np
import pandas as pd
import requests


class YahooException(Exception):

szs8

def diffCols(df1, df2):
    """ Find columns in df1 not present in df2
    Return df1.columns  - df2.columns maintaining the order which the resulting
    columns appears in df1.

    Parameters:
    ----------
    df1 : pandas dataframe object
    df2 : pandas dataframe objct
    Pandas already offers df1.columns - df2.columns, but unfortunately

szs8

#!/usr/bin/env python

import wx
from wx import EVT_CLOSE
import wx.grid as gridlib

EVEN_ROW_COLOUR = '#CCE6FF'
GRID_LINE_COLOUR = '#ccc'

class PandasTable(wx.Frame):

szs8

In [25]: quotes.head()
Out[25]: 
                              bid    ask  bsize  asize
2012-09-06 09:30:00.026000  13.34  13.44      3     16
2012-09-06 09:30:00.043000  13.34  13.44      3     17
2012-09-06 09:30:00.121000  13.36  13.65      1     10
2012-09-06 09:30:00.386000  13.36  13.52     21      1
2012-09-06 09:30:00.440000  13.40  13.44     15     17

In [26]: quotes["mid"] = 0.5 * (quotes.bid + quotes.ask)

szs8

# -*- coding: utf-8 -*-
"""
example use of pandas with oracle mysql postgresql sqlite
    lightly tested.

    to do:  
            save/restore index (how to check table existence? just do select count(*)?), 
            finish odbc, 
            add booleans?, 
            sql_server?

szs8

L1 cache reference                  0.5 ns
Branch mispredict                   5 ns
L2 cache reference                  7 ns
Mutex lock/unlock                   25 ns
Main memory reference               100 ns
Compress 1K bytes with Zippy        3,000 ns
Send 2K bytes over 1 Gbps network   20,000 ns
Read 1 MB sequentially from memory  250,000 ns
Round trip within same datacenter   500,000 ns
Disk seek                           10,000,000 ns