Skip to content

Instantly share code, notes, and snippets.

View semyont's full-sized avatar

Semyon semyont

View GitHub Profile
@semyont
semyont / gevent_wsgi_tornado.py
Created June 4, 2017 08:35
gevent wsgi tornado example
import os.path
import tornado.web
import tornado.wsgi
import gevent.wsgi
class MainHandler(tornado.web.RequestHandler):
def get(self):
self.render('main.html', page_title="", body_id="", messages="whatever",title="home")
settings = {

Performance of Flask, Tornado, GEvent, and their combinations

Wensheng Wang, 10/1/11

Source: http://blog.wensheng.org/2011/10/performance-of-flask-tornado-gevent-and.html

When choosing a web framework, I pretty much have eyes set on Tornado. But I heard good things about Flask and Gevent. So I tested the performance of each and combinations of the three. I chose something just a little more advanced than a "Hello World" program to write - one that use templates. Here are the codes:

1, Pure Flask (pure_flask.py)

@semyont
semyont / mytasks.py
Created June 11, 2017 08:33 — forked from demoray/mytasks.py
A custom complete method for luigi tasks that will re-run the task if any prerequisite task is newer
class MyTask(luigi.Task):
def complete(self):
def _get_last(outputs):
last = 0.0
for item in outputs:
if not item.exists():
continue
current = os.path.getmtime(item.path)
if current > last:
last = current
@semyont
semyont / luigi_server.cfg
Created June 14, 2017 14:53
luigi server config explained
[core]
# These parameters control core luigi behavior, such as error e-mails and
# interactions between the worker and scheduler.
default-scheduler-host: localhost
# Hostname of the machine running the scheduler. Defaults to localhost.
default-scheduler-port: 8082
# Port of the remote scheduler api process. Defaults to 8082.
@semyont
semyont / s3_bucket_policy.json
Created June 20, 2017 12:43
IAM Simple Bucket Policy For AWS
"Statement": [
{
"Effect": "Allow",
"Action": "s3:ListAllMyBuckets",
"Resource": "arn:aws:s3:::*"
},
{
"Effect": "Allow",
"Action": "s3:*",
"Resource": [
@semyont
semyont / World_Map_Service_url_kibana_map_coordinates.txt
Last active February 8, 2018 12:36
Free WMS Kibana Map Coordinates
WMS url => http://ows.mundialis.de/services/service?
WMS layers => TOPO-OSM-WMS
WMS version => 1.1.1
WMS format => image/png
@semyont
semyont / clean.sh
Last active December 25, 2018 11:04
Clean UBUNTU Space
sudo apt-get clean
sudo apt-get autoremove --purge -y
@semyont
semyont / s3_to_redshift_upsert.py
Created December 5, 2018 10:02
Redshift Upsert
import uuid
import psycopg2
from secret import REDSHIFT_CREDS
from secret import AWS_ACCESS_KEY, AWS_SECRET_KEY
def get_primary_keys(tablename, db):
c = db.cursor()
sql = "select indexdef from pg_indexes where tablename = '%s';" % tablename
c.execute(sql)
result = c.fetchall()[0][0]
@semyont
semyont / example_etl.py
Created January 2, 2019 08:39 — forked from dlstadther/example_etl.py
Example ETL Using Luigi
# import python core modules
import datetime
import logging
# import external modules
import pandas as pd
import requests
# import luigi modules
import luigi
@semyont
semyont / pandas_weighted_avg_example.py
Created January 17, 2019 10:27
Pandas Group By Aggregate CTR Weighted Average on Campaigns Mediums Example
#!/usr/bin/python
__author__='Semyon Teplisky'
import functools
import pandas as pd
import numpy as np
raw_data = [{'campaign_id':1111,'medium_id':'msn','campaign_name': 'viral #1', 'clicks':100, 'ctr':0.7},
{'campaign_id':1111,'medium_id':'aol','campaign_name': 'viral #1', 'clicks':250, 'ctr':0.4},
{'campaign_id':1111,'medium_id':'google','campaign_name': 'viral #1', 'clicks':500, 'ctr':0.1}]