This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/bin/bash | |
| mkdir -p ~/.ssh | |
| # generate new personal ed25519 ssh keys | |
| ssh-keygen -o -a 100 -t ed25519 -f ~/.ssh/id_ed25519 -C "rob thijssen <rthijssen@gmail.com>" | |
| ssh-keygen -o -a 100 -t ed25519 -f ~/.ssh/id_robtn -C "rob thijssen <rob@rob.tn>" | |
| # generate new host cert authority (host_ca) ed25519 ssh key | |
| # used for signing host keys and creating host certs |
The following gist is an extract of the article Building a simple crawler. It allows crawling from a URL and for a given number of bounce.
from crawler import Crawler
crawler = Crawler()
crawler.crawl('http://techcrunch.com/')
If you see Access denied, or other error messages when trying to access a git repository or SSH server, here are some steps you can take to resolve the problem.
When in doubt, run ssh -vvv <other options...> to enable more verbose logging.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # -*- coding: utf-8 -*- | |
| import StringIO | |
| import math | |
| import boto3 | |
| # custom hash function that hashes the filename | |
| from generator import generate_file_hash |
A little info about your project and/ or overview that explains what the project is about.
A short description of the motivation behind the creation and maintenance of the project. This should explain why the project exists.
Build status of continus integration i.e. travis, appveyor etc. Ex. -
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| """ | |
| Downloads folders from github repo | |
| Requires PyGithub | |
| pip install PyGithub | |
| """ | |
| import os | |
| import sys | |
| import base64 | |
| import shutil |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # List unique values in a DataFrame column | |
| pd.unique(df.column_name.ravel()) | |
| # Convert Series datatype to numeric, getting rid of any non-numeric values | |
| df['col'] = df['col'].astype(str).convert_objects(convert_numeric=True) | |
| # Grab DataFrame rows where column has certain values | |
| valuelist = ['value1', 'value2', 'value3'] | |
| df = df[df.column.isin(valuelist)] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # The initial version | |
| if [ ! -f .env ] | |
| then | |
| export $(cat .env | xargs) | |
| fi | |
| # My favorite from the comments. Thanks @richarddewit & others! | |
| set -a && source .env && set +a |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # A simple cheat sheet of Spark Dataframe syntax | |
| # Current for Spark 1.6.1 | |
| # import statements | |
| #from pyspark.sql import SQLContext | |
| #from pyspark.sql.types import * | |
| #from pyspark.sql.functions import * | |
| from pyspark.sql import functions as F | |
| #SparkContext available as sc, HiveContext available as sqlContext. |
OlderNewer

