Skip to content

Instantly share code, notes, and snippets.

Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@joshlk
joshlk / faster_toPandas.py
Last active September 19, 2025 16:11
PySpark faster toPandas using mapPartitions
import pandas as pd
def _map_to_pandas(rdds):
""" Needs to be here due to pickling issues """
return [pd.DataFrame(list(rdds))]
def toPandas(df, n_partitions=None):
"""
Returns the contents of `df` as a local `pandas.DataFrame` in a speedy fashion. The DataFrame is
repartitioned if `n_partitions` is passed.
@evenv
evenv / Spark Dataframe Cheat Sheet.py
Last active August 3, 2025 19:50
Cheat sheet for Spark Dataframes (using Python)
# A simple cheat sheet of Spark Dataframe syntax
# Current for Spark 1.6.1
# import statements
from pyspark.sql import SQLContext
from pyspark.sql.types import *
from pyspark.sql.functions import *
#creating dataframes
df = sqlContext.createDataFrame([(1, 4), (2, 5), (3, 6)], ["A", "B"]) # from manual data
@borgar
borgar / .block
Last active September 15, 2025 05:01
PRS B1919+21
license: cc-by-nc-sa-4.0
height: 500
border: no
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@dbapl
dbapl / .gitattributes
Last active August 1, 2024 14:33 — forked from kbaird/git_binary_diff_gist
Diff setup for ~/.gitconfig, .gitattributes file for repository and helper scripts.
*.ods diff=odf
*.odt diff=odf
*.odp diff=odf
*.pdf diff=pdf
*.PDF diff=pdf
*.apk diff=apk
*.bz2 diff=bz2
*.gz diff=gzip
*.zip diff=zip
*.tar diff=tar
anonymous
anonymous / Completer Widget.ipynb
Created February 3, 2016 22:17
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
anonymous
anonymous / Theming nbpresent.ipynb
Created January 28, 2016 04:35
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@blark
blark / completer.py
Last active April 2, 2020 02:53
A really basic completer for prompt_toolkit
from __future__ import print_function, unicode_literals
from prompt_toolkit.completion import Completer, Completion
class PwnCompleter(Completer):
cmds = {'quit': None,
'exit': None,
'use': ['/foo/bar', '/herp/derp', '/baz/qux'],