Yi Wang yiwang

Spark "Timestamp" Behavior

Reading data in different timezones

Note that the ansi sql standard defines "timestamp" as equivalent to "timestamp without time zone". However Spark's behavior depends on both the version of spark and the file format

format \ spark version	<= 2.0.0	>= 2.0.1

Serializing Binary Data in Rust

The way I like to serialize data in Rust into binary formats is to let a data structure blit itself into a mutable buffer. This is a relatively composable, low level way to work that lends itself to having other abstractions built on top of it. I recently was serializing network packets, so let's make up a small packet format that illustrates how we can do this.

+-------------+-------------+
|  Tag (u16)  | Count (u16) |
+-------------+-------------+
|                           |
~        Entry (u32)        ~

Installation of NixOS with encrypted root

These are my notes on instaling NixOS 16.03 on a Lenovo ThinkPad X1 Carbon (4th generation) with an encrypted root file system using UEFI.

Most of this is scrambled from the following pages:

Install HomeBrew first

brew update
brew tap caskroom/cask
brew install brew-cask

If you get the error "already installed", follow the instructions to unlink it, then install again:

require-from-twitter

Since Twitter doesn't have an edit button, it's a suitable host for JavaScript modules.

How to use

Source tweet: https://twitter.com/rauchg/status/712799807073419264

const leftPad = await requireFromTwitter('712799807073419264');

Rapid mobile application development with Python

While there are examples of Python-based apps in mobile App stores, the knowledge of how to actually create a mobile app in Python hasn't been well documented, or simplified for mass use - until now.

In this talk, Dr Russell Keith-Magee will demonstrate a collection of tools from the BeeWare Project that enable you to build a cross-platform mobile app using Python in a matter of minutes.

	from h3 import h3

	geoJson = {'type': 'Polygon',
	'coordinates': [[[37.813318999983238, -122.4089866999972145],
	[ 37.7866302000007224, -122.3805436999997056 ],
	[37.7198061999978478, -122.3544736999993603],
	[ 37.7076131999975672, -122.5123436999983966 ],
	[37.7835871999971715, -122.5247187000021967],
	[37.8151571999998453, -122.4798767000009008]]] }

	/// An extension to provide conversion to and from HSL (hue, saturation, lightness) colors.
	extension UIColor {

	/// The HSL (hue, saturation, lightness) components of a color.
	struct HSL: Hashable {

	/// The hue component of the color, in the range [0, 360°].
	var hue: CGFloat
	/// The saturation component of the color, in the range [0, 100%].
	var saturation: CGFloat

	[package]
	name = "demo-ssb-client"
	version = "0.1.0"
	authors = ["AljoschaMeyer <[email protected]>"]

	[dependencies]
	tokio = "0.1.3"
	futures = "0.1.18"

	sodiumoxide = "0.0.16"

	object DataFrameFunctions {

	final val TEMP_TABLE_PLACEHOLDER = "~tbl~"

	/** Executes a SQL statement on the dataframe.
	* Behind the scenes, it registers and cleans up a temporary table.
	*
	* @param df input dataframe
	* @param stmtTemplate SQL statement template that uses the value of
	* `TEMP_TABLE_PLACEHOLDER` for the table name.