Greg Rahn gregrahn

Recent versions of Cloudera's Impala added NDV, a "number of distinct values" aggregate function that uses the HyperLogLog algorithm to estimate this number, in parallel, in a fixed amount of space.

This can make a really, really big difference: in a large table I tested this on, which had roughly 100M unique values of mycolumn, using NDV(mycolumn) got me an approximate answer in 27 seconds, whereas the exact answer using count(distinct mycolumn) took ... well, I don't know how long, because I got tired of waiting for it after 45 minutes.

It's fun to note, though, that because of another recent addition to Impala's dialect of SQL, the fnv_hash function, you don't actually need to use NDV; instead, you can build HyperLogLog yourself from mathematical primitives.

HyperLogLog hashes each value it sees, and then assigns them to a bucket based on the low order bits of the hash. It's common to use 1024 buckets, so we can get the bucket by using a bitwise & with 1023:

select

Notes:

I've tried to break up in to separate pieces, but it's not always possible: e.g. knowledge of data structures and subsetting are tidy intertwined.
Level of Bloom's taxonomy listed in square brackets, e.g. http://bit.ly/15gqPEx. Few categories currently assess components higher in the taxonomy.

Programming R curriculum

Data structures

A friend asked me for a few pointers to interesting, mostly recent papers on data warehousing and "big data" database systems, with an eye towards real-world deployments. I figured I'd share the list. It's biased and rather incomplete but maybe of interest to someone. While many are obvious choices (I've omitted several, like MapReduce), I think there are a few underappreciated gems.

###Dataflow Engines:

Dryad--general-purpose distributed parallel dataflow engine
http://research.microsoft.com/en-us/projects/dryad/eurosys07.pdf

Spark--in memory dataflow
http://www.cs.berkeley.edu/~matei/papers/2012/nsdi_spark.pdf

	#! /usr/bin/env python3

	# This script is explained here: <https://sixcolors.com/post/2024/10/putting-my-solar-system-in-my-menu-bar/>
	# Adapted from [email protected] <https://github.com/grzegorz914/homebridge-enphase-envoy>

	# <xbar.title>Enphase Solar (Averages)</bitbar.title>
	# <xbar.version>v1.2</xbar.version>
	# <xbar.author>Jason Snell</xbar.author>
	# <xbar.author.github>jasonsnell</xbar.author.github>
	# <xbar.desc>Display local Enphase solar stats.</xbar.desc>

	#include <stdio.h>
	#include <stdint.h>
	// Philips Sonicare NFC Head Password calculation by @atc1441 Video manual: https://www.youtube.com/watch?v=EPytrn8i8sc
	uint16_t CRC16(uint16_t crc, uint8_t *buffer, int len) // Default CRC16 Algo
	{
	while(len--)
	{
	crc ^= *buffer++ << 8;
	int bits = 0;
	do

	#---------------- Plotting Daily Cumulative Cases of the Coronavirus----------------
	# Installing the most update version of the coronavirus
	# install.packages("devtools")
	devtools::install_github("RamiKrispin/coronavirus")
	data("coronavirus")


	# Reformat and aggregate the data to daily by country and type of case
	df_daily <- coronavirus %>%
	dplyr::group_by(date, type) %>%

	This guide assumes that you have already set up a Ubiquiti Unifi VPN following the guide:
	https://help.ubnt.com/hc/en-us/articles/115005445768-UniFi-L2TP-Remote-Access-VPN-with-USG-as-RADIUS-Server

	To configure a Linux machine to be able to connect remotely I followed these steps. This guide was written for Debian 8.

	- In Debian install the "xl2tpd" and "strongswan" packages.

	- Edit /etc/ipsec.conf to add the connection:

	conn YOURVPNCONNECTIONNAME

	<!--
	~ Copyright (c) 2014 Scaling Data. All rights reserved. This gist is licensed under the Apache Software License v2.0.
	-->

	<assembly
	xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.2 http://maven.apache.org/xsd/assembly-1.1.2.xsd">

	<id>dist</id>

	##### ##### ##### ##### ##### ##### ##### ##### ##### ##### ##### ##### ##### ##### ##### ##### #####

	### Shell script to download Oracle JDK / JRE / Java binaries from Oracle website using terminal / command / shell prompt using wget.
	### You can download all the binaries one-shot by just giving the BASE_URL.
	### Script might be useful if you need Oracle JDK on Amazon EC2 env.
	### Script is updated for every JDK release.

	### Features:-
	# 1. Resumes a broken / interrupted [previous] download, if any.
	# 2. Renames the file to a proper name with including platform info.

	# Compiled source #
	###################
	*.com
	*.class
	*.dll
	*.exe
	*.o
	*.so

	# Packages #