Skip to content

Instantly share code, notes, and snippets.

View beckyconning's full-sized avatar

Becky Conning beckyconning

View GitHub Profile

What it does

It streams all tables (excluding archived ones) into Google Big Query.

Scheduling this in the cloud

This will start up a server which will stream data from data sources like Atlas, MongoDB, S3 and Azure Blob Storage into Google Big Query on a schedule e.g. every day at midnight or every hour. You don't need anything other than this server. When this process is complete it will shut down the server so if you want to add more tables you'll need to start it back up. Note that if the server is on when its scheduled to begin the etl process it will restart.

  • https://console.cloud.google.com/compute/instancesAdd Create a fast high memory Google Cloud VM Instance called slamdata-bigquery with access scopes set to "Allow full access to all Cloud API" and in the Management, security, disks, networking, sole tenancy" section, in the "Networking" section add the network tag my-ip-web then edit the network interface and create an external ip address called external, make a note of this ip ad

Before |Column Number|Column String|Column Number|Column DateTime|

A B C D

After |Column Number|Column String|Column Number|Column DateTime| |A|B|B|C|D|D| ||Self Join A|Self Join C||Self Join A|Self Join C|

{-# LANGUAGE RankNTypes #-}
import System.IO
import System.Console.ANSI
import Control.Monad.Loops
import Data.Maybe
import Data.Foldable (all)
import Control.Lens
data Mark = X | O deriving (Eq, Show)
import System.IO
import System.Console.ANSI
import Control.Monad.Loops
import Data.Maybe
import Data.Foldable (all)
data Mark = X | O deriving (Eq, Show)
data Status = InProgress | Won | Drawn
type Cell = Maybe Mark
module Main where
import Prelude
import Control.Monad.Eff (Eff)
import Control.Monad.Eff.Console (CONSOLE, log)
import Control.Monad.Eff.Random (RANDOM, random)
import Control.Monad.Loops (iterateUntilM)
import Data.Array as Array
import Data.Either (Either)
document.querySelector("body").appendChild(document.createElement("button"));
document.querySelector("button").addEventListener("mouseover", function() {
document.querySelector("button").appendChild(document.createTextNode("😎"));
});
document.querySelector("button")
.dispatchEvent(new Event('mouseover', { 'bubbles': true, 'cancelable': true }));
parseRow =
many1Till (parseNumber >>= \n -> char "," *> pure n)
>>= \features -> manyTil eol
>>= \label -> eol
*> pure (Iris { features, label })
5.1,3.5,1.4,0.2,Iris-setosa
4.9,3.0,1.4,0.2,Iris-setosa
4.7,3.2,1.3,0.2,Iris-setosa
4.6,3.1,1.5,0.2,Iris-setosa
5.0,3.6,1.4,0.2,Iris-setosa
5.4,3.9,1.7,0.4,Iris-setosa
4.6,3.4,1.4,0.3,Iris-setosa
5.0,3.4,1.5,0.2,Iris-setosa
4.4,2.9,1.4,0.2,Iris-setosa
4.9,3.1,1.5,0.1,Iris-setosa
module Main where
import Prelude ((++), putStrLn)
data Unit = Unit
data TrueFalse = True | False
data List a = Nil | Cons a (List a)
db.getCollection("__quasar\\.tmp_b6dfc67c_44").aggregate(↵[↵{↵
"$project": {↵
"s": "$$ROOT",
↵"o": {↵
"$objectToArray": {↵
"$cond": [↵{↵
"$and": [↵{
"$lte": [{
"$literal": {}
}, "$$ROOT"]