Andrew Stewart andrewcstewart

Using GitHub Actions to run dbt

This example shows you how to use GitHub Actions to run dbt against BigQuery.

Follow the instructions on getdbt.com for installing and initializing a dbt project.
Copy this action (dbt.yml) into the workflows directory.
```
 mkdir .github
 mkdir .github/workflows
```

cp ~/Downloads/dbt.yml .github/workflows/

Why build Apache Arrow from source on ARM?

Apache Arrow is an in-memory data structure used in several projects. It's python module can be used to save what's on the memory to the disk via python code, commonly used in the Machine Learning projects. With low RAM, ARM devices can make use of it but there seems to be an configuration error with the packaged binaries as of version 0.15.1 and so we're forced to build and install from the source.

The installation build steps are based on official guidelines but modified for ARM and has taken clues from building Ray for ARM.

My setup

I'm using Nvidia Jetson nano.

Quad-core ARM® Cortex®-A57 MPCore processor

EN685.648 Data Science

This course requires knowledge of Python and SQL (the requirement is listed in the course description). If you do not know Python, you will not do well and the course will be that much harder.

Infrastructure

You are very strongly encouraged to use a computer upon which you have administrator/superuser privileges. I cannot help you with problems associated with the installation of software and libraries.
You are encouraged to use "Unix"-style operating system (MacOS or Linux flavor) either directly or in a virual environment (Docker or VirtualBox). It's not required but you should be multi-hosted when it comes to OSes and the examples of command line utilities will be in 'Nix. This is not necessary to excel in the class but it is helpful. Many platforms are built on Linux and you should learn to use it.
Ideally, you should have your environment up and running before the semester starts but no later than the 2nd day of class (that first Friday). There is a test a

	ARG MELTANO_IMAGE=meltano/meltano:latest
	FROM $MELTANO_IMAGE

	WORKDIR /project

	# Install chamber
	RUN curl -s https://packagecloud.io/install/repositories/segment/chamber/script.deb.sh \| /bin/bash \
	&& apt-get install -y chamber \
	&& apt-get clean

	brew install pandoc
	brew tap homebrew/cask
	brew install --cask basictex
	eval "$(/usr/libexec/path_helper)"
	# Update $PATH to include `/usr/local/texlive/2022basic/bin/universal-darwin`
	sudo tlmgr update --self
	sudo tlmgr install texliveonfly
	sudo tlmgr install xelatex
	sudo tlmgr install adjustbox
	sudo tlmgr install tcolorbox

	xcode-select --install

	if [ ! -x /opt/homebrew/bin/brew ]; then
	/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
	fi

	brew tap 'homebrew/bundle'
	brew tap 'homebrew/services'

	brew install automake clang-format cmake icu4c libressl libxml2 openssl pkg-config readline sqlite yajl

	#!/usr/bin/env python
	"""

	Sample call:
	python synapse_galaxy_sync.py <api_key> <api_url> syn12345

	NOTE: The upload method used requires the data library filesystem upload allow_library_path_paste
	"""
	import os
	import shutil

	>>> from pandas import DataFrame
	>>> from sklearn.feature_extraction.text import CountVectorizer
	>>> docs = ["You can catch more flies with honey than you can with vinegar.",
	... "You can lead a horse to water, but you can't make him drink."]
	>>> vect = CountVectorizer(min_df=0., max_df=1.0)
	>>> X = vect.fit_transform(docs)
	>>> print(DataFrame(X.A, columns=vect.get_feature_names()).to_string())
	but can catch drink flies him honey horse lead make more than to vinegar water with you
	0 0 2 1 0 1 0 1 0 0 0 1 1 0 1 0 2 2
	1 1 2 0 1 0 1 0 1 1 1 0 0 1 0 1 0 2

	python ../nbconvert/nbconvert.py -f reveal reveal-test.ipynb
	reveal-test_slides.html

	-- Find the note in omnifocus for the current project / folder
	-- (or create a new one) and display it.

	-- Created by Mark Fowler ([email protected])
	-- Help from http://veritrope.com/code/search-evernote-for-notes-containing-link-to-selected-omnifocus-item/
	-- on getting the name / url for the current omnifocus item


	tell front window of application "OmniFocus"
	try