Gilson Filho gilsondev

The Problem

The documentation for how to deploy a pipeline with extra, non-PyPi, pure Python packages on GCP is missing some detail. This gist shows how to package and deploy an external pure-Python, non-PyPi dependency to a managed dataflow pipeline on GCP.

TL;DR: You external package needs to be a python (source/binary) distro properly packaged and shipped alongside your pipeline. It is not enough to only specify a tar file with a setup.py.

Preparing the External Package

Your external package must have a proper setup.py. What follow is an example setup.py for our ETL package. This is used to package version 1.1.1 of the etl library. The library requires 3 native PyPi packages to run. These are specified in the install_requires field. This package also ships with custom external JSON data, declared in the package_data section. Last, the setuptools.find_packages function searches for all available packages and returns that

Spark - High availability

Components in play

As a reminder, here are the components in play to run an application:

The cluster:
- Spark Master: coordinates the resources
- Spark Workers: offer resources to run the applications
The application:

Using py.test is great and the support for test fixtures is pretty awesome. However, in order to share your fixtures across your entire module, py.test suggests you define all your fixtures within one single conftest.py file. This is impractical if you have a large quantity of fixtures -- for better organization and readibility, you would much rather define your fixtures across multiple, well-named files. But how do you do that? ...No one on the internet seemed to know.

Turns out, however, you can define fixtures in individual files like this:

tests/fixtures/add.py

import pytest

@pytest.fixture

	First you have to ensure the Java JRE is installed on your system.
	Remember the java plugin will only work on Firefox. If you prefer to install from fedora/rhel repos:

	sudo dnf (yum) install java-1.8.0-openjdk icedtea-web

	download the snx script installer from the Checkpoint VPN page:
	https://<your company vpn ip addr>/sslvpn/SNX/INSTALL/snx_install.sh

	run int as root to install

	# taken from https://github.com/niteshpatel/habitica-slack/
	language: python

	python:
	- "2.7"

	env:
	- DJANGO=1.10.1

	install:

	#!/usr/bin/python
	"""
	Copyright (C) 2016 Jeffrey S. McAnarney

	This program is free software; you can redistribute it and/or
	modify it under the terms of the GNU General Public License
	as published by the Free Software Foundation; either version 2
	of the License, or (at your option) any later version.

	This program is distributed in the hope that it will be useful,

	function zip(a, b) {
	var arr = [];
	for (var key in a) arr.push([a[key], b[key]]);
	return arr;
	}

	console.log(zip('foo', 'bar'));
	//=> [ [ 'f', 'b' ], [ 'o', 'a' ], [ 'o', 'r' ] ]

	console.log(zip(['a', 'b', 'c'], ['x', 'y', 'z']));

	'''
	Spider for IMDb
	- Retrieve most popular movies & TV series with rating of 8.0 and above that have at least 5 award nominations
	- Crawl next pages recursively
	- Follow the details pages of scraped films to retrieve more information of each film
	'''

	from scrapy.contrib.spiders import CrawlSpider, Rule
	from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
	from scrapy.selector import Selector

	docker run --name gitlab-postgresql -d \
	--env 'DB_NAME=gitlabhq_production' \
	--env 'DB_USER=gitlab' \
	--env 'DB_PASS=DBPASSWORD' \
	--volume /local/db/location/postgresql:/var/lib/postgresql \
	sameersbn/postgresql

	docker run --name gitlab-redis -d \
	--volume /local/cache/location/redis:/var/lib/redis \
	sameersbn/redis

	mkdir temp && cd temp

	# for linux 'amd64' architecture install those packages:
	sudo apt-get install libx11-6:i386 libpam0g:i386 libstdc++5:i386 lib32z1 lib32ncurses5 lib32bz2-1.0

	wget https://vpnportal.aktifbank.com.tr/SNX/INSTALL/snx_install.sh
	sudo ./snx_install.sh

	cd .. && rm -rf temp/