onefoursix’s gists

onefoursix / CreateSnowflakeTablesFromOracle.json

Last active March 12, 2022 04:59

CreateSnowflakeTablesFromOracle.json

	{
	"pipelineConfig": {
	"schemaVersion" : 6,
	"version" : 32,
	"pipelineId" : "CreateSnowflakeTablesFromOraclecopyc6377e36-fbd3-4892-8bea-3dbd18c66c10",
	"uuid" : "18c1e180-b3cd-47ce-9d9d-7fce22b120fe",
	"title" : "CreateSnowflakeTablesFromOracle",
	"description" : "",
	"configuration" : [ {
	"name" : "executionMode",

onefoursix / sdc-with-hpa.yaml

Created January 12, 2022 18:56

Example manifest for an SDC Deployment with an HPA

	apiVersion: 1
	kind: List
	items:
	- apiVersion: apps/v1
	kind: Deployment
	metadata:
	name: sdc
	namespace: ns1
	labels:
	app: sdc-microservice

onefoursix / SFTP_to_SFTP.json

Created December 17, 2021 19:53

SDC pipeline that uses Whole File Format to move files between SFTP sites with a date filter

	{
	"pipelineConfig" : {
	"schemaVersion" : 6,
	"version" : 22,
	"pipelineId" : "SFTPtoSFc2a6666c-1b01-4521-807f-6c0b71a826fc",
	"title" : "SFTP to SFTP",
	"description" : "",
	"uuid" : "71aaf3ad-b439-4a3d-88c7-0926733192b2",
	"configuration" : [ {
	"name" : "executionMode",

onefoursix / get-job-metrics.json

Created November 2, 2021 15:40

StreamSets Pipeline that captures Job metrics

	{
	"pipelineConfig": {
	"schemaVersion" : 6,
	"version" : 22,
	"pipelineId" : "GetJobMec005d568-8e20-4797-96fa-1074ff68dd19",
	"uuid" : "aff7822e-10e1-49ac-9fb8-e6f0aaf0d2d8",
	"title" : "Get Job Metrics",
	"description" : "",
	"configuration" : [ {
	"name" : "executionMode",

onefoursix / build.sh

Last active November 5, 2022 17:10

builds an StreamSets Data Collector Docker image with a set of standard and enterprise stage libraries

	#!/usr/bin/env bash

	# This script builds a custom Docker image that extends the base SDC image
	# It downloads a set of SDC stage libs and enterprise stage libs to a local directory.
	# The Dockerfile copies those libs into the SDC image as well as a custom sdc.properties file
	# The last line assumes you have executed a $ docker login for the push to work

	# Use this build script with a Dockerfile that has this content:
	# ARG SDC_VERSION=4.1.0
	# FROM streamsets/datacollector:${SDC_VERSION}

onefoursix / DynamicDataValidation.json

Last active October 5, 2021 00:11

SDC Pipeline that validates data types for single-field records by lookups into a RDBMS table with caching

	{
	"pipelineConfig": {
	"schemaVersion" : 6,
	"version" : 22,
	"pipelineId" : "DynamicDae95dbfe5-59f3-4f1c-a1cd-777ac6d26440",
	"uuid" : "05e2119a-6a8a-452f-ba86-dbacacffaa97",
	"title" : "Dynamic Data Validation",
	"description" : "",
	"configuration" : [ {
	"name" : "executionMode",

onefoursix / create-sdc-deployment.sh

Last active September 16, 2021 19:35

StreamSets REST API Example of creating an SDC Deployment

	#!/usr/bin/env bash

	# This script creates an SDC Deployment based on the manifest file pointed to
	# by the DEPLOYMENT_MANIFEST_FILE environment variable set below.

	# The deployment manifest file should use '\n' (without the quotes) rather than new line characters
	# and all quote characters should be escaped.

	# Here is an example deployment.yaml file that can be used:
	# https://gist.githubusercontent.com/onefoursix/f5a1084079f5b80c97c3418c9c267dca/raw/ac2dce6f69beb74d4ee6951d4cc83ec5e39c141e/deployment.yaml

onefoursix / deployment.yaml

Created September 16, 2021 19:23

SDC deployment yaml that can be used when creating SDC Deployment using the Control Hub REST API

apiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: datacollector-deployment\n  namespace: ns1\nspec:\n  replicas: 1\n  selector:\n    matchLabels:\n      app: datacollector-deployment\n  template:\n    metadata:\n      labels:\n        app: datacollector-deployment\n    spec:\n      containers:\n      - name: datacollector\n        image: streamsets/datacollector:latest\n\n

onefoursix / CreateSnowflakeTablesFromSQLServer.json

Created August 25, 2021 15:08

CreateSnowflakeTablesFromSQLServer.json

	{
	"pipelineConfig": {
	"schemaVersion" : 6,
	"version" : 23,
	"pipelineId" : "CreateSnowflakeTablesFromSQLServer73648919-02ed-4f43-a9fa-43d6579794f0",
	"uuid" : "18c1e180-b3cd-47ce-9d9d-7fce22b120fe",
	"title" : "CreateSnowflakeTablesFromSQLServer",
	"description" : "",
	"configuration" : [ {
	"name" : "executionMode",

onefoursix / sdc.yaml

Created August 18, 2021 00:10

SDC K8s Deployment manifest with path-based routing for ingress-nginx

	apiVersion: 1
	kind: List
	items:
	- apiVersion: apps/v1
	kind: Deployment
	metadata:
	name: sdc1
	namespace: ns1
	labels:
	app: sdc1

Mark Brooks onefoursix