onefoursix’s gists

onefoursix / jython-origin-with-offset.txt

Created March 10, 2021 00:24

	try:
	sdc.importLock()
	from datetime import datetime
	finally:
	sdc.importUnlock()

	date_time_format = '%Y-%m-%d %H:%M:%S'

	## Offsets will be stored with the key 'offsetKey' as a String with a value like this:
	## startDate=1609459200&endDate=1609462800

onefoursix / SQLServerCDCHelper_v5.json

Last active March 4, 2021 21:05

SQLServer CDC Helper v5

	{
	"pipelineConfig": {
	"schemaVersion" : 6,
	"version" : 22,
	"pipelineId" : "SQLServerCDCOffsetHelper793fd8c6-8568-4291-9c6f-dbb1bb63c38a",
	"uuid" : "3d9ec1a4-ae19-484b-9c7b-5b8e606d546e",
	"title" : "SQLServer-CDC-Offset-Helper",
	"description" : "",
	"configuration" : [ {
	"name" : "executionMode",

onefoursix / set-component-ids.sh

Last active February 23, 2021 22:55

A utility to initialize config files for HA instances of StreamSets Control Hub

	#!/usr/bin/env bash

	## set-component-ids.sh
	## Run this script in the $DPM_CONF dir
	##
	## Pass in values to replace old componentId suffixes like "000" with new values like "001"
	##
	## Usage: ./set-component-ids.sh <OLD_SUFFIX> <NEW_SUFFIX>
	## Example: ./set-component-ids.sh 000 001

onefoursix / SQLServer_CDC_Offset_Helper_JDBC_Audit.json

Created February 23, 2021 04:34

An SDC orchestration pipeline that corrects SQLServer CDC Offsets after CDC clean up processes

	{
	"pipelineConfig": {
	"schemaVersion" : 6,
	"version" : 22,
	"pipelineId" : "SQLServerCd7c461df-919f-4790-bbd4-91189c05ed20",
	"uuid" : "a6c1d8b2-3258-418d-8e73-22437f011a31",
	"title" : "SQLServer_CDC_Offset_Helper_JDBC_Audit",
	"description" : "",
	"configuration" : [ {
	"name" : "executionMode",

onefoursix / ngnix.conf

Created February 15, 2021 20:47

ngnix.conf for single instance Control Hub

	worker_processes 1;
	events {
	worker_connections 1024;
	}
	http {
	include mime.types;
	default_type application/octet-stream;
	server {
	listen 443 ssl;
	server_name schlb.onefoursix.com;

onefoursix / restart-streamsets-job.sh

Last active March 3, 2021 20:51

bash script to restart StreamSets Job that waits for Job to deactivate before starting it again

	#!/usr/bin/env bash

	## restart-streamsets-job.sh

	## A bash script that stops a StreamSets Job, waits for the Job to transition
	## to an INACTIVE state and then starts the Job. If the Job is already in an
	## INACTIVE state it will simply be started.

	## Dependencies:

onefoursix / update-jdbc.sh

Created February 9, 2021 22:41

Util for SCH install

	declare -a dbs=("connection" "jobrunner" "messaging" "notification" "pipelinestore" "policy" "provisioning" "reporting" "scheduler" "sdp_classification" "security" "sla" "timeseries" "topology" "dynamic_preview")

	for i in "${dbs[@]}"
	do
	sed -i "s/^db.openjpa.ConnectionURL=/db.openjpa.ConnectionURL=jdbc\:mysql\:\/\/localhost:3306\/${i}?useSSL=false/" ${i}-app.properties
	sed -i "s/^db.openjpa.ConnectionUserName=/db.openjpa.ConnectionUserName=sch/" ${i}-app.properties
	sed -i "s/^db.openjpa.ConnectionPassword=/db.openjpa.ConnectionPassword=Mysql12345!/" ${i}-app.properties
	done

onefoursix / sdc.yaml

Created February 9, 2021 17:58

Deployment manifest for SDC that downloads stage libs at deployment time

onefoursix / sdc-stage-libs-configmap.yaml

Created February 9, 2021 16:41

StreamSets Stage Lib configmap for k8s deployment

	apiVersion: v1
	kind: ConfigMap
	metadata:
	name: sdc-stage-libs-list
	data:
	sdc-stage-libs: \|
	streamsets-datacollector-aws-lib
	streamsets-datacollector-basic-lib
	streamsets-datacollector-bigtable-lib
	streamsets-datacollector-dataformats-lib

onefoursix / test_azure_stages_with_modified_tests.py

Last active January 4, 2021 03:14

Modified STF Tests for Event Hub WebSockets Support

	# Copyright 2019 StreamSets Inc.
	#
	# Licensed under the Apache License, Version 2.0 (the "License");
	# you may not use this file except in compliance with the License.
	# You may obtain a copy of the License at
	#
	# http://www.apache.org/licenses/LICENSE-2.0
	#
	# Unless required by applicable law or agreed to in writing, software
	# distributed under the License is distributed on an "AS IS" BASIS,

Mark Brooks onefoursix