onefoursix’s gists

onefoursix / sdc.jython

Created August 11, 2021 21:00

Jython example for SDC Scripting Origin

	## Example Scripting Origin to call a REST API with pagination and offset handling

	# Imports
	try:
	sdc.importLock()
	import sys

	## Set this path to where we can load the Requests module
	sys.path.append('/usr/local/lib/python3.9/site-packages')

onefoursix / OracleCDCtoBigQuerywithSoftDeletes.json

Last active July 30, 2021 15:47

StreamSets Data Collector pipeline Oracle CDC to Oracle BigQuery with Soft Deletes

	{
	"pipelineConfig": {
	"schemaVersion" : 6,
	"version" : 23,
	"pipelineId" : "OracleCDCtoBigQuerywithSoftDeletes97c001d2-9e55-4aae-9cbe-fb047069f73c",
	"uuid" : "28b9babb-44b6-4952-8a4d-692d45f31dfa",
	"title" : "Oracle CDC to BigQuery with Soft Deletes",
	"description" : "",
	"configuration" : [ {
	"name" : "executionMode",

onefoursix / Scripting-Origin-REST-API-with-Pagination-and-Offset-Handling.json

Last active July 23, 2021 02:38

StreamSets Summer 21 Example: Pipeline with a scripting origin that reads a REST API with custom pagination and offset management

	{
	"pipelineConfig": {
	"schemaVersion" : 6,
	"version" : 23,
	"pipelineId" : "ScriptingOriginRESTAPIwithPaginationandOffsetHandling9d21bcdc-ba2b-4c9a-b3c9-29f099d6a5cb",
	"uuid" : "f808cb09-dbe5-4350-a81b-e88504dc513a",
	"title" : "Scripting Origin REST API with Pagination and Offset Handling",
	"description" : "",
	"configuration" : [ {
	"name" : "executionMode",

onefoursix / pom.xml

Last active July 4, 2021 18:24

pom.xml to package MetaMind/quickstart Java client into a jar without runtime checks

	<?xml version="1.0" encoding="UTF-8"?>
	<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<groupId>com.salesforce.ps</groupId>
	<artifactId>quickstart-java</artifactId>
	<version>1.0-SNAPSHOT</version>
	<name>quickstart-java</name>
	<properties>
	<dependencyCheck.skip>true</dependencyCheck.skip>
	<dependencyCheck.failBuildOnCVSS>9</dependencyCheck.failBuildOnCVSS>

onefoursix / eks-alb-sdc-svc-ingress.yaml

Created July 2, 2021 06:32

	apiVersion: 1
	kind: List
	items:
	- apiVersion: apps/v1
	kind: Deployment
	metadata:
	name: sdc
	namespace: ns1
	labels:
	app: sdc

onefoursix / custom_control_hub_notification_handler.jy

Last active June 28, 2021 14:53

	import json
	import time

	# Conver the pipeline parameter JOB_STATUS_CHANGE_SUPPRESS_DUPES_MINUTES from minutes to seconds
	SUPPRESS_DUPE_NOTIFICATIONS_SECONDS = ${JOB_STATUS_CHANGE_SUPPRESS_DUPES_MINUTES} * 60

	for record in sdc.records:
	try:

	# Assume we will forward this notification to the endpoint targets

onefoursix / sdc-with-runAsUser.yaml

Last active June 16, 2021 02:50

SDC Deployment YAML with a runAsUser ID to prevent issue with PodSecurityPolicy "'MustRunAsNonRoot'"

	apiVersion: apps/v1
	kind: Deployment
	metadata:
	name: sdc
	labels:
	app : sdc
	spec:
	replicas: 1
	selector:
	matchLabels:

onefoursix / psp.yaml

Created June 16, 2021 02:21

Pod Security Policy for Minikube

	---
	apiVersion: policy/v1beta1
	kind: PodSecurityPolicy
	metadata:
	name: privileged
	annotations:
	seccomp.security.alpha.kubernetes.io/allowedProfileNames: "*"
	labels:
	addonmanager.kubernetes.io/mode: EnsureExists
	spec:

onefoursix / restart-streamsets-oracle-cdc-job.sh

Created March 29, 2021 03:55

A bash script that stops a StreamSets Job for an Oracle CDC pipeline, waits for the Job to transition to an INACTIVE state and then starts the Job with the Runtime Parameter ORACLE_CDC_DICTIONARY_SOURCE set to either "DICT_FROM_ONLINE_CATALOG" or "DICT_FROM_REDO_LOGS"

	#!/usr/bin/env bash

	## restart-streamsets-oracle-cdc-job.sh

	## A bash script that stops a StreamSets Job for an Oracle CDC pipeline, waits for the Job
	## to transition to an INACTIVE state and then starts the Job with the Runtime Parameter
	## ORACLE_CDC_DICTIONARY_SOURCE set to either "DICT_FROM_ONLINE_CATALOG" or "DICT_FROM_REDO_LOGS"
	##
	## If the Job is currently in an INACTIVE state, the Job is started immediately
	##

onefoursix / jython-origin-REST-API-with-pagination.json

Last active March 17, 2021 07:22

an example StreamSets pipeline that uses a Jython Scripting Origin to call a REST API with custom pagination

	{
	"pipelineConfig": {
	"schemaVersion" : 6,
	"version" : 22,
	"pipelineId" : "JythonOri974259df-0475-4bd1-809a-0d70115aaa0b",
	"uuid" : "7223c35c-e891-4af1-b205-fffffe7ea275",
	"title" : "Jython Origin REST API Call with Pagination",
	"description" : "",
	"configuration" : [ {
	"name" : "executionMode",

Mark Brooks onefoursix