Skip to content

Instantly share code, notes, and snippets.

@alces
alces / pause_connectors.py
Created August 14, 2019 04:35
Pause or resume all connectors on a Kafka Connect cluster
#!/usr/bin/env python
CONNECT_ENDPOINT = 'http://10.0.0.24:8083'
import httplib
import json
import os
import sys
action = 'pause' if os.path.basename(sys.argv[0]).startswith('pause') else 'resume'
@alces
alces / curl_with_client_auth.sh
Created April 1, 2019 09:36
Provide SSL client authentication via curl command line
curl --cacert /path/to/ca-root.crt \
--cert /path/to/client.crt \
--key /path/to/client.key \
https://service.with.client.auth
@alces
alces / UrlConnectTest.java
Last active October 9, 2019 10:13
Java "one-liner" for testing SSL connections (truststores, client auth, etc)
import java.io.*;
import java.net.*;
/**
* Open URL passed as the first command-line argument and read the first line from it
*
* Example usage for testing client auth:
* java -Djavax.net.ssl.trustStore=/path/to/truststore.jks \
* -Djavax.net.ssl.trustStorePassword=myword \
* -Djavax.net.ssl.keyStore=/path/to/keystore.jks \
@alces
alces / kafka-jdbc-src-reset.sh
Created January 29, 2019 07:31
Reset value of incrementing field for Kafka JDBC source connector
#!/bin/bash
echo '["my-connector-name",{"protocol":"1","table":"my_db_name.my_table_name"}]@{"incrementing":5}' | kafka-console-producer --topic _connect-offsets --broker-list 127.0.0.1:9092 --property parse.key=true --property key.separator=@
# And after issuing this command, restart the connector's task (pausing/resuming the connector isn't enoght)
@alces
alces / words_count.pig
Created November 30, 2018 06:08
Count words' occurrences in a text file
data = load '$file' using TextLoader();
tokens = foreach data generate FLATTEN(TOKENIZE($0));
words = filter tokens by $0 MATCHES '[A-Za-z]+';
lowers = foreach words generate LOWER($0);
groups = group lowers by $0;
counts = foreach groups generate group, COUNT(lowers.$0);
by_count = order counts by $1 DESC;
store by_count into '$file-by-count';
by_word = order counts by $0 ASC;
store by_word into '$file-by-word';
@alces
alces / jenkinsfile_pr_check.md
Created September 26, 2018 06:35
Using Jenkins pipelines as Github pull request cheks
  1. in Jenkins set up a multibranch pipeline with Branch source of type Github (under it, set up endpoint, credentials, repo name, etc.);
  2. in Github go to the repository Settings and add the user chosen on the previous step to the repository's colaborators;
  3. go to the Hooks menu and add a webhook pointing to <your-jenkins-host>/github-webhook/ and select pull request event under Let me select individual events option;
  4. create a pull request - after that Jenkins should automatically start a build;
  5. go to Branches menu under Settings and add the target branch to Protected branches;
  6. choose Require status checks to pass before merging and continuous-integration/jenkins/pr-merge under it
  7. commit a change into the pull request and see the Jenkins build result on the page.
@alces
alces / wrest_dict.py
Created April 4, 2018 11:47
Wrest a dictionary of lists (e.g., turn a dict of nodes as keys and lists of groups as values into a dict of groups as keys and lists of nodes as values)
def wrest_dict(orig):
pairs = [(g, n) for n in orig.keys() for g in orig[n]]
nodes4group = lamdba g: sorted(n[1] for n in pairs if n[0] == g)
return {g[0]: nodes4group(g[0]) for g in pairs}
@alces
alces / list_reduce.go
Created November 29, 2017 11:26
Adding Reduce method to List class
package main
import "container/list"
type MyList struct {
*list.List
}
func (l *MyList) Reduce(f func(interface{}, interface{}) interface{}) interface{} {
s := l.Front()
@alces
alces / list_map.go
Last active November 27, 2017 14:12
Adding Map method to List class
package main
import "container/list"
type MyList struct {
*list.List
}
func (l *MyList) Map(f func(interface{}) interface{}) *MyList {
nl := &MyList {list.New()}
@alces
alces / list_filter.go
Last active November 27, 2017 14:12
Adding Filter method to List
package main
import "container/list"
type MyList struct {
*list.List
}
func (l *MyList) Filter(f func(interface{}) bool) *MyList {
nl := &MyList{list.New()}