Andrew Kaiser andykais
andykais
/ Dockerfile
Created
August 1, 2017 15:49
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| FROM node:8-alpine | |
| COPY entrypoint.sh /usr/bin/entrypoint.sh | |
| # create npmrc with enviroment variable | |
| RUN echo "\$NPM_TOKEN" > /root/.npmrc | |
| # entrypoint script performs envsubst on npmrc | |
| RUN printf '#!/bin/sh\n\ | |
| sed -i "s~\$NPM_TOKEN~$NPM_TOKEN~" /root/.npmrc\n\ | |
| $@' > /usr/bin/entrypoint.sh \ |
andykais
/ observable-queue.js
Last active
June 12, 2018 22:36
continually accept values and perform some async task on them until there are no more to add
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| const EventEmitter = require('events') | |
| const { fromEvent } = require('rxjs') | |
| const { tap, takeUntil, mergeMap } = require('rxjs/operators') | |
| class Queuer { | |
| constructor({ maxConcurrent = 1, debug = false } = {}) { | |
| // node event emitter | |
| const queueEmitter = new EventEmitter() | |
| // event emitter to keep track of each task finishing | |
| const taskEmitter = new EventEmitter() |
andykais
/ .travis.yml
Created
July 5, 2018 23:44
travis config to test against multiple node versions but lint only once
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| language: node_js | |
| stages: | |
| - test | |
| - lint | |
| cache: | |
| directories: | |
| - $HOME/.npm |
andykais
/ make-url-from-zipped-string.js
Created
July 25, 2018 17:25
store (compressed) text input on a static page in the url
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import LZUTF8 from 'lzutf8' | |
| const config = { | |
| // scrape... | |
| } | |
| // on config text box input | |
| const configUrlParam = LZUTF8.compress(config, { outputEncoding: "Base64" }) | |
| history.replaceState(null, '', "?config=" + configUrlParam) |
andykais
/ .bash_aliases.sh
Created
August 7, 2018 22:51
test a npm package locally in as close a manner to a published package as possible
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| function test_package_local() { | |
| local cwd="$PWD" | |
| local package_folder=$1 | |
| cd $1 | |
| local tarball=$(npm pack) | |
| local fullpath="$package_folder/$tarball" | |
| npm install "$fullpath" | |
| cd "$cwd" | |
| } |
andykais
/ login.config.js
Last active
September 18, 2018 16:28
Possible new spec for passing vars like login to scraper. Essentially passing one scraper as input variables to another scraper. New syntax includes limiting the number of parsed values ("max"), parsing headers ("expect": "header") and "asInput" clause,
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "input": ["username","password"], | |
| "scrape": { | |
| "download": { | |
| "urlTemplate": "http://example-site.com/login", | |
| "headers": { | |
| "username": "{username}", | |
| "password": "{password" | |
| }, | |
| "parse": { |
andykais
/ example-defs-scraper.config.yml
Created
December 5, 2018 03:54
separate scraper-step definitions from the downloading structure for readability, still will compile down to inline structure: https://gist.github.com/andykais/04a02b61bb3b6d92aa3388c45ea816bd
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| input: 'username' | |
| defs: | |
| - name: 'home' | |
| download: 'https://ifunny.co/user/{{ username }}' | |
| parse: | |
| name: 'batch-id' | |
| selector: '.stream__item:first-child' | |
| attribute: 'data-next' | |
| - name: 'gallery' |
andykais
/ placeholder magic.js
Last active
December 7, 2018 22:08
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| // index.js | |
| const objectPath = Symbol("object-path"); | |
| const objectCrawler = Symbol("object-crawler"); | |
| const handler = { | |
| get: (object, prop) => { | |
| if (prop === objectPath) { | |
| return object[objectPath]; | |
| } else if (String(prop) === "Symbol(util.inspect.custom)") { | |
| return object; |
andykais
/ example-log-transport.js
Created
January 4, 2019 22:08
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import * as Rx from 'rxjs' | |
| import * as ops from 'rxjs/operators' | |
| import fs from 'fs' | |
| import VError from 'verror' | |
| /** | |
| * for now, there can only be one transport at a time | |
| * the cli will have to handle an option to directly log to terminal while also including a | |
| * "terminal ui" output default (progress bars, queued, in progress, completed, etc) | |
| * this can be accomplished by log options being done directly as command line args | |
| */ |
andykais
/ api-proposal.js
Last active
January 16, 2019 01:48
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| // current design | |
| import ScrapePages from '../src' | |
| import { normalizeConfig } from '../lib/normalize-config' | |
| const config = { scrape: {} } | |
| const options = { | |
| input: { | |
| username: 'bob' | |
| }, | |
| optionsEach: { | |
| gallery: { |