WooodHead

How we incorporate next and cloudfront (2018-04-21)

Feel free to contact me at [email protected] or tweet at me @statisticsftw

This is a rough outline of how we utilize next.js and S3/Cloudfront. Hope it helps!

It assumes some knowledge of AWS.

Goals

This is a collection of scripts to restore a full GHTorrent MongoDB database from the dumps available at http://ghtorrent-downloads.ewi.tudelft.nl.

To do the restore:

Open a MongoDB terminal and run the createCollections.js script to create the necessary collections. You can block_compressor to either snappy or zlib to make your databases compressed. I am using none here, as I am using compression at the filesystem level.
Run restore-cummulative-dumps.sh to restore the cummulative dumps. Wait 3-4 days.

OS: Ubuntu 18.04 Apache/2.4.18 1.0.2g-1ubuntu4.10
Aim: to install Aria2 with WebUI and secure Token.
IP Addr of your Aria2 server is 192.168.0.111
Your local IP network is 192.168.0.0/24

Aria 2

1. Installation

Install aria2 package:

AMAZON API GATEWAY REGIONS

US East (Ohio) us-east-2 apigateway.us-east-2.amazonaws.com HTTPS ZOJJZC49E0EPZ

US East (N. Virginia) us-east-1 apigateway.us-east-1.amazonaws.com HTTPS Z1UJRXOUMOOFQ8

US West (N. California) us-west-1 apigateway.us-west-1.amazonaws.com HTTPS Z2MUQ32089INYE

US West (Oregon) us-west-2 apigateway.us-west-2.amazonaws.com HTTPS Z2OJLYMUO9EFXC

	const fs = require('fs');
	const puppeteer = require('puppeteer');

	const GIFEncoder = require('gifencoder');
	const PNG = require('png-js');

	function decode(png) {
	return new Promise(r => {png.decode(pixels => r(pixels))});
	}

	"""
	Convert YouTube subtitles(vtt) to human readable text.

	Download only subtitles from YouTube with youtube-dl:
	youtube-dl --skip-download --convert-subs vtt <video_url>

	Note that default subtitle format provided by YouTube is ass, which is hard
	to process with simple regex. Luckily youtube-dl can convert ass to vtt, which
	is easier to process.

	deb http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse
	deb http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse
	deb http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse
	deb http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse
	deb http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
	deb-src http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse
	deb-src http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse
	deb-src http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse
	deb-src http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse
	deb-src http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse

	#!/bin/bash

	set -e
	#
	# Create a cluster of workers on exoscale, sharing a common data folder using a NFS exports.
	# (not battle tested)
	#

	# BASED ON THE ENTIRE WEB and those sources
	# https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nfs-mount-on-ubuntu-16-04

	query{
	__type(name:"Repository") {
	fields {
	name
	description
	type {
	kind
	name
	description
	}

	/*
	* This script will create base62_encode() and base62_decode() in the current schema of a postgresql database.
	* Give it a star if you find it useful.
	*/

	CREATE OR REPLACE FUNCTION base62_encode( long_number bigint )
	RETURNS text
	AS $BODY$
	/*
	* base62_encode()