Adam Kocoloski kocolosk

Heterogeneous Schema Discovery

Moving documents from Cloudant into dashDB means transforming JSON documents into relational records. To build the correct relational representation, we first have to describe the schema of the JSON document itself. That is a trivial process for a single document but becomes very complex if many JSON documents have to be mapped to many relational records while still fitting a limited set of tables.

The Schema Discovery Process (SDP described in this playbook) has been designed to find such a set of tables suitable to hold as many (ideally all) JSON documents in a Cloudant database. The question is how to compute this set of tables?

The answer is again fairly simple if all JSON documents in a Cloudant database implement the same homogenous schema (i.e. have the same attributes, values of the same type, the same levels of nested objects and arrays). The answer is more complex if the documents have heterogenous schemata.

	#!/usr/bin/env python3

	import json
	import random
	import requests
	import threading

	URL = 'http://127.0.0.1:15984/snapshots-for-indexing'
	USER = 'adm'
	PASS = 'pass'

	function (keys, values, rereduce) {
	var topTenPlusBoundaryKeys = function(partitions) {
	// preserve boundary keys because we may not have the correct count for them yet
	// not that it matters, but the array is reversed so these labels are correct
	var first = partitions.pop();
	var last = partitions.shift();

	// sort the remaining entries by value
	partitions.sort(function(p1, p2) { return p2.count - p1.count; });

	[notice] 2019-08-13T17:51:13.420803Z nonode@nohost <0.4242.0> -------- 127.0.0.1 - - GET /db-7d3b1fc07b063c0fece110b7f013b3ca 200
	[notice] 2019-08-13T17:51:13.433480Z nonode@nohost <0.4353.0> -------- 127.0.0.1 - - GET /db-b114040ecd03e61e2b576ce1edbf9310 200
	[notice] 2019-08-13T17:51:13.435036Z nonode@nohost <0.4347.0> -------- Received {[{<<"source">>,<<"http://127.0.0.1:40837/db-7d3b1fc07b063c0fece110b7f013b3ca">>},{<<"target">>,<<"http://127.0.0.1:40837/db-b114040ecd03e61e2b576ce1edbf9310">>}]} {user_ctx,null,[<<"_admin">>],undefined}

	[notice] 2019-08-13T17:51:13.631469Z nonode@nohost <0.4347.0> -------- Checking authorization

	[notice] 2019-08-13T17:51:13.631626Z nonode@nohost <0.4347.0> -------- Starting listener

	[notice] 2019-08-13T17:51:13.631844Z nonode@nohost <0.4347.0> -------- Starting replication loop

	apt-get update
	apt-get install -y \
	apt-transport-https \
	ca-certificates \
	curl \
	gnupg2 \
	software-properties-common
	curl -fsSL https://download.docker.com/linux/debian/gpg \| apt-key add -
	add-apt-repository \
	"deb [arch=amd64] https://download.docker.com/linux/debian \

	➜ ~ ibmcloud sl vs help
	NAME:
	ibmcloud sl vs - Gen1 infrastructure Virtual Servers
	USAGE:
	ibmcloud sl vs command [arguments...] [command options]

	COMMANDS:
	cancel Cancel virtual server instance
	capture Capture virtual server instance into an image
	create Create virtual server instance

	function (doc) {
	var partition = doc._id.slice(0, doc._id.indexOf(':'))
	emit(partition, 1);
	}

	# This file just grabs the list of pod IP addresses for the service
	# called "couchdb" and feeds those as `couchdb@PODIP` nodes to mem3.
	# It's just a proof of concept; the proper solution is likely a module
	# in mem3 itself.

	import json
	import os
	import requests
	import time

	kind: ConfigMap
	apiVersion: v1
	metadata:
	name: couchdb
	data:
	# Erlang VM settings. The -name flag activates the Erlang distribution; there
	# should be no reason to change this setting. The -setcookie flag is used to
	# control the Erlang magic cookie. CouchDB cluster nodes can only establish a
	# connection with one another if they share the same magic cookie.
	erlflags: >

	# Start a 3 node cluster and join it together automatically. Uses
	# local ephemeral disk for database storage.

	apiVersion: extensions/v1beta1
	kind: Deployment
	metadata:
	name: couchdb
	spec:
	replicas: 3
	template:

Adam Kocoloski kocolosk

Heterogeneous Schema Discovery

What is a heterogenous sc