Ettore Rizza ettorerizza

How to control OpenRefine 3.3+ with cURL (and jq) in Bash scripts

tested on Fedora 32 with bash 5.0.17 and curl 7.69.1

git clone https://gist.github.com/d76bd27fbc4b8ab6d683822cdf61f81d.git bash-refine

layout

title

description

	#!/usr/bin/env python

	# Get all the database objects and permissions.
	# Can be used after running snowflake_roles.py to create the required roles
	# 2018-10-23 jfrink added ddl for roles, modified connection parameters
	# 2019-01-15 jfrink added Roles and permissions report by object.
	# 2019-03-07 jfrink added extract script to create a dump of all the tables to a stage
	# and also the corresponding script to load all the data.
	# Converted show tables over to using information schema for cases greater then 10k rows.
	# Converted show views over to using information schema for cases greater then 10k rows.

	<?php
	// TODO
	//
	// Change dc:subject to tags as with OpenLearn
	// Add notes at top of each output file
	// Add namespaces at top of each output file
	// Add Table of Contents?

	$books = "";
	$serials = "";

	# Count rows within CSV files, in a specified directory.
	Get-ChildItem "C:\myfolder\" -re -in "*.csv" \|
	Foreach-Object {
	$fileStats = Get-Content $_.FullName \| Measure-Object -line
	$linesInFile = $fileStats.Lines -1
	Write-Host "$_,$linesInFile"
	}


	/*
	===================================================
	Generate MERGE Statements for All Tables
	===================================================
	Copyright: Eitan Blumin (C) 2012
	Email: [email protected]
	Source: www.madeiradata.com
	Disclaimer:
	The author is not responsible for any damage this

	I followed these two blogs to install server
	1. http://koo.fi/blog/2015/03/19/openstreetmap-nominatim-server-for-geocoding/#Compile_Nominatim
	This explains ( and is the main blog which I followed) various steps
	2. https://www.linuxbabe.com/linux-server/openstreetmap-tile-server-ubuntu-16-04
	This explains how to setup swap files and install tiles if needed.

	I will use Ubuntu 14.04 LTS as the platform. Just a basic install with ssh server. We will install Apache to serve http requests. Make sure you have enough disk space and RAM to hold the data and serve it efficiently. I used the Finland extract, which was about a 200 MB download. The resulting database was 26 GB after importing, indexing and adding Wikipedia data. The Wikipedia data probably actually took more disk space than the OSM data. My server has 4 GB RAM, which seems to be enough for this small data set.

	1. Sofware requirements

	#!/bin/sh

	gunzip -c viaf-20191104-links.txt.gz \| perl viaf-links-count.pl > viaf-links-count-201911.txt