Eric Phetteplace phette23

Almost immediately after declaring a hiatus seems like a great time for a blog post.

Inspired by nina de jesus and Ruth Tillman's libtech level up project, here's something on the value of command-line text processing. Some of these common UNIX tools that have been around since practically the 1980s are great for the sort of data wrangling that many librarians find themselves doing, whether their responsibilities lie with systems, the web, metadata, or other areas. But the command prompt has a learning curve and if you already use text editor tools to accomplish some tasks, it might be tough to see why you should invest in learning. Here's one case I've found.

Scenario: our digital repository needs to maintain several vocabularies of faculty who teach in different departments. That information is, of course, within a siloed vendor product that has no viable APIs. I'm only able to export CSVs that looks like this:

"Namerer, Name","username"

"Othern

	// take http://digital2.library.unt.edu/vocabularies/agent-qualifiers/json/
	// and insert into an EQUELLA taxonomy, preserving URL & definition info
	// EQUELLA taxo format is CSV-like:
	// term1/term/term2,key,value,key2,value2…
	// can then upload with their included Python scripts or write your own
	var fs = require('fs')
	var data = JSON.parse(fs.readFileSync('agent-qualifiers.json'))
	var terms = data.terms

	var getDescription = function(term) {

	<#-- NOTE this style should be removed for home page portlet
	also should really use an ID rather than hide all portlet headings -->
	<style>
	/* hide header */
	.portlet_freemarker_content .box_title_wrapper h3 {
	display: none !important;
	}
	</style>
	<#-- these role IDs will need to be researched & changed -->
	<#if user.hasRole('490b1b93-10cd-b8fa-3291-93c357efe57b')>

	#!/usr/bin/env python

	# note final, more robust version at https://github.com/cca/millennium_patron_import

	import sys
	import csv # https://docs.python.org/2/library/csv.html
	# PCODE mappings, etc.
	from mapping import *
	import datetime
	import re

	var clipboard = require('child_process').spawn('pbcopy')
	, data = 'put whatever data here';

	clipboard.stdin.write(data);
	clipboard.stdin.end();

	// taken from https://medium.com/html5-css3/7c80a4b731f8
	// and expanded, unreadable one-liners are kind of pointless

	//pad zeroes
	function pad(num) {
	return ('0' + num).split('').reverse().splice(0,2).reverse().join('')
	}

	// get page query params
	var qp = document.location.search.replace(/(^\?)/,'')

	// stupid Node.js tricks
	// accepts JS object literal of commands & their args
	// executes each in series, pipes child process output to parent
	// e.g. `node hellspawn.js '{"ls": "-al", "pwd": null, "echo": "Hello world!"}'`
	var spawn = require('child_process').spawn
	, cmds = JSON.parse(process.argv[2])
	, cmd
	, key
	, val;

	#!/usr/bin/env fish
	# similar script in Fish
	# still under construction, need to quiet `git status` more effectively

	function update -d 'Update git repo'
	git stash --quiet
	git pull
	git stash apply --quiet
	end

	<?php
	// need to download simple_html_dom.php from SourceForge
	// http://sourceforge.net/projects/simplehtmldom/files/
	// and place it in the same directory as this script
	require_once( 'simple_html_dom.php' );
	// http://simplehtmldom.sourceforge.net/manual_api.htm

	$base = 'http://www.doaj.org/doaj?func=search&template=&uiLanguage=en&query=';
	$query = urlencode( 'librarianship' );

	for f in *.tif
	convert $f (basename -s .tif $f).jpg
	end