billdueber

Subject: ANNOUNCEMENT: Traject MARC->Solr indexer beta release

Jonathan Rochkind (Johns Hopkins), along with Bill Dueber (University of Michigan), is happy to announce a first beta release of "traject," a framework for indexing MARC data to Solr.

traject, in the vein of solrmarc, allows you to define your indexing rules using simple macro and translation files. However, traject runs under JRuby and is "ruby all the way down," so you can easily provide additional logic by simply requring ruby files.

traject is currently in a beta release, but is already being used in production to generated the HathiTrust Catalog (http://www.hathitrust.org/). traject was developed under a test-first mentality and has undergone both continuous integration and an extensive benchmarking/profiling period to keep it fast.

You can view the code[1] on github, and easily install it as a (jruby) gem using "gem install traject".

billdueber

module Traject
  class MarcExtractor
    
    # A set of Spec object, with knowlege about the collection as a whole
    class SpecSet
      
      attr_reader :interesting_tags, :options

      def initialize(opts = {})
        @specs = {}

billdueber

indy slowdown was due to me havint JRUBY_OPTS include -J-XX:+TieredCompilation and
-J-XX:TieredStopAtLevel=1, supposedly to make startup faster. Removing them 
removes the performance issue. Jira ticket closed.

billdueber

# I just nabbed the source of marc4j and built it with "ant jar"

require 'marc4j-2.5.1-beta.jar'

# Conveniently add Enumerable to the reader interface so I can get #each, #each_with_index, etc.
# This would be automatic if MarcReader were specified as an iterable, as per a recent github issue
# on the marc4j repo (https://github.com/marc4j/marc4j/issues/11)

module org.marc4j::MarcReader
  include Enumerable

billdueber

# Not saying this is optimal, just what I currently do.

# Start out by logging what the hell we're doing: what config files are loaded, where we're sending documents, etc.

INFO    08:48:20         1252   ROOT                            Loading files in /l/solr-vufind/apps/marc2solr_example/umich/lib
INFO    08:48:23         4113   MARC2Solr.Conf                  Set suss url to http://localhost:8024/solr/biblio
INFO    08:48:23         4114   MARC2Solr.Conf                  Using 3 threads for the suss
INFO    08:48:24         4334   ROOT                            Using 4 threads; activiating threach
INFO    08:48:24         4335   ROOT                            Indexing file /l/solr-vufind/data/vufind_full_20130715.seq.gz
INFO    08:48:24         4335   MARC2Solr.Conf                  Sniffed marc file type as seq

billdueber

Lessons learned from marc2solr

Things I did wrong

These are the things off the top of my head that drive me crazy and/or that I’ve had to work around. I’m sure there are more that I’ll come up with later.

The fundamental problem, it feels to me, is that the (equivalent of the) MARC::Reader.each loop is hidden. Pretty much all the rest of these problems flow from that. Basically, I want to give up on the idea of hiding the primary loop from the user, and just assume the user is both a programmer and a non-idiot.

billdueber

require 'parslet' 

# A complex field-selection syntax to get MARC fields. Something I'm messing around
# with for a marc indexing process I'm thinking of building to replace marcspec
#
#
# spec := <tag>
#         <tag>!<ind><ind>
# tag  := '245' # literal string
#      := '6##' # use hashes to mean "any character"

billdueber

require 'marc'
require 'marc4j4r'
require 'benchmark'
 

iterations = 1
 
xmlsourcefile = 'topics.xml' # 18k records as a MARC-XML collection

puts RUBY_DESCRIPTION

billdueber

      <fieldtype name="text" class="solr.TextField" positionIncrementGap="1000">
        <analyzer type="index">
          <charFilter class="solr.PatternReplaceCharFilterFactory"
            pattern="&amp;" replacement=" and " />
          <charFilter class="solr.PatternReplaceCharFilterFactory"
            pattern="\b([A-Ga-g])[\#♯](\s+|\Z)" replacement="$1 sharp$2" />                
          <charFilter class="solr.PatternReplaceCharFilterFactory"
            pattern="\b([A-Ga-g])\s*[b♭](\s+|\Z)" replacement="$1 flat$2" />                
          <charFilter class="solr.PatternReplaceCharFilterFactory"
            pattern="\b[Cc]\+\+" replacement="cplusplus" />                

billdueber

    <!-- 
      #########################
        TEXT FIELD TYPES
      ######################### 

      In all cases, we want to perform NFKC unicode normalization,
      case folding, and ASCII-folding (i.e., removal of accents so
      ü => u).                                   
      
      ICUFoldingFilterFactory will give us *all* of those things.