Preparing ERA Data for DSpace

_README.md

The Ask

March 2022, Weiwei asked,

We are looking to create an extract of around 4 dozen items in ERA as sample data for Atmire to evaluate and propose for migration charges. See Leah's message below. I wonder if this could be something you coordinate with Leah and someone from the metadata team? You are probably the one who has the most comprehensive knowledge of ERA and the history of some of these items?

Ignace Deroost of Atmire said,

A CSV format usually works best. So if you could export your samples in CSV that would be great!

From our DSpace investigation I had observed a csv export from their software (dspace_sample_export)

The Work

Using the rails c in my jupiter development environment I worked out code snippets that would export the content we would need.

Knowing that we used acts_as_rdfable to annotate our ActiveRecord attributes with rdf predicates it was easy to use this as the csv headings: RDF::URI(Thesis.rdf_annotation_for_attr(key).first.predicate).pname.to_s

I captured the Item and Thesis export in the create_data.rb found below.

Mariana and I worked out the types of metadata that we would need in DSpace migration - export packages metadata checklist and sample data using comments and in conversation with Leah and Sharon in email.

This allowed me to fill out examples of the exports for bitstreams, communities, collections, statistics and version history. I'm not satisfied with the presentation of the version history but this might be good enough?

Mariana picked a number of items from ERA which met our criteria. I refined my scripts to just pick those Entities (pick_items_data.rb). Then I used the rails console in production to export the data:

ssh era-app-prd-3
sudo su apache -s /bin/bash
cd /var/www/sites/jupiter/
RAILS_ENV=production rails c --sandbox

I used scp to copy the exported csv files to my desktop and then can email to Leah, Sharon, Mariana and Weiwei for discussion before emailing Atmire.

Metadata	Locations
Community metadata (descriptive)	atmire_community_data.csv
Collection metadata (descriptive)	atmire_collection_data.csv
Resource metadata (descriptive, generic items)	atmire_item_data.csv
Resource metadata (descriptive, theses)	atmire_thesis_data.csv
Bitstreams / technical metadata	atmire_bitstreams_data.csv (filename, file order, md5, size, content type)
Usage metadata	atmire_statistics_data.csv - views and downloads
Workflow/process and provenance metadata	atmire_item_history_data.csv (object and object_changes used as basis for diff displayed in current UI)
Access info	atmire_item_data.csv and atmire_thesis_data.csv - dc:accessRights

Memberships	Locations
Memberships between resource (thesis or item) metadata and a community + collection pair	Match between atmire_community_data.csv / atmire_collection_data.csv andatmire_item_data.csv / atmire_thesis_data.csv (path field)
Memberships between communities and collections	Match between atmire_community_data.csv and atmire_collection_data.csv (path field)
Memberships between bitstreams and resource metadata	atmire_bitstreams_data.csv and atmire_item_data.csv atmire_thesis_data.csv (id fields)

Alternate Approaches

We use pmpy and our AIP api to create a preservation copy (eg: e75e4ad4-7301-41f8-9127-8d293f469c8b) in Open Stack Swift. All the information required for migration might be available via that format.

Exporting Bitstreams

Download from era.library.ualberta.ca

This is what I've opted to do for the sample data, using Rails.application.routes.url_helpers.file_download_item_url(id: file.record_id, file_set_id: file.fileset_uuid) to get the download link for each bitstream.

One concern for this approach might be the performance impact this would have on the website. The application was designed with the intent to be indexed by Google Scholar and LAC using the OAI protocol. As long as the process used by Atmire has similar throttling and retry policies we should be okay.

Swift Container Dumped to Disk

This makes a lot of sense to me. The content is already organized on disk by entity uuid and contains contextual files. We could make a copy, put it on a large enough disk and mail it to Atmire.

/era_assets Dumped to Disk

Similarly to the Swift Container this is the local Rails ActiveStorage directory. The path from the entities uuid to the bitstream is obfuscated but not difficult to export.

atmire_bitstreams_data.csv

atmire_collection_data.csv

atmire_community_data.csv

atmire_item_data.csv

atmire_item_history_data.csv

atmire_statistics_data.csv

atmire_thesis_data.csv

create_data.rb

CSV.open('atmire_item_data.csv', 'wb') do |csv|
  keys = Item.first.attributes.keys
  csv << keys.map {|key|  Item.rdf_annotation_for_attr(key).present? ? RDF::URI(Item.rdf_annotation_for_attr(key).first.predicate).pname.to_s : key }
  Item.find_each do |item|
    csv << item.values_at(keys)
  end
end

CSV.open('atmire_thesis_data.csv', 'wb') do |csv|
  keys = Thesis.first.attributes.keys
  csv << keys.map {|key|  Thesis.rdf_annotation_for_attr(key).present? ? RDF::URI(Thesis.rdf_annotation_for_attr(key).first.predicate).pname.to_s : key }
  Thesis.find_each do |item|
    csv << item.values_at(keys)
  end
end

CSV.open('atmire_bitstreams_data.csv', 'wb') do |csv|
  csv << ['id', 'type', 'filename', 'file order', 'md5', 'size in bytes', 'content type']
  [Item,Thesis].each do |klass|
    klass.find_each do |item|
      item.files.each_with_index do |file, index|
        csv << [item.id, item.class, file.filename.to_s, index, file.checksum, file.byte_size, file.content_type]
      end
    end
  end
end

CSV.open('atmire_collection_data.csv', 'wb') do |csv|
  keys = Collection.first.attributes.keys
  csv << keys.map {|key|  Collection.rdf_annotation_for_attr(key).present? ? RDF::URI(Collection.rdf_annotation_for_attr(key).first.predicate).pname.to_s : key }
  Collection.find_each do |item|
    csv << item.values_at(keys)
  end
end

CSV.open('atmire_community_data.csv', 'wb') do |csv|
  keys = Community.first.attributes.keys
  csv << keys.map {|key|  Community.rdf_annotation_for_attr(key).present? ? RDF::URI(Community.rdf_annotation_for_attr(key).first.predicate).pname.to_s : key }
  Community.find_each do |item|
    csv << item.values_at(keys)
  end
end

CSV.open('atmire_statistics_data.csv', 'wb') do |csv|
    csv << ['id', 'type', 'views', 'downloads']
    Item.find_each do |item|
      csv << [item.id, item.class, Statistics.for(item_id: item.id)].flatten
    end
    Thesis.find_each do |item|
      csv << [item.id, item.class, Statistics.for(item_id: item.id)].flatten
    end
end

CSV.open('atmire_item_history_data.csv', 'wb') do |csv|
  keys = ["id", "item_type", "item_id", "event", "whodunnit", "object", "created_at", "object_changes"]
  csv << keys
  [Item,Thesis].each do |klass|
    klass.find_each do |item|
      item.versions.each do |version|
        csv << version.values_at(keys)
      end
    end
  end
end

dspace_sample_export.csv

e75e4ad4-7301-41f8-9127-8d293f469c8b

https://gist.github.com/pgwillia/eed7dd858e17a9a67f9d90cb1d703adb#file-atmire_item_history_data-csvtmp/work/e75e4ad4-7301-41f8-9127-8d293f469c8b
├── bag-info.txt
├── bagit.txt
├── data
│   ├── logs
│   │   ├── aipcreation.log
│   │   └── files_logs
│   │       ├── a3a81495-1725-4277-a6b6-5c75538f96ee
│   │       │   └── content_fixity_report.n3
│   │       ├── d6dc3572-a33c-4559-bd4a-73567eca9fe0
│   │       │   └── content_fixity_report.n3
│   │       └── de42fe9e-0fb3-4c5f-9bed-8d4886a0f42c
│   │           └── content_fixity_report.n3
│   └── objects
│       ├── files
│       │   ├── a3a81495-1725-4277-a6b6-5c75538f96ee
│       │   │   └── theses.jpg
│       │   ├── d6dc3572-a33c-4559-bd4a-73567eca9fe0
│       │   │   └── era-logo.png
│       │   └── de42fe9e-0fb3-4c5f-9bed-8d4886a0f42c
│       │       └── image-sample.jpeg
│       └── metadata
│           ├── files_metadata
│           │   ├── a3a81495-1725-4277-a6b6-5c75538f96ee
│           │   │   └── file_set_metadata.n3
│           │   ├── d6dc3572-a33c-4559-bd4a-73567eca9fe0
│           │   │   └── file_set_metadata.n3
│           │   ├── de42fe9e-0fb3-4c5f-9bed-8d4886a0f42c
│           │   │   └── file_set_metadata.n3
│           │   └── file_order.xml
│           └── object_metadata.n3
├── manifest-md5.txt
├── manifest-sha1.txt
├── tagmanifest-md5.txt
└── tagmanifest-sha1.txt

pick_items_data.rb

items_to_pick = []
theses_to_pick = []

CSV.open('atmire_item_data.csv', 'wb') do |csv|
  keys = Item.first.attributes.keys
  csv << keys.map {|key|  Item.rdf_annotation_for_attr(key).present? ? RDF::URI(Item.rdf_annotation_for_attr(key).first.predicate).pname.to_s : key }
  Item.find(items_to_pick).each do |item|
    csv << item.values_at(keys)
  end
end

CSV.open('atmire_thesis_data.csv', 'wb') do |csv|
  keys = Thesis.first.attributes.keys
  csv << keys.map {|key|  Thesis.rdf_annotation_for_attr(key).present? ? RDF::URI(Thesis.rdf_annotation_for_attr(key).first.predicate).pname.to_s : key }
  Thesis.find(theses_to_pick).each do |item|
    csv << item.values_at(keys)
  end
end

CSV.open('atmire_bitstreams_data.csv', 'wb') do |csv|
  csv << ['id', 'type', 'download target', 'filename', 'file order', 'md5', 'size in bytes', 'content type']
  Item.find(items_to_pick).each do |item|
    item.files.each_with_index do |file, index|
      csv << [item.id, item.class, Rails.application.routes.url_helpers.file_download_item_url(id: file.record_id, file_set_id: file.fileset_uuid), file.filename.to_s, index, file.checksum, file.byte_size, file.content_type]
    end
  end
  Thesis.find(theses_to_pick).each do |item|
    item.files.each_with_index do |file, index|
      csv << [item.id, item.class, Rails.application.routes.url_helpers.file_download_item_url(id: file.record_id, file_set_id: file.fileset_uuid), file.filename.to_s, index, file.checksum, file.byte_size, file.content_type]
    end
  end
end

collections_to_pick = []
communities_to_pick = []
Item.find(items_to_pick).each do |item|
  item.member_of_paths.each do |community_collection|
    communities_to_pick << community_collection.split('/')[0]
    collections_to_pick << community_collection.split('/')[1]
  end
end
Thesis.find(theses_to_pick).each do |item|
  item.member_of_paths.each do |community_collection|
    communities_to_pick << community_collection.split('/')[0]
    collections_to_pick << community_collection.split('/')[1]
  end
end

CSV.open('atmire_collection_data.csv', 'wb') do |csv|
  keys = Collection.first.attributes.keys
  csv << keys.map {|key|  Collection.rdf_annotation_for_attr(key).present? ? RDF::URI(Collection.rdf_annotation_for_attr(key).first.predicate).pname.to_s : key }
  Collection.find(collections_to_pick).each do |item|
    csv << item.values_at(keys)
  end
end

CSV.open('atmire_community_data.csv', 'wb') do |csv|
  keys = Community.first.attributes.keys
  csv << keys.map {|key|  Community.rdf_annotation_for_attr(key).present? ? RDF::URI(Community.rdf_annotation_for_attr(key).first.predicate).pname.to_s : key }
  Community.find(communities_to_pick).each do |item|
    csv << item.values_at(keys)
  end
end

CSV.open('atmire_statistics_data.csv', 'wb') do |csv|
    csv << ['id', 'type', 'views', 'downloads']
    Item.find(items_to_pick).each do |item|
      csv << [item.id, item.class, Statistics.for(item_id: item.id)].flatten
    end
    Thesis.find(theses_to_pick).each do |item|
      csv << [item.id, item.class, Statistics.for(item_id: item.id)].flatten
    end
end

CSV.open('atmire_item_history_data.csv', 'wb') do |csv|
  keys = ["id", "item_type", "item_id", "event", "whodunnit", "object", "created_at", "object_changes"]
  csv << keys
  Item.find(items_to_pick).each do |item|
    item.versions.each do |version|
     csv << version.values_at(keys)
    end
  end
  Thesis.find(theses_to_pick).each do |item|
    item.versions.each do |version|
     csv << version.values_at(keys)
    end
  end
end