ingesting activities into stumpf

doc.md

The way to ingest activities is to take the relevant json files from the m1-annotations repo and run a python script over them that uploads them to Stumpf.

To set this up on your machine you need a few things:

• Python and pip (if running the command 'pip' or 'python -m pip' prints the help page, you're good)

• The m1-annotations repo
  Typically you'll want to be on the latest version of master without uncommitted changes, since the files are going from your computer to Stumpf.

• A few python packages
  These can be installed via pip, so running one of the 2 commands should work:
  pip install --user requests awscli boto3
  python -m pip install --user requests awscli boto3

• AWS credentials
  Once you have those python packages, you should be able to run the 'aws configure' command (this might be 'aws.cmd configure' if you're on Windows, I'm not sure). It'll ask you a few questions, most you can just accept the defaults for (press enter). The primary concern is when it asks you for an access key id and secret access key you use the credentials obtained under "Access Keys" here: https://console.aws.amazon.com/iam/home?region=us-east-2#/security_credentials

At this point, you have all the dependencies for running the script.

1. Go to the main directory inside the m1-annotations repository

2. Create a text file with the list of activities to import, for example:
   m1-annotations/master_grouper/2018-03-08/11/2018-03-08.10-55-00.11-00-00.bus.G327/refiner_activities_pass2

3. Run 'python tools/stumpf-ingest.py $file' where $file is the location of the text file with your activities.

Here there's 2 likely outcomes:

1. It spits out an error about the data
   It won't accept when there's an invalid track/activity type (often capitalization errors), or if there's a file with no tracks in it. In this case, fixing the json and trying again is best. If it says videos aren't ingested, then contact Jon/Dan/Brandon/Mike with the list to ingest.

2. It says 'Uploaded all activities...' - in this case, you're all set.
   Nothing left to do. You can send an email to Dan or Jon Beezley if you want these processed immediately, but we're currently checking the queue every day - so it'll get picked up from here.