Reporting app sketches

sketches.md

Getting data for frontend

NOTE: This section is out-of-date.

Getting campaigns, questions, responses, and custom fields. from Spoke.

First import the module:

import rlabsreporting

Instantiate the class:

# Requires a .env file with db credentials w/ the following keys: 
# dbname, host port, user, password
#
# Also needs a list of campaign IDs (IDs should be ints)
pg_reporting = rlabsreporting.pg.PgReporting(campaign_ids)

Getting campaigns:

pg_reporting.get_campaigns()

==> creates a pg_reporting.campaigns object, which is a dictionary with campaign ID as keys and campaign titles as values

Getting survey responses:

pg_reporting.get_survey()

==> create a pg_reporting.survey object, which is a dictionary with campaign ID as keys, with dictionaries of question-keys and response-values. (So, a nested ditionary of dictionaries -- not sure how much I like nesting though, but that's the idea for now.)

Getting custom fields:

pg_reporting.get_custom_fields()

==> creates a pg_reporting.custom_fields object, which is a dictionary of campaign IDs and the names of their custom fields.

Final generation of report .xlsx files

Parameters:

1. Path to CSV w/ "flat" campaign export (file format might change, but CSV is ok for now)
2. Excel sheet names (for each desired type of response), with the specific Spoke survey questions and associated reponses
3. Desired custom fields

Example of #2:

{
  "Know when they will vote": {
    "Do they know when they will vote?": [
      "Yes, they know when to vote",
      "Yes"
    ],
    "Do they know when to vote?": [
      "Yes"
    ]
  },
  "Negative responses": {
    "Do they know when they will vote?": [
      "No",
      "No they do not know when they will vote"
    ],
    "Do they know when to vote?": [
      "No"
    ]
  }
}

Example of #3:

["DWID", "VAN_ID"]

Report statuses

• -1: Error
• 0: Requested (Default)
• 1: In progress
• 2: Completed
• 3: Cancelled

Loading data into Redis

NOTE: Requires a .env file w/ Redshift credentials in current directory.

import rlabsreporting

rput = rlabsreporting.RedisPut()

rput.get_campaigns()
rput.get_custom_fields()
rput.get_survey()

rput.make_all_campaign_json()

rput.load_all_campaigns_into_redis()

API sketches

TODO

New field: Priority (integer) Default: 0

New field: URL of object (will be stored on S3, but that's an unneeded detail in this context)

Endpoints: /api/v1.0/reports

GET

• Returns list of all report tasks
• Example:

{"reports": [{"id": 0, "c_ids": "6000,6001,6002", "name": "somename", "created_at": unixtime, "status" 0}]}

Probably should turn c_ids value into a JSON list.

POST

• Accepts JSON object with four fields. Example:

{"name": somename, "surveys": [], "c_ids", [6000, 6001, 6002], "custom_fields": []}  

Return 400 on bad request.

/api/v1.0/reports/<int:rid>

GET

• Returns dict of JSON

PUT

One field: status Updates status of a report. (Cancelled, done, etc)

In future could allow arbitrary changing of fields.

Notes:

[email protected] needs its EnvironmentFile value edited to a environment with all Redshift credentials

Can also update Docker container and docker-dev.sh to also have credentials.

This looks clear. Would you consider returning lists of objects for get_campaigns(), get_survey(), and get_custom_fields() rather than a single object, where the campaign_id is stored as a value in each object? I've been finding that kind of interface much simpler for iterations, etc.

This looks clear. Would you consider returning lists of objects for get_campaigns(), get_survey(), and get_custom_fields() rather than a single object, where the campaign_id is stored as a value in each object? I've been finding that kind of interface much simpler for iterations, etc.

Sure -- Could you sketch out what those lists of objects would look like so I know what I'm aiming for? (Want to make sure I understand the intent)

get_campaigns() returns:
[
    {'campaign_id': 2734, 'title': 'BVM Voter Reg'},
    {'campaign_id': 2891, 'title': 'Chockablocka Hammertime'}
]

get_survey() returns:
[
    {
        'campaign_id': 2834,
        'questions': [
           {
               'question': 'Who did you vote for?',
               'responses': ['Biden', 'Trump', 'Not voting']
           },
           {
               'question': 'Will you text three friends?',
               'responses': ['Yes', 'No']
           }
         ]
   },
    {
        'campaign_id': 7254,
        'questions': [
           {
               'question': 'When is the party?',
               'responses': ['Monday', 'Tueday', 'No party']
           },
           {
               'question': 'Is it cold?',
               'responses': ['Yes', 'No']
           }
        ]
   }
]


get_custom_fields() returns:
[
    {
        'campaign_id': 2348,
        'custom_fields': ['DWID', 'District', 'Eligibility']
    },
    {
        'campaign_id': 8280,
        'custom_fields': ['PID', 'middle_name']
    },
]

From Austin:

sketching it out -
rlabsreporting could have this:

def get_campaigns(c_ids):
  pg_reporting = pg.PgReporting(creds, c_ids)
  campaigns = pg_reporting.get_campaigns()
  return campaigns

instead of Flask importing the class and then instantiating it, flask could run:

from rlabsreporting import get_campaigns
campaigns = get_campaigns(c_ids)

The reports database ORM model AFTER a user has requested a report and aggregated question / response versions.

There are 6 objects/tables: Report, Question, Questionversion, Response, Responseversion, CustomField. Each of these have one-to-many relationships structured as follows:

• Report (id, c_ids, name, created_at, fname, status, questions, customfields)
  • Question (id, text, versions, responses, report_id)
    • Questionversion (id, text, question_id)
    • Response (id, text, versions, question_id)
      • Responseversion (id, text, response_id)
  • CustomField(id, text, report_id)

Child relationships are bold
Parent relationships are italic

For any given single question, there may be many versions of it in a series of campaigns: "Did you vote?", "Have you voted?", "Did you vote yet?" Similarly for responses: "Yes", "Yes, I voted."

The surveys (question / response pairs) can be passed in a JSON object or python dictionary structured like so:

[
  {
    'questions': ['Did you vote?', 'Have you voted?'],
     'responses': [
                    ['Yes', 'yes', 'yes I voted'],
                    ['No', 'no']
                  ]
  },
  {
    'questions': ['will you remind 3 friends?'],
    'responses': [
                   ['yes', 'Yes'],
                   ['no']
                 ]
  }
]

The API model has been updated to integrate with the format expected by the report module, as detailed below:

The reports database ORM model AFTER a user has requested a report and aggregated responses into tabs.

There are 5 objects/tables: Report, Tab, Question, Response, CustomField. Each of these have one-to-many relationships structured as follows:

• Report (id, c_ids, name, created_at, fname, status, tabs, customfields)
  • Tab (id, text, questions, report_id)
    • Question (id, text, responses, tab_id)>
      • Response (id, text, question_id)
  • CustomField(id, text, report_id)

Child relationships are bold
Parent relationships are italic