- Create election in elections admin
- Enter race meta data
- Set up clarity scrapers for local county races
- Set up any Collin County race results to be clerked, including clerks
- Pair race meta data with race aliases from scrapers
- Pair candidates
- Create a new homepage in the django admin
- Set up the featured races for the homepage module
- !!! Need Allan to remind me how to preview the homepage modules and get the embed codes
- Create a new liveblog in the datalab/admin
- Create new metadata for results, supplying the liveblog id
- Update the hashtags the live blog pulls
- Publish and pray
Last active
January 18, 2023 03:07
-
-
Save johndhancock/89288c2bbc92f545ad26a445200f46e2 to your computer and use it in GitHub Desktop.
Elections
Some races cross county lines, and need to have the results combined, i.e., abstract tabulations. To set up an abstract tabulation for a race, go to https://elections-admin.dallasnews.com/admin -> results -> abstract tabulations -> add abstract tabulation.
You need to pull the race id number from the url in the race's page in the admin. Reporting unit should be USA > Texas > Texas (Abstract). Then you want to select the level to tabulate up from, in this case County
Note: If your race crosses county lines with Collin County, you'll need to enter that race to be clerked before setting up the abstract tabulation.
jhancock 7:51 AM Hey Allan, I know you’ve probably long-since forgotten how the elections rig here works, but I’m preparing for AP Super Tuesday testing and was hoping you could shed a little light on the process for setting up our rig for an AP feed, if you remember anything. Things I’m unclear about include: Do I need to manually enter the race data first, and then pair it with the ap feed, like we do with the county clarity sites, or, does the AP feed make that step unnecessary? Is capturing results just as simple as setting up the AP in the Data Sources for the election, like in the screenshot attached? The documentation makes it look like there’s several python management commands to run, but I get the impression this is maybe for capturing results in local development? Screen Shot 2020-02-07 at 7.50.22 AM.png Screen Shot 2020-02-07 at 7.50.22 AM.png
7:52 Realize you’ve got your hands full with nation-wide primary stuff now, and this is all probably a distant memory, but could use anything you can remember. (edited)
ajv 8:04 AM Hey John! Definitely a little slammed with election prep, but I’ll take a look at this in the next two hours and get back to you then.
ajv 11:51 AM Looking at this now. Sorry for delay — we keep getting kind of (but not really) significant updates from The Caucus That Refuses To Die.
jhancock 11:51 AM No worries. I know you’re swamped. If you have time, great, but if not, I totally get it.
ajv 12:00 PM So I believe you'd want to create your Election instance and then run pipenv run python manage.py import_ap_metadata 2020-03-03. 12:00 That ought to create races and candidates for you, and tie them to the AP's IDs as well. (edited)
jhancock 12:01 PM when you say create my Election instance, you mean just create a new election in the elections admin, like this: Screen Shot 2020-02-07 at 12.01.23 PM.png Screen Shot 2020-02-07 at 12.01.23 PM.png
ajv 12:03 PM Yep! That's all you need. You will want the "Use test-data loaders?" option to be checked at this stage, though. 12:05 The one issue with all this is, the AP doesn't send around finalized metadata (race and candidate names, etc.) until usually midnight on the day of the election. So what I've tended to do is, either have two totally separate Heroku instances of the code (one for staging/test data and one for the real thing) or blow away the database after the last test. 12:05 So, in other words, you'll have to run import_ap_metadata at least twice — once for the testing cycle and once for the real thing. That make sense?
jhancock 12:06 PM Yeah. We haven’t gotten a Texas Super Tuesday test schedule yet, so I’m just trying to think ahead. So, for the races included in the AP feed, I don’t need to enter the race data beforehand and then pair races, like we do with the local stuff?
ajv 12:10 PM Well, if you haven't gotten a test schedule you must have some clairvoyance — our nationwide testing for Super Tuesday starts on Monday. Here's what I have on that in case it's useful (obviously they could have different timing for one-state-only tests): Screen Shot 2020-02-07 at 1.08.56 PM.png Screen Shot 2020-02-07 at 1.08.56 PM.png
12:10 Yeah, exactly. Since AP gives you more data than Clarity does, the code can construct most of that race data automatically.
jhancock 12:11 PM Okay. Ugh. I called the AP this morning and they said testing doesn’t usually happen until two weeks out. 12:11 Now I’m questioning that.
ajv 12:14 PM Yeah, it was like pulling teeth to get this doc out of them, and I still haven't formally gotten any information about their New Hampshire or Nevada or South Carolina tests. Their broad-ranging communication isn't totally the best right now (although to their credit, they've been great about quickly passing along Iowa information whenever the state party deigns to say anything).
jhancock 12:14 PM So, if I ran that import_ap_metadata command now, would nothing happen since tests aren’t going on?
ajv 12:15 PM Think you'll have to reach out through Konradi if you want to escalate (assuming he still manages that relationship). Do you have a current API key?
jhancock 12:15 PM Regarding API key: I only have the one you got last time. I asked about that too this morning and they said they’d get back to me.
ajv 12:16 PM Hard to say, actually. If you wanted, you could send a test request to their API directly. Here's their documentation on their API: 12:16 PDF AP_Elections_API_Developer_Guide.pdf 769 kB PDF — Click to view
12:16 So let me send you what a test request would look like. 12:18 https://api.ap.org/v2/elections/2020-03-03?apiKey=yourKey&format=json 12:18 If you have time, now, want to pop in your API key and see what that gives you back? 12:18 From just my experience, I don't think POLITICO's key changed from 2018. (edited)
ajv 12:20 PM OK. Well once they send you a key, that endpoint will look more like this: 12:21 2020-03-03.json {"electionDate":"2020-03-03","timestamp":"2020-02-07T18:19:42.032Z","races":[{"raceID":"44001","statePostal":"TX","raceTypeID":"D","officeID":"P","officeName":"President","party":"Dem","national":true,"lastUpdated":"2020-02-06T20:10:27.940Z","candidates":[{"first":"Joe","last":"Biden","party":"Dem","candidateID":"61613","polID":"1036","ballotOrder":1,"polNum":"52388"},{"first":"Cory","last":"Booker","party":"Dem","candidateID":"61614","polID":"63133","ballotOrder":3,"polNum":"52389"},{"first":"Pete","last":"Buttigieg","abbrv":"Buttigig","party":"Dem","candidateID":"61615","polID":"67933","ballotOrder":4,"polNum":"52390"},{"first":"Tulsi","last":"Gabbard","party":"Dem","candidateID":"61616","polID":"62589","ballotOrder":5,"polNum":"52391"},{"first":"Amy","last":"Klobuchar","abbrv":"Klobuchr","party":"Dem","candidateID":"61618","polID":"51526","ballotOrder":6,"polNum":"52393"},{"first":"Bernie","last":"Sanders","party":"Dem","candidateID":"61619","polID":"1445","ballotOrder":7,"polNum":"51500"},{"first":"Tom","... Click to expand inline (1 line)
jhancock 12:21 PM Okay. I think I might call back this afternoon just to double check. 12:22 I got a testing schedule for the races after Iowa but before Super Tuesday, but haven’t seen a Super Tuesday one yet. 12:24 Again, know you’re super busy, but once I get to the point where I’m running that import_ap_metadata command, is it cool if I reach out to you if I stumble?
ajv 12:26 PM Sure thing. Happy to field questions. May have to time-shift a little if I'm right in the middle of something big, but absolutely feel free to reach out.
jhancock 12:26 PM Yeah, totally get that.
ajv 12:27 PM Before you run that import_ap_metadata command, I'd hit that JSON endpoint to see what kind of stuff it'll be pulling in. 12:27 The JSON you get back from that request may contain the senate/congressional primaries too. There may also be another parameter you can set for state-level versus national-level results — I think national only contains presidential race. That'd be listed in the developer guide I sent you above. 12:27 And you may actually need to add something like &test=true to the querystring — again, would be in that doc. 12:29 As to your other question about actually pulling in results, that should be just a push-button thing that starts working when you change the Test capture status or Result polling status values. (You'll need to have imported the metadata first, of course.) 12:30 That capture process happens in Celery, though (apologies if this is a review, but that's essentially a separate server on Heroku that only handles scheduled tasks and doesn't serve any HTTP responses). You'll need to make sure you have at least one worker node set up for this project in Heroku any time you want to pull results down. 12:31 Actually, you'll probably need that set up whenever you pull metadata too.
jhancock 12:31 PM Right. Guess one other question is where do I actually run that import_ap_metadata command from? 12:31 My .env file in elex-results doesn’t seem to have any databases associated with it 12:32 At least my local copy doesn’t.
ajv 12:32 PM Hm. I'd go into the configuration on Heroku and copy whatever's in the elex-results app's DATABASE_URL env var.
jhancock 12:33 PM So I’d run it from elex-results ?
ajv 12:34 PM Although you may not want to mess around with your live database in production, in which case you could just create a new DB locally with createdb my-database-name, and then set the .env to be postgres:///my-database-name. (edited)
jhancock 12:35 PM Well, I guess what I’m asking is in production to get that ap metadata, where do I run that command from?
ajv 12:36 PM
The easiest thing would just be to go into the folder where this lives on your computer and then run heroku run bash. That will spin up (and log you into) a short-lived server which will have all the same environment variables as the production stuff.
heroku run bash --app elex-results
12:37 Then you can run python manage.py import_ap_metadata 2020-03-03 from there.
jhancock 12:37 PM Okay, that makes sense.
ajv 12:37 PM (You don't need to prefix with pipenv run in the production setting) 12:38 Good! Let me know if you have any questions as you go through this. Hopefully it'll be a fairly easy process, but I'm around if not. And best of luck with the AP people.
Set up your election as you normally would through elex-results. You'll also need the AP Elections API key, which the AP usually emails out the newsapps email address. Make sure this api key matches the one in elex-results in heroku, under the settings tab, under config variables.
First step is grab the race metadata from AP. The way to do this is to jump into the project on heroku through the command line by running heroku run bash --app elex-results. Once in the app, you can run the command python manage.py import_ap_metadata <<election-date as YYYY-MM-MM>>
Example: python manage.py import_ap_metadata 2020-03-03
This will set up the race metadata, which you should then be able to see in the elex-results homepage.
Then you'll need to create the FIPS codes and area reports for the election: python manage.py fips.
Make sure you have a data source set up within your election. For state results, the area would be 48000, the loader type Associated Press, the enabled box checked.
For national results, the area would be 00000.
Then, when AP is running tests, change the results polling status dropdown to a time interval of your choice. Also check use-test-data-loaders. This box can be unchecked for live results.
If you need to update the AP race metadata at any point, follow the same steps above under setting up AP race metadata, except run the python command with the update flag: python manage.py import_ap_metadata 2020-03-03 --update
For primaries, you can enter in any local race metadata as you normally would. However, after the race is entered through elex-results, find the race through the admin panel, and select the appropriate party for the race.
After the race meta data has been entered, go to https://elections-admin.dallasnews.com/ -> results entry -> enter results. Select the race name and county from the dropdowns. Repeat.
The template for the xml output is found at elex-results > print_feeds > templates > print_feeds > fip > report.xml. You can edit this file to make changes to the print box format. This will usually be done in relation to the legends associated with the tables (rep, dem, runoff, etc).
You can see a difference file here to see what the format has been in the past: https://github.com/DallasMorningNews/elex-results/commit/724f68b461c388e2a19594b26d12b5e52f2e6e5e#diff-c08784c1c168f277a9e2c2cb1ec4c74a
Once elex-results is changed, pushing to master will kick off a new redeploy. There are a couple steps in verifying this happened correctly:
- Check elex-results in Circle-Ci to see if the tests pass
- Check the Heroku dashboard to verify a build and deploy
- Then check LOGDNA to verify that celery tasks get restarted.
In some instances, like when detail page winner and runoff declarations don't match list pages, you may need to generate new race reports for a given election. To do, in the terminal, run
heroku run bash -a elex-results
python manage.py shell
from governments.models import Race
from meta.models import Election
from results.tasks.race_reports import generate_race_reports
this_election = Election.objects.get_by_date_str('<<election date>>')
these_races = list(Race.objects.filter(election_id=this_election.pk).values_list('uuid', flat=True))
generate_race_reports(this_election.pk, these_races)
This should results in several .json files being listed in the terminal
Go to https://elections-admin.dallasnews.com/ -> selection election -> web publishing -> web feeds -> add web feed.
For the elections homepage, the slug is elections-dot. For the homepage module, the slug is homepage-races. Set the maximum number of entries, then the races you want to display.
A quick rundown of what the different key/values mean in the election-metadata.json file mean.
isDefault: true or false, determines which elections is being displayed
title, richTitle, logo, date, url, shareImage, fb, parselyID, authors, keywords, description should all be self explanatory
liveBogID: the id number of the liveblog that's been created for a given election
electionType: not completely sure what this covers or what the accepted values are.
verboseDate: Date of the election in standard format
resultsIndexLayout: an array of the blocks that make up the home page. each block takes a few different values:
divID: the id that will be given to that block
type: the type of block. can be a raceSearch, lastUpdated, featuredRaces, countyList, resultsGrid or linkList
title: the heading that will be displayed over the block
featuredCounties: If a countyList, you'll need to supply an array of fips codes for the counties you want to display
races: if a resultGrid or linkList, an array of race ids will need to be supplied to display the races
trailingRule: true or false, optional
displayField: for link list, you can specify what field is used as the link text. optional. left unset will use the name
searchPlaceholder: the placeholder text in the search field
endpoints: where the results and the liveblog data live. Note, during testing, if results are captured using staging, the url for the resultsBase will be "//s3.amazonaws.com/elections.dallasnews.com.test/<>/<>
hideResults: optional, but if set to true will not dispaly results on the election home page, only the blog. helpful if they want the blog up and running earlier than results are ready
At https://elections-admin.dallasnews.com/admin/, go to Homepage -> Add homepage On the following form screen, select the election and give the homepage a name. Typical homepage names are "test homepage" or "LIVE homepage."
Leave View state as full-width display, and set the view for the blog state. Check results active, and enter in the serif tag ex: local-politics. We typically set up a test homepage and a live homepage for each election.
After a homepage is created, you must then create the models for the regions (centerpiece, storyrail and rightrail) for each homepage.
Homepages can have slugs of testing-homepage or homepage.
At https://elections-admin.dallasnews.com/admin/, go to Homepage -> Regions -> Add Region. For most homepages, we'll create a region for the centerpiece, story rail and right sidebar.
Name each region, select the homepage and priority.
Centerpiece's have 1 allowed stub count and take a slug of centerpiece. For local elections, centerpieces get one region mode:
name: Simple results
slug: results
icon: ballot
Story rails's have 3 allowed stub count and take a slug of storyRail. For local elections, story rails get one region mode:
name: Election results
slug: results
icon: ballot
Right sidebars's have 0 allowed stub count and take a slug of rightRail. For local elections, right rails get two region mode:
name: Liveblog
slug: liveblog
icon: offline_bolt
name: Election results
slug: results
icon: ballot
Note: Any changes to the modles of the hompage or regions requires a republish of the homepage frontend. To see the testing homepage, the url is /homepage.html?test=true
To set up the elections rig to catch AP results, you'll need the AP results key for state and national. This goes in the .env file locally for elex-results, and also in the Heroku settings. They are usually sent by the AP to the newsapps email address at the beginning of testing season.
- In the elections admin, for an election, add two data sources
state, with a fips of 48000, select Associated Press as the loader and click enabled
national, fips of 00000, select Associated Press as the loader, and click enabled and national
Check use test-data loaders
- Run the management commands listed in the bootstrapping documentation here: https://github.com/DallasMorningNews/elex-results/blob/master/docs/first-steps-for-new-election.md
It's a good idea to delete all races from the test data a couple of weeks out from the live elections and reconstitute the database, then re-run the import_ap_metadata command again.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment