I was uploading many csv's in parallel to MyCampaign, for a total of ~660k contacts across 17 uploads, each with 39,998 contacts per csv.
In one of the uploads, this is what the csv sheet_01.csv looks like:
person__id,voterfile_vanid,occupation,employer,prefix,first,middle,last,suffix,nickname
...
3646c0f3-39ba-4756-bf2f-c33f2109eedb,228485,,,MWWIWFX,KEWNEQE,DIEMIWZ,AHCLCXE,BZYMPZI,JZISXJJ
...
The person__id is a UUID in my internal database, and is stored as a MemberID external ID in NGP. The occupation and employer are fairly straightforward. The name fields are randomly generated names that I use for the initial upload to MyCampaign, to prevent NGP from linking on names, so that NGP will create exactly 1 contact for every row uploaded. (In a second followup phase, I upload again with the correct name, linked by member ID). I hope you can see why I would expect there to be exactly one contact created for every uploaded row.
OK, so then one of the other uploaded CSVs, sheet_16.csv, looks like
person__id,voterfile_vanid,occupation,employer,prefix,first,middle,last,suffix,nickname
...
ba567c5b-c60a-4b16-aabf-f3c15551b681,228484,,,KQJYHRK,ESCSQPB,HXFOHQF,AOWJNXD,IDPGHWM,CGCIBV
....
Notice that:
- the person__ids are different.
- the voterfile_vanid are different by 1.
- all the other fields are different.
I uploaded in mode "Load new and/or make changes to existing contacts", and applied the mappings
- first name
- last name
- Voterfile VANID
- Apply External ID
I discovered this bug by looking at the successful uploads, and all of them have between 50-150 contacts that were "updated".
What I expected was that 0 contacts to be updated, and all of them to be created.
What happens is NGP somehow messes up and bogusly links/garbles these two records from sheet_01.csv and sheet_16.csv.
After the uploads, there is only a single contact created:
- has name ESCSQPB HXFOHQF AOWJNXD, from
sheet_16.csv - has voterfile VANID of 228484, from
sheet_16.csv - has Member ID 3646c0f3-39ba-4756-bf2f-c33f2109eedb, from
sheet_01.csvI would expected two contacts to be created, one for each row.
There is also another bug, which could be related, so I'll put it here too. Notice how the "complete" count for each upload is always just a few less than 39,998, the number which I would expect.
For example, if I look at the "Processed" count for sheet_01.csv, it says 39,994.
So, I clicked on that link to load that list of processed contacts.
Then I exported this list to debug, including the Member ID, call it processed_01.csv.
I compared this to sheet_01.csv, the 39,998 contacts that I uploaded.
Here is the weird thing:
out of the 39,994 member IDs in processed_01.csv, 150 weren't even in sheet_01.csv!
Those member IDs were actually coming from other uploads, such as sheet_05.csv.
So, it looks like your logic that ascribes each processed contact to an upload job is
wrong and scrambling things.
Since there doesn't seem to be good isolation between upload jobs that are happening in parrallel, this is why I am wondering if these two bugs are related.
Please have your engineering team reach out to me at n.crewsshipcreekgroup.com to schedule a call so I can help you fix this.