Partners sadtime

gistfile1.md

Day 2 begins! Day 1 notes here

https://technonaturalist.net/blog/2011/01/drupal-7-notes-dealing-ghost-fields

https://technonaturalist.net/comment/reply/43/1223 -- Comment from Jen Lampton Nov 2013 Running cron once doesn’t always do the trick. If it doesn’t work the first time, you may need to run cron 10 or 20 times to get the fields to actually be deleted. Its faster to just manually delete the two tables (field_data and field_revision) as well as the two records from field_instance and field_instance_config. Still not resolved as of the latest version of D7.

https://drupal.org/comment/5865292#comment-5865292

• Delete the records from field_config / field_config_instance

Staging DB 11:28am Sunday. Dev branch. Let's get a lay of the land as far as what needs to be deleted as is, without any changes. Feeling like there's a conflict because we deleted field_partners before purging the field_deleted tables, which is why it says "cannot rename."

Data tables:

• field_data_field_partners | 49 rows
• field_data_field_partners_fc | 0 rows today

Deleted tables:

• field_deleted_data_82 | field_image_partner_target_id | 0 rows
• field_deleted_data_83 | field_is_sponsor_value | 0
• field_deleted_data_84 | field_partner_target_id | 0
• field_deleted_data_85 | field_partner_copy_value, copy_format | 0
• field_deleted_data_87 | field_partners_value, field_partners_revision_id | 0

SELECT * 
FROM field_config 
WHERE deleted = 1;
#0 rows;

SELECT * 
FROM field_config_instance
WHERE deleted = 1;
#0 rows;

SELECT * 
FROM field_config
WHERE field_name LIKE '%partner%';
#5 rows, expected fields and data types.

SELECT * 
FROM field_config_instance
WHERE field_name LIKE '%partner%';
#same

First, disable apachesolr.

Do you really want to continue? (y/n): y
apachesolr was disabled successfully.                                [ok]
apachesolr_search was disabled successfully.                         [ok]
dosomething_search was disabled successfully.                        [ok]
vagrant@dev:/vagrant/html$ 

Ok, so there's two issues here, one more critical than the other.

1. field_data_partners has weird data in it, which is what's fucking up the campaign preprocess
2. ghost tables aren't deleted
   #1 may be solvable by just deleting all previous traces of field_partners data.

I think #2 was coming into play because I was trying to delete field_partners to just re-do the whole thing as field_partners_fc.

So, all we've done so far is disable apachesolr. Let's see what happens on field_purge_batch, if we get errors.

drush php-eval "field_purge_batch(1000);"

it's taking a while.. and completed with no errors. let's try this again, to see if they actually get purged after 10-20 tries as per suggested here https://technonaturalist.net/comment/reply/43/1223

2nd time - much quicker. the tables all still exist
3rd time: quick. all tables are still there. still, i want to see if this will work at all. let's try `

drush cron

taking a long while.

In the meantime, one thing that's bizarre is why Partner Copy/Value are fields in field_deleted_data_85. Also for field_deleted_data_87 -- that very much feels like it was created because of me deleting the field locally, and possibly apachesolr writing the datatable. for the same reason that the field_partners_fc data table exists (was never deployed to stage environment). FYI partners_fc datatable still exists as well. And no record of it in field_config or field_config_instance.

Ok,
Cron run successful.

Refresh tables in SequelPro and i still see the ghost tables. No dice. let's try to run cron again.

2nd drush cron
Cron run successful, much quicker. Tables still there.
3rd - still there.
4th- still there.
5th - still there
6th - still there.

Ok. Let's try field_purge_batch(10) for shits n gigs.

drush php-eval "field_purge_batch(10);"

quick. no they are still there.

fuck it, lets get up to 20.
7th cron - no
8 th
9th
10th
11th
12th
13th

This guy looks like the guy who actually takes care of cleanup -- field_sql_storage_field_storage_purge_field.

14th
15th

Ok that game is not fun. Let's look into why this data is fucked in field_partners.

For node 43:

field_partners (Array, 1 element)
und (Array, 1 element)
0 (Array, 2 elements)
value (String, 3 characters ) 493
revision_id (String, 4 characters ) 2111

SELECT * 
FROM field_collection_item
WHERE item_id = 493;

493 2111 field_partners 0 6c038f33-cf4c-41bf-bd9f-1050f766bef1

OK. Weird.
Let's try to save a new node without filling in the Partners FC, because last I remember it'd create a crazy value there.

Right. new node, 589.
There's a value in field_partners for it.
There is not a value in say field_faq. So that's what's fucking this up.

Let's attempt this next. Manually truncate the field_deleted tables, and then attempt to get rid of this evil field_partners field. I suppose this begs the question of the best way to delete a field captured in features.

I think what was happening yesterday is that i'd do a drush -y fra, and it wouldn't be able to delete the field_partners because it's already been deleted. Actually let's test that, we found this interesting function _field_sql_storage_tablename

Here goes the test. Switched to dev branch, and added an install hook to remove the field.

Running the install hook:

vagrant@dev:/vagrant/html$ drush updb
 Dosomething_taxonomy  7001  Remove field_partners.
Do you wish to run all pending updates? (y/n): yy
Do you wish to run all pending updates? (y/n): y
Cannot rename <em class="placeholder">field_data_field_partners</em> [error]
to <em class="placeholder">field_deleted_data_87</em>: table <em
class="placeholder">field_deleted_data_87</em> already exists.
Performed update: dosomething_taxonomy_update_7001                   [ok]
'all' cache was cleared.                                             [success]
Finished performing updates. 

I expected that. Let's look at the actual field_partners table.

It exists, and all the "deleted" values are set to 1. Still only 10 field_deleted tables.

Let's take a look at the node 589 now. Ok, there's no more field_partners in the array. That's promising.

SELECT * 
FROM field_config
WHERE field_name LIKE '%partner%';

^^ What's interesting here is that all the field_config records have deleted = 0.

SELECT * 
FROM field_config_instance
WHERE field_name LIKE '%partner%';

^^ In INSTANCE, field_partners.deleted = 1 but all of the other guys have deleted = 0.

Once you delete a field collection instance, you aren't able to delete the rest of the fields in it. My bet is that deleting the field collection won't be deleting its child fields, resulting in orphans... sad :(

Creating a new node, and works fine. No record inserted into field_partners.

Let's try to see what field_purge_batch does, if we still get the "cannot rename table" error.

 drush php-eval "field_purge_batch(10);"

it completes with no errors. We still have a field_data_field partners with all deleted values=1 and 10 revision tables. Nothing changed.

Beginning to think may need to delete all child fields first and then attempt to delete field_partners, AFTER first manually dropping all of the field_deleted _ tabesl

This sounds a lot like this: https://drupal.org/comment/8012101#comment-8012101

Going to test patch to field_collection http://drupalcode.org/project/field_collection.git/commit/ded2a96

field_collection cloned from                                         [ok]
http://git.drupal.org/project/field_collection.git.
Checked out revision a31d81a55ffaf7f83db607ed1c802bd8c17441d4.

 Dosomething_taxonomy  7001  Remove field_partners.                           
 Field_collection      7003  Update field_collection_field_schema columns for 
                             all tables.

Performed update: field_collection_update_7003   

Taking forever here....

WD Apache Solr: HTTP Status: 0; Message: Request failed: Connection  [error]
timed out; Response: ; Request: Unknown; Caller:
call_user_func_array() (line Unknown of Unknown)
WD Apache Solr: HTTP 0; Request failed: Connection timed out         [error]
Performed update: dosomething_taxonomy_update_7001   

Interesting...

No shit. THere's 8 tables now, 83, 84, 85, 87.

• field_deleted_data_83 | field_is_sponsor_value | 39 records

field_collection_item field_partners 1 369 2070 und 0 0

• field_deleted_data_84 | field_partner_target_id | 0 records
• field_deleted_data_85 | field_partner_copy_value | 0 records
• field_deleted_data_87 | field_partners_value, field_partners_value_revision_id | 37 record

node campaign 1 19 752 und 0 408 2065
node campaign 1 43 758 und 0 493 2111 -- this matches wack data from above with value 493

Promising. The field_partners table is removed.

node/43 is taking forever to load.

In field_config and field_config_instance we have all of the orphaned children.

disabling apachesolr to see what happens with running the purge.

node/43 loads! no error. same with node/73

Refreshed tables and now we're down to two deleted tables: 83 and 87. that's weird. They both have data still. Let's try

drush php-eval "field_purge_batch(10);"

Taking a while....completed. Huh. They are still there but less rows. Oh, duh. the batch_size is the records to run. Motherfucker.

drush php-eval "field_purge_batch(100);"

Ok, still less data but still tables.

drush php-eval "field_purge_batch(1000);"

SHAZAM THem tables SHIT IS GONE MOTHERFUCKER

Ok, so here's the final test. Let's see if that field_partners_fc renaming shit will work now, because we can properly delete field_partners.

Downloading stage, and will run a ds build from the partners_fix branch to verify.

HERE IT GOES

Added the administrator role to uid 1                                [success]
Updating database...
 Field_collection  7003  Update field_collection_field_schema columns for all 
                         tables.                                              
 Dosomething_taxo  7001  Drops evil forsaken field_partners field_collection  
 nomy                    field.
Do you wish to run all pending updates? (y/n): y
WD Apache Solr: HTTP Status: 0; Message: Request failed: Connection  [error]
timed out; Response: ; Request: Unknown; Caller:
call_user_func_array() (line Unknown of Unknown)
WD Apache Solr: HTTP 0; Request failed: Connection timed out         [error]
Performed update: dosomething_taxonomy_update_7001                   [ok]
Performed update: field_collection_update_7003                       [ok]

-- Possibly should disable apachesolr locally before running build scripts. --

Interesting. 4 deleted tables, 83, 84, 85, 87.
87 is old field_partners, with field_partners_value, and field_partners_revision_id.

5 rows left in field_deleted_data_87
20 rows in field_deleted_revision_87

we have "deleted" records in field_config and field_config_instance, which would make sense because features haven't been reverted to inform of the new partners_fc.

build finished.

http://dev.dosomething.org:8888/node/43 works! here's the moment of truth

drush -y fra

Not a surprise. This probably happens because there's no corresponding record in field_config / field_config_instance. Let's see if they exist now, and if everything's working.

array_merge(): Argument #1 is not an array image.module:586          [warning]
The following modules will be reverted: dosomething_taxonomy, dosomething_image, dosomething_campaign, dosomething_static_content
Do you really want to continue? (y/n): y
WD php: DatabaseSchemaObjectExistsException: Table                   [error]
field_data_field_partners_fc already exists. in
DatabaseSchema->createTable() (line 657 of
/vagrant/html/includes/database/schema.inc).
Cannot modify header information - headers already sent by (output   [warning]
started at /opt/drush-6.2.0/includes/output.inc:38)
bootstrap.inc:1217
DatabaseSchemaObjectExistsException: Table <em class="placeholder">field_data_field_partners_fc</em> already exists. in DatabaseSchema->createTable() (line 657 of /vagrant/html/includes/database/schema.inc).
Drush command terminated abnormally due to an unrecoverable error.

Not seeing records in field_config or field_config_instance for field_partners_fc, and all those field_partners records in field_config/config_instance are still marked to be deleted.

Let's drop tables and drush -y fra again.

vagrant@dev:/vagrant/html$ drush -y fra
The following modules will be reverted: dosomething_image, dosomething_campaign, dosomething_static_content
Do you really want to continue? (y/n): y
Reverted dosomething_image.image.                                    [ok]
Reverted dosomething_campaign.field_instance.                        [ok]
Reverted dosomething_static_content.user_permission.                 [ok]
Reverted dosomething_static_content.variable.                        [ok]

(not seeing taxonomy changing)

vagrant@dev:/vagrant/html$ drush -y fra
The following modules will be reverted: dosomething_taxonomy, dosomething_image, dosomething_campaign
Do you really want to continue? (y/n): y
Reverted dosomething_taxonomy.field_base.                            [ok]
Reverted dosomething_taxonomy.field_instance.                        [ok]
Reverted dosomething_taxonomy.user_permission.                       [ok]
Reverted dosomething_image.image.                                    [ok]
Reverted dosomething_campaign.field_instance.                        [ok]
vagrant@dev:/vagrant/html$ 

This looks good. the partners_fc tables are in the database now.

Let's do one more drush -y fra. everything should be up to date.

Do you really want to continue? (y/n): y
Reverted dosomething_image.image.

uh ok. that's weird but let's try to hit up node 43 again.

it works.

so it seems this is the jam.

1. ds build -- grabs dev branch of field_collection, ds module updates, drush updb
2. DROP TABLE field_data_field_partners_fc; DROP TABLE field_revision_field_partners_fc;
3. drush -y fra
4. drush php-eval "field_purge_batch(1000);"

Last step will remove all the field_deleted_ tables and remove all records from field_config and field_config_instance where deleted = 1.

nope, sorry! just created a new campaign node. didn't fill out the parners info. just got a

EntityMetadataWrapperException: Invalid data value given. Be sure it matches the required data type and format. in EntityDrupalWrapper->set() (line 736 of /vagrant/html/profiles/dosomething/modules/contrib/entity/includes/entity.wrapper.inc).

AND i didn't fill out partners info. AND there's data in the field_partners_fc with value 568. what the fuck.

Fuck. Was it just this the whole time?

https://drupal.org/node/1239946

someone says here is the answer https://drupal.org/node/1498746#comment-6734208 -- which says both modules should be updated to latest. and, now realizing we need to upgrade entity from 1.2 to 1.3.

so. next test is to see if i remove video_id from the partners_fc field collection, should i be able to successfully save without a ghost entity id value in field_partners_fc ?

either way we'll want to delete field_partners because it's full of bogus data now. so, all of that work around deleting a field_collectoin field is still relevant. (tell yourself that)

Removing video from the field_partners_fc field collection. It'd make sense if this fixes it, we probably haven't used any FC's with default values set.

The field Video has been deleted from the Field collection field_partners_fc content type.

Ok lets try this again. node/add/campaign

No I still get this shit

EntityMetadataWrapperException: Invalid data value given. Be sure it matches the required data type and format. in EntityDrupalWrapper->set() (line 736 of /vagrant/html/profiles/dosomething/modules/contrib/entity/includes/entity.wrapper.inc).

Lets try to upgrade entity and just grab the latest version of field collection. Fuck!

Attempted build on same db with updating to Entity 1.3 and the latest commit on field collection.

Updating database...
 Field_collection  7004  Add index on {$field_collection_field}_revision_id 
                         column for all tables.
Do you wish to run all pending updates? (y/n): y
Performed update: field_collection_update_7004                       [ok]
'all' cache was cleared.                                             [success]
Finished performing updates.                                         [ok]
Clearing caches...

no, nothing. still get a false entity upon creating a new campaign.

new plan. just remove the field_partners for good. try to rebuild the field collection over again but WTF.

i suppose i need to now test to see what happens just deleting the whole thing and never adding the partners_fc in.

Testing remove_partners branch with staging db. Upgraded entity to 1.3 and field_collection to latest dev commit.

Plan is to updb and then see if at least all is well with completely removing this guy. Tackle the incorrect default entity another day.

This is what's crazy. I just grabbed the latest stage db and now there are no field_deleted tables present. This is while drush updb is runnig but I haven't answered yes yet. This is where I seem to be magically manipulating the database on staging by doing all this stuff locally.

Do you really want to continue? (y/n): y
vagrant@dev:/vagrant$ cd /vagrant/html
vagrant@dev:/vagrant/html$ drush updb
 Field_collection  7003  Update field_collection_field_schema columns for all 
                         tables.                                              
 Field_collection  7004  Add index on {$field_collection_field}_revision_id   
                         column for all tables.                               
 Dosomething_taxo  7001  Remove field_partners.                               
 nomy
Do you wish to run all pending updates? (y/n): y

Proof. The update hasn't run yet. Maybe it's cron running on staging that's purging. But.. I don't know. I still have my mystery partners_fc table which exists and still hasn't been committed into dev.

^^ Should have tested on deleting the is_sponsored field too. That could be the one that's providing the default value that's creating the false entity. (only tried deleting the video_id field)

http://dev.dosomething.org:8888/node/43 works. seeing no deleted tables. lets try the add a few campaigns trick

here we go, update from stage then disable apachesolr. let's also test a record on staging db to maek sure deleted tables don't exist. Aaron Test 1010

ds update-db

Actually just looking at a node 591 created and there's a "Learn more about " link... meaning there's probably a bogus entity for "More Facts" too. Already did a ds update-db but worth looking into. Might be happening on a few different nodes?

Confirmed no more field_deleted tables in new db pull, and updated campaign title is Aaron Test 1010" . All partners tables exist in the db, including partners_fc

drush -y dis apachesolr

drush -y en devel

drush updb

Took a lonnnng time to ask if i really want to continue. YES

Pretty sure that this guy is a separate issue https://drupal.org/node/1239946 that started the whole thing.

vagrant@dev:/vagrant/html$ drush updb
Field_collection 7003 Update field_collection_field_schema columns for all
tables.
Field_collection 7004 Add index on {$field_collection_field}_revision_id
column for all tables.
Dosomething_taxo 7001 Remove field_partners.
nomy
Do you wish to run all pending updates? (y/n): y
Performed update: field_collection_update_7003 [ok]
Performed update: dosomething_taxonomy_update_7001 [ok]
Performed update: field_collection_update_7004 [ok]
'all' cache was cleared.

drush -y fra

array_merge(): Argument #1 is not an array image.module:586          [warning]
The following modules will be reverted: dosomething_taxonomy, dosomething_image, dosomething_campaign, dosomething_static_content
Do you really want to continue? (y/n): y
Reverted dosomething_taxonomy.user_permission.                       [ok]
Reverted dosomething_image.image.                                    [ok]
Reverted dosomething_campaign.variable.                              [ok]
Reverted dosomething_static_content.user_permission.                 [ok]
Reverted dosomething_static_content.variable.                        [ok]
vagrant@dev:/vagrant/html$ 

Multiple ndoe creation seems ok. Going to try save existing.

the deleted tables are back!

vagrant@dev:/vagrant/html$ drush -y dis apachesolr
The following extensions will be disabled: apachesolr, apachesolr_search, dosomething_search
Do you really want to continue? (y/n): y
apachesolr was disabled successfully.                                [ok]
apachesolr_search was disabled successfully.                         [ok]
dosomething_search was disabled successfully.                        [ok]
vagrant@dev:/vagrant/html$ drush -y updb
Killed
vagrant@dev:/vagrant/html$ drush -y updb
 Field_collection  7003  Update field_collection_field_schema columns for all 
                         tables.                                              
 Field_collection  7004  Add index on {$field_collection_field}_revision_id   
                         column for all tables.                               
 Dosomething_taxo  7001  Remove field_partners.                               
 nomy
Do you wish to run all pending updates? (y/n): y
Performed update: field_collection_update_7003                       [ok]
Cannot rename <em class="placeholder">field_data_field_partners</em> [error]
to <em class="placeholder">field_deleted_data_87</em>: table <em
class="placeholder">field_deleted_data_87</em> already exists.
Performed update: dosomething_taxonomy_update_7001   

this is fun.

maybe the shitty fix is just that when you check to see if partners exist, you have to actually check and see if the term tid is set. and accept that it just creates empty field_collection_item id's for every single node.