uhlhosting · March 6, 2016 09:57
diff --git a/cpanel analyst reply b/cpanel analyst reply
 Please disregard Terrance's reply above. He replied to the wrong ticket by mistake. 

 I am sorry to hear about the problems that you are having. I do see that Jesse was able to help and get MySQL back online for you. Your ticket was brought to my attention by one of our Level 1's on shift who has been monitoring the ticket at Jesse's request. 

 Please note as both Jesse and Kaden stated repairing corruption is not something cPanel can do because we are not experts in database corruption repair and would not want to cause any more damage to your data. 

 I see that you are still seeking some guidance on where to go from here in getting your sites back online and databases working. Since Jesse got MySQL/MariaDB reinstalled via our RPMs the database system itself is operational so this is good. 

 The problem is that the current databases are likely still corrupted beyond repair and if the guide to repairing InnoDB issues by forcing InnoDB recovery did not work, there is not much else that can be done. Have you tried removing the innodb=OFF option in my.cnf and trying to force recovery further. 

 If 'innodb_force_recovery=1' does not fix the corruption, the '1' should be incremented by one all the way to 6 until the corruption is repaired, per MySQL's documentation:

 http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html

 Please also note that "innodb_force_recovery" should *not* be present in /etc/my.cnf unless you are attempting recovery. A functional InnoDB server should not use innodb_force_recovery in its functional configuration. I see that this is still in my.cnf

 Can you try this. Can you try turnning InnoDB back on by removing the option Jesse added and try running through the recovery option. 

 innodb=OFF

 Essentially you would remove the above. Edit innodb_force_recovery=1 to innodb_force_recovery=2, then restart mysql and see if it comes back online. If it does not come back online change this to innodb_force_recovery=3 and restart, and continue this same pattern until you hit innodb_force_recovery=6 and restart each time checking if MySQL is accessible via the command line and if your sites are working. 

 If after doing the above you still run into problems we may need to consider, dropping the databases to completely get rid of any trace of it and the corruption in it, and restoring from backups. The last thing I would want is you to waste time trying to restore data on top of an already corrupted database. 

 I see that you have several backups under /backup going over the last week. Since these issues started occurring between March 5th and 6th I would recommend that we use the backups from March 3rd or one of the earlier two backups on the 1st or 28th

 I would also recommend restoring these backups in full and overwriting the account. 

 This can be done via SSH by going to /backup/ and choosing the date you wish to restore from. Then simply run the following for each account. 

 /scripts/restorepkg --force username.tar.gz

 You can also do this via WHM > Backups > Backup Restoration >> Restore by Date and select each account and add it to the restore queue. Then process the restore and it should do the same as the command above and replace the accounts, there files and the databases in full. 

 If you do not wish to restore the entire account what you can do is drop all of its databases and then restore each MySQL database. Then recreate an empty database with the same name via the users account. 

 You would then go to the same backup directory above and extract the backup for the user and navigate to the mysql directory and restore the databases with the command below

 mysql databasename < databasename.sql

 You should do this one by one and remove the extracted folder after extracting and restoring the databases to avoid issues with disk space. You should be okay however as the compressed backup will still exist. You do not want to remove that. 

 The goal here is that by dropping the database, creating a blank one and restoring or restoring the account over itself would be that it completely replaces the database and any corruption that exists at this time. Please note however it is possible that if the corruption existed prior to today / yesterday it could be reflected in these backups. I did extract one from March 3rd and it does appear to have mysql databases so you should be okay hopefully. 

 Please let us know if you have further questions. 

 Please note that I do not normally work today and was asked to hop on to address your concerns after your replies with Jesse. If for some reason I do not reply directly to you between now and 7am CST a Level 3 analyst will address your concerns further. Jesse will be back in the office at 7am on Monday should you have further questions for him. 

 Regards

 --
 Matthew Vetter
 Technical Analyst II
 cPanel, Inc.
	Please disregard Terrance's reply above. He replied to the wrong ticket by mistake.

	I am sorry to hear about the problems that you are having. I do see that Jesse was able to help and get MySQL back online for you. Your ticket was brought to my attention by one of our Level 1's on shift who has been monitoring the ticket at Jesse's request.

	Please note as both Jesse and Kaden stated repairing corruption is not something cPanel can do because we are not experts in database corruption repair and would not want to cause any more damage to your data.

	I see that you are still seeking some guidance on where to go from here in getting your sites back online and databases working. Since Jesse got MySQL/MariaDB reinstalled via our RPMs the database system itself is operational so this is good.

	The problem is that the current databases are likely still corrupted beyond repair and if the guide to repairing InnoDB issues by forcing InnoDB recovery did not work, there is not much else that can be done. Have you tried removing the innodb=OFF option in my.cnf and trying to force recovery further.

	If 'innodb_force_recovery=1' does not fix the corruption, the '1' should be incremented by one all the way to 6 until the corruption is repaired, per MySQL's documentation:

	http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html

	Please also note that "innodb_force_recovery" should not be present in /etc/my.cnf unless you are attempting recovery. A functional InnoDB server should not use innodb_force_recovery in its functional configuration. I see that this is still in my.cnf

	Can you try this. Can you try turnning InnoDB back on by removing the option Jesse added and try running through the recovery option.

	innodb=OFF

	Essentially you would remove the above. Edit innodb_force_recovery=1 to innodb_force_recovery=2, then restart mysql and see if it comes back online. If it does not come back online change this to innodb_force_recovery=3 and restart, and continue this same pattern until you hit innodb_force_recovery=6 and restart each time checking if MySQL is accessible via the command line and if your sites are working.

	If after doing the above you still run into problems we may need to consider, dropping the databases to completely get rid of any trace of it and the corruption in it, and restoring from backups. The last thing I would want is you to waste time trying to restore data on top of an already corrupted database.

	I see that you have several backups under /backup going over the last week. Since these issues started occurring between March 5th and 6th I would recommend that we use the backups from March 3rd or one of the earlier two backups on the 1st or 28th

	I would also recommend restoring these backups in full and overwriting the account.

	This can be done via SSH by going to /backup/ and choosing the date you wish to restore from. Then simply run the following for each account.

	/scripts/restorepkg --force username.tar.gz

	You can also do this via WHM > Backups > Backup Restoration >> Restore by Date and select each account and add it to the restore queue. Then process the restore and it should do the same as the command above and replace the accounts, there files and the databases in full.

	If you do not wish to restore the entire account what you can do is drop all of its databases and then restore each MySQL database. Then recreate an empty database with the same name via the users account.

	You would then go to the same backup directory above and extract the backup for the user and navigate to the mysql directory and restore the databases with the command below

	mysql databasename < databasename.sql

	You should do this one by one and remove the extracted folder after extracting and restoring the databases to avoid issues with disk space. You should be okay however as the compressed backup will still exist. You do not want to remove that.

	The goal here is that by dropping the database, creating a blank one and restoring or restoring the account over itself would be that it completely replaces the database and any corruption that exists at this time. Please note however it is possible that if the corruption existed prior to today / yesterday it could be reflected in these backups. I did extract one from March 3rd and it does appear to have mysql databases so you should be okay hopefully.

	Please let us know if you have further questions.

	Please note that I do not normally work today and was asked to hop on to address your concerns after your replies with Jesse. If for some reason I do not reply directly to you between now and 7am CST a Level 3 analyst will address your concerns further. Jesse will be back in the office at 7am on Monday should you have further questions for him.

	Regards

	--
	Matthew Vetter
	Technical Analyst II
	cPanel, Inc.