GitLab error OpenSSL::Cipher::CipherError () in app/controllers/admin/application_settings_controller.rb:40:in `update'

gitlab-error-encryption.md

The issue

It happened after secrets file was lost during GitLab upgrade.

The case described in documentation When the secrets file is lost

But not completely clear.

From log file /var/log/gitlab/gitlab-rails/production.log:

Started PATCH "/admin/application_settings" for ::1 at 2019-08-01 17:14:35 +0200
Processing by Admin::ApplicationSettingsController#update as HTML
  Parameters: {"utf8"=>"✓", "authenticity_token"=>"[FILTERED]", "application_setting"=>{"allow_local_requests_from_hooks_and_services"=>"[FILTERED]"}}
Completed 500 Internal Server Error in 47ms (ActiveRecord: 20.2ms)
  
OpenSSL::Cipher::CipherError ():
  
lib/gitlab/crypto_helper.rb:27:in `aes256_gcm_decrypt'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:55:in `get_token'
app/models/concerns/token_authenticatable_strategies/base.rb:27:in `ensure_token'
app/models/concerns/token_authenticatable_strategies/encrypted.rb:42:in `ensure_token'
app/models/concerns/token_authenticatable.rb:38:in `block in add_authentication_token_field'
app/services/application_settings/update_service.rb:18:in `execute'
app/controllers/admin/application_settings_controller.rb:40:in `update'

Issue reported on bug tracker (I just found it)

Getting 500 Internal error on /admin/application_settings after Gitlab upgrade [11.5.2->11.6.2]

Solution

Solution has been provided in one of the comments:

sudo gitlab-rails c


-------------------------------------------------------------------------------------
 GitLab:       11.7.0 (1d9280e)
 GitLab Shell: 8.4.4
 postgresql:   9.6.11
-------------------------------------------------------------------------------------
Loading production environment (Rails 5.0.7.1)
irb(main):001:0> 
irb(main):002:0> 
irb(main):003:0> settings = ApplicationSetting.last
......
irb(main):004:0> settings.update_column(:runners_registration_token_encrypted, nil)
=> true
irb(main):005:0> exit

gitlab-ctl restart

Proof:

Started PATCH "/admin/application_settings" for ::1 at 2019-08-01 17:52:17 +0200
Processing by Admin::ApplicationSettingsController#update as HTML
  Parameters: {"utf8"=>"✓", "authenticity_token"=>"[FILTERED]", "application_setting"=>{"allow_local_requests_from_hooks_and_services"=>"[FILTERED]"}}
Redirected to https://gitlab.company.com/admin/application_settings/network
Completed 302 Found in 64ms (ActiveRecord: 25.9ms)

Me help reset:
settings.update_column(:runners_registration_token_encrypted, nil)
settings.update_column(:encrypted_ci_jwt_signing_key, nil)

sudo gitlab-rails c
settings = ApplicationSetting.last
settings.update_column(:runners_registration_token_encrypted, nil)
settings.update_column(:encrypted_ci_jwt_signing_key, nil)
settings.save!

sudo gitlab-rails c
settings = ApplicationSetting.last
settings.update_column(:runners_registration_token_encrypted, nil)
settings.update_column(:encrypted_ci_jwt_signing_key, nil)
settings.save!

When I try to do this I get an Cipher error.
The few things I tried so far all failed

irb(main):001:0> settings = ApplicationSetting.last
=> #<ApplicationSetting id: 1, default_projects_limit: 100000, signup_enabled: true, gravatar_enabled: true, sign_in_text: nil, created_at: "2023-01-19 03:46:08.732363000 +0000", updated_at: "2023-01-19 03:46:10.815025000 +0000", home_page_url: nil, default_branch_pr...
irb(main):002:0> settings.update_column(:runners_registration_token_encrypted, nil)
=> true
irb(main):003:0> settings.update_column(:encrypted_ci_jwt_signing_key, nil)
=> true
irb(main):004:0> settings.save!
Traceback (most recent call last):
14: from (irb):4
13: from lib/gitlab/database/load_balancing/connection_proxy.rb:71:in transaction' 12: from lib/gitlab/database/load_balancing/connection_proxy.rb:119:in write_using_load_balancer'
11: from lib/gitlab/database/load_balancing/load_balancer.rb:118:in read_write' 10: from lib/gitlab/database/load_balancing/load_balancer.rb:200:in retry_with_backoff'
9: from lib/gitlab/database/load_balancing/load_balancer.rb:129:in block in read_write' 8: from lib/gitlab/database/load_balancing/connection_proxy.rb:120:in block in write_using_load_balancer'
7: from app/models/concerns/token_authenticatable.rb:49:in block in add_authentication_token_field' 6: from app/models/concerns/token_authenticatable_strategies/encrypted.rb:38:in ensure_token'
5: from app/models/concerns/token_authenticatable_strategies/base.rb:40:in ensure_token' 4: from app/models/concerns/token_authenticatable_strategies/encrypted.rb:117:in token_set?'
3: from app/models/concerns/token_authenticatable_strategies/encrypted.rb:78:in get_encrypted_token' 2: from app/models/concerns/token_authenticatable_strategies/encryption_helper.rb:18:in decrypt_token'
1: from lib/gitlab/crypto_helper.rb:28:in `aes256_gcm_decrypt'
OpenSSL::Cipher::CipherError ()

I get an Cipher error, too.

Traceback (most recent call last):
14: from (irb):5
13: from lib/gitlab/database/load_balancing/connection_proxy.rb:71:in transaction' 12: from lib/gitlab/database/load_balancing/connection_proxy.rb:119:in write_using_load_balancer'
11: from lib/gitlab/database/load_balancing/load_balancer.rb:118:in read_write' 10: from lib/gitlab/database/load_balancing/load_balancer.rb:193:in retry_with_backoff'
9: from lib/gitlab/database/load_balancing/load_balancer.rb:122:in block in read_write' 8: from lib/gitlab/database/load_balancing/connection_proxy.rb:120:in block in write_using_load_balancer'
7: from app/models/concerns/token_authenticatable.rb:49:in block in add_authentication_token_field' 6: from app/models/concerns/token_authenticatable_strategies/encrypted.rb:38:in ensure_token'
5: from app/models/concerns/token_authenticatable_strategies/base.rb:40:in ensure_token' 4: from app/models/concerns/token_authenticatable_strategies/encrypted.rb:117:in token_set?'
3: from app/models/concerns/token_authenticatable_strategies/encrypted.rb:78:in get_encrypted_token' 2: from app/models/concerns/token_authenticatable_strategies/encryption_helper.rb:18:in decrypt_token'
1: from lib/gitlab/crypto_helper.rb:28:in `aes256_gcm_decrypt'
OpenSSL::Cipher::CipherError ()

ref gitlab doc: When the secrets file is lost

I recently migrated from Centos Omnibus installation to sameersbn/gitlab on k8s. When I ran ApplicationSetting.current.reset_runners_registration_token! in Rails console, it did not work for me. So, I had to delete the application settings by following the below steps:

1. Enter the container:

kubectl exec -it -n <namespace> <pod name> -- bash

2. Enter the Rails console:

./bin/rails console

3. Run the following Ruby statements:

ApplicationSetting.first.delete
ApplicationSetting.first
exit

I recently migrated from Centos Omnibus installation to sameersbn/gitlab on k8s. When I ran ApplicationSetting.current.reset_runners_registration_token! in Rails console, it did not work for me. So, I had to delete the application settings by following the below steps:

1. Enter the container:

kubectl exec -it -n <namespace> <pod name> -- bash

2. Enter the Rails console:

./bin/rails console

3. Run the following Ruby statements:

ApplicationSetting.first.delete
ApplicationSetting.first
exit

thanks!

I recently migrated from Centos Omnibus installation to sameersbn/gitlab on k8s. When I ran ApplicationSetting.current.reset_runners_registration_token! in Rails console, it did not work for me. So, I had to delete the application settings by following the below steps:

1. Enter the container:

kubectl exec -it -n <namespace> <pod name> -- bash

2. Enter the Rails console:

./bin/rails console

3. Run the following Ruby statements:

ApplicationSetting.first.delete
ApplicationSetting.first
exit

This works for me. Thank you so much.