Skip to content

Instantly share code, notes, and snippets.

@johnko
Last active August 29, 2015 14:01
Show Gist options
  • Save johnko/f22c87931e31246598eb to your computer and use it in GitHub Desktop.
Save johnko/f22c87931e31246598eb to your computer and use it in GitHub Desktop.
These were my error logs. but it turns out Line 4 of var-log-mysql-error2.log was the hint I needed to get this working. /server/mysql/data/wsrep_recovery.lBRkgo could not be created because galera could not write to that directory
# grep mysql /etc/rc.conf
mysql_limits="YES"
mysql_dbdir="/server/mysql/data"
mysql_optfile="/server/mysql/my.cnf"
mysql_enable="YES"
mysql_args=""
# cat /server/mysql/my.cnf
[mysqld]
datadir = /server/mysql/data
tmpdir = /server/mysql/tmp
log-error = /var/log/mysql-error.log
slow-query-log-file = /var/log/mysql-slow.log
port = 3306
innodb_autoinc_lock_mode = 2
wsrep_sst_method = xtrabackup-v2
wsrep_provider = /usr/local/lib/libgalera_smm.so
wsrep_provider_options = "gmcast.listen_addr=tcp://127.0.0.1:4567"
wsrep_cluster_name = mycluster
wsrep_node_name = NXQCSO-NXQCSO
wsrep_node_address = 127.0.0.1
wsrep_slave_threads = 8
wsrep_sst_auth = "sst:25awfaSADF4toih4t"
server-id = 1
wsrep_cluster_address=gcomm://
# cat /var/log/mysql-error1.log
2014-05-26 00:37:59 85437 [Note] WSREP: Read nil XID from storage engines, skipping position init
2014-05-26 00:37:59 85437 [Note] WSREP: wsrep_load(): loading provider library 'none'
2014-05-26 00:37:59 85437 [Note] InnoDB: The InnoDB memory heap is disabled
2014-05-26 00:37:59 85437 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2014-05-26 00:37:59 85437 [Note] InnoDB: Compressed tables use zlib 1.2.3
2014-05-26 00:37:59 85437 [Note] InnoDB: Using CPU crc32 instructions
2014-05-26 00:37:59 85437 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2014-05-26 00:37:59 85437 [Note] InnoDB: Completed initialization of buffer pool
2014-05-26 00:37:59 85437 [Note] InnoDB: The first specified data file ./ibdata1 did not exist: a new database to be created!
2014-05-26 00:37:59 85437 [Note] InnoDB: Setting file ./ibdata1 size to 12 MB
2014-05-26 00:37:59 85437 [Note] InnoDB: Database physically writes the file full: wait...
2014-05-26 00:38:00 85437 [Note] InnoDB: Setting log file ./ib_logfile101 size to 48 MB
2014-05-26 00:38:01 85437 [Note] InnoDB: Setting log file ./ib_logfile1 size to 48 MB
2014-05-26 00:38:06 85437 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
2014-05-26 00:38:06 85437 [Warning] InnoDB: New log files created, LSN=45781
2014-05-26 00:38:06 85437 [Note] InnoDB: Doublewrite buffer not found: creating new
2014-05-26 00:38:07 85437 [Note] InnoDB: Doublewrite buffer created
2014-05-26 00:38:07 85437 [Note] InnoDB: 128 rollback segment(s) are active.
2014-05-26 00:38:07 85437 [Warning] InnoDB: Creating foreign key constraint system tables.
2014-05-26 00:38:07 85437 [Note] InnoDB: Foreign key constraint system tables created
2014-05-26 00:38:07 85437 [Note] InnoDB: Creating tablespace and datafile system tables.
2014-05-26 00:38:08 85437 [Note] InnoDB: Tablespace and datafile system tables created.
2014-05-26 00:38:08 85437 [Note] InnoDB: Waiting for purge to start
2014-05-26 00:38:08 85437 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.15-60.3 started; log sequence number 0
2014-05-26 00:38:09 85437 [Note] WSREP: Service disconnected.
2014-05-26 00:38:10 85437 [Note] WSREP: Some threads may fail to exit.
2014-05-26 00:38:10 85437 [Note] Binlog end
2014-05-26 00:38:10 85437 [Note] InnoDB: FTS optimize thread exiting.
2014-05-26 00:38:10 85437 [Note] InnoDB: Starting shutdown...
2014-05-26 00:38:12 85437 [Note] InnoDB: Shutdown completed; log sequence number 1625977
2014-05-26 00:38:12 86819 [Note] WSREP: Read nil XID from storage engines, skipping position init
2014-05-26 00:38:12 86819 [Note] WSREP: wsrep_load(): loading provider library 'none'
2014-05-26 00:38:12 86819 [Note] InnoDB: The InnoDB memory heap is disabled
2014-05-26 00:38:12 86819 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2014-05-26 00:38:12 86819 [Note] InnoDB: Compressed tables use zlib 1.2.3
2014-05-26 00:38:12 86819 [Note] InnoDB: Using CPU crc32 instructions
2014-05-26 00:38:12 86819 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2014-05-26 00:38:12 86819 [Note] InnoDB: Completed initialization of buffer pool
2014-05-26 00:38:12 86819 [Note] InnoDB: Highest supported file format is Barracuda.
2014-05-26 00:38:13 86819 [Note] InnoDB: 128 rollback segment(s) are active.
2014-05-26 00:38:13 86819 [Note] InnoDB: Waiting for purge to start
2014-05-26 00:38:13 86819 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.15-60.3 started; log sequence number 1625977
2014-05-26 00:38:13 86819 [Note] WSREP: Service disconnected.
2014-05-26 00:38:14 86819 [Note] WSREP: Some threads may fail to exit.
2014-05-26 00:38:14 86819 [Note] Binlog end
2014-05-26 00:38:14 86819 [Note] InnoDB: FTS optimize thread exiting.
2014-05-26 00:38:14 86819 [Note] InnoDB: Starting shutdown...
2014-05-26 00:38:16 86819 [Note] InnoDB: Shutdown completed; log sequence number 1625987
140526 00:38:16 mysqld_safe Starting mysqld daemon with databases from /server/mysql/data
140526 00:38:16 mysqld_safe Skipping wsrep-recover for empty datadir: /server/mysql/data
140526 00:38:16 mysqld_safe Assigning 00000000-0000-0000-0000-000000000000:-1 to wsrep_start_position
2014-05-26 00:38:17 0 [Note] WSREP: wsrep_start_position var submitted: '00000000-0000-0000-0000-000000000000:-1'
2014-05-26 00:38:17 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2014-05-26 00:38:17 88308 [Note] WSREP: Read nil XID from storage engines, skipping position init
2014-05-26 00:38:17 88308 [Note] WSREP: wsrep_load(): loading provider library '/usr/local/lib/libgalera_smm.so'
2014-05-26 00:38:17 88308 [Note] WSREP: wsrep_load(): Galera 3.5(rXXXX) by Codership Oy <[email protected]> loaded successfully.
2014-05-26 00:38:17 88308 [Note] WSREP: CRC-32C: using hardware acceleration.
2014-05-26 00:38:17 88308 [Warning] WSREP: Could not open saved state file for reading: /server/mysql/data//grastate.dat
2014-05-26 00:38:17 88308 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
2014-05-26 00:38:18 88308 [Note] WSREP: Passing config to GCS: base_host = 127.0.0.1; base_port = 4567; cert.log_conflicts = no; debug = no; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 1; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /server/mysql/data/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /server/mysql/data//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://127.0.0.1:4567; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.version = 0; pc.wait_prim = true; pc.
2014-05-26 00:38:18 88308 [Note] WSREP: Service thread queue flushed.
2014-05-26 00:38:18 88308 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
2014-05-26 00:38:18 88308 [Note] WSREP: wsrep_sst_grab()
2014-05-26 00:38:18 88308 [Note] WSREP: Start replication
2014-05-26 00:38:18 88308 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
2014-05-26 00:38:18 88308 [Note] WSREP: protonet asio version 0
2014-05-26 00:38:18 88308 [Note] WSREP: Using CRC-32C (optimized) for message checksums.
2014-05-26 00:38:18 88308 [Note] WSREP: backend: asio
2014-05-26 00:38:18 88308 [Note] WSREP: GMCast version 0
2014-05-26 00:38:18 88308 [Note] WSREP: (07f39324-e46e-11e3-a1da-f3840078dc28, 'tcp://127.0.0.1:4567') listening at tcp://127.0.0.1:4567
2014-05-26 00:38:18 88308 [Note] WSREP: (07f39324-e46e-11e3-a1da-f3840078dc28, 'tcp://127.0.0.1:4567') multicast: , ttl: 1
2014-05-26 00:38:18 88308 [Note] WSREP: EVS version 0
2014-05-26 00:38:18 88308 [Note] WSREP: PC version 0
2014-05-26 00:38:18 88308 [Note] WSREP: gcomm: connecting to group 'mycluster', peer ''
00:38:18 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https://bugs.launchpad.net/percona-xtradb-cluster
key_buffer_size=0
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 60961 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
140526 00:38:18 mysqld_safe mysqld from pid file /server/mysql/data/NXQCSO-NXQCSO.pid ended
# /usr/local/etc/rc.d/mysql-server start
# cat /var/log/mysql-error2.log
140526 00:44:04 mysqld_safe Starting mysqld daemon with databases from /server/mysql/data
140526 00:44:04 mysqld_safe WSREP: Running position recovery with --log_error='/server/mysql/data/wsrep_recovery.lBRkgo' --pid-file='/server/mysql/data/NXQCSO-NXQCSO-recover.pid'
2014-05-26 00:44:05 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
140526 00:44:07 mysqld_safe WSREP: Recovered position 00000000-0000-0000-0000-000000000000:-1
2014-05-26 00:44:08 0 [Note] WSREP: wsrep_start_position var submitted: '00000000-0000-0000-0000-000000000000:-1'
2014-05-26 00:44:08 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2014-05-26 00:44:08 63584 [Note] WSREP: Read nil XID from storage engines, skipping position init
2014-05-26 00:44:08 63584 [Note] WSREP: wsrep_load(): loading provider library '/usr/local/lib/libgalera_smm.so'
2014-05-26 00:44:08 63584 [Note] WSREP: wsrep_load(): Galera 3.5(rXXXX) by Codership Oy <[email protected]> loaded successfully.
2014-05-26 00:44:08 63584 [Note] WSREP: CRC-32C: using hardware acceleration.
2014-05-26 00:44:08 63584 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
2014-05-26 00:44:08 63584 [Note] WSREP: Passing config to GCS: base_host = 127.0.0.1; base_port = 4567; cert.log_conflicts = no; debug = no; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 1; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /server/mysql/data/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /server/mysql/data//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = tcp://127.0.0.1:4567; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.version = 0; pc.wait_prim = true; pc.
2014-05-26 00:44:08 63584 [Note] WSREP: Service thread queue flushed.
2014-05-26 00:44:08 63584 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
2014-05-26 00:44:08 63584 [Note] WSREP: wsrep_sst_grab()
2014-05-26 00:44:08 63584 [Note] WSREP: Start replication
2014-05-26 00:44:08 63584 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
2014-05-26 00:44:08 63584 [Note] WSREP: protonet asio version 0
2014-05-26 00:44:08 63584 [Note] WSREP: Using CRC-32C (optimized) for message checksums.
2014-05-26 00:44:08 63584 [Note] WSREP: backend: asio
2014-05-26 00:44:08 63584 [Note] WSREP: GMCast version 0
2014-05-26 00:44:08 63584 [Note] WSREP: (d87b6a66-e46e-11e3-add3-2375d3f17430, 'tcp://127.0.0.1:4567') listening at tcp://127.0.0.1:4567
2014-05-26 00:44:08 63584 [Note] WSREP: (d87b6a66-e46e-11e3-add3-2375d3f17430, 'tcp://127.0.0.1:4567') multicast: , ttl: 1
2014-05-26 00:44:08 63584 [Note] WSREP: EVS version 0
2014-05-26 00:44:08 63584 [Note] WSREP: PC version 0
2014-05-26 00:44:08 63584 [Note] WSREP: gcomm: connecting to group 'mycluster', peer ''
...
...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment