Skip to content

Instantly share code, notes, and snippets.

@Inkimar

Inkimar/gist:1e186a98726d212846161dee5bcafd80

Created November 9, 2022 14:09
Show Gist options
  • Save Inkimar/1e186a98726d212846161dee5bcafd80 to your computer and use it in GitHub Desktop.
Save Inkimar/1e186a98726d212846161dee5bcafd80 to your computer and use it in GitHub Desktop.
Download ZIP
2021-11-9, SNIC - biocache -> dr632
Raw
gistfile1.txt
1. load has been done
2. biocache> process -dr dr632
@Inkimar
Copy link
Author

Inkimar commented Nov 9, 2022 •
edited
Loading

Failed to connect to Zookeeper

solrcloud_zookeeper1.1.bkgihw11jnv1@uat-solr-1    | java.net.UnknownHostException: zookeeper3: Name or service not known
solrcloud_zookeeper1.1.bkgihw11jnv1@uat-solr-1    | 	at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
solrcloud_zookeeper1.1.bkgihw11jnv1@uat-solr-1    | 	at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929)
solrcloud_zookeeper1.1.bkgihw11jnv1@uat-solr-1    | 	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324)
solrcloud_zookeeper1.1.bkgihw11jnv1@uat-solr-1    | 	at java.net.InetAddress.getAllByName0(InetAddress.java:1277)
solrcloud_zookeeper1.1.bkgihw11jnv1@uat-solr-1    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1193)
solrcloud_zookeeper1.1.bkgihw11jnv1@uat-solr-1    | 	at java.net.InetAddress.getAllByName(InetAddress.java:1127)
solrcloud_zookeeper1.1.bkgihw11jnv1@uat-solr-1    | 	at java.net.InetAddress.getByName(InetAddress.java:1077)
solrcloud_zookeeper1.1.bkgihw11jnv1@uat-solr-1    | 	at org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer.recreateSocketAddresses(QuorumPeer.java:181)
solrcloud_zookeeper1.1.bkgihw11jnv1@uat-solr-1    | 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:591)
solrcloud_zookeeper1.1.bkgihw11jnv1@uat-solr-1    | 	at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:610)
solrcloud_zookeeper1.1.bkgihw11jnv1@uat-solr-1    | 	at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:838)
solrcloud_zookeeper1.1.bkgihw11jnv1@uat-solr-1    | 	at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:958)
solrcloud_zookeeper1.1.bkgihw11jnv1@uat-solr-1    | 2022-11-09 14:28:11,236 [myid:1] - INFO  [QuorumPeer[myid=1]/0.0.0.0:2181:FastLeaderElection@847] - Notification time out: 12800

@Inkimar
Copy link
Author

Inkimar commented Nov 9, 2022

Try to restart with this command:

  1. docker service update --force solrcloud_solr1
  2. docker service update --force solrcloud_solr4
  3. docker service update --force solrcloud_solr2
  4. docker service update --force solrcloud_solr3

Error when it comes to mounting the disk
docker service update --force solrcloud_zookeeper1
solrcloud_zookeeper1
overall progress: 0 out of 1 tasks
1/1: failed to mount local volume: mount /docker/var/volumes/solrcloud/data_zoo…
service update paused: update paused due to failure or early termination of task jt8o9waf5o7hprxhmzmclmwsk

@Inkimar
Copy link
Author

Inkimar commented Nov 9, 2022

2022-11-09

ingimar@uat-manager-1:~$ docker service ps solrcloud_zookeeper1
ID             NAME                         IMAGE              NODE            DESIRED STATE   CURRENT STATE             ERROR                              PORTS
71cr8nk8eqp3   solrcloud_zookeeper1.1       zookeeper:3.4.14   uat-solr-1      Running         Running 26 minutes ago                                       *:3888->3888/tcp,*:3888->3888/tcp,*:2888->2888/tcp,*:2888->2888/tcp,*:2181->2181/tcp,*:2181->2181/tcp
x0bvmny1q5ts    \_ solrcloud_zookeeper1.1   zookeeper:3.4.14   uat-manager-3   Shutdown        Rejected 26 minutes ago   "failed to mount local volume:…"   
zbo6mq2m4n6y    \_ solrcloud_zookeeper1.1   zookeeper:3.4.14   uat-manager-3   Shutdown        Rejected 26 minutes ago   "failed to mount local volume:…"   
qo17l309mlwb    \_ solrcloud_zookeeper1.1   zookeeper:3.4.14   uat-solr-4      Shutdown        Rejected 26 minutes ago   "failed to mount local volume:…"   
ql8rh45jsg5t    \_ solrcloud_zookeeper1.1   zookeeper:3.4.14   uat-solr-4      Shutdown        Rejected 26 minutes ago   "failed to mount local volume:…"

@Inkimar
Copy link
Author

Inkimar commented Nov 9, 2022

Jag kan se att disken finns på uat-solr1

                          Size  Used Avail Use% Mounted on
udev                                            16G     0   16G   0% /dev
tmpfs                                          3.2G  1.5M  3.2G   1% /run
/dev/vda1                                       20G   12G  7.3G  63% /
tmpfs                                           16G     0   16G   0% /dev/shm
tmpfs                                          5.0M     0  5.0M   0% /run/lock
tmpfs                                           16G     0   16G   0% /sys/fs/cgroup
/dev/loop1                                      56M   56M     0 100% /snap/core18/2620
/dev/mapper/vg_docker_local-lv_var_lib_docker   79G   19G   56G  25% /docker_var_lib
/dev/loop0                                      56M   56M     0 100% /snap/core18/2566
/dev/loop3                                      64M   64M     0 100% /snap/core20/1634
/dev/loop4                                      71M   71M     0 100% /snap/lxd/19647
/dev/loop5                                      68M   68M     0 100% /snap/lxd/22753
/dev/loop6                                      48M   48M     0 100% /snap/snapd/17336
/dev/loop7                                      48M   48M     0 100% /snap/snapd/17029
/dev/mapper/vg_docker_local-lv_docker_local    200G  1.7G  199G   1% /docker_local
/dev/vda15                                     105M  5.2M  100M   5% /boot/efi
uat-manager-1:/docker                          796G  491G  306G  62% /docker
/dev/loop8                                      64M   64M     0 100% /snap/core20/1695
tmpfs                                          3.2G     0  3.2G   0% /run/user/1700

@Inkimar
Copy link
Author

Inkimar commented Nov 9, 2022

Fungerar nu klockan 16:35 - 2021-11-09

biocache> index -dr dr632
There is an incremental row key file for this resource.  Would you like to perform an incremental process (y/n) ?
y
7e1638f6c05d 2022-11-09 15:33:48,009 INFO : [IndexRecords] - Starting the reindex by row key....
7e1638f6c05d 2022-11-09 15:33:48,102 INFO : [SolrIndexDAO] - Initialising the solr server zookeeper1:2181,zookeeper2:2181 cloudserver:null solrServer:null
7e1638f6c05d 2022-11-09 15:33:49,477 INFO : [TaxonSpeciesListDAO] - Loading list information...
7e1638f6c05d 2022-11-09 15:33:49,958 INFO : [TaxonSpeciesListDAO] - Loading data from dr174
7e1638f6c05d 2022-11-09 15:33:49,959 INFO : [TaxonSpeciesListDAO] - Downloading list data from https://lists.biodiversitydata.se/speciesListItem/downloadList/dr174?fetch=%7BkvpValues%3Dselect%7D
7e1638f6c05d 2022-11-09 15:33:50,301 INFO : [TaxonSpeciesListDAO] - Loading data from dr204
7e1638f6c05d 2022-11-09 15:33:50,301 INFO : [TaxonSpeciesListDAO] - Downloading list data from https://lists.biodiversitydata.se/speciesListItem/downloadList/dr204?fetch=%7BkvpValues%3Dselect%7D
7e1638f6c05d 2022-11-09 15:34:16,035 INFO : [TaxonSpeciesListDAO] - Loading list information...completed.
7e1638f6c05d 2022-11-09 15:34:16,343 INFO : [SpeciesGroups] - Using default legacy species groups
7e1638f6c05d 2022-11-09 15:34:24,219 INFO : [IndexRecords] - 1000 >> Last key : 4840f76f-f82c-4f65-b276-ae432b79eaec, records per sec: 27.617443
7e1638f6c05d 2022-11-09 15:34:29,020 INFO : [IndexRecords] - 2000 >> Last key : 50cb9031-f2ab-4d49-9eeb-334ce2943953, records per sec: 208.33333
7e1638f6c05d 2022-11-09 15:34:33,188 INFO : [IndexRecords] - 3000 >> Last key : 696978cb-183a-4387-9357-3be6e8420098, records per sec: 239.92322
7e1638f6c05d 2022-11-09 15:34:37,362 INFO : [IndexRecords] - 4000 >> Last key : dcc58958-561b-404e-99a0-f4ec286abbdc, records per sec: 239.63576
7e1638f6c05d 2022-11-09 15:34:41,972 INFO : [IndexRecords] - 5000 >> Last key : 07214217-36cf-4da8-81c0-0be5ab070a1f, records per sec: 216.9668
7e1638f6c05d 2022-11-09 15:34:46,001 INFO : [IndexRecords] - 6000 >> Last key : 522f08a8-9ce9-47e7-b589-5c2c710dfa81, records per sec: 248.20056
7e1638f6c05d 2022-11-09 15:34:50,078 INFO : [IndexRecords] - 7000 >> Last key : 524e2ff9-b0f3-423a-b5d9-13b1fe7ba6fa, records per sec: 245.33855
7e1638f6c05d 2022-11-09 15:34:54,052 INFO : [IndexRecords] - 8000 >> Last key : 613d9f3c-6f84-4e3a-a3ca-5ec8d4c935a3, records per sec: 251.63564
7e1638f6c05d 2022-11-09 15:34:57,989 INFO : [IndexRecords] - 9000 >> Last key : a11d46be-0227-4711-b4f7-75a8f6462474, records per sec: 254.0005
7e1638f6c05d 2022-11-09 15:35:01,950 INFO : [IndexRecords] - 10000 >> Last key : cced1cdc-77ff-4c33-bf4b-1ef18d6b5f41, records per sec: 252.4615
7e1638f6c05d 2022-11-09 15:35:06,087 INFO : [IndexRecords] - 11000 >> Last key : f214a31d-8b8a-4c01-9a2f-98db86dc630c, records per sec: 241.7795
7e1638f6c05d 2022-11-09 15:35:07,607 INFO : [SolrIndexDAO] - Performing index commit....
7e1638f6c05d 2022-11-09 15:35:11,397 INFO : [SolrIndexDAO] - Performing index commit....done
7e1638f6c05d 2022-11-09 15:35:11,544 INFO : [SolrIndexDAO] - >>>> Document count of index: 16343
7e1638f6c05d 2022-11-09 15:35:11,545 INFO : [SolrIndexDAO] - Shutting down the indexing...
7e1638f6c05d 2022-11-09 15:35:11,663 INFO : [SolrIndexDAO] - Finalise finished.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment