conversation.txt

Junaid Farooq  1:39 PM
Hi Chris
leveldb/table: corruption on data-block (pos=0): checksum mismatch, want=0x20258dff got=0x426a0169 [file=14883696.ldb]

Junaid Farooq  8:36 AM
When I try to move this file 14883696.ldb
from _main folder, filer dont work
8:37
and after a restart it works fine but a few hours after, this error starts to come again
8:37
I have read over the internet on GH , if you delete this file, everything will be fine.
I removed that fine and then filer just waits to connect with 18888 port.
8:37
can you give me help on this?

Junaid Farooq  8:49 AM
is there anyway to clean this block?

Chris  9:32 AM
I do not really know the internal details about leveldb.
Do you still have the original copy before any move?


Junaid Farooq  9:32 AM
filer.zip

Junaid Farooq  9:33 AM
I still have the same copy and this is just ldbs from _main, I just moved the file and started it but filer didnt connect to 1888 port.
9:33
and then I moved the file back. and it started.

Chris  9:33 AM
did you shutdown the existing filer before moving?

Junaid Farooq  9:35 AM
hmm, I actually just moved the file.
and the restarted it.
but I didnt shut down. before moving
9:38
should I try this?
9:38
this is my current version 30GB 2.79

Junaid Farooq  9:46 AM
I stopped filer and then moved the file now its loading, lets see
9:46
wait to connect to filer 88.99.139.88:8888 grpc address 88.99.139.88:18888
Dec 30 05:46:16 Ubuntu-2004-focal-64-minimal seaweedfs[2885284]: I1230 05:46:16 85284 s3.go:162] wait to connect to filer 88.99.139.88:8888 grpc address 88.99.139.88:18888
Dec 30 05:46:17 Ubuntu-2004-focal-64-minimal seaweedfs[2885284]: I1230 05:46:17 85284 s3.go:162] wait to connect to filer 88.99.139.88:8888 grpc address 88.99.139.88:18888
Dec 30 05:46:18 Ubuntu-2004-focal-64-minimal seaweedfs[2885284]: I1230 05:46:18 85284 s3.go:162] wait to connect to filer 88.99.139.88:8888 grpc address 88.99.139.88:18888
Dec 30 05:46:19 Ubuntu-2004-focal-64-minimal seaweedfs[2885284]: I1230 05:46:19 85284 s3.go:162] wait to connect to filer 88.99.139.88:8888 grpc address 88.99.139.88:18888
Dec 30 05:46:20 Ubuntu-2004-focal-64-minimal seaweedfs[2885284]: I1230 05:46:20 85284 s3.go:162] wait to connect to filer 88.99.139.88:8888 grpc address 88.99.139.88:18888
Dec 30 05:46:21 Ubuntu-2004-focal-64-minimal seaweedfs[2885284]: I1230 05:46:21 85284 s3.go:162] wait to connect to filer 88.99.139.88:8888 grpc address 88.99.139.88:18888
Dec 30 05:46:22 Ubuntu-2004-focal-64-minimal seaweedfs[2885284]: I1230 05:46:22 85284 s3.go:162] wait to connect to filer 88.99.139.88:8888 grpc address 88.99.139.88:18888
Dec 30 05:46:23 Ubuntu-2004-focal-64-minimal seaweedfs[2885284]: I1230 05:46:23 85284 s3.go:162] wait to connect to filer 88.99.139.88:8888 grpc address 88.99.139.88:18888
Dec 30 05:46:24 Ubuntu-2004-focal-64-minimal seaweedfs[2885284]: I1230 05:46:24 85284 s3.go:162] wait to connect to filer 88.99.139.88:8888 grpc address 88.99.139.88:18888
Dec 30 05:46:25 Ubuntu-2004-focal-64-minimal seaweedfs[2885284]: I1230 05:46:25 85284 s3.go:162] wait to connect to filer 88.99.139.88:8888 grpc address 88.99.139.88:18888
Dec 30 05:46:26 Ubuntu-2004-focal-64-minimal seaweedfs[2885284]: I1230 05:46:26 85284 s3.go:162] wait to connect to filer 88.99.139.88:8888 grpc address 88.99.139.88:18888
9:46
this is what it says after loading.
9:48
So I replaced the same file there again.. and now it says
5] connected to filer 88.99.139.88:8888 grpc address 88.99.139.88:18888
9:49
this file file=14883696.ldb
is in /storage/filer/_main
is there any connection of filer to such file?

Chris  9:57 AM
these two are not related to each other.

Junaid Farooq  9:58 AM
this seems like to be connected then
9:58
can you reproduce it? while replacing a file from _main?

Chris  9:58 AM
may just mean the filer is not started up successfully

Junaid Farooq  10:00 AM
yes but after removing this file only

Junaid Farooq  10:09 AM
I think you can easily reproduce it, while moving a file from _main folder, any ldb file.

Junaid Farooq  10:38 AM
I think this needs to be fixed, and I am pretty sure you can reproduce it while just moving one file.

Chris  10:43 AM
maybe need to ask leveldb library to support this. I do not understand why needs to take one ldb file and treat it as a db.

Junaid Farooq  10:43 AM
hmm but can you reproduce it?

Junaid Farooq  7:59 AM
Hi Chris that issue was gone after 2 3 restarts but it has come in again.
8:01
were you able to to reproduce?

Chris  8:31 AM
this is not supposed to work.

Junaid Farooq  8:31 AM
My only bet right now is to have a new filer by this end of the day today
8:32
because I see there is no fix available to that issue
8:32
or make that file read only? that one leveldb file?

Chris  8:33 AM
need to shutdown before copying the leveldb data.

Junaid Farooq  8:33 AM
I actually shut down and moved the file and the started the filer, filer didnct connect with 18888 (edited) 
8:34
without that file filer wont start. Its very easy if you can reproduce it, just start the filer. upload a file
stop the filer
remove any ldb file
start filer again
it wont work

Chris  8:38 AM
what is your command to “remove any ldb file”?

Junaid Farooq  8:38 AM
remove= mv
8:38
mv /storage/filer/_main/xxxxx.ldb /root/
8:38
thats it.
8:39
/opt/go/bin/weed -v=0 server -garbageThreshold=0.1 -s3 -ip=88.99.139.88 -volume.port=8081 -dir=/storage -volume.max=1000 -volume.images.fix.orientation=false
8:39
this is how it started.

Chris  8:39 AM
what is your version?

Junaid Farooq  8:39 AM
30GB 2.79

Chris  8:39 AM
what is the filer.toml?

Junaid Farooq  8:40 AM
root@Ubuntu-2004-focal-64-minimal ~ # cat /etc/seaweedfs/filer.toml 
[memory]
# local in memory, mostly for testing purpose
enabled = false

[leveldb3]
# local on disk, mostly for simple single-machine setup, fairly scalable
enabled = true
dir = "/storage/filer"

Chris  8:44 AM
the “/storage/filer” is the leveldb folder. you can not just copy “/storage/filer/_main/xxxxx.ldb”, which is one part of it.

Junaid Farooq  8:44 AM
i didnnt copy anything, I just moved a file
8:44
which is corrupt
8:44
leveldb/table: corruption on data-block (pos=0): checksum mismatch, want=0x20258dff got=0x426a0169 [file=14883696.ldb]
8:45
file=14883696.ldb
 this one
8:45
what do you mean by copy?
I just move that one file to another directory so it cannot be present in /storage/filer thats it.

Chris  8:47 AM
saw this https://github.com/ethereum/go-ethereum/issues/20062
#20062 err="leveldb/table: corruption on data-block
Hi there,
please note that this is an issue tracker reserved for bug reports and feature requests.
For general questions please use the gitter channel or the Ethereum stack exchange at https://ethereum.stackexchange.com.
System information
Geth  
Version: 1.9.3-stable  
Git Commit: cfbb969da803d4cc92e1a64fc1b3c06db299b564  
Architecture: amd64  
Protocol Versions: [63]  
Show more
Comments
3
<https://github.com/ethereum/go-ethereum|ethereum/go-ethereum>ethereum/go-ethereum | Sep 12th, 2019 | Added by GitHub

Junaid Farooq  8:48 AM
I did a check of disk as well

Chris  8:54 AM
seems you met with this problem before :slightly_smiling_face: https://github.com/syndtr/goleveldb/issues/184
#184 Can not start filer in dir /storage/filer : leveldb/storage: corrupted or incomplete meta file.
Just as a brief history of the issue,
• chrislusf/seaweedfs#576
• chrislusf/seaweedfs#578
we are using SeaweedFS, for image storage, and this tool is using your repo for handling all leveldb queries and methodologies
• https://github.com/chrislusf/seaweedfs/blob/master/weed/filer/embedded_filer/files_in_leveldb.go#L8
Issues:
Our Disk File system was XFS, and it got corrupt, we used xfs_repair, to repair it and get it back to life, But while all these operations, our ldb files got corrupt, and we couldn't start the leveldb, from db directory with an error as
I1101 12:42:19 31159 volume.go:110] loading index file /storage/1082.idx readonly false
F1101 12:42:19 31159 filer_server.go:53] Can not start f…
Show more
Comments
13
<https://github.com/syndtr/goleveldb|syndtr/goleveldb>syndtr/goleveldb | Nov 19th, 2017 | Added by GitHub

Junaid Farooq  8:55 AM
yes but that was different
8:55
and I know about that. that was disk issue. but now its a just one file.

Chris  8:57 AM
may need to do “leveldb.RecoverFile()“, for folder /storage/filer/_main

Junaid Farooq  8:57 AM
leveldb.RecoverFile() this comes from?

Chris  8:57 AM
https://github.com/syndtr/goleveldb/issues/184#issuecomment-346782863
Comment on #184 Can not start filer in dir /storage/filer : leveldb/storage: corrupted or incomplete meta file.
Also, right now we are just repairing the files using no options like paranoid checks and compaction.
Is there any possibility to add those option to this
package main

import (
    "log"

Show more
<https://github.com/syndtr/goleveldb|syndtr/goleveldb>syndtr/goleveldb | Nov 24th, 2017 | Added by GitHub

Junaid Farooq  8:59 AM
hmm, I think this is going to be messed up so its better to have a new filer by today