Skip to content

Instantly share code, notes, and snippets.

@kazuhisya
Created August 16, 2011 07:22
Show Gist options
  • Save kazuhisya/1148606 to your computer and use it in GitHub Desktop.
Save kazuhisya/1148606 to your computer and use it in GitHub Desktop.
glusterfs error
[2011-08-16 15:51:28.232732] I [dht-common.c:523:dht_revalidate_cbk] 0-vol3-dht: mismatching layouts for /
[2011-08-16 15:51:28.664589] I [dht-layout.c:655:dht_layout_dir_mismatch] 0-vol3-dht: / - disk layout missing
[2011-08-16 15:51:28.664637] I [dht-common.c:523:dht_revalidate_cbk] 0-vol3-dht: mismatching layouts for /
[2011-08-16 15:51:28.665225] I [dht-layout.c:581:dht_layout_normalize] 0-vol3-dht: found anomalies in /. holes=1 overlaps=0
[2011-08-16 15:51:28.665260] I [dht-common.c:361:dht_lookup_root_dir_cbk] 0-vol3-dht: fixing assignment on /
[2011-08-16 15:51:33.624713] I [dht-layout.c:682:dht_layout_dir_mismatch] 0-vol3-dht: subvol: vol3-stripe-0; inode layout - 0 - 0; disk layout - 0 - 2147483646
[2011-08-16 15:51:33.624764] I [dht-common.c:523:dht_revalidate_cbk] 0-vol3-dht: mismatching layouts for /
[2011-08-16 15:52:30.537803] I [dht-layout.c:655:dht_layout_dir_mismatch] 0-vol3-dht: / - disk layout missing
[2011-08-16 15:52:30.537851] I [dht-common.c:523:dht_revalidate_cbk] 0-vol3-dht: mismatching layouts for /
[2011-08-16 15:52:30.538383] I [dht-common.c:361:dht_lookup_root_dir_cbk] 0-vol3-dht: fixing assignment on /
[2011-08-16 15:52:30.538763] I [dht-layout.c:192:dht_layout_search] 0-vol3-dht: no subvolume for hash (value) = 648307008
[2011-08-16 15:52:30.539495] I [dht-layout.c:192:dht_layout_search] 0-vol3-dht: no subvolume for hash (value) = 648307008
[2011-08-16 15:52:30.539543] W [fuse-bridge.c:1570:fuse_create_cbk] 0-glusterfs-fuse: 415: /hoge-fuga0111111111 => -1 (No such file or directory)
[2011-08-16 15:52:30.539709] I [dht-layout.c:192:dht_layout_search] 0-vol3-dht: no subvolume for hash (value) = 648307008
[2011-08-16 15:52:43.451162] I [dht-layout.c:192:dht_layout_search] 0-vol3-dht: no subvolume for hash (value) = 648307008
[2011-08-16 15:52:43.452049] I [dht-layout.c:192:dht_layout_search] 0-vol3-dht: no subvolume for hash (value) = 648307008
[2011-08-16 15:52:43.452087] W [fuse-bridge.c:1570:fuse_create_cbk] 0-glusterfs-fuse: 419: /hoge-fuga0111111111 => -1 (No such file or directory)
[2011-08-16 15:52:43.452306] I [dht-layout.c:192:dht_layout_search] 0-vol3-dht: no subvolume for hash (value) = 648307008
[2011-08-16 15:52:57.210979] I [dht-layout.c:192:dht_layout_search] 0-vol3-dht: no subvolume for hash (value) = 648307008
[2011-08-16 15:52:57.211904] I [dht-layout.c:192:dht_layout_search] 0-vol3-dht: no subvolume for hash (value) = 648307008
[2011-08-16 15:52:57.211943] W [fuse-bridge.c:1570:fuse_create_cbk] 0-glusterfs-fuse: 423: /hoge-fuga0111111111 => -1 (No such file or directory)
[2011-08-16 15:52:57.212138] I [dht-layout.c:192:dht_layout_search] 0-vol3-dht: no subvolume for hash (value) = 648307008
[2011-08-16 15:53:25.932058] I [dht-layout.c:192:dht_layout_search] 0-vol3-dht: no subvolume for hash (value) = 648307008
[2011-08-16 15:53:25.932834] I [dht-layout.c:192:dht_layout_search] 0-vol3-dht: no subvolume for hash (value) = 648307008
[2011-08-16 15:53:25.932876] W [fuse-bridge.c:1570:fuse_create_cbk] 0-glusterfs-fuse: 427: /hoge-fuga0111111111 => -1 (No such file or directory)

状況

  • マシンは2台、同型機(glus01, glus02 | HDD 各1発)、同構成
  • LVMでGlusterFS用にパーティションを切り、 /data配下に data1〜4として配置
  • 各dataN直下をBrickにしようと思ったが、lost+foundが邪魔だと判断したため、更にdataNディレクトリを掘った
  • volumeはstripe(2+2)で作成
  • 検証中で3回目だったからvol3
  • geo-replicationの検証をしてしていた(geo-replicationを止めてもtouchはできないままだった)
  • geo-replication自体は意図する挙動のように見えた
  • touchコマンドは、mount先にcdで移動して実行
  • mkdir, vimで編集はOK, touch, echo "hoge" > hoge.txtはNG(タイミングの問題?)
  • touchでNGでも同じコマンドを根気よく何度も叩くと通るときもあった
  • 既にVolumeを作りなおしてしまったのでstatus, infoはlogよりサルベージしたもの

version

glusterfs-geo-replication-3.2.2-1.el6.x86_64
glusterfs-fuse-3.2.2-1.el6.x86_64
glusterfs-core-3.2.2-1.el6.x86_64
glusterfs-rdma-3.2.2-1.el6.x86_64
fuse-2.8.3-3.el6_1.x86_64
rsync-3.0.8-1.el6.x86_64  <-- Fedora15よりbackportしたもの

作成コマンド

# gluster volume create vol3 stripe 2 glus01:/data/data1/data1 glus01:/data/data2/data2 glus02:/data/data1/data1 glus02:/data/data2/data2
# gluster volume geo-replication vol3 ssh://root@glus01:file:///root/geo-r start

mount コマンド(glus01にて)

# mount.glusterfs glus01:vol3 /mnt/vol3

df -hT

Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/mapper/vg_glus01-lv_root
ext4     50G  1.3G   46G   3% /
tmpfs        tmpfs    5.9G     0  5.9G   0% /dev/shm
/dev/sda1     ext4    485M   31M  430M   7% /boot
/dev/mapper/vg_glus01-lv_data1
ext4     20G  173M   19G   1% /data/data1
/dev/mapper/vg_glus01-lv_data2
ext4     20G  173M   19G   1% /data/data2
/dev/mapper/vg_glus01-lv_data3
ext4     20G  172M   19G   1% /data/data3
/dev/mapper/vg_glus01-lv_data4
ext4     20G  172M   19G   1% /data/data4

mount

/dev/mapper/vg_glus01-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda1 on /boot type ext4 (rw)
/dev/mapper/vg_glus01-lv_data1 on /data/data1 type ext4 (rw)
/dev/mapper/vg_glus01-lv_data2 on /data/data2 type ext4 (rw)
/dev/mapper/vg_glus01-lv_data3 on /data/data3 type ext4 (rw)
/dev/mapper/vg_glus01-lv_data4 on /data/data4 type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)

volume info

Volume Name: vol3
Type: Distributed-Stripe
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: glus01:/data/data1/data1
Brick2: glus01:/data/data2/data2
Brick3: glus02:/data/data1/data1
Brick4: glus02:/data/data2/data2
Options Reconfigured:
geo-replication.indexing: on
Given volfile:
+------------------------------------------------------------------------------+
1: volume vol3-client-0
2:     type protocol/client
3:     option remote-host glus01
4:     option remote-subvolume /data/data1/data1
5:     option transport-type tcp
6: end-volume
7: 
8: volume vol3-client-1
9:     type protocol/client
10:     option remote-host glus01
11:     option remote-subvolume /data/data2/data2
12:     option transport-type tcp
13: end-volume
14: 
15: volume vol3-client-2
16:     type protocol/client
17:     option remote-host glus02
18:     option remote-subvolume /data/data1/data1
19:     option transport-type tcp
20: end-volume
21: 
22: volume vol3-client-3
23:     type protocol/client
24:     option remote-host glus02
25:     option remote-subvolume /data/data2/data2
26:     option transport-type tcp
27: end-volume
28: 
29: volume vol3-stripe-0
30:     type cluster/stripe
31:     subvolumes vol3-client-0 vol3-client-1
32: end-volume
33: 
34: volume vol3-stripe-1
35:     type cluster/stripe
36:     subvolumes vol3-client-2 vol3-client-3
37: end-volume
38: 
39: volume vol3-dht
40:     type cluster/distribute
41:     subvolumes vol3-stripe-0 vol3-stripe-1
42: end-volume
43: 
44: volume vol3-write-behind
45:     type performance/write-behind
46:     subvolumes vol3-dht
47: end-volume
48: 
49: volume vol3-read-ahead
50:     type performance/read-ahead
51:     subvolumes vol3-write-behind
52: end-volume
53: 
54: volume vol3-io-cache
55:     type performance/io-cache
56:     subvolumes vol3-read-ahead
57: end-volume
58: 
59: volume vol3-quick-read
60:     type performance/quick-read
61:     subvolumes vol3-io-cache
62: end-volume
63: 
64: volume vol3-stat-prefetch
65:     type performance/stat-prefetch
66:     subvolumes vol3-quick-read
67: end-volume
68: 
69: volume vol3
70:     type debug/io-stats
71:     option latency-measurement off
72:     option count-fop-hits off
73:     subvolumes vol3-stat-prefetch
74: end-volume

+------------------------------------------------------------------------------+
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment