Was ich so treibe...

Uli's IT-Blog - Konzeption, Entwicklung, Betrieb

Experimente mit ZFS unter Ubuntu-12.04

DEBs installieren

1
2
3
4
5
6
7
8
9
10
11
12
13

sudo apt-get install make build-essential dpkg-dev
sudo apt-get install -f
sudo dpkg -i zfs-dkms_0.6.1-1~precise_amd64.deb \
  spl-dkms_0.6.1-1~precise_all.deb              \
  dkms_2.2.0.3-1ubuntu3.1+zfs6~precise1_all.deb
sudo dpkg -i zfsutils_0.6.1-1~precise_amd64.deb \
  libnvpair1_0.6.1-1~precise_amd64.deb          \
  libuutil1_0.6.1-1~precise_amd64.deb           \
  libzfs1_0.6.1-1~precise_amd64.deb             \
  libzpool1_0.6.1-1~precise_amd64.deb
sudo dpkg -i ubuntu-zfs_7~precise1_amd64.deb    \
  spl_0.6.1-1~precise_amd64.deb
sudo dpkg -i mountall_2.48build1-zfs2_amd64.deb

Kurzexperimente mit ZPOOL

Pool anlegen

1
2
3
4
5
6
7
8
9
10
11
12
13

$ sudo zpool create -f zfsdata /dev/sda
$ sudo zpool status
  pool: zfsdata
 state: ONLINE
  scan: none requested
config:

  NAME        STATE     READ WRITE CKSUM
  zfsdata     ONLINE       0     0     0
    sda       ONLINE       0     0     0

errors: No known data errors
$ sudo zpool destroy zfsdata

Pool mit RaidZ anlegen

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

$ sudo zpool create -f zfsdata raidz /dev/sda /dev/sdb
$ sudo zpool status
  pool: zfsdata
 state: ONLINE
  scan: none requested
config:

  NAME        STATE     READ WRITE CKSUM
  zfsdata     ONLINE       0     0     0
    raidz1-0  ONLINE       0     0     0
      sda     ONLINE       0     0     0
      sdb     ONLINE       0     0     0

errors: No known data errors
$ sudo zpool destroy zfsdata

Pool und Reboot

Dieser Test soll sicherstellen, dass ein ZPOOL auch nach einem Reboot verfügbar ist.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

$ sudo zpool create -f zfsdata raidz /dev/sda /dev/sdb
$ sudo zpool status
  pool: zfsdata
 state: ONLINE
  scan: none requested
config:

  NAME        STATE     READ WRITE CKSUM
  zfsdata     ONLINE       0     0     0
    raidz1-0  ONLINE       0     0     0
      sda     ONLINE       0     0     0
      sdb     ONLINE       0     0     0

errors: No known data errors
$ df|grep zfs
zfsdata                  2873622144    128 2873622016   1% /zfsdata
$ sudo reboot
...
$ df|grep zfs
zfsdata                  2873622144    128 2873622016   1% /zfsdata
$ sudo zpool destroy zfsdata

Das funktioniert offenbar nur dann, wenn das “mountall”-Paket installiert ist!

Pool mit RaidZ und Spare Disk (manuell)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

$ sudo zpool create -f zfsdata raidz /dev/sda /dev/sdb spare /dev/sdc
$ sudo zpool status
  pool: zfsdata
 state: ONLINE
  scan: none requested
config:

  NAME        STATE     READ WRITE CKSUM
  zfsdata     ONLINE       0     0     0
    raidz1-0  ONLINE       0     0     0
      sda     ONLINE       0     0     0
      sdb     ONLINE       0     0     0
  spares
    sdc       AVAIL

errors: No known data errors

Jetzt habe ich die Platte /dev/sda rausgezogen. Ideal wäre nun, wenn sich das RAIDZ automatisch auf /dev/sdc verlagern würde…

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58

$ sudo zpool scrub zfsdata
$ sudo zpool status
  pool: zfsdata
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
  invalid.  Sufficient replicas exist for the pool to continue
  functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-4J
  scan: scrub repaired 0 in 0h0m with 0 errors on Sat Jul 20 11:07:11 2013
config:

  NAME        STATE     READ WRITE CKSUM
  zfsdata     DEGRADED     0     0     0
    raidz1-0  DEGRADED     0     0     0
      sda     UNAVAIL      0     0     0  corrupted data
      sdb     ONLINE       0     0     0
  spares
    sdc       AVAIL

errors: No known data errors
$ sudo zpool replace zfsdata sda sdc
$ sudo zpool status
  pool: zfsdata
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
  invalid.  Sufficient replicas exist for the pool to continue
  functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-4J
  scan: resilvered 532K in 0h0m with 0 errors on Sat Jul 20 11:12:19 2013
config:

  NAME         STATE     READ WRITE CKSUM
  zfsdata      DEGRADED     0     0     0
    raidz1-0   DEGRADED     0     0     0
      spare-0  UNAVAIL      0     0     0
        sda    UNAVAIL      0     0     0  corrupted data
        sdc    ONLINE       0     0     0
      sdb      ONLINE       0     0     0
  spares
    sdc        INUSE     currently in use

errors: No known data errors
$ sudo zpool detach zfsdata sda
$ sudo zpool status
  pool: zfsdata
 state: ONLINE
  scan: resilvered 532K in 0h0m with 0 errors on Sat Jul 20 11:12:19 2013
config:

  NAME        STATE     READ WRITE CKSUM
  zfsdata     ONLINE       0     0     0
    raidz1-0  ONLINE       0     0     0
      sdc     ONLINE       0     0     0
      sdb     ONLINE       0     0     0

errors: No known data errors

Jetzt habe ich /dev/sda wieder eingebaut und mit fdisk geprüft, ob sie auch ansprechbar ist.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

$ sudo zpool add zfsdata spare sda
$ sudo zpool status
  pool: zfsdata
 state: ONLINE
  scan: resilvered 532K in 0h0m with 0 errors on Sat Jul 20 11:12:19 2013
config:

  NAME        STATE     READ WRITE CKSUM
  zfsdata     ONLINE       0     0     0
    raidz1-0  ONLINE       0     0     0
      sdc     ONLINE       0     0     0
      sdb     ONLINE       0     0     0
  spares
    sda       AVAIL

errors: No known data errors

Pool mit RaidZ und Spare Disk (automatisch)

Nachfolgend meine Versuche mit “autoreplace” – leider wenig erfolgreich. Hier http://forums.nas4free.org/viewtopic.php?f=66&t=3873 werden die gleichen Probleme mit FreeBSD beschrieben. Anscheinend unterstützt FreeBSD dieses Feature nicht richtig, vermutlich ist’s bei Linux ebenso.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

$ sudo zpool create -f zfsdata raidz /dev/sda /dev/sdb spare /dev/sdc
$ sudo zpool set autoreplace=on zfsdata
$ sudo zpool status
  pool: zfsdata
 state: ONLINE
  scan: none requested
config:

  NAME        STATE     READ WRITE CKSUM
  zfsdata     ONLINE       0     0     0
    raidz1-0  ONLINE       0     0     0
      sda     ONLINE       0     0     0
      sdb     ONLINE       0     0     0
  spares
    sdc       AVAIL

errors: No known data errors

Jetzt habe ich die Platte /dev/sda rausgezogen. Ideal wäre nun, wenn sich das RAIDZ automatisch auf /dev/sdc verlagern würde…

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80

$ sudo zpool scrub zfsdata
$ sudo zpool status
  pool: zfsdata
 state: DEGRADED
status: One or more devices has been removed by the administrator.
  Sufficient replicas exist for the pool to continue functioning in a
  degraded state.
action: Online the device using 'zpool online' or replace the device with
  'zpool replace'.
  scan: scrub repaired 0 in 0h0m with 0 errors on Sat Jul 20 11:25:59 2013
config:

  NAME        STATE     READ WRITE CKSUM
  zfsdata     DEGRADED     0     0     0
    raidz1-0  DEGRADED     0     0     0
      sda     REMOVED      0     0     0
      sdb     ONLINE       0     0     0
  spares
    sdc       AVAIL

errors: No known data errors
$ sudo zpool replace zfsdata sda sdc
$ sudo zpool status
  pool: zfsdata
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
  continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sat Jul 20 11:33:09 2013
    146M scanned out of 1,03G at 8,61M/s, 0h1m to go
    72,9M resilvered, 13,94% done
config:

  NAME         STATE     READ WRITE CKSUM
  zfsdata      DEGRADED     0     0     0
    raidz1-0   DEGRADED     0     0     0
      spare-0  UNAVAIL      0     0     0
        sda    UNAVAIL      0     0     0  corrupted data
        sdc    ONLINE       0     0     0  (resilvering)
      sdb      ONLINE       0     0     0
  spares
    sdc        INUSE     currently in use

errors: No known data errors
$ sudo zpool status
  pool: zfsdata
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
  invalid.  Sufficient replicas exist for the pool to continue
  functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-4J
  scan: resilvered 526M in 0h1m with 0 errors on Sat Jul 20 11:34:28 2013
config:

  NAME         STATE     READ WRITE CKSUM
  zfsdata      DEGRADED     0     0     0
    raidz1-0   DEGRADED     0     0     0
      spare-0  UNAVAIL      0     0     0
        sda    UNAVAIL      0     0     0  corrupted data
        sdc    ONLINE       0     0     0
      sdb      ONLINE       0     0     0
  spares
    sdc        INUSE     currently in use

errors: No known data errors
$ sudo zpool detach zfsdata sda
$ sudo zpool status
  pool: zfsdata
 state: ONLINE
  scan: resilvered 526M in 0h1m with 0 errors on Sat Jul 20 11:34:28 2013
config:

  NAME        STATE     READ WRITE CKSUM
  zfsdata     ONLINE       0     0     0
    raidz1-0  ONLINE       0     0     0
      sdc     ONLINE       0     0     0
      sdb     ONLINE       0     0     0

errors: No known data errors

Jetzt habe ich /dev/sda wieder eingebaut und mit fdisk geprüft, ob sie auch ansprechbar ist.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

$ sudo zpool add zfsdata spare sda
$ sudo zpool status
  pool: zfsdata
 state: ONLINE
  scan: resilvered 526M in 0h1m with 0 errors on Sat Jul 20 11:34:28 2013
config:

  NAME        STATE     READ WRITE CKSUM
  zfsdata     ONLINE       0     0     0
    raidz1-0  ONLINE       0     0     0
      sdc     ONLINE       0     0     0
      sdb     ONLINE       0     0     0
  spares
    sda       AVAIL

errors: No known data errors

Platten-IDs ermitteln

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

$ ls /dev/disk/by-id/scsi-SATA_WDC*|grep -v part
/dev/disk/by-id/scsi-SATA_WDC_WD30EFRX-68_WD-WCC1T0755994
/dev/disk/by-id/scsi-SATA_WDC_WD30EFRX-68_WD-WCC1T0770511
/dev/disk/by-id/scsi-SATA_WDC_WD30EFRX-68_WD-WCC1T0771577
/dev/disk/by-id/scsi-SATA_WDC_WD30EFRX-68_WD-WCC1T0777780

$ ls /dev/disk/by-id/scsi-SATA_WDC*|grep -v part|xargs --verbose -n1 udevadm info -q path -n
udevadm info -q path -n /dev/disk/by-id/scsi-SATA_WDC_WD30EFRX-68_WD-WCC1T0755994
/devices/pci0000:00/0000:00:11.0/host3/target3:0:0/3:0:0:0/block/sdd
udevadm info -q path -n /dev/disk/by-id/scsi-SATA_WDC_WD30EFRX-68_WD-WCC1T0770511
/devices/pci0000:00/0000:00:11.0/host2/target2:0:0/2:0:0:0/block/sdc
udevadm info -q path -n /dev/disk/by-id/scsi-SATA_WDC_WD30EFRX-68_WD-WCC1T0771577
/devices/pci0000:00/0000:00:11.0/host0/target0:0:0/0:0:0:0/block/sda
udevadm info -q path -n /dev/disk/by-id/scsi-SATA_WDC_WD30EFRX-68_WD-WCC1T0777780
/devices/pci0000:00/0000:00:11.0/host1/target1:0:0/1:0:0:0/block/sdb

Aus den Ausgaben kann man diese Zuordnung erkennen:

  • sda – scsi-SATA_WDC_WD30EFRX-68_WD-WCC1T0771577
  • sdb – scsi-SATA_WDC_WD30EFRX-68_WD-WCC1T0777780
  • sdc – scsi-SATA_WDC_WD30EFRX-68_WD-WCC1T0770511
  • sdd – scsi-SATA_WDC_WD30EFRX-68_WD-WCC1T0755994

Die Platten-IDs sollten am besten auch auf den jeweiligen Einbaurahmen geschrieben werden, damit es zu keinen Verwechslungen kommen kann.

ZPool über Platten-IDs

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55

$ sudo zpool create -f zfsdata raidz scsi-SATA_WDC_WD30EFRX-68_WD-WCC1T0771577 \
   scsi-SATA_WDC_WD30EFRX-68_WD-WCC1T0777780 \
   spare scsi-SATA_WDC_WD30EFRX-68_WD-WCC1T0770511
$ sudo zpool status
  pool: zfsdata
 state: ONLINE
  scan: none requested
config:

  NAME                                           STATE     READ WRITE CKSUM
  zfsdata                                        ONLINE       0     0     0
    raidz1-0                                     ONLINE       0     0     0
      scsi-SATA_WDC_WD30EFRX-68_WD-WCC1T0771577  ONLINE       0     0     0
      scsi-SATA_WDC_WD30EFRX-68_WD-WCC1T0777780  ONLINE       0     0     0
  spares
    scsi-SATA_WDC_WD30EFRX-68_WD-WCC1T0770511    AVAIL

errors: No known data errors
$ sudo zpool scrub zfsdata
$ sudo zpool status
  pool: zfsdata
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
  invalid.  Sufficient replicas exist for the pool to continue
  functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-4J
  scan: scrub repaired 0 in 0h0m with 0 errors on Sat Jul 20 12:19:07 2013
config:

  NAME                                           STATE     READ WRITE CKSUM
  zfsdata                                        DEGRADED     0     0     0
    raidz1-0                                     DEGRADED     0     0     0
      scsi-SATA_WDC_WD30EFRX-68_WD-WCC1T0771577  UNAVAIL      0     0     0  corrupted data
      scsi-SATA_WDC_WD30EFRX-68_WD-WCC1T0777780  ONLINE       0     0     0
  spares
    scsi-SATA_WDC_WD30EFRX-68_WD-WCC1T0770511    AVAIL

errors: No known data errors

$ sudo zpool replace zfsdata scsi-SATA_WDC_WD30EFRX-68_WD-WCC1T0771577 scsi-SATA_WDC_WD30EFRX-68_WD-WCC1T0770511
$ sudo zpool detach zfsdata scsi-SATA_WDC_WD30EFRX-68_WD-WCC1T0771577
$ sudo zpool status
  pool: zfsdata
 state: ONLINE
  scan: resilvered 532K in 0h0m with 0 errors on Sat Jul 20 12:19:36 2013
config:

  NAME                                           STATE     READ WRITE CKSUM
  zfsdata                                        ONLINE       0     0     0
    raidz1-0                                     ONLINE       0     0     0
      scsi-SATA_WDC_WD30EFRX-68_WD-WCC1T0770511  ONLINE       0     0     0
      scsi-SATA_WDC_WD30EFRX-68_WD-WCC1T0777780  ONLINE       0     0     0

errors: No known data errors

Finale

1
2
3
4
5
6
7
8
9
10
11
12
13
14

$ sudo zpool create -o ashift=12 -f zfsdata scsi-SATA_WDC_WD30EFRX-68_WD-WCC1T0771577 scsi-SATA_WDC_WD30EFRX-68_WD-WCC1T0777780 scsi-SATA_WDC_WD30EFRX-68_WD-WCC1T0770511
$ sudo zpool status
  pool: zfsdata
 state: ONLINE
  scan: none requested
config:

  NAME                                         STATE     READ WRITE CKSUM
  zfsdata                                      ONLINE       0     0     0
    scsi-SATA_WDC_WD30EFRX-68_WD-WCC1T0771577  ONLINE       0     0     0
    scsi-SATA_WDC_WD30EFRX-68_WD-WCC1T0777780  ONLINE       0     0     0
    scsi-SATA_WDC_WD30EFRX-68_WD-WCC1T0770511  ONLINE       0     0     0

errors: No known data errors

Die Platte “scsi-SATA_WDC_WD30EFRX-68_WD-WCC1T0755994” bleibt auf Vorrat nicht eingesteckt im Rahmen. Beim Ausfall einer Platte wird sie eingesteckt und mittels zpool replace zfsdata {kaputte_platte} scsi-SATA_WDC_WD30EFRX-68_WD-WCC1T0755994 aktiviert.

Probleme

Dateisystem nach Neustart nicht eingebunden

Nachdem ich den Kernel aktualisiert habe von 3.2 nach 3.5 klappt das automatische Einhängen des ZFS-Dateisystems nicht mehr.

Abhilfe: Standard-Paket “mountall” wieder installieren und ZFS umkonfigurieren!

1
2

sudo apt-get install mountall=2.36.4
sudo sed -i s/ZFS_MOUNT='no'/ZFS_MOUNT='yes'/ /etc/default/zfs

Änderungen

2013-07-20

  • Formatierung korrigiert
  • Reboot-Test
  • Tests mit Spare Disk
  • Tests mit Platten-IDs
  • Finale

Comments