CEPH 세로 확장 - Nautilus 버전 가이드

0 변경 기록

순번 변경 내용 변경 날짜
1 추가 작성 2021/2/6

1 개요

이 문서는 CentOS 7.6 환경에서 CEPH Nautilus 버전을 설치한 후, 세로 확장을 수행하는 방법을 설명합니다. 이전 글인 'Linux 기반 3대 서버로 CEPH Nautilus 설치'와 'CEPH 수평 확장 가이드'의 내용을 바탕으로 진행됩니다.

2 환경 정보

호스트명 상태 IP 주소 디스크 구성 역할
cephtest001.ceph.kxdigit.com 완료 10.3.176.10 시스템 디스크: /dev/sda 데이터 디스크: /dev/sdb /dev/sdc /dev/sdd ceph-deploy, monitor, mgr, mds, osd
cephtest002.ceph.kxdigit.com 완료 10.3.176.16 시스템 디스크: /dev/sda 데이터 디스크: /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf monitor, mgr, mds, osd
cephtest003.ceph.kxdigit.com 완료 10.3.176.44 시스템 디스크: /dev/sda 데이터 디스크: /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg monitor, mgr, mds, osd
cephtest004.ceph.kxdigit.com 이번 확장 대상 10.3.176.36 시스템 디스크: /dev/sda 데이터 디스크: /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf (확장 대상) monitor, mgr, mds, osd

3 실행 절차

(1) 확장 전 CEPH 상태 확인

3.1.1 클러스터 상태 확인

[root@cephtest001 ~]# ceph -s
  cluster:
    id:     6cd05235-66dd-4929-b697-1562d308d5c3
    health: HEALTH_WARN
            1 pools have many more objects per pg than average

  services:
    mon: 4 daemons, quorum cephtest001,cephtest002,cephtest003,cephtest004 (age 47h)
    mgr: cephtest001(active, since 6w), standbys: cephtest002, cephtest003, cephtest004
    osd: 18 osds: 18 up (since 47h), 18 in (since 47h)
    rgw: 1 daemon active (cephtest004)

  task status:

  data:
    pools:   8 pools, 400 pgs
    objects: 24.07k objects, 167 GiB
    usage:   520 GiB used, 66 TiB / 67 TiB avail
    pgs:     400 active+clean

  io:
    client:   147 KiB/s rd, 21 KiB/s wr, 166 op/s rd, 3 op/s wr

[root@cephtest001 ~]#

3.1.2 OSD 구성 확인

cephtest004 노드는 현재 4개의 OSD를 가지고 있으며, 여기에 하나 더 추가할 예정입니다.

[root@cephtest001 ~]# ceph osd tree
ID CLASS WEIGHT   TYPE NAME            STATUS REWEIGHT PRI-AFF
-1       66.93628 root default
-3        3.26669     host cephtest001
 0   hdd  1.08890         osd.0            up  1.00000 1.00000
 1   hdd  1.08890         osd.1            up  1.00000 1.00000
 2   hdd  1.08890         osd.2            up  1.00000 1.00000
-5        5.45547     host cephtest002
 3   hdd  1.09109         osd.3            up  1.00000 1.00000
 4   hdd  1.09109         osd.4            up  1.00000 1.00000
 5   hdd  1.09109         osd.5            up  1.00000 1.00000
 6   hdd  1.09109         osd.6            up  1.00000 1.00000
 7   hdd  1.09109         osd.7            up  1.00000 1.00000
-7       43.66132     host cephtest003
 8   hdd  7.27689         osd.8            up  1.00000 1.00000
 9   hdd  7.27689         osd.9            up  1.00000 1.00000
10   hdd  7.27689         osd.10           up  1.00000 1.00000
11   hdd  7.27689         osd.11           up  1.00000 1.00000
12   hdd  7.27689         osd.12           up  1.00000 1.00000
13   hdd  7.27689         osd.13           up  1.00000 1.00000
-9       14.55280     host cephtest004
14   hdd  3.63820         osd.14           up  1.00000 1.00000
15   hdd  3.63820         osd.15           up  1.00000 1.00000
16   hdd  3.63820         osd.16           up  1.00000 1.00000
17   hdd  3.63820         osd.17           up  1.00000 1.00000
[root@cephtest001 ~]#

(2) 확장 준비 작업

3.2.1 대상 서버 하드웨어 준비

해당 서버를 종료하고 새로운 디스크를 추가한 뒤 RAID0 설정을 진행합니다. cephtest004.ceph.kxdigit.com 서버에는 새 디스크 /dev/sdf가 추가되었습니다.

[root@cephtest004 ~]# lsblk
NAME                                                                                                  MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                                                                                                     8:0    0  1.8T  0 disk
├─sda1                                                                                                  8:1    0    1G  0 part /boot
└─sda2                                                                                                  8:2    0  1.8T  0 part
  ├─centos-root                                                                                       253:0    0  1.7T  0 lvm  /
  ├─centos-swap                                                                                       253:1    0    4G  0 lvm  [SWAP]
  └─centos-home                                                                                       253:5    0  100G  0 lvm  /home
sdb                                                                                                     8:16   0  3.7T  0 disk
└─ceph--b1011de5--e580--4eb0--b477--70a7a56e6c76-osd--block--f43f5d64--bb23--4a4b--aaa2--1138bca784ea 253:4    0  3.7T  0 lvm
sdc                                                                                                     8:32   0  3.7T  0 disk
└─ceph--42d1e684--fadc--46e8--8816--2aedaeed1725-osd--block--5e1cd70d--470b--4821--9f7f--195a65e95c86 253:6    0  3.7T  0 lvm
sdd                                                                                                     8:48   0  3.7T  0 disk
└─ceph--74ce24e0--92a0--4913--8127--03c6565574ee-osd--block--f1e446eb--557a--43c0--b82d--f4503dd93e18 253:3    0  3.7T  0 lvm
sde                                                                                                     8:64   0  3.7T  0 disk
└─ceph--34f94761--91a8--4f5a--8d29--8829fe5561de-osd--block--0d229ba1--f7fb--4660--827a--571cc41e20cc 253:2    0  3.7T  0 lvm
sdf                                                                                                     8:80   0  3.7T  0 disk
[root@cephtest004 ~]#

배포 노드에서 다음 명령어로 디스크 정보를 확인합니다.

[cephadmin@cephtest001 cephcluster]$ ceph-deploy disk list cephtest004
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadmin/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /bin/ceph-deploy disk list cephtest004
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  debug                         : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : list
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f82ecc8fc20>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  host                          : ['cephtest004']
[ceph_deploy.cli][INFO  ]  func                          : <function disk at 0x7f82ed0e2cf8>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[cephtest004][DEBUG ] connection detected need for sudo
[cephtest004][DEBUG ] connected to host: cephtest004
[cephtest004][DEBUG ] detect platform information from remote host
[cephtest004][DEBUG ] detect machine type
[cephtest004][DEBUG ] find the location of an executable
[cephtest004][INFO  ] Running command: sudo fdisk -l
[cephtest004][INFO  ] Disk /dev/sda: 1999.8 GB, 1999844147200 bytes, 3905945600 sectors
[cephtest004][INFO  ] Disk /dev/sdb: 4000.2 GB, 4000225165312 bytes, 7812939776 sectors
[cephtest004][INFO  ] Disk /dev/sdc: 4000.2 GB, 4000225165312 bytes, 7812939776 sectors
[cephtest004][INFO  ] Disk /dev/sdd: 4000.2 GB, 4000225165312 bytes, 7812939776 sectors
[cephtest004][INFO  ] Disk /dev/sde: 4000.2 GB, 4000225165312 bytes, 7812939776 sectors
[cephtest004][INFO  ] Disk /dev/sdf: 4000.2 GB, 4000225165312 bytes, 7812939776 sectors
[cephtest004][INFO  ] Disk /dev/mapper/centos-root: 1879.0 GB, 1879048192000 bytes, 3670016000 sectors
[cephtest004][INFO  ] Disk /dev/mapper/centos-swap: 4294 MB, 4294967296 bytes, 8388608 sectors
[cephtest004][INFO  ] Disk /dev/mapper/ceph--34f94761--91a8--4f5a--8d29--8829fe5561de-osd--block--0d229ba1--f7fb--4660--827a--571cc41e20cc: 4000.2 GB, 4000220971008 bytes, 7812931584 sectors
[cephtest004][INFO  ] Disk /dev/mapper/ceph--74ce24e0--92a0--4913--8127--03c6565574ee-osd--block--f1e446eb--557a--43c0--b82d--f4503dd93e18: 4000.2 GB, 4000220971008 bytes, 7812931584 sectors
[cephtest004][INFO  ] Disk /dev/mapper/ceph--b1011de5--e580--4eb0--b477--70a7a56e6c76-osd--block--f43f5d64--bb23--4a4b--aaa2--1138bca784ea: 4000.2 GB, 4000220971008 bytes, 7812931584 sectors
[cephtest004][INFO  ] Disk /dev/mapper/centos-home: 107.4 GB, 107374182400 bytes, 209715200 sectors
[cephtest004][INFO  ] Disk /dev/mapper/ceph--42d1e684--fadc--46e8--8816--2aedaeed1725-osd--block--5e1cd70d--470b--4821--9f7f--195a65e95c86: 4000.2 GB, 4000220971008 bytes, 7812931584 sectors
[cephadmin@cephtest001 cephcluster]$

(3) 세로 확장 수행

3.3.1 디스크 초기화 (배포 노드에서 실행)

cephadmin 사용자로 배포 노드에 접속하여 아래 명령어를 실행합니다.

ceph-deploy disk zap cephtest004 /dev/sdf

[cephadmin@cephtest001 cephcluster]$ pwd
/home/cephadmin/cephcluster
[cephadmin@cephtest001 cephcluster]$ ceph-deploy disk zap cephtest004 /dev/sdf
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadmin/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /bin/ceph-deploy disk zap cephtest004 /dev/sdf
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  debug                         : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : zap
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7fc618f5dc20>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  host                          : cephtest004
[ceph_deploy.cli][INFO  ]  func                          : <function disk at 0x7fc6193b0cf8>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  disk                          : ['/dev/sdf']
[ceph_deploy.osd][DEBUG ] zapping /dev/sdf on cephtest004
[cephtest004][DEBUG ] connection detected need for sudo
[cephtest004][DEBUG ] connected to host: cephtest004
[cephtest004][DEBUG ] detect platform information from remote host
[cephtest004][DEBUG ] detect machine type
[cephtest004][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO  ] Distro info: CentOS Linux 7.6.1810 Core
[cephtest004][DEBUG ] zeroing last few blocks of device
[cephtest004][DEBUG ] find the location of an executable
[cephtest004][INFO  ] Running command: sudo /usr/sbin/ceph-volume lvm zap /dev/sdf
[cephtest004][WARNIN] --> Zapping: /dev/sdf
[cephtest004][WARNIN] --> --destroy was not specified, but zapping a whole device will remove the partition table
[cephtest004][WARNIN] Running command: /bin/dd if=/dev/zero of=/dev/sdf bs=1M count=10 conv=fsync
[cephtest004][WARNIN]  stderr: 10+0 records in
[cephtest004][WARNIN] 10+0 records out
[cephtest004][WARNIN] 10485760 bytes (10 MB) copied
[cephtest004][WARNIN]  stderr: , 0.0190883 s, 549 MB/s
[cephtest004][WARNIN] --> Zapping successful for: <Raw Device: /dev/sdf>
[cephadmin@cephtest001 cephcluster]$

3.3.2 OSD 생성

[cephadmin@cephtest001 cephcluster]$ ceph-deploy osd create cephtest004 --data /dev/sdf
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadmin/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /bin/ceph-deploy osd create cephtest004 --data /dev/sdf

4 검증

새로 추가된 디스크가 클러스터에 성공적으로 추가되었습니다.

[cephadmin@cephtest001 cephcluster]$ ceph -s
  cluster:
    id:     6cd05235-66dd-4929-b697-1562d308d5c3
    health: HEALTH_WARN
            1 pools have many more objects per pg than average
            Degraded data redundancy: 1725/72222 objects degraded (2.388%), 7 pgs degraded

  services:
    mon: 4 daemons, quorum cephtest001,cephtest002,cephtest003,cephtest004 (age 30h)
    mgr: cephtest001(active, since 6w), standbys: cephtest002, cephtest003, cephtest004
    osd: 19 osds: 19 up (since 35s), 19 in (since 35s); 43 remapped pgs
    rgw: 1 daemon active (cephtest004)

  task status:

  data:
    pools:   8 pools, 400 pgs
    objects: 24.07k objects, 167 GiB
    usage:   522 GiB used, 70 TiB / 71 TiB avail
    pgs:     1725/72222 objects degraded (2.388%)
             10462/72222 objects misplaced (14.486%)
             355 active+clean
             31  active+remapped+backfill_wait
             7   active+recovery_wait+undersized+degraded+remapped
             4   active+remapped+backfilling
             2   active+recovering+undersized+remapped
             1   active+remapped

  io:
    client:   741 KiB/s rd, 22 KiB/s wr, 795 op/s rd, 2 op/s wr
    recovery: 101 MiB/s, 18 objects/s

[cephadmin@cephtest001 cephcluster]$ ceph osd tree
ID CLASS WEIGHT   TYPE NAME            STATUS REWEIGHT PRI-AFF
-1       70.57448 root default
-3        3.26669     host cephtest001
 0   hdd  1.08890         osd.0            up  1.00000 1.00000
 1   hdd  1.08890         osd.1            up  1.00000 1.00000
 2   hdd  1.08890         osd.2            up  1.00000 1.00000
-5        5.45547     host cephtest002
 3   hdd  1.09109         osd.3            up  1.00000 1.00000
 4   hdd  1.09109         osd.4            up  1.00000 1.00000
 5   hdd  1.09109         osd.5            up  1.00000 1.00000
 6   hdd  1.09109         osd.6            up  1.00000 1.00000
 7   hdd  1.09109         osd.7            up  1.00000 1.00000
-7       43.66132     host cephtest003
 8   hdd  7.27689         osd.8            up  1.00000 1.00000
 9   hdd  7.27689         osd.9            up  1.00000 1.00000
10   hdd  7.27689         osd.10           up  1.00000 1.00000
11   hdd  7.27689         osd.11           up  1.00000 1.00000
12   hdd  7.27689         osd.12           up  1.00000 1.00000
13   hdd  7.27689         osd.13           up  1.00000 1.00000
-9       18.19099     host cephtest004
14   hdd  3.63820         osd.14           up  1.00000 1.00000
15   hdd  3.63820         osd.15           up  1.00000 1.00000
16   hdd  3.63820         osd.16           up  1.00000 1.00000
17   hdd  3.63820         osd.17           up  1.00000 1.00000
18   hdd  3.63820         osd.18           up  1.00000 1.00000
[cephadmin@cephtest001 cephcluster]$

5 참고 자료

https://docs.ceph.com/en/nautilus/rados/deployment/ceph-deploy-osd/#list-osds

태그: Ceph nautilus storage osd disk-expansion

10월 12일 05:22에 게시됨