0 변경 기록
| 순번 | 변경 내용 | 변경 날짜 |
|---|---|---|
| 1 | 추가 작성 | 2021/2/6 |
1 개요
이 문서는 CentOS 7.6 환경에서 CEPH Nautilus 버전을 설치한 후, 세로 확장을 수행하는 방법을 설명합니다. 이전 글인 'Linux 기반 3대 서버로 CEPH Nautilus 설치'와 'CEPH 수평 확장 가이드'의 내용을 바탕으로 진행됩니다.
2 환경 정보
| 호스트명 | 상태 | IP 주소 | 디스크 구성 | 역할 |
|---|---|---|---|---|
| cephtest001.ceph.kxdigit.com | 완료 | 10.3.176.10 | 시스템 디스크: /dev/sda 데이터 디스크: /dev/sdb /dev/sdc /dev/sdd | ceph-deploy, monitor, mgr, mds, osd |
| cephtest002.ceph.kxdigit.com | 완료 | 10.3.176.16 | 시스템 디스크: /dev/sda 데이터 디스크: /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf | monitor, mgr, mds, osd |
| cephtest003.ceph.kxdigit.com | 완료 | 10.3.176.44 | 시스템 디스크: /dev/sda 데이터 디스크: /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg | monitor, mgr, mds, osd |
| cephtest004.ceph.kxdigit.com | 이번 확장 대상 | 10.3.176.36 | 시스템 디스크: /dev/sda 데이터 디스크: /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf (확장 대상) | monitor, mgr, mds, osd |
3 실행 절차
(1) 확장 전 CEPH 상태 확인
3.1.1 클러스터 상태 확인
[root@cephtest001 ~]# ceph -s
cluster:
id: 6cd05235-66dd-4929-b697-1562d308d5c3
health: HEALTH_WARN
1 pools have many more objects per pg than average
services:
mon: 4 daemons, quorum cephtest001,cephtest002,cephtest003,cephtest004 (age 47h)
mgr: cephtest001(active, since 6w), standbys: cephtest002, cephtest003, cephtest004
osd: 18 osds: 18 up (since 47h), 18 in (since 47h)
rgw: 1 daemon active (cephtest004)
task status:
data:
pools: 8 pools, 400 pgs
objects: 24.07k objects, 167 GiB
usage: 520 GiB used, 66 TiB / 67 TiB avail
pgs: 400 active+clean
io:
client: 147 KiB/s rd, 21 KiB/s wr, 166 op/s rd, 3 op/s wr
[root@cephtest001 ~]#
3.1.2 OSD 구성 확인
cephtest004 노드는 현재 4개의 OSD를 가지고 있으며, 여기에 하나 더 추가할 예정입니다.
[root@cephtest001 ~]# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 66.93628 root default
-3 3.26669 host cephtest001
0 hdd 1.08890 osd.0 up 1.00000 1.00000
1 hdd 1.08890 osd.1 up 1.00000 1.00000
2 hdd 1.08890 osd.2 up 1.00000 1.00000
-5 5.45547 host cephtest002
3 hdd 1.09109 osd.3 up 1.00000 1.00000
4 hdd 1.09109 osd.4 up 1.00000 1.00000
5 hdd 1.09109 osd.5 up 1.00000 1.00000
6 hdd 1.09109 osd.6 up 1.00000 1.00000
7 hdd 1.09109 osd.7 up 1.00000 1.00000
-7 43.66132 host cephtest003
8 hdd 7.27689 osd.8 up 1.00000 1.00000
9 hdd 7.27689 osd.9 up 1.00000 1.00000
10 hdd 7.27689 osd.10 up 1.00000 1.00000
11 hdd 7.27689 osd.11 up 1.00000 1.00000
12 hdd 7.27689 osd.12 up 1.00000 1.00000
13 hdd 7.27689 osd.13 up 1.00000 1.00000
-9 14.55280 host cephtest004
14 hdd 3.63820 osd.14 up 1.00000 1.00000
15 hdd 3.63820 osd.15 up 1.00000 1.00000
16 hdd 3.63820 osd.16 up 1.00000 1.00000
17 hdd 3.63820 osd.17 up 1.00000 1.00000
[root@cephtest001 ~]#
(2) 확장 준비 작업
3.2.1 대상 서버 하드웨어 준비
해당 서버를 종료하고 새로운 디스크를 추가한 뒤 RAID0 설정을 진행합니다. cephtest004.ceph.kxdigit.com 서버에는 새 디스크 /dev/sdf가 추가되었습니다.
[root@cephtest004 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 1.8T 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 1.8T 0 part
├─centos-root 253:0 0 1.7T 0 lvm /
├─centos-swap 253:1 0 4G 0 lvm [SWAP]
└─centos-home 253:5 0 100G 0 lvm /home
sdb 8:16 0 3.7T 0 disk
└─ceph--b1011de5--e580--4eb0--b477--70a7a56e6c76-osd--block--f43f5d64--bb23--4a4b--aaa2--1138bca784ea 253:4 0 3.7T 0 lvm
sdc 8:32 0 3.7T 0 disk
└─ceph--42d1e684--fadc--46e8--8816--2aedaeed1725-osd--block--5e1cd70d--470b--4821--9f7f--195a65e95c86 253:6 0 3.7T 0 lvm
sdd 8:48 0 3.7T 0 disk
└─ceph--74ce24e0--92a0--4913--8127--03c6565574ee-osd--block--f1e446eb--557a--43c0--b82d--f4503dd93e18 253:3 0 3.7T 0 lvm
sde 8:64 0 3.7T 0 disk
└─ceph--34f94761--91a8--4f5a--8d29--8829fe5561de-osd--block--0d229ba1--f7fb--4660--827a--571cc41e20cc 253:2 0 3.7T 0 lvm
sdf 8:80 0 3.7T 0 disk
[root@cephtest004 ~]#
배포 노드에서 다음 명령어로 디스크 정보를 확인합니다.
[cephadmin@cephtest001 cephcluster]$ ceph-deploy disk list cephtest004
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadmin/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (2.0.1): /bin/ceph-deploy disk list cephtest004
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] debug : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] subcommand : list
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f82ecc8fc20>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] host : ['cephtest004']
[ceph_deploy.cli][INFO ] func : <function disk at 0x7f82ed0e2cf8>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[cephtest004][DEBUG ] connection detected need for sudo
[cephtest004][DEBUG ] connected to host: cephtest004
[cephtest004][DEBUG ] detect platform information from remote host
[cephtest004][DEBUG ] detect machine type
[cephtest004][DEBUG ] find the location of an executable
[cephtest004][INFO ] Running command: sudo fdisk -l
[cephtest004][INFO ] Disk /dev/sda: 1999.8 GB, 1999844147200 bytes, 3905945600 sectors
[cephtest004][INFO ] Disk /dev/sdb: 4000.2 GB, 4000225165312 bytes, 7812939776 sectors
[cephtest004][INFO ] Disk /dev/sdc: 4000.2 GB, 4000225165312 bytes, 7812939776 sectors
[cephtest004][INFO ] Disk /dev/sdd: 4000.2 GB, 4000225165312 bytes, 7812939776 sectors
[cephtest004][INFO ] Disk /dev/sde: 4000.2 GB, 4000225165312 bytes, 7812939776 sectors
[cephtest004][INFO ] Disk /dev/sdf: 4000.2 GB, 4000225165312 bytes, 7812939776 sectors
[cephtest004][INFO ] Disk /dev/mapper/centos-root: 1879.0 GB, 1879048192000 bytes, 3670016000 sectors
[cephtest004][INFO ] Disk /dev/mapper/centos-swap: 4294 MB, 4294967296 bytes, 8388608 sectors
[cephtest004][INFO ] Disk /dev/mapper/ceph--34f94761--91a8--4f5a--8d29--8829fe5561de-osd--block--0d229ba1--f7fb--4660--827a--571cc41e20cc: 4000.2 GB, 4000220971008 bytes, 7812931584 sectors
[cephtest004][INFO ] Disk /dev/mapper/ceph--74ce24e0--92a0--4913--8127--03c6565574ee-osd--block--f1e446eb--557a--43c0--b82d--f4503dd93e18: 4000.2 GB, 4000220971008 bytes, 7812931584 sectors
[cephtest004][INFO ] Disk /dev/mapper/ceph--b1011de5--e580--4eb0--b477--70a7a56e6c76-osd--block--f43f5d64--bb23--4a4b--aaa2--1138bca784ea: 4000.2 GB, 4000220971008 bytes, 7812931584 sectors
[cephtest004][INFO ] Disk /dev/mapper/centos-home: 107.4 GB, 107374182400 bytes, 209715200 sectors
[cephtest004][INFO ] Disk /dev/mapper/ceph--42d1e684--fadc--46e8--8816--2aedaeed1725-osd--block--5e1cd70d--470b--4821--9f7f--195a65e95c86: 4000.2 GB, 4000220971008 bytes, 7812931584 sectors
[cephadmin@cephtest001 cephcluster]$
(3) 세로 확장 수행
3.3.1 디스크 초기화 (배포 노드에서 실행)
cephadmin 사용자로 배포 노드에 접속하여 아래 명령어를 실행합니다.
ceph-deploy disk zap cephtest004 /dev/sdf
[cephadmin@cephtest001 cephcluster]$ pwd
/home/cephadmin/cephcluster
[cephadmin@cephtest001 cephcluster]$ ceph-deploy disk zap cephtest004 /dev/sdf
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadmin/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (2.0.1): /bin/ceph-deploy disk zap cephtest004 /dev/sdf
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] debug : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] subcommand : zap
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7fc618f5dc20>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] host : cephtest004
[ceph_deploy.cli][INFO ] func : <function disk at 0x7fc6193b0cf8>
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] disk : ['/dev/sdf']
[ceph_deploy.osd][DEBUG ] zapping /dev/sdf on cephtest004
[cephtest004][DEBUG ] connection detected need for sudo
[cephtest004][DEBUG ] connected to host: cephtest004
[cephtest004][DEBUG ] detect platform information from remote host
[cephtest004][DEBUG ] detect machine type
[cephtest004][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO ] Distro info: CentOS Linux 7.6.1810 Core
[cephtest004][DEBUG ] zeroing last few blocks of device
[cephtest004][DEBUG ] find the location of an executable
[cephtest004][INFO ] Running command: sudo /usr/sbin/ceph-volume lvm zap /dev/sdf
[cephtest004][WARNIN] --> Zapping: /dev/sdf
[cephtest004][WARNIN] --> --destroy was not specified, but zapping a whole device will remove the partition table
[cephtest004][WARNIN] Running command: /bin/dd if=/dev/zero of=/dev/sdf bs=1M count=10 conv=fsync
[cephtest004][WARNIN] stderr: 10+0 records in
[cephtest004][WARNIN] 10+0 records out
[cephtest004][WARNIN] 10485760 bytes (10 MB) copied
[cephtest004][WARNIN] stderr: , 0.0190883 s, 549 MB/s
[cephtest004][WARNIN] --> Zapping successful for: <Raw Device: /dev/sdf>
[cephadmin@cephtest001 cephcluster]$
3.3.2 OSD 생성
[cephadmin@cephtest001 cephcluster]$ ceph-deploy osd create cephtest004 --data /dev/sdf
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephadmin/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (2.0.1): /bin/ceph-deploy osd create cephtest004 --data /dev/sdf
4 검증
새로 추가된 디스크가 클러스터에 성공적으로 추가되었습니다.
[cephadmin@cephtest001 cephcluster]$ ceph -s
cluster:
id: 6cd05235-66dd-4929-b697-1562d308d5c3
health: HEALTH_WARN
1 pools have many more objects per pg than average
Degraded data redundancy: 1725/72222 objects degraded (2.388%), 7 pgs degraded
services:
mon: 4 daemons, quorum cephtest001,cephtest002,cephtest003,cephtest004 (age 30h)
mgr: cephtest001(active, since 6w), standbys: cephtest002, cephtest003, cephtest004
osd: 19 osds: 19 up (since 35s), 19 in (since 35s); 43 remapped pgs
rgw: 1 daemon active (cephtest004)
task status:
data:
pools: 8 pools, 400 pgs
objects: 24.07k objects, 167 GiB
usage: 522 GiB used, 70 TiB / 71 TiB avail
pgs: 1725/72222 objects degraded (2.388%)
10462/72222 objects misplaced (14.486%)
355 active+clean
31 active+remapped+backfill_wait
7 active+recovery_wait+undersized+degraded+remapped
4 active+remapped+backfilling
2 active+recovering+undersized+remapped
1 active+remapped
io:
client: 741 KiB/s rd, 22 KiB/s wr, 795 op/s rd, 2 op/s wr
recovery: 101 MiB/s, 18 objects/s
[cephadmin@cephtest001 cephcluster]$ ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 70.57448 root default
-3 3.26669 host cephtest001
0 hdd 1.08890 osd.0 up 1.00000 1.00000
1 hdd 1.08890 osd.1 up 1.00000 1.00000
2 hdd 1.08890 osd.2 up 1.00000 1.00000
-5 5.45547 host cephtest002
3 hdd 1.09109 osd.3 up 1.00000 1.00000
4 hdd 1.09109 osd.4 up 1.00000 1.00000
5 hdd 1.09109 osd.5 up 1.00000 1.00000
6 hdd 1.09109 osd.6 up 1.00000 1.00000
7 hdd 1.09109 osd.7 up 1.00000 1.00000
-7 43.66132 host cephtest003
8 hdd 7.27689 osd.8 up 1.00000 1.00000
9 hdd 7.27689 osd.9 up 1.00000 1.00000
10 hdd 7.27689 osd.10 up 1.00000 1.00000
11 hdd 7.27689 osd.11 up 1.00000 1.00000
12 hdd 7.27689 osd.12 up 1.00000 1.00000
13 hdd 7.27689 osd.13 up 1.00000 1.00000
-9 18.19099 host cephtest004
14 hdd 3.63820 osd.14 up 1.00000 1.00000
15 hdd 3.63820 osd.15 up 1.00000 1.00000
16 hdd 3.63820 osd.16 up 1.00000 1.00000
17 hdd 3.63820 osd.17 up 1.00000 1.00000
18 hdd 3.63820 osd.18 up 1.00000 1.00000
[cephadmin@cephtest001 cephcluster]$
5 참고 자료
https://docs.ceph.com/en/nautilus/rados/deployment/ceph-deploy-osd/#list-osds