Gluterfs安装部署

2017年12月15日11:58:31Gluterfs安装部署已关闭评论 366 views

1.1 安装gluterfs

1.1.1 环境准备

1.1.1.1 系统环境准备

1
2
3
4
5
6
7
8
9
10
# 系统信息
[root@gluterfs ~]# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
[root@gluterfs ~]# uname -r
3.10.0-327.el7.x86_64
# 网络信息
[root@gluterfs01 ~]# hostname -I
10.0.0.211 172.16.1.211
[root@gluterfs02 ~]# hostname -I
10.0.0.212 172.16.1.212
提示:先在虚拟机中挂载一块新的硬盘作为非系统盘,实际生产中视情况是否使用独立的存储设备。

1.1.1.2 分区并格式化硬盘

1
2
3
4
5
6
7
8
9
10
# 重新分区
[root@gluterfs ~]# fdisk /dev/sdb
Command (m for help): n
Select (default p): p
Partition number (1-4, default 1):
First sector (2048-20971519, default 2048):
Last sector, +sectors or +size{K,M,G} (2048-20971519, default 20971519):
Command (m for help): w
# 格式化硬盘
[root@gluterfs ~]# mkfs.xfs /dev/sdb1

1.1.1.3 挂载并创建共享目录

1
2
3
4
5
6
7
8
# 挂载硬盘
[root@gluterfs ~]# mkdir -p /data/
[root@gluterfs ~]# mount /dev/sdb1 /data/
# 创建共享目录
[root@gluterfs01 ~]# mkdir -p /data/{distributed01,replicated01,striped01}
[root@gluterfs01 ~]# mkdir -p /data/distributed-replicated0{1,2}
[root@gluterfs02 ~]# mkdir -p /data/{distributed02,replicated02,striped02}
[root@gluterfs02 ~]# mkdir -p /data/distributed-replicated0{3,4}

1.1.1.4 制作本地yum源文件

1
2
3
4
5
6
7
[root@gluterfs ~]# cat > /etc/yum.repos.d/CentOS-Gluster.repo << EOF
[centos-gluster]
name=centos-gluster
baseurl=http://10.0.0.1/centos/7/storage/x86_64/gluster-3.10/
enabled=1
gpgcheck=0
EOF

1.1.2 安装部署软件

1.1.2.1 安装软件

1
2
[root@gluterfs ~]# yum install -y centos-release-gluster    # 如果没有本地源时安装此源
[root@gluterfs ~]# yum install -y glusterfs-server

1.1.2.2 启动服务

1
2
[root@gluterfs ~]# systemctl enable glusterd
[root@gluterfs ~]# systemctl start glusterd

1.2 配置信任池(即所有节点地址池)

1
2
[root@gluterfs01 ~]# gluster peer probe 10.0.0.212
[root@gluterfs02 ~]# gluster peer probe 10.0.0.211

1.3 创建卷

提示:glusterfs的基础卷分为分布式卷、复制卷、条带卷,并且可以由这三种基础卷组合为其他扩展卷。

1.3.1 创建分布式卷

1.3.1.1 语法及示例

1
2
# gluster volume create  [transport tcp | rdma | tcp,rdma]
# gluster volume create test-volume server1:/exp1 server2:/exp2 server3:/exp3 server4:/exp4

1.3.1.2 命令实例

1
[root@gluterfs01 ~]# gluster volume create distributed-volume 10.0.0.211:/data/distributed01 10.0.0.212:/data/distributed02

1.3.2 创建复制卷(RAID 1)

1.3.2.1 语法及示例

1
2
# gluster volume create  [replica ] [transport tcp | rdma | tcp,rdma]
# gluster volume create test-volume replica 2 transport tcp server1:/exp1 server2:/exp2

1.3.2.2 命令实例

1
[root@gluterfs01 ~]# gluster volume create replicated-volume replica 2 10.0.0.211:/data/replicated01 10.0.0.212:/data/replicated02

1.3.3 创建条带卷(RAID 0)

1.3.3.1 语法及示例

1
2
# gluster volume create  [stripe ] [transport tcp | rdma | tcp,rdma]
# gluster volume create test-volume stripe 2 transport tcp server1:/exp1 server2:/exp2

1.3.3.2 命令实例

1
[root@gluterfs01 ~]# gluster volume create striped-volume stripe 2 10.0.0.211:/data/striped01 10.0.0.212:/data/striped02

1.3.4 创建分布式条带卷

1.3.4.1 语法及示例(必须偶数个节点)

1
2
# gluster volume create  [stripe ] [transport tcp | rdma | tcp,rdma]
# gluster volume create test-volume stripe 4 transport tcp server1:/exp1 server2:/exp2 server3:/exp3 server4:/exp4 server5:/exp5 server6:/exp6 server7:/exp7 server8:/exp8

1.3.5 创建分布式复制卷

1.3.5.1 语法及示例(必须偶数个节点,生产中常用方式)

1
2
# gluster volume create  [replica ] [transport tcp | rdma | tcp,rdma]
# gluster volume create test-volume replica 2 transport tcp server1:/exp1 server2:/exp2 server3:/exp3 server4:/exp4 server5:/exp5 server6:/exp6

1.3.5.2 命令实例(此处使用两个节点模拟四个节点时的情况)

1
[root@gluterfs01 ~]# gluster volume create distributed-replicated-volume replica 2 transport tcp 10.0.0.211:/data/distributed-replicated01 10.0.0.212:/data/distributed-replicated03 10.0.0.211:/data/distributed-replicated02 10.0.0.212:/data/distributed-replicated04 force
提示:复制的顺序和创建的顺序有关:server1:/exp1 server2:/exp2互为复制关系,server1:/exp1 server2:/exp2互为复制关系,以此类推。

1.3.5.3 问题解决

【问题现象】:

1
2
[root@gluterfs02 ~]# gluster volume create distributed-replicated-volume replica 2 transport tcp 10.0.0.211:/data/distributed-replicated01 10.0.0.211:/data/distributed-replicated02 10.0.0.212:/data/distributed-replicated03 10.0.0.212:/data/distributed-replicated04
volume create: distributed-replicated-volume: failed: Multiple bricks of a replicate volume are present on the same server. This setup is not optimal. Use 'force' at the end of the command if you want to override this behavior.

【解决办法】:

1
gluster volume create distributed-replicated-volume replica 2 transport tcp 10.0.0.211:/data/distributed-replicated01 10.0.0.212:/data/distributed-replicated03 10.0.0.211:/data/distributed-replicated02 10.0.0.212:/data/distributed-replicated04 force
提示:如果执行命令时提示“Use 'force' at the end of the command if you want to override this behavior.”时,在命令最后添加force参数执行即可。

1.3.6 创建分布式条带复制卷

1.3.6.1 语法及示例

1
2
# gluster volume create  [stripe ] [replica ] [transport tcp | rdma | tcp,rdma]
# gluster volume create test-volume stripe 2 replica 2 transport tcp server1:/exp1 server2:/exp2 server3:/exp3 server4:/exp4 server5:/exp5 server6:/exp6 server7:/exp7 server8:/exp8

1.4 启动卷并查看状态

1.4.1 启动卷

1
2
3
4
[root@gluterfs01 ~]# gluster volume start distributed-volume
[root@gluterfs01 ~]# gluster volume start replicated-volume
[root@gluterfs01 ~]# gluster volume start striped-volume
[root@gluterfs01 ~]# gluster volume start distributed-replicated-volume

1.4.2 查看卷状态

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
[root@gluterfs01 ~]# gluster volume info

Volume Name: distributed-replicated-volume
Type: Distributed-Replicate
Volume ID: 60b17fbc-309e-4893-82ab-208f946b7b74
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 10.0.0.211:/data/distributed-replicated01
Brick2: 10.0.0.212:/data/distributed-replicated03
Brick3: 10.0.0.211:/data/distributed-replicated02
Brick4: 10.0.0.212:/data/distributed-replicated04
Options Reconfigured:
transport.address-family: inet
nfs.disable: on

Volume Name: distributed-volume
Type: Distribute
Volume ID: 4bb1530e-99a1-44fd-9fd1-b90e387262dc
Status: Started
Snapshot Count: 0
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: 10.0.0.211:/data/distributed01
Brick2: 10.0.0.212:/data/distributed02
Options Reconfigured:
transport.address-family: inet
nfs.disable: on

Volume Name: replicated-volume
Type: Replicate
Volume ID: 10517a05-7fdd-47d5-8018-d1788c0482b5
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: 10.0.0.211:/data/replicated01
Brick2: 10.0.0.212:/data/replicated02
Options Reconfigured:
transport.address-family: inet
nfs.disable: on

Volume Name: striped-volume
Type: Stripe
Volume ID: 50be9ee6-f8c5-424b-ba2d-9812d3cd6270
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: 10.0.0.211:/data/striped01
Brick2: 10.0.0.212:/data/striped02
Options Reconfigured:
transport.address-family: inet
nfs.disable: on

1.5 使用gluterfs

1.5.1 创建共享目录

1
[root@gluterfs01 ~]# mkdir -p /mnt/{distributed,replicated,striped,distributed-replicated}

1.5.2 挂载共享卷

1
2
3
4
[root@gluterfs01 ~]# mount.glusterfs 10.0.0.211:/distributed-volume /mnt/distributed/
[root@gluterfs01 ~]# mount.glusterfs 10.0.0.211:/replicated-volume /mnt/replicated/
[root@gluterfs01 ~]# mount.glusterfs 10.0.0.211:/striped-volume /mnt/striped/
[root@gluterfs01 ~]# mount.glusterfs 10.0.0.211:/distributed-replicated-volume /mnt/distributed-replicated/

1.5.3 查看挂载情况

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[root@gluterfs01 ~]# df -h
Filesystem                                 Size  Used Avail Use% Mounted on
/dev/sda3                                   98G  2.3G   96G   3% /
devtmpfs                                   228M     0  228M   0% /dev
tmpfs                                      237M     0  237M   0% /dev/shm
tmpfs                                      237M  4.6M  233M   2% /run
tmpfs                                      237M     0  237M   0% /sys/fs/cgroup
/dev/sda1                                 1014M  110M  905M  11% /boot
tmpfs                                       48M     0   48M   0% /run/user/0
/dev/sdb1                                   10G   33M   10G   1% /data
10.0.0.211:/distributed-volume              20G   66M   20G   1% /mnt/distributed
10.0.0.211:/replicated-volume               10G   33M   10G   1% /mnt/replicated
10.0.0.211:/striped-volume                  20G   66M   20G   1% /mnt/striped
10.0.0.211:/distributed-replicated-volume   20G   66M   20G   1% /mnt/distributed-replicated

1.6 测试卷

1.6.1 测试分布式卷

1.6.1.1 写入文件

1
2
[root@gluterfs01 ~]# echo "distributed01" > /mnt/distributed/distributed01.txt
[root@gluterfs01 ~]# echo "distributed02" > /mnt/distributed/distributed02.txt

1.6.1.2 查看写入的文件

1
2
3
4
[root@gluterfs01 ~]# ll /data/distributed01/*
-rw-r--r-- 2 root root 14 Dec 14 15:30 /data/distributed01/distributed02.txt
[root@gluterfs02 ~]# ll /data/distributed02/*
-rw-r--r-- 2 root root 14 Dec 14 15:30 /data/distributed02/distributed01.txt

1.6.2 测试复制卷

1.6.2.1 写入文件

1
[root@gluterfs01 ~]# echo "replicated" > /mnt/replicated/replicated.txt

1.6.2.2 查看写入的文件

1
2
3
4
[root@gluterfs01 ~]# ll /data/replicated01/*
-rw-r--r-- 2 root root 11 Dec 14 15:34 /data/replicated01/replicated.txt
[root@gluterfs02 ~]# ll /data/replicated02/*
-rw-r--r-- 2 root root 11 Dec 14 15:34 /data/replicated02/replicated.txt

1.6.3 测试条带卷

1.6.3.1 写入文件

1
2
3
[root@gluterfs01 ~]# ll -h /var/log/messages-20171214
-rw-------. 1 root root 736K Dec 14 15:19 /var/log/messages-20171214
[root@gluterfs01 ~]# cat /var/log/messages-20171214 >> /mnt/striped/striped.txt

1.6.3.2 查看写入的文件

1
2
3
4
[root@gluterfs01 ~]# ll -h /data/striped01/*
-rw-r--r-- 2 root root 384K Dec 14 15:40 /data/striped01/striped.txt
[root@gluterfs02 ~]# ll -h /data/striped02/*
-rw-r--r-- 2 root root 352K Dec 14 15:40 /data/striped02/striped.txt

1.6.4 测试分布式复制卷

1.6.4.1 写入文件

1
2
3
4
5
6
7
8
9
[root@gluterfs01 ~]# cat /var/log/messages-20171214 > /mnt/distributed-replicated/distributed-replicated01.txt
[root@gluterfs01 ~]# cat /var/log/messages-20171214 > /mnt/distributed-replicated/distributed-replicated02.txt
[root@gluterfs01 ~]# cat /var/log/messages-20171214 > /mnt/distributed-replicated/distributed-replicated03.txt
[root@gluterfs01 ~]# cat /var/log/messages-20171214 > /mnt/distributed-replicated/distributed-replicated04.txt
[root@gluterfs01 ~]# cat /var/log/messages-20171214 > /mnt/distributed-replicated/distributed-replicated05.txt
[root@gluterfs01 ~]# cat /var/log/messages-20171214 > /mnt/distributed-replicated/distributed-replicated05.txt
[root@gluterfs01 ~]# cat /var/log/messages-20171214 > /mnt/distributed-replicated/distributed-replicated06.txt
[root@gluterfs01 ~]# cat /var/log/messages-20171214 > /mnt/distributed-replicated/distributed-replicated07.txt
[root@gluterfs01 ~]# cat /var/log/messages-20171214 > /mnt/distributed-replicated/distributed-replicated08.txt

1.6.4.2 查看写入的文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
# gluterfs01节点查看
[root@gluterfs01 ~]# tree /data/distributed-replicated0*
/data/distributed-replicated01
├── distributed-replicated01.txt
├── distributed-replicated02.txt
├── distributed-replicated03.txt
└── distributed-replicated04.txt
/data/distributed-replicated02
├── distributed-replicated05.txt
├── distributed-replicated06.txt
├── distributed-replicated07.txt
└── distributed-replicated08.txt

0 directories, 8 files
# gluterfs02节点查看
[root@gluterfs02 ~]# tree /data/distributed-replicated0*
/data/distributed-replicated03
├── distributed-replicated01.txt
├── distributed-replicated02.txt
├── distributed-replicated03.txt
└── distributed-replicated04.txt
/data/distributed-replicated04
├── distributed-replicated05.txt
├── distributed-replicated06.txt
├── distributed-replicated07.txt
└── distributed-replicated08.txt

0 directories, 8 files
weinxin
我的微信
如果有技术上的问题可以扫一扫我的微信