Gluster安装

Environment

Centos 7
Gluster 4.1

使用三台机器

192.168.1.100 server1
192.168.1.101 server2
192.168.1.102 server3

Foreward

kubernetes的持久化需要使用gluster.这里基于Centos提供的Quick-Start教程验证后的记录.

Install

Prepare

DNS和NTP

DNS如果没有特殊要求,采用默认配置即可,每台服务器对时间进行校对和统一时区.

Hosts Set

建议使用服务器名的方式管理gluster,则需要配置ip和名称的映射.每台机器都需要配置.

$ vim /etc/hosts
192.168.1.100 server1
192.168.1.101 server2
192.168.1.102 server3

Add Yum Repository

添加gluster下载仓库

1	$ yum install centos-release-gluster

Use XFS

推荐使用XFS文件系统,这里我还是使用exts4,没有重新格式化文件系统.格式化参考教程网上很多.

Install Gluster And Start

1 2	$ yum install glusterfs-server $ systemctl enable glusterd && systemctl start glusterd

Set Firewalld

○ By default, glusterd will listen on tcp/24007. But each time you add a brick, it will open a new port (that you’ll be able to see with “gluster volume status”)

☆ 默认是24007端口,但是每增加brick都会新增监听端口,具体端口可以通过gluster volume status查看.不过在配置防火墙时,我使用授权ip方式.

# 这里的TCP Port指定的49152就是需要开放的端口.
$ gluster volume status
Status of volume: gv0
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick server1:/bricks/brick1/gv0            49152     0          Y       6566
Brick server2:/bricks/brick1/gv0            49152     0          Y       26880
Self-heal Daemon on localhost               N/A       N/A        Y       6590
Self-heal Daemon on server2                 N/A       N/A        Y       26903

Task Status of Volume gv0
------------------------------------------------------------------------------
There are no active volume tasks

centos7默认使用firewalld,

1
2
3

# 允许指定ip.我使用三台机器做测试,每台机器要配置另外两台服务器的ip.
$ firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='192.168.1.101' accept"
$ firewall-cmd --reload

Set Trusted Pool

这里使用三台机器,可以在任意一台服务器,将另外两台服务器添加到Trust Pool即可.

如在server1上将server2和server3添加到Trust Pool.

1
2
3

# 注:/etc/hosts需要配置映射.
$ gluster peer probe server2
$ gluster peer probe server3

Create a Volume

在三台服务器上分别创建/bricks/brick1/gv0目录

1 2	# 官网推荐brick命名方式:/data/glusterfs/<volume>/<brick>/brick $ mkdir -p /bricks/brick1/gv0

挂载目录到gluster上.(任意节点执行命令)

1
2
3

# 这里选择的replicas模式,保证数据丢失,其他模式不在这里讨论.
$ gluster volume create gv0 replica 3 server1:/bricks/brick1/gv0 server2:/bricks/brick1/gv0 server3:/bricks/brick1/gv0
$ gluster volume start gv0

查看volume信息

$ gluster volume info
Volume Name: gv0
Type: Replicate
Volume ID: 5835a014-b598-467d-ba34-3301d6730d6f
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: server1:/bricks/brick1/gv0
Brick2: server2:/bricks/brick1/gv0
Brick3: server3:/bricks/brick1/gv0
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off

Testing

# 将server1:/gv0 挂在到/mnt目录
$ mount -t glusterfs server1:/gv0 /mnt
# 生成100个copy-test的文件
$ for i in `seq -w 1 100`; do cp -rp /var/log/messages /mnt/copy-test-$i; done
# 在三台机器上查看是否都有100个文件.
$ ls -lA /bricks/brick1/gv0

总结

推荐部署机器为奇数,不然会出现脑裂现象.(猜测使用了poxis算法)
gluster推荐使用xfs文件系统,centos/Red Hat Enterprise Linux 7默认使用,磁盘格式化时可能没有指定.