18.Keepalived安装配置文档

前提

1. 了解lo网卡与eth0网卡区别?

lo为本地网卡,但可配置多个IP,也就是后面配置的realServer.sh的配置即配置的lo网卡,为的是抑制ipv4的apr.如果不设置,即会出现,keepalived选它为realserver的时候,他会一直说,我不是这个IP啊.我不是这个IP啊.我怎么给你返回信息呢.

2. 了解LVS-DR的原理?

LVS有LVS-DR,LVS-NAT,LVS-TUN三种模式,DR是最常用,最好用的.keepalived配置的realserver就是与LVS的天生结合.具体可年查看下面的[12]链接了解下.

3. 常用工具

tcpdump监听tcp数据,以进一步调试问题所在

查看设置情况ipvsadm -S -n

查看负载转向情况ipvsadm -L -n -c

场景

Keepalived即想处理nginx负载也想处理mysql负载,常规做法keepalived单独部署+nginx(lvs)+mysql(lvs),然后在keeplived配置80和3306端口的realServer.

另一种场景是keepalived只管理nginx且只处理高可用的情况,那样keepalived可以和nginx部署在一台,且不用设置LVS.

我们是第一种方式,但是为了节省系统资源,减少服务器数量LVS+NGINX+keepalived前端负载在同一台机器上(需要处理请求递归),LVS MYSQL负载在另外两台服务器上.网络结构图如下:

下载安装ipvsadm

1
2
3
4
5


yum install ipvsadm

modprobe ip_vs

lsmod |grep ip_vs 

下载安装keepalived

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


yum install popt-devel libnl libnl-devel libnfnetlink-devel

wget http://www.keepalived.org/software/keepalived-1.3.5.tar.gz

tar -zxvf keepalived-1.3.5.tar.gz

cd keepalived-1.3.5

./configure --prefix=/usr/local/keepalived
--with-kernel-dir=/usr/src/kernels/\`uname -r\`

make & make install

配置

将配置文件拷贝到系统对应的目录下:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31



cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/

mkdir /etc/keepalived

ln -s /usr/local/keepalived/sbin/keepalived /usr/sbin/

cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/

cp ./keepalived-1.3.5/keepalived/etc/init.d/keepalived /etc/init.d/

##再次强调:Keepalived的所有功能是配置keepalived.conf文件来实现的.

chkconfig keepalived on

service keepalived start

service keepalived stop

service keepalived restart

##注意:keepalive启动需要时间,稍等一会.如果出现failed: No route to host,如果你确定你的网络配置和DNS都设置正确的话,需要注释iptables的两行配置,最好是先关闭防火墙.

#-A INPUT -j REJECT --reject-with icmp-host-prohibited

#-A FORWARD -j REJECT --reject-with icmp-host-prohibited

#添加上一句

-A INPUT -p vrrp -j ACCEPT

BACKUP配置

注意修改:

router_id ###标识本节点,通常为hostname###

state ###备用节点设置为BACKUP###

priority ###备用节点必须比主节点优先级低###

RealServer脚本

LVS-DR模式中,后端真实服务器(RealServer)无需安装相关软件,只需要对VIP进行绑定和路由设置等一系列操作,可以理解为安装LVS的客户端.

这里整理为一个脚本:realserver,需要注册为服务,开机启动,顺序靠前.需要注意windows换行符和Linux不同,脚本容易报错可以使用以下命令去掉 \r

1
2
3
4
5


vi -b filename

:%s/r$//

:x

realserver.sh 脚本如下:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


#add for chkconfig
#chkconfig: 2345 70 30  ###234都是文本界面,5就是图形界面X,70启动顺序号,30系统关闭###
#description: RealServer's script  ###关于脚本的简短描述###
#processname: realserver       ###第一个进程名,后边设置自动时会用到###
#!/bin/bash
VIP=10.0.70.128 
##source /etc/rc.d/init.d/functions
case "$1" in
start)  
       ifconfig lo:0 $VIP netmask 255.255.255.255 broadcast $VIP  
       /sbin/route add -host $VIP dev lo:0  
       echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore  
       echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce  
       echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore  
       echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce  
       echo "realserver Start OK"  
       ;;  
stop)  
       ifconfig lo:0 down  
       route del $VIP >/dev/null 2>&1  
       echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore  
       echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce  
       echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore  
       echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce  
       echo "realserver Stoped"  
       ;;
       
restart)
        stop
        start  
        ;;   
        
       *)  
       echo "Usage: $0 {start|stop}"  
       exit 1  
esac  
exit 0  

同一台服务器LVS+KEEPALIVED+NGINX的问题

同一台做高可用即可,不建议做负载,需要制定复杂的防火墙策略

主要是因为:两台KeepAvlied都具有负载功能,同时具有realserver,会造成KeepAvlied1--KeepAvlied2--KeepAvlied1这样死循环.

需要防火墙iptable在所有的请求根据条件(如果是另一台keepalived mac过来的就不走LVS),给iptables的mangle表做一个mark,keepalived只fwmark相匹配的才进LVS,否则就进真实的IP.

在BACKUP10.0.70.32上执行

1
2


iptables -t mangle -I PREROUTING -d $VIP -p tcp -m tcp --dport 80 -m
mac ! --mac-source $33MAC -j MARK --set-mark 0x3

在MASTER10.0.70.33上执行

1
2


iptables -t mangle -I PREROUTING -d $VIP -p tcp -m tcp --dport 80 -m
mac ! --mac-source $22MAC -j MARK --set-mark 0x4

维护

ipvsadm -ln --stats

查看端口流量情况

查看设置情况

ipvsadm -S -n

查看负载转向情况

ipvsadm -L -n -c

配置文件

IP虚拟配置(主用)

keepalived.conf

check_nginx.sh脚本如下

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


#!/bin/bash
if [ "$(ps -ef | grep "nginx: master process"| grep -v grep )" == "" ]
then
 /usr/local/nginx/sbin/nginx
 sleep 5
 if [ "$(ps -ef | grep "nginx: master process"| grep -v grep )" == "" ]
 then
 killall keepalived
 fi
fi