KubernetesV1.12二进制部署多Master

4.2部署apiserver组件

下载二进制包：https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.12.md
下载这个包（kubernetes-server-linux-amd64.tar.gz）就够了，包含了所需的所有组件。

# cd ~/k8s
# tar -zxvf kubernetes-server-linux-amd64.tar.gz
# cd ~/k8s/kubernetes/server/bin/
# cp kube-apiserver kube-scheduler kube-controller-manager kubectl /opt/kubernetes/bin/

创建token文件：

生成token：
# head -c 16 /dev/urandom | od -An -t x | tr -d ' '
# vim /opt/kubernetes/cfg/token.csv
    2f7a15198f7c0c3af3ba7f264b6885c2,kubelet-bootstrap,10001,"system:kubelet-bootstrap"

第一列：随机字符串，自己可生成
第二列：用户名
第三列：UID
第四列：用户组
创建apiserver配置文件：（注意修改master地址，etcd服务）

cat </opt/kubernetes/cfg/kube-apiserver

KUBE_APISERVER_OPTS="--logtostderr=true \\
--v=4 \\
--etcd-servers=https://192.168.1.43:2379,https://192.168.1.30:2379,https://192.168.1.51:2379 \\
--bind-address=192.168.1.43 \\
--secure-port=6443 \\
--advertise-address=192.168.1.43 \\
--allow-privileged=true \\
--service-cluster-ip-range=10.0.0.0/24 \\
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \\
--authorization-mode=RBAC,Node \\
--kubelet-https=true \\
--enable-bootstrap-token-auth \\
--token-auth-file=/opt/kubernetes/cfg/token.csv \\
--service-node-port-range=30000-50000 \\
--tls-cert-file=/opt/kubernetes/ssl/server.pem  \\
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \\
--client-ca-file=/opt/kubernetes/ssl/ca.pem \\
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--etcd-cafile=/opt/etcd/ssl/ca.pem \\
--etcd-certfile=/opt/etcd/ssl/server.pem \\
--etcd-keyfile=/opt/etcd/ssl/server-key.pem"

EOF

配置好前面生成的证书，确保能连接etcd。

参数说明：

• --logtostderr 启用日志
• ---v 日志等级
• --etcd-servers etcd集群地址
• --bind-address 监听地址
• --secure-port https安全端口
• --advertise-address 集群通告地址
• --allow-privileged 启用授权
• --service-cluster-ip-range Service虚拟IP地址段
• --enable-admission-plugins 准入控制模块
• --authorization-mode 认证授权，启用RBAC授权和节点自管理
• --enable-bootstrap-token-auth 启用TLS bootstrap功能，后面会讲到
• --token-auth-file token文件
• --service-node-port-range Service Node类型默认分配端口范围

systemd管理apiserver：

cat </usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-apiserver
ExecStart=/opt/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

启动

# systemctl daemon-reload && systemctl enable kube-apiserver && systemctl restart kube-apiserver
# ps -ef | grep kube-apiserver

4.3部署kube-scheduler

创建配置文件：

# cat </opt/kubernetes/cfg/kube-scheduler

KUBE_SCHEDULER_OPTS="--logtostderr=true \\
--v=4 \\
--master=127.0.0.1:8080 \\
--leader-elect"

EOF

参数说明：

• --master 连接本地apiserver
• --leader-elect 当该组件启动多个时，自动选举（HA）

创建systemd管理文件：

# cat </usr/lib/systemd/system/kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-scheduler
ExecStart=/opt/kubernetes/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

启动：

# systemctl daemon-reload  && systemctl enable kube-scheduler && systemctl restart kube-scheduler
# ps -ef | grep kube-scheduler

4.4部署controller-manager组件

创建controller-manager配置文件：

cat </opt/kubernetes/cfg/kube-controller-manager

KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=true \\
--v=4 \\
--master=127.0.0.1:8080 \\
--leader-elect=true \\
--address=127.0.0.1 \\
--service-cluster-ip-range=10.0.0.0/24 \\
--cluster-name=kubernetes \\
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \\
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem  \\
--root-ca-file=/opt/kubernetes/ssl/ca.pem \\
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--experimental-cluster-signing-duration=87600h0m0s"

EOF

systemd管理controller-manager组件：

cat </usr/lib/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-controller-manager
ExecStart=/opt/kubernetes/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

启动

# systemctl daemon-reload && systemctl enable kube-controller-manager && systemctl restart kube-controller-manager
# ps -ef | grep kube-controller-manager

所有组件都已经启动成功，通过kubectl工具查看当前集群组件状态：

# /opt/kubernetes/bin/kubectl get cs
NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok                  
controller-manager   Healthy   ok                  
etcd-1               Healthy   {"health":"true"}   
etcd-0               Healthy   {"health":"true"}   
etcd-2               Healthy   {"health":"true"} 
如上输出说明组件都正常。

5.部署node组件

Master apiserver启用TLS认证后，Node节点kubelet组件想要加入集群，必须使用CA签发的有效证书才能与apiserver通信，当Node节点很多时，签署证书是一件很繁琐的事情，因此有了TLS Bootstrapping机制，kubelet会以一个低权限用户自动向apiserver申请证书，kubelet的证书由apiserver动态签署。

认证大致工作流程如图所示：

5.1将kubelet-bootstrap用户绑定到系统集群角色（在master上）

kubectl create clusterrolebinding kubelet-bootstrap \
  --clusterrole=system:node-bootstrapper \
  --user=kubelet-bootstrap

5.2创建kubeconfig文件

创建kubelet bootstrapping kubeconfig（在master上）

# cd ~/k8s
# mkdir kubeconfig
# cd kubeconfig/

设置kubectl环境变量：
# vi /etc/profile
# export PATH=$PATH:/opt/kubernetes/bin/
# source /etc/profile

# 设置集群参数
kubectl config set-cluster kubernetes \
  --certificate-authority=/root/k8s/k8s-cert/ca.pem \
  --embed-certs=true \
  --server=https://192.168.1.43:6443 \
  --kubeconfig=bootstrap.kubeconfig

# 设置客户端认证参数
kubectl config set-credentials kubelet-bootstrap \
  --token=2f7a15198f7c0c3af3ba7f264b6885c2 \
  --kubeconfig=bootstrap.kubeconfig

# 设置上下文参数
kubectl config set-context default \
  --cluster=kubernetes \
  --user=kubelet-bootstrap \
  --kubeconfig=bootstrap.kubeconfig

# 设置默认上下文
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig

创建kube-proxy kubeconfig文件:（在master上）

kubectl config set-cluster kubernetes \
  --certificate-authority=/root/k8s/k8s-cert/ca.pem \
  --embed-certs=true \
  --server=https://192.168.1.43:6443 \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config set-credentials kube-proxy \
  --client-certificate=/root/k8s/k8s-cert/kube-proxy.pem \
  --client-key=/root/k8s/k8s-cert/kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig

kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

# ls
    bootstrap.kubeconfig  kube-proxy.kubeconfig

拷贝配置文件到node

# scp kube-proxy.kubeconfig bootstrap.kubeconfig root@192.168.1.30:/opt/kubernetes/cfg/
# scp kube-proxy.kubeconfig bootstrap.kubeconfig root@192.168.1.51:/opt/kubernetes/cfg/

5.3部署kubelet组件

将前面下载的二进制包中的kubelet和kube-proxy拷贝到/opt/kubernetes/bin目录下。

# cd ~/k8s/kubernetes/server/bin
# scp kubelet  kube-proxy root@192.168.1.30:/opt/kubernetes/bin/
# scp kubelet  kube-proxy root@192.168.1.51:/opt/kubernetes/bin/

创建kubelet配置文件：

cat </opt/kubernetes/cfg/kubelet

KUBELET_OPTS="--logtostderr=true \\
--v=4 \\
--hostname-override=192.168.1.30 \\
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \\
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \\
--config=/opt/kubernetes/cfg/kubelet.config \\
--cert-dir=/opt/kubernetes/ssl \\
--pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0"

EOF

参数说明：

• --hostname-override 在集群中显示的主机名
• --kubeconfig 指定kubeconfig文件位置，会自动生成
• --bootstrap-kubeconfig 指定刚才生成的bootstrap.kubeconfig文件
• --cert-dir 颁发证书存放位置
• --pod-infra-container-image 管理Pod网络的镜像

其中/opt/kubernetes/cfg/kubelet.config配置文件如下：

cat </opt/kubernetes/cfg/kubelet.config

kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 192.168.1.30 
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS:
- 10.0.0.2
clusterDomain: cluster.local.
failSwapOn: false
authentication:
  anonymous:
    enabled: true
EOF

systemd管理kubelet组件：

cat </usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
After=docker.service
Requires=docker.service

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kubelet
ExecStart=/opt/kubernetes/bin/kubelet \$KUBELET_OPTS
Restart=on-failure
KillMode=process

[Install]
WantedBy=multi-user.target
EOF

启动：

# systemctl daemon-reload && systemctl enable kubelet && systemctl restart kubelet
# ps -ef | grep kubelet

在Master审批Node加入集群：
启动后还没加入到集群中，需要手动允许该节点才可以。
在Master节点查看请求签名的Node：

# kubectl get csr
# kubectl certificate approve XXXXID
# kubectl get node

5.4部署kube-proxy组件

创建kube-proxy配置文件：

cat </opt/kubernetes/cfg/kube-proxy

KUBE_PROXY_OPTS="--logtostderr=true \\
--v=4 \\
--hostname-override=192.168.1.30 \\
--cluster-cidr=10.0.0.0/24 \\
--proxy-mode=ipvs \\
--masquerade-all=true \\
--kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig"

EOF

systemd管理kube-proxy组件：

cat </usr/lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Proxy
After=network.target

[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-proxy
ExecStart=/opt/kubernetes/bin/kube-proxy \$KUBE_PROXY_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

启动：

# systemctl daemon-reload && systemctl enable kube-proxy && systemctl restart kube-proxy
# ps -ef | grep kube-proxy

拷贝配置文件到其他node：

配置文件：
# scp -r /opt/kubernetes/ root@192.168.1.51:/opt/
systemd管理文件：
# scp /usr/lib/systemd/system/{kubelet,kube-proxy}.service root@192.168.1.51:/usr/lib/systemd/system/
删除ssl文件（master颁发）：
# rm -f /opt/kubernetes/ssl/*
修改配置文件(节点ip)：
# cd /opt/kubernetes/cfg
    kubelet，kubelet.config，kube-proxy，

启动：

# systemctl daemon-reload && systemctl enable kube-proxy && systemctl restart kube-proxy
# ps -ef | grep kube-proxy
# systemctl daemon-reload && systemctl enable kubelet && systemctl restart kubelet
# ps -ef | grep kubelet

在Master审批Node加入集群：
启动后还没加入到集群中，需要手动允许该节点才可以。
在Master节点查看请求签名的Node：

# kubectl get csr
# kubectl certificate approve XXXXID
# kubectl get node

6.查看集群状态

# kubectl get node
NAME           STATUS   ROLES    AGE   VERSION
192.168.1.30   Ready       14h   v1.12.7
192.168.1.51   Ready       23s   v1.12.7

# kubectl get cs
NAME                 STATUS    MESSAGE             ERROR
controller-manager   Healthy   ok                  
scheduler            Healthy   ok                  
etcd-1               Healthy   {"health":"true"}   
etcd-2               Healthy   {"health":"true"}   
etcd-0               Healthy   {"health":"true"}

至此单master搭建完毕，下面拓展多master

7.master02部署

拷贝所有组件到master02：
# scp -r /opt/kubernetes/ root@192.168.1.63:/opt
拷贝systemd文件拷贝：
# scp /usr/lib/systemd/system/{kube-apiserver,kube-controller-manager,kube-scheduler}.service root@192.168.1.63:/usr/lib/systemd/system/
拷贝etcd文件：
# scp -r /opt/etcd/ root@192.168.1.63:/opt/
修改apiserver地址（address）：
# vi /opt/kubernetes/cfg/kube-apiserver

启动：

启动kube-apiserver：
# systemctl daemon-reload && systemctl enable kube-apiserver && systemctl restart kube-apiserver
启动kube-scheduler：
# systemctl daemon-reload  && systemctl enable kube-scheduler && systemctl restart kube-scheduler
启动kube-controller-manager：
# systemctl daemon-reload && systemctl enable kube-controller-manager && systemctl restart kube-controller-manager
查看三个组件启动：
#ps -ef | grep kube

查看集群状态：

设置kubectl环境变量：
# vi /etc/profile
# export PATH=$PATH:/opt/kubernetes/bin/
# source /etc/profile

# kubectl get node
NAME           STATUS   ROLES    AGE   VERSION
192.168.1.30   Ready       15h   v1.12.7
192.168.1.51   Ready       53m   v1.12.7
# kubectl get cs
NAME                 STATUS    MESSAGE             ERROR
controller-manager   Healthy   ok                  
scheduler            Healthy   ok                  
etcd-0               Healthy   {"health":"true"}   
etcd-1               Healthy   {"health":"true"}   
etcd-2               Healthy   {"health":"true"}

8.Load Balancer 部署

8.1部署nginx

nginx-master：

配置源：
# vim /etc/yum.repos.d/nginx.repo
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/7/$basearch/
gpgcheck=0
安装nginx：
# yum -y install nginx
添加L4负载均衡：
# vim /etc/nginx/nginx.conf
stream {

   log_format  main  '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';
    access_log  /var/log/nginx/k8s-access.log  main;

    upstream k8s-apiserver {
        server 192.168.1.43:6443;
        server 192.168.1.63:6443;
    }
    server {
                listen 6443;
                proxy_pass k8s-apiserver;
    }
    }

启动：

关闭selinux：
# setenforce 0
# vi /etc/selinux/config
将SELINUX=enforcing改为SELINUX=disabled

#systemctl start nginx
# netstat -anpt | grep 6443

# echo "master" > /usr/share/nginx/html/index.html

nginx-backup：

配置源：
# vim /etc/yum.repos.d/nginx.repo
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/7/$basearch/
gpgcheck=0
安装nginx：
# yum -y install nginx

拷贝到backup：
# scp /etc/nginx/nginx.conf root@192.168.1.31:/etc/nginx/

关闭selinux：
# setenforce 0
# vi /etc/selinux/config
将SELINUX=enforcing改为SELINUX=disabled

#systemctl start nginx
# netstat -anpt | grep 6443

# echo "backup" > /usr/share/nginx/html/index.html

8.2部署keepalived

master和backup安装keeplived：

# yum -y install keepalived

master的keeplived配置文件：

# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
   # 接收邮件地址
   notification_email {
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   # 邮件发送地址
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id NGINX_MASTER
}

vrrp_script check_nginx {
    script "/etc/nginx/check_nginx.sh"
}

vrrp_instance VI_1 {
    state MASTER
    interface ens32
    virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的
    priority 100    # 优先级，备服务器设置 90
    advert_int 1    # 指定VRRP 心跳包通告间隔时间，默认1秒
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    virtual_ipaddress {
        192.168.1.230/24
    }
    track_script {
        check_nginx
    }
}

backup的keeplived配置文件：

! Configuration File for keepalived 

global_defs { 
   # 接收邮件地址 
   notification_email { 
     acassen@firewall.loc 
     failover@firewall.loc 
     sysadmin@firewall.loc 
   } 
   # 邮件发送地址 
   notification_email_from Alexandre.Cassen@firewall.loc  
   smtp_server 127.0.0.1 
   smtp_connect_timeout 30 
   router_id NGINX_BACKUP
} 

vrrp_script check_nginx {
    script "/etc/nginx/check_nginx.sh"
}

vrrp_instance VI_1 { 
    state BACKUP
    interface ens32
    virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的 
    priority 90    # 优先级，备服务器设置 90 
    advert_int 1    # 指定VRRP 心跳包通告间隔时间，默认1秒 
    authentication { 
        auth_type PASS      
        auth_pass 1111 
    }  
    virtual_ipaddress { 
        192.168.1.230/24 
    } 
    track_script {
        check_nginx
    } 
}

nginx检查脚本：

# vim /etc/nginx/check_nginx.sh
count=$(ps -ef |grep nginx |egrep -cv "grep|$$")

if [ "$count" -eq 0 ];then
    systemctl stop keepalived
fi

启动：

# systemctl start keepalived
关闭master的nginx进行测试：
# systemctl stop nginx

9.修改node节点apiserver为负载均衡ip

# cd /opt/kubernetes/cfg
# vi bootstrap.kubeconfig
# vi kubelet.kubeconfig
# vi kube-proxy.kubeconfig

# systemctl restart kubelet
# systemctl restart kube-proxy

10.运行一个测试示例

10.1授权kubelet：

kubectl create clusterrolebinding system:anonymous --clusterrole=cluster-admin --user=system:anonymous

10.2创建一个Nginx Web，测试集群是否正常工作：

# kubectl run nginx --image=nginx --replicas=3
# kubectl expose deployment nginx --port=88 --target-port=80 --type=NodePort

查看Pod，Service：

# kubectl get pod
NAME                    READY   STATUS    RESTARTS   AGE
nginx-dbddb74b8-j4bjq   1/1     Running   0          19m
nginx-dbddb74b8-kpqht   1/1     Running   0          19m
nginx-dbddb74b8-xjn5k   1/1     Running   0          19m
# kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.0.0.1             443/TCP        16h
nginx        NodePort    10.0.0.33            88:32694/TCP   20m

11.部署Web UI（Dashboard）

地址：https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dashboard

# cd /k8s/Dashboard
# ls
    dashboard-configmap.yaml  dashboard-controller.yaml  dashboard-rbac.yaml  dashboard-secret.yaml  dashboard-service.yaml  k8s-admin.yaml
# kubectl apply -f .

# kubectl get pod,svc -o wide --all-namespaces | grep dashboard
kube-system   pod/kubernetes-dashboard-65f974f565-crvwj   1/1     Running   1          6m1s   172.17.75.2   192.168.1.30   
kube-system   service/kubernetes-dashboard   NodePort    10.0.0.192           443:30001/TCP   6m      k8s-app=kubernetes-dashboard

访问(尽量用火狐)：https://192.168.1.30:30001

查看token：

#  kubectl get secret --all-namespaces | grep dashboard
kube-system   dashboard-admin-token-nrvzx        kubernetes.io/service-account-token   3      9m16s
kube-system   kubernetes-dashboard-certs         Opaque                                0      9m17s
kube-system   kubernetes-dashboard-key-holder    Opaque                                2      9m17s
kube-system   kubernetes-dashboard-token-cqqm8   kubernetes.io/service-account-token   3      9m17s

# kubectl describe secret dashboard-admin-token-nrvzx -n kube-system