1 源码分析
如果全部前提工作准备完毕,启动和停止k8s的方法非常简单分别单独一条命令(在cluster目录下):
KUBERNETES_PROVIDER=ubuntu ./kube-up.sh
KUBERNETES_PROVIDER=ubuntu ./kube-down.sh
下面分析下这条命令具体做了哪些事情。
#!/bin/bash
# Copyright 2014 The Kubernetes Authors All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Bring up a Kubernetes cluster.
#
# If the full release name (gs://<bucket>/<release>) is passed in then we take
# that directly. If not then we assume we are doing development stuff and take
# the defaults in the release config.
set -o errexit
set -o nounset
set -o pipefail
KUBE_ROOT=$(dirname "${BASH_SOURCE}")/..
if [ -f "${KUBE_ROOT}/cluster/env.sh" ]; then
source "${KUBE_ROOT}/cluster/env.sh"
fi
source "${KUBE_ROOT}/cluster/kube-env.sh"
source "${KUBE_ROOT}/cluster/kube-util.sh"
echo "... Starting cluster using provider: $KUBERNETES_PROVIDER" >&2
echo "... calling verify-prereqs" >&2
verify-prereqs
echo "... calling kube-up" >&2
kube-up
echo "... calling validate-cluster" >&2
validate-cluster
echo -e "Done, listing cluster services:
" >&2
"${KUBE_ROOT}/cluster/kubectl.sh" cluster-info
echo
exit 0
脚本执行结构图:
kube-up.sh
- env.sh
- kube-env.sh
- kube-util.sh
- util.sh
- config-default.sh
- build.sh
首先设置KUBERNETES_PROVIDER环境变量为ubuntu(默认此环境变量在kube-env.sh中初始化),目的是指定我们要使用cluster/ubuntu下的脚本。真正工作的脚本是util.sh,里面定义了后面要执行的一些函数,例如,verify-prereqs,kube-up,validate-cluster。
(1) verify-prereqs
作用:在客户端和服务器端通过使用ssh建立安全通信,同时使用ssh-agent避免了每次都输入密码的麻烦。
过程:首先执行ssh-add -L命令,显示所有身份认证的公钥,如果echo $?显示返回值为2,说明ssh-agent服务没有启动,则执行ssh-agent命令来启动ssh-agent服务。然后再执行ssh-add -L判断,如果返回值是1(The agent has no identities.),则执行ssh-add添加一个默认的identities。最后再执行ssh-add -L判断添加是否成功。关于ssh-agent的用法可以参考链接[3],Anyway, here is how to set up a pair of keys for passwordless authentication via ssh-agent. 。
# Verify ssh prereqs
function verify-prereqs {
local rc
rc=0
ssh-add -L 1> /dev/null 2> /dev/null || rc="$?"
# "Could not open a connection to your authentication agent."
if [[ "${rc}" -eq 2 ]]; then
eval "$(ssh-agent)" > /dev/null
trap-add "kill ${SSH_AGENT_PID}" EXIT
fi
rc=0
ssh-add -L 1> /dev/null 2> /dev/null || rc="$?"
# "The agent has no identities."
if [[ "${rc}" -eq 1 ]]; then
# Try adding one of the default identities, with or without passphrase.
ssh-add || true
fi
# Expect at least one identity to be available.
if ! ssh-add -L 1> /dev/null 2> /dev/null; then
echo "Could not find or add an SSH identity."
echo "Please start ssh-agent, add your identity, and retry."
exit 1
fi
}
在/etc/profile配置中添加如下代码,实现每次登陆终端后都自动启动ssh-agent服务。
# ssh-agent auto-load
SSH_ENV="$HOME/.ssh/environment"
function start_agent {
echo "Initialising new SSH agent..."
/usr/bin/ssh-agent | sed 's/^echo/#echo/' > "${SSH_ENV}"
echo succeeded
chmod 600 "${SSH_ENV}"
. "${SSH_ENV}" > /dev/null
/usr/bin/ssh-add;
}
# Source SSH settings, if applicable
if [ -f "${SSH_ENV}" ]; then
. "${SSH_ENV}" > /dev/null
#ps ${SSH_AGENT_PID} doesn't work under cywgin
ps -ef | grep ${SSH_AGENT_PID} | grep ssh-agent$ > /dev/null || {
start_agent;
}
else
start_agent;
fi
(2) kube-up
作用:在ubuntu上初始化k8s cluster。
过程:先默认加载config-default.sh配置,然后检查ubuntu/binaries/master/kube-apiserver是否存在,如果没有则执行build.sh下载需要的二进制文件。setClusterInfo根据config-default.sh中配置的节点信息设置master和node的IP。设置完分簇信息后,再根据配置的节点类型a/i/ai分别调用provision-master/provision-node/provision-masterandnode函数,通过scp方法向每个目标节点复制需要的配置和可执行文件(在~/kube目录下),完成master/node/master-node节点的部署。
# Instantiate a kubernetes cluster on ubuntu
function kube-up() {
source "${KUBE_ROOT}/cluster/ubuntu/${KUBE_CONFIG_FILE-"config-default.sh"}"
# ensure the binaries are well prepared
if [ ! -f "ubuntu/binaries/master/kube-apiserver" ]; then
echo "No local binaries for kube-up, downloading... "
"${KUBE_ROOT}/cluster/ubuntu/build.sh"
fi
setClusterInfo
ii=0
for i in ${nodes}
do
{
if [ "${roles[${ii}]}" == "a" ]; then
provision-master
elif [ "${roles[${ii}]}" == "ai" ]; then
provision-masterandnode
elif [ "${roles[${ii}]}" == "i" ]; then
provision-node $i
else
echo "unsupported role for ${i}. please check"
exit 1
fi
}
((ii=ii+1))
done
wait
verify-cluster
detect-master
export CONTEXT="ubuntu"
export KUBE_SERVER="http://${KUBE_MASTER_IP}:8080"
source "${KUBE_ROOT}/cluster/common.sh"
# set kubernetes user and password
load-or-gen-kube-basicauth
create-kubeconfig
}
关于build.sh:
此脚本可以在kube-up.sh脚本前执行,在下载需要的二进制文件前,可以通过环境变量指定需要下载的具体版本,在执行build.sh前可以先执行下面的脚本:
#cd kubernetes/cluster/ubuntu
#source init_version.sh
#init_version.sh
#!/bin/bash
export KUBE_VERSION=1.1.2
export FLANNEL_VERSION=0.5.5
export ETCD_VERSION=2.2.1
echo "done"
关于setClusterInfo:
nodes和roles变量都在config-default.sh初始化,然后根据初始化结果设置MASTER_IP和NODE_IPS。
# From user input set the necessary k8s and etcd configuration information
function setClusterInfo() {
# Initialize NODE_IPS in setClusterInfo function
# NODE_IPS is defined as a global variable, and is concatenated with other nodeIP
# When setClusterInfo is called for many times, this could cause potential problems
# Such as, you will have NODE_IPS=192.168.0.2,192.168.0.3,192.168.0.2,192.168.0.3 which is obviously wrong
NODE_IPS=""
ii=0
for i in $nodes; do
nodeIP=${i#*@}
if [[ "${roles[${ii}]}" == "ai" ]]; then
MASTER_IP=$nodeIP
MASTER=$i
NODE_IPS="$nodeIP"
elif [[ "${roles[${ii}]}" == "a" ]]; then
MASTER_IP=$nodeIP
MASTER=$i
elif [[ "${roles[${ii}]}" == "i" ]]; then
if [[ -z "${NODE_IPS}" ]];then
NODE_IPS="$nodeIP"
else
NODE_IPS="$NODE_IPS,$nodeIP"
fi
else
echo "unsupported role for ${i}. please check"
exit 1
fi
((ii=ii+1))
done
}
关于provision-masterandnode:
function provision-masterandnode() {
# copy the binaries and scripts to the ~/kube directory on the master
echo "Deploying master and node on machine ${MASTER_IP}"
echo "SSH_OPTS=$SSH_OPTS"
echo "MASTER=$MASTER"
echo "SERVICE_CLUSTER_IP_RANGE=$SERVICE_CLUSTER_IP_RANGE"
echo "ADMISSION_CONTROL=$ADMISSION_CONTROL"
echo "SERVICE_NODE_PORT_RANGE=$SERVICE_NODE_PORT_RANGE"
echo "NODE_IPS=$NODE_IPS"
echo "DNS_SERVER_IP=$DNS_SERVER_IP"
echo "DNS_DOMAIN=$DNS_DOMAIN"
echo "FLANNEL_NET=$FLANNEL_NET"
echo
ssh $SSH_OPTS $MASTER "mkdir -p ~/kube/default"
# scp order matters
scp -r $SSH_OPTS ubuntu/config-default.sh ubuntu/util.sh ubuntu/minion/* ubuntu/master/* ubuntu/reconfDocker.sh ubuntu/binaries/master/ ubuntu/binaries/minion "${MASTER}:~/kube"
# remote login to the node and use sudo to configue k8s
ssh $SSH_OPTS -t $MASTER "source ~/kube/util.sh;
setClusterInfo;
create-etcd-opts;
create-kube-apiserver-opts "${SERVICE_CLUSTER_IP_RANGE}" "${ADMISSION_CONTROL}" "${SERVICE_NODE_PORT_RANGE}";
create-kube-controller-manager-opts "${NODE_IPS}";
create-kube-scheduler-opts;
create-kubelet-opts "${MASTER_IP}" "${MASTER_IP}" "${DNS_SERVER_IP}" "${DNS_DOMAIN}";
create-kube-proxy-opts "${MASTER_IP}";
create-flanneld-opts "127.0.0.1";
sudo -p '[sudo] password to start master: ' cp ~/kube/default/* /etc/default/ && sudo cp ~/kube/init_conf/* /etc/init/ && sudo cp ~/kube/init_scripts/* /etc/init.d/ ;
sudo mkdir -p /opt/bin/ && sudo cp ~/kube/master/* /opt/bin/ && sudo cp ~/kube/minion/* /opt/bin/;
sudo service etcd start;
sudo FLANNEL_NET=${FLANNEL_NET} -b ~/kube/reconfDocker.sh "ai";"
}
关于verify-cluster:
function verify-cluster {
ii=0
for i in ${nodes}
do
if [ "${roles[${ii}]}" == "a" ]; then
verify-master
elif [ "${roles[${ii}]}" == "i" ]; then
verify-node $i
elif [ "${roles[${ii}]}" == "ai" ]; then
verify-master
verify-node $i
else
echo "unsupported role for ${i}. please check"
exit 1
fi
((ii=ii+1))
done
echo
echo "Kubernetes cluster is running. The master is running at:"
echo
echo " http://${MASTER_IP}:8080"
echo
}
关于verify-master:
function verify-master(){
# verify master has all required daemons
printf "Validating master"
local -a required_daemon=("kube-apiserver" "kube-controller-manager" "kube-scheduler")
local validated="1"
local try_count=1
local max_try_count=3
until [[ "$validated" == "0" ]]; do
validated="0"
local daemon
for daemon in "${required_daemon[@]}"; do
ssh $SSH_OPTS "$MASTER" "pgrep -f ${daemon}" >/dev/null 2>&1 || {
printf "."
validated="1"
((try_count=try_count+1))
if [[ ${try_count} -gt ${max_try_count} ]]; then
printf "
Warning: Process "${daemon}" failed to run on ${MASTER}, please check.
"
exit 1
fi
sleep 2
}
done
done
printf "
"
}
关于verify-node:
function verify-node(){
# verify node has all required daemons
printf "Validating ${1}"
local -a required_daemon=("kube-proxy" "kubelet" "docker")
local validated="1"
local try_count=1
local max_try_count=3
until [[ "$validated" == "0" ]]; do
validated="0"
local daemon
for daemon in "${required_daemon[@]}"; do
ssh $SSH_OPTS "$1" "pgrep -f $daemon" >/dev/null 2>&1 || {
printf "."
validated="1"
((try_count=try_count+1))
if [[ ${try_count} -gt ${max_try_count} ]]; then
printf "
Warning: Process "${daemon}" failed to run on ${1}, please check.
"
exit 1
fi
sleep 2
}
done
done
printf "
"
}
3 测试
可以使用kubectl工具以命令行的方式操作k8s。
root@gerryyang:~/k8s/test/kubernetes/k8s_1.1.3/kubernetes/cluster/ubuntu/binaries# ./kubectl --help
kubectl controls the Kubernetes cluster manager.
Find more information at https://github.com/kubernetes/kubernetes.
Usage:
kubectl [flags]
kubectl [command]
Available Commands:
get Display one or many resources
describe Show details of a specific resource or group of resources
create Create a resource by filename or stdin
replace Replace a resource by filename or stdin.
patch Update field(s) of a resource by stdin.
delete Delete resources by filenames, stdin, resources and names, or by resources and label selector.
edit Edit a resource on the server
apply Apply a configuration to a resource by filename or stdin
namespace SUPERSEDED: Set and view the current Kubernetes namespace
logs Print the logs for a container in a pod.
rolling-update Perform a rolling update of the given ReplicationController.
scale Set a new size for a Replication Controller.
attach Attach to a running container.
exec Execute a command in a container.
port-forward Forward one or more local ports to a pod.
proxy Run a proxy to the Kubernetes API server
run Run a particular image on the cluster.
stop Deprecated: Gracefully shut down a resource by name or filename.
expose Take a replication controller, service or pod and expose it as a new Kubernetes Service
autoscale Auto-scale a replication controller
label Update the labels on a resource
annotate Update the annotations on a resource
config config modifies kubeconfig files
cluster-info Display cluster info
api-versions Print the supported API versions on the server, in the form of "group/version".
version Print the client and server version information.
help Help about any command
Flags:
--alsologtostderr[=false]: log to standard error as well as files
--api-version="": The API version to use when talking to the server
--certificate-authority="": Path to a cert. file for the certificate authority.
--client-certificate="": Path to a client key file for TLS.
--client-key="": Path to a client key file for TLS.
--cluster="": The name of the kubeconfig cluster to use
--context="": The name of the kubeconfig context to use
--insecure-skip-tls-verify[=false]: If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure.
--kubeconfig="": Path to the kubeconfig file to use for CLI requests.
--log-backtrace-at=:0: when logging hits line file:N, emit a stack trace
--log-dir="": If non-empty, write log files in this directory
--log-flush-frequency=5s: Maximum number of seconds between log flushes
--logtostderr[=true]: log to standard error instead of files
--match-server-version[=false]: Require server version to match client version
--namespace="": If present, the namespace scope for this CLI request.
--password="": Password for basic authentication to the API server.
-s, --server="": The address and port of the Kubernetes API server
--stderrthreshold=2: logs at or above this threshold go to stderr
--token="": Bearer token for authentication to the API server.
--user="": The name of the kubeconfig user to use
--username="": Username for basic authentication to the API server.
--v=0: log level for V logs
--vmodule=: comma-separated list of pattern=N settings for file-filtered logging
Use "kubectl [command] --help" for more information about a command.
kube*
etcd
flannel
4 参考
[1] https://github.com/kubernetes/kubernetes/blob/master/docs/getting-started-guides/ubuntu.md
[2] https://github.com/kubernetes/kubernetes/releases
[3] http://mah.everybody.org/docs/ssh