Horizontal Pod Autoscaling,简称HPA,是Kubernetes中实现POD水平自动伸缩的功能。

简介

K8S集群可以通过Replication Controller的scale机制完成服务的扩容或缩容,实现具有伸缩性的服务。
K8S自动伸缩分为:

自动扩展主要分为两种:

  • 水平扩展(scale out),针对于实例数目的增减。
  • 垂直扩展(scal up),即单个实例可以使用的资源的增减, 比如增加cpu和增大内存。

HPA属于前者。它可以根据CPU使用率或应用自定义metrics自动扩展Pod数量(支持 replication controller、deployment 和 replica set)。
HPA架构

获取metrics的两种方式:

  • Heapster:heapster提供metrics服务,但是在v1(autoscaling/v1)版本中仅支持以CPU作为扩展度量指标。而其他比如:内存,网络流量,qps等目前处于beta阶段(autoscaling/v2beta1)。
  • Cousom:同样处于beta阶段(autoscaling/v2beta1),但是涉及到自定义的REST API的开发,复杂度会大一些,并且当需要从自定义的监控中获取数据时,只能设置绝对值,无法设置使用率。

工作流程

  • 创建HPA资源,设定目标CPU使用率限额,以及最大/最小实例数,一定要设置Pod的资源限制参数: request,否则HPA不会工作。
  • 控制管理器每隔30s(在kube-controller-manager.service中可以通过–horizontal-pod-autoscaler-sync-period修改)查询metrics的资源使用情况。
  • 然后与创建时设定的值和指标做对比(平均值之和/限额),求出目标调整的实例个数。
  • 目标调整的实例数不能超过第一条中设定的最大/最小实例数。如果没有超过,则扩容;超过,则扩容至最大的实例个数。
  • 重复第2-4步。

自动伸缩算法

HPA Controller会通过调整副本数量使得CPU使用率尽量向期望值靠近,而且不是完全相等。另官方考虑到自动扩展的决策可能需要一段时间才会生效:例如当pod所需要的CPU负荷过大,从而在创建一个新pod的过程中,系统的CPU使用量可能会同样在有一个攀升的过程。所以在每一次作出决策后的一段时间内,将不再进行扩展决策。对于扩容而言,这个时间段为3分钟,缩容为5分钟(可以通过--horizontal-pod-autoscaler-downscale-delay--horizontal-pod-autoscaler-upscale-delay进行调整)。

  • HPA Controller中有一个tolerance(容忍力)的概念,它允许一定范围内的使用量的不稳定,现在默认为0.1,这也是出于维护系统稳定性的考虑。例如设定HPA调度策略为cpu使用率高于50%触发扩容,那么只有当使用率大于55%或者小于45%才会触发伸缩活动,HPA会尽力把Pod的使用率控制在这个范围之间。
  • 具体的每次扩容或者缩容的多少Pod的算法为:Ceil(前采集到的使用率 / 用户自定义的使用率) * Pod数量)。
  • 每次最大扩容pod数量不会超过当前副本数量的2倍。

环境说明

角色 IP 操作系统版本
master 192.168.1.201 centos 7.4
etcd1 192.168.1.201 centos 7.4
etcd2 192.168.1.202 centos 7.4
etcd3 192.168.1.203 centos 7.4
node1 192.168.1.204 centos 7.4
node2 192.168.1.205 centos 7.4
环境 软件版本
kubectl server v1.9.2
kubectl client v1.9.2
Go go1.9.2
etcdctl 3.2.15
etcd 3.2.15
flanneld v0.10.0
cfssl 1.2.0
docker 18.09.1-beta1
1
2
3
4
5
6
7
[root@master ~]# kubectl cluster-info
Kubernetes master is running at https://192.168.1.201:6443
Heapster is running at https://192.168.1.201:6443/api/v1/namespaces/kube-system/services/heapster/proxy
monitoring-grafana is running at https://192.168.1.201:6443/api/v1/namespaces/kube-system/services/monitoring-grafana/proxy
monitoring-influxdb is running at https://192.168.1.201:6443/api/v1/namespaces/kube-system/services/monitoring-influxdb/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
[root@master ~]#
1
2
3
4
5
6
7
8
[root@master ~]# kubectl -s http://192.168.1.201:8080 get componentstatuses
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
etcd-2 Healthy {"health": "true"}
etcd-1 Healthy {"health": "true"}
scheduler Healthy ok
etcd-0 Healthy {"health": "true"}
[root@master ~]#
1
2
3
4
5
[root@master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
192.168.1.204 Ready <none> 21h v1.9.2
192.168.1.205 Ready <none> 21h v1.9.2
[root@master ~]#

部署HPA

先准备一套K8S集群环境,环境部署略。

创建Deployment POD应用nginx
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
[root@master ~]# cat nginx.yml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx
spec:
replicas: 1
template:
metadata:
labels:
app: nginx-hpa
spec:
containers:
- name: nginx
image: nginx:latest
ports:
- containerPort: 80
name: http
protocol: TCP
resources:
requests:
cpu: 0.01
memory: 25Mi
limits:
cpu: 0.05
memory: 60Mi
---
apiVersion: v1
kind: Service
metadata:
name: nginx
labels:
app: nginx-hpa
spec:
selector:
app: nginx-hpa
type: NodePort
ports:
- name: http
protocol: TCP
port: 80
targetPort: 80
nodePort: 30080
[root@master ~]#
1
2
3
4
5
[root@master ~]# kubectl apply -f nginx.yml
[root@master ~]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE
nginx-5dcf548595-bk9cr 1/1 Running 1 14h 172.30.94.2 192.168.1.205
[root@master ~]#
创建nginx应用的HPA
1
2
3
4
5
6
7
8
9
10
11
12
13
14
[root@master ~]# cat nginx-hpa-cpu.yml
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: nginx-hpa
spec:
scaleTargetRef:
apiVersion: extensions/v1beta1
kind: Deployment
name: nginx
minReplicas: 1
maxReplicas: 5
targetCPUUtilizationPercentage: 70
[root@master ~]#
1
2
3
4
5
[root@master ~]# kubectl apply -f nginx-hpa-cpu.yml
[root@master ~]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
nginx-hpa Deployment/nginx <unknown> / 70% 1 5 1 14h
[root@master ~]#
Q1

这时发现nginx-hpa获取不到当前的CPU情况(TARGETS)。等待几分钟后执行kubectl describe hpa发现HPA报错信息如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
[root@master ~]# kubectl describe hpa
Name: nginx-hpa
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"au
toscaling/v1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"nginx-hpa","namespace":"default"},"spec":{"maxReplic...
CreationTimestamp: Sat, 26 Jan 2019 22:23:08 +0800
Reference: Deployment/nginx
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): <unknown> / 70%
Min replicas: 1
Max replicas: 5
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetResourceMetric the HPA was unable to compute the replica count: unable to get metrics for resource cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.k8s.io)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedComputeMetricsReplicas 1m (x12 over 3m) horizontal-pod-autoscaler failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.k8s.io)
Warning FailedGetResourceMetric 1m (x13 over 3m) horizontal-pod-autoscaler unable to get metrics for resource cpu: unable to fetch metrics from API: the server could not find the requested resource (get pods.metrics.k8s.io)
[root@master ~]#

大概意思是HPA无法通过API获取到metrics值。
解决办法:
/etc/systemd/system/kube-controller-manager.service配置文件中新增--horizontal-pod-autoscaler-use-rest-clients=false配置参数。然后重启kube-controller-manager服务即可。

1
2
kube-controller-manager's parameter --horizontal-pod-autoscaler-use-rest-clients in k8s 1.9.0 default value is true , while in k8s 1.8.x is false
change it to false and it works.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
[root@master ~]# cat /etc/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]
ExecStart=/usr/local/k8s/bin/kube-controller-manager \
--address=127.0.0.1 \
--master=http://192.168.1.201:8080 \
--allocate-node-cidrs=true \
--service-cluster-ip-range=172.16.0.0/16 \
--cluster-cidr=172.30.0.0/16 \
--cluster-name=kubernetes \
--cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \
--service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \
--root-ca-file=/etc/kubernetes/ssl/ca.pem \
--leader-elect=true \
--horizontal-pod-autoscaler-use-rest-clients=false \
--v=2
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
[root@master ~]#
1
2
[root@master ~]# systemctl daemon-reload
[root@master ~]# systemctl restart kube-controller-manager
Q2

配置并重启完成kube-controller-manager服务后,执行kubectl delete -f nginx-hpa-cpu.ymlkubectl apply -f nginx-hpa-cpu.yml重新创建服务后,发现出现新的错误,信息如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
[root@master ~]# kubectl describe hpa
Name: nginx-hpa
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"au
scaling/v1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"nginx-hpa","namespace":"default"},"spec{"maxRepl...
CreationTimestamp: Sun, 27 Jan 2019 00:18:02 +0800
Reference: Deployment/nginx
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): <unknown> / 70%
Min replicas: 1
Max replicas: 5
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetResourceMetric the HPA was unable to compute the replica count: unable to get metrics r resource cpu: failed to get pod resource metrics: an error on the server ("Error: 'dial tcp 172.30.9.4:8082: getsockoptconnection timed out'\nTrying to reach: 'http://172.30.9.4:8082/apis/metrics/v1alpha1/namespaces/default/pods?labelSelect=app%3Dnginx-hpa'") has prevented the request from succeeding (get services http:heapster:)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedUpdateStatus 2m horizontal-pod-autoscaler Operation cannot be fulfilled on hozontalpodautoscalers.autoscaling "nginx-hpa": the object has been modified; please apply your changes to the latest versi and try again
Warning FailedGetResourceMetric 24s (x3 over 4m) horizontal-pod-autoscaler unable to get metrics for resource u: failed to get pod resource metrics: an error on the server ("Error: 'dial tcp 172.30.9.4:8082: getsockopt: connection med out'\nTrying to reach: 'http://172.30.9.4:8082/apis/metrics/v1alpha1/namespaces/default/pods?labelSelector=app%3Dnginhpa'") has prevented the request from succeeding (get services http:heapster:)
Warning FailedComputeMetricsReplicas 24s (x3 over 4m) horizontal-pod-autoscaler failed to get cpu utilization: unab to get metrics for resource cpu: failed to get pod resource metrics: an error on the server ("Error: 'dial tcp 172.30.9.4:8082: getsockopt: connection timed out'\nTrying to reach: 'http://172.30.9.4:8082/apis/metrics/v1alpha1/namespaces/defaulpods?labelSelector=app%3Dnginx-hpa'") has prevented the request from succeeding (get services http:heapster:)
[root@master ~]#

意思是HPA无法连接heapster服务。于是检查heapster服务是否异常。

1
2
3
4
5
6
7
[root@master ~]# kubectl get pod -o wide -n kube-system
NAME READY STATUS RESTARTS AGE IP NODE
heapster-6d5c495969-2rgcr 1/1 Running 2 20h 172.30.9.4 192.168.1.204
kubernetes-dashboard-cbbf9945c-bkvbk 1/1 Running 2 20h 172.30.9.3 192.168.1.204
monitoring-grafana-67d68bf9c6-zv928 1/1 Running 2 20h 172.30.9.2 192.168.1.204
monitoring-influxdb-7c4c46745f-kbxgb 1/1 Running 0 <invalid> 172.30.9.5 192.168.1.204
[root@master ~]#

访问kube-dashboard发现POD是可以通过heapster获取到CPU内存的信息的。如下,说明heapster工作正常。
kube-dashboard
于是到node节点手动curl访问连接异常的URL。经测试在node1节点上访问正常。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
[root@node1 ~]# curl 'http://172.30.9.4:8082/apis/metrics/v1alpha1/namespaces/default/pods?labelSelector=app%3Dnginx-hpa'
{
"metadata": {},
"items": [
{
"metadata": {
"name": "nginx-5dcf548595-bk9cr",
"namespace": "default",
"creationTimestamp": "2019-01-27T07:29:43Z"
},
"timestamp": "2019-01-27T07:29:00Z",
"window": "1m0s",
"containers": [
{
"name": "nginx",
"usage": {
"cpu": "0",
"memory": "2820Ki"
}
}
]
}
]
}
[root@node1 ~]#

于是到kube-master上访问测试,发现HPA无法访问到heapster。

1
2
3
[root@master ~]# curl 'http://172.30.9.4:8082/apis/metrics/v1alpha1/namespaces/default/pods?labelSelector=app%3Dnginx-hpa'
curl: (7) Failed connect to 172.30.9.4:8082; Connection timed out
[root@master ~]#

接下来我们来测试下网络情况,发现kube-master无法Ping通heapster的POD地址。

1
2
3
4
5
6
7
8
9
10
[root@master ~]# ping 172.30.9.4
PING 172.30.9.4 (172.30.9.4) 56(84) bytes of data.
^C
--- 172.30.9.4 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1002ms
[root@master ~]# telnet 172.30.9.4 8082
Trying 172.30.9.4...
telnet: connect to address 172.30.9.4: Connection timed out
[root@master ~]#

测试发现是网络不通导致的。解决办法是在kube-master上安装flannel网络。
如果flannel网络的IP地址丢失,重启flannel网卡systemctl restart flanneld即可解决。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
[root@localhost ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:48:f6:1d brd ff:ff:ff:ff:ff:ff
inet 192.168.1.201/24 brd 192.168.1.255 scope global ens33
valid_lft forever preferred_lft forever
inet6 fe80::22d8:9dda:6705:ec09/64 scope link
valid_lft forever preferred_lft forever
3: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN
link/ether 6e:05:c0:9c:34:3f brd ff:ff:ff:ff:ff:ff
inet 172.30.13.0/32 scope global flannel.1
valid_lft forever preferred_lft forever
inet6 fe80::6c05:c0ff:fe9c:343f/64 scope link
valid_lft forever preferred_lft forever
[root@localhost ~]#

再测试下kube-master到heapster POD的网络情况:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[root@master ~]# ping 172.30.9.4 -c 4
PING 172.30.9.4 (172.30.9.4) 56(84) bytes of data.
64 bytes from 172.30.9.4: icmp_seq=1 ttl=63 time=2.15 ms
64 bytes from 172.30.9.4: icmp_seq=2 ttl=63 time=1.27 ms
64 bytes from 172.30.9.4: icmp_seq=3 ttl=63 time=1.30 ms
64 bytes from 172.30.9.4: icmp_seq=4 ttl=63 time=1.66 ms
--- 172.30.9.4 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3003ms
rtt min/avg/max/mdev = 1.277/1.599/2.150/0.354 ms
[root@master ~]# telnet 172.30.9.4 8082
Trying 172.30.9.4...
telnet: connect to address 172.30.9.4: Connection refused
[root@master ~]#

重新导入nginx-hpa-cpu.yml文件,然后等待几分钟…

1
2
3
4
5
6
[root@localhost ~]# kubectl delete -f nginx-hpa-cpu.yml
horizontalpodautoscaler "nginx-hpa" deleted
[root@localhost ~]#
[root@localhost ~]# kubectl apply -f nginx-hpa-cpu.yml
horizontalpodautoscaler "nginx-hpa" created
[root@localhost ~]#

OK,HPA连接heapster成功。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
[root@localhost ~]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
nginx-hpa Deployment/nginx 0% / 70% 1 5 1 39s
[root@localhost ~]#
[root@localhost ~]# kubectl describe hpa
Name: nginx-hpa
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"au
toscaling/v1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"nginx-hpa","namespace":"default"},"spec":{"maxRepl...
CreationTimestamp: Sun, 27 Jan 2019 01:04:25 +0800
Reference: Deployment/nginx
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): 0% (0) / 70%
Min replicas: 1
Max replicas: 5
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True ReadyForNewScale the last scale time was sufficiently old as to warrant a new scale
ScalingActive True ValidMetricFound the HPA was able to succesfully calculate a replica count from cpu resource utilization (percentage of request)
ScalingLimited True TooFewReplicas the desired replica count is increasing faster than the maximum scale rate
Events: <none>
[root@localhost ~]#

HPA测试

截至目前,HPA支持的API版本有三个。分别是autoscaling/v1autoscaling/v2beta1autoscaling/v2beta2。其中autoscaling/v1只支持CPU一种伸缩指标;在autoscaling/v2beta1中增加支持custom metrics;在autoscaling/v2beta2中增加支持external metrics。
详细说明参考:

官方说明,在k8s 1.11版本,HPA将不再从heapster上获取指标。

1
2
3
4
5
The HorizontalPodAutoscaler normally fetches metrics from a series of aggregated APIs (metrics.k8s.io, custom.metrics.k8s.io, and external.metrics.k8s.io). The metrics.k8s.io API is usually provided by metrics-server, which needs to be launched separately. See metrics-server for instructions. The HorizontalPodAutoscaler can also fetch metrics directly from Heapster.
Note:
FEATURE STATE: Kubernetes 1.11 deprecated
Fetching metrics from Heapster is deprecated as of Kubernetes 1.11.

autoscaling/v1
1
2
3
4
5
6
7
8
9
10
11
12
13
14
[root@master ~]# cat nginx-hpa-cpu.yml
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: nginx-hpa
spec:
scaleTargetRef:
apiVersion: extensions/v1beta1
kind: Deployment
name: nginx
minReplicas: 1
maxReplicas: 5
targetCPUUtilizationPercentage: 70
[root@master ~]#

这里只针对CPU的HPA 压力测试。
压测命令

1
2
3
4
5
6
[root@node1 ~]# cat test.sh
while true
do
wget -q -O- http://192.168.1.204:30080
done
[root@node1 ~]# sh test.sh

观察HPA当前负载和POD的情况

1
2
3
4
[root@master ~]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
nginx-hpa Deployment/nginx 0% / 70% 1 5 1 14h
[root@master ~]#

1
2
3
4
[root@master ~]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
nginx-hpa Deployment/nginx 14% / 70% 1 5 1 14h
[root@master ~]#

当负载飙升时,HPA会按照定义的规则开始创建新的POD副本(定义POD的CPU阈值为70%)。

1
2
3
4
5
6
7
8
9
10
[root@master ~]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
nginx-hpa Deployment/nginx 180% / 70% 1 5 3 14h
[root@master ~]#
[root@master ~]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE
nginx-5dcf548595-bk9cr 1/1 Running 1 15h 172.30.94.2 192.168.1.205
nginx-5dcf548595-pdndb 1/1 Running 0 1m 172.30.94.4 192.168.1.205
nginx-5dcf548595-z9d6h 1/1 Running 0 1m 172.30.94.3 192.168.1.205
[root@master ~]#

继续压测,会发现POD副本数量继续增加(REPLICAS从3到5)。

1
2
3
4
5
6
7
8
9
10
11
12
[root@master ~]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
nginx-hpa Deployment/nginx 139% / 70% 1 5 5 14h
[root@master ~]#
[root@master ~]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE
nginx-5dcf548595-9gmqf 0/1 ContainerCreating 0 39s <none> 192.168.1.204
nginx-5dcf548595-bk9cr 1/1 Running 1 15h 172.30.94.2 192.168.1.205
nginx-5dcf548595-pdndb 1/1 Running 0 10m 172.30.94.4 192.168.1.205
nginx-5dcf548595-r7n4b 1/1 Running 0 39s 172.30.94.5 192.168.1.205
nginx-5dcf548595-z9d6h 1/1 Running 0 10m 172.30.94.3 192.168.1.205
[root@master ~]#

当REPLICAS达到定义的上限时,即使当前CPU的压力仍然很大,REPLICAS也不会再增加了。

1
2
3
4
5
6
7
8
9
10
11
12
[root@master ~]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
nginx-hpa Deployment/nginx 112% / 70% 1 5 5 14h
[root@master ~]#
[root@master ~]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE
nginx-5dcf548595-9gmqf 1/1 Running 0 2m 172.30.9.6 192.168.1.204
nginx-5dcf548595-bk9cr 1/1 Running 1 15h 172.30.94.2 192.168.1.205
nginx-5dcf548595-pdndb 1/1 Running 0 12m 172.30.94.4 192.168.1.205
nginx-5dcf548595-r7n4b 1/1 Running 0 2m 172.30.94.5 192.168.1.205
nginx-5dcf548595-z9d6h 1/1 Running 0 12m 172.30.94.3 192.168.1.205
[root@master ~]#

停止压测,当CPU负载降低时,HPA会自动减少POD的数量。

1
2
3
4
5
6
7
8
9
10
[root@master ~]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
nginx-hpa Deployment/nginx 40% / 70% 1 5 3 14h
[root@master ~]#
[root@master ~]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE
nginx-5dcf548595-pdndb 1/1 Running 0 16m 172.30.94.4 192.168.1.205
nginx-5dcf548595-r7n4b 1/1 Running 0 6m 172.30.94.5 192.168.1.205
nginx-5dcf548595-z9d6h 1/1 Running 0 16m 172.30.94.3 192.168.1.205
[root@master ~]#

慢慢的,HPA会减少POD的数量,直到降低到最小POD数(MINPODS)。

1
2
3
4
5
6
7
8
[root@master ~]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
nginx-hpa Deployment/nginx 0% / 70% 1 5 1 15h
[root@master ~]#
[root@master ~]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE
nginx-5dcf548595-z9d6h 1/1 Running 0 1h 172.30.94.3 192.168.1.205
[root@master ~]#

通过kube-dashboard观察这个过程的变化。
kube-dashboard
kube-dashboard
kube-dashboard
通过HPA的日志信息查看到它伸缩的过程。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
[root@master ~]# kubectl describe hpa
Name: nginx-hpa
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"au
toscaling/v1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"nginx-hpa","namespace":"default"},"spec":{"maxRepl...CreationTimestamp: Sun, 27 Jan 2019 01:04:25 +0800
Reference: Deployment/nginx
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): 0% (0) / 70%
Min replicas: 1
Max replicas: 5
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale False BackoffDownscale the time since the previous scale is still within the downscale forbidden window
ScalingActive True ValidMetricFound the HPA was able to succesfully calculate a replica count from cpu resource utilization (percentage of request) ScalingLimited True TooFewReplicas the desired replica count is increasing faster than the maximum scale rate
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulRescale 41m (x2 over 1h) horizontal-pod-autoscaler New size: 5; reason: cpu resource utilization (percentage of request) above target
Normal SuccessfulRescale 29m (x2 over 1h) horizontal-pod-autoscaler New size: 3; reason: All metrics below target
Normal SuccessfulRescale 17m horizontal-pod-autoscaler New size: 2; reason: All metrics below target
Normal SuccessfulRescale 8m (x2 over 1h) horizontal-pod-autoscaler New size: 3; reason: cpu resource utilization (percentage of request) above target
Normal SuccessfulRescale 3m (x2 over 12m) horizontal-pod-autoscaler New size: 1; reason: All metrics below target
[root@master ~]#

autoscaling/v2beta1

autoscaling/v2beta1中增加支持custom metrics。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
[root@master ~]# cat nginx-hpa-v2beta1.yml
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: nginx-hpa
spec:
scaleTargetRef:
apiVersion: extensions/v1beta1
kind: Deployment
name: nginx
minReplicas: 1
maxReplicas: 5
metrics:
- type: Resource
resource:
name: memory
targetAverageUtilization: 70
- type: Resource
resource:
name: cpu
targetAverageUtilization: 70
[root@master ~]#

1
[root@master ~]# kubectl apply -f nginx-hpa-v2beta1.yml

等待几分钟后…
观察发现前面10%是内存的使用百分比,后面0%是CPU的使用百分比。

1
2
3
4
[root@master ~]# kubectl get hpa nginx-hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
nginx-hpa Deployment/nginx 10% / 70%, 0% / 70% 1 5 1 51s
[root@master ~]#

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
[root@master ~]# kubectl describe hpa nginx-hpa
Name: nginx-hpa
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":
"autoscaling/v2beta1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"nginx-hpa","namespace":"default"},"spec":{"ma...CreationTimestamp: Mon, 28 Jan 2019 22:22:01 +0800
Reference: Deployment/nginx
Metrics: ( current / target )
resource memory on pods (as a percentage of request): 10% (2670592) / 70%
resource cpu on pods (as a percentage of request): 0% (0) / 70%
Min replicas: 1
Max replicas: 5
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True ReadyForNewScale the last scale time was sufficiently old as to warrant a new scale
ScalingActive True ValidMetricFound the HPA was able to succesfully calculate a replica count from memory resou
rce utilization (percentage of request) ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
Events: <none>
[root@master ~]#
autoscaling/v2beta2

autoscaling/v2beta2测试发现目前k8s 1.9.2暂不支持这个API版本。

1
2
3
[root@master ~]# kubectl get hpa.v2beta2.autoscaling -o yaml
the server doesn't have a resource type "hpa" in group "v2beta2.autoscaling"
[root@master ~]#

参考:
http://blog.51cto.com/ylw6006/2113848
https://blog.frognew.com/2017/01/kubernetes-pod-scale.html
https://k8smeetup.github.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/
https://blog.csdn.net/qq_17016649/article/details/79297796
https://github.com/kubernetes/kubernetes/issues/57673

附件:
HPA测试配置文件.zip


本作品采用知识共享署名 2.5 中国大陆许可协议进行许可,欢迎转载,但转载请注明来自Jack Wang Blog,并保持转载后文章内容的完整。本人保留所有版权相关权利。
打赏
本文出自”Jack Wang Blog”:http://www.yfshare.vip/2019/01/28/k8s集群水平扩展-HPA/