.NET Core + Kubernetes:StatefulSet

在 Kubernetes 中,Pod 资源的控制器 Deployment、Replicaset、Daemonset 等常用于管理无状态应用,它们所管理的 Pod 对应的 IP、名字,启停顺序等都是随机的,Pod 之间也并不存在任何关联关系。而实际情况下,在应用集群部署时,实例彼此之间可能是需要存在关联关系的(启动顺序、角色),如 MySQL、MongoDB,所以 StatefulSet 就是为了运行有状态服务引入的一种资源类型,StatefulSet 为每个 Pod 维持一个唯一且固定的标识符,必要时还会为其创建专用的存储卷,当 Pod 被重建时,也依然能保持原来的标识符和存储卷。

完整的 StatefulSet 通常由三部分构成:StatefulSetVolumeClaimTemplateHeadless Service

StatefulSet 用于 Pod 资源定义与管控,在 StatefulSet 模式下,Pod 有自己固定的命名规则(StatfulSet 名称 + Pod 创建时所在的索引),假设设置的 StatefulSet 名称为 k8sdemo,replicas 为3,则对应的 Pod 名称将分别是 k8sdemo-0k8sdemo-1k8sdemo-0,同时在进行 Pod 副本伸缩时也能做到按序号进行升降。

VolumeClaimTemplate 用于定义 Pod 所需存储的 PVC 声明 ,PVC 与 PV 进行绑定,提供专有固定的存储卷。

Headless ServiceclusterIP: None)用于为 Pod 生成可解析的 DNS 域名记录,基于 Pod 名称的有序规则,Pod 域名是不会变的(Pod 名称.serviceName),这也保证了 Pod 网络标识的稳定性。

下面继续以 .NET Core 项目构建的 beckjin/k8sdemo:1.2.0 镜像为例,增加了接口访问日志记录的功能。通过集成 log4net 将接口访问日志进行文件记录,日志将输出到 /Data/ 目录,每个 Pod 都会拥有自己的一份日志文件(这只是一个假设的场景,切勿较真,实际情况下日志记录一般都会使用统一的日志采集工具)。

定义资源

k8sdemo-statefulset.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: k8sdemo
spec:
serviceName: "k8sdemo-service" # 需要与创建的 service name 一致
replicas: 3
selector:
matchLabels:
name: k8sdemo
template:
metadata:
labels:
name: k8sdemo
spec:
containers:
- name: k8sdemo
image: beckjin/k8sdemo:1.2.0
imagePullPolicy: IfNotPresent
volumeMounts:
- name: data
mountPath: /app/Data # 将容器内的 Data 目录进行挂载
volumeClaimTemplates: # 定义模板,自动创建 PVC
- metadata:
name: data
spec:
accessModes:
- ReadOnlyMany
resources:
requests:
storage: 100Mi
storageClassName: "k8sdemo-sc" # 将自动与集群内 storageClassName 匹配的 PV 进行绑定

k8sdemo-service.yaml

1
2
3
4
5
6
7
8
9
10
11
apiVersion: v1
kind: Service
metadata:
name: k8sdemo-service
spec:
clusterIP: None
ports:
- port: 80
targetPort: 80
selector:
name: k8sdemo

StatefulSet 模式下需要设置 serviceName 字段,用来告诉 StatefulSet 控制器具体使用哪个 service 来解析它所管理的 Pod。同时通过 volumeClaimTemplates 字段进行 PVC 定义,StatefulSet 控制器会自动创建与 Pod 对应的 PVC,PVC 的名称为 (volumeClaimTemplateName)-(podName),然后 PVC 会自动与满足要求的 PV 进行绑定,PV 如果不支持自动创建可手动完成。另外当 Pod 被删除时 PVC 与 PV 依然会被保留,Pod 重建时会重新关联之前对应的 PVC 与 PV。

这里还是使用的 NFS 创建 PV 来实现存储,分别创建 3 个(data-k8sdemo-pv-[1~3])满足定义要求的 PV,如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
apiVersion: v1
kind: PersistentVolume
metadata:
name: data-k8sdemo-pv-1
spec:
nfs:
server: 192.168.124.21
path: /statefulset/data1
accessModes:
- ReadOnlyMany
capacity:
storage: 100Mi
storageClassName: k8sdemo-sc

部署与测试

创建 PV 与 StatefulSet:

1
2
3
4
kubectl apply -f k8sdemo-statefulset-pv1.yaml
kubectl apply -f k8sdemo-statefulset-pv2.yaml
kubectl apply -f k8sdemo-statefulset-pv3.yaml
kubectl apply -f k8sdemo-statefulset.yaml

注意:: PV 命名顺序并不代表被 PVC 的绑定顺序,这两者没有关系,所以不用对上图的数字编号对应关系有疑问。

创建 Service:

1
kubectl apply -f k8sdemo-service.yaml

因为 Service 定义的是 Headless 模式,所以需要进去 Pod 内进行接口访问测试,如:kubectl exec -it k8sdemo-0 bash 进入 k8sdemo-0 这个 Pod,通过域名 Pod 名称.serviceName 来访问,如下:

1
2
3
curl k8sdemo-0.k8sdemo-service/weatherforecast
curl k8sdemo-1.k8sdemo-service/weatherforecast
curl k8sdemo-2.k8sdemo-service/weatherforecast

在 NFS 挂载目录中查看接口访问日志,以下是 Pod k8sdemo-1 中的日志:

1
2
3
4
5
6
7
2020-09-20 06:01:17,451 [17] INFO  [k8sdemo-1] - Request starting HTTP/1.1 GET http://k8sdemo-1.k8sdemo-service/weatherforecast  
2020-09-20 06:01:17,455 [17] INFO [k8sdemo-1] - Executing endpoint 'T.K8SDemo.Controllers.WeatherForecastController.Get (T.K8SDemo)'
2020-09-20 06:01:17,458 [17] INFO [k8sdemo-1] - Route matched with {action = "Get", controller = "WeatherForecast"}. Executing controller action with signature System.Collections.Generic.IEnumerable`1[T.K8SDemo.WeatherForecast] Get() on controller T.K8SDemo.Controllers.WeatherForecastController (T.K8SDemo).
2020-09-20 06:01:17,459 [17] INFO [k8sdemo-1] - Executing ObjectResult, writing value of type 'T.K8SDemo.WeatherForecast[]'.
2020-09-20 06:01:17,460 [17] INFO [k8sdemo-1] - Executed action T.K8SDemo.Controllers.WeatherForecastController.Get (T.K8SDemo) in 2.3627ms
2020-09-20 06:01:17,460 [17] INFO [k8sdemo-1] - Executed endpoint 'T.K8SDemo.Controllers.WeatherForecastController.Get (T.K8SDemo)'
2020-09-20 06:01:17,461 [17] INFO [k8sdemo-1] - Request finished in 9.9194ms 200 application/json; charset=utf-8

执行 kubectl delete pod k8sdemo-1 删除 Pod k8sdemo-1,等待一会 k8sdemo-1 会自动恢复,然后重新访问 curl k8sdemo-1.k8sdemo-service/weatherforecast,日志依然向原来的文件内追加,也说明保留了原来的状态。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
2020-09-20 06:01:17,451 [17] INFO  [k8sdemo-1] - Request starting HTTP/1.1 GET http://k8sdemo-1.k8sdemo-service/weatherforecast  
2020-09-20 06:01:17,455 [17] INFO [k8sdemo-1] - Executing endpoint 'T.K8SDemo.Controllers.WeatherForecastController.Get (T.K8SDemo)'
2020-09-20 06:01:17,458 [17] INFO [k8sdemo-1] - Route matched with {action = "Get", controller = "WeatherForecast"}. Executing controller action with signature System.Collections.Generic.IEnumerable`1[T.K8SDemo.WeatherForecast] Get() on controller T.K8SDemo.Controllers.WeatherForecastController (T.K8SDemo).
2020-09-20 06:01:17,459 [17] INFO [k8sdemo-1] - Executing ObjectResult, writing value of type 'T.K8SDemo.WeatherForecast[]'.
2020-09-20 06:01:17,460 [17] INFO [k8sdemo-1] - Executed action T.K8SDemo.Controllers.WeatherForecastController.Get (T.K8SDemo) in 2.3627ms
2020-09-20 06:01:17,460 [17] INFO [k8sdemo-1] - Executed endpoint 'T.K8SDemo.Controllers.WeatherForecastController.Get (T.K8SDemo)'
2020-09-20 06:01:17,461 [17] INFO [k8sdemo-1] - Request finished in 9.9194ms 200 application/json; charset=utf-8
2020-09-20 06:17:06,467 [12] INFO [k8sdemo-1] - Request starting HTTP/1.1 GET http://k8sdemo-1.k8sdemo-service/weatherforecast
2020-09-20 06:17:06,494 [12] INFO [k8sdemo-1] - Executing endpoint 'T.K8SDemo.Controllers.WeatherForecastController.Get (T.K8SDemo)'
2020-09-20 06:17:06,527 [12] INFO [k8sdemo-1] - Route matched with {action = "Get", controller = "WeatherForecast"}. Executing controller action with signature System.Collections.Generic.IEnumerable`1[T.K8SDemo.WeatherForecast] Get() on controller T.K8SDemo.Controllers.WeatherForecastController (T.K8SDemo).
2020-09-20 06:17:06,533 [12] INFO [k8sdemo-1] - Executing ObjectResult, writing value of type 'T.K8SDemo.WeatherForecast[]'.
2020-09-20 06:17:06,548 [12] INFO [k8sdemo-1] - Executed action T.K8SDemo.Controllers.WeatherForecastController.Get (T.K8SDemo) in 17.1904ms
2020-09-20 06:17:06,549 [12] INFO [k8sdemo-1] - Executed endpoint 'T.K8SDemo.Controllers.WeatherForecastController.Get (T.K8SDemo)'
2020-09-20 06:17:06,550 [12] INFO [k8sdemo-1] - Request finished in 84.3414ms 200 application/json; charset=utf-8

另外对 Pod 副本进行伸缩时效果也是一样的,都会保持 Pod 具有的状态。当然文中的例子和一些组件的集群部署不太一样,比如像 MySQL 这类组件,各实例间还会做数据同步来实现数据的一致性,当然最终也是每个实例关联自己的数据存储卷。

如果对你有帮助就好