Kubernetes:Prometheus Operator手动安装

Posted by zhangshun on June 18, 2020

一、prometheus-operator 介绍和功能

prometheus-operator 介绍


为了在Kubernetes能够方便的管理和部署Prometheus,我们使用ConfigMap了管理Prometheus配置文件。每次对Prometheus配置文件进行升级时,我们需要手动移除已经运行的Pod实例,从而让Kubernetes可以使用最新的配置文件创建Prometheus。 而如果当应用实例的数量更多时,通过手动的方式部署和升级Prometheus过程繁琐并且效率低下。

从本质上来讲Prometheus属于是典型的有状态应用,而其有包含了一些自身特有的运维管理和配置管理方式。而这些都无法通过Kubernetes原生提供的应用管理概念实现自动化。为了简化这类应用程序的管理复杂度,CoreOS率先引入了Operator的概念,并且首先推出了针对在Kubernetes下运行和管理Etcd的Etcd Operator。并随后推出了Prometheus Operator。

prometheus-operator的工作原理


在Kubernetes中我们使用Deployment、DamenSet,StatefulSet来管理应用Workload,使用Service,Ingress来管理应用的访问方式,使用ConfigMap和Secret来管理应用配置。我们在集群中对这些资源的创建,更新,删除的动作都会被转换为事件(Event),Kubernetes的Controller Manager负责监听这些事件并触发相应的任务来满足用户的期望。这种方式我们成为声明式,用户只需要关心应用程序的最终状态,其它的都通过Kubernetes来帮助我们完成,通过这种方式可以大大简化应用的配置管理复杂度。

而除了这些原生的Resource资源以外,Kubernetes还允许用户添加自己的自定义资源(Custom Resource)。并且通过实现自定义Controller来实现对Kubernetes的扩展。

如下所示,是Prometheus Operator的架构示意图: prometheus-operator架构图

Prometheus的本职就是一组用户自定义的CRD资源以及Controller的实现,Prometheus Operator负责监听这些自定义资源的变化,并且根据这些资源的定义自动化的完成如Prometheus Server自身以及配置的自动化管理工作。

prometheus-operator能做什么


要了解Prometheus Operator能做什么,其实就是要了解Prometheus Operator为我们提供了哪些自定义的Kubernetes资源,列出了Prometheus Operator目前提供的️4类资源:

  • Prometheus:声明式创建和管理Prometheus Server实例;
  • ServiceMonitor:负责声明式的管理监控配置;
  • PrometheusRule:负责声明式的管理告警配置;
  • Alertmanager:声明式的创建和管理Alertmanager实例。

简言之,Prometheus Operator能够帮助用户自动化的创建以及管理Prometheus Server以及其相应的配置。

二、下载 prometheus-operator 配置

下载官方kube-prometheus代码,官方把所有文件都放在一起,这里我分类下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
git clone https://github.com/prometheus-operator/kube-prometheus.git
cd manifests/

# 新建对应的服务目录
$ mkdir -p node-exporter alertmanager grafana kube-state-metrics prometheus serviceMonitor adapter thanos

# 把对应的服务配置文件移动到相应的服务目录
$ mv *-serviceMonitor* serviceMonitor/
$ mv grafana-* grafana/
$ mv kube-state-metrics-* kube-state-metrics/
$ mv alertmanager-* alertmanager/
$ mv node-exporter-* node-exporter/
$ mv prometheus-adapter* adapter/
$ mv prometheus-* prometheus/

# 新创建了两个目录,存放钉钉配置和其它配置
mkdir other dingtalk-hook

三、部署operator

首先创建CRD等基础资源


kubectl apply -f setup/.

查看是否正常部署


1
2
3
4
$ kubectl -n monitoring get pod

NAME                                   READY     STATUS    RESTARTS   AGE
prometheus-operator-56954c76b5-qm9ww   1/1       Running   0          24s

查看是否正常部署自定义资源定义(CRD)


1
2
3
4
5
6
7
$ kubectl get crd -n monitoring

NAME                                    CREATED AT
alertmanagers.monitoring.coreos.com     2019-04-16T06:22:20Z
prometheuses.monitoring.coreos.com      2019-04-16T06:22:20Z
prometheusrules.monitoring.coreos.com   2019-04-16T06:22:20Z
servicemonitors.monitoring.coreos.com   2019-04-16T06:22:21Z

四、部署整套CRD

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
kubectl apply -f adapter/
kubectl apply -f alertmanager/
kubectl apply -f node-exporter/
kubectl apply -f kube-state-metrics/
kubectl apply -f grafana/
kubectl apply -f prometheus/
kubectl apply -f serviceMonitor/
 
 
# 查看是否正常部署
kubectl -n monitoring get all -o wide

# 修改svc(grafana、prometheus-k8s)类型为NodePort
kubectl -n monitoring edit svc grafana
kubectl -n monitoring edit svc prometheus-k8s

五、完善相关组件

修改grafana dashboard的时区

vim grafana/grafana-dashboardDefinitions.yaml

默认dashboard使用的是UTC时间,将每个dashboard中的timezone改为空
示例: "timezone": ""

增加coredns、controller-manager、scheduler的监控

部署完会发现在prometheus server的页面上发现kube-controller和kube-schedule、coredns的target为0/0。需要手动创建资源

  • 增加coredns

    vim orther/core-svc-ep.yaml.yaml
    
    apiVersion: v1
    kind: Service
    metadata:
      labels:
        k8s-app: kube-dns
      name: prometheus-coredns
      namespace: kube-system
    spec:
      ports:
      - name: metrics
        port: 9153
        protocol: TCP
        targetPort: 9153
      selector:
        k8s-app: kube-dns
      sessionAffinity: None
      type: ClusterIP
    
  • 增加controller-manager

    vim other/kube-controller-manager-svc-ep.yaml
    
    apiVersion: v1
    kind: Service
    metadata:
      namespace: kube-system
      name: prometheus-kube-controller-manager
      labels:
        k8s-app: kube-controller-manager
    spec:
      type: ClusterIP
      clusterIP: None
      ports:
      - name: http-metrics
        port: 10252
        targetPort: 10252
        protocol: TCP
    ---
    apiVersion: v1
    kind: Endpoints
    metadata:
      labels:
        k8s-app: kube-controller-manager
      name: kube-controller-manager
      namespace: kube-system
    subsets:
    - addresses:
      - ip: 172.16.3.9   # master ip 地址
      ports:
      - name: http-metrics
        port: 10252
        protocol: TCP
    
  • 增加schedule

    vim other/kube-scheduler-svc-ep.yaml
    
    apiVersion: v1
    kind: Service
    metadata:
      namespace: kube-system
      name: kube-scheduler
      labels:
        k8s-app: kube-scheduler
    spec:
      type: ClusterIP
      clusterIP: None
      ports:
      - name: http-metrics
        port: 10251
        targetPort: 10251
        protocol: TCP
    ---
    apiVersion: v1
    kind: Endpoints
    metadata:
      labels:
        k8s-app: kube-scheduler
      name: kube-scheduler
      namespace: kube-system
    subsets:
    - addresses:
      - ip: 172.16.3.9 # master ip 地址
      ports:
      - name: http-metrics
        port: 10251
        protocol: TCP
    

自定义Prometheus配置文件(自动发现)

自带的prometheus.yaml是只读的,没法修改 ,我们可以添加配置到prometheus CRD资源中

1.创建prometheus配置文件

vim other/prometheus-additional.yaml

- job_name: 'kubernetes-service-endpoints'
  kubernetes_sd_configs:
  - role: endpoints
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
    action: replace
    target_label: __scheme__
    regex: (https?)
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
    action: replace
    target_label: __address__
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
  - action: labelmap
    regex: __meta_kubernetes_service_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_service_name]
    action: replace
    target_label: kubernetes_name
   

自动发现并监控具有prometheus.io/scrape=true这个 annotations 的 Service

2.通过这个文件创建一个对应的 Secret 对象

kubectl create secret generic additional-configs --from-file=prometheus-additional.yaml -n monitoring

3.需要在声明 prometheus 的资源对象文件中添加上这个额外的配置

vim prometheus/prometheus-prometheus.yaml

   apiVersion: monitoring.coreos.com/v1
   kind: Prometheus
   metadata:
     labels:
       prometheus: k8s
     name: k8s
     namespace: monitoring
   spec:
     alerting:
       alertmanagers:
       - name: alertmanager-main
         namespace: monitoring
         port: web
     image: quay.io/prometheus/prometheus:v2.20.0
     nodeSelector:
       kubernetes.io/os: linux
     podMonitorNamespaceSelector: {}
     podMonitorSelector: {}
     probeNamespaceSelector: {}
     probeSelector: {}
     replicas: 2
     resources:
       requests:
         memory: 400Mi
     ruleSelector:
       matchLabels:
         prometheus: k8s
         role: alert-rules
     securityContext:
       fsGroup: 2000
       runAsNonRoot: true
       runAsUser: 1000
     additionalScrapeConfigs:	# 添加配置文件
       name: additional-configs
       key: prometheus-additional.yaml
     serviceAccountName: prometheus-k8s
     serviceMonitorNamespaceSelector: {}
     serviceMonitorSelector: {}
     version: v2.20.0
     thanos:
       baseImage: quay.io/thanos/thanos
       version: v0.8.1
       objectStorageConfig:
         key: thanos.yaml
         name: thanos-objstore-config
   

最后在prometheus configure界面可以看到新增配置

在 Prometheus Dashboard 的配置页面下面我们可以看到已经有了对应的的配置信息了,但是我们切换到 targets 页面下面却并没有发现对应的监控任务,查看 Prometheus 的 Pod 日志:

1
2
3
4
5
$ kubectl logs -f prometheus-k8s-0 prometheus -n monitoring
level=error ts=2018-12-20T15:14:06.772903214Z caller=main.go:240 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:302: Failed to list *v1.Pod: pods is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list pods at the cluster scope"
level=error ts=2018-12-20T15:14:06.773096875Z caller=main.go:240 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:301: Failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list services at the cluster scope"
level=error ts=2018-12-20T15:14:06.773212629Z caller=main.go:240 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:300: Failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list endpoints at the cluster scope"
......

可以看到有很多错误日志出现,都是xxx is forbidden,这说明是 RBAC 权限的问题,通过 prometheus 资源对象的配置可以知道 Prometheus 绑定了一个名为 prometheus-k8s 的 ServiceAccount 对象,而这个对象绑定的是一个名为 prometheus-k8s 的 ClusterRole:(prometheus-clusterRole.yaml)

vim prometheus/prometheus-clusterRole.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-k8s
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - services
  - endpoints
  - pods
  - nodes/proxy
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - configmaps
  - nodes/metrics
  verbs:
  - get
- nonResourceURLs:
  - /metrics
  verbs:
  - get

数据持久化

通过 prometheus 这个 CRD 创建的 Prometheus 并没有做数据的持久化, 可以看到 Prometheus 的数据目录 /prometheus 实际上是通过 emptyDir 进行挂载的,当prometheus重启后抓取的指标会清空

这里采用的腾讯云CFS文件存储

vim prometheus/prometheus-prometheus.yaml

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    prometheus: k8s
  name: k8s
  namespace: monitoring
spec:
  alerting:
    alertmanagers:
    - name: alertmanager-main
      namespace: monitoring
      port: web
  image: quay.io/prometheus/prometheus:v2.20.0
  nodeSelector:
    kubernetes.io/os: linux
  podMonitorNamespaceSelector: {}
  podMonitorSelector: {}
  probeNamespaceSelector: {}
  probeSelector: {}
  replicas: 2
  resources:
    requests:
      memory: 400Mi
  ruleSelector:
    matchLabels:
      prometheus: k8s
      role: alert-rules
  securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  additionalScrapeConfigs:	# 增加prometheus配置文件
    name: additional-configs
    key: prometheus-additional.yaml
  serviceAccountName: prometheus-k8s
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  version: v2.20.0
  thanos:	# 与thanos集成
    baseImage: quay.io/thanos/thanos
    version: v0.8.1
    objectStorageConfig:
      key: thanos.yaml
      name: thanos-objstore-config
  storage:	# 增加持久化配置
    volumeClaimTemplate:
      spec:
        accessModes:
          - ReadWriteMany
        storageClassName: "cfs-storageclass"
        resources:
          requests:
            storage: 10Gi

六、完整的prometheus CRD资源

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    prometheus: k8s
  name: k8s
  namespace: monitoring
spec:
  alerting:
    alertmanagers:
    - name: alertmanager-main
      namespace: monitoring
      port: web
  image: quay.io/prometheus/prometheus:v2.20.0
  nodeSelector:
    kubernetes.io/os: linux
  podMonitorNamespaceSelector: {}
  podMonitorSelector: {}
  probeNamespaceSelector: {}
  probeSelector: {}
  replicas: 2
  resources:
    requests:
      memory: 400Mi
  ruleSelector:
    matchLabels:
      prometheus: k8s
      role: alert-rules
  securityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  additionalScrapeConfigs:
    name: additional-configs
    key: prometheus-additional.yaml
  serviceAccountName: prometheus-k8s
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  version: v2.20.0
  thanos:
    baseImage: quay.io/thanos/thanos
    version: v0.8.1
    objectStorageConfig:
      key: thanos.yaml
      name: thanos-objstore-config
  storage:
    volumeClaimTemplate:
      spec:
        accessModes:
          - ReadWriteMany
        storageClassName: "cfs-storageclass"
        resources:
          requests:
            storage: 10Gi