# Aggregated APIServer(AA)
## Kubernetes API 聚合层.
使用聚合层（Aggregation Layer），用户可以通过附加的 API 扩展 Kubernetes， 而不局限于 Kubernetes 核心 API 提供的功能。 这里的附加 API 可以是现成的解决方案，比如 [metrics server](https://github.com/kubernetes-sigs/metrics-server)， 或者你自己开发的 API。
聚合层在 kube-apiserver 进程内运行。在扩展资源注册之前，聚合层不做任何事情。 要注册 API，你可以添加一个 APIService 对象，用它来 “申领” Kubernetes API 中的 URL 路径。 自此以后，聚合层将把发给该 API 路径的所有内容（例如 /apis/myextension.mycompany.io/v1/…） 转发到已注册的 APIService。
APIService 的最常见实现方式是在集群中某 Pod 内运行 **扩展 API 服务器**。
## K8s API三种类型
kube-apiserver 其实包含三种 APIServer：
- AggregatorServer：负责处理 apiregistration.k8s.io 组下的 APIService 资源请求，同时将来自用户的请求拦截转发给 Aggregated APIServer(AA)；
- KubeAPIServer：负责对请求的一些通用处理，包括：认证、鉴权以及各个内建资源(pod, deployment，service)的 REST 服务等；
- ApiExtensionsServer：负责 CustomResourceDefinition（CRD）apiResources 以及 apiVersions 的注册，同时处理 CRD 以及相应 CustomResource（CR）的REST请求(如果对应 CR 不能被处理的话则会返回404)，也是 apiserver Delegation 的最后一环；
## 选择 CRDs 还是 Aggregated APIServer？
除了聚合 API，官方还提供了另一种方式以实现对标准 kubernetes API 接口的扩展：CRD（Custom Resource Definition ），能达到与聚合 API 基本一样的功能，而且更加易用，开发成本更小，但相较而言聚合 API 则更为灵活。针对这两种扩展方式如何选择，官方也提供了相应的参考。
通常，如果存在以下情况，CRD 可能更合适：
- 定制资源的字段不多；
- 你在组织内部使用该资源或者在一个小规模的开源项目中使用该资源，而不是在商业产品中使用；
- 聚合 API 可提供更多的高级 API 特性，也可对其他特性进行定制；例如，对存储层进行定制、对 protobuf 协议支持、对 logs、patch 等操作支持。
两种方式的核心区别是定义 api-resource 的方式不同。在 Aggregated APIServer 方式中，api-resource 是通过代码向 API 注册资源类型，而 Custom Resource 是直接通过 yaml 文件向 API 注册资源类型。
CRD 是让 kube-apiserver 认识更多的对象类别（Kind），Aggregated APIServer 是构建自己的 APIServer 服务。虽然 CRD 更简单，但是缺少更多的灵活性，更详细的 CRDs 与 Aggregated API 的对比可参考[官方文档](https://kubernetes.io/zh/docs/concepts/extend-kubernetes/api-extension/custom-resources/#compare-ease-of-use)。
## Metrics Server
Metrics-server 是 K8s监控体系中的核心组件之一,它负责从kubelet 收集资源指标, 然后对这些指标监控数据进行聚合(依赖kube-aggregator), 并在K8s Apiserver中通过Metrics API(/apis/metrics.k8s.io/)公开暴露它们, 但是Metrics-server 只存储最新的指标数据.
**kubectl具体执行过程**
GET https://kubemaster.cluster:6443/apis/metrics.k8s.io/v1beta1/nodes/k8smaster01-application.ali
```shell
[root@k8smaster01-application ~]# kubectl top node k8smaster01-application.ali -v 9
I0807 15:42:25.647138 13405 loader.go:359] Config loaded from file /root/.kube/config
I0807 15:42:25.648877 13405 round_trippers.go:419] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.14.7 (linux/amd64) kubernetes/8fca2ec" 'https://kubemaster.cluster:6443/api?timeout=32s'
I0807 15:42:25.692029 13405 round_trippers.go:438] GET https://kubemaster.cluster:6443/api?timeout=32s 200 OK in 43 milliseconds
I0807 15:42:25.692053 13405 round_trippers.go:444] Response Headers:
I0807 15:42:25.692060 13405 round_trippers.go:447] Content-Length: 136
I0807 15:42:25.692064 13405 round_trippers.go:447] Date: Sun, 07 Aug 2022 07:42:25 GMT
I0807 15:42:25.692069 13405 round_trippers.go:447] Audit-Id: 9d4ed495-fb82-4cc5-bdc8-fff4e5a0eb6b
I0807 15:42:25.692073 13405 round_trippers.go:447] Content-Type: application/json
I0807 15:42:25.692104 13405 request.go:942] Response Body: {"kind":"APIVersions","versions":["v1"],"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"172.26.25.137:6443"}]}
I0807 15:42:25.692363 13405 round_trippers.go:419] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.14.7 (linux/amd64) kubernetes/8fca2ec" 'https://kubemaster.cluster:6443/apis?timeout=32s'
I0807 15:42:25.714723 13405 round_trippers.go:438] GET https://kubemaster.cluster:6443/apis?timeout=32s 200 OK in 22 milliseconds
I0807 15:42:25.714745 13405 round_trippers.go:444] Response Headers:
I0807 15:42:25.714751 13405 round_trippers.go:447] Audit-Id: a7dd4e83-e5d0-421b-b5b5-0f4de6fe3909
I0807 15:42:25.714756 13405 round_trippers.go:447] Content-Type: application/json
I0807 15:42:25.714761 13405 round_trippers.go:447] Date: Sun, 07 Aug 2022 07:42:25 GMT
I0807 15:42:25.714845 13405 request.go:942] Response Body: {"kind":"APIGroupList","apiVersion":"v1","groups":[{"name":"apiregistration.k8s.io","versions":[{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"},{"groupVersion":"apiregistration.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"}},{"name":"extensions","versions":[{"groupVersion":"extensions/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"extensions/v1beta1","version":"v1beta1"}},{"name":"apps","versions":[{"groupVersion":"apps/v1","version":"v1"},{"groupVersion":"apps/v1beta2","version":"v1beta2"},{"groupVersion":"apps/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"apps/v1","version":"v1"}},{"name":"events.k8s.io","versions":[{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"events.k8s.io/v1beta1","version":"v1beta1"}},{"name":"authentication.k8s.io","versions":[{"groupVersion":"authentication.k8s.io/v1","version":"v1"},{"groupVersion":"authentication.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"authentication.k8s.io/v1","version":"v1"}},{"name":"authorization.k8s.io","versions":[{"groupVersion":"authorization.k8s.io/v1","version":"v1"},{"groupVersion":"authorization.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"authorization.k8s.io/v1","version":"v1"}},{"name":"autoscaling","versions":[{"groupVersion":"autoscaling/v1","version":"v1"},{"groupVersion":"autoscaling/v2beta1","version":"v2beta1"},{"groupVersion":"autoscaling/v2beta2","version":"v2beta2"}],"preferredVersion":{"groupVersion":"autoscaling/v1","version":"v1"}},{"name":"batch","versions":[{"groupVersion":"batch/v1","version":"v1"},{"groupVersion":"batch/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"batch/v1","version":"v1"}},{"name":"certificates.k8s.io","versions":[{"groupVersion":"certificates.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"certificates.k8s.io/v1beta1","version":"v1beta1"}},{"name":"networking.k8s.io","versions":[{"groupVersion":"networking.k8s.io/v1","version":"v1"},{"groupVersion":"networking.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"networking.k8s.io/v1","version":"v1"}},{"name":"policy","versions":[{"groupVersion":"policy/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"policy/v1beta1","version":"v1beta1"}},{"name":"rbac.authorization.k8s.io","versions":[{"groupVersion":"rbac.authorization.k8s.io/v1","version":"v1"},{"groupVersion":"rbac.authorization.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"rbac.authorization.k8s.io/v1","version":"v1"}},{"name":"storage.k8s.io","versions":[{"groupVersion":"storage.k8s.io/v1","version":"v1"},{"groupVersion":"storage.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"storage.k8s.io/v1","version":"v1"}},{"name":"admissionregistration.k8s.io","versions":[{"groupVersion":"admissionregistration.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"admissionregistration.k8s.io/v1beta1","version":"v1beta1"}},{"name":"apiextensions.k8s.io","versions":[{"groupVersion":"apiextensions.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"apiextensions.k8s.io/v1beta1","version":"v1beta1"}},{"name":"scheduling.k8s.io","versions":[{"groupVersion":"scheduling.k8s.io/v1","version":"v1"},{"groupVersion":"scheduling.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"scheduling.k8s.io/v1","version":"v1"}},{"name":"coordination.k8s.io","versions":[{"groupVersion":"coordination.k8s.io/v1","version":"v1"},{"groupVersion":"coordination.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"coordination.k8s.io/v1","version":"v1"}},{"name":"node.k8s.io","versions":[{"groupVersion":"node.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"node.k8s.io/v1beta1","version":"v1beta1"}},{"name":"argoproj.io","versions":[{"groupVersion":"argoproj.io/v1alpha1","version":"v1alpha1"}],"preferredVersion":{"groupVersion":"argoproj.io/v1alpha1","version":"v1alpha1"}},{"name":"authentication.istio.io","versions":[{"groupVersion":"authentication.istio.io/v1alpha1","version":"v1alpha1"}],"preferredVersion":{"groupVersion":"authentication.istio.io/v1alpha1","version":"v1alpha1"}},{"name":"rbac.istio.io","versions":[{"groupVersion":"rbac.istio.io/v1alpha1","version":"v1alpha1"}],"preferredVersion":{"groupVersion":"rbac.istio.io/v1alpha1","version":"v1alpha1"}},{"name":"traefik.containo.us","versions":[{"groupVersion":"traefik.containo.us/v1alpha1","version":"v1alpha1"}],"preferredVersion":{"groupVersion":"traefik.containo.us/v1alpha1","version":"v1alpha1"}},{"name":"config.istio.io","versions":[{"groupVersion":"config.istio.io/v1alpha2","version":"v1alpha2"}],"preferredVersion":{"groupVersion":"config.istio.io/v1alpha2","version":"v1alpha2"}},{"name":"networking.istio.io","versions":[{"groupVersion":"networking.istio.io/v1alpha3","version":"v1alpha3"}],"preferredVersion":{"groupVersion":"networking.istio.io/v1alpha3","version":"v1alpha3"}},{"name":"security.istio.io","versions":[{"groupVersion":"security.istio.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"security.istio.io/v1beta1","version":"v1beta1"}},{"name":"metrics.k8s.io","versions":[{"groupVersion":"metrics.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"metrics.k8s.io/v1beta1","version":"v1beta1"}}]}
I0807 15:42:25.715357 13405 round_trippers.go:419] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.14.7 (linux/amd64) kubernetes/8fca2ec" 'https://kubemaster.cluster:6443/apis/metrics.k8s.io/v1beta1/nodes/k8smaster01-application.ali'
I0807 15:42:25.734775 13405 round_trippers.go:438] GET https://kubemaster.cluster:6443/apis/metrics.k8s.io/v1beta1/nodes/k8smaster01-application.ali 200 OK in 19 milliseconds
I0807 15:42:25.734797 13405 round_trippers.go:444] Response Headers:
I0807 15:42:25.734802 13405 round_trippers.go:447] Audit-Id: 2a891333-1141-415b-995e-e2c5d02fe2fe
I0807 15:42:25.734807 13405 round_trippers.go:447] Content-Type: application/json
I0807 15:42:25.734811 13405 round_trippers.go:447] Date: Sun, 07 Aug 2022 07:42:25 GMT
I0807 15:42:25.734815 13405 round_trippers.go:447] Content-Length: 331
I0807 15:42:25.734841 13405 request.go:942] Response Body: {"kind":"NodeMetrics","apiVersion":"metrics.k8s.io/v1beta1","metadata":{"name":"k8smaster01-application.ali","selfLink":"/apis/metrics.k8s.io/v1beta1/nodes/k8smaster01-application.ali","creationTimestamp":"2022-08-07T07:42:25Z"},"timestamp":"2022-08-07T07:41:33Z","window":"30s","usage":{"cpu":"1102333605n","memory":"9202624Ki"}}
I0807 15:42:25.748983 13405 round_trippers.go:419] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.14.7 (linux/amd64) kubernetes/8fca2ec" 'https://kubemaster.cluster:6443/api/v1/nodes/k8smaster01-application.ali'
I0807 15:42:25.997639 13405 round_trippers.go:438] GET https://kubemaster.cluster:6443/api/v1/nodes/k8smaster01-application.ali 200 OK in 248 milliseconds
I0807 15:42:25.997662 13405 round_trippers.go:444] Response Headers:
I0807 15:42:25.997667 13405 round_trippers.go:447] Audit-Id: 3af02c5e-69a6-4816-9359-dfabce9d6f9a
I0807 15:42:25.997672 13405 round_trippers.go:447] Content-Type: application/json
I0807 15:42:25.997677 13405 round_trippers.go:447] Date: Sun, 07 Aug 2022 07:42:25 GMT
I0807 15:42:25.997759 13405 request.go:942] Response Body: {"kind":"Node","apiVersion":"v1","metadata":{"name":"k8smaster01-application.ali","selfLink":"/api/v1/nodes/k8smaster01-application.ali","uid":"2fa8762d-013c-11ea-81ab-00163e06bb08","resourceVersion":"1164983866","creationTimestamp":"2019-11-07T08:54:19Z","labels":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","k8s_cluster":"aliyun-bj-online-01","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"k8smaster01-application.ali","kubernetes.io/os":"linux","node-role.kubernetes.io/master":""},"annotations":{"flannel.alpha.coreos.com/backend-data":"{\"VtepMAC\":\"22:c3:2f:2a:d8:bf\"}","flannel.alpha.coreos.com/backend-type":"vxlan","flannel.alpha.coreos.com/kube-subnet-manager":"true","flannel.alpha.coreos.com/public-ip":"172.26.25.137","kubeadm.alpha.kubernetes.io/cri-socket":"/var/run/dockershim.sock","node.alpha.kubernetes.io/ttl":"15","volumes.kubernetes.io/controller-managed-attach-detach":"true"}},"spec":{"podCIDR":"10.64.0.0/24","taints":[{"key":"node-role.kubernetes.io/master","effect":"NoSchedule"}]},"status":{"capacity":{"cpu":"4","ephemeral-storage":"103080204Ki","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"16155100Ki","pods":"110"},"allocatable":{"cpu":"4","ephemeral-storage":"94998715850","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"16052700Ki","pods":"110"},"conditions":[{"type":"MemoryPressure","status":"False","lastHeartbeatTime":"2022-08-07T07:41:38Z","lastTransitionTime":"2019-11-07T08:54:17Z","reason":"KubeletHasSufficientMemory","message":"kubelet has sufficient memory available"},{"type":"DiskPressure","status":"False","lastHeartbeatTime":"2022-08-07T07:41:38Z","lastTransitionTime":"2020-08-13T03:40:38Z","reason":"KubeletHasNoDiskPressure","message":"kubelet has no disk pressure"},{"type":"PIDPressure","status":"False","lastHeartbeatTime":"2022-08-07T07:41:38Z","lastTransitionTime":"2019-11-07T08:54:17Z","reason":"KubeletHasSufficientPID","message":"kubelet has sufficient PID available"},{"type":"Ready","status":"True","lastHeartbeatTime":"2022-08-07T07:41:38Z","lastTransitionTime":"2021-06-09T08:24:53Z","reason":"KubeletReady","message":"kubelet is posting ready status"}],"addresses":[{"type":"InternalIP","address":"172.26.25.137"},{"type":"Hostname","address":"k8smaster01-application.ali"}],"daemonEndpoints":{"kubeletEndpoint":{"Port":10250}},"nodeInfo":{"machineID":"f0f31005fb5a436d88e3c6cbf54e25aa","systemUUID":"10CB0DA3-8BCC-4245-9CF3-9062B4D3BEC8","bootID":"6621ee1d-534e-4c53-a44f-f32248abdf56","kernelVersion":"4.14.1-1.el7.elrepo.x86_64","osImage":"CentOS Linux 7 (Core)","containerRuntimeVersion":"docker://19.3.4","kubeletVersion":"v1.14.7","kubeProxyVersion":"v1.14.7","operatingSystem":"linux","architecture":"amd64"},"images":[{"names":["harbor.hualala.com/ci/baseservice/shop-base-service@sha256:9c7b5bcacfd76f89a7ad418e7cd25fe5ca50c10cc43d7ba5d227c2b190e6293f","registry2.hualala.com/shop-base-service@sha256:9c7b5bcacfd76f89a7ad418e7cd25fe5ca50c10cc43d7ba5d227c2b190e6293f","harbor.hualala.com/ci/baseservice/shop-base-service:bb5b969.235","harbor.hualala.com/ci/baseservice/shop-base-service:bb5b969.235sh","registry2.hualala.com/shop-base-service:bb5b969.235"],"sizeBytes":1085599691},{"names":["harbor.hualala.com/base/node@sha256:e56cbff0a409c97873fd0249adefa3bec7161e4bf486f01ff058b4734ab9b199","harbor.hualala.com/base/node:10.20.1"],"sizeBytes":911604404},{"names":["harbor.hualala.com/ci/baseservice/shop-base-service@sha256:591be1fe00f0b01b5a6aca845f46727071dee55f7b9ad72c0af3629a7dd718cc"],"sizeBytes":551545966},{"names":["harbor.hualala.com/base/node@sha256:06e9fd7a363e2f7d9253b91831d2d4f056429ab51551be1ba19ad78858e36907","harbor.hualala.com/base/node:16.13.2"],"sizeBytes":547886955},{"names":["harbor.hualala.com/base/node@sha256:0fa09c4055e707878f8459c1976cb056f2369462d317d2710abc0d64c91f9001","harbor.hualala.com/base/node:8.15.1"],"sizeBytes":514263667},{"names":["harbor.hualala.com/ci/filebeat/filebeat@sha256:a20270d0b4f07b083758f67bb39c775289fbb8f88c57257137f6e4216e0a7e12","harbor.hualala.com/ci/filebeat/filebeat:7.11.2"],"sizeBytes":464611372},{"names":["registry.cn-hangzhou.aliyuncs.com/google_containers/etcd@sha256:7872edc2929aa009ddcfeae6ff9a55b779890f46f5720fb3f01d3559c659db1b","registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.2"],"sizeBytes":281880100},{"names":["registry.aliyuncs.com/google_containers/kube-apiserver@sha256:a098084d3e65fe42acf886ae2a3c46a91cf60f74ea0d934b3cb9989b725faf3a","registry.aliyuncs.com/google_containers/kube-apiserver:v1.14.7"],"sizeBytes":209478462},{"names":["registry.aliyuncs.com/google_containers/kube-controller-manager@sha256:38c38df63ba9e765709a5d8737f59ac37ea56a311e7cd941c6215ce0667401f9","registry.aliyuncs.com/google_containers/kube-controller-manager:v1.14.7"],"sizeBytes":157503518},{"names":["harbor.hualala.com/ci/ocean/crab@sha256:565892d866cb737edef80c5abf984ba932241c23afe02013302a00fc757bbfc7","harbor.hualala.com/ci/ocean/crab:f1229da.66"],"sizeBytes":124654529},{"names":["registry.aliyuncs.com/google_containers/kube-proxy@sha256:6e09cc1d370b296cf19771f3112794cbf4bd59188f7cb4d37b3983d657b3bd2f","registry.aliyuncs.com/google_containers/kube-proxy:v1.14.7"],"sizeBytes":82106236},{"names":["registry.aliyuncs.com/google_containers/kube-scheduler@sha256:2e7dd61ef77805ea7af69697ca1a7727b1d5803c02e83f3463edb724029e120a","registry.aliyuncs.com/google_containers/kube-scheduler:v1.14.7"],"sizeBytes":81579742},{"names":["registry.cn-shanghai.aliyuncs.com/gcr-k8s/flannel@sha256:25e23320b5965ec8d5063ecf9f5a154372f6c230334dd11d76a0290184e789be","registry.cn-shanghai.aliyuncs.com/gcr-k8s/flannel:v0.10.0-amd64"],"sizeBytes":44598861},{"names":["registry.cn-beijing.aliyuncs.com/bj-aliyun/check@sha256:44ae2c9bc689e62d27c61e3484aa19731d993dc37ff4fb9a672c9d74632eb79a","registry.cn-beijing.aliyuncs.com/bj-aliyun/check:v1.0"],"sizeBytes":17997252},{"names":["registry2.hualala.com/apm@sha256:a0c2357e3aed55be6195187756ca5900ca570f0675b027bb198efd1604435bae","registry2.hualala.com/apm:052101"],"sizeBytes":14270047},{"names":["registry.aliyuncs.com/google_containers/pause@sha256:759c3f0f6493093a9043cc813092290af69029699ade0e3dbe024e968fcb7cca","registry.aliyuncs.com/google_containers/pause:3.1"],"sizeBytes":742472}]}}
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
k8smaster01-application.ali 1103m 27% 8986Mi 57%
[root@k8smaster01-application ~]#
```
K8s API扩展-Aggregated APIServer
Copyright: 采用 知识共享署名4.0 国际许可协议进行许可
Links: https://blog.zs-fighting.cn/archives/k8sapi扩展-aggregatedapiserver
