Prometheus,Grafana 설치(manifest)

프로메테우스 설치시 필요한 것

namespace

rbac

configmap

prometheus-deployment

prometheus-service

 

그라파나 설치시 필요한 것

garafana_deployment

grafana_service

node-exporter-deamonset

---

프로메테우스

vi namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: monitoring

rometheus를 설치할 네임스페이스를 생성

rbac.yaml
  ---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/proxy
  - nodes/metrics
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups:
  - extensions
  - networking.k8s.io
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: monitoring

Prometheus가 Kubernetes API 서버와 상호작용할 수 있도록 권한을 설정

vi configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
 name: prometheus-config
 namespace: monitoring
data:
 prometheus.yml: |
   global:
     scrape_interval: 15s
     evaluation_interval: 15s
   scrape_configs:
     - job_name: 'kubernetes-apiservers'
       kubernetes_sd_configs:
       - role: endpoints
       scheme: https
       tls_config:
         ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
       bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
       relabel_configs:
       - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
         action: keep
         regex: default;kubernetes;https
     - job_name: 'kubernetes-nodes'
       scheme: https
       tls_config:
         ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
       bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
       kubernetes_sd_configs:
       - role: node
     - job_name: 'kubernetes-pods'
       kubernetes_sd_configs:
       - role: pod
       relabel_configs:
       - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
         action: keep
         regex: true
     - job_name: 'kubernetes-cadvisor'
       scheme: https
       tls_config:
         ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
       bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
       kubernetes_sd_configs:
       - role: node
       relabel_configs:
       - target_label: __metrics_path__
         replacement: /metrics/cadvisor
       - job_name: 'node-exporter'
       kubernetes_sd_configs:
       - role: endpoints
       relabel_configs:
       - source_labels: [__meta_kubernetes_pod_label_k8s_app]
         action: keep
         regex: node-exporter

Prometheus 설정을 담을 ConfigMap을 생성

애플리케이션의 설정을 ConfigMap에 담고, 이를 Kubernetes 리소스와 연동하여 애플리케이션이 설정을 읽도록 함

API 서버, 노드, 파드, cAdvisor, Node Exporter 등을 모니터링 대상으로 포함

vi prometheus-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: prometheus
  replicas: 1
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      serviceAccountName: prometheus
      containers:
      - name: prometheus
        image: prom/prometheus
        ports:
        - containerPort: 9090
          name: http
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "2Gi"
            cpu: "1000m"
        volumeMounts:
        - name: config-volume
          mountPath: /etc/prometheus/
        - name: storage-volume
          mountPath: /prometheus/
        args:
          - "--config.file=/etc/prometheus/prometheus.yml"
          - "--storage.tsdb.path=/prometheus/"
          - "--web.enable-lifecycle"
      volumes:
      - name: config-volume
        configMap:
          name: prometheus-config
      - name: storage-volume
        emptyDir: {}

Prometheus 서버를 실행하는 Deployment 리소스를 설정

Prometheus 인스턴스를 Kubernetes 클러스터 내에서 실행하기 위해 필요함

vi service.yaml
apiVersion: v1
kind: Service
metadata:
  name: prometheus
  namespace: monitoring
spec:
  selector:
    app: prometheus
  ports:
  - port: 9090
    targetPort: 9090
    protocol: TCP
  type: NodePort

Prometheus 서버에 대한 접근을 제공하는 Service를 설정

외부 또는 내부 클러스터에서 Prometheus에 접근할 수 있게 해줌

---

그라파나

vi garafana_deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      containers:
      - name: grafana
        image: grafana/grafana:latest
        ports:
        - containerPort: 3000
          name: http
        resources:
          limits:
            cpu: 200m
            memory: 512Mi
          requests:
            cpu: 100m
            memory: 256Mi
        volumeMounts:
        - mountPath: /var/lib/grafana
          name: grafana-storage
      volumes:
      - name: grafana-storage
        emptyDir: {}

Grafana를 실행하는 Deployment 리소스를 설정

대시보드를 제공하고 Prometheus와 통합하여 시각화를 제공함

vi grafana_service.yaml
apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: monitoring
spec:
  selector:
    app: grafana
  ports:
  - port: 3000
    targetPort: 3000
    protocol: TCP
  type: NodePort

Grafana에 접근할 수 있는 Service를 설정함.

이 서비스를 통해 Grafana 웹 UI에 접근할 수 있음.

vi node-exporter-deamonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: monitoring
  labels:
    k8s-app: node-exporter
spec:
  selector:
    matchLabels:
      k8s-app: node-exporter
  template:
    metadata:
      labels:
        k8s-app: node-exporter
    spec:
      hostNetwork: true
      hostPID: true
      containers:
      - name: node-exporter
        image: prom/node-exporter
        ports:
        - containerPort: 9100
          protocol: TCP
          name: http
        volumeMounts:
        - name: proc
          mountPath: /host/proc
          readOnly: true
        - name: sys
          mountPath: /host/sys
          readOnly: true
        - name: root
          mountPath: /host/root
          readOnly: true
      volumes:
      - name: proc
        hostPath:
          path: /proc
      - name: sys
        hostPath:
          path: /sys
      - name: root
        hostPath:
          path: /

노드의 메트릭을 수집하는 에이전트.

DaemonSet으로 배포하여 모든 노드에 설치하고 메트릭을 수집합니다.

daemonset은 Kubernetes에서 모든 노드에 하나씩 Pod를 배포하기 위해 사용되는 리소스임.