This article is this in-progress and more details to be added
Step 1 - Install Prometheus, Grafana, Alert Manager with prometheus-operator, which packaged the installation and provide CRDs to manipulate these resources in Kubernetes (i.e. kubectl get Prometheus
)
Step 2 - Configure Dashboard, presenting metrics with PromQL
Step 3 - Setup AlertManager
Others:
Setup Thanos for persistent S3 backend for Prometheus
Setup Loki from Grafana for logging
References:
https://training.promlabs.com/training/prometheus-and-kubernetes/ (paid)