Prometheus rule support
This commit is contained in:
@@ -4,7 +4,7 @@ metadata:
|
|||||||
name: {{ .Release.Name }}
|
name: {{ .Release.Name }}
|
||||||
namespace: {{ .Release.Namespace }}
|
namespace: {{ .Release.Namespace }}
|
||||||
labels:
|
labels:
|
||||||
{{ include "helm-chart.labels" . | nindent 4 }}
|
{{- include "helm-chart.labels" . | nindent 4 }}
|
||||||
spec:
|
spec:
|
||||||
schedule: {{ .Values.schedule | quote }}
|
schedule: {{ .Values.schedule | quote }}
|
||||||
successfulJobsHistoryLimit: 1
|
successfulJobsHistoryLimit: 1
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
suite: test deployment
|
suite: test cron job
|
||||||
templates:
|
templates:
|
||||||
- cronJob.yaml
|
- cronJob.yaml
|
||||||
- configmap.yaml
|
- configmap.yaml
|
||||||
|
|||||||
@@ -15,4 +15,4 @@ type: application
|
|||||||
# This is the chart version. This version number should be incremented each time you make changes
|
# This is the chart version. This version number should be incremented each time you make changes
|
||||||
# to the chart and its templates, including the app version.
|
# to the chart and its templates, including the app version.
|
||||||
# Versions are expected to follow Semantic Versioning (https://semver.org/)
|
# Versions are expected to follow Semantic Versioning (https://semver.org/)
|
||||||
version: 0.5.1
|
version: 0.6.0
|
||||||
|
|||||||
@@ -0,0 +1,24 @@
|
|||||||
|
{{- if .Values.prometheusRules }}
|
||||||
|
apiVersion: monitoring.coreos.com/v1
|
||||||
|
kind: PrometheusRule
|
||||||
|
metadata:
|
||||||
|
name: {{ .Release.Name }}
|
||||||
|
namespace: {{ .Release.Namespace }}
|
||||||
|
labels:
|
||||||
|
{{- include "helm-chart.labels" . | nindent 4 }}
|
||||||
|
annotations:
|
||||||
|
spec:
|
||||||
|
groups:
|
||||||
|
- name: kubernetes-apps
|
||||||
|
rules:
|
||||||
|
{{- range $rule := .Values.prometheusRules }}
|
||||||
|
- alert: {{ $rule.name }}
|
||||||
|
annotations:
|
||||||
|
message: {{ $rule.message | quote }}
|
||||||
|
runbook_url: {{ $rule.runBookURL | quote }}
|
||||||
|
expr: {{ $rule.expression | quote }}
|
||||||
|
for: {{ $rule.for | quote }}
|
||||||
|
labels:
|
||||||
|
{{- toYaml $rule.labels | nindent 12 }}
|
||||||
|
{{- end }}
|
||||||
|
{{- end }}
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
suite: test prometheus rule
|
||||||
|
templates:
|
||||||
|
- prometheus-rule.yaml
|
||||||
|
tests:
|
||||||
|
- it: Should set rules
|
||||||
|
set:
|
||||||
|
prometheusRules:
|
||||||
|
- name: KubePodCrashLooping
|
||||||
|
message: Pod {{`{{`}} $labels.namespace {{`}}`}}/{{`{{`}} $labels.pod {{`}}`}} ({{`{{`}} $labels.container {{`}}`}}) is restarting {{`{{`}} printf "%.2f" $value {{`}}`}} times / 5 minutes.
|
||||||
|
runBookURL:
|
||||||
|
expression: rate(kube_pod_container_status_restarts_total{job="kube-state-metrics", namespace=~"{{ $targetNamespace }}"}[15m]) * 60 * 5 > 0
|
||||||
|
for: 1h
|
||||||
|
labels:
|
||||||
|
severity: criticalv
|
||||||
|
asserts:
|
||||||
|
- equal:
|
||||||
|
path: spec.groups[0].name
|
||||||
|
value: kubernetes-apps
|
||||||
|
- equal:
|
||||||
|
path: spec.groups[0].rules[0].alert
|
||||||
|
value: KubePodCrashLooping
|
||||||
|
- equal:
|
||||||
|
path: spec.groups[0].rules[0].expr
|
||||||
|
value: rate(kube_pod_container_status_restarts_total{job="kube-state-metrics", namespace=~"{{ $targetNamespace }}"}[15m]) * 60 * 5 > 0
|
||||||
@@ -22,7 +22,7 @@ image:
|
|||||||
ingress:
|
ingress:
|
||||||
annotations:
|
annotations:
|
||||||
kubernetes.io/ingress.class: nginx
|
kubernetes.io/ingress.class: nginx
|
||||||
host: chart-example.local
|
host: my-release.mycompany.com
|
||||||
EOF
|
EOF
|
||||||
|
|
||||||
helm template my-release onechart/onechart -f values.yaml
|
helm template my-release onechart/onechart -f values.yaml
|
||||||
|
|||||||
@@ -0,0 +1,53 @@
|
|||||||
|
Feature branch deploys in large part is a templating question:
|
||||||
|
|
||||||
|
- Names should be unique to avoid collision between application instances
|
||||||
|
- Names should follow some convention
|
||||||
|
- It's driven by CI, and git branch name is a typical input parameter
|
||||||
|
|
||||||
|
### Avoiding name collisions
|
||||||
|
|
||||||
|
With OneChart, you can drive the naming of most resources by setting a unique release name.
|
||||||
|
Release name is unique in Helm too, so it makes it a good tool to drive resource names.
|
||||||
|
|
||||||
|
One good practice can be to add a `-$BRANCH` suffix to the feature branch instance:
|
||||||
|
|
||||||
|
```
|
||||||
|
helm template my-release-my-branch onechart/onechart -f values.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
### Avoiding domain name collision
|
||||||
|
|
||||||
|
The release name will make all Kubernetes objects unique, but the domain name configuration remains static:
|
||||||
|
|
||||||
|
```
|
||||||
|
image:
|
||||||
|
repository: nginx
|
||||||
|
tag: 1.19.3
|
||||||
|
|
||||||
|
ingress:
|
||||||
|
annotations:
|
||||||
|
kubernetes.io/ingress.class: nginx
|
||||||
|
host: my-release.mycompany.com
|
||||||
|
|
||||||
|
helm template my-release-my-branch onechart/onechart -f values.yaml
|
||||||
|
```
|
||||||
|
|
||||||
|
The `ingress.host` name should also be dynamic to avoid the collision:
|
||||||
|
|
||||||
|
```
|
||||||
|
helm template my-release-my-branch onechart/onechart\
|
||||||
|
-f values.yaml \
|
||||||
|
--set ingress.host=my-release-my-branch.mycompany.com
|
||||||
|
```
|
||||||
|
|
||||||
|
### CI
|
||||||
|
|
||||||
|
In CI the above command needs to be templated:
|
||||||
|
|
||||||
|
```
|
||||||
|
helm template my-release-$BRANCH onechart/onechart\
|
||||||
|
-f values.yaml \
|
||||||
|
--set ingress.host=my-release-$BRANCH.mycompany.com
|
||||||
|
```
|
||||||
|
|
||||||
|
where $BRANCH is a built-in CI variable.
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
This page shows how you can add a `PrometheusRule` to your app deployment.
|
||||||
|
|
||||||
|
!!! Note
|
||||||
|
This is a feature only supported by the [kube-stack-prometheus stack (formerly known as the Prometheus Operator)](https://github.com/prometheus-operator/kube-prometheus)
|
||||||
|
|
||||||
|
The following Prometheus rule alerts if a pod is crashlooping:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
image:
|
||||||
|
repository: nginx
|
||||||
|
tag: 1.19.3
|
||||||
|
|
||||||
|
prometheusRules:
|
||||||
|
- name: KubePodCrashLooping
|
||||||
|
message: "Pod {{`{{`}} $labels.namespace {{`}}`}}/{{`{{`}} $labels.pod {{`}}`}} ({{`{{`}} $labels.container {{`}}`}}) is restarting {{`{{`}} printf \"%.2f\" $value {{`}}`}} times / 5 minutes."
|
||||||
|
runBookURL: myrunbook.com
|
||||||
|
expression: "rate(kube_pod_container_status_restarts_total{job=\"kube-state-metrics\", namespace=~\"{{ $targetNamespace }}\"}[15m]) * 60 * 5 > 0"
|
||||||
|
for: 1h
|
||||||
|
labels:
|
||||||
|
severity: critical
|
||||||
|
```
|
||||||
|
|
||||||
|
Check the Kubernetes manifest:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
helm template my-release onechart/onechart -f values.yaml
|
||||||
|
```
|
||||||
+3
-1
@@ -28,7 +28,7 @@ site_description: >-
|
|||||||
# Repository
|
# Repository
|
||||||
repo_name: onechart/onechart
|
repo_name: onechart/onechart
|
||||||
repo_url: https://github.com/onechart/onechart
|
repo_url: https://github.com/onechart/onechart
|
||||||
edit_uri: https://github.com/onechart/onechart/website
|
edit_uri: https://github.com/onechart/onechart/tree/master/website/docs
|
||||||
|
|
||||||
extra_css:
|
extra_css:
|
||||||
- stylesheets/extra.css
|
- stylesheets/extra.css
|
||||||
@@ -138,3 +138,5 @@ nav:
|
|||||||
- Volumes: examples/volumes.md
|
- Volumes: examples/volumes.md
|
||||||
- Custom command: examples/custom-command.md
|
- Custom command: examples/custom-command.md
|
||||||
- Cron job: examples/cron-job.md
|
- Cron job: examples/cron-job.md
|
||||||
|
- Feature branch deploys: examples/feature-branch-deploys.md
|
||||||
|
- Prometheus monitoring rules: examples/prometheus-monitoring-rules.md
|
||||||
|
|||||||
Reference in New Issue
Block a user