Monitoring

Exposing data

There are a few ways to expose both the monitoring services to view the health and the metrics of the cluster.

Port forwarding

The easiest way to expose the monitoring services is to use port forwarding using the built-in kubectl command.

Prometheus

kubectl -n monitoring port-forward svc/kube-prometheus-stack-prometheus 9090

Once you run the command above, you'll be able to open http://localhost:9090 on your local system and view the Prometheus UI.

AlertManager

kubectl -n monitoring port-forward svc/kube-prometheus-stack-alertmanager 9093

Once you run the command above, you'll be able to open http://localhost:9093 on your local system and view the AlertManager UI.

Unprotected access

If you want to expose the monitoring services, you can use the following overrides which will create an Ingress for all the services.

!!! danger

This will expose the monitoring services without any authentication or
authorization. This is not recommended for production environments or any
environment where the monitoring services are exposed to the public internet.

kube_prometheus_stack_values:
  alertmanager:
    ingress:
      enabled: true
      annotations:
        cert-manager.io/cluster-issuer: atmosphere
      hosts:
        - alertmanager.example.com
      tls:
        - secretName: alertmanager-tls
          hosts:
            - alertmanager.example.com
    alertmanagerSpec:
      externalUrl: https://alertmanager.example.com
  prometheus:
    ingress:
      enabled: true
      ingressClassName: openstack
      annotations:
        cert-manager.io/cluster-issuer: atmosphere
      hosts:
        - prometheus.example.com
      tls:
        - secretName: prometheus-certs
          hosts:
            - prometheus.example.com
    prometheusSpec:
      externalUrl: https://prometheus.example.com

Protected access

If you want to expose the monitoring services, you can use the following overrides which will create an Ingress for all the services.

kube_prometheus_stack_values:
  alertmanager:
    ingress:
      enabled: true
      annotations:
        cert-manager.io/cluster-issuer: atmosphere
        nginx.ingress.kubernetes.io/auth-type: basic
        nginx.ingress.kubernetes.io/auth-secret: prometheus-auth
        nginx.ingress.kubernetes.io/auth-realm: Prometheus
      hosts:
        - alertmanager.example.com
      tls:
        - secretName: alertmanager-tls
          hosts:
            - alertmanager.example.com
    alertmanagerSpec:
      externalUrl: https://alertmanager.example.com
  prometheus:
    ingress:
      enabled: true
      ingressClassName: openstack
      annotations:
        cert-manager.io/cluster-issuer: atmosphere
        nginx.ingress.kubernetes.io/auth-type: basic
        nginx.ingress.kubernetes.io/auth-secret: prometheus-auth
        nginx.ingress.kubernetes.io/auth-realm: Prometheus
      hosts:
        - prometheus.example.com
      tls:
        - secretName: prometheus-certs
          hosts:
            - prometheus.example.com
    prometheusSpec:
      externalUrl: https://prometheus.example.com

Once you've deployed with the overrides above, you'll need to create a secret with the username and password you want to use to access the monitoring services.

htpasswd -c auth monitoring

The above will generate a file called auth with the username and password, in this case the username is monitoring. You'll need to create a secret with the contents of the file.

kubectl -n monitoring create secret generic prometheus-auth --from-file=auth

Once you're done, you'll be able to access the monitoring services using the username and password you created.

!!! info

In the future, Atmosphere will support automating the process of exposing
the monitoring stack with basic authentication.

Integrations

OpsGenie

Atmosphere can be integrated with OpsGenie in order to send all alerts to it, this is useful if you want to have a single place to manage all your alerts.

In order to get started, you will need to complete the following steps inside OpsGenie:

1. Create an integration inside OpsGenie, you can do this by going to Settings > Integrations > Add Integration and selecting Prometheus.
2. Copy the API key that is generated for you and setup correct assignment rules inside OpsGenie.
3. Create a new heartbeat inside OpsGenie, you can do this by going to Settings > Heartbeats > Create Heartbeat. Set the interval to 1 minute.

Afterwards, you can configure the following options for the Atmosphere config:

atmosphere_opsgenie_config:
  enabled: true
  api_key: <your-api-key>
  heartbeat: <your-heartbeat-name>

Once this is done and deployed, you'll start to see alerts inside OpsGenie and you can also verify that the heartbeat is listed as ACTIVE.

Alerts

etcdDatabaseHighFragmentationRatio

kubectl -n kube-system exec svc/kube-prometheus-stack-kube-etcd -- \
  etcdctl defrag \
  --cluster \
  --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --key /etc/kubernetes/pki/etcd/server.key \
  --cert /etc/kubernetes/pki/etcd/server.crt