In order to create a silence, you'll need to login to your Grafana instance that is deployed as part of Atmosphere as an admin user.
Click on the hamburger menu in the top left corner and select "Alerting" and then "Silences" from the menu.
Ensure that you select "AlertManager" on the top right corner of the page, this will make sure that you create a silence inside of the AlertManager that is managed by the Prometheus operator instead of the built-in Grafana AlertManager which is not used.
!!! warning
It's important that you select the AlertManager that is managed by the Prometheus operator, otherwise your silence will not be applied to the Prometheus instance that is deployed as part of Atmosphere.
Click the "Add Silence" button and use the AlertManager format to create your silence, which you can test by seeing if it matches any alerts in the list labeled "Affected alert instances".
!!! note
It is strongly recommended that you create a silence with the least amount of needed labels which will make sure that small minor changes to the alerts will not break your silence.
For Grafana, rather than enabling persistence through the application's user interface or manual Helm chart modifications, dashboards should be managed directly via the Helm chart values.
!!! warning
It is important to avoid manual persistence configurations, especially for services like Grafana, where dashboards and data sources can be saved. Such practices are not captured in version control and pose a risk of data loss, configuration drift, and upgrade complications.
To manage Grafana dashboards through Helm, you can include the dashboard definitions within your configuration file. By doing so, you facilitate version-controlled dashboard configurations that can be replicated across different deployments without manual intervention.
For example, a dashboard can be defined in the Helm values like this:
kube_prometheus_stack_helm_values: grafana: dashboards: default: my-dashboard: gnetId: 10000 revision: 1 datasource: Prometheus
This instructs Helm to fetch and configure the specified dashboard from Grafana.com, using Prometheus as the data source.
You can find more examples of how to do this in the Grafana Helm chart Documentation.