kylebrandt

Grafana Managed Alert Rule Issue Debugging

Needed context

In order to figure what is going on with Grafana managed alerts generic issue, we can use (or need) the following information.

Less may be needed depending how far along in the pipeline the issue is (if it is at the end like notifications, then can need all the stuff that takes us to that point).

Configuration

kylebrandt

{
  "template_files": {},
  "alertmanager_config": {
    "route": {
      "receiver": "webhook_test",
      "group_by": [
        "alertname"
      ]
    },
    "templates": [],

kylebrandt

Alert Migration Notes

List of things the migration does or will need to do

• Load the dashboard alert from alert table and create rules in the alert_rule table
  • Translate conditions and queries into SSE for the alert_rule row
  • Translate other settings like NoData,FOR,Interval,alertRuleTags,(notifications?) for alert_rule row
  • Match Permissions of dashboard alert (create folders as needed):
    • Alerts will not have permissions, Folders will
    • If alert's dashboard has permissions, create corresponding folder to match permissions

kylebrandt

POST http://admin:admin@localhost:3000/api/alert-definitions/
Content-Type: application/json

{
"title": "classic many",
"condition": "G",
"data": [
  {
    "refId": "A",

kylebrandt

POST http://admin:admin@localhost:3000/ruler/grafana/api/v1/rules/ng-alert-demo
Content-Type: application/json

{
  "name": "test-group2",
  "interval": "10s",
  "rules": [
    {
      "grafana_alert": {
        "title": "pp query with SSE classic condition",

kylebrandt

POST http://admin:admin@localhost:3000/ruler/grafana/api/v1/rules/ng-alert-demo
Content-Type: application/json

{
  "name": "test-group",
  "interval": "10s",
  "rules": [
    {
      "grafana_alert": {
        "title": "prom query with SSE classic condition",

kylebrandt

kylebrandt

xclip -o -selection clipboard | jq '.conditions | { "conditions": . }' |  curl -H 'Content-Type: application/json' -X POST --data-binary @-  http://admin:admin@localhost:3000/api/alert-definitions/evalOld | jq -r '.instances[0]' | base64 -d > ~/tmp/af && arrow-cat ~/tmp/af

kylebrandt

PUT http://admin:admin@localhost:3000/api/alert-definitions/HH7cv0yMk
Content-Type: application/json

{
  "id": 24,
  "orgId": 1,
  "title": "First CC Alert",
  "condition": "B",
  "data": [
    {

kylebrandt

"conditions": [
   {
     "evaluator": {
       "params": [
         0
       ],
       "type": "gt"
     },
     "operator": {
       "type": "and"