How to use Grafana Alerts with the Telegram

Grafana Alert Template.md

• Template name: telegram.message
• Content:

  {{- /* Telegram message to use: {{ template "telegram.message2" . }} */ -}}
  {{ define "__alerts_list" -}}
  {{ range . }}
  {{if ne (index .Labels "alertname") "" -}}
  {{ if eq .Status "firing" }}🔴{{ else }}🟢{{ end }}
      {{- if ne (index .Labels "severity") "" -}}
          <u><b>P{{ index .Labels "severity" }}</b></u> {{ end -}}
  <b>{{ index .Labels "alertname" }}</b> 🕙 {{ .StartsAt.Format "15:04:05    🗓️ 2006-01-02" }}{{ end -}}
  {{ if len .Annotations }}
  <i>Annotations:</i>
      {{ range .Annotations.SortedPairs -}}
      - {{ .Name }}: {{ .Value }}
      {{ end -}}
  {{ end }}
  {{ if len .Labels -}}
  <i>Labels:</i>
      {{ range .Labels.SortedPairs -}}
      - {{ .Name }}: {{ .Value }}
      {{ end -}}
  {{ end }}
  <i>Value:</i> <pre>{{ .ValueString }}</pre>
      {{- if gt (len .GeneratorURL) 0 }}<a href="{{ .GeneratorURL }}">source</a>  |  {{ end }}
      {{- if gt (len .SilenceURL) 0 }}<a href="{{ .SilenceURL }}">🔕 silence</a>  |  {{ end }}
      {{- if gt (len .DashboardURL) 0 }}📁 <a href="{{ .DashboardURL }}">dashboard</a>  |  {{ end }}
      {{- if gt (len .PanelURL) 0 }}<a href="{{ .PanelURL }}">panel</a> {{- end -}}
      <pre>--------</pre>
  {{- end -}} {{- /* range */ -}}
  {{- end -}} {{- /* define __alerts_list */ -}}
  
  {{ define "__telegram.title" -}}
  {{ if ne (index .CommonLabels "severity") "" }} <u><b>P{{ index .CommonLabels "severity" }}</b></u> {{ end -}}
  {{ if ne (index .CommonLabels "alertname") "" -}}
      [{{ index .CommonLabels "alertname" }}]
  {{- end -}}
  {{- end -}}{{- /* define __telegram */ -}}
  
  {{ define "telegram.message" }}
      {{ if gt (len .Alerts.Firing) 0 }}
      🚨 <b>ALARM</b> (#{{ .Alerts.Firing | len }})
      {{- template "__alerts_list" .Alerts.Firing }}{{ end -}}
      {{ if gt (len .Alerts.Resolved) 0 }}
      ✅ <b>RESOLVED</b>{{ template "__telegram.title" . }} (#{{ .Alerts.Resolved | len }})
      {{- template "__alerts_list" .Alerts.Resolved }}{{ end }}
  
  <a href="{{ .ExternalURL }}">📲 Grafana</a>
  {{- end -}}

- Contact point | Optional Telegram settings | Message: `{{ template "telegram.message" . }}`
- Add verbose annotations to your alerts. Example: `The <code>{{ $labels.host }}</code> host is low on free space! Used: {{ $values.display.Value }}%` (where `display` is one of the queries).
- I usually add the following queries: `data` (the datasource query), `metric` (the reduce expression), `condition`, and `display`, which is `round($metric)`. [Multi-dimensional rules documentation](https://grafana.com/docs/grafana/next/alerting/alerting-rules/create-grafana-managed-rule).

Hi! Thanks for the template, can you send an example of an alert rule in yaml to work with this template?

Hello @gelldur
Is it possible to include the server in value parameter of solved? The empty value will make ambiguity to identify which system have been solved.
Another question is, is it possible to change summary of solved? For now in both cases (firing and solved) it uses the same summary.

Updated for Grafana 9 where now we have .ValueString instead of nice map of values (just tell my why :/) This is how it looks:

thanks a lot for the developed template. this is very useful.
just a question, the "href"s, they are not shown as links in my telegram channel, even tho i have set the parse mode to HTML in the contact point telegram optional settings. i am getting just texts

source | 🔕 silence | 📁 dashboard | panel

can you guys tell me where and what settings i need to apply to get as hyperlinks? i have two channels (webhook + telegram) and in my webhook it is properly received as hyperlinks, but not in telegram

thanks in advance

Hello @gelldur Is it possible to include the server in value parameter of solved? The empty value will make ambiguity to identify which system have been solved. Another question is, is it possible to change summary of solved? For now in both cases (firing and solved) it uses the same summary.

Not sure, I really lost a lot of time on this...

Updated for Grafana 9 where now we have .ValueString instead of nice map of values (just tell my why :/) This is how it looks:

thanks a lot for the developed template. this is very useful. just a question, the "href"s, they are not shown as links in my telegram channel, even tho i have set the parse mode to HTML in the contact point telegram optional settings. i am getting just texts

source | 🔕 silence | 📁 dashboard | panel

can you guys tell me where and what settings i need to apply to get as hyperlinks? i have two channels (webhook + telegram) and in my webhook it is properly received as hyperlinks, but not in telegram

thanks in advance

I don't see any configuration in my grafana instance for this. Maybe this is issue on telegram client side? Trusted/untrusted stuff?

thanks for quick turnaround. i guess the issue is in here:

{{- if gt (len .GeneratorURL) 0 }} <a href={{ .GeneratorURL }}> source </a>  |  {{ end }}
{{- if gt (len .SilenceURL) 0 }} <a href="{{ .SilenceURL }}"> 🔕 silence </a>  |  {{ end }}
{{- if gt (len .DashboardURL) 0 }} <a href="{{ .DashboardURL }}"> 📁 dashboard</a>  |  {{ end }}
{{- if gt (len .PanelURL) 0 }} <a href="http://{{ .PanelURL }}"> panel </a> {{- end -}}

because when i put just {{ .GeneratorURL }} in the template the URL is properly printed, but when it is in the if and in a href it is not showing up at all, leading to print just "source" in the alert. is it properly working on your side? I am on grafana 10. i guess i need to play around to see how i will overcome this.

thanks again!

Tested on Grafana 9.X only, so maybe yes on Grafana 10 it may not work.

thanks again, i have also tried to downgrade to 9.5.18, and it is still doing the same. i guess i am missing a step :(

{{- /* Telegram message to use: {{ template "telegram.message2" . }} */ -}}
{{ define "__alerts_list" -}}
{{ range . }}
{{if ne (index .Labels "alertname") "" -}}
{{ if eq .Status "firing" }}🔴{{ else }}🟢{{ end }}
    {{- if ne (index .Labels "severity") "" -}}
        <u><b>P{{ index .Labels "severity" }}</b></u> {{ end -}}
<b>{{ index .Labels "alertname" }}</b> 🕙 {{ .StartsAt.Format "15:04:05    🗓️ 2006-01-02" }}{{ end -}}
{{ if len .Annotations }}
<i>Annotations:</i>
    {{ range .Annotations.SortedPairs -}}
    - {{ .Name }}: {{ .Value }}
    {{ end -}}
{{ end }}
{{ if len .Labels -}}
<i>Labels:</i>
    {{ range .Labels.SortedPairs -}}
    - {{ .Name }}: {{ .Value }}
    {{ end -}}
{{ end }}
<i>Value:</i> <pre>{{ .ValueString }}</pre>
    {{- if gt (len .GeneratorURL) 0 }} <a href="{{ .GeneratorURL }}"> source </a>  |  {{ end }}
    {{- if gt (len .SilenceURL) 0 }} <a href="{{ .SilenceURL }}"> 🔕 silence </a>  |  {{ end }}
    {{- if gt (len .DashboardURL) 0 }} <a href="{{ .DashboardURL }}"> 📁 dashboard</a>  |  {{ end }}
    {{- if gt (len .PanelURL) 0 }} <a href="{{ .PanelURL }}"> panel </a> {{- end -}}
    <pre>--------</pre>
{{- end -}} {{- /* range */ -}}
{{- end -}} {{- /* define __alerts_list */ -}}

{{ define "__telegram.title" -}}
{{ if ne (index .CommonLabels "severity") "" }} <u><b>P{{ index .CommonLabels "severity" }}</b></u> {{ end -}}
{{ if ne (index .CommonLabels "alertname") "" -}}
    [{{ index .CommonLabels "alertname" }}]
{{- end -}}
{{- end -}}{{- /* define __telegram */ -}}

{{ define "telegram.message" }}
    {{ if gt (len .Alerts.Firing) 0 }}
    🚨 <b>ALARM</b> (#{{ .Alerts.Firing | len }})
    {{- template "__alerts_list" .Alerts.Firing }}{{ end -}}
    {{ if gt (len .Alerts.Resolved) 0 }}
    ✅ <b>RESOLVED</b>{{ template "__telegram.title" . }} (#{{ .Alerts.Resolved | len }})
    {{- template "__alerts_list" .Alerts.Resolved }}{{ end }}
<a href="{{ .ExternalURL }}">📲 Grafana</a>
{{- end -}}

in the message i have put:

{{ template "telegram.message" . }}

and in parse mode i have put HTML.

not sure what else i need to change, but i am getting plain text in telegram without hyperlinks:

 source   |    🔕 silence   |    📁 dashboard  |    panel 

copy
--------

📲 Grafana

when i switch the parse mode to markdown i am clearly getting hyperlinks but not html formatted:

silence </a>  |   <a href="http://localhost:3000/d/bdjhure0cbvggc?orgId=1"> 📁 dashboard</a>  |   <a href="http://localhost:3000/d/bdjhure0cbvggc?orgId=1&viewPanel=1"> panel </a><pre>--------</pre>

<a href="http://localhost:3000/">📲 Grafana</a>

thanks anyways, i will try to find the root cause for this.

by the way, which version of grafana are you using exactly ? i can try to reproduce on that one

@arturucraft I'm using v9.2.20. Even so In this version I don't have option "parse mode". No such option in contact points or templates.

@arturucraft I'm using v9.2.20. Even so In this version I don't have option "parse mode". No such option in contact points or templates.

ok, thanks! i am struggling with this till now. i have opened a bug, to see why my html content is not normally rendering

@arturucraft just had a thought. Maybe your message is too long and thats why telegram not parsing it as full html? Try sending only single link message to confirm.

i think that's not the problem, because even i add another html code, it parses/renders it as expected, when i put {{ .DashboardURL }} it displaysthe variable correctly, if i put <a href="http://example.com">aaaa</a> it renders it properly, the problem is only in combination of a href and {{ .DashboardURL }} where it does not render <a href="{{ .DashboardURL }}"> 📁 dashboard</a>

Last think I could suggest remove emoji? Other things to test:

<a href="{{ .DashboardURL }}">{{ .DashboardURL }}<\a>

As I understand this variable is working for you only not rendering(you can't click on it).
Maybe you will find something in docs: https://grafana.com/docs/grafana/latest/alerting/manage-notifications/template-notifications/reference/

Best luck.

Last think I could suggest remove emoji? Other things to test:

<a href="{{ .DashboardURL }}">{{ .DashboardURL }}<\a>

As I understand this variable is working for you only not rendering(you can't click on it). Maybe you will find something in docs: https://grafana.com/docs/grafana/latest/alerting/manage-notifications/template-notifications/reference/

Best luck.

thanks again!

i have two other lines which do not contain emojies, but still does not work as expected:

    {{- if gt (len .GeneratorURL) 0 }} <a href="{{ .GeneratorURL }}"> source </a>  |  {{ end }}
    {{- if gt (len .PanelURL) 0 }} <a href="{{ .PanelURL }}"> panel </a> {{- end -}}

so i still believe it is smth related to combination of those two, or i am missing something, not sure what :(

Hi,

i had the same problem and yesterday i was able to resolve the issue and the links are finally working.

I'm using the docker container and after i set the following env variable it started to work:

GF_SERVER_ROOT_URL=https://my-domain.com