How to use Grafana Alerts with the Telegram

Grafana Alert Template.md

• Template name: telegram.message
• Content:

  {{- /* Telegram message to use: {{ template "telegram.message2" . }} */ -}}
  {{ define "__alerts_list" -}}
  {{ range . }}
  {{if ne (index .Labels "alertname") "" -}}
  {{ if eq .Status "firing" }}🔴{{ else }}🟢{{ end }}
      {{- if ne (index .Labels "severity") "" -}}
          <u><b>P{{ index .Labels "severity" }}</b></u> {{ end -}}
  <b>{{ index .Labels "alertname" }}</b> 🕙 {{ .StartsAt.Format "15:04:05    🗓️ 2006-01-02" }}{{ end -}}
  {{ if len .Annotations }}
  <i>Annotations:</i>
      {{ range .Annotations.SortedPairs -}}
      - {{ .Name }}: {{ .Value }}
      {{ end -}}
  {{ end }}
  {{ if len .Labels -}}
  <i>Labels:</i>
      {{ range .Labels.SortedPairs -}}
      - {{ .Name }}: {{ .Value }}
      {{ end -}}
  {{ end }}
  <i>Value:</i> <pre>{{ .ValueString }}</pre>
      {{- if gt (len .GeneratorURL) 0 }}<a href="{{ .GeneratorURL }}">source</a>  |  {{ end }}
      {{- if gt (len .SilenceURL) 0 }}<a href="{{ .SilenceURL }}">🔕 silence</a>  |  {{ end }}
      {{- if gt (len .DashboardURL) 0 }}📁 <a href="{{ .DashboardURL }}">dashboard</a>  |  {{ end }}
      {{- if gt (len .PanelURL) 0 }}<a href="{{ .PanelURL }}">panel</a> {{- end -}}
      <pre>--------</pre>
  {{- end -}} {{- /* range */ -}}
  {{- end -}} {{- /* define __alerts_list */ -}}
  
  {{ define "__telegram.title" -}}
  {{ if ne (index .CommonLabels "severity") "" }} <u><b>P{{ index .CommonLabels "severity" }}</b></u> {{ end -}}
  {{ if ne (index .CommonLabels "alertname") "" -}}
      [{{ index .CommonLabels "alertname" }}]
  {{- end -}}
  {{- end -}}{{- /* define __telegram */ -}}
  
  {{ define "telegram.message" }}
      {{ if gt (len .Alerts.Firing) 0 }}
      🚨 <b>ALARM</b> (#{{ .Alerts.Firing | len }})
      {{- template "__alerts_list" .Alerts.Firing }}{{ end -}}
      {{ if gt (len .Alerts.Resolved) 0 }}
      ✅ <b>RESOLVED</b>{{ template "__telegram.title" . }} (#{{ .Alerts.Resolved | len }})
      {{- template "__alerts_list" .Alerts.Resolved }}{{ end }}
  
  <a href="{{ .ExternalURL }}">📲 Grafana</a>
  {{- end -}}

- Contact point | Optional Telegram settings | Message: `{{ template "telegram.message" . }}`
- Add verbose annotations to your alerts. Example: `The <code>{{ $labels.host }}</code> host is low on free space! Used: {{ $values.display.Value }}%` (where `display` is one of the queries).
- I usually add the following queries: `data` (the datasource query), `metric` (the reduce expression), `condition`, and `display`, which is `round($metric)`. [Multi-dimensional rules documentation](https://grafana.com/docs/grafana/next/alerting/alerting-rules/create-grafana-managed-rule).

thanks for quick turnaround. i guess the issue is in here:

{{- if gt (len .GeneratorURL) 0 }} <a href={{ .GeneratorURL }}> source </a>  |  {{ end }}
{{- if gt (len .SilenceURL) 0 }} <a href="{{ .SilenceURL }}"> 🔕 silence </a>  |  {{ end }}
{{- if gt (len .DashboardURL) 0 }} <a href="{{ .DashboardURL }}"> 📁 dashboard</a>  |  {{ end }}
{{- if gt (len .PanelURL) 0 }} <a href="http://{{ .PanelURL }}"> panel </a> {{- end -}}

because when i put just {{ .GeneratorURL }} in the template the URL is properly printed, but when it is in the if and in a href it is not showing up at all, leading to print just "source" in the alert. is it properly working on your side? I am on grafana 10. i guess i need to play around to see how i will overcome this.

thanks again!

Tested on Grafana 9.X only, so maybe yes on Grafana 10 it may not work.

thanks again, i have also tried to downgrade to 9.5.18, and it is still doing the same. i guess i am missing a step :(

{{- /* Telegram message to use: {{ template "telegram.message2" . }} */ -}}
{{ define "__alerts_list" -}}
{{ range . }}
{{if ne (index .Labels "alertname") "" -}}
{{ if eq .Status "firing" }}🔴{{ else }}🟢{{ end }}
    {{- if ne (index .Labels "severity") "" -}}
        <u><b>P{{ index .Labels "severity" }}</b></u> {{ end -}}
<b>{{ index .Labels "alertname" }}</b> 🕙 {{ .StartsAt.Format "15:04:05    🗓️ 2006-01-02" }}{{ end -}}
{{ if len .Annotations }}
<i>Annotations:</i>
    {{ range .Annotations.SortedPairs -}}
    - {{ .Name }}: {{ .Value }}
    {{ end -}}
{{ end }}
{{ if len .Labels -}}
<i>Labels:</i>
    {{ range .Labels.SortedPairs -}}
    - {{ .Name }}: {{ .Value }}
    {{ end -}}
{{ end }}
<i>Value:</i> <pre>{{ .ValueString }}</pre>
    {{- if gt (len .GeneratorURL) 0 }} <a href="{{ .GeneratorURL }}"> source </a>  |  {{ end }}
    {{- if gt (len .SilenceURL) 0 }} <a href="{{ .SilenceURL }}"> 🔕 silence </a>  |  {{ end }}
    {{- if gt (len .DashboardURL) 0 }} <a href="{{ .DashboardURL }}"> 📁 dashboard</a>  |  {{ end }}
    {{- if gt (len .PanelURL) 0 }} <a href="{{ .PanelURL }}"> panel </a> {{- end -}}
    <pre>--------</pre>
{{- end -}} {{- /* range */ -}}
{{- end -}} {{- /* define __alerts_list */ -}}

{{ define "__telegram.title" -}}
{{ if ne (index .CommonLabels "severity") "" }} <u><b>P{{ index .CommonLabels "severity" }}</b></u> {{ end -}}
{{ if ne (index .CommonLabels "alertname") "" -}}
    [{{ index .CommonLabels "alertname" }}]
{{- end -}}
{{- end -}}{{- /* define __telegram */ -}}

{{ define "telegram.message" }}
    {{ if gt (len .Alerts.Firing) 0 }}
    🚨 <b>ALARM</b> (#{{ .Alerts.Firing | len }})
    {{- template "__alerts_list" .Alerts.Firing }}{{ end -}}
    {{ if gt (len .Alerts.Resolved) 0 }}
    ✅ <b>RESOLVED</b>{{ template "__telegram.title" . }} (#{{ .Alerts.Resolved | len }})
    {{- template "__alerts_list" .Alerts.Resolved }}{{ end }}
<a href="{{ .ExternalURL }}">📲 Grafana</a>
{{- end -}}

in the message i have put:

{{ template "telegram.message" . }}

and in parse mode i have put HTML.

not sure what else i need to change, but i am getting plain text in telegram without hyperlinks:

 source   |    🔕 silence   |    📁 dashboard  |    panel 

copy
--------

📲 Grafana

when i switch the parse mode to markdown i am clearly getting hyperlinks but not html formatted:

silence </a>  |   <a href="http://localhost:3000/d/bdjhure0cbvggc?orgId=1"> 📁 dashboard</a>  |   <a href="http://localhost:3000/d/bdjhure0cbvggc?orgId=1&viewPanel=1"> panel </a><pre>--------</pre>

<a href="http://localhost:3000/">📲 Grafana</a>

thanks anyways, i will try to find the root cause for this.

by the way, which version of grafana are you using exactly ? i can try to reproduce on that one

@arturucraft I'm using v9.2.20. Even so In this version I don't have option "parse mode". No such option in contact points or templates.

@arturucraft I'm using v9.2.20. Even so In this version I don't have option "parse mode". No such option in contact points or templates.

ok, thanks! i am struggling with this till now. i have opened a bug, to see why my html content is not normally rendering

@arturucraft just had a thought. Maybe your message is too long and thats why telegram not parsing it as full html? Try sending only single link message to confirm.

i think that's not the problem, because even i add another html code, it parses/renders it as expected, when i put {{ .DashboardURL }} it displaysthe variable correctly, if i put <a href="http://example.com">aaaa</a> it renders it properly, the problem is only in combination of a href and {{ .DashboardURL }} where it does not render <a href="{{ .DashboardURL }}"> 📁 dashboard</a>

Last think I could suggest remove emoji? Other things to test:

<a href="{{ .DashboardURL }}">{{ .DashboardURL }}<\a>

As I understand this variable is working for you only not rendering(you can't click on it).
Maybe you will find something in docs: https://grafana.com/docs/grafana/latest/alerting/manage-notifications/template-notifications/reference/

Best luck.

Last think I could suggest remove emoji? Other things to test:

<a href="{{ .DashboardURL }}">{{ .DashboardURL }}<\a>

As I understand this variable is working for you only not rendering(you can't click on it). Maybe you will find something in docs: https://grafana.com/docs/grafana/latest/alerting/manage-notifications/template-notifications/reference/

Best luck.

thanks again!

i have two other lines which do not contain emojies, but still does not work as expected:

    {{- if gt (len .GeneratorURL) 0 }} <a href="{{ .GeneratorURL }}"> source </a>  |  {{ end }}
    {{- if gt (len .PanelURL) 0 }} <a href="{{ .PanelURL }}"> panel </a> {{- end -}}

so i still believe it is smth related to combination of those two, or i am missing something, not sure what :(

Hi,

i had the same problem and yesterday i was able to resolve the issue and the links are finally working.

I'm using the docker container and after i set the following env variable it started to work:

GF_SERVER_ROOT_URL=https://my-domain.com