If you have applications in Azure, there is a good chance you're making use of Azure Monitor with Application Insights or Log Analytics. These can provide useful diagnostic information, healthchecks, and alerting for VM's, serverless function apps, containers, and other services. The alerts can send email and call webhooks, but for the most flexibility, delivering them to an Azure Function allows for robust processing with custom F# code. With the azurts library, this can be used to filter alerts based on content and deliver them to different Slack channels.
Azurts uses a "railway oriented programming" technique that is popular in several F# libraries to compose a series of hooks together, where a Hook is a simple function that accepts one type as input and optionally returns Some value or None.
type Hook<'a, 'b> = 'a -> 'b optionA compose operator >=> is included that will bind two Hook functions together, meaning that if the first one returns Some value, then it will pass that value to the second function, which then returns an option as well. Chain as many together as needed to filter or enhance data before the final Hook in the chain, which is typically the one that will send it to Slack. For example
Alert.tryParse >=> Payload.ofAzureAlert "alerts-channel" >=> Payload.sendToSlack webHookConfigBy hosting this in an Azure function, Azure Monitor alerts can be delivered to the function, which will convert them to Slack messages and deliver to the channel.
namespace AzureAlertWebhook
open System.IO
open Microsoft.AspNetCore.Mvc
open Microsoft.Azure.WebJobs
open Microsoft.AspNetCore.Http
open Microsoft.Extensions.Logging
open azurts.Hook
open azurts.AzureAlert
open azurts.SlackWebHook
module WebHook =
let http = new System.Net.Http.HttpClient ()
let Run([<HttpTrigger(Methods=[|"POST"|])>] req:HttpRequest) (log:ILogger) =
async {
use reader = new StreamReader (req.Body)
let! alertJson = reader.ReadToEndAsync () |> Async.AwaitTask
let config =
{
Client = http
WebHookUri = System.Uri "https://hooks.slack.com/services/..."
Token = Some "UakM4mV1GMsDhMvA4KxKEyvnV5d7Q3qmwfScC1PuLmg="
ErrorCallback = fun (httpStatus, err) -> log.LogError (err)
}
let hook = Alert.tryParse >=> Payload.ofAzureAlert "alerts-channel" >=> Payload.sendToSlack config
match alertJson |> hook with
| Some send -> do! send
| None -> ()
return OkResult ()
} |> Async.StartAsTaskTo actually make use of this, create an Azure FunctionApp for this function and create an HTTP trigger, using IDE, the CLI tools or even building from scratch in F#. Once you publish the FunctionApp, you are provided with a URL to send messages.
From Azure Monitor in the Azure Portal, configure Alert Actions:
Create a new "Action Group" for sending to the FunctionApp, either selecting the FunctionApp directly or by specifying a Webhook where you can provide the URL that is generated when publishing the FunctionApp.
Now you can create an Alert using a Custom Log Search and choose the Action Group that sends to the FunctionApp. When the Alert is fired, it will send a JSON payload for the alert with records for all of the log entries that triggered that alert in the polling period. So if it polls every 30 minutes and there are 4 items that would trigger that alert in that 30 minutes, it will send a single alert payload that contains those 4 items. The Payload.ofAzureAlert will convert each of them into a Slack message, all with the same Context that shows what log search sent them.
The basic workflow of 1. parse, 2. build payload, 3. send to Slack is a nice introduction, but in real world use, richer functionality like filtering and broadcasting are often necessary. For example, alerts may need to be delivered to different channels depending on severity or even custom trace properties. Many included Hook functions can be composed together to define the alert processing logic. The broadcast function accepts a list of other Hook functions and will iterate through each of them, executing each Hook that doesn't return a None value for the option. Combined with Hook functions from the Filters module, alerts can be routed to one or more channels based on content.
For example, this hook sends high severity alerts to the critical-alerts-channel, sends lower severity alerts to regular-alerts-channel and also sends all alerts for ResourceOne to the resource-one-alerts channel.
Alert.tryParse >=> broadcast
[
Filters.minimumSeverity 3 >=> Payload.ofAzureAlert "critical-alerts-channel" >=> sendToSlack config
Filters.maximumSeverity 2 >=> Payload.ofAzureAlert "regular-alerts-channel" >=> sendToSlack config
Filters.fieldValue "customDimensions_ResourceName" "ResourceOne" >=> Payload.ofAzureAlert "resource-one-alerts" >=> sendToSlack config
]It's useful to build Slack messages outside of Azure Monitor alerting, so it's worth noting that the Slack WebHook DSL is usable on its own to be composed with other solutions.
A Slack message payload is just the channel and a list of Blocks:
type Payload =
{
Channel : string
Blocks : Block list
}The Blocks used to build messages are implemented by the Block discriminated union:
type Block =
| Context of Context
| Divider of Divider
| Section of SectionThe message itself is made of a lot of text, which can be either simple markdown or some labeled text elements which are rendered to show a labeled field that Slack will layout in one or two columns.
type Text =
| Markdown of string
| LabeledText of Label:string * Content:stringThe divider is pretty simple and a single case just for dividing messages:
type Divider =
| Divider`The context can provide a simple bit of subtext in a message, built from a list of Text elements:
type Context =
| Elements of Text listA section can contain either a single Text element or Fields which is a list of Text elements.
type Section =
| Text of Text
| Fields of Text listThere is a function included in the SlackPayload module to create a payload from an Azure Monitor Alert that can be used as a guide to implement your own message payloads.
It's worth mentioning that this is useful for webhooks beyond just Azure Monitor alerts because composition makes it possible to bring your own records and objects, and format them into a Slack message. Or instead of delivering to Slack, the could deliver to other services, like certain critical messages could deliver to a PagerDuty hook. Anything that can be described as type Hook<'a, 'b> = 'a -> 'b option can be integrated into the alert processing pipeline.
Building the Slack webhook DSL was fun but also sometimes frustrating. The responses returned from the API aren't very useful when there is a payload that isn't structured properly. There are also some lightly documented rules like how a text field has a maximum length of 3000 characters. The API reference is very thorough, and the SlackWebHook.fs module implements it to ensure messages meet Slack's API requirements.
I also learned that the built in webhook payload formatting built into Azure Monitor is relatively limited. I initially asked myself Why not use a Slack webhook directly - why put F# in the middle? While that can work, the resulting Slack message leaves a lot to be desired. Basic formatting of alert information is possible, but little else, meaning anyone watching the Slack channel will have to go back to Azure Monitor to get the details.
Since we moved from using the basic webhook to using the formatted messages through azurts, it's been a lot easier for people to see and act on alerts, and as a result, people resolve and take steps to prevent them. The expressiveness of F# and "railway composition" makes it easy for people to understand how alerts are delivered, and the type safety enables others to contribute without fear of breaking the alerts that are in place. Please use it as you like and contributions are welcome to extend the alert processing and message formatting capabilities!
Logo's are used per the guidelines from Azure and Slack and originals can be found on the Azure and Slack websites.