Alert: Lambda High Error Rate
Date: 2026-07-15
Status: Resolved (all instances Normal)
Severity: P4
Slack: #sp-monitoring-alerts-prod
Spark group: UK Sky Protect IoT SRE
Runbook: https://sky-group.atlassian.net/wiki/x/nWHi
Monitors all Lambda functions in account 446133167807. Fires when:
(SUM(Errors) / SUM(Invocations)) * 100 > 10%
...sustained for 3 consecutive minutes, evaluated over a rolling 10-minute window, grouped by FunctionName.
noDataState: OK | execErrState: OK
The alert is currently inactive -- all 11 monitored function instances are Normal. However, investigation reveals a persistent daily failure in one function that has been broken for at least 7 consecutive days.
100% error rate -- every invocation times out at the Lambda maximum (15 minutes).
| Metric | Value |
|---|---|
| Error rate | 100% (3/3 invocations fail every day) |
| Max duration | 900,000 ms (15 min Lambda timeout) |
| Concurrency | 1 per invocation |
| Schedule | Daily, 3 invocations between ~05:00--05:33 UTC |
| Broken since | At least 2026-07-09 (7 consecutive days) |
| Date | Errors | Invocations | Duration (avg) | Error rate |
|---|---|---|---|---|
| 2026-07-09 04:00 | 3 | 3 | 900,000 ms | 100% |
| 2026-07-10 04:00 | 3 | 3 | 900,000 ms | 100% |
| 2026-07-11 04:00 | 3 | 3 | 900,000 ms | 100% |
| 2026-07-12 04:00 | 3 | 3 | 900,000 ms | 100% |
| 2026-07-13 04:00 | 3 | 3 | 900,000 ms | 100% |
| 2026-07-14 04:00 | 3 | 3 | 900,000 ms | 100% |
| 2026-07-15 05:00 | 3 | 3 | 900,000 ms | 100% |
The alert fires daily but auto-resolves within ~30 minutes because no further invocations enter the evaluation window.
Sporadic single-invocation failures -- not severe enough to trigger the alert.
| Metric | Value |
|---|---|
| 7-day error rate | 1.8% (12 errors / 683 invocations) |
| Schedule | Every 15 minutes |
| Pattern | Isolated single failures, no sustained bursts |
| Time (UTC) | Errors | Invocations |
|---|---|---|
| Jul 08, 17:00 | 1 | 5 |
| Jul 10, 20:00 | 1 | 5 |
| Jul 12, 14:00 | 2 | 5 |
| Jul 12, 15:00 | 1 | 5 |
| Jul 13, 02:00 | 2 | 6 |
| Jul 13, 22:00 | 2 | 6 |
| Jul 14, 05:00 | 1 | 5 |
| Jul 14, 21:00 | 1 | 5 |
| Jul 15, 01:00 | 1 | 5 |
All remaining functions had zero errors and zero throttles in the last 24 hours:
device-telemetry-firehose-transformer, event-data-exporter, event-ingester, event-stream-transformer, ice-events-exporter, kvs-streams-count-alarm, request-path-redirect, thing-state-notifier, transit-gateway-state-collector, underwriting-engine-existing-policy-cache, websocket-authoriser
| Time (UTC) | Event |
|---|---|
| Jul 09, 04:00 | p2p-server-logs-lambda first observed failing (3/3 timeouts) |
| Jul 10--14, ~04:00 | Same pattern repeats daily |
| Jul 15, 05:00 | Today's 1st invocation -- timeout at 900s |
| Jul 15, 05:16 | 2nd invocation -- timeout at 900s |
| Jul 15, 05:33 | 3rd invocation -- timeout at 900s |
| Jul 15, ~05:36 | Alert auto-resolves (no more invocations in window) |
| Jul 15, ~08:15 | Investigation -- alert inactive, all instances Normal |
| Check | Finding |
|---|---|
| Other Sky Protect alerts firing | AWS Costs (6 instances, since Jul 11--13), Synthetic check failure for Admin Portal (since Apr 28) -- unrelated |
| Lambda throttles | Zero across all functions |
| Concurrent executions | Max 1 for p2p-server-logs-lambda -- not a concurrency issue |
p2p-server-logs-lambda is hitting the 15-minute Lambda timeout on every invocation.
-
Downstream dependency unresponsive (most likely) -- the function connects to a P2P server log source that is unreachable or hanging, causing the function to block until timeout. Consistent 900s duration across all invocations supports a hard timeout rather than variable processing.
-
Data volume exceeds processing capacity -- the volume of P2P server logs has grown beyond what can be processed within the 15-minute Lambda limit. Less likely given the exact-timeout pattern (would expect some variance).
-
Code or dependency change -- a deployment broke the function. Would need deployment history to confirm, but the 7-day persistence suggests this isn't being actively worked.
- Pull CloudWatch Logs for
p2p-server-logs-lambdaaround 05:00 UTC today to see where execution hangs. - Check the function's event source (likely EventBridge/CloudWatch Events) for the schedule and input payload.
- Check deployment history to see if anything changed around Jul 8--9.
- p2p-server-logs-lambda: Whatever P2P server log processing this function performs has been completely non-functional for 7+ days. Impact depends on downstream consumers of the output.
- underwriting-engine-flood-alert: Sporadic failures at 1.8% -- low impact, 98.2% success rate maintained.
| Priority | Action |
|---|---|
| P1 | Investigate p2p-server-logs-lambda CloudWatch Logs to determine where it hangs |
| P1 | If the data source is dead, disable the scheduled trigger to stop daily noise |
| P2 | Fix the root cause (reconnect to data source, increase timeout, or refactor to handle larger volumes) |
| P3 | Add a dedicated alert or dashboard for this function's daily success/failure |
| P3 | Monitor underwriting-engine-flood-alert error trend -- investigate if rate increases |
| # | Query | Source | Window |
|---|---|---|---|
| 1 | Alert rule definition | GET /api/v1/provisioning/alert-rules/ffihc6bh7fawwe |
-- |
| 2 | All alert states | GET /api/prometheus/grafana/api/v1/rules |
-- |
| 3 | Lambda Errors (SQL, all functions) | CloudWatch AWS/Lambda |
6h, 5m period |
| 4 | Lambda Errors per function | CloudWatch AWS/Lambda |
24h, 1m period |
| 5 | Lambda Invocations per function | CloudWatch AWS/Lambda |
6h, 5m period |
| 6 | Lambda Errors + Invocations (p2p) | CloudWatch AWS/Lambda |
7d, 1h period |
| 7 | Lambda Errors + Invocations (flood-alert) | CloudWatch AWS/Lambda |
7d, 1h period |
| 8 | Lambda Duration (p2p) | CloudWatch AWS/Lambda |
12h, 1m period |
| 9 | Lambda ConcurrentExecutions (p2p) | CloudWatch AWS/Lambda |
12h, 1m period |
| 10 | Lambda Throttles (4 functions) | CloudWatch AWS/Lambda |
24h, 5m period |
| 11 | Sky Protect firing alerts | Prometheus rules API | current |