Skip to content

Instantly share code, notes, and snippets.

Search indexing performance — findings & fix plan

TL;DR

Bonfire enqueues each indexable object to Meilisearch as a single-document HTTP PUT, synchronously, in the publish/delete Epic. Meilisearch (LMDB-backed) performs one full write transaction per task, so every single-doc submission costs roughly the same as a 100-doc batch would. In production this sustains ~12 single-doc commits/second on the prod_closed index (2.4M docs, fed largely by federation), consuming ~200% CPU and ~1.5 TB of cumulative disk writes on the search container. The host shows load average 6+ while CPU is 80% idle — classic bursty-contention symptom caused by the search container starving neighbours for I/O and short CPU bursts.

The fix is to coalesce single-doc submissions into batched PUTs, fronted by a durable Oban queue so nothing is lost on a restart. Expected impact on the search container: roughly 50–100× reduction in CPU and disk writes for the same indexing throughput.


@ivanminutillo
ivanminutillo / .md
Created May 11, 2026 15:09
messages timeout

Bonfire boundary EXISTS subquery is the dominant DB bottleneck under federation load

TL;DR

  • Every page that checks "can user see this?" runs the same complex boundary EXISTS subquery (bonfire_boundaries_summary view). It's a 6-way cross-join + custom aggregate re-evaluated on every check.
  • Under federation pressure, this subquery's cost compounds: federation workers hammer ap_object and a per-peer "silencing circle" lookup, saturating the DB pool. The boundary EXISTS queues behind them, normal pages slow, /messages specifically times out at 20 s (statement_timeout).
  • /messages is the canary, not the patient. The same EXISTS runs on every feed, profile, thread, MRF filter, etc.

Original symptom (from production logs)

@ivanminutillo
ivanminutillo / .js
Created February 4, 2026 09:29
k6_liveview.js
// Bonfire LiveView Load Test for k6
//
// Tests: "Can the server handle N concurrent LiveView connections to feed/explore?"
//
// Usage:
// 1. Login to Bonfire in browser, copy session cookie from DevTools
// 2. Run: k6 run -e COOKIE="your_cookie" -e HOST="localhost:4000" benchmarks/load_test/k6_liveview.js
//
// Options:
// HOST - Target host (default: localhost:4000)
:21:22.228738383Z
?2026-01-15T11:21:22.229191006Z 11:21:22.229 [debug] SQL query: ok db=2ms source="bonfire_data_identity_user" repo=Elixir.Bonfire.Common.Repo
2026-01-15T11:21:22.229198626Z SELECT b0."id", b4."id", b4."name", b4."summary", b4."website", b4."location", b4."icon_id", b4."image_id", b5."id", b5."creator_id", b5."path", b5."file", b5."size", b5."media_type", b5."metadata", b5."deleted_at", b3."id", b3."username", b3."username_hash", b3."outbox_id", b3."inbox_id", b3."notifications_id", b6."id", b6."json", b2."id", b7."id", b7."json", b8."id", b8."is_instance_admin", b8."user_id", b9."id", b9."label" FROM "bonfire_data_identity_user" AS b0 LEFT OUTER JOIN "bonfire_data_identity_accounted" AS b1 ON (b1."account_id" = '01JSC3XEX8Z9XW642ACRQK9Y4T') AND (b1."id" = b0."id") LEFT OUTER JOIN "bonfire_data_identity_account" AS b2 ON b1."account_id" = b2."id" LEFT OUTER JOIN "bonfire_data_identity_character" AS b3 ON b3."id" = b0."id" LEFT OUTER JOIN "bonfire_data_social_profile" AS b4 ON b4."id" = b0.
[debug] LiveHandler: handle_event "Bonfire.Social.Boosts:boost" via Elixir.Bonfire.UI.Social.Activity.ActionsLive @ bonfire_ui_common/lib/live_handlers.ex:218 @ Bonfire.UI.Common.LiveHandlers.handle_event/5
[debug] LiveHandler: handle_event {"Bonfire.Social.Boosts", "boost"} via delegation @ bonfire_ui_common/lib/live_handlers.ex:218 @ Bonfire.UI.Common.LiveHandlers.handle_event/5
[debug] getting from cache: Application.get_application(Bonfire.Social.Boosts.LiveHandler) @ bonfire_common/lib/cache/cache.ex:257 @ Bonfire.Common.Cache.do_maybe_apply_cached/4
[debug] Get config for app bonfire_ui_reactions: [:modularity] @ bonfire_common/lib/config_settings/config.ex:82 @ Bonfire.Common.Config.__get__/3
[debug] :bonfire_ui_reactions: false @ bonfire_common/lib/modularity/extend.ex:341 @ Bonfire.Common.Extend.is_disabled?/2
[debug] getting from cache: Application.get_application(Bonfire.Social.Boosts.LiveHandler) @ bonfire_common/lib/cache/cache.ex:257 @ Bonfire.Common.Cache.do_maybe_apply_cached/4
[debug] Get con
[debug] LiveHandler: handle_event "Bonfire.Social.Likes:like" via Elixir.Bonfire.UI.Social.Activity.ActionsLive @ bonfire_ui_common/lib/live_handlers.ex:218 @ Bonfire.UI.Common.LiveHandlers.handle_event/5
[debug] LiveHandler: handle_event {"Bonfire.Social.Likes", "like"} via delegation @ bonfire_ui_common/lib/live_handlers.ex:218 @ Bonfire.UI.Common.LiveHandlers.handle_event/5
[debug] getting from cache: Application.get_application(Bonfire.Social.Likes.LiveHandler) @ bonfire_common/lib/cache/cache.ex:257 @ Bonfire.Common.Cache.do_maybe_apply_cached/4
[debug] Get config for app bonfire_ui_reactions: [:modularity] @ bonfire_common/lib/config_settings/config.ex:82 @ Bonfire.Common.Config.__get__/3
[debug] :bonfire_ui_reactions: false @ bonfire_common/lib/modularity/extend.ex:341 @ Bonfire.Common.Extend.is_disabled?/2
[debug] getting from cache: Application.get_application(Bonfire.Social.Likes.LiveHandler) @ bonfire_common/lib/cache/cache.ex:257 @ Bonfire.Common.Cache.do_maybe_apply_cached/4
[debug] Get config fo
[debug] Get config for app bonfire: [:repo_module] @ bonfire_common/lib/config_settings/config.ex:82 @ Bonfire.Common.Config.__get__/3
[debug] Get config for app bonfire: [:env] @ bonfire_common/lib/config_settings/config.ex:82 @ Bonfire.Common.Config.__get__/3
[debug] Get config for app bonfire: [:endpoint_module] @ bonfire_common/lib/config_settings/config.ex:82 @ Bonfire.Common.Config.__get__/3
[debug] Get config for app bonfire: [:env] @ bonfire_common/lib/config_settings/config.ex:82 @ Bonfire.Common.Config.__get__/3
[info] GET /feed/remote
[info] Request from ::ffff:127.0.0.1
[debug] Get config for app sentry: [:modularity] @ bonfire_common/lib/config_settings/config.ex:82 @ Bonfire.Common.Config.__get__/3
[debug] :sentry: true @ bonfire_common/lib/modularity/extend.ex:341 @ Bonfire.Common.Extend.is_disabled?/2
[debug] Processing with Bonfire.UI.Social.FeedsLive.__live__/0
Parameters: %{"tab" => "remote"}
[debug] Get config for app bonfire: [:repo_module] @ bonfire_common/lib/config_settings/config.ex:82 @ Bonfire.Common.Config.__get__/3
[debug] Get config for app bonfire: [:env] @ bonfire_common/lib/config_settings/config.ex:82 @ Bonfire.Common.Config.__get__/3
[debug] Get config for app bonfire: [:endpoint_module] @ bonfire_common/lib/config_settings/config.ex:82 @ Bonfire.Common.Config.__get__/3
[debug] Get config for app bonfire: [:env] @ bonfire_common/lib/config_settings/config.ex:82 @ Bonfire.Common.Config.__get__/3
[info] GET /@Sporer_LLC
[info] Request from ::ffff:127.0.0.1
[debug] Get config for app sentry: [:modularity] @ bonfire_common/lib/config_settings/config.ex:82 @ Bonfire.Common.Config.__get__/3
[debug] :sentry: true @ bonfire_common/lib/modularity/extend.ex:341 @ Bonfire.Common.Extend.is_disabled?/2
[debug] Processing with Bonfire.UI.Me.ProfileLive.__live__/0
Parameters: %{"username" => "Sporer_LLC"}
iex(localenv@127.0.0.1)1>
nil
[info] [source: "oban", duration: 4712, event: "plugin:stop", jobs: [], plugin: "Oban.Plugins.Cron"]
[debug] LiveHandler: handle_event "Bonfire.Social.Graph.Follows:follow" via Elixir.Bonfire.UI.Social.Graph.FollowButtonLive @ bonfire_ui_common/lib/live_handlers.ex:218 @ Bonfire.UI.Common.LiveHandlers.handle_event/5
[debug] LiveHandler: handle_event {"Bonfire.Social.Graph.Follows", "follow"} via delegation @ bonfire_ui_common/lib/live_handlers.ex:218 @ Bonfire.UI.Common.LiveHandlers.handle_event/5
[debug] getting from cache: Application.get_application(h:HYMGwrIbieJTu7nkloWfeg) @ bonfire_common/lib/cache/cache.ex:257 @ Bonfire.Common.Cache.do_maybe_apply_cached/4
[debug] Get config for app bonfire_ui_social_graph: [:modularity] @ bonfire_common/lib/config_settings/config.ex:82 @ Bonfire.Common.Config.__get__/3
[debug] :bonfire_ui_social_graph: false @ bonfire_common/lib/modularity/extend.ex:341 @ Bonfire.Common.Extend.is_disabled?/2
[debug] getting from cache: Application.get_ap
[debug] LiveHandler: handle_event "Bonfire.Social.Graph.Follows:unfollow" via Elixir.Bonfire.UI.Social.Graph.FollowButtonLive @ bonfire_ui_common/lib/live_handlers.ex:218 @ Bonfire.UI.Common.LiveHandlers.handle_event/5
[debug] LiveHandler: handle_event {"Bonfire.Social.Graph.Follows", "unfollow"} via delegation @ bonfire_ui_common/lib/live_handlers.ex:218 @ Bonfire.UI.Common.LiveHandlers.handle_event/5
[debug] getting from cache: Application.get_application(h:HYMGwrIbieJTu7nkloWfeg) @ bonfire_common/lib/cache/cache.ex:257 @ Bonfire.Common.Cache.do_maybe_apply_cached/4
[debug] Get config for app bonfire_ui_social_graph: [:modularity] @ bonfire_common/lib/config_settings/config.ex:82 @ Bonfire.Common.Config.__get__/3
[debug] :bonfire_ui_social_graph: false @ bonfire_common/lib/modularity/extend.ex:341 @ Bonfire.Common.Extend.is_disabled?/2
[debug] getting from cache: Application.get_application(h:HYMGwrIbieJTu7nkloWfeg) @ bonfire_common/lib/cache/cache.ex:257 @ Bonfire.Common.Cache.do_maybe_apply_cached/4
[debu