2026-04-13 `libxmtp` Modularity Exploration Status Check

2026-04-13-libxmtp-modularity-exploration-status-check.md

libxmtp Modularity Exploration: Status and Findings

Context

Over the weekend I ran a structured exploration into whether libxmtp's internals, specifically mls_sync and the groups module, could be made meaningfully more modular without forcing a broad public crate split or breaking downstream consumers. These are the two largest and most complex modules in the codebase, and their size has been making it harder to reason about behavior, isolate bugs, and onboard new contributors.

I used my team of agents (Codex & Claude) to do the heavy lifting: planning milestones, executing the refactors, writing tests, and validating the results. The goal was to answer the modularity question with code, not architecture diagrams. Prove or disprove specific internal boundaries, then make a grounded recommendation. The work followed an incremental milestone plan (M0–M8), where each milestone produced reviewable commits and a retro that informed the next step.

By the numbers

• 107 commits across 121 files (~13.9k insertions / ~9.2k deletions)
• 30 test-specific commits adding +3,215 net lines of test coverage
• 19 replay/idempotence proofs, runtime-backed tests against the local XMTP stack, not just compile checks
• 2 real regressions caught and fixed during the process (details below)

Where the seams are

The exploration identified which internal boundaries are real and which are aspirational. Here's where I cut and how things shaped up.

mls_sync: from monolith to modules

The single 4,219-line mls_sync.rs file was decomposed into 14 focused modules:

Module	Lines	Responsibility
processing.rs	1,273	Message processing spine and durable-apply flows
publish.rs	715	Intent publishing and send finalization
update_group_membership.rs	460	Membership update orchestration
persistence.rs	452	Transaction bodies and bookkeeping
removals.rs	416	Removal handling and pending-admin flows
tests.rs	351	Dedicated sync-level test coverage
membership.rs	302	Membership refresh and key package lookups
resolution.rs	272	Intent resolution query helpers
welcomes.rs	271	Welcome handling and sync filtering
receive.rs	183	Receive loop and cursor management
types.rs	176	Shared types and enums
helpers.rs	138	Shared utility functions
events.rs	131	Deferred event emission
orchestration.rs	98	Top-level sync orchestration

To be clear about where this stands: most of these modules are still impl MlsGroup method blocks in separate files. The file reorganization makes the seams visible and navigable, but it's not yet a true functional decomposition with independent interface boundaries. The agents did extract ~15 standalone functions behind narrower capability traits (ForkDetectionContext, PauseResolutionContext, MembershipUpdateContext, CommitValidationContext, and others) that define minimal capability surfaces rather than requiring the full XmtpSharedContext. Those are real interface boundaries, but they're at the internal call boundary, not the public API surface.

The goal for the next phase is the real decomposition: pulling functional components out of MlsGroup into independently testable units with their own interface contracts. The file reorganization was the prerequisite. You can't decompose what you can't see, and now the seams are visible enough to do that work with confidence.

groups/mod.rs: 89% reduction

The groups shell went from 2,842 lines to 318, an 89% reduction. The bulk was extracted into focused siblings:

• creation.rs — group and DM creation flows
• message_ops.rs — send, prepare, and process message paths
• metadata_ops.rs — metadata and consent helpers
• membership_ops.rs — add, remove, and update members
• state_ops.rs — group state queries and admin checks
• proposal_ops.rs — proposal support helpers
• group_config.rs — configuration and permissions
• message_settings.rs — disappearing message settings

Other subsystems clarified

• Device sync was decomposed from a single large worker file into lifecycle, inbound, outbound, archive_receive, catalog, and sync_group modules with clear boundaries.
• Subscriptions split into facade, events, stream_all, stream_conversations, and stream_messages, each with co-located tests.

Seams that proved weaker than hoped

• A standalone xmtp_messages crate: not enough separation from group state
• Consent as its own subsystem: too interleaved with group operations
• A broad multi-crate split: the code doesn't support it yet and it would be premature to head down that path. That said, the internal modularity work makes a future split more tractable than before. The seams are now visible, named, and tested, which is exactly the groundwork you'd want in place before attempting it.

Testing and correctness

This work meaningfully strengthened the testing story in xmtp_mls. Beyond the structural refactoring, the agents added runtime-backed proofs for behavior that was previously only assumed correct.

Replay and idempotence coverage

19 new tests prove that critical sync paths are replay-stable, meaning re-processing the same messages or intents produces identical state without duplicates, corruption, or side effects. These run against the local XMTP stack, not mocks:

• Own-message replay: replaying self-authored application envelopes preserves message IDs, keeps DeliveryStatus::Published, leaves commit logs byte-for-byte unchanged, and holds cursor snapshots fixed
• External message replay: external application, membership, metadata, and removal messages all replay cleanly
• Proposal/commit replay: staged commit finalization is idempotent across reloads
• Cursor and pause behavior: protocol-version pauses, retryable aborts, and invalid proposals all consume cursors correctly on replay

Regressions caught and fixed

Two real bugs were surfaced during the exploration and promoted to standing coverage:

1. Retryable send poisoning. A transient send failure on a published intent was permanently poisoning that intent, preventing all future retries. Fixed and covered by test_retryable_commit_log_recovery.
2. Late same-inbox welcome recovery. When a second installation from the same inbox joined a group after the first was removed, the pending-remove state wasn't inherited correctly. Fixed and covered by test_late_install_self_removal_recovery.

Subscription and event coverage

New tests cover consent-transition stream behavior, DM deduplication, device-sync consent propagation, and paused-stream recovery. These are areas that previously had no direct coverage.

Binding compatibility

The compatibility-first approach held throughout. Internal restructuring did not require downstream API changes, and runtime smoke confirmed it:

• Node bindings: vitest passed on a send/list message flow
• Mobile bindings: DM and group creation tests passed against xmtpv3
• WASM: compiles clean against wasm32-unknown-unknown. One pre-existing test failure in test_welcome_cursor (unrelated to this work), but the build and the rest of the test suite pass.

Recommendation

The conclusion from this work: compatibility-first internal modularity is the right direction for libxmtp. The code supports a better-organized monolith with opportunistic extraction where seams prove real, not a broad package graph rewrite.

Concretely:

• Keep the internal modularity wins as-is. The decomposition makes each module easier to reason about, test, and modify independently.
• Keep xmtp_mls as the stable facade. Bindings and consumers shouldn't need to know about internal module boundaries.
• Extract subsystems only when needed. Device sync and subscriptions have real boundaries, but extraction should be driven by a concrete consumer need, not done preemptively.
• Hold off on aggressive crate splitting for now. The deepest remaining coupling (in processing.rs / persistence.rs) appears genuinely protocol/data-flow required, not just packaging convenience. But the internal module boundaries are now clean enough that a future split would be a much smaller leap than before this work.