MVP-808: Feed Preload & Pagination Fix

20260524-mvp-808-feed-preload-pagination.md

MVP-808: Feed Preload & Pagination Fix

Goal: Faster perceived startup, fix broken Trending/Recent pagination, and add feed-cap recycling so the user never hits a dead end.

---

Root-cause audit (current state)

Why Trending/Recent never scroll past page 1

_execute_feed_query() (feeds_http.py:475) computes offset = int(q.since) correctly but never passes it to TrendingAdapter.fetch() or RecentAdapter.fetch().

# Current — offset is computed but ignored for these two paths
if q.feed_id == "trending" or q.kind == "trending":
    _merge(TrendingAdapter().fetch(q.limit, session, ...))   # no offset!
    next_cursor = str(offset + q.limit) if items else None
    return PaginatedList(items=items, next=next_cursor)

TrendingAdapter._fetch_most_viewed() also caps the DB candidate pool at limit * 2, so even if offset were passed, fetching page 2 would re-score the same candidates. The next cursor advances ("10" → "20" → …) but every response is identical — the top-N trending posts.

RecentAdapter._fetch_newest() has the same bug: it slices post_ids[:limit] from the 1 000-entry cache regardless of the requested offset.

Why initial load feels slow

Both Main Feed and Trending are called for the first render. Main Feed fetches a 4× candidate pool (SGC mixing), hydrates images/tags/trees, and only then returns to the client. The cold-start DB hit is noticeable. Triggering Trending in parallel (or bundling both in one request) removes one RTT.

---

Proposed changes

1 — Fix Trending pagination

File: app/sherpahealthy/posts/feed_adapters.py

Change _fetch_most_viewed() signature to accept offset:

def _fetch_most_viewed(self, limit, session, exclude_ids=None,
                       feed_session=None, offset=0) -> List[PostModel]:

Expand the DB candidate pool to cover the full range up to the requested page:

# Fetch enough candidates to cover the requested page.
# Double for headroom after Python-side scoring/filtering.
db_limit = max((offset + limit) * 2, 1000)
stmt = text("""
    SELECT pv.post_id, COUNT(*) AS view_count, MAX(p.created_at) AS created_at
    FROM post_views pv
    JOIN posts p ON p.id = pv.post_id
    WHERE pv.viewed_at > :cutoff
      AND p.visible = TRUE
    GROUP BY pv.post_id
    LIMIT :limit
""")
rows = session.execute(stmt, {"cutoff": cutoff, "limit": db_limit}).fetchall()

Apply Python offset after scoring:

# existing sort:
post_ids = sorted(base_scores, key=lambda pid: base_scores[pid], reverse=True)
# NEW — paginate after Python-side scoring
post_ids = post_ids[offset : offset + limit * 2]   # keep headroom for filters

Propagate offset from the dispatcher:

# feeds_http.py — trending path
if q.feed_id == "trending" or q.kind == "trending":
    _merge(TrendingAdapter().fetch(q.limit, session,
                                   feed_session=feed_session, offset=offset))

2 — Fix Recent pagination

File: app/sherpahealthy/posts/feed_adapters.py

_fetch_newest() already has a 1 000-entry cache — just slice it correctly:

def _fetch_newest(self, limit, session, exclude_ids=None,
                  feed_session=None, offset=0) -> List[PostModel]:
    ...
    post_ids: List[int] = list(_trending_new_cache["ids"])
    if exclude_ids:
        post_ids = [pid for pid in post_ids if pid not in exclude_ids]

    # NEW — paginate cache slice
    page_ids = post_ids[offset : offset + limit]
    page_ids = _apply_session_filters(page_ids, feed_session, session)
    return _batch_fetch_posts(session, page_ids, self.source_label)

Propagate offset from dispatcher (same pattern as Trending above).

3 — New bundle endpoint

Add GET /v1/posts/bundle that returns the Main Feed initial page AND the Trending initial page in a single round trip. This enables the "4 posts render → background fetch Main+Trending simultaneously" pattern without two parallel HTTP connections.

File: app/sherpahealthy/posts/feeds_http.py

GET /v1/posts/bundle
Headers: X-Profile-Id: <profile_id>        (optional)
Query:   main_limit   int  default=10
         trending_limit int default=10
         main_since   str  default=None     (cursor for main feed)
         trending_since str default=None

Response 200:
{
  "main":     { "items": [...PostModel], "next": "10" },
  "trending": { "items": [...PostModel], "next": "10" }
}

Implementation: call _execute_feed_query() twice (main + trending) and return both results. Because _execute_feed_query opens its own session it is safe to call in sequence. No new adapter logic needed.

4 — Feed caps and recycling

Caps are client-enforced — the backend does not reset offsets. When the client detects next == null it resets since to null (start over). The backend naturally returns next=null when the DB result is empty.

Feed	Client cap guidance
Main Feed	Reset after since reaches 10 000
Trending	Reset after since reaches 1 000 (matches cache size)
Recent	Reset after since reaches 1 000 (matches cache size)

"Fresh first, then recycle": once reset, the adapter fetches from offset 0 again. New posts added since the previous session appear at the top naturally. Per-session seen-post deduplication is out of scope (would require storing up to 10 K IDs per session in DB or Redis).

---

API contract changes

Endpoint	Change
GET /v1/posts/	No contract change. since pagination already works correctly.
GET /v1/posts/by-feed/trending	since now actually paginates (was broken). Existing callers that passed since will now get the correct page.
GET /v1/posts/by-feed/recent	Same — since now paginates correctly.
GET /v1/posts/bundle	New endpoint (additive, non-breaking).

---

Mobile / frontend guidance (separate ticket)

These are not backend changes but are needed for the full MVP-808 experience:

Step	Action
App open	Call GET /v1/posts/bundle?main_limit=4&trending_limit=0 for the 4-post initial splash
After first render	Background call GET /v1/posts/bundle?main_since=4&main_limit=10&trending_limit=10
Prefetch trigger	Move from "user at post #10" → "user at post #3"
Feed exhausted (next=null)	Reset since=null and restart

Alternatively the client can make two parallel requests instead of using /bundle — the backend supports both patterns.

---

Implementation order

1. Fix Trending pagination (highest user impact — "never gets more than 10")
2. Fix Recent pagination (same fix pattern)
3. Add /bundle endpoint (enables faster startup without client-side changes)
4. Mobile/FE prefetch strategy (separate mobile ticket, depends on #3)

---

Testing checklist

• GET /v1/posts/by-feed/trending?since=0&limit=10 and ?since=10&limit=10 return different posts (regression test for the broken-pagination bug)
• GET /v1/posts/by-feed/recent?since=0&limit=10 and ?since=10&limit=10 return different posts
• GET /v1/posts/bundle returns both feeds in a single response with correct cursors
• Main Feed pagination still works correctly (no regression)
• TrendingAdapter with offset >= 1000 returns empty → next=null
• RecentAdapter with offset >= 1000 returns empty → next=null