newsArchiveUpdatedPlan.md

News Archive - Project Spec

Background

News websites publish new stories daily, but older content quickly becomes hard to find — buried behind pagination, missing from site search, or removed entirely. This project solves that by continuously collecting articles from RSS feeds and making them searchable through a simple web interface, creating a persistent archive that grows over time.

Target users: Anyone who wants to search or browse archived news stories across multiple sources.

Concept

An RSS-based news archiving and search application with two parts:

1. Ingestion engine — a background worker that reads RSS feeds on a schedule, fetches the summary content provided by each feed, and stores new items in a database. Only the content provided by the RSS feed itself is stored; full article scraping from the source website is out of scope.
2. Search interface — a public-facing web UI where users search for stories by keyword and browse results.
3. Admin interface — a protected web UI for monitoring the system and managing feeds.

Tech Stack

Layer	Technology
Frontend	Next.js (App Router), HTML/CSS/JS
Backend API	Next.js API routes
Feed worker	Standalone Node.js process (runs on a cron schedule)
Database	MySQL (utf8mb4 encoding, InnoDB engine)
Testing	Red/green TDD — unit tests for the feed parser and DB layer; integration tests for API routes

UI conventions: Light-mode theme; responsive design that works on desktop and mobile without requiring a keyboard (touch-friendly).

Architecture note: Next.js handles both the UI and the API layer. The feed ingestion worker is a separate Node.js script invoked on a schedule independently of the Next.js server. For local machine deployment on Windows, use Windows Task Scheduler to run the worker hourly; on macOS/Linux, use a crontab entry.

Database Schema

All tables use utf8mb4 encoding. Index feed_id and pub_date on the items table for query performance. Add a FULLTEXT index on items(title, description) for keyword search.

feeds table

Column	Type	Notes
id	INT AUTO_INCREMENT	Primary key
title	VARCHAR(255)	Feed name
url	VARCHAR(512)	URL to the RSS XML file
site_url	VARCHAR(512)	Feed's parent website homepage
description	TEXT	Feed summary
last_fetched	DATETIME	Timestamp of last successful poll

items table

Column	Type	Notes
id	INT AUTO_INCREMENT	Primary key
feed_id	INT	Foreign key → feeds.id
guid	VARCHAR(512)	Unique identifier from the feed; used to detect duplicates. If the feed provides no guid, use a hash of the item URL.
title	VARCHAR(512)	Article headline
link	VARCHAR(512)	URL to the original article
description	TEXT	Summary or snippet from the feed
content	LONGTEXT	Full content if provided by the feed (e.g., <content:encoded>); otherwise null
pub_date	DATETIME	Original publication timestamp

admin table

Single-row table holding the one admin account. No user registration or multi-user support.

Column	Type	Notes
id	INT AUTO_INCREMENT	Primary key (always 1)
username	VARCHAR(255)	Admin username
password_hash	VARCHAR(255)	Bcrypt hash

Development Roadmap

---

Milestone 1: Feed ingestion and admin interface

Goal: The system can collect RSS feed items automatically and an admin can monitor the system.

Feed ingestion worker

• On first run, the worker reads a list of RSS feeds from an OPML file and populates the feeds table.
• The worker runs on an hourly cron schedule (e.g., node worker/fetchFeeds.js called by the OS cron or a process manager).
• For each feed, the worker fetches the RSS XML, parses each item, and inserts it into items only if its guid does not already exist in the database (no duplicates).
• Worker logs a summary on each run: feeds checked, new items added, errors encountered.

Admin interface

• Protected by username/password login (session-based auth) using the single admin account.
• Admin can: update the admin password, and view a dashboard showing total feeds tracked, total items stored, and the timestamp of the last successful poll per feed.

Done when:

• Running the worker once against a sample OPML file correctly populates feeds and items.
• Running the worker a second time adds no duplicate items.
• Admin can log in, view the dashboard report, and update the admin password.
• Hourly worker invocation is configured and documented for Windows Task Scheduler.

---

Milestone 2: Public search interface

Goal: Users can find archived stories by searching keywords.

Search UI

• Publicly accessible — no login required.
• A search bar accepts one or more keywords and queries the title and description fields of the items table using MySQL FULLTEXT search (MATCH ... AGAINST).
• Results are displayed as a list of article cards, each showing: headline (linked to the original article), source feed name, publication date, and description snippet.
• Results are sorted by pub_date descending (most recent first) by default.
• Results are paginated (e.g., 20 items per page) with next/previous controls.
• Users can filter results by date range (from / to) and by feed source.

Done when:

• A keyword search returns relevant results from the database.
• Results display headline, source, date, and snippet for each item.
• Pagination works correctly at result set boundaries.
• Date and source filters correctly narrow results.
• UI is usable on a mobile viewport without horizontal scroll.

---

Milestone 3: Export and reporting

Goal: Users can export search results for offline use.

• Search results can be exported as an HTML file or plain-text file.
• The exported file includes all matching results (not just the current page).
• Export is triggered from a button on the search results page.

Done when:

• HTML export renders a readable, self-contained file of all results for the current query.
• Plain-text export lists headline, source, date, and link for each result.
• Export respects active search filters (date range, source).

---

Milestone 4: Feed management UI

Goal: Admin can manage RSS feeds through the UI without editing the OPML file.

• Admin interface gains a feed management section showing all feeds currently in the database.
• Admin can add a new feed by entering a URL (the worker fetches and validates the RSS before saving).
• Admin can remove a feed (and optionally its associated items) from the database.
• The OPML file remains supported as the initial seed mechanism, but is no longer the only way to manage feeds.

Done when:

• Admin can add a new feed URL via the UI and the next worker run collects its items.
• Admin can remove a feed; it no longer appears in search results.
• Feed list in the UI reflects the current state of the feeds table.