Skip to content

Instantly share code, notes, and snippets.

@caius

caius/feed server.md

Last active December 12, 2017 09:03
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save caius/e21470d24920e6425003149e3741c5c8 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save caius/e21470d24920e6425003149e3741c5c8 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
feed server.md

Feed Server

Fetcher

  • Reads file containing feed URLs on each line
  • Has a feedstore directory it places data into
  • feedstore is structured
    • {{feed}}.json - file, metadata about the feed, name consistently generated from URL
    • {{feed}} - folder, name consistently generated from URL
      • {{guid}}.json - file, one per item, JSON format, contains item details (optionally .json.gz)
  • Basic operation is 0. Feed URL is scraped
    1. Datastore has folder created for this feed (if not already there)
    2. New file created per feed item, named after item guid (filesystem safe?)
    3. Feed item files are JSON
      • title
      • link
      • published at date
      • updated at date
      • actual guid (filename is consistently altered guid)
      • description (body of item)
    4. Feed folder is touched at the end of handling it to timestamp last checked date

Server

  • Exposes [Fever API][] for readers to connect via
  • Configured via JSON
    • Given the datastore from Fetcher as datasource
  • Watches filesystem for changes, updating in-memory map of posts/etc
  • Reads in feeds from folders on disk, posts from files on disk
  • As posts are marked read, stores state somewhere (sqlite? json formatted cache file?) - separate data to feedstore
  • Multiple users handled somehow? Should probably be independant of feedstore, server controls who subscribes to what feeds, fetcher just fetches
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment