Skip to content

Instantly share code, notes, and snippets.

@jwaiswa7

jwaiswa7/system_design.md

Last active December 9, 2024 14:35
Show Gist options
  • Save jwaiswa7/0363bebd1e240d5188dc1b433653ee71 to your computer and use it in GitHub Desktop.
Save jwaiswa7/0363bebd1e240d5188dc1b433653ee71 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
system_design.md

I scrapping Application

Services

  • Crawl websites for information about jobs
  • Use AI to extract specific information based on the schema
  • Save the information to a database
  • Information about specific jobs can then be accessed via a Rails application

Steps to get data from a website

  • Get the URL and pass it to the AI
  • AI passes the data to a function, the function will scrap the website
  • Once the function scraps the website, AI will find the careers/job page link
  • AI passes the job page link to another function that will scrape information on the link
  • AI passes the information and will build a list of links with different Jobs returned
  • AI will pass the link back to a function to scrap information about the job like skills, etc
  • The function will return the data and AI will use it to process the job details and in a form that will be saved in the database.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment