Skip to content

Instantly share code, notes, and snippets.

@thoroc
thoroc / README.md
Last active September 6, 2026 17:52
How I scraped 2.1 million jobs (including 5,335 data science jobs)

How I scraped 2.1 million jobs (including 5,335 data science jobs)

original: https://www.reddit.com/r/datascience/comments/1iynyco/how_i_scraped_21_million_jobs_including_5335_data/

Background

During my PhD in Data Science at Stanford, I got sick and tired of ghost jobs & 3rd party offshore agencies on LinkedIn & Indeed. So I wrote a script that fetches jobs from 30k+ company websites' career pages and uses GPT4o-mini to extract relevant information (ex salary, remote, etc.) from job descriptions. You can use it here: (HiringCafe). Here is a filter for Data science jobs