During my PhD in Data Science at Stanford, I got sick and tired of ghost jobs & 3rd party offshore agencies on LinkedIn & Indeed. So I wrote a script that fetches jobs from 30k+ company websites' career pages and uses GPT4o-mini to extract relevant information (ex salary, remote, etc.) from job descriptions. You can use it here: (HiringCafe). Here is a filter for Data science jobs (5,335 and counting). I scrape every company 3x/day, so the results stay fresh if you check back the next day.
You can follow my progress on r/hiringcafe
-
I identified company career pages with active job listings. I used the Apollo.io to search for companies across various industries, and get their company URLs. To narrow these down, I wrote a web crawler (using Node.js, and a combination of Cheerio + Puppeteer depending on site complexity) to find the career page of the company. I discovered that I could dump the raw HTML and prompt ChatGPT o1-mini to classify (as a binary classification) whether each page contained a job description or not. I thus compiled a list of verified job page if it contains a job description or not. If it contains a job description, I add it to a list and proceed to step 2
-
Verifying legit companies. This part I had to do manually, but it was crucial that I exclude any recruiting firms, 3rd party offshore agencies, etc. because I wanted only high-quality companies directly hiring for roles at their firm. I manually sorted through the 30,000 company career pages (this took several weeks) and picked the ones that looked legit. At Stanford, we call this technique "occular regression" :) It was doable because I only had to verify each company a single time and then I trust it moving forward.
-
Removing ghost jobs. I discovered that a strong predictor of if a job is a ghost job is that if it keeps being reposted. I was able to identify reposting by doing a embedding text similarity search for jobs from the same company. If 2 job descriptions overlap too much, I only show the date posted for the earliest listing. This allowed me to weed out most ghost jobs simply by using a date filter (for example, excluding any jobs posted over a month ago). In my anecdotal, experience this means that I get a higher response rate for data science jobs compared to LinkedIn or Indeed.
-
Scraping fresh jobs 3x/day. To ensure that my database is reflective of the company career page, I check each company career page 3x/day. Many career pages do not have rate limits because it is in their best interest to allow web scrapers, which is great. For the few that do, I was able to use a rotating proxy. I use Oxylabs for now, but I've heard good things about ScraperAPI, Crawlera.
-
Building advanced NLP text filters. After playing with GPT4o-mini API, I realized I could can effectively dump raw job descriptions (in HTML) and ask it to give me back formatted information back in JSON (ex salary, yoe, etc). I used this technique to extract a variety of information, including technical keywords, job industry, required licenses & security clearance, if the company sponsors visa, etc.
-
Powerful search. Once I had the structured JSON data (containing salary, years of experience, remote status, job title, company name, location, and other relevant fields) from ChatGPT's extraction process, I needed a robust search engine to allow users to query and filter jobs efficiently. I chose Elasticsearch due to its powerful full-text search capabilities, filtering, and aggregation features. My favorite feature with Elasticsearch is that it allows me to do Boolean queries. For instance, I can search for job descriptions with technical keywords of "Pandas" or "R" (example link here).
Beyond job search, one thing I'm really excited about this 2.1 million job dataset is to be able to do a yearly or quarterly trend report. For instance, to look at what technical skills are growing in demand. What kinds of cool job trends analyses would you do if you had access to this data.
Thank you for sharing! I am building jseek.co with an identical motivation in mind! The HiringCafe is an inspiration for me, but I do think that it has a lot of limitations, especially concerning the EU market.
I follow a very different approach though. I use an agentic workflow to configure prebuilt generalizable scrapers. Have a look at the scrapers I have for inspiration! I've noticed you don't really have a lot of jobs from behemoths like Amazon or Accenture, I would guess because it is very hard to scrape them using straight-forward approaches.
The deduplication for ghost jobs detection is a great feature, I'll definitely implement it! Did you consider using something similar for deduplication of job postings by the same company? If I had to guess, not every company has its own career page some must rely solely on job platforms like LinkedIn and Indeed to distribute job ads.