Skip to content

Instantly share code, notes, and snippets.

@lobstrio
lobstrio / twitter_scraper.py
Last active March 23, 2024 16:21
Collect 100 first tweets from any user at any time — and export to a .csv file 🐦
# =============================================================================
# Title: Twitter Users Tweets Scraper
# Language: Python
# Description: This script does scrape the first 100 tweets
# of any Twitter User.
# Author: Sasha Bouloudnine
# Date: 2023-08-08
#
# Usage:
# - Make sure you have the required libraries installed by running:
@lobstrio
lobstrio / chatgpt_powered_product_page_universal_scraper.py
Created April 28, 2023 12:48
Scrape price and title from *any* product page, with Python and ChatGPT 🤖
import os
import requests
import html2text
import re
import argparse
OPENAI_API_KEY = 'YOUR_OPEN_AI_API_KEY'
COMPLETION_URL = 'https://api.openai.com/v1/chat/completions'
PROMPT = """Find the main article from this product page, and return from this text content, as JSON format:
@lobstrio
lobstrio / cdiscount_scraper.py
Last active August 11, 2023 21:57
Scrape all products from a cDiscount hot barbecue category URL 🍖
import requests
import re
import json
from lxml import html
import time
from retry import retry
import csv
URL = 'https://www.cdiscount.com/search/10/barbecue.html'
from curl_cffi import requests
from lxml import html
import json
import csv
import time
import argparse
HEADERS = {
'authority': 'www.doctolib.fr',
'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',
@lobstrio
lobstrio / yelp_scraper_2023.py
Created July 17, 2023 10:47
🍝 Collect all listings from Yelp from a Search URL e.g. https://www.yelp.fr/search?find_desc=Pizza&find_loc=marseille — phones included!
import requests
import csv
from lxml import html
import argparse
import time
class YelpSearchScraper:
def iter_listings(self, url):
response = requests.get(url)
if response.status_code != 200:
@lobstrio
lobstrio / growthhackingfr_scraper.py
Created August 11, 2023 17:43
🧙 Scrape all topics from the famous French GrowthHacking.fr forum — 'scraping' category only!
"""
GrowthHacking.fr Forum Scraper
This script is used to scrape data from the GrowthHacking.fr forum, specifically from the "Scraping" category.
It retrieves information about forum topics and saves it as CSV data.
Usage:
1. Install the required library using the following command:
$ pip install requests