Fun with qwen2.5-coder:14b

downloader.md

Write a python script to scrape all linked files of a certain file extension on a web page under the same domain as the page. Follow best practices. Handle errors, make strings OS-independent, etc. Be persnickety. Be pythonic.

Certainly! Below is a Python script that scrapes all linked files with a specified file extension from a given webpage and ensures they are from the same domain. This script uses requests to fetch web content and BeautifulSoup from bs4 for parsing HTML. It also handles errors gracefully, sanitizes paths, and follows best practices.

First, make sure you have the necessary packages installed:

pip install requests beautifulsoup4 urllib3

Now, here's the script:

import os
import re
from urllib.parse import urljoin, urlparse
import requests
from bs4 import BeautifulSoup

def is_valid_url(url):
    """Check if a URL has a valid domain."""
    parsed = urlparse(url)
    return bool(parsed.netloc) and bool(parsed.scheme)

def sanitize_filename(filename):
    """Make a filename safe to use on the OS."""
    # Remove any characters that are not alphanumeric or underscores
    sanitized = re.sub(r'[^\w]', '_', filename)
    return sanitized

def download_file(session, url, save_dir, file_extension):
    """Download and save a file from a URL if it has the specified extension."""
    try:
        response = session.get(url, stream=True)
        response.raise_for_status()

        # Extract filename from URL
        path = urlparse(url).path
        filename = os.path.basename(path)
        sanitized_filename = sanitize_filename(filename)

        # Ensure the file has the correct extension
        if not sanitized_filename.endswith(file_extension):
            sanitized_filename += file_extension

        save_path = os.path.join(save_dir, sanitized_filename)

        with open(save_path, 'wb') as f:
            for chunk in response.iter_content(chunk_size=8192):
                f.write(chunk)

        print(f"Downloaded: {save_path}")
    except requests.exceptions.RequestException as e:
        print(f"Failed to download {url}: {e}")

def scrape_and_download_files(url, file_extension, save_dir):
    """Scrape and download all files with a given extension from the same domain."""
    if not is_valid_url(url):
        raise ValueError("Invalid URL provided.")

    # Ensure the directory exists
    os.makedirs(save_dir, exist_ok=True)

    session = requests.Session()

    try:
        response = session.get(url)
        response.raise_for_status()
    except requests.exceptions.RequestException as e:
        print(f"Failed to fetch {url}: {e}")
        return

    soup = BeautifulSoup(response.content, 'html.parser')
    base_url = urlparse(url).netloc
    visited_urls = set()

    for tag in soup.find_all(['a', 'link']):
        href = tag.get('href')
        if href and is_valid_url(href):
            parsed_href = urlparse(href)
            if parsed_href.netloc == base_url:
                file_path = urljoin(url, href)

                # Avoid duplicate URLs
                if file_path not in visited_urls:
                    visited_urls.add(file_path)

                    # Download the file if it matches the extension
                    if file_path.endswith(file_extension):
                        download_file(session, file_path, save_dir, file_extension)

if __name__ == "__main__":
    url = "http://example.com"  # Replace with your target URL
    file_extension = ".pdf"   # Change to the desired file extension
    save_dir = "./downloaded_files"  # Directory where files will be saved

    scrape_and_download_files(url, file_extension, save_dir)

Key Points:

1. URL Validation: The is_valid_url function ensures that URLs are valid.
2. Filename Sanitization: The sanitize_filename function removes invalid characters from filenames to make them safe for use on the OS.
3. Error Handling: The script uses try-except blocks to handle potential errors during HTTP requests and file operations.
4. Session Management: A requests.Session is used to manage cookies and reuse connections, which can be more efficient.
5. Directory Creation: The os.makedirs function ensures the save directory exists before attempting to write files.
6. URL Joining: The urljoin function is used to handle relative URLs correctly.

This script should be run in a Python environment with internet access and appropriate permissions to create directories and write files. Adjust the url, file_extension, and save_dir variables as needed for your specific use case.