Python has evolved from a simple scripting language to the backbone of modern computing, powering everything from web applications to artificial intelligence and scientific research. As we navigate through 2026, Python's relevance continues to grow, with emerging fields like quantum computing, edge AI, and advanced cybersecurity relying heavily on Python's versatility.
This comprehensive guide is designed for learners at all levels—from absolute beginners taking their first steps in programming to experienced developers seeking to deepen their understanding of advanced concepts. The journey through these pages will transform you from a Python novice into a proficient developer capable of building sophisticated systems and contributing to cutting-edge research.
Python's story begins in the late 1980s when Guido van Rossum, a Dutch programmer, started working on a new scripting language during his Christmas holidays. He wanted to create a language that was both powerful and easy to use, bridging the gap between C and shell scripting. The name "Python" wasn't inspired by the snake but by Monty Python's Flying Circus, reflecting van Rossum's desire for a language that would be fun to work with.
Python 0.9.0 was released in February 1991, already featuring many of the core concepts we use today: classes with inheritance, exception handling, functions, and the core data types list, dict, and str. The language was designed with a clear philosophy emphasizing code readability and simplicity.
The journey through major versions marks Python's evolution:
Python 1.x (1994-2000): Established the foundation with functional programming tools (lambda, map, filter, reduce) and the first module system. Python 1.4 introduced the Modula-3 inspired keyword arguments.
Python 2.x (2000-2010): A monumental release that brought garbage collection, list comprehensions, and Unicode support. Python 2.2 unified types and classes, introducing new-style classes. Python 2.5 added with statements and conditional expressions. However, Python 2.x also carried design flaws that would later necessitate a major overhaul.
Python 3.x (2008-present): A deliberate break from backward compatibility to fix fundamental design issues. Python 3.0 cleaned up the language by removing redundant constructs, changing print to a function, improving integer division behavior, and overhauling string handling. The transition was painful but necessary for Python's long-term health. Python 3.3 introduced yield from and venv, 3.5 added async/await and matrix multiplication operators, 3.6 brought formatted string literals, 3.7 introduced dataclasses, 3.8 added assignment expressions, 3.9 brought dictionary merge operators, 3.10 introduced pattern matching, and 3.11 focused on performance improvements. Python 3.12 and beyond continue to refine the language with better error messages, improved performance, and enhanced typing capabilities.
Python's design philosophy is encapsulated in "The Zen of Python," a collection of 19 guiding principles written by Tim Peters. You can view them anytime by typing import this in a Python interpreter:
Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!
These principles guide Python's development and influence how Python programmers write code. "Readability counts" explains why Python uses indentation for block structure. "Explicit is better than implicit" justifies why Python requires self as the first parameter in instance methods. "There should be one obvious way to do it" contrasts with Perl's "there's more than one way to do it" philosophy.
As we progress through 2026, Python's position in the programming landscape is stronger than ever. Several factors contribute to its continued dominance:
Artificial Intelligence and Machine Learning Dominance: Python has become the lingua franca of AI research and development. Frameworks like TensorFlow, PyTorch, and JAX are Python-first, with extensive ecosystems for data preprocessing, model training, and deployment. The rise of large language models and generative AI has only increased Python's importance, with libraries like Hugging Face Transformers and LangChain built around Python.
Data Science Ecosystem: The PyData stack—NumPy, Pandas, Matplotlib, SciPy—remains unparalleled for data analysis and visualization. Organizations across industries rely on Python for business intelligence, analytics, and reporting.
Scientific Computing Renaissance: Python has largely replaced MATLAB and commercial alternatives in academic research. Fields from astrophysics to computational biology use Python for simulations, data analysis, and visualization.
Web Development Maturity: Django and Flask continue to evolve, with FastAPI gaining significant traction for high-performance APIs. Python's web ecosystem now competes effectively with Node.js and Go for many use cases.
DevOps and Cloud Computing: Python is the language of choice for infrastructure automation, with tools like Ansible, Terraform (via Python providers), and cloud vendor SDKs all offering Python interfaces.
Education and Accessibility: Python's gentle learning curve makes it the most taught introductory programming language worldwide. This educational dominance ensures a steady stream of new developers entering the Python ecosystem.
Emerging Fields: Quantum computing frameworks like Qiskit and Cirq are Python-based. Edge computing and IoT leverage MicroPython. Cybersecurity professionals use Python for automation and tool development.
Community and Ecosystem: The Python Package Index (PyPI) hosts over 400,000 packages, providing solutions for virtually any programming task. The community's commitment to documentation, inclusivity, and knowledge sharing creates a welcoming environment for newcomers.
The Python 2 to Python 3 transition represents one of the most significant events in programming language history. Python 3 was designed to fix fundamental flaws that couldn't be addressed without breaking backward compatibility. Key differences include:
Print Function: Python 2 used print "Hello" as a statement. Python 3 makes print() a function, allowing for more flexibility with arguments like sep and end.
Integer Division: Python 2's / performed floor division with integers (3/2 = 1). Python 3 uses true division (3/2 = 1.5), with // for floor division. This change eliminated a common source of bugs in scientific computing.
Unicode Handling: Python 2 had separate str (bytes) and unicode types, leading to encoding confusion. Python 3 made all strings Unicode by default, with a separate bytes type for binary data. This change dramatically simplified internationalization and text processing.
Range Behavior: Python 2's range() returned a list, while xrange() provided lazy evaluation. Python 3's range() behaves like xrange(), saving memory. Similarly, zip(), map(), and filter() return iterators instead of lists.
Exception Syntax: Python 2 used except Exception, e, while Python 3 requires except Exception as e. The older syntax caused subtle bugs with tuple unpacking.
Input Function: Python 2's input() evaluated user input as Python code (dangerous!), while raw_input() read strings. Python 3's input() always returns strings, with eval(input()) required for evaluation.
Iterators and Generators: Python 3 made many functions return iterators rather than lists, promoting memory efficiency. Methods like dict.keys() and dict.items() return view objects instead of lists.
Type Annotations: Python 3.5 introduced optional type hints, enabling better tooling and static analysis. Python 2 had no comparable feature.
Python 2 officially reached end-of-life on January 1, 2020. While some legacy systems still run Python 2, all new development should use Python 3. The migration, though painful, positioned Python for long-term growth and maintainability.
Python's versatility makes it valuable across virtually every industry:
Technology and Software Development: Tech companies use Python for web applications (Instagram, Pinterest), APIs (Spotify), internal tools, and automation. Google, Netflix, and Dropbox have significant Python codebases.
Finance and FinTech: Quantitative analysts use Python for algorithmic trading, risk modeling, and portfolio optimization. Libraries like Zipline and QuantLib support financial analysis. JPMorgan Chase, Goldman Sachs, and Bloomberg employ Python extensively.
Healthcare and Biotechnology: Python powers drug discovery pipelines, genomic analysis, medical imaging, and electronic health record systems. The Broad Institute uses Python for cancer research. COVID-19 modeling efforts relied heavily on Python.
Aerospace and Defense: NASA uses Python for mission planning and data analysis. SpaceX employs Python for rocket telemetry and simulation. Lockheed Martin integrates Python into defense systems.
Automotive Industry: Tesla and other manufacturers use Python for autonomous vehicle research, sensor data processing, and simulation. Python's role in the autonomous vehicle stack continues to grow.
Entertainment and Media: Netflix uses Python for content recommendation and infrastructure management. Disney employs Python for animation and visual effects. Video game companies use Python for tooling and scripting.
Energy and Utilities: Python optimizes power grid operations, models renewable energy systems, and analyzes seismic data for oil exploration. Shell and BP have significant Python investments.
Government and Public Sector: The U.S. federal government uses Python for data analysis, security automation, and citizen services. The UK Government Digital Service builds services with Python.
Retail and E-commerce: Amazon uses Python for recommendation engines and supply chain optimization. Python powers inventory management, customer analytics, and pricing algorithms across the retail sector.
Telecommunications: Python handles network configuration, traffic analysis, and customer experience monitoring. Ericsson and Nokia incorporate Python into their network solutions.
The Python ecosystem extends far beyond the core language. Understanding the landscape helps developers choose appropriate tools:
Package Management: pip is the default package installer, pulling packages from PyPI. Poetry and Pipenv provide more sophisticated dependency management with lock files and virtual environment integration. Conda serves the scientific community with binary package management.
Development Environments: VS Code with the Python extension offers a lightweight, feature-rich experience. PyCharm provides comprehensive IDE capabilities. Jupyter Notebook and JupyterLab dominate data science workflows. Google Colab offers cloud-based notebooks with free GPU access.
Web Frameworks: Django provides a "batteries-included" approach for full-featured web applications. Flask offers minimalism and flexibility. FastAPI delivers high performance with automatic OpenAPI documentation. Pyramid and Tornado serve specific niches.
Data Science Stack: NumPy provides array computing, Pandas offers data structures and analysis, Matplotlib enables visualization, and Scikit-learn delivers machine learning algorithms. These libraries form the foundation of scientific Python.
Machine Learning and AI: TensorFlow and PyTorch dominate deep learning. Hugging Face Transformers provides pre-trained models for NLP. XGBoost and LightGBM excel at gradient boosting. JAX enables high-performance numerical computing.
Testing Tools: unittest comes built-in. pytest offers a more feature-rich testing experience with fixtures and plugins. Hypothesis performs property-based testing. Selenium automates browser testing.
Documentation: Sphinx generates documentation from docstrings. MkDocs builds project documentation from Markdown. Read the Docs hosts documentation for open-source projects.
Deployment and Operations: Docker containerizes Python applications. Kubernetes orchestrates container deployments. Gunicorn and uWSGI serve Python web applications. Celery handles distributed task queues.
Code Quality: Black and autopep8 auto-format code. Flake8 and Pylint perform static analysis. mypy checks type hints. pre-commit automates quality checks.
Scientific Computing: SciPy adds scientific algorithms, SymPy enables symbolic mathematics, and Biopython serves bioinformatics. Astropy supports astronomy research.
This rich ecosystem, combined with Python's readability and learning curve, explains why Python continues to thrive across domains and industries.
Windows users have several options for installing Python, each with advantages for different use cases:
Official Python Installer: The most straightforward approach downloads the installer from python.org. During installation, critically check "Add Python to PATH" to enable command-line access from any directory. The installer offers customization options including installation location and feature selection.
After installation, verify success by opening Command Prompt or PowerShell and typing:
python --version
pip --versionWindows Subsystem for Linux (WSL): For developers working across platforms or requiring Linux compatibility, WSL provides a Linux environment within Windows. After enabling WSL, install Ubuntu or another distribution, then use Linux package managers:
sudo apt update
sudo apt install python3 python3-pipChocolatey Package Manager: For users who prefer package management, Chocolatey offers Python installation:
choco install pythonMicrosoft Store: Python is available through the Microsoft Store, providing easy installation and automatic updates. However, this version may have path limitations and restricted filesystem access.
Anaconda Distribution: Data scientists often prefer Anaconda, which includes Python, Conda package manager, and pre-installed scientific libraries. Download the installer from anaconda.com and follow the graphical installer instructions.
Linux distributions typically include Python pre-installed, but the version may be older. Package managers provide the most integrated installation:
Debian/Ubuntu and derivatives:
sudo apt update
sudo apt install python3 python3-pip python3-venvRed Hat/CentOS/Fedora:
# RHEL/CentOS
sudo yum install python3 python3-pip
# Fedora
sudo dnf install python3 python3-pipArch Linux:
sudo pacman -S python python-pipopenSUSE:
sudo zypper install python3 python3-pipAfter installation, verify with:
python3 --version
pip3 --versionNote that many Linux systems use python3 and pip3 explicitly to distinguish from Python 2, which may still be present for system tools.
macOS includes Python 2.7 for system compatibility, but users should install Python 3 separately:
Official Installer: Download from python.org and run the package installer. The installer adds Python to PATH and optionally installs IDLE and documentation.
Homebrew Package Manager: For developers already using Homebrew, installation is simple:
brew install pythonThis installs the latest Python 3, along with pip and setuptools.
Xcode Command Line Tools: Many Python packages compile C extensions, requiring Xcode tools:
xcode-select --installmacOS-Specific Considerations: Apple's System Integrity Protection may affect Python installations in protected directories. Installing Python in /usr/local or using Homebrew avoids these issues.
pyenv solves the challenge of managing multiple Python versions on the same system. It allows per-project version selection without interfering with system Python:
Installation on macOS/Linux:
curl https://pyenv.run | bashAdd to shell configuration (.bashrc, .zshrc):
export PATH="$HOME/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"Windows users can use pyenv-win:
Invoke-WebRequest -UseBasicParsing -Uri "https://raw.githubusercontent.com/pyenv-win/pyenv-win/master/pyenv-win/install-pyenv-win.ps1" -OutFile "./install-pyenv-win.ps1"; &"./install-pyenv-win.ps1"Common pyenv Commands:
# List available Python versions
pyenv install --list
# Install specific version
pyenv install 3.11.6
pyenv install 3.10.13
# Set global version (default)
pyenv global 3.11.6
# Set local version (per directory)
pyenv local 3.10.13
# View current versions
pyenv versionspyenv integrates with virtual environments through pyenv-virtualenv, allowing project-specific environments with precise Python versions.
Virtual environments isolate project dependencies, preventing conflicts between projects requiring different package versions. Python 3 includes venv built-in:
Creating Virtual Environments with venv:
# Create environment
python -m venv myproject_env
# Activate on Windows
myproject_env\Scripts\activate
# Activate on macOS/Linux
source myproject_env/bin/activate
# Deactivate
deactivatevirtualenv offers additional features and works with Python 2:
pip install virtualenv
virtualenv myproject_envBest Practices:
- Create a virtual environment for each project
- Exclude environment directories from version control (add to .gitignore)
- Generate requirements.txt files:
pip freeze > requirements.txt - Install from requirements:
pip install -r requirements.txt
Advanced venv Usage:
# Create environment with specific Python version
python3.10 -m venv project_env
# Create environment without system site packages
python -m venv --system-site-packages project_envpip remains the fundamental package installer:
# Install package
pip install requests
# Install specific version
pip install django==4.2.7
# Install from requirements file
pip install -r requirements.txt
# Upgrade packages
pip install --upgrade requests
# Uninstall packages
pip uninstall requests
# List installed packages
pip list
# Show package information
pip show requestspipenv combines package management and virtual environments:
# Install pipenv
pip install pipenv
# Create environment and install packages
pipenv install requests
pipenv install --dev pytest
# Activate environment
pipenv shell
# Run command in environment
pipenv run python script.py
# Generate lock file
pipenv lockPipenv maintains Pipfile for dependencies and Pipfile.lock for reproducible builds.
poetry offers modern dependency management with pyproject.toml:
# Install poetry
curl -sSL https://install.python-poetry.org | python3 -
# Create new project
poetry new myproject
# Add dependencies
poetry add requests
poetry add --dev pytest
# Install dependencies
poetry install
# Activate environment
poetry shell
# Build package
poetry build
# Publish to PyPI
poetry publishPoetry handles dependency resolution more intelligently than pip and manages virtual environments automatically.
Visual Studio Code has become the most popular Python editor due to its balance of features and performance:
- Download VS Code from code.visualstudio.com
- Install the Python extension from Microsoft
- Configure settings.json for Python path:
{ "python.defaultInterpreterPath": "path/to/python", "python.linting.enabled": true, "python.linting.pylintEnabled": true, "python.formatting.provider": "black" } - Enable features:
- IntelliSense for code completion
- Debugging with breakpoints
- Jupyter notebook integration
- Git integration
- Testing discovery
PyCharm provides a full-featured Python IDE with two editions:
Community Edition (free):
- Intelligent code completion
- Debugging and testing
- Version control integration
- Django support
Professional Edition (paid):
- Database tools
- Professional web frameworks
- Remote development
- Scientific tools
Setup steps:
- Download from jetbrains.com/pycharm
- Configure Python interpreter (File > Settings > Project > Python Interpreter)
- Install plugins as needed
- Set up code style (PEP 8 enforcement)
Other Notable IDEs:
- Spyder for scientific computing
- Thonny for beginners
- IDLE (included with Python)
- JupyterLab for notebooks
The Python Read-Eval-Print Loop (REPL) provides interactive exploration:
Starting the REPL:
pythonBasic usage:
>>> print("Hello, World!")
Hello, World!
>>> 2 + 2
4
>>> import this
# The Zen of Python appearsIPython enhances the REPL experience:
pip install ipython
ipythonIPython features:
- Tab completion
- Syntax highlighting
- Magic commands (%time, %debug, %run)
- Shell integration (!ls, !pwd)
- History navigation
- Object introspection (object? for documentation)
Example IPython session:
In [1]: import numpy as np
In [2]: arr = np.random.randn(1000)
In [3]: %time np.mean(arr)
CPU times: user 0 ns, sys: 0 ns, total: 0 ns
Wall time: 0 ns
Out[3]: 0.0123456789
In [4]: arr? # Show documentationJupyter Notebook revolutionized data science workflows by combining code, output, and documentation:
Installation:
pip install jupyter notebook
# Or with Anaconda (pre-installed)Starting Jupyter:
jupyter notebookThis opens a browser interface showing the file system. Create new notebooks with Python 3 kernel.
Notebook Components:
- Code cells execute Python
- Markdown cells provide documentation
- Raw cells contain unexecuted text
- Output includes text, images, plots, and interactive widgets
JupyterLab offers an improved interface:
pip install jupyterlab
jupyter labKernels allow multiple languages:
# Install R kernel
pip install r-irkernel
# Install Julia kernel
using Pkg; Pkg.add("IJulia")Magic Commands in notebooks:
# Time execution
%time function_call()
# Run external script
%run script.py
# Debug code
%debug
# Write cell contents to file
%%writefile output.txt
contentExtensions enhance functionality:
pip install jupyter_contrib_nbextensions
jupyter contrib nbextension install --userSharing Notebooks:
- GitHub renders notebooks natively
- nbviewer renders notebooks online
- Convert to other formats:
jupyter nbconvert --to html notebook.ipynb jupyter nbconvert --to pdf notebook.ipynb jupyter nbconvert --to slides notebook.ipynb
Python reserves specific words that have special meaning in the language. These keywords cannot be used as identifiers (variable names, function names, etc.). Python 3.12 includes 35 keywords:
import keyword
print(keyword.kwlist)Control Flow Keywords:
if,elif,else: Conditional executionfor,while: Looping constructsbreak,continue: Loop controlreturn: Function returnyield: Generator function returntry,except,finally,raise: Exception handlingwith: Context managerspass: No-operation placeholder
Logical Keywords:
and,or,not: Boolean operationsTrue,False: Boolean literalsNone: Null valueis: Identity comparisonin: Membership testing
Function and Class Keywords:
def: Function definitionlambda: Anonymous functionclass: Class definitionglobal: Global variable declarationnonlocal: Nonlocal variable declaration
Import Keywords:
import: Module importfrom: Partial importas: Import alias
Asynchronous Keywords:
async: Asynchronous function definitionawait: Asynchronous result waiting
Other Keywords:
del: Object deletionassert: Debugging assertions
Understanding keyword usage is fundamental to Python programming. For example, using and versus & (bitwise and) demonstrates the difference between logical and bitwise operations.
Python uniquely uses indentation to define code blocks, rejecting the brace-based syntax of C-family languages. This design choice enforces readable code:
# Correct indentation
def calculate_average(numbers):
total = 0
count = 0
for number in numbers:
total += number
count += 1
return total / count
# IndentationError: expected an indented block
def calculate_average(numbers):
total = 0 # This line must be indented
return totalIndentation Rules:
- Use 4 spaces per indentation level (PEP 8 recommendation)
- Never mix tabs and spaces
- The first line after a colon must be indented
- All lines in a block must have the same indentation
- Blank lines don't break indentation
Common Indentation Errors:
# Inconsistent indentation
if condition:
print("True branch")
print("Still true?") # Indentation mismatch
# Mixing tabs and spaces (Python 3 disallows)
def function():
print("Tab indented")
print("Space indented") # TabErrorModern editors can be configured to convert tabs to spaces and show whitespace characters, preventing these errors.
Comments and docstrings serve different purposes in Python documentation:
Single-line Comments begin with #:
# Calculate factorial recursively
def factorial(n):
# Base case
if n <= 1:
return 1
# Recursive case
return n * factorial(n - 1)Inline Comments appear on the same line:
x = x + 1 # Increment counterMulti-line Comments use triple quotes but are technically string literals:
"""
This is a multi-line comment
that spans multiple lines.
Python ignores string literals not assigned to variables.
"""Docstrings document modules, functions, classes, and methods:
def calculate_mean(numbers):
"""
Calculate the arithmetic mean of a list of numbers.
Args:
numbers (list): List of numeric values
Returns:
float: The mean average
Raises:
ValueError: If the list is empty
"""
if not numbers:
raise ValueError("Cannot calculate mean of empty list")
return sum(numbers) / len(numbers)Docstring conventions:
- Triple quotes regardless of length
- One-line summary ending with period
- Blank line after summary for multi-line docstrings
- Document arguments, returns, and exceptions
- Follow PEP 257 guidelines
Access docstrings with help() or .__doc__:
help(calculate_mean)
print(calculate_mean.__doc__)Python offers two primary execution modes:
Script Mode executes complete files:
python script.py arg1 arg2Scripts are ideal for:
- Production code
- Reusable modules
- Batch processing
- Application deployment
Interactive Mode (REPL) executes statements immediately:
python
>>> x = 10
>>> print(x * 2)
20Interactive mode excels at:
- Learning and experimentation
- Testing code snippets
- Debugging
- Data exploration
- Quick calculations
IPython enhances interactive mode with:
In [1]: %run script.py # Run script in interactive context
In [2]: %debug # Enter debugger after exception
In [3]: %timeit [x**2 for x in range(1000)] # Time executionScript vs Module Distinction: Python files can act as both scripts (executed directly) and modules (imported elsewhere). The __name__ variable distinguishes these contexts:
# mymodule.py
def useful_function():
return "Useful result"
if __name__ == "__main__":
# This code runs only when script executed directly
print("Running as script")
result = useful_function()
print(result)PEP 8 is Python's official style guide, promoting consistency across Python code:
Naming Conventions:
variable_name: lowercase with underscoresfunction_name: lowercase with underscoresClassName: CapWords (PascalCase)CONSTANT_NAME: UPPERCASE with underscores_internal_use: single leading underscore for "private"__strong_private: double leading underscore for name mangling__magic__: double underscores for special methods
Line Length: Maximum 79 characters for code, 72 for docstrings/comments
Blank Lines:
- Two blank lines between top-level functions and classes
- One blank line between methods within a class
- Use blank lines sparingly within functions for logical grouping
Whitespace:
- Avoid extraneous whitespace inside parentheses, brackets, braces
- One space around assignments and comparisons
- No space before commas, semicolons, or colons
- One space after commas
Example PEP 8 Compliant Code:
import os
import sys
from datetime import datetime
class DataProcessor:
"""Process data according to specified rules."""
MAX_ITEMS = 1000
def __init__(self, name: str):
self.name = name
self.items = []
def add_item(self, item: object) -> bool:
"""Add item to processor if under limit."""
if len(self.items) < self.MAX_ITEMS:
self.items.append(item)
return True
return False
def process_all(self) -> list:
"""Process all items and return results."""
results = []
for item in self.items:
result = self._process_single(item)
results.append(result)
return results
def _process_single(self, item: object) -> object:
"""Internal method for single item processing."""
# Implementation here
return item
def main():
"""Main entry point."""
processor = DataProcessor("Test")
processor.add_item(42)
results = processor.process_all()
print(f"Processed {len(results)} items")
if __name__ == "__main__":
main()Tools like black automatically format code to PEP 8 standards, while flake8 and pylint check compliance.
Beyond PEP 8, clean code principles enhance maintainability:
Meaningful Names:
# Bad
def calc(a, b):
return a * b * 0.5
# Good
def calculate_triangle_area(base, height):
return base * height * 0.5Single Responsibility Principle:
# Bad: Function does too much
def process_user_data(user_data):
# Validate
# Clean
# Save to database
# Send email
pass
# Good: Separated responsibilities
def validate_user_data(data): ...
def clean_user_data(data): ...
def save_user_to_db(user): ...
def send_welcome_email(user): ...DRY (Don't Repeat Yourself):
# Bad: Repeated code
def process_male_users(users):
male_users = [u for u in users if u.gender == 'M']
for user in male_users:
print(f"Processing {user.name}")
# complex processing
def process_female_users(users):
female_users = [u for u in users if u.gender == 'F']
for user in female_users:
print(f"Processing {user.name}")
# complex processing
# Good: Reusable function
def process_users_by_gender(users, gender):
filtered = [u for u in users if u.gender == gender]
for user in filtered:
print(f"Processing {user.name}")
# complex processingFunction Length: Functions should do one thing and be short enough to understand at a glance
Comments Should Explain Why, Not What:
# Bad: Explains what (obvious from code)
x = x + 1 # Increment x by 1
# Good: Explains why
# Adjust for zero-based indexing in the database
user_id = api_response.user_id - 1Error Handling:
# Bad: Silent failure
try:
data = load_file(filename)
except FileNotFoundError:
pass
# Good: Explicit handling
try:
data = load_file(filename)
except FileNotFoundError:
logger.error(f"Configuration file {filename} not found")
raise SystemExit("Cannot start without configuration")Clean code practices make programs easier to understand, debug, and modify, reducing technical debt and improving team productivity.
Variables in Python are references to objects in memory. Unlike statically-typed languages, Python variables don't have fixed types; they simply point to objects that carry type information:
x = 42 # x references an integer
x = "hello" # Now x references a string
x = [1, 2, 3] # Now x references a listNaming Rules:
- Must start with letter or underscore
- Can contain letters, numbers, underscores
- Case-sensitive (age, Age, AGE are different)
- Cannot use Python keywords
- Should follow PEP 8 conventions
Valid Names:
user_name = "Alice"
_user_id = 42
user2 = "Bob"
camelCase = "not recommended" # but allowedInvalid Names:
2user = "Bob" # Cannot start with number
user-name = "Bob" # Hyphen not allowed
class = "Math" # 'class' is keywordVariable Assignment:
# Multiple assignment
a, b, c = 1, 2, 3
# Chained assignment
x = y = z = 0
# Swapping values
a, b = b, a
# Unpacking sequences
coordinates = (10, 20)
x, y = coordinatesPython's dynamic typing means variables can reference any type, and type checking occurs at runtime:
def process(value):
if isinstance(value, int):
return value * 2
elif isinstance(value, str):
return value.upper()
else:
return str(value)
print(process(10)) # 20
print(process("hello")) # HELLO
print(process([1,2])) # [1, 2]Type Inference: Python determines types during execution:
x = 10 # Python knows x is int
print(type(x)) # <class 'int'>
x = x + 0.5 # Now x becomes float
print(type(x)) # <class 'float'>Dynamic Typing Advantages:
- Flexibility in function arguments
- Easier prototyping
- Duck typing ("If it walks like a duck...")
Dynamic Typing Challenges:
- Runtime type errors
- Performance overhead
- Less self-documenting code
Python's primitive types are fundamental building blocks:
int (Integers):
# Unlimited precision (only limited by memory)
x = 42
y = -100
z = 123456789012345678901234567890
# Different bases
binary = 0b1010 # 10 in decimal
octal = 0o12 # 10 in decimal
hexadecimal = 0xA # 10 in decimal
# Underscores for readability
million = 1_000_000float (Floating-point numbers):
x = 3.14159
y = -0.001
z = 2.5e-4 # Scientific notation: 0.00025
# Special values
inf = float('inf')
neg_inf = float('-inf')
nan = float('nan')
# Precision limitations
print(0.1 + 0.2) # 0.30000000000000004 (floating-point arithmetic)bool (Boolean values):
is_valid = True
is_complete = False
# Boolean operations
result = is_valid and is_complete
result = is_valid or is_complete
result = not is_valid
# Truth value testing
bool(0) # False
bool(1) # True
bool("") # False
bool("hello") # True
bool([]) # False
bool([1,2]) # True
bool(None) # Falsecomplex (Complex numbers):
z = 3 + 4j
print(z.real) # 3.0
print(z.imag) # 4.0
print(z.conjugate()) # (3-4j)
# Operations
z1 = 2 + 3j
z2 = 1 - 2j
print(z1 + z2) # (3+1j)
print(z1 * z2) # (8-1j)NoneType (Null value):
result = None
if result is None:
print("No result available")Type conversion (casting) changes between data types:
Implicit Conversion (automatic):
x = 10 # int
y = 3.14 # float
z = x + y # z becomes float (12.14)
# Boolean conversion in conditions
if 42: # True
print("42 is truthy")Explicit Conversion (manual):
# To integer
int(3.14) # 3 (truncates)
int("42") # 42
int(True) # 1
int(False) # 0
# To float
float(3) # 3.0
float("3.14") # 3.14
float("inf") # inf
# To string
str(42) # "42"
str(3.14) # "3.14"
str(True) # "True"
# To boolean
bool(0) # False
bool(42) # True
bool("") # False
bool("hello") # True
bool([]) # False
bool([1,2]) # True
bool(None) # False
# Complex conversion
complex(3, 4) # (3+4j)
complex("3+4j") # (3+4j)Handling Conversion Errors:
try:
value = int("not a number")
except ValueError as e:
print(f"Conversion failed: {e}")Type hints, introduced in Python 3.5, enable optional static typing:
Basic Type Hints:
name: str = "Alice"
age: int = 30
height: float = 1.75
is_student: bool = False
def greet(name: str) -> str:
return f"Hello, {name}"Collection Type Hints:
from typing import List, Dict, Set, Tuple
names: List[str] = ["Alice", "Bob", "Charlie"]
scores: Dict[str, int] = {"Alice": 95, "Bob": 87}
unique_ids: Set[int] = {1, 2, 3, 4}
coordinates: Tuple[float, float] = (40.7128, -74.0060)Optional and Union Types:
from typing import Optional, Union
def find_user(user_id: int) -> Optional[dict]:
# Returns dict or None
if user_id in database:
return database[user_id]
return None
def process_value(value: Union[int, str]) -> str:
if isinstance(value, int):
return str(value * 2)
return value.upper()Type Aliases:
from typing import List, Tuple
Coordinate = Tuple[float, float]
Polygon = List[Coordinate]
def calculate_area(shape: Polygon) -> float:
# Implementation here
passGeneric Types:
from typing import TypeVar, Generic
T = TypeVar('T')
class Stack(Generic[T]):
def __init__(self) -> None:
self.items: List[T] = []
def push(self, item: T) -> None:
self.items.append(item)
def pop(self) -> T:
return self.items.pop()
# Usage
int_stack = Stack[int]()
int_stack.push(42)Type Checking Tools:
# Install mypy
pip install mypy
# Run type checker
mypy script.pyType hints don't affect runtime behavior but enable better IDE support, documentation, and static analysis.
Python provides standard arithmetic operations:
a = 10
b = 3
# Basic operations
print(a + b) # 13 (addition)
print(a - b) # 7 (subtraction)
print(a * b) # 30 (multiplication)
print(a / b) # 3.3333333333333335 (division)
print(a // b) # 3 (floor division)
print(a % b) # 1 (modulus/remainder)
print(a ** b) # 1000 (exponentiation)
# Operator with assignment
x = 5
x += 3 # x = x + 3
x -= 2 # x = x - 2
x *= 4 # x = x * 4
x /= 2 # x = x / 2
x //= 3 # x = x // 3
x %= 2 # x = x % 2
x **= 3 # x = x ** 3Special Cases:
# Division by zero
try:
result = 10 / 0
except ZeroDivisionError:
print("Cannot divide by zero")
# Integer division with negatives
print(-10 // 3) # -4 (floor, not truncate)
print(10 // -3) # -4
# Modulo with negatives
print(-10 % 3) # 2 (result has same sign as divisor)
print(10 % -3) # -2Comparison operators return boolean values:
x = 10
y = 20
print(x == y) # False (equal)
print(x != y) # True (not equal)
print(x < y) # True (less than)
print(x > y) # False (greater than)
print(x <= y) # True (less than or equal)
print(x >= y) # False (greater than or equal)
# Chained comparisons
print(5 < x < 15) # True (5 < 10 and 10 < 15)
print(5 < x < 8) # False
print(10 == x == 10) # True
# Identity comparisons (is, is not)
a = [1, 2, 3]
b = [1, 2, 3]
c = a
print(a == b) # True (same value)
print(a is b) # False (different objects)
print(a is c) # True (same object)
# Membership (in, not in)
numbers = [1, 2, 3, 4, 5]
print(3 in numbers) # True
print(10 not in numbers) # TrueLogical operators combine boolean expressions:
a = True
b = False
print(a and b) # False (both must be True)
print(a or b) # True (at least one True)
print(not a) # False (negation)
# Short-circuit evaluation
def expensive_check():
print("Performing expensive check...")
return True
# Second operand never evaluated because first is False
result = False and expensive_check() # expensive_check not called
# Practical examples
user = get_user()
if user is not None and user.is_active:
print("User is active")
# Truthiness in logical operations
print(0 and 42) # 0 (first false value)
print(42 and 100) # 100 (last value if all true)
print(0 or 42) # 42 (first true value)
print(42 or 100) # 42 (first true value)Bitwise operators work on integer bits:
a = 0b1100 # 12 in decimal
b = 0b1010 # 10 in decimal
print(bin(a & b)) # 0b1000 (8) - AND
print(bin(a | b)) # 0b1110 (14) - OR
print(bin(a ^ b)) # 0b0110 (6) - XOR
print(bin(~a)) # -0b1101 (-13) - NOT (two's complement)
print(bin(a << 2)) # 0b110000 (48) - left shift
print(bin(a >> 2)) # 0b11 (3) - right shift
# Practical use: flag combinations
READ = 0b001
WRITE = 0b010
EXECUTE = 0b100
permissions = READ | WRITE # 0b011
if permissions & READ:
print("Can read")
if permissions & WRITE:
print("Can write")Beyond simple assignment, Python offers compound operators:
# Basic assignment
x = 10
# Compound assignment (works with all arithmetic operators)
x += 5 # x = x + 5
x -= 3 # x = x - 3
x *= 2 # x = x * 2
x /= 4 # x = x / 4
x //= 2 # x = x // 2
x %= 3 # x = x % 3
x **= 2 # x = x ** 2
# Bitwise compound assignment
x &= 0b1010 # x = x & 0b1010
x |= 0b1100 # x = x | 0b1100
x ^= 0b1111 # x = x ^ 0b1111
x <<= 2 # x = x << 2
x >>= 1 # x = x >> 1
# Walrus operator (Python 3.8+)
# Assign and use in expression
if (n := len(data)) > 10:
print(f"Processing {n} items")
while (line := file.readline()) != "":
process(line)Python follows standard mathematical precedence:
# Precedence (highest to lowest):
# 1. Parentheses: ()
# 2. Exponentiation: **
# 3. Unary operators: +x, -x, ~x
# 4. Multiplication, division, modulo: *, /, //, %
# 5. Addition, subtraction: +, -
# 6. Bitwise shifts: <<, >>
# 7. Bitwise AND: &
# 8. Bitwise XOR: ^
# 9. Bitwise OR: |
# 10. Comparisons: ==, !=, <, >, <=, >=, is, in
# 11. Logical NOT: not
# 12. Logical AND: and
# 13. Logical OR: or
# 14. Assignment: =, +=, -=, etc.
# Examples
result = 2 + 3 * 4 # 14 (not 20)
result = (2 + 3) * 4 # 20 (parentheses override)
result = 2 ** 3 ** 2 # 512 (right-associative: 2 ** (3 ** 2))
result = (2 ** 3) ** 2 # 64 (parentheses change associativity)
# Complex expressions benefit from parentheses
x = 10
y = 5
z = 2
# Clear version
result = ((x + y) * z) / (x - y)
# Unclear version (same result but harder to read)
result = x + y * z / x - yConditional execution forms the backbone of decision-making:
temperature = 25
if temperature > 30:
print("It's hot outside!")
print("Stay hydrated.")
elif temperature > 20:
print("It's warm and pleasant.")
elif temperature > 10:
print("It's a bit cool.")
else:
print("It's cold!")
# Multiple conditions
age = 25
has_license = True
if age >= 18 and has_license:
print("You can drive")
elif age >= 18 and not has_license:
print("You need a license first")
else:
print("Too young to drive")
# Nested conditions
score = 85
if score >= 60:
print("You passed!")
if score >= 90:
print("Grade: A")
elif score >= 80:
print("Grade: B")
elif score >= 70:
print("Grade: C")
else:
print("Grade: D")
else:
print("You failed")Conditional Expressions (Ternary Operator):
# Traditional if-else
if age >= 18:
status = "adult"
else:
status = "minor"
# Ternary expression
status = "adult" if age >= 18 else "minor"
# Nested ternary (use sparingly)
category = "child" if age < 13 else "teen" if age < 20 else "adult"Python 3.10 introduced structural pattern matching, similar to switch statements in other languages:
# Basic match-case
def describe_value(value):
match value:
case 0:
return "Zero"
case 1:
return "One"
case 2:
return "Two"
case _: # Default case
return "Something else"
# Matching with OR patterns
def get_weekend_day(day):
match day:
case "Saturday" | "Sunday":
return "Weekend"
case _:
return "Weekday"
# Matching sequences
def process_command(command):
match command.split():
case ["quit"]:
print("Goodbye!")
return True
case ["hello", name]:
print(f"Hello, {name}!")
case ["add", x, y]:
result = int(x) + int(y)
print(f"Result: {result}")
case _:
print("Unknown command")
# Matching with guards
def categorize_number(n):
match n:
case 0:
return "Zero"
case x if x > 0:
return "Positive"
case x if x < 0:
return "Negative"
# Matching objects
from dataclasses import dataclass
@dataclass
class Point:
x: int
y: int
def describe_point(point):
match point:
case Point(0, 0):
return "Origin"
case Point(0, y):
return f"On y-axis at {y}"
case Point(x, 0):
return f"On x-axis at {x}"
case Point(x, y):
return f"Point at ({x}, {y})"For loops iterate over sequences:
# Iterating over a list
fruits = ["apple", "banana", "cherry"]
for fruit in fruits:
print(fruit)
# Iterating with index
for i, fruit in enumerate(fruits):
print(f"{i}: {fruit}")
# Iterating over range
for i in range(5): # 0,1,2,3,4
print(i)
for i in range(2, 8): # 2,3,4,5,6,7
print(i)
for i in range(0, 10, 2): # 0,2,4,6,8 (step)
print(i)
# Iterating over dictionary
person = {"name": "Alice", "age": 30, "city": "New York"}
for key in person: # keys
print(key)
for value in person.values(): # values
print(value)
for key, value in person.items(): # key-value pairs
print(f"{key}: {value}")
# Nested loops
for i in range(3):
for j in range(3):
print(f"({i},{j})", end=" ")
print() # newline
# Loop with zip (parallel iteration)
names = ["Alice", "Bob", "Charlie"]
ages = [25, 30, 35]
cities = ["NYC", "LA", "Chicago"]
for name, age, city in zip(names, ages, cities):
print(f"{name} is {age} and lives in {city}")While loops continue as long as a condition is true:
# Basic while loop
count = 0
while count < 5:
print(count)
count += 1
# Input validation
user_input = ""
while user_input.lower() not in ["yes", "no", "quit"]:
user_input = input("Enter yes, no, or quit: ")
if user_input.lower() == "quit":
print("Goodbye!")
break
# Infinite loop with break
while True:
response = input("Continue? (y/n): ")
if response.lower() == 'n':
break
print("Continuing...")
# Game loop example
import random
target = random.randint(1, 100)
attempts = 0
guessed = False
while not guessed:
guess = int(input("Guess a number (1-100): "))
attempts += 1
if guess < target:
print("Too low!")
elif guess > target:
print("Too high!")
else:
print(f"Correct! You took {attempts} attempts.")
guessed = TrueLoop control statements modify execution flow:
break: Exits the loop entirely
# Find first even number
numbers = [1, 3, 5, 7, 8, 9, 11]
for num in numbers:
if num % 2 == 0:
print(f"Found even number: {num}")
break
print(f"{num} is odd")
# Output: 1 is odd, 3 is odd, 5 is odd, 7 is odd, Found even number: 8
# break in nested loops (breaks only innermost loop)
for i in range(3):
for j in range(3):
if i == j == 1:
break
print(f"({i},{j})")continue: Skips to next iteration
# Print odd numbers only
for num in range(10):
if num % 2 == 0:
continue
print(num) # 1,3,5,7,9
# Skip specific items
items = [1, None, 3, None, 5]
for item in items:
if item is None:
continue
print(item * 2) # 2,6,10pass: Does nothing (placeholder)
# Placeholder for future code
def function_not_implemented_yet():
pass
class FutureClass:
pass
# In loops when syntax requires statement
for i in range(10):
if i % 2 == 0:
pass # Will handle even numbers later
else:
print(f"Odd: {i}")Python's for-else and while-else execute when loops complete normally (without break):
# Search example - else executes if item not found
def find_item(items, target):
for item in items:
if item == target:
print(f"Found {target}")
break
else:
print(f"{target} not found")
find_item([1, 2, 3, 4, 5], 3) # Found 3
find_item([1, 2, 3, 4, 5], 10) # 10 not found
# Prime number checker
def is_prime(n):
if n < 2:
return False
for i in range(2, int(n ** 0.5) + 1):
if n % i == 0:
print(f"{n} is divisible by {i}")
break
else:
print(f"{n} is prime!")
return True
return False
is_prime(17) # 17 is prime!
is_prime(15) # 15 is divisible by 3
# while-else example
attempts = 0
max_attempts = 3
while attempts < max_attempts:
password = input("Enter password: ")
if password == "secret":
print("Access granted")
break
attempts += 1
else:
print("Too many failed attempts")Functions encapsulate reusable code blocks:
# Basic function definition
def greet():
"""Simple greeting function."""
print("Hello, World!")
# Function with parameters
def greet_person(name):
"""Greet a specific person."""
print(f"Hello, {name}!")
# Function with return value
def add(a, b):
"""Return sum of two numbers."""
return a + b
# Multiple return values
def get_min_max(numbers):
"""Return minimum and maximum from list."""
return min(numbers), max(numbers)
# Calling functions
greet()
greet_person("Alice")
result = add(5, 3)
min_val, max_val = get_min_max([1, 5, 2, 8, 3])Function Anatomy:
def function_name(parameters):
"""
Docstring explaining function purpose.
Args:
parameters: description
Returns:
description of return value
"""
# Function body
# ... processing ...
return valuePython offers flexible parameter handling:
Positional Arguments:
def describe_person(name, age, city):
print(f"{name} is {age} years old and lives in {city}")
# Must match order
describe_person("Alice", 30, "New York")Keyword Arguments:
# Order doesn't matter with keywords
describe_person(age=30, name="Alice", city="New York")
# Mix positional and keyword (positional first)
describe_person("Alice", city="New York", age=30)Default Parameters:
def greet(name, greeting="Hello", punctuation="!"):
print(f"{greeting}, {name}{punctuation}")
greet("Alice") # Hello, Alice!
greet("Bob", "Hi") # Hi, Bob!
greet("Charlie", "Hey", ".") # Hey, Charlie.
greet(name="Dave", punctuation="?") # Hello, Dave?Important: Default parameters are evaluated once at function definition:
def add_item(item, items=[]): # BAD: list shared across calls
items.append(item)
return items
print(add_item(1)) # [1]
print(add_item(2)) # [1, 2] (shared list!)
# Correct approach
def add_item(item, items=None):
if items is None:
items = []
items.append(item)
return itemsArbitrary positional arguments (*args):
def sum_all(*numbers):
"""Sum any number of arguments."""
return sum(numbers)
print(sum_all(1, 2, 3)) # 6
print(sum_all(1, 2, 3, 4, 5)) # 15
def log_message(level, *messages):
"""Log messages with severity level."""
for msg in messages:
print(f"[{level}] {msg}")
log_message("INFO", "Starting", "Processing", "Done")**Arbitrary keyword arguments (kwargs):
def create_profile(name, **details):
"""Create user profile with arbitrary details."""
profile = {"name": name}
profile.update(details)
return profile
profile = create_profile(
"Alice",
age=30,
city="New York",
occupation="Engineer",
hobby="Photography"
)
print(profile)
# {'name': 'Alice', 'age': 30, 'city': 'New York',
# 'occupation': 'Engineer', 'hobby': 'Photography'}
def print_config(**settings):
for key, value in settings.items():
print(f"{key}: {value}")
print_config(host="localhost", port=8080, debug=True)Combining parameter types:
def complex_function(
required, # Positional parameter
default="default", # Default parameter
*args, # Variable positional
**kwargs # Variable keyword
):
print(f"required: {required}")
print(f"default: {default}")
print(f"args: {args}")
print(f"kwargs: {kwargs}")
complex_function("must", "optional", 1, 2, 3, extra="data", flag=True)Lambda functions are anonymous, single-expression functions:
# Basic lambda
square = lambda x: x ** 2
print(square(5)) # 25
# Lambda with multiple parameters
add = lambda x, y: x + y
print(add(3, 4)) # 7
# Common use with sorting
students = [
{"name": "Alice", "grade": 85},
{"name": "Bob", "grade": 92},
{"name": "Charlie", "grade": 78}
]
# Sort by grade
students.sort(key=lambda student: student["grade"])
print(students)
# With map, filter, reduce
numbers = [1, 2, 3, 4, 5]
squared = list(map(lambda x: x ** 2, numbers))
evens = list(filter(lambda x: x % 2 == 0, numbers))
from functools import reduce
product = reduce(lambda x, y: x * y, numbers) # 120
# Conditional lambda
status = lambda age: "adult" if age >= 18 else "minor"
print(status(20)) # adult
print(status(15)) # minorFunctions calling themselves for repetitive problems:
# Factorial recursively
def factorial(n):
"""Calculate n! recursively."""
if n <= 1: # Base case
return 1
return n * factorial(n - 1) # Recursive case
print(factorial(5)) # 120
# Fibonacci sequence
def fibonacci(n):
"""Return nth Fibonacci number."""
if n <= 1:
return n
return fibonacci(n - 1) + fibonacci(n - 2)
# Optimized with memoization
from functools import lru_cache
@lru_cache(maxsize=None)
def fibonacci_optimized(n):
if n <= 1:
return n
return fibonacci_optimized(n - 1) + fibonacci_optimized(n - 2)
# Directory tree traversal
import os
def list_files(path, indent=""):
"""Recursively list files in directory."""
print(f"{indent}{os.path.basename(path)}/")
for item in os.listdir(path):
item_path = os.path.join(path, item)
if os.path.isdir(item_path):
list_files(item_path, indent + " ")
else:
print(f"{indent} {item}")Recursion limits and tail recursion:
import sys
print(sys.getrecursionlimit()) # Usually 1000
# Python doesn't optimize tail recursion
def tail_factorial(n, accumulator=1):
if n <= 1:
return accumulator
return tail_factorial(n - 1, n * accumulator)
# Still uses stack for each callProper documentation enhances code usability:
Comprehensive docstring:
def calculate_bmi(weight: float, height: float) -> float:
"""
Calculate Body Mass Index (BMI).
BMI is a measure of body fat based on height and weight.
Args:
weight (float): Weight in kilograms
height (float): Height in meters
Returns:
float: BMI value rounded to 2 decimal places
Raises:
ValueError: If weight or height is not positive
Examples:
>>> calculate_bmi(70, 1.75)
22.86
>>> calculate_bmi(0, 1.75)
Traceback (most recent call last):
...
ValueError: Weight must be positive
"""
if weight <= 0:
raise ValueError("Weight must be positive")
if height <= 0:
raise ValueError("Height must be positive")
bmi = weight / (height ** 2)
return round(bmi, 2)Type hints with complex types:
from typing import List, Dict, Optional, Union, Callable, Any
from datetime import datetime
def process_data(
data: List[Dict[str, Any]],
filter_func: Optional[Callable[[Dict[str, Any]], bool]] = None,
sort_key: Optional[str] = None,
limit: int = 100
) -> List[Dict[str, Any]]:
"""
Process a list of data dictionaries.
Args:
data: List of dictionaries to process
filter_func: Optional function to filter items
sort_key: Optional key to sort by
limit: Maximum number of items to return
Returns:
Processed list of dictionaries
"""
result = data.copy()
if filter_func:
result = [item for item in result if filter_func(item)]
if sort_key:
result.sort(key=lambda x: x.get(sort_key, ''))
return result[:limit]
# Function type hints
def create_multiplier(factor: int) -> Callable[[int], int]:
"""Return a function that multiplies by factor."""
def multiplier(x: int) -> int:
return x * factor
return multiplierFunction annotations provide metadata about parameters and return values:
def greet(name: str, age: int = 0) -> str:
return f"{name} is {age} years old"
# Access annotations
print(greet.__annotations__)
# {'name': <class 'str'>, 'age': <class 'int'>, 'return': <class 'str'>}Advanced annotations:
from typing import List, Tuple, Optional
def process_items(
items: List[str],
callback: Optional[callable] = None,
*args: Tuple[str, ...],
**kwargs: dict
) -> List[str]:
"""Process items with optional callback."""
result = []
for item in items:
processed = item.upper()
if callback:
processed = callback(processed, *args, **kwargs)
result.append(processed)
return result
# Type aliases with annotations
Vector = List[float]
Matrix = List[Vector]
def matrix_multiply(a: Matrix, b: Matrix) -> Matrix:
"""Multiply two matrices."""
# Implementation here
passFunction design principles:
Single Responsibility:
# BAD: Does too many things
def process_user_data(user_data):
# Validate
if not user_data.get('email'):
return None
# Clean
user_data['email'] = user_data['email'].lower().strip()
# Save to DB
db.save(user_data)
# Send email
send_welcome_email(user_data['email'])
return user_data
# GOOD: Separated concerns
def validate_user_data(data):
return bool(data.get('email'))
def clean_user_data(data):
cleaned = data.copy()
if 'email' in cleaned:
cleaned['email'] = cleaned['email'].lower().strip()
return cleaned
def save_user(data):
return db.save(data)
def process_user_data(user_data):
if not validate_user_data(user_data):
raise ValueError("Invalid user data")
cleaned = clean_user_data(user_data)
user = save_user(cleaned)
send_welcome_email(user.email)
return userPure functions when possible:
# IMPURE (modifies global state or input)
total = 0
def add_to_total(x):
global total
total += x
return total
# PURE (no side effects)
def add(x, y):
return x + y
# PURE (creates new object rather than modifying input)
def add_to_list(lst, item):
return lst + [item] # Creates new list
original = [1, 2, 3]
new_list = add_to_list(original, 4)
print(original) # [1, 2, 3] (unchanged)Explicit over implicit:
# BAD: Implicit dependencies
def calculate():
global data # Where does data come from?
return sum(data) / len(data)
# GOOD: Explicit parameters
def calculate_average(data):
return sum(data) / len(data)Error handling:
def divide_numbers(a, b):
"""
Safely divide two numbers.
Args:
a: Numerator
b: Denominator
Returns:
float result or None if division impossible
Raises:
TypeError: If inputs aren't numbers
"""
try:
if not isinstance(a, (int, float)) or not isinstance(b, (int, float)):
raise TypeError("Arguments must be numbers")
return a / b
except ZeroDivisionError:
print("Warning: Division by zero")
return None
except Exception as e:
print(f"Unexpected error: {e}")
return NoneFunction length:
# BAD: Too long, does too much
def process_order(order):
# 100+ lines of validation, calculation, payment, shipping, logging...
pass
# GOOD: Decomposed into smaller functions
def validate_order(order): ...
def calculate_total(order): ...
def process_payment(order, total): ...
def arrange_shipping(order): ...
def log_order(order): ...
def process_order(order):
if not validate_order(order):
raise ValueError("Invalid order")
total = calculate_total(order)
payment_result = process_payment(order, total)
if payment_result.success:
shipping_result = arrange_shipping(order)
log_order(order)
return shipping_result
return payment_resultDocumentation:
def complex_algorithm(data, iterations=100):
"""
Implement complex algorithm with clear documentation.
This algorithm uses the Smith-Waterman local alignment
approach with optimized scoring matrices.
For mathematical background, see: Smith & Waterman (1981)
Performance considerations: O(n²) time, O(n) space
Example usage:
>>> data = [1, 2, 3, 4, 5]
>>> complex_algorithm(data, iterations=50)
[2, 4, 6, 8, 10]
"""
# Implementation
passStrings in Python are immutable sequences of Unicode characters. The extensive set of built-in methods makes string manipulation powerful and intuitive:
Basic String Operations:
# Creating strings
single_quotes = 'Hello'
double_quotes = "World"
triple_quotes = """This can span
multiple lines"""
# String concatenation
first = "Hello"
second = "World"
result = first + " " + second # "Hello World"
# String repetition
dash_line = "-" * 50 # 50 dashes
# String length
text = "Python"
print(len(text)) # 6
# Accessing characters (strings are sequences)
print(text[0]) # 'P'
print(text[-1]) # 'n'
print(text[1:4]) # 'yth' (slicing)Case Conversion Methods:
text = "Python Programming"
print(text.upper()) # "PYTHON PROGRAMMING"
print(text.lower()) # "python programming"
print(text.capitalize()) # "Python programming"
print(text.title()) # "Python Programming"
print(text.swapcase()) # "pYTHON pROGRAMMING"
# Case-insensitive comparison
user_input = "Yes"
if user_input.lower() == "yes":
print("User confirmed")
# Checking case
print("hello".islower()) # True
print("HELLO".isupper()) # True
print("Hello World".istitle()) # TrueSearching and Finding:
text = "Python is amazing, Python is powerful"
# Find substring (returns first index or -1)
print(text.find("Python")) # 0
print(text.find("Java")) # -1
# rfind searches from right
print(text.rfind("Python")) # 22
# index raises ValueError if not found
try:
print(text.index("Java"))
except ValueError:
print("Substring not found")
# Count occurrences
print(text.count("Python")) # 2
print(text.count("is")) # 2
# Startswith / Endswith
filename = "document.pdf"
print(filename.startswith("doc")) # True
print(filename.endswith(".pdf")) # True
print(filename.endswith((".txt", ".pdf", ".doc"))) # True (tuple)Validation Methods:
# Check character types
print("123".isdigit()) # True
print("123.45".isdigit()) # False (contains decimal point)
print("123".isnumeric()) # True
print("½".isnumeric()) # True (unicode fraction)
print("abc123".isalnum()) # True (letters and numbers)
print("abc123!".isalnum()) # False (contains !)
print("abc".isalpha()) # True
print("abc123".isalpha()) # False
print(" ".isspace()) # True
print("Hello".isprintable()) # True
# Practical validation
def validate_username(username):
if not username:
return False
if not username[0].isalpha():
return False
if not username.isalnum():
return False
return TrueManipulation Methods:
text = " Python Programming "
# Stripping whitespace
print(text.strip()) # "Python Programming"
print(text.lstrip()) # "Python Programming "
print(text.rstrip()) # " Python Programming"
# Stripping specific characters
url = "www.python.org"
print(url.strip("worg.")) # "python"
# Replacing
text = "I like Python, Python is great"
print(text.replace("Python", "Java")) # Replace all
print(text.replace("Python", "Java", 1)) # Replace first only
# Splitting
sentence = "Python is awesome"
words = sentence.split() # ["Python", "is", "awesome"]
csv = "apple,banana,orange"
fruits = csv.split(",") # ["apple", "banana", "orange"]
parts = "one::two::three".split("::") # ["one", "two", "three"]
# Joining
words = ["Python", "is", "awesome"]
sentence = " ".join(words) # "Python is awesome"
path = "/".join(["home", "user", "docs"]) # "home/user/docs"
# Partition
text = "hello-world-123"
print(text.partition("-")) # ('hello', '-', 'world-123')
print(text.rpartition("-")) # ('hello-world', '-', '123')Padding and Alignment:
text = "Python"
# Center
print(text.center(20)) # " Python "
print(text.center(20, "*")) # "*******Python*******"
# Left/Right justify
print(text.ljust(20)) # "Python "
print(text.rjust(20)) # " Python"
print(text.rjust(20, '-')) # "--------------Python"
# Zfill (zero padding)
number = "42"
print(number.zfill(5)) # "00042"
print("-42".zfill(5)) # "-0042"Formatted string literals (f-strings) provide the most readable string interpolation:
Basic f-strings:
name = "Alice"
age = 30
print(f"My name is {name} and I am {age} years old")
# Expressions inside braces
width = 10
height = 5
print(f"Area: {width * height}") # Area: 50
# Function calls
def double(x):
return x * 2
value = 21
print(f"Double of {value} is {double(value)}") # Double of 21 is 42Format Specifiers:
# Numbers
pi = 3.14159265359
print(f"Pi to 2 decimals: {pi:.2f}") # Pi to 2 decimals: 3.14
print(f"Pi to 4 decimals: {pi:.4f}") # Pi to 4 decimals: 3.1416
print(f"Percentage: {0.25:.2%}") # Percentage: 25.00%
# Width and alignment
name = "Alice"
print(f"|{name:>10}|") # Right align: | Alice|
print(f"|{name:<10}|") # Left align: |Alice |
print(f"|{name:^10}|") # Center: | Alice |
# Padding with fill character
print(f"|{name:*^10}|") # |**Alice***|
print(f"|{name:*>10}|") # |*****Alice|
print(f"|{name:*<10}|") # |Alice*****|
# Number formatting
number = 1234567
print(f"{number:,}") # 1,234,567 (thousands separator)
print(f"{number:_}") # 1_234_567
# Binary, hex, octal
value = 42
print(f"Binary: {value:b}") # Binary: 101010
print(f"Hex: {value:x}") # Hex: 2a
print(f"Octal: {value:o}") # Octal: 52Advanced f-string Features:
# Date formatting
from datetime import datetime
now = datetime.now()
print(f"{now:%Y-%m-%d %H:%M:%S}") # 2026-02-28 15:30:45
# Nested formatting
precision = 3
value = 1.23456789
print(f"{value:.{precision}f}") # 1.235
# Debugging (Python 3.8+)
x = 10
y = 20
print(f"{x=}, {y=}") # x=10, y=20
print(f"{x + y=}") # x + y=30
# Multiline f-strings
name = "Alice"
age = 30
occupation = "Engineer"
info = (
f"Name: {name}\n"
f"Age: {age}\n"
f"Occupation: {occupation}"
)
print(info)
# Dictionary access
person = {"name": "Bob", "age": 25}
print(f"{person['name']} is {person['age']} years old")
# Object attributes
class User:
def __init__(self, name, email):
self.name = name
self.email = email
def __str__(self):
return f"User({self.name}, {self.email})"
user = User("Charlie", "charlie@email.com")
print(f"Contact: {user}") # Uses __str__String encoding converts text to bytes, and decoding converts bytes to text:
Basic Encoding/Decoding:
# String to bytes (encoding)
text = "Hello, 世界"
utf8_bytes = text.encode('utf-8')
print(utf8_bytes) # b'Hello, \xe4\xb8\x96\xe7\x95\x8c'
print(type(utf8_bytes)) # <class 'bytes'>
# Bytes to string (decoding)
decoded = utf8_bytes.decode('utf-8')
print(decoded) # Hello, 世界
# Different encodings
text = "Café"
print(text.encode('utf-8')) # b'Caf\xc3\xa9'
print(text.encode('latin-1')) # b'Caf\xe9'
print(text.encode('ascii', errors='ignore')) # b'Caf'Error Handling:
text = "Café"
# Strict (default) - raises error for unsupported characters
try:
text.encode('ascii', errors='strict')
except UnicodeEncodeError as e:
print(f"Strict failed: {e}")
# Ignore - removes unsupported characters
print(text.encode('ascii', errors='ignore')) # b'Caf'
# Replace - replaces with placeholder
print(text.encode('ascii', errors='replace')) # b'Caf?'
# Xmlcharrefreplace - replaces with XML entity
print(text.encode('ascii', errors='xmlcharrefreplace')) # b'Café'
# Backslashreplace - uses Python escape sequences
print(text.encode('ascii', errors='backslashreplace')) # b'Caf\\xe9'Working with Files:
# Writing text with specific encoding
with open('file.txt', 'w', encoding='utf-8') as f:
f.write("Hello, 世界")
# Reading with encoding detection
import chardet
# Detect encoding of unknown bytes
with open('unknown_file.txt', 'rb') as f:
raw_data = f.read()
result = chardet.detect(raw_data)
encoding = result['encoding']
confidence = result['confidence']
print(f"Detected encoding: {encoding} with {confidence} confidence")
# Decode with detected encoding
text = raw_data.decode(encoding)Regular expressions provide powerful pattern matching:
Basic Pattern Matching:
import re
# Search for pattern
text = "The rain in Spain"
pattern = r"rain"
match = re.search(pattern, text)
if match:
print(f"Found '{match.group()}' at position {match.start()}")
# Find all matches
matches = re.findall(r"ai", text) # ['ai', 'ai']
print(matches)
# Find all with positions
for match in re.finditer(r"ai", text):
print(f"Match '{match.group()}' at {match.start()}-{match.end()}")Pattern Syntax:
# Character classes
pattern = r"[aeiou]" # Any vowel
print(re.findall(pattern, "hello")) # ['e', 'o']
pattern = r"[^aeiou]" # Not a vowel (consonants)
print(re.findall(pattern, "hello")) # ['h', 'l', 'l']
# Predefined classes
print(re.findall(r"\d", "User123")) # ['1', '2', '3'] (digits)
print(re.findall(r"\D", "User123")) # ['U', 's', 'e', 'r'] (non-digits)
print(re.findall(r"\w", "Hello123")) # ['H','e','l','l','o','1','2','3'] (word chars)
print(re.findall(r"\W", "Hello 123!")) # [' ', '!'] (non-word chars)
print(re.findall(r"\s", "Hello world")) # [' '] (whitespace)
print(re.findall(r"\S", "Hello world")) # All non-whitespace
# Anchors
text = "cat category catalog"
print(re.findall(r"^cat", text)) # ['cat'] (start of string)
print(re.findall(r"cat$", text)) # [] (end of string)
print(re.findall(r"\bcat\b", text)) # ['cat'] (word boundary)
# Quantifiers
text = "color colour colouur"
print(re.findall(r"colou?r", text)) # ['color', 'colour'] (0 or 1 u)
print(re.findall(r"colou*r", text)) # ['color', 'colour', 'colouur'] (0 or more)
print(re.findall(r"colou+r", text)) # ['colour', 'colouur'] (1 or more)
print(re.findall(r"colou{2}r", text)) # ['colouur'] (exactly 2)
print(re.findall(r"colou{1,2}r", text)) # ['colour', 'colouur'] (1 to 2)Groups and Capturing:
# Capturing groups
text = "John: 25, Jane: 30, Bob: 35"
pattern = r"(\w+): (\d+)"
matches = re.findall(pattern, text)
print(matches) # [('John', '25'), ('Jane', '30'), ('Bob', '35')]
# Named groups
pattern = r"(?P<name>\w+): (?P<age>\d+)"
for match in re.finditer(pattern, text):
print(f"{match.group('name')} is {match.group('age')} years old")
# Non-capturing groups
text = "color colour"
pattern = r"col(?:ou)?r" # (?:...) doesn't capture
print(re.findall(pattern, text)) # ['color', 'colour']
# Backreferences
text = "hello hello world world"
pattern = r"(\w+) \1" # Match repeated word
print(re.findall(pattern, text)) # ['hello', 'world']Substitution and Splitting:
# Substitution
text = "Hello, World!"
result = re.sub(r"World", "Python", text)
print(result) # Hello, Python!
# Substitution with function
def uppercase(match):
return match.group(0).upper()
text = "hello world python"
result = re.sub(r"\b\w+\b", uppercase, text)
print(result) # HELLO WORLD PYTHON
# Limiting substitutions
text = "one one two two three three"
result = re.sub(r"one|two", "X", text, count=2)
print(result) # X X two two three three
# Splitting
text = "apple,banana;orange:grape"
result = re.split(r"[,;:]", text)
print(result) # ['apple', 'banana', 'orange', 'grape']
# Split with max splits
text = "one:two:three:four"
result = re.split(r":", text, maxsplit=2)
print(result) # ['one', 'two', 'three:four']Compilation for Performance:
# Compile pattern for reuse
email_pattern = re.compile(r"^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$")
# Validate multiple emails
emails = ["user@example.com", "invalid-email", "another@test.org"]
for email in emails:
if email_pattern.match(email):
print(f"{email} is valid")
else:
print(f"{email} is invalid")
# Flags
pattern = re.compile(r"python", re.IGNORECASE)
print(pattern.findall("Python PYTHON python")) # ['Python', 'PYTHON', 'python']
# Multiline mode
text = "Line1\nLine2\nLine3"
pattern = re.compile(r"^Line\d", re.MULTILINE)
print(pattern.findall(text)) # ['Line1', 'Line2', 'Line3']Practical Examples:
# Email validation
def validate_email(email):
pattern = r"^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$"
return bool(re.match(pattern, email))
# Phone number extraction
def extract_phone_numbers(text):
pattern = r"\(?\d{3}\)?[-.\s]?\d{3}[-.\s]?\d{4}"
return re.findall(pattern, text)
# URL parsing
def parse_url(url):
pattern = r"^(?P<protocol>https?)://(?P<domain>[^/:]+)(?::(?P<port>\d+))?(?P<path>/.*)?$"
match = re.match(pattern, url)
return match.groupdict() if match else None
# HTML tag stripping
def strip_tags(html):
pattern = r"<[^>]+>"
return re.sub(pattern, "", html)
# Password strength validation
def check_password_strength(password):
patterns = {
'length': r".{8,}",
'uppercase': r"[A-Z]",
'lowercase': r"[a-z]",
'digit': r"\d",
'special': r"[!@#$%^&*]"
}
score = 0
for name, pattern in patterns.items():
if re.search(pattern, password):
score += 1
return scorePython 3 uses Unicode for all strings, making internationalization straightforward:
Unicode Basics:
# Unicode characters directly in strings
text = "Hello, 世界" # Chinese characters
print(text)
# Unicode escape sequences
print("\u0041") # 'A' (hex)
print("\x41") # 'A' (hex byte)
print("\U0001F600") # 😀 (smiley emoji)
# Unicode names
import unicodedata
char = 'é'
print(unicodedata.name(char)) # 'LATIN SMALL LETTER E WITH ACUTE'
print(unicodedata.lookup('LATIN SMALL LETTER E WITH ACUTE')) # 'é'
# Character properties
print(unicodedata.category('A')) # 'Lu' (Letter, uppercase)
print(unicodedata.category('1')) # 'Nd' (Number, decimal)
print(unicodedata.category('!')) # 'Po' (Punctuation)Normalization:
import unicodedata
# Different representations of the same character
e_acute1 = 'é' # Single character
e_acute2 = 'e\u0301' # e + combining acute accent
print(len(e_acute1)) # 1
print(len(e_acute2)) # 2
print(e_acute1 == e_acute2) # False (different representations)
# Normalize to NFC (composed form)
normalized = unicodedata.normalize('NFC', e_acute2)
print(len(normalized)) # 1
print(normalized == e_acute1) # True
# Normalize to NFD (decomposed form)
decomposed = unicodedata.normalize('NFD', e_acute1)
print(len(decomposed)) # 2
# Use cases for normalization
def normalize_text(text, form='NFC'):
"""Normalize text for consistent comparison."""
return unicodedata.normalize(form, text)
# Case-insensitive comparison with normalization
def case_insensitive_equal(s1, s2):
"""Compare strings case-insensitively with Unicode normalization."""
s1 = unicodedata.normalize('NFC', s1.lower())
s2 = unicodedata.normalize('NFC', s2.lower())
return s1 == s2Unicode Properties and Manipulation:
# Checking Unicode categories
def get_script(char):
"""Get Unicode script of a character."""
try:
return unicodedata.name(char).split()[0]
except (TypeError, ValueError):
return 'Unknown'
# Remove diacritical marks
import unicodedata
def remove_diacritics(text):
"""Remove accents and diacritical marks."""
normalized = unicodedata.normalize('NFD', text)
# Remove combining characters (diacritical marks)
return ''.join(c for c in normalized
if unicodedata.category(c) != 'Mn')
text = "Café crème"
print(remove_diacritics(text)) # "Cafe creme"
# Unicode-aware string operations
def reverse_string_unicode(s):
"""Reverse string respecting Unicode grapheme clusters."""
# Simple reversal may break grapheme clusters
# For proper handling, use regex with grapheme cluster matching
import regex # pip install regex
graphemes = regex.findall(r'\X', s)
return ''.join(reversed(graphemes))
# Example with emoji and combining characters
text = "e\u0301" # 'e' + combining acute
print(reverse_string_unicode(text)) # Preserves graphemeUnicode in Practice:
# Reading files with unknown encoding
def read_file_with_fallback(filename):
encodings = ['utf-8', 'latin-1', 'cp1252', 'iso-8859-1']
for encoding in encodings:
try:
with open(filename, 'r', encoding=encoding) as f:
return f.read()
except UnicodeDecodeError:
continue
raise ValueError(f"Could not decode {filename}")
# Writing with BOM for Windows compatibility
with open('file.txt', 'w', encoding='utf-8-sig') as f:
f.write("Hello with BOM") # Adds UTF-8 BOM
# Unicode-aware slug generation
import unicodedata
import re
def slugify(text):
"""Convert text to URL-friendly slug."""
# Normalize and remove diacritics
text = unicodedata.normalize('NFKD', text)
text = text.encode('ascii', 'ignore').decode('ascii')
# Convert to lowercase and replace spaces
text = text.lower()
text = re.sub(r'[^\w\s-]', '', text)
text = re.sub(r'[-\s]+', '-', text)
return text.strip('-')
print(slugify("Café crème, s'il vous plaît!")) # "cafe-creme-sil-vous-plait"Lists are mutable sequences, perfect for ordered collections that may change:
Creating Lists:
# Empty list
empty_list = []
empty_list = list()
# List with initial values
fruits = ["apple", "banana", "cherry"]
numbers = [1, 2, 3, 4, 5]
mixed = [1, "hello", 3.14, True]
# Using list() constructor
chars = list("Python") # ['P', 'y', 't', 'h', 'o', 'n']
range_list = list(range(5)) # [0, 1, 2, 3, 4]
# List comprehension
squares = [x**2 for x in range(10)]Adding Elements:
fruits = ["apple", "banana"]
# Append to end
fruits.append("cherry")
print(fruits) # ['apple', 'banana', 'cherry']
# Insert at specific position
fruits.insert(1, "blueberry")
print(fruits) # ['apple', 'blueberry', 'banana', 'cherry']
# Extend with another list
more_fruits = ["date", "elderberry"]
fruits.extend(more_fruits)
print(fruits) # ['apple', 'blueberry', 'banana', 'cherry', 'date', 'elderberry']
# Using + operator (creates new list)
combined = fruits + ["fig", "grape"]
print(combined)
# Using * operator (repetition)
repeated = [1, 2] * 3 # [1, 2, 1, 2, 1, 2]Removing Elements:
fruits = ["apple", "banana", "cherry", "banana", "date"]
# Remove by value (first occurrence)
fruits.remove("banana")
print(fruits) # ['apple', 'cherry', 'banana', 'date']
# Remove by index with pop (returns removed value)
popped = fruits.pop(1) # Removes and returns index 1
print(popped) # 'cherry'
print(fruits) # ['apple', 'banana', 'date']
# Pop from end
last = fruits.pop()
print(last) # 'date'
print(fruits) # ['apple', 'banana']
# Delete by index or slice
del fruits[0]
print(fruits) # ['banana']
# Clear all elements
fruits.clear()
print(fruits) # []Searching and Counting:
numbers = [1, 2, 3, 2, 4, 2, 5]
# Count occurrences
print(numbers.count(2)) # 3
# Find index of first occurrence
print(numbers.index(2)) # 1
print(numbers.index(2, 2)) # 3 (start from index 2)
print(numbers.index(2, 4)) # 5 (start from index 4)
# Handle missing elements safely
try:
pos = numbers.index(10)
except ValueError:
print("Element not found")
# Check membership
print(3 in numbers) # True
print(10 in numbers) # FalseSorting and Reversing:
numbers = [3, 1, 4, 1, 5, 9, 2]
# sort() modifies list in-place
numbers.sort()
print(numbers) # [1, 1, 2, 3, 4, 5, 9]
# sort descending
numbers.sort(reverse=True)
print(numbers) # [9, 5, 4, 3, 2, 1, 1]
# sorted() returns new sorted list
original = [3, 1, 4, 1]
sorted_list = sorted(original)
print(original) # [3, 1, 4, 1] (unchanged)
print(sorted_list) # [1, 1, 3, 4]
# Custom sorting with key
words = ["banana", "apple", "cherry", "date"]
words.sort(key=len) # Sort by length
print(words) # ['date', 'apple', 'banana', 'cherry']
words.sort(key=lambda x: x[-1]) # Sort by last character
print(words) # ['banana', 'apple', 'date', 'cherry']
# Reverse list
numbers = [1, 2, 3, 4, 5]
numbers.reverse()
print(numbers) # [5, 4, 3, 2, 1]
# reversed() returns iterator
for num in reversed([1, 2, 3]):
print(num) # 3, 2, 1Other Useful Methods:
# Copying lists
original = [1, 2, 3]
# Shallow copy
copy1 = original.copy()
copy2 = original[:]
copy3 = list(original)
# Deep copy for nested lists
import copy
nested = [[1, 2], [3, 4]]
deep_copy = copy.deepcopy(nested)
# Length
print(len([1, 2, 3])) # 3
# Minimum and maximum
print(min([3, 1, 4, 2])) # 1
print(max([3, 1, 4, 2])) # 4
# Sum (for numeric lists)
print(sum([1, 2, 3, 4])) # 10
# Any and All
print(any([False, True, False])) # True
print(all([True, True, False])) # FalseList comprehensions provide concise syntax for creating lists:
Basic Comprehensions:
# Traditional approach
squares = []
for x in range(10):
squares.append(x**2)
# List comprehension
squares = [x**2 for x in range(10)]
print(squares) # [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
# With conditional
evens = [x for x in range(20) if x % 2 == 0]
print(evens) # [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]
# With if-else (ternary)
parity = ["even" if x % 2 == 0 else "odd" for x in range(5)]
print(parity) # ['even', 'odd', 'even', 'odd', 'even']Nested Comprehensions:
# Nested loops
matrix = [[i * j for j in range(1, 4)] for i in range(1, 4)]
print(matrix) # [[1, 2, 3], [2, 4, 6], [3, 6, 9]]
# Flattening a matrix
matrix = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
flattened = [num for row in matrix for num in row]
print(flattened) # [1, 2, 3, 4, 5, 6, 7, 8, 9]
# Cartesian product
colors = ["red", "green"]
sizes = ["S", "M", "L"]
products = [(color, size) for color in colors for size in sizes]
print(products) # [('red', 'S'), ('red', 'M'), ('red', 'L'), ('green', 'S'), ('green', 'M'), ('green', 'L')]Advanced Comprehensions:
# Transforming data
words = ["hello", "world", "python"]
uppercase = [word.upper() for word in words]
print(uppercase) # ['HELLO', 'WORLD', 'PYTHON']
# Filtering with multiple conditions
numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
filtered = [x for x in numbers if x % 2 == 0 and x > 5]
print(filtered) # [6, 8, 10]
# Using functions
def is_prime(n):
if n < 2:
return False
for i in range(2, int(n**0.5) + 1):
if n % i == 0:
return False
return True
primes = [x for x in range(50) if is_prime(x)]
print(primes) # [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47]
# String manipulation
sentence = "the quick brown fox jumps over the lazy dog"
words = sentence.split()
word_lengths = [len(word) for word in words]
print(word_lengths) # [3, 5, 5, 3, 4, 4, 3, 4, 3]
# Removing duplicates while preserving order (Python 3.7+)
items = [3, 1, 2, 1, 3, 4, 2, 5]
unique = list(dict.fromkeys(items))
print(unique) # [3, 1, 2, 4, 5]Dictionary and Set Comprehensions:
# Dictionary comprehension
squares_dict = {x: x**2 for x in range(5)}
print(squares_dict) # {0: 0, 1: 1, 2: 4, 3: 9, 4: 16}
# Set comprehension
numbers = [1, 2, 2, 3, 3, 3, 4, 4, 4, 4]
unique_squares = {x**2 for x in numbers}
print(unique_squares) # {16, 1, 4, 9}
# Filtering dictionary
ages = {"Alice": 30, "Bob": 25, "Charlie": 35, "Diana": 28}
over_30 = {name: age for name, age in ages.items() if age > 30}
print(over_30) # {'Alice': 30, 'Charlie': 35}
# Transforming dictionary
names = ["Alice", "Bob", "Charlie"]
name_lengths = {name: len(name) for name in names}
print(name_lengths) # {'Alice': 5, 'Bob': 3, 'Charlie': 7}Performance Considerations:
import timeit
# List comprehension vs. loop
def with_loop():
result = []
for i in range(1000):
result.append(i**2)
return result
def with_comprehension():
return [i**2 for i in range(1000)]
# Comprehensions are generally faster
loop_time = timeit.timeit(with_loop, number=10000)
comp_time = timeit.timeit(with_comprehension, number=10000)
print(f"Loop: {loop_time:.4f}, Comprehension: {comp_time:.4f}")
# Generator expressions for memory efficiency
# (lazy evaluation)
large_sum = sum(x**2 for x in range(1000000)) # No list createdLists can contain other lists, creating multi-dimensional structures:
Creating Nested Lists:
# Matrix (2D list)
matrix = [
[1, 2, 3],
[4, 5, 6],
[7, 8, 9]
]
# Jagged array (rows of different lengths)
jagged = [
[1, 2, 3],
[4, 5],
[6, 7, 8, 9]
]
# Creating with comprehensions
size = 3
identity = [[1 if i == j else 0 for j in range(size)] for i in range(size)]
print(identity) # [[1, 0, 0], [0, 1, 0], [0, 0, 1]]Accessing Elements:
matrix = [
[1, 2, 3],
[4, 5, 6],
[7, 8, 9]
]
# Access single element
print(matrix[0][1]) # 2 (row 0, column 1)
# Access entire row
print(matrix[1]) # [4, 5, 6]
# Access column (using comprehension)
column1 = [row[1] for row in matrix]
print(column1) # [2, 5, 8]
# Access submatrix
submatrix = [row[1:3] for row in matrix[1:3]]
print(submatrix) # [[5, 6], [8, 9]]Modifying Nested Lists:
matrix = [
[1, 2, 3],
[4, 5, 6],
[7, 8, 9]
]
# Modify single element
matrix[1][1] = 50
print(matrix) # [[1, 2, 3], [4, 50, 6], [7, 8, 9]]
# Modify entire row
matrix[0] = [10, 20, 30]
print(matrix) # [[10, 20, 30], [4, 50, 6], [7, 8, 9]]
# Add new row
matrix.append([10, 11, 12])
# Add column to each row
for row in matrix:
row.append(0)Deep vs Shallow Copy:
import copy
original = [[1, 2, 3], [4, 5, 6]]
# Shallow copy (copies references to inner lists)
shallow = copy.copy(original)
shallow[0][0] = 99
print(original) # [[99, 2, 3], [4, 5, 6]] (modified!)
# Deep copy (copies all nested structures)
original = [[1, 2, 3], [4, 5, 6]]
deep = copy.deepcopy(original)
deep[0][0] = 99
print(original) # [[1, 2, 3], [4, 5, 6]] (unchanged)Common Operations on Nested Lists:
# Flatten nested list
def flatten(nested_list):
"""Flatten a list of lists."""
return [item for sublist in nested_list for item in sublist]
matrix = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
flat = flatten(matrix)
print(flat) # [1, 2, 3, 4, 5, 6, 7, 8, 9]
# Deep flatten for arbitrarily nested lists
def deep_flatten(nested):
"""Flatten arbitrarily nested lists."""
result = []
for item in nested:
if isinstance(item, list):
result.extend(deep_flatten(item))
else:
result.append(item)
return result
nested = [1, [2, [3, 4], 5], 6, [7, 8]]
print(deep_flatten(nested)) # [1, 2, 3, 4, 5, 6, 7, 8]
# Transpose matrix
def transpose(matrix):
"""Transpose a matrix (swap rows and columns)."""
return [[row[i] for row in matrix] for i in range(len(matrix[0]))]
matrix = [[1, 2, 3], [4, 5, 6]]
transposed = transpose(matrix)
print(transposed) # [[1, 4], [2, 5], [3, 6]]
# Matrix multiplication
def matrix_multiply(A, B):
"""Multiply two matrices."""
result = [[0 for _ in range(len(B[0]))] for _ in range(len(A))]
for i in range(len(A)):
for j in range(len(B[0])):
for k in range(len(B)):
result[i][j] += A[i][k] * B[k][j]
return result
A = [[1, 2], [3, 4]]
B = [[5, 6], [7, 8]]
print(matrix_multiply(A, B)) # [[19, 22], [43, 50]]Sorting Techniques:
# Basic sorting
numbers = [3, 1, 4, 1, 5, 9, 2]
numbers.sort()
print(numbers) # [1, 1, 2, 3, 4, 5, 9]
# Sort with key function
words = ["banana", "apple", "Cherry", "date"]
words.sort(key=str.lower) # Case-insensitive sort
print(words) # ['apple', 'banana', 'Cherry', 'date']
# Sort by multiple criteria
students = [
{"name": "Alice", "grade": 85, "age": 20},
{"name": "Bob", "grade": 92, "age": 19},
{"name": "Charlie", "grade": 85, "age": 21},
{"name": "Diana", "grade": 92, "age": 18}
]
# Sort by grade descending, then age ascending
students.sort(key=lambda x: (-x["grade"], x["age"]))
print(students)
# Stable sort (preserves order of equal keys)
pairs = [(1, 'a'), (2, 'b'), (1, 'c'), (2, 'd')]
pairs.sort(key=lambda x: x[0])
print(pairs) # [(1, 'a'), (1, 'c'), (2, 'b'), (2, 'd')] (stable)Custom Sorting with functools.cmp_to_key:
from functools import cmp_to_key
# Custom comparison function
def compare_people(p1, p2):
"""Compare by age, then by name."""
if p1["age"] != p2["age"]:
return p1["age"] - p2["age"]
return -1 if p1["name"] < p2["name"] else 1
people = [
{"name": "Alice", "age": 30},
{"name": "Bob", "age": 25},
{"name": "Charlie", "age": 30},
{"name": "Diana", "age": 25}
]
sorted_people = sorted(people, key=cmp_to_key(compare_people))
print(sorted_people)Searching Algorithms:
Linear Search:
def linear_search(arr, target):
"""Find index of target using linear search."""
for i, value in enumerate(arr):
if value == target:
return i
return -1
# For sorted arrays, can early exit
def linear_search_sorted(arr, target):
"""Linear search with early exit for sorted arrays."""
for i, value in enumerate(arr):
if value == target:
return i
if value > target: # Array is sorted
break
return -1
# Example
numbers = [3, 7, 1, 9, 4, 2, 8]
print(linear_search(numbers, 9)) # 3
print(linear_search(numbers, 5)) # -1Binary Search (requires sorted array):
def binary_search(arr, target):
"""Find index of target using binary search."""
left, right = 0, len(arr) - 1
while left <= right:
mid = (left + right) // 2
if arr[mid] == target:
return mid
elif arr[mid] < target:
left = mid + 1
else:
right = mid - 1
return -1 # Not found
# Recursive binary search
def binary_search_recursive(arr, target, left=0, right=None):
if right is None:
right = len(arr) - 1
if left > right:
return -1
mid = (left + right) // 2
if arr[mid] == target:
return mid
elif arr[mid] < target:
return binary_search_recursive(arr, target, mid + 1, right)
else:
return binary_search_recursive(arr, target, left, mid - 1)
# Example
numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9]
print(binary_search(numbers, 7)) # 6
print(binary_search(numbers, 10)) # -1
# Using bisect module (Python's built-in binary search)
import bisect
sorted_list = [1, 3, 5, 7, 9]
# Find insertion point
pos = bisect.bisect_left(sorted_list, 6)
print(pos) # 3 (index where 6 would be inserted)
# Check if element exists
def binary_search_bisect(arr, target):
pos = bisect.bisect_left(arr, target)
return pos if pos < len(arr) and arr[pos] == target else -1Finding Extremes:
# Minimum and maximum
numbers = [3, 7, 1, 9, 4, 2, 8]
print(min(numbers)) # 1
print(max(numbers)) # 9
# Index of minimum
def argmin(arr):
return min(range(len(arr)), key=lambda i: arr[i])
print(argmin(numbers)) # 2 (index of 1)
# n largest/smallest
import heapq
numbers = [3, 7, 1, 9, 4, 2, 8]
print(heapq.nlargest(3, numbers)) # [9, 8, 7]
print(heapq.nsmallest(3, numbers)) # [1, 2, 3]
# For complex objects
students = [
{"name": "Alice", "grade": 85},
{"name": "Bob", "grade": 92},
{"name": "Charlie", "grade": 78},
{"name": "Diana", "grade": 95}
]
top_2 = heapq.nlargest(2, students, key=lambda x: x["grade"])
print(top_2) # [{'name': 'Diana', 'grade': 95}, {'name': 'Bob', 'grade': 92}]Tuples are immutable sequences, ideal for fixed collections:
Creating Tuples:
# Empty tuple
empty_tuple = ()
empty_tuple = tuple()
# Single-element tuple (note the comma)
single = (1,) # Without comma, it's just an integer
print(type(single)) # <class 'tuple'>
# Multiple elements
coordinates = (10, 20)
person = ("Alice", 30, "Engineer")
# Without parentheses (tuple packing)
point = 10, 20
print(type(point)) # <class 'tuple'>
# Using tuple() constructor
numbers = tuple([1, 2, 3])
chars = tuple("hello") # ('h', 'e', 'l', 'l', 'o')Tuple Packing and Unpacking:
# Packing
person = "Alice", 30, "Engineer"
# Unpacking
name, age, profession = person
print(name) # "Alice"
print(age) # 30
print(profession) # "Engineer"
# Unpacking with *
first, *rest = (1, 2, 3, 4, 5)
print(first) # 1
print(rest) # [2, 3, 4, 5]
*beginning, last = (1, 2, 3, 4, 5)
print(beginning) # [1, 2, 3, 4]
print(last) # 5
first, *middle, last = (1, 2, 3, 4, 5)
print(first) # 1
print(middle) # [2, 3, 4]
print(last) # 5
# Swapping variables (uses tuple packing/unpacking)
a, b = 10, 20
a, b = b, a
print(a, b) # 20 10
# Returning multiple values from function
def get_min_max(numbers):
return min(numbers), max(numbers)
values = [3, 1, 4, 1, 5, 9, 2]
minimum, maximum = get_min_max(values)
print(minimum, maximum) # 1 9Tuple Operations:
# Concatenation
t1 = (1, 2, 3)
t2 = (4, 5, 6)
t3 = t1 + t2
print(t3) # (1, 2, 3, 4, 5, 6)
# Repetition
t = (1, 2) * 3
print(t) # (1, 2, 1, 2, 1, 2)
# Membership
print(3 in (1, 2, 3)) # True
print(5 not in (1, 2, 3)) # True
# Indexing and slicing
t = (10, 20, 30, 40, 50)
print(t[0]) # 10
print(t[-1]) # 50
print(t[1:4]) # (20, 30, 40)
print(t[::-1]) # (50, 40, 30, 20, 10)
# Length
print(len((1, 2, 3))) # 3
# Count and index
t = (1, 2, 3, 2, 4, 2)
print(t.count(2)) # 3
print(t.index(3)) # 2
print(t.index(2, 2)) # 3 (start from index 2)Tuple Immutability:
t = (1, 2, 3)
# This would raise TypeError
# t[0] = 10
# But if tuple contains mutable objects, those objects can change
t = ([1, 2], [3, 4])
t[0].append(3)
print(t) # ([1, 2, 3], [3, 4])
# Reassigning the variable (different tuple)
t = (1, 2, 3) # This is fine - new tuple assignedNamed tuples provide readable, lightweight data structures:
Basic Named Tuples:
from collections import namedtuple
# Define a named tuple type
Point = namedtuple('Point', ['x', 'y'])
# Create instances
p1 = Point(10, 20)
p2 = Point(x=30, y=40)
# Access by attribute
print(p1.x) # 10
print(p1.y) # 20
# Access by index (still works)
print(p1[0]) # 10
# Unpacking
x, y = p1
print(x, y) # 10 20
# Field names can be strings with spaces
Person = namedtuple('Person', 'name age city')
alice = Person('Alice', 30, 'New York')
print(alice.name) # AliceNamed Tuple Methods:
# _asdict() - convert to dictionary
person = Person('Bob', 25, 'Los Angeles')
person_dict = person._asdict()
print(person_dict) # {'name': 'Bob', 'age': 25, 'city': 'Los Angeles'}
# _replace() - create new instance with replaced fields
person2 = person._replace(age=26, city='San Francisco')
print(person2) # Person(name='Bob', age=26, city='San Francisco')
# _fields - get field names
print(Point._fields) # ('x', 'y')
# _make() - create from iterable
values = ['Charlie', 35, 'Chicago']
charlie = Person._make(values)
print(charlie) # Person(name='Charlie', age=35, city='Chicago')Advanced Named Tuple Features:
# Default values (Python 3.7+)
from typing import NamedTuple
class Employee(NamedTuple):
"""Employee record with default values."""
name: str
id: int
department: str = 'Engineering'
salary: float = 50000.0
emp1 = Employee('Alice', 1001)
print(emp1) # Employee(name='Alice', id=1001, department='Engineering', salary=50000.0)
emp2 = Employee('Bob', 1002, 'Marketing', 60000.0)
print(emp2)
# Docstrings and type hints included
help(Employee)
# Named tuples are still tuples
print(isinstance(emp1, tuple)) # True
# Can have methods
class Point(NamedTuple):
x: float
y: float
def distance_from_origin(self):
return (self.x**2 + self.y**2)**0.5
def __str__(self):
return f"Point({self.x}, {self.y})"
p = Point(3, 4)
print(p.distance_from_origin()) # 5.0
print(p) # Point(3, 4)Practical Examples:
# Database record representation
from collections import namedtuple
Stock = namedtuple('Stock', 'symbol price volume')
# Reading data
stocks = [
Stock('AAPL', 150.25, 1000000),
Stock('GOOGL', 2750.50, 500000),
Stock('MSFT', 300.75, 750000)
]
# Processing
total_value = sum(s.price * s.volume for s in stocks)
print(f"Total value: ${total_value:,.2f}")
# Filtering
large_trades = [s for s in stocks if s.volume > 600000]
print(large_trades)
# Returning multiple values from function
def get_user_info(user_id):
# Simulate database lookup
return User('Alice', 30, 'alice@email.com')
User = namedtuple('User', 'name age email')
user = get_user_info(123)
print(f"{user.name} ({user.age}) - {user.email}")Sets store unique, unordered collections:
Creating Sets:
# Empty set (must use set(), {} creates empty dict)
empty_set = set()
# Set with values
fruits = {'apple', 'banana', 'cherry'}
print(fruits) # Order may vary
# From list (removes duplicates)
numbers = set([1, 2, 2, 3, 3, 3])
print(numbers) # {1, 2, 3}
# From string
chars = set('hello')
print(chars) # {'h', 'e', 'l', 'o'} (note: only one 'l')
# Set comprehension
squares = {x**2 for x in range(10)}
print(squares) # {0, 1, 4, 9, 16, 25, 36, 49, 64, 81}Basic Set Operations:
# Adding elements
s = {1, 2, 3}
s.add(4)
print(s) # {1, 2, 3, 4}
s.add(2) # No effect (already present)
# Adding multiple elements
s.update([5, 6, 7])
print(s) # {1, 2, 3, 4, 5, 6, 7}
# Removing elements
s = {1, 2, 3, 4, 5}
# remove() raises KeyError if not present
s.remove(3)
print(s) # {1, 2, 4, 5}
# discard() doesn't raise error
s.discard(10) # No error
# pop() removes and returns arbitrary element
element = s.pop()
print(f"Removed: {element}")
print(s)
# clear() removes all elements
s.clear()
print(s) # set()
# Copy sets
original = {1, 2, 3}
copy1 = original.copy()
copy2 = set(original)Set Mathematics:
A = {1, 2, 3, 4, 5}
B = {4, 5, 6, 7, 8}
# Union (elements in A or B)
print(A | B) # {1, 2, 3, 4, 5, 6, 7, 8}
print(A.union(B))
# Intersection (elements in both)
print(A & B) # {4, 5}
print(A.intersection(B))
# Difference (elements in A but not B)
print(A - B) # {1, 2, 3}
print(A.difference(B))
# Symmetric difference (elements in A or B but not both)
print(A ^ B) # {1, 2, 3, 6, 7, 8}
print(A.symmetric_difference(B))
# Subset and superset
C = {1, 2, 3}
print(C.issubset(A)) # True (C ⊆ A)
print(A.issuperset(C)) # True (A ⊇ C)
# Disjoint sets (no common elements)
D = {9, 10, 11}
print(A.isdisjoint(D)) # True
# Update operations (modify original)
A.update(B) # Add elements from B
A.intersection_update(B) # Keep only elements in both
A.difference_update(B) # Remove elements in B
A.symmetric_difference_update(B) # Keep symmetric differenceSet Comparisons:
# Equality (same elements, order irrelevant)
print({1, 2, 3} == {3, 2, 1}) # True
# Proper subset
A = {1, 2, 3}
B = {1, 2, 3, 4, 5}
C = {1, 2, 3}
print(A < B) # True (A is proper subset of B)
print(A < C) # False (A equals C, not proper subset)
print(A <= C) # True (A is subset of C, allows equality)
print(B > A) # True (B is proper superset of A)
print(B >= A) # True (B is superset of A)Set Membership and Operations:
# Membership testing (very fast - O(1))
fruits = {'apple', 'banana', 'cherry'}
print('banana' in fruits) # True
print('grape' in fruits) # False
# Set operations with multiple sets
set1 = {1, 2, 3, 4}
set2 = {3, 4, 5, 6}
set3 = {4, 5, 6, 7}
# Union of multiple sets
print(set1 | set2 | set3)
# Intersection of multiple sets
print(set1 & set2 & set3) # {4}
# Checking if any common elements
if set1 & set2:
print("Sets intersect")
else:
print("Sets are disjoint")Practical Set Examples:
# Remove duplicates from sequence
def unique_elements(sequence):
return list(set(sequence))
numbers = [1, 2, 2, 3, 3, 3, 4, 4, 4, 4]
print(unique_elements(numbers)) # [1, 2, 3, 4]
# Find common elements
def find_common(list1, list2):
return list(set(list1) & set(list2))
list1 = [1, 2, 3, 4, 5]
list2 = [4, 5, 6, 7, 8]
print(find_common(list1, list2)) # [4, 5]
# Find unique elements in first list
def unique_in_first(list1, list2):
return list(set(list1) - set(list2))
print(unique_in_first(list1, list2)) # [1, 2, 3]
# Check if two sequences contain same elements (ignoring order)
def same_elements(seq1, seq2):
return set(seq1) == set(seq2)
print(same_elements([1, 2, 3], [3, 2, 1])) # True
print(same_elements([1, 2, 3], [1, 2, 3, 3])) # True (duplicates ignored)
# Word uniqueness in text
def unique_words(text):
# Convert to lowercase and split
words = text.lower().split()
# Remove punctuation
words = [word.strip('.,!?;:') for word in words]
return set(words)
sentence = "The quick brown fox jumps over the lazy dog"
print(unique_words(sentence))
# Find anagrams
def find_anagrams(word, candidates):
"""
Find anagrams of word from candidates.
"""
word_sorted = sorted(word.lower())
return [c for c in candidates
if sorted(c.lower()) == word_sorted
and c.lower() != word.lower()]
word = "listen"
candidates = ["enlist", "google", "inlets", "banana", "silent"]
print(find_anagrams(word, candidates)) # ['enlist', 'inlets', 'silent']
# Track visited items (for loops/recursion)
visited = set()
def dfs(graph, node, visited):
if node in visited:
return
visited.add(node)
print(f"Visiting {node}")
for neighbor in graph[node]:
dfs(graph, neighbor, visited)Frozen sets are immutable sets, usable as dictionary keys:
Creating Frozen Sets:
# Empty frozen set
empty_frozen = frozenset()
# From iterable
numbers = frozenset([1, 2, 3, 3, 4])
print(numbers) # frozenset({1, 2, 3, 4})
# From regular set
s = {1, 2, 3}
fs = frozenset(s)Frozen Set Operations:
fs1 = frozenset([1, 2, 3, 4])
fs2 = frozenset([3, 4, 5, 6])
# Read-only operations work as with regular sets
print(fs1 | fs2) # frozenset({1, 2, 3, 4, 5, 6})
print(fs1 & fs2) # frozenset({3, 4})
print(fs1 - fs2) # frozenset({1, 2})
print(fs1 ^ fs2) # frozenset({1, 2, 5, 6})
print(fs1.issubset(fs2)) # False
# Membership testing
print(3 in fs1) # True
# Modification operations are not allowed
# fs1.add(5) # AttributeError
# fs1.remove(1) # AttributeErrorUsing Frozen Sets as Dictionary Keys:
# Regular sets can't be dictionary keys (unhashable)
# d = {{1,2}: "value"} # TypeError
# Frozen sets can be keys
d = {
frozenset([1, 2]): "set of 1 and 2",
frozenset([3, 4]): "set of 3 and 4"
}
print(d[frozenset([1, 2])]) # "set of 1 and 2"
# Use case: caching function results based on set of arguments
def expensive_computation(items):
# Convert to frozen set for cache key
items_key = frozenset(items) if isinstance(items, set) else items
# ... computation
return result
# Store sets in sets
set_of_sets = {frozenset([1, 2]), frozenset([3, 4])}
print(set_of_sets)Practical Frozen Set Applications:
# Graph representation with frozenset edges
class Graph:
def __init__(self):
self.edges = set() # Set of frozenset edges
def add_edge(self, u, v):
# Undirected graph: edge is frozenset of vertices
edge = frozenset([u, v])
self.edges.add(edge)
def has_edge(self, u, v):
return frozenset([u, v]) in self.edges
graph = Graph()
graph.add_edge(1, 2)
graph.add_edge(2, 3)
print(graph.has_edge(1, 2)) # True
print(graph.has_edge(2, 1)) # True (undirected)
print(graph.has_edge(1, 3)) # False
# Configuration with immutable sets
VALID_CONFIGS = {
frozenset(['debug', 'verbose']): "Debug mode with verbose output",
frozenset(['debug']): "Debug mode only",
frozenset(['production', 'optimized']): "Production optimized",
frozenset(['production']): "Production standard"
}
def get_config_description(flags):
"""Get description for a set of configuration flags."""
flags_key = frozenset(flags)
return VALID_CONFIGS.get(flags_key, "Unknown configuration")
print(get_config_description(['debug', 'verbose']))
print(get_config_description(['production']))
# Cache with set-based keys
from functools import lru_cache
@lru_cache(maxsize=128)
def analyze_subset(elements):
"""Analyze a subset (passed as frozenset)."""
# Convert back to set for operations if needed
elements_set = set(elements)
# ... analysis
return sum(elements_set) # Simplified example
# Usage (must pass hashable frozenset)
result = analyze_subset(frozenset([1, 2, 3, 4, 5]))
print(result)Dictionaries store key-value pairs with fast lookup:
Creating Dictionaries:
# Empty dictionary
empty_dict = {}
empty_dict = dict()
# With initial values
person = {
"name": "Alice",
"age": 30,
"city": "New York"
}
# Using dict() constructor
person = dict(name="Alice", age=30, city="New York")
# From list of tuples
pairs = [("name", "Alice"), ("age", 30), ("city", "New York")]
person = dict(pairs)
# From zip
keys = ["name", "age", "city"]
values = ["Alice", 30, "New York"]
person = dict(zip(keys, values))
# Dictionary comprehension
squares = {x: x**2 for x in range(5)}
print(squares) # {0: 0, 1: 1, 2: 4, 3: 9, 4: 16}Accessing Values:
person = {"name": "Alice", "age": 30, "city": "New York"}
# Direct access (raises KeyError if missing)
print(person["name"]) # "Alice"
# get() method (returns None or default if missing)
print(person.get("age")) # 30
print(person.get("country")) # None
print(person.get("country", "Unknown")) # "Unknown"
# setdefault() - get value or set default if missing
city = person.setdefault("city", "Boston") # Returns "New York" (existing)
country = person.setdefault("country", "USA") # Sets and returns "USA"
print(person) # Now includes "country": "USA"
# Check if key exists
print("name" in person) # True
print("salary" in person) # FalseModifying Dictionaries:
person = {"name": "Alice", "age": 30}
# Add or update single key
person["city"] = "New York" # Add new
person["age"] = 31 # Update existing
print(person) # {'name': 'Alice', 'age': 31, 'city': 'New York'}
# Update with another dictionary
updates = {"age": 32, "country": "USA"}
person.update(updates)
print(person) # {'name': 'Alice', 'age': 32, 'city': 'New York', 'country': 'USA'}
# Update with key-value pairs
person.update(occupation="Engineer", salary=75000)
print(person)
# Update with list of tuples
person.update([("hobby", "photography"), ("languages", ["English", "Spanish"])])
# Merge dictionaries (Python 3.9+)
dict1 = {"a": 1, "b": 2}
dict2 = {"c": 3, "d": 4}
merged = dict1 | dict2
print(merged) # {'a': 1, 'b': 2, 'c': 3, 'd': 4}
# Update with |= operator
dict1 |= dict2
print(dict1) # {'a': 1, 'b': 2, 'c': 3, 'd': 4}Removing Items:
person = {"name": "Alice", "age": 30, "city": "New York", "country": "USA"}
# pop() - remove and return value
age = person.pop("age")
print(age) # 30
print(person) # {'name': 'Alice', 'city': 'New York', 'country': 'USA'}
# pop with default (avoids KeyError)
salary = person.pop("salary", 0) # Returns 0
print(salary)
# popitem() - remove and return last inserted item (Python 3.7+)
key, value = person.popitem()
print(f"Removed: {key}: {value}")
# del statement
del person["city"]
print(person) # {'name': 'Alice', 'country': 'USA'}
# clear() - remove all items
person.clear()
print(person) # {}Dictionary Views:
person = {"name": "Alice", "age": 30, "city": "New York"}
# keys() - view of keys
keys = person.keys()
print(keys) # dict_keys(['name', 'age', 'city'])
print(list(keys)) # Convert to list
# values() - view of values
values = person.values()
print(values) # dict_values(['Alice', 30, 'New York'])
# items() - view of key-value pairs
items = person.items()
print(items) # dict_items([('name', 'Alice'), ('age', 30), ('city', 'New York')])
# Views reflect changes to dictionary
person["country"] = "USA"
print(keys) # dict_keys(['name', 'age', 'city', 'country'])
# Iterating over dictionary
for key in person:
print(key)
for value in person.values():
print(value)
for key, value in person.items():
print(f"{key}: {value}")Dictionary Copying:
original = {"a": 1, "b": [2, 3], "c": 4}
# Shallow copy
shallow = original.copy()
shallow["b"].append(4) # Modifies original's list!
print(original) # {'a': 1, 'b': [2, 3, 4], 'c': 4}
# Deep copy
import copy
original = {"a": 1, "b": [2, 3], "c": 4}
deep = copy.deepcopy(original)
deep["b"].append(4)
print(original) # {'a': 1, 'b': [2, 3], 'c': 4} (unchanged)
print(deep) # {'a': 1, 'b': [2, 3, 4], 'c': 4}Dictionary comprehensions provide concise dictionary creation:
Basic Comprehensions:
# Square numbers
squares = {x: x**2 for x in range(5)}
print(squares) # {0: 0, 1: 1, 2: 4, 3: 9, 4: 16}
# Filtering
even_squares = {x: x**2 for x in range(10) if x % 2 == 0}
print(even_squares) # {0: 0, 2: 4, 4: 16, 6: 36, 8: 64}
# Transforming keys/values
words = ["hello", "world", "python"]
word_lengths = {word: len(word) for word in words}
print(word_lengths) # {'hello': 5, 'world': 5, 'python': 6}
# Swapping keys and values
original = {"a": 1, "b": 2, "c": 3}
swapped = {value: key for key, value in original.items()}
print(swapped) # {1: 'a', 2: 'b', 3: 'c'}Advanced Comprehensions:
# Conditional value assignment
numbers = [1, 2, 3, 4, 5]
parity = {n: "even" if n % 2 == 0 else "odd" for n in numbers}
print(parity) # {1: 'odd', 2: 'even', 3: 'odd', 4: 'even', 5: 'odd'}
# Nested comprehensions
matrix = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
position_values = {(i, j): matrix[i][j] for i in range(3) for j in range(3)}
print(position_values) # {(0,0):1, (0,1):2, (0,2):3, (1,0):4, (1,1):5, (1,2):6, (2,0):7, (2,1):8, (2,2):9}
# From two lists with zip
names = ["Alice", "Bob", "Charlie"]
ages = [30, 25, 35]
name_age = {name: age for name, age in zip(names, ages)}
print(name_age) # {'Alice': 30, 'Bob': 25, 'Charlie': 35}
# With conditions on both key and value
scores = {"Alice": 85, "Bob": 92, "Charlie": 78, "Diana": 95}
passed = {name: score for name, score in scores.items() if score >= 80}
print(passed) # {'Alice': 85, 'Bob': 92, 'Diana': 95}
# Transforming values
celsius = {"Monday": 20, "Tuesday": 22, "Wednesday": 19}
fahrenheit = {day: temp * 9/5 + 32 for day, temp in celsius.items()}
print(fahrenheit)Set and Dictionary Combination:
# Counting occurrences (building frequency dictionary)
text = "hello world hello python world hello"
words = text.split()
word_count = {word: words.count(word) for word in set(words)}
print(word_count) # {'hello': 3, 'world': 2, 'python': 1}
# Better way using collections.Counter
from collections import Counter
word_count = Counter(words)
print(word_count) # Counter({'hello': 3, 'world': 2, 'python': 1})
# Grouping items
from collections import defaultdict
numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
# Group by even/odd
grouped = defaultdict(list)
for n in numbers:
grouped["even" if n % 2 == 0 else "odd"].append(n)
print(dict(grouped)) # {'odd': [1, 3, 5, 7, 9], 'even': [2, 4, 6, 8, 10]}OrderedDict maintains insertion order (Python's dict now does too, but OrderedDict has extra features):
from collections import OrderedDict
# Creating OrderedDict
od = OrderedDict()
od['a'] = 1
od['b'] = 2
od['c'] = 3
print(od) # OrderedDict([('a', 1), ('b', 2), ('c', 3)])
# From regular dict (preserves order in Python 3.7+)
regular_dict = {'a': 1, 'b': 2, 'c': 3}
od = OrderedDict(regular_dict)
# Moving items
od.move_to_end('b')
print(od) # OrderedDict([('a', 1), ('c', 3), ('b', 2)])
od.move_to_end('b', last=False) # Move to beginning
print(od) # OrderedDict([('b', 2), ('a', 1), ('c', 3)])
# Pop item from either end
last = od.popitem(last=True) # Pop from end
print(last) # ('c', 3)
first = od.popitem(last=False) # Pop from beginning
print(first) # ('b', 2)
# Equality comparison (OrderedDict cares about order)
od1 = OrderedDict([('a', 1), ('b', 2)])
od2 = OrderedDict([('b', 2), ('a', 1)])
print(od1 == od2) # False (order differs)
# Regular dict equality ignores order
d1 = {'a': 1, 'b': 2}
d2 = {'b': 2, 'a': 1}
print(d1 == d2) # TrueDefaultDict provides default values for missing keys:
from collections import defaultdict
# Default factory for missing keys
# list factory
dd_list = defaultdict(list)
dd_list['fruits'].append('apple')
dd_list['fruits'].append('banana')
dd_list['vegetables'].append('carrot')
print(dd_list) # defaultdict(<class 'list'>, {'fruits': ['apple', 'banana'], 'vegetables': ['carrot']})
# int factory (for counting)
dd_int = defaultdict(int)
dd_int['a'] += 1
dd_int['b'] += 2
print(dd_int) # defaultdict(<class 'int'>, {'a': 1, 'b': 2})
# set factory
dd_set = defaultdict(set)
dd_set['fruits'].add('apple')
dd_set['fruits'].add('banana')
dd_set['vegetables'].add('carrot')
print(dd_set) # defaultdict(<class 'set'>, {'fruits': {'banana', 'apple'}, 'vegetables': {'carrot'}})
# Custom default factory
def default_value():
return "N/A"
dd_custom = defaultdict(default_value)
dd_custom['name'] = 'Alice'
print(dd_custom['name']) # 'Alice'
print(dd_custom['age']) # 'N/A' (not 'age' key)Practical DefaultDict Examples:
# Grouping items
data = [('fruit', 'apple'), ('fruit', 'banana'), ('veg', 'carrot'),
('fruit', 'cherry'), ('veg', 'broccoli')]
grouped = defaultdict(list)
for category, item in data:
grouped[category].append(item)
print(dict(grouped)) # {'fruit': ['apple', 'banana', 'cherry'], 'veg': ['carrot', 'broccoli']}
# Counting occurrences
words = ['apple', 'banana', 'apple', 'cherry', 'banana', 'apple']
word_count = defaultdict(int)
for word in words:
word_count[word] += 1
print(dict(word_count)) # {'apple': 3, 'banana': 2, 'cherry': 1}
# Nested defaultdict (for multi-level grouping)
data = [
('USA', 'NY', 100),
('USA', 'CA', 200),
('Canada', 'ON', 150),
('USA', 'NY', 50)
]
nested_dict = defaultdict(lambda: defaultdict(int))
for country, state, value in data:
nested_dict[country][state] += value
print(dict(nested_dict))
# {'USA': {'NY': 150, 'CA': 200}, 'Canada': {'ON': 150}}
# Graph adjacency list
graph = defaultdict(list)
edges = [('A', 'B'), ('A', 'C'), ('B', 'C'), ('C', 'D')]
for u, v in edges:
graph[u].append(v)
graph[v].append(u) # For undirected graph
print(dict(graph))
# {'A': ['B', 'C'], 'B': ['A', 'C'], 'C': ['A', 'B', 'D'], 'D': ['C']}Counter is a specialized dictionary for counting hashable objects:
from collections import Counter
# Creating counters
# From sequence
cnt = Counter(['a', 'b', 'c', 'a', 'b', 'a'])
print(cnt) # Counter({'a': 3, 'b': 2, 'c': 1})
# From string
cnt = Counter("hello world")
print(cnt) # Counter({'l': 3, 'o': 2, 'h': 1, 'e': 1, ' ': 1, 'w': 1, 'r': 1, 'd': 1})
# From dictionary
cnt = Counter({'a': 3, 'b': 2, 'c': 1})
# With keyword arguments
cnt = Counter(a=3, b=2, c=1)
# Counter methods
cnt = Counter(a=3, b=2, c=1, d=0)
# elements() - iterator over elements repeating count times
print(list(cnt.elements())) # ['a', 'a', 'a', 'b', 'b', 'c']
# most_common() - n most common elements
print(cnt.most_common(2)) # [('a', 3), ('b', 2)]
# subtract() - subtract counts
cnt2 = Counter(a=1, b=1, c=1)
cnt.subtract(cnt2)
print(cnt) # Counter({'a': 2, 'b': 1, 'c': 0, 'd': 0})
# update() - add counts
cnt.update(cnt2)
print(cnt) # Back to original
# Mathematical operations
c1 = Counter(a=3, b=2, c=1)
c2 = Counter(a=1, b=2, c=3)
# Addition
print(c1 + c2) # Counter({'a': 4, 'b': 4, 'c': 4})
# Subtraction (keeps positive counts only)
print(c1 - c2) # Counter({'a': 2})
# Intersection (min)
print(c1 & c2) # Counter({'b': 2, 'a': 1, 'c': 1})
# Union (max)
print(c1 | c2) # Counter({'a': 3, 'c': 3, 'b': 2})Practical Counter Examples:
# Word frequency analysis
def analyze_text(text):
"""Analyze word frequencies in text."""
words = text.lower().split()
# Remove punctuation
words = [word.strip('.,!?;:()[]{}"\'') for word in words]
return Counter(words)
text = """
Python is awesome. Python is powerful.
Python is easy to learn. Python is popular!
"""
word_counts = analyze_text(text)
print(word_counts.most_common(5))
# Find anagrams
def find_anagrams(words):
"""Group words that are anagrams."""
anagram_groups = defaultdict(list)
for word in words:
# Use sorted word as key
key = ''.join(sorted(word))
anagram_groups[key].append(word)
return {k: v for k, v in anagram_groups.items() if len(v) > 1}
words = ['listen', 'silent', 'enlist', 'google', 'gogleo', 'banana']
print(find_anagrams(words))
# Check if two strings are anagrams
def are_anagrams(s1, s2):
return Counter(s1) == Counter(s2)
print(are_anagrams('listen', 'silent')) # True
print(are_anagrams('hello', 'world')) # False
# Inventory management
inventory = Counter(apples=10, bananas=5, oranges=8)
# Sell some items
sold = Counter(apples=3, oranges=2)
inventory -= sold
print(inventory) # Counter({'apples': 7, 'oranges': 6, 'bananas': 5})
# Restock
restock = Counter(apples=5, bananas=3)
inventory += restock
print(inventory) # Counter({'apples': 12, 'bananas': 8, 'oranges': 6})
# Find most common items
print(inventory.most_common(2)) # [('apples', 12), ('bananas', 8)]
# Find total items
total_items = sum(inventory.values())
print(f"Total items: {total_items}")
# Find items with low stock
low_stock = [item for item, count in inventory.items() if count < 7]
print(f"Low stock items: {low_stock}")The collections module provides specialized container datatypes:
ChainMap - Groups multiple dictionaries into a single view:
from collections import ChainMap
# Creating ChainMap
defaults = {'theme': 'light', 'language': 'en', 'show_sidebar': True}
user_prefs = {'theme': 'dark', 'language': 'fr'}
session_prefs = {'language': 'es'}
# Order matters: earlier dicts have higher priority
config = ChainMap(session_prefs, user_prefs, defaults)
# Access values (searches in order)
print(config['theme']) # 'dark' (from user_prefs)
print(config['language']) # 'es' (from session_prefs)
print(config['show_sidebar']) # True (from defaults)
# Get missing key
print(config.get('font_size', 12)) # 12
# Update (affects first dictionary only)
config['theme'] = 'custom'
print(session_prefs) # {'language': 'es', 'theme': 'custom'}
# Add new dictionary
new_prefs = {'font_size': 14}
config = config.new_child(new_prefs)
print(config['font_size']) # 14
# Get list of maps
print(config.maps) # [{'font_size': 14}, {'language': 'es', 'theme': 'custom'},
# {'theme': 'light', 'language': 'en', 'show_sidebar': True}]
# Reverse ChainMap
parent = config.parents
print(parent['theme']) # 'custom' (from previous level)Practical ChainMap Examples:
# Configuration management with layered overrides
class ConfigManager:
def __init__(self):
self.defaults = {
'host': 'localhost',
'port': 8080,
'debug': False,
'timeout': 30
}
self._chain = ChainMap({}, self.defaults)
def load_file_config(self, file_path):
"""Load configuration from file."""
# Simulate loading config
file_config = {'port': 9000, 'debug': True}
self._chain = self._chain.new_child(file_config)
def set_env_config(self, env_vars):
"""Set environment-specific configuration."""
self._chain = self._chain.new_child(env_vars)
def get(self, key, default=None):
return self._chain.get(key, default)
def set(self, key, value):
self._chain[key] = value
@property
def current_config(self):
return dict(self._chain)
# Usage
config = ConfigManager()
print(config.get('port')) # 8080
config.load_file_config('config.json')
print(config.get('port')) # 9000 (file overrides default)
config.set_env_config({'host': '192.168.1.100', 'debug': False})
print(config.get('host')) # 192.168.1.100
print(config.get('debug')) # False (env overrides file)
config.set('timeout', 60) # Runtime override
print(config.current_config)deque - Double-ended queue for fast appends/pops from both ends:
from collections import deque
# Creating deque
d = deque([1, 2, 3, 4, 5])
print(d) # deque([1, 2, 3, 4, 5])
# Append to right
d.append(6)
print(d) # deque([1, 2, 3, 4, 5, 6])
# Append to left
d.appendleft(0)
print(d) # deque([0, 1, 2, 3, 4, 5, 6])
# Pop from right
right = d.pop()
print(right) # 6
print(d) # deque([0, 1, 2, 3, 4, 5])
# Pop from left
left = d.popleft()
print(left) # 0
print(d) # deque([1, 2, 3, 4, 5])
# Extend
d.extend([6, 7, 8])
print(d) # deque([1, 2, 3, 4, 5, 6, 7, 8])
d.extendleft([0, -1]) # Note: extends in reverse order
print(d) # deque([-1, 0, 1, 2, 3, 4, 5, 6, 7, 8])
# Rotate
d.rotate(2) # Rotate right by 2
print(d) # deque([7, 8, -1, 0, 1, 2, 3, 4, 5, 6])
d.rotate(-3) # Rotate left by 3
print(d) # deque([1, 2, 3, 4, 5, 6, 7, 8, -1, 0])
# Max length (bounded deque)
bounded = deque(maxlen=3)
for i in range(5):
bounded.append(i)
print(bounded) # Shows sliding window
# Access elements
print(bounded[0]) # First element
print(bounded[-1]) # Last element
# Count occurrences
d = deque([1, 2, 3, 2, 1, 2])
print(d.count(2)) # 3
# Remove first occurrence
d.remove(2)
print(d) # deque([1, 3, 2, 1, 2])
# Reverse in-place
d.reverse()
print(d) # deque([2, 1, 2, 3, 1])Practical deque Examples:
# Sliding window maximum
def sliding_window_maximum(nums, k):
"""Find maximum in each sliding window of size k."""
result = []
window = deque()
for i, num in enumerate(nums):
# Remove elements outside current window
while window and window[0] <= i - k:
window.popleft()
# Remove smaller elements from back
while window and nums[window[-1]] < num:
window.pop()
window.append(i)
# Add to result when window is formed
if i >= k - 1:
result.append(nums[window[0]])
return result
nums = [1, 3, -1, -3, 5, 3, 6, 7]
k = 3
print(sliding_window_maximum(nums, k)) # [3, 3, 5, 5, 6, 7]
# Task scheduler (round-robin)
class TaskScheduler:
def __init__(self):
self.tasks = deque()
def add_task(self, task, priority=0):
self.tasks.append((priority, task))
# Keep sorted by priority (simple version)
self.tasks = deque(sorted(self.tasks, reverse=True))
def run_next(self):
if self.tasks:
priority, task = self.tasks.popleft()
print(f"Running task: {task} (priority {priority})")
# Re-add with lower priority? (for round-robin)
if priority > 0:
self.tasks.append((priority - 1, task))
else:
print("No tasks to run")
# Recent items cache
class RecentCache:
def __init__(self, maxsize=100):
self.cache = deque(maxlen=maxsize)
def add(self, item):
# Remove if already exists
if item in self.cache:
self.cache.remove(item)
self.cache.append(item)
def get_recent(self, n=10):
return list(self.cache)[-n:]
def __contains__(self, item):
return item in self.cache
cache = RecentCache(maxsize=5)
for i in range(10):
cache.add(f"item{i}")
print(cache.get_recent())UserDict, UserList, UserString - Wrapper classes for creating custom containers:
from collections import UserDict, UserList, UserString
# Custom dictionary with validation
class ValidatedDict(UserDict):
"""Dictionary that validates keys and values."""
def __init__(self, key_type=None, value_type=None, **kwargs):
super().__init__(**kwargs)
self.key_type = key_type
self.value_type = value_type
self._validate_all()
def _validate_key(self, key):
if self.key_type and not isinstance(key, self.key_type):
raise TypeError(f"Key must be of type {self.key_type.__name__}")
def _validate_value(self, value):
if self.value_type and not isinstance(value, self.value_type):
raise TypeError(f"Value must be of type {self.value_type.__name__}")
def _validate_all(self):
for key, value in self.data.items():
self._validate_key(key)
self._validate_value(value)
def __setitem__(self, key, value):
self._validate_key(key)
self._validate_value(value)
super().__setitem__(key, value)
def update(self, *args, **kwargs):
# Validate before updating
other = dict(*args, **kwargs)
for key, value in other.items():
self._validate_key(key)
self._validate_value(value)
super().update(other)
# Usage
vd = ValidatedDict(key_type=str, value_type=int)
vd['age'] = 30
vd['count'] = 5
# vd[123] = 10 # TypeError: Key must be of type str
# vd['score'] = 'A' # TypeError: Value must be of type int
# Custom list with logging
class LoggedList(UserList):
"""List that logs all modifications."""
def __init__(self, *args, logger=None):
super().__init__(*args)
self.logger = logger or print
def _log(self, operation, item=None):
self.logger(f"{operation}: {item if item else ''}")
def append(self, item):
self._log("APPEND", item)
super().append(item)
def extend(self, items):
self._log("EXTEND", items)
super().extend(items)
def insert(self, i, item):
self._log(f"INSERT at {i}", item)
super().insert(i, item)
def remove(self, item):
self._log("REMOVE", item)
super().remove(item)
def pop(self, i=-1):
item = super().pop(i)
self._log(f"POP from {i}", item)
return item
def __setitem__(self, i, item):
self._log(f"SET at {i}", item)
super().__setitem__(i, item)
# Usage
logged = LoggedList([1, 2, 3])
logged.append(4)
logged.insert(1, 5)
logged.pop()Heap queue (priority queue) algorithm:
import heapq
# Creating a heap
numbers = [3, 1, 4, 1, 5, 9, 2]
heapq.heapify(numbers) # Transform list into heap (in-place)
print(numbers) # [1, 1, 2, 3, 5, 9, 4] (heap property satisfied)
# Push onto heap
heapq.heappush(numbers, 0)
print(numbers) # [0, 1, 1, 3, 2, 9, 4, 5]
# Pop smallest
smallest = heapq.heappop(numbers)
print(smallest) # 0
print(numbers) # [1, 1, 2, 3, 5, 9, 4]
# Push then pop
result = heapq.heappushpop(numbers, 1.5)
print(result) # 1
print(numbers) # [1, 1.5, 2, 3, 5, 9, 4]
# Pop then push
result = heapq.heapreplace(numbers, 0.5)
print(result) # 1
print(numbers) # [0.5, 1.5, 2, 3, 5, 9, 4]
# n largest/smallest
numbers = [3, 1, 4, 1, 5, 9, 2, 6, 5]
print(heapq.nlargest(3, numbers)) # [9, 6, 5]
print(heapq.nsmallest(3, numbers)) # [1, 1, 2]
# With custom key
students = [
{'name': 'Alice', 'grade': 85},
{'name': 'Bob', 'grade': 92},
{'name': 'Charlie', 'grade': 78},
{'name': 'Diana', 'grade': 95}
]
top_2 = heapq.nlargest(2, students, key=lambda x: x['grade'])
print(top_2)Priority Queue Implementation:
class PriorityQueue:
"""Priority queue using heapq."""
def __init__(self):
self._heap = []
self._count = 0
def push(self, item, priority):
"""Push item with given priority."""
# Negative priority for max-heap behavior
heapq.heappush(self._heap, (-priority, self._count, item))
self._count += 1
def pop(self):
"""Pop item with highest priority."""
if self.is_empty():
raise IndexError("Pop from empty priority queue")
return heapq.heappop(self._heap)[-1]
def peek(self):
"""Return highest priority item without removing."""
if self.is_empty():
return None
return self._heap[0][-1]
def is_empty(self):
return len(self._heap) == 0
def __len__(self):
return len(self._heap)
# Usage
pq = PriorityQueue()
pq.push("task1", 3)
pq.push("task2", 1)
pq.push("task3", 5)
pq.push("task4", 2)
while not pq.is_empty():
print(pq.pop()) # task3, task1, task4, task2Practical Heap Examples:
# Merge k sorted lists
def merge_k_sorted(lists):
"""Merge k sorted lists using heap."""
result = []
heap = []
# Push first element from each list
for i, lst in enumerate(lists):
if lst:
heapq.heappush(heap, (lst[0], i, 0))
while heap:
val, list_idx, elem_idx = heapq.heappop(heap)
result.append(val)
# Push next element from same list
if elem_idx + 1 < len(lists[list_idx]):
next_val = lists[list_idx][elem_idx + 1]
heapq.heappush(heap, (next_val, list_idx, elem_idx + 1))
return result
lists = [[1, 4, 5], [1, 3, 4], [2, 6]]
print(merge_k_sorted(lists)) # [1, 1, 2, 3, 4, 4, 5, 6]
# Find k closest points to origin
def k_closest(points, k):
"""Find k closest points to origin (0,0)."""
heap = []
for x, y in points:
dist = -(x*x + y*y) # Negative for max-heap
if len(heap) < k:
heapq.heappush(heap, (dist, x, y))
else:
heapq.heappushpop(heap, (dist, x, y))
return [(x, y) for _, x, y in heap]
points = [(1, 3), (-2, 2), (5, 8), (0, 1), (3, 3)]
print(k_closest(points, 2)) # [(-2, 2), (0, 1)]
# Running median
class RunningMedian:
"""Track median of a stream of numbers."""
def __init__(self):
self.left = [] # Max-heap (negate values)
self.right = [] # Min-heap
def add(self, num):
if not self.left or num <= -self.left[0]:
heapq.heappush(self.left, -num)
else:
heapq.heappush(self.right, num)
# Balance heaps
if len(self.left) > len(self.right) + 1:
val = -heapq.heappop(self.left)
heapq.heappush(self.right, val)
elif len(self.right) > len(self.left):
val = heapq.heappop(self.right)
heapq.heappush(self.left, -val)
def median(self):
if len(self.left) > len(self.right):
return -self.left[0]
return (-self.left[0] + self.right[0]) / 2
rm = RunningMedian()
for num in [5, 2, 8, 1, 9]:
rm.add(num)
print(f"After adding {num}, median: {rm.median()}")Dataclasses provide a concise way to create classes that primarily store data:
from dataclasses import dataclass, field, asdict, astuple
from typing import List, Optional
import inspect
# Basic dataclass
@dataclass
class Person:
name: str
age: int
city: str = "Unknown"
# Usage
p1 = Person("Alice", 30)
p2 = Person("Bob", 25, "New York")
print(p1) # Person(name='Alice', age=30, city='Unknown')
print(p1 == p2) # False (compares all fields)
# Default values and mutability
@dataclass
class Employee:
name: str
id: int
department: str = "Engineering"
skills: List[str] = field(default_factory=list) # Mutable default
salary: Optional[float] = None
active: bool = field(default=True, init=False) # Not in __init__
def __post_init__(self):
"""Initialize after __init__."""
if self.salary is None:
self.salary = 50000.0
emp = Employee("Alice", 1001)
print(emp) # Employee(name='Alice', id=1001, department='Engineering',
# skills=[], salary=50000.0, active=True)Dataclass Features:
from dataclasses import dataclass, field, InitVar
# Field customization
@dataclass
class Product:
name: str
price: float = field(metadata={'unit': 'USD'})
quantity: int = field(default=0, repr=False) # Exclude from repr
discount: float = field(default=0.0, compare=False) # Exclude from comparisons
# Computed field (not in __init__)
total_value: float = field(init=False)
def __post_init__(self):
self.total_value = self.price * self.quantity * (1 - self.discount)
p = Product("Laptop", 999.99, 5, 0.1)
print(p) # Product(name='Laptop', price=999.99, discount=0.1)
print(p.total_value) # 4499.955
# Frozen (immutable) dataclass
@dataclass(frozen=True)
class Point:
x: float
y: float
def distance(self, other):
return ((self.x - other.x)**2 + (self.y - other.y)**2)**0.5
p1 = Point(1, 2)
p2 = Point(4, 6)
# p1.x = 10 # FrozenInstanceError
print(p1.distance(p2)) # 5.0
# Inheritance
@dataclass
class Base:
x: int = 0
y: int = 0
@dataclass
class Derived(Base):
z: int = 0
def __post_init__(self):
super().__post_init__()
# Additional initialization
d = Derived(1, 2, 3)
print(d) # Derived(x=1, y=2, z=3)
# Init-only variables
@dataclass
class DatabaseConnection:
host: str
port: int
connection: object = field(init=False)
db_name: InitVar[str] = None # Passed to __init__ but not stored
def __post_init__(self, db_name):
self.connection = f"Connected to {self.host}:{self.port}/{db_name}"
db = DatabaseConnection("localhost", 5432, "mydb")
print(db.connection) # "Connected to localhost:5432/mydb"Converting Dataclasses:
from dataclasses import dataclass, asdict, astuple
import json
@dataclass
class Address:
street: str
city: str
zipcode: str
@dataclass
class User:
name: str
age: int
address: Address
emails: List[str]
# Create user
user = User(
"Alice",
30,
Address("123 Main St", "Springfield", "12345"),
["alice@email.com", "alice@work.com"]
)
# Convert to dictionary
user_dict = asdict(user)
print(user_dict)
# {
# 'name': 'Alice',
# 'age': 30,
# 'address': {'street': '123 Main St', 'city': 'Springfield', 'zipcode': '12345'},
# 'emails': ['alice@email.com', 'alice@work.com']
# }
# Convert to tuple
user_tuple = astuple(user)
print(user_tuple) # ('Alice', 30, ('123 Main St', 'Springfield', '12345'), [...])
# Serialize to JSON
json_str = json.dumps(asdict(user), indent=2)
print(json_str)
# Deserialize from JSON (with validation)
def from_dict(cls, data):
"""Create dataclass from dictionary, handling nested dataclasses."""
if hasattr(cls, '__dataclass_fields__'):
field_types = {f: cls.__dataclass_fields__[f].type for f in data}
kwargs = {}
for field_name, field_type in field_types.items():
if field_name in data:
# Check if field_type is another dataclass
if hasattr(field_type, '__dataclass_fields__'):
kwargs[field_name] = from_dict(field_type, data[field_name])
else:
kwargs[field_name] = data[field_name]
return cls(**kwargs)
return data
user2 = from_dict(User, user_dict)
print(user2 == user) # TruePractical Dataclass Examples:
from dataclasses import dataclass, field
from typing import List, Optional
from datetime import datetime
import uuid
@dataclass
class Book:
"""Book model."""
title: str
author: str
isbn: str
published_year: int
genres: List[str] = field(default_factory=list)
id: str = field(default_factory=lambda: str(uuid.uuid4()))
created_at: datetime = field(default_factory=datetime.now)
def __post_init__(self):
if self.published_year < 0:
raise ValueError("Published year cannot be negative")
@property
def age(self):
"""Calculate age of book."""
return datetime.now().year - self.published_year
@dataclass
class Library:
"""Library containing books."""
name: str
location: str
books: List[Book] = field(default_factory=list)
def add_book(self, book: Book):
self.books.append(book)
def find_by_author(self, author: str) -> List[Book]:
return [b for b in self.books if b.author.lower() == author.lower()]
def find_by_genre(self, genre: str) -> List[Book]:
return [b for b in self.books if genre.lower() in [g.lower() for g in b.genres]]
def total_books(self) -> int:
return len(self.books)
def __len__(self):
return self.total_books()
# Usage
library = Library("City Library", "Downtown")
book1 = Book(
"The Python Programming",
"John Smith",
"978-1234567890",
2020,
["Programming", "Education"]
)
book2 = Book(
"Data Science Handbook",
"Jane Doe",
"978-0987654321",
2021,
["Data Science", "Programming"]
)
library.add_book(book1)
library.add_book(book2)
print(f"Library has {len(library)} books")
print(library.find_by_author("john smith"))
print([b.title for b in library.find_by_genre("Programming")])
# Configuration management
@dataclass
class DatabaseConfig:
host: str = "localhost"
port: int = 5432
database: str = "app"
username: str = "user"
password: str = field(default="", repr=False) # Hide in repr
@property
def connection_string(self):
return f"postgresql://{self.username}:****@{self.host}:{self.port}/{self.database}"
@dataclass
class AppConfig:
name: str
debug: bool = False
database: DatabaseConfig = field(default_factory=DatabaseConfig)
features: List[str] = field(default_factory=list)
# Load from dictionary
config_dict = {
"name": "MyApp",
"debug": True,
"database": {
"host": "db.example.com",
"port": 5432,
"database": "prod"
},
"features": ["auth", "logging"]
}
config = from_dict(AppConfig, config_dict)
print(config.database.connection_string)Object-oriented programming (OOP) organizes code around objects that contain both data (attributes) and behavior (methods). Python's implementation of OOP is elegant and powerful:
Defining a Simple Class:
class Dog:
"""A simple Dog class."""
# Class attribute (shared by all instances)
species = "Canis familiaris"
# Constructor (initializer)
def __init__(self, name, age):
# Instance attributes (unique to each instance)
self.name = name
self.age = age
# Instance method
def description(self):
return f"{self.name} is {self.age} years old"
def bark(self, sound):
return f"{self.name} says {sound}"
# Creating objects (instances)
my_dog = Dog("Buddy", 3)
your_dog = Dog("Lucy", 5)
# Accessing attributes
print(my_dog.name) # "Buddy"
print(my_dog.age) # 3
print(my_dog.species) # "Canis familiaris" (class attribute)
# Calling methods
print(my_dog.description()) # "Buddy is 3 years old"
print(my_dog.bark("Woof!")) # "Buddy says Woof!"
# Each instance has its own attribute values
print(your_dog.description()) # "Lucy is 5 years old"Understanding self:
The self parameter refers to the instance itself. When you call a method, Python automatically passes the instance as the first argument:
class Example:
def method(self, arg):
print(f"self: {self}")
print(f"arg: {arg}")
obj = Example()
obj.method("hello")
# This is equivalent to:
Example.method(obj, "hello")Instance vs Class Attributes:
class Counter:
# Class attribute
count = 0
def __init__(self):
# Instance attribute
self.instance_count = 0
Counter.count += 1 # Modify class attribute
def increment(self):
self.instance_count += 1
c1 = Counter()
c1.increment()
c1.increment()
c2 = Counter()
c2.increment()
print(f"Class count: {Counter.count}") # 2 (shared across instances)
print(f"c1 instance count: {c1.instance_count}") # 2
print(f"c2 instance count: {c2.instance_count}") # 1
# Accessing class attribute through instance
print(c1.count) # 2 (looks up in class if not found in instance)
# Modifying class attribute through instance creates instance attribute
c1.count = 10 # Creates instance attribute 'count'
print(c1.count) # 10 (instance attribute)
print(Counter.count) # 2 (class attribute unchanged)Python provides flexible ways to define and access attributes and methods:
Instance Methods:
class Rectangle:
def __init__(self, width, height):
self.width = width
self.height = height
# Instance method
def area(self):
return self.width * self.height
def perimeter(self):
return 2 * (self.width + self.height)
def scale(self, factor):
self.width *= factor
self.height *= factor
rect = Rectangle(5, 3)
print(rect.area()) # 15
print(rect.perimeter()) # 16
rect.scale(2)
print(rect.area()) # 60Class Methods: Class methods operate on the class itself, not instances. They receive the class as the first argument:
class Employee:
company = "Tech Corp"
def __init__(self, name, salary):
self.name = name
self.salary = salary
@classmethod
def change_company(cls, new_name):
cls.company = new_name
@classmethod
def from_string(cls, emp_string):
"""Alternative constructor - create Employee from string."""
name, salary = emp_string.split("-")
return cls(name, float(salary))
@classmethod
def get_company_info(cls):
return f"Company: {cls.company}"
# Using class methods
print(Employee.get_company_info()) # "Company: Tech Corp"
Employee.change_company("New Tech Inc")
print(Employee.get_company_info()) # "Company: New Tech Inc"
# Using alternative constructor
emp = Employee.from_string("Alice-75000")
print(emp.name) # "Alice"
print(emp.salary) # 75000.0Static Methods: Static methods don't receive self or cls. They behave like regular functions but belong to the class namespace:
class MathUtils:
@staticmethod
def add(x, y):
return x + y
@staticmethod
def multiply(x, y):
return x * y
@staticmethod
def is_even(n):
return n % 2 == 0
# Called on the class, no instance needed
print(MathUtils.add(5, 3)) # 8
print(MathUtils.is_even(10)) # True
# Can also be called on instances, but this is unusual
math = MathUtils()
print(math.multiply(4, 5)) # 20Property Methods: Properties allow method calls to look like attribute access:
class Circle:
def __init__(self, radius):
self._radius = radius
self._diameter = radius * 2
@property
def radius(self):
"""Get the radius."""
return self._radius
@radius.setter
def radius(self, value):
if value <= 0:
raise ValueError("Radius must be positive")
self._radius = value
self._diameter = value * 2 # Update diameter
@property
def diameter(self):
"""Get the diameter."""
return self._diameter
@property
def area(self):
"""Calculate area (computed property)."""
return 3.14159 * self._radius ** 2
@property
def circumference(self):
"""Calculate circumference."""
return 2 * 3.14159 * self._radius
circle = Circle(5)
print(circle.radius) # 5 (looks like attribute)
print(circle.diameter) # 10
print(circle.area) # 78.53975 (computed)
circle.radius = 10
print(circle.diameter) # 20 (automatically updated)
# circle.radius = -5 # Raises ValueErrorSpecial Methods (Magic Methods): Special methods customize class behavior:
class Book:
def __init__(self, title, author, pages):
self.title = title
self.author = author
self.pages = pages
self._read_pages = 0
def __str__(self):
"""String representation for users."""
return f"'{self.title}' by {self.author}"
def __repr__(self):
"""String representation for developers."""
return f"Book('{self.title}', '{self.author}', {self.pages})"
def __len__(self):
"""Return length (number of pages)."""
return self.pages
def __contains__(self, item):
"""Check if item is in book (in title or author)."""
return item.lower() in self.title.lower() or \
item.lower() in self.author.lower()
def __iter__(self):
"""Make book iterable (iterate through pages)."""
self._current_page = 1
return self
def __next__(self):
"""Next page in iteration."""
if self._current_page > self.pages:
raise StopIteration
page_num = self._current_page
self._current_page += 1
return f"Page {page_num}"
def __call__(self, pages):
"""Make object callable (read pages)."""
self._read_pages = min(pages, self.pages)
return f"Read {self._read_pages} pages of {self.title}"
def __eq__(self, other):
"""Equality comparison."""
if not isinstance(other, Book):
return False
return self.title == other.title and self.author == other.author
def __lt__(self, other):
"""Less than comparison (by page count)."""
return self.pages < other.pages
# Usage
book = Book("1984", "George Orwell", 328)
print(str(book)) # '1984' by George Orwell
print(repr(book)) # Book('1984', 'George Orwell', 328)
print(len(book)) # 328
print("Orwell" in book) # True
print("python" in book) # False
# Iteration
for page in book:
if page == "Page 5": # Stop after page 5
break
print(page)
# Callable object
result = book(50) # "Read 50 pages of 1984"
# Comparisons
book2 = Book("1984", "George Orwell", 328)
book3 = Book("Animal Farm", "George Orwell", 112)
print(book == book2) # True
print(book < book3) # False (328 < 112)
print(book > book3) # TruePython uses __init__ as the constructor and __del__ as the destructor:
Advanced Constructor Patterns:
class DatabaseConnection:
# Class-level pool
_pool = []
def __new__(cls, *args, **kwargs):
"""Control instance creation (rarely overridden)."""
print("1. Creating instance")
instance = super().__new__(cls)
return instance
def __init__(self, host, port, database):
"""Initialize the instance."""
print("2. Initializing instance")
self.host = host
self.port = port
self.database = database
self.connection = None
self._connected = False
DatabaseConnection._pool.append(self)
def __del__(self):
"""Destructor - called when object is garbage collected."""
print(f"3. Destroying connection to {self.host}")
if self._connected:
self.disconnect()
if self in DatabaseConnection._pool:
DatabaseConnection._pool.remove(self)
@classmethod
def from_url(cls, url):
"""Alternative constructor from database URL."""
# Parse URL: postgresql://user:pass@host:port/db
import re
pattern = r"postgresql://([^:]+):([^@]+)@([^:]+):(\d+)/(.+)"
match = re.match(pattern, url)
if match:
user, password, host, port, db = match.groups()
return cls(host, int(port), db)
raise ValueError("Invalid database URL")
@classmethod
def get_pool_stats(cls):
"""Get connection pool statistics."""
return {
'total_connections': len(cls._pool),
'active_connections': sum(1 for c in cls._pool if c._connected)
}
# Usage
db1 = DatabaseConnection("localhost", 5432, "myapp")
db2 = DatabaseConnection.from_url(
"postgresql://user:pass@remotehost:5432/production"
)
print(DatabaseConnection.get_pool_stats())
del db1 # Triggers __del__
print(DatabaseConnection.get_pool_stats())Singleton Pattern with Constructor Control:
class Singleton:
"""Singleton pattern using __new__."""
_instance = None
_initialized = False
def __new__(cls, *args, **kwargs):
if cls._instance is None:
print("Creating singleton instance")
cls._instance = super().__new__(cls)
return cls._instance
def __init__(self, value=None):
if not self._initialized:
print("Initializing singleton")
self.value = value
self._initialized = True
def __str__(self):
return f"Singleton(value={self.value})"
# Test
s1 = Singleton(10)
s2 = Singleton(20)
print(s1 is s2) # True
print(s1) # Singleton(value=10) (initialized only once)
print(s2.value) # 10 (same instance)Understanding the distinction between instance and class variables is crucial:
class Student:
# Class variables (shared)
school = "Python University"
total_students = 0
grade_levels = ["Freshman", "Sophomore", "Junior", "Senior"]
def __init__(self, name, grade_level):
# Instance variables (unique)
self.name = name
self.grade_level = grade_level
self.grades = [] # Instance-specific
# Modify class variable
Student.total_students += 1
def add_grade(self, grade):
self.grades.append(grade)
def average(self):
return sum(self.grades) / len(self.grades) if self.grades else 0
@classmethod
def get_school_info(cls):
return f"{cls.school} has {cls.total_students} students"
# Create students
alice = Student("Alice", "Freshman")
bob = Student("Bob", "Sophomore")
# Instance variables are unique
alice.add_grade(85)
alice.add_grade(90)
bob.add_grade(92)
print(f"Alice's average: {alice.average():.1f}") # 87.5
print(f"Bob's average: {bob.average():.1f}") # 92.0
# Class variables are shared
print(alice.school) # "Python University"
print(bob.school) # "Python University"
Student.school = "Advanced Python Institute"
print(alice.school) # "Advanced Python Institute"
# Class methods operate on class variables
print(Student.get_school_info()) # "Advanced Python Institute has 2 students"
# Beware of mutable class variables
class Problematic:
shared_list = [] # Shared by all instances!
def add_item(self, item):
self.shared_list.append(item)
p1 = Problematic()
p2 = Problematic()
p1.add_item(1)
p2.add_item(2)
print(p1.shared_list) # [1, 2] (not what you might expect!)
# Correct approach - use instance variables
class Correct:
def __init__(self):
self.items = [] # Each instance gets its own list
def add_item(self, item):
self.items.append(item)Name Mangling for "Private" Attributes:
class BankAccount:
def __init__(self, owner, balance):
self.owner = owner
self.__balance = balance # Name mangled to _BankAccount__balance
self._transaction_history = [] # Single underscore: "protected" by convention
def deposit(self, amount):
if amount > 0:
self.__balance += amount
self._transaction_history.append(f"Deposited: ${amount}")
def withdraw(self, amount):
if 0 < amount <= self.__balance:
self.__balance -= amount
self._transaction_history.append(f"Withdrew: ${amount}")
return True
return False
def get_balance(self):
return self.__balance
def get_history(self):
return self._transaction_history.copy()
account = BankAccount("Alice", 1000)
account.deposit(500)
account.withdraw(200)
# Can't access __balance directly
# print(account.__balance) # AttributeError
# But it's still accessible (just name-mangled)
print(account._BankAccount__balance) # 1300 (not recommended!)
# Single underscore is just a convention
print(account._transaction_history) # Accessible but "please don't"Inheritance allows classes to inherit attributes and methods from parent classes:
Basic Inheritance:
class Animal:
"""Base class for all animals."""
def __init__(self, name, age):
self.name = name
self.age = age
def speak(self):
raise NotImplementedError("Subclass must implement this method")
def move(self):
return f"{self.name} is moving"
def __str__(self):
return f"{self.name} ({self.__class__.__name__}), age {self.age}"
class Dog(Animal):
"""Dog inherits from Animal."""
def speak(self):
return f"{self.name} says Woof!"
def wag_tail(self):
return f"{self.name} is wagging tail"
class Cat(Animal):
"""Cat inherits from Animal."""
def speak(self):
return f"{self.name} says Meow!"
def purr(self):
return f"{self.name} is purring"
# Usage
dog = Dog("Buddy", 3)
cat = Cat("Whiskers", 2)
print(dog.move()) # Inherited from Animal
print(dog.speak()) # Implemented in Dog
print(dog.wag_tail()) # Specific to Dog
print(cat.speak()) # Implemented in Cat
print(cat.purr()) # Specific to Cat
# Polymorphism: treating objects by their common interface
animals = [dog, cat]
for animal in animals:
print(animal.speak()) # Each speaks appropriatelyExtending Parent Class Methods:
class Vehicle:
def __init__(self, make, model, year):
self.make = make
self.model = model
self.year = year
self._mileage = 0
def start(self):
return f"{self.make} {self.model} is starting"
def stop(self):
return f"{self.make} {self.model} is stopping"
def drive(self, miles):
self._mileage += miles
return f"Drove {miles} miles"
def info(self):
return f"{self.year} {self.make} {self.model}, {self._mileage} miles"
class Car(Vehicle):
def __init__(self, make, model, year, doors):
# Call parent constructor
super().__init__(make, model, year)
self.doors = doors
self._fuel_level = 100
# Override parent method
def start(self):
# Extend parent behavior
parent_start = super().start()
return f"{parent_start} (Fuel: {self._fuel_level}%)"
# Add new method
def refuel(self, amount):
self._fuel_level = min(100, self._fuel_level + amount)
return f"Fuel level: {self._fuel_level}%"
# Override with additional functionality
def info(self):
parent_info = super().info()
return f"{parent_info}, {self.doors} doors"
class ElectricCar(Car):
def __init__(self, make, model, year, doors, battery_capacity):
super().__init__(make, model, year, doors)
self.battery_capacity = battery_capacity
self._charge_level = 100
def start(self):
return f"{self.make} {self.model} starts silently"
def refuel(self, amount):
# Electric cars don't refuel, they charge
return self.charge(amount)
def charge(self, minutes):
self._charge_level = min(100, self._charge_level + minutes // 10)
return f"Charged to {self._charge_level}%"
def info(self):
return f"{super().info()}, {self.battery_capacity} kWh battery"
# Test
car = Car("Toyota", "Camry", 2022, 4)
print(car.start())
print(car.drive(100))
print(car.info())
tesla = ElectricCar("Tesla", "Model 3", 2023, 4, 75)
print(tesla.start())
print(tesla.charge(30))
print(tesla.info())The super() Function:
class Base:
def __init__(self):
print("Base.__init__")
self.x = 10
class A(Base):
def __init__(self):
print("A.__init__")
super().__init__()
self.x += 5
class B(Base):
def __init__(self):
print("B.__init__")
super().__init__()
self.x *= 2
class C(A, B):
def __init__(self):
print("C.__init__")
super().__init__()
print(f"x = {self.x}")
# What happens? Let's see MRO
print(C.__mro__)
c = C()
# Output shows method resolution order and super() callsAbstract Base Classes:
from abc import ABC, abstractmethod
import math
class Shape(ABC):
"""Abstract base class for shapes."""
@abstractmethod
def area(self):
"""Calculate area - must be implemented by subclasses."""
pass
@abstractmethod
def perimeter(self):
"""Calculate perimeter - must be implemented."""
pass
def describe(self):
"""Concrete method available to all subclasses."""
return f"{self.__class__.__name__} - Area: {self.area():.2f}, Perimeter: {self.perimeter():.2f}"
class Circle(Shape):
def __init__(self, radius):
self.radius = radius
def area(self):
return math.pi * self.radius ** 2
def perimeter(self):
return 2 * math.pi * self.radius
class Rectangle(Shape):
def __init__(self, width, height):
self.width = width
self.height = height
def area(self):
return self.width * self.height
def perimeter(self):
return 2 * (self.width + self.height)
# shape = Shape() # TypeError: Can't instantiate abstract class
circle = Circle(5)
rectangle = Rectangle(4, 6)
print(circle.describe())
print(rectangle.describe())Python supports multiple inheritance, allowing a class to inherit from multiple parent classes:
class Flyer:
def fly(self):
return "Flying through the air"
def move(self):
return "Moving through air"
class Swimmer:
def swim(self):
return "Swimming through water"
def move(self):
return "Moving through water"
class Walker:
def walk(self):
return "Walking on ground"
def move(self):
return "Moving on ground"
class Duck(Flyer, Swimmer, Walker):
def __init__(self, name):
self.name = name
def move(self):
# Explicitly choose which parent's move method to use
return f"{self.name}: {Walker.move(self)}"
duck = Duck("Donald")
print(duck.fly())
print(duck.swim())
print(duck.walk())
print(duck.move()) # Uses Walker's move due to explicit choiceDiamond Problem and MRO:
class A:
def method(self):
return "A"
class B(A):
def method(self):
return "B"
class C(A):
def method(self):
return "C"
class D(B, C):
pass
d = D()
print(d.method()) # "B" - follows MRO
print(D.__mro__) # Shows order: D, B, C, A, object
# Complex diamond
class X:
def func(self):
return "X"
class Y(X):
def func(self):
return "Y"
class Z(X):
def func(self):
return "Z"
class M(Y, Z):
def func(self):
return super().func() # Calls next in MRO
m = M()
print(m.func()) # "Y"
print(M.__mro__) # M, Y, Z, X, objectMixins - Reusable Components:
class TimestampMixin:
"""Add timestamp functionality to any class."""
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.created_at = None
self.updated_at = None
def touch(self):
import datetime
if self.created_at is None:
self.created_at = datetime.datetime.now()
self.updated_at = datetime.datetime.now()
def age(self):
if self.created_at:
return (datetime.datetime.now() - self.created_at).total_seconds()
return 0
class JSONMixin:
"""Add JSON serialization to any class."""
def to_json(self):
import json
# Convert object attributes to dict
data = {k: v for k, v in self.__dict__.items() if not k.startswith('_')}
return json.dumps(data, default=str, indent=2)
@classmethod
def from_json(cls, json_str):
import json
data = json.loads(json_str)
return cls(**data)
class LoggingMixin:
"""Add logging to method calls."""
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.log = []
def log_call(self, method_name, *args, **kwargs):
entry = f"Called {method_name} with args={args}, kwargs={kwargs}"
self.log.append(entry)
print(entry)
class Person(TimestampMixin, JSONMixin, LoggingMixin):
def __init__(self, name, age):
super().__init__()
self.name = name
self.age = age
self.touch() # From TimestampMixin
def celebrate_birthday(self):
self.log_call("celebrate_birthday")
self.age += 1
self.touch()
# Using mixins
person = Person("Alice", 30)
print(person.to_json()) # From JSONMixin
person.celebrate_birthday()
print(f"Age: {person.age}")
print(f"Account age: {person.age():.0f} seconds")Understanding MRO is crucial for multiple inheritance:
class A:
def process(self):
return "A"
class B(A):
def process(self):
return f"B -> {super().process()}"
class C(A):
def process(self):
return f"C -> {super().process()}"
class D(B, C):
def process(self):
return f"D -> {super().process()}"
d = D()
print(d.process()) # "D -> B -> C -> A"
# Visualize MRO
import inspect
print(inspect.getmro(D))
# (__main__.D, __main__.B, __main__.C, __main__.A, object)
# Complex example
class X: pass
class Y: pass
class Z: pass
class A(X, Y): pass
class B(Y, Z): pass
class C(A, B): pass
print(C.__mro__)
# (C, A, X, Y, B, Z, object)Practical MRO Example:
class Logger:
def __init__(self, name):
self.name = name
self.logs = []
def log(self, message):
entry = f"[{self.name}] {message}"
self.logs.append(entry)
print(entry)
class FileHandler:
def __init__(self, filename):
self.filename = filename
def write(self, data):
with open(self.filename, 'a') as f:
f.write(data + '\n')
def read(self):
with open(self.filename, 'r') as f:
return f.read()
class LogFileHandler(Logger, FileHandler):
def __init__(self, name, filename):
# Careful: both parents have __init__
Logger.__init__(self, name)
FileHandler.__init__(self, filename)
def log_to_file(self, message):
self.log(message)
self.write(message)
# MRO determines method lookup
handler = LogFileHandler("test", "log.txt")
handler.log("Starting") # From Logger
handler.write("Direct write") # From FileHandler
handler.log_to_file("Both logging and writing")
print(LogFileHandler.__mro__)Polymorphism allows objects of different types to respond to the same interface:
Duck Typing: "If it walks like a duck and quacks like a duck, it's a duck."
class Duck:
def quack(self):
return "Duck quacks"
def walk(self):
return "Duck waddles"
class Person:
def quack(self):
return "Person imitates a duck"
def walk(self):
return "Person walks"
class Robot:
def quack(self):
return "Robot says 'Quack sequence initiated'"
def walk(self):
return "Robot moves on wheels"
def make_it_quack(entity):
"""Works with any object that has a quack method."""
print(entity.quack())
print(entity.walk())
# All work because they implement the required interface
make_it_quack(Duck())
make_it_quack(Person())
make_it_quack(Robot())Polymorphic Methods:
class PaymentProcessor:
def process_payment(self, amount):
raise NotImplementedError
class CreditCardProcessor(PaymentProcessor):
def __init__(self, card_number, expiry):
self.card_number = card_number
self.expiry = expiry
def process_payment(self, amount):
# Credit card processing logic
return f"Processing ${amount} via Credit Card ****{self.card_number[-4:]}"
class PayPalProcessor(PaymentProcessor):
def __init__(self, email):
self.email = email
def process_payment(self, amount):
# PayPal processing logic
return f"Processing ${amount} via PayPal account {self.email}"
class CryptoProcessor(PaymentProcessor):
def __init__(self, wallet_address):
self.wallet_address = wallet_address
def process_payment(self, amount):
# Crypto processing logic
return f"Processing ${amount} worth of crypto to {self.wallet_address[:10]}..."
def checkout(cart, payment_processor):
"""Process checkout with any payment processor."""
total = sum(cart.values())
return payment_processor.process_payment(total)
# Usage
cart = {"item1": 25.99, "item2": 34.50, "item3": 12.75}
cc_processor = CreditCardProcessor("1234567890123456", "12/25")
paypal_processor = PayPalProcessor("user@example.com")
crypto_processor = CryptoProcessor("0x742d35Cc6634C0532925a3b844Bc454e4438f44e")
print(checkout(cart, cc_processor))
print(checkout(cart, paypal_processor))
print(checkout(cart, crypto_processor))Encapsulation bundles data and methods and restricts direct access to some components:
Public, Protected, and Private:
class BankAccount:
"""Demonstrate encapsulation levels."""
# Class constant (public)
INTEREST_RATE = 0.03
def __init__(self, account_holder, initial_balance):
# Public attribute
self.account_holder = account_holder
# Protected attribute (convention: single underscore)
self._account_number = self._generate_account_number()
# Private attribute (name mangling: double underscore)
self.__balance = initial_balance
# Protected list of transactions
self._transactions = []
# Record initial deposit
self._record_transaction("OPEN", initial_balance)
def _generate_account_number(self):
"""Protected method for internal use."""
import random
return f"ACC-{random.randint(10000, 99999)}"
def _record_transaction(self, type_, amount):
"""Protected method to record transactions."""
from datetime import datetime
self._transactions.append({
'type': type_,
'amount': amount,
'balance': self.__balance,
'timestamp': datetime.now()
})
# Public interface
def deposit(self, amount):
"""Public method to deposit money."""
if amount <= 0:
raise ValueError("Deposit amount must be positive")
self.__balance += amount
self._record_transaction("DEPOSIT", amount)
return f"Deposited ${amount}. New balance: ${self.__balance}"
def withdraw(self, amount):
"""Public method to withdraw money."""
if amount <= 0:
raise ValueError("Withdrawal amount must be positive")
if amount > self.__balance:
raise ValueError("Insufficient funds")
self.__balance -= amount
self._record_transaction("WITHDRAW", amount)
return f"Withdrew ${amount}. New balance: ${self.__balance}"
def get_balance(self):
"""Public method to check balance."""
return self.__balance
def get_statement(self):
"""Get transaction history."""
return self._transactions.copy() # Return copy to protect internal list
@property
def account_number(self):
"""Public property for read-only access."""
return self._account_number
# Usage
account = BankAccount("Alice Smith", 1000)
# Public interface
print(account.account_holder) # Public - accessible
print(account.deposit(500))
print(account.withdraw(200))
print(f"Balance: ${account.get_balance()}")
# Protected - accessible but "please don't"
print(account._account_number) # Works but not recommended
print(account._transactions) # Works but not recommended
# Private - name mangling makes it harder to access accidentally
# print(account.__balance) # AttributeError
# But still accessible if you know the mangled name
print(account._BankAccount__balance) # Works but don't do this!
# Properties provide controlled access
print(account.account_number) # Read-only property
# account.account_number = "NEW" # AttributeError (no setter)Property Decorators for Encapsulation:
class Temperature:
def __init__(self, celsius=0):
self._celsius = celsius
@property
def celsius(self):
"""Get temperature in Celsius."""
return self._celsius
@celsius.setter
def celsius(self, value):
"""Set temperature in Celsius with validation."""
if value < -273.15:
raise ValueError("Temperature below absolute zero")
self._celsius = value
@property
def fahrenheit(self):
"""Get temperature in Fahrenheit (computed)."""
return (self._celsius * 9/5) + 32
@fahrenheit.setter
def fahrenheit(self, value):
"""Set temperature using Fahrenheit."""
self.celsius = (value - 32) * 5/9
@property
def kelvin(self):
"""Get temperature in Kelvin."""
return self._celsius + 273.15
@kelvin.setter
def kelvin(self, value):
"""Set temperature using Kelvin."""
if value < 0:
raise ValueError("Kelvin cannot be negative")
self.celsius = value - 273.15
# Usage
temp = Temperature(25)
print(temp.celsius) # 25
print(temp.fahrenheit) # 77.0
print(temp.kelvin) # 298.15
temp.fahrenheit = 100
print(temp.celsius) # 37.777...
# temp.celsius = -300 # ValueError: Temperature below absolute zeroAbstraction hides complex implementation details and shows only essential features:
Abstract Base Classes:
from abc import ABC, abstractmethod
from typing import List, Dict, Any
import json
import csv
class DataExporter(ABC):
"""Abstract base class for data exporters."""
@abstractmethod
def export(self, data: List[Dict[str, Any]], destination: str) -> bool:
"""Export data to destination."""
pass
@abstractmethod
def validate(self, data: List[Dict[str, Any]]) -> bool:
"""Validate data before export."""
pass
def export_with_validation(self, data: List[Dict[str, Any]], destination: str) -> bool:
"""Template method with validation."""
if not self.validate(data):
print("Validation failed")
return False
try:
result = self.export(data, destination)
print(f"Export successful: {result}")
return True
except Exception as e:
print(f"Export failed: {e}")
return False
class JSONExporter(DataExporter):
def validate(self, data: List[Dict[str, Any]]) -> bool:
"""Validate JSON-serializable data."""
try:
json.dumps(data)
return True
except:
return False
def export(self, data: List[Dict[str, Any]], destination: str) -> bool:
"""Export data to JSON file."""
with open(destination, 'w') as f:
json.dump(data, f, indent=2)
return True
class CSVExporter(DataExporter):
def validate(self, data: List[Dict[str, Any]]) -> bool:
"""Validate CSV-suitable data."""
if not data:
return False
# Check all dicts have same keys
keys = set(data[0].keys())
return all(set(d.keys()) == keys for d in data)
def export(self, data: List[Dict[str, Any]], destination: str) -> bool:
"""Export data to CSV file."""
if not data:
return False
fieldnames = data[0].keys()
with open(destination, 'w', newline='') as f:
writer = csv.DictWriter(f, fieldnames=fieldnames)
writer.writeheader()
writer.writerows(data)
return True
# Usage
data = [
{"name": "Alice", "age": 30, "city": "New York"},
{"name": "Bob", "age": 25, "city": "Los Angeles"}
]
json_exporter = JSONExporter()
csv_exporter = CSVExporter()
json_exporter.export_with_validation(data, "data.json")
csv_exporter.export_with_validation(data, "data.csv")Interface-like Classes:
class Drawable(ABC):
"""Interface for drawable objects."""
@abstractmethod
def draw(self, canvas):
"""Draw the object on canvas."""
pass
@abstractmethod
def get_bounding_box(self):
"""Get bounding box coordinates."""
pass
class Clickable(ABC):
"""Interface for clickable objects."""
@abstractmethod
def on_click(self, x, y):
"""Handle click event."""
pass
@abstractmethod
def is_point_inside(self, x, y):
"""Check if point is inside object."""
pass
class Button(Drawable, Clickable):
"""Concrete class implementing multiple interfaces."""
def __init__(self, x, y, width, height, text):
self.x = x
self.y = y
self.width = width
self.height = height
self.text = text
self.is_pressed = False
def draw(self, canvas):
"""Draw button on canvas."""
# Simulated drawing
print(f"Drawing button '{self.text}' at ({self.x}, {self.y})")
if self.is_pressed:
print(" Button is pressed")
def get_bounding_box(self):
"""Return bounding box as (min_x, min_y, max_x, max_y)."""
return (self.x, self.y, self.x + self.width, self.y + self.height)
def on_click(self, x, y):
"""Handle click event."""
if self.is_point_inside(x, y):
self.is_pressed = not self.is_pressed
print(f"Button '{self.text}' {'pressed' if self.is_pressed else 'released'}")
return True
return False
def is_point_inside(self, x, y):
"""Check if point is inside button."""
return (self.x <= x <= self.x + self.width and
self.y <= y <= self.y + self.height)
# Usage
button = Button(10, 10, 100, 30, "Click Me")
button.draw()
button.on_click(50, 25)
button.draw()Python provides special methods that begin and end with double underscores:
Comprehensive Magic Methods:
class Vector:
"""A 2D vector with comprehensive magic methods."""
def __init__(self, x, y):
self.x = x
self.y = y
# String representations
def __str__(self):
"""String representation for users."""
return f"({self.x}, {self.y})"
def __repr__(self):
"""String representation for developers."""
return f"Vector({self.x}, {self.y})"
# Arithmetic operations
def __add__(self, other):
"""Addition: v1 + v2."""
if isinstance(other, Vector):
return Vector(self.x + other.x, self.y + other.y)
return Vector(self.x + other, self.y + other)
def __sub__(self, other):
"""Subtraction: v1 - v2."""
if isinstance(other, Vector):
return Vector(self.x - other.x, self.y - other.y)
return Vector(self.x - other, self.y - other)
def __mul__(self, other):
"""Multiplication: v * scalar or v * v (dot product)."""
if isinstance(other, (int, float)):
return Vector(self.x * other, self.y * other)
if isinstance(other, Vector):
return self.x * other.x + self.y * other.y # Dot product
return NotImplemented
def __rmul__(self, other):
"""Right multiplication: scalar * v."""
return self.__mul__(other)
def __truediv__(self, other):
"""Division: v / scalar."""
if isinstance(other, (int, float)):
return Vector(self.x / other, self.y / other)
return NotImplemented
# Unary operations
def __neg__(self):
"""Unary negation: -v."""
return Vector(-self.x, -self.y)
def __abs__(self):
"""Absolute value: magnitude."""
return (self.x**2 + self.y**2) ** 0.5
# Comparison operations
def __eq__(self, other):
"""Equality: v1 == v2."""
if not isinstance(other, Vector):
return False
return self.x == other.x and self.y == other.y
def __ne__(self, other):
"""Inequality: v1 != v2."""
return not self.__eq__(other)
def __lt__(self, other):
"""Less than (by magnitude): v1 < v2."""
if not isinstance(other, Vector):
return NotImplemented
return abs(self) < abs(other)
def __le__(self, other):
"""Less than or equal."""
return self < other or self == other
def __gt__(self, other):
"""Greater than."""
return not self <= other
def __ge__(self, other):
"""Greater than or equal."""
return not self < other
# Container emulation
def __len__(self):
"""Length (number of components)."""
return 2
def __getitem__(self, index):
"""Get component by index."""
if index == 0 or index == -2:
return self.x
if index == 1 or index == -1:
return self.y
raise IndexError("Vector index out of range")
def __setitem__(self, index, value):
"""Set component by index."""
if index == 0 or index == -2:
self.x = value
elif index == 1 or index == -1:
self.y = value
else:
raise IndexError("Vector index out of range")
def __contains__(self, item):
"""Check if value equals any component."""
return item == self.x or item == self.y
# Callable
def __call__(self, scalar):
"""Scale vector when called."""
return Vector(self.x * scalar, self.y * scalar)
# Context manager
def __enter__(self):
"""Enter context."""
print(f"Entering context with {self}")
return self
def __exit__(self, exc_type, exc_val, exc_tb):
"""Exit context."""
print(f"Exiting context with {self}")
if exc_type:
print(f"Exception: {exc_val}")
return False # Don't suppress exceptions
# Iterator
def __iter__(self):
"""Make vector iterable."""
yield self.x
yield self.y
# Hashable
def __hash__(self):
"""Make vector hashable."""
return hash((self.x, self.y))
# Boolean
def __bool__(self):
"""Boolean value (True if non-zero)."""
return self.x != 0 or self.y != 0
# Test all magic methods
v1 = Vector(2, 3)
v2 = Vector(4, 5)
print(str(v1)) # (2, 3)
print(repr(v1)) # Vector(2, 3)
print(v1 + v2) # (6, 8)
print(v1 - v2) # (-2, -2)
print(v1 * 3) # (6, 9)
print(3 * v1) # (6, 9)
print(v1 * v2) # 2*4 + 3*5 = 23 (dot product)
print(v1 / 2) # (1.0, 1.5)
print(-v1) # (-2, -3)
print(abs(v1)) # 3.6055...
print(v1 == Vector(2, 3)) # True
print(v1 != v2) # True
print(v1 < v2) # True (3.6 < 6.4)
print(v1 > Vector(1, 1)) # True
print(len(v1)) # 2
print(v1[0]) # 2
print(v1[1]) # 3
v1[0] = 10
print(v1) # (10, 3)
print(10 in v1) # True
scaled = v1(2) # Callable
print(scaled) # (20, 6)
# Iteration
for component in Vector(1, 2):
print(component) # 1, 2
# Hashable
s = {Vector(1, 2), Vector(3, 4)}
print(s)
# Boolean
print(bool(Vector(0, 0))) # False
print(bool(Vector(1, 0))) # True
# Context manager
with Vector(5, 5) as v:
print(v * 2)
# Entering context with (5, 5)
# (10, 10)
# Exiting context with (5, 5)Building on magic methods, we can overload operators for custom classes:
class Fraction:
"""A fraction class with operator overloading."""
def __init__(self, numerator, denominator=1):
if denominator == 0:
raise ValueError("Denominator cannot be zero")
# Reduce fraction
g = self._gcd(abs(numerator), abs(denominator))
self.numerator = numerator // g
self.denominator = denominator // g
# Keep denominator positive
if self.denominator < 0:
self.numerator = -self.numerator
self.denominator = -self.denominator
@staticmethod
def _gcd(a, b):
"""Euclidean algorithm for GCD."""
while b:
a, b = b, a % b
return a
def __str__(self):
return f"{self.numerator}/{self.denominator}"
def __repr__(self):
return f"Fraction({self.numerator}, {self.denominator})"
def __float__(self):
"""Convert to float."""
return self.numerator / self.denominator
def __int__(self):
"""Convert to int (floor division)."""
return self.numerator // self.denominator
# Arithmetic
def __add__(self, other):
"""Addition: f1 + f2 or f + number."""
if isinstance(other, Fraction):
num = (self.numerator * other.denominator +
other.numerator * self.denominator)
den = self.denominator * other.denominator
return Fraction(num, den)
elif isinstance(other, (int, float)):
return self + Fraction(other)
return NotImplemented
def __radd__(self, other):
"""Right addition: number + f."""
return self.__add__(other)
def __sub__(self, other):
"""Subtraction."""
if isinstance(other, Fraction):
return self + (-other)
return self - Fraction(other)
def __rsub__(self, other):
"""Right subtraction: number - f."""
return Fraction(other) - self
def __mul__(self, other):
"""Multiplication."""
if isinstance(other, Fraction):
return Fraction(
self.numerator * other.numerator,
self.denominator * other.denominator
)
return self * Fraction(other)
def __rmul__(self, other):
"""Right multiplication."""
return self.__mul__(other)
def __truediv__(self, other):
"""Division."""
if isinstance(other, Fraction):
return self * Fraction(other.denominator, other.numerator)
return self / Fraction(other)
def __rtruediv__(self, other):
"""Right division."""
return Fraction(other) / self
def __neg__(self):
"""Unary negation."""
return Fraction(-self.numerator, self.denominator)
def __pos__(self):
"""Unary plus."""
return self
def __abs__(self):
"""Absolute value."""
return Fraction(abs(self.numerator), self.denominator)
def __pow__(self, power):
"""Exponentiation."""
if isinstance(power, int):
return Fraction(
self.numerator ** abs(power),
self.denominator ** abs(power)
)
return NotImplemented
# Comparisons
def __eq__(self, other):
"""Equality."""
if not isinstance(other, Fraction):
try:
other = Fraction(other)
except:
return False
return (self.numerator == other.numerator and
self.denominator == other.denominator)
def __lt__(self, other):
"""Less than."""
if not isinstance(other, Fraction):
other = Fraction(other)
return (self.numerator * other.denominator <
other.numerator * self.denominator)
def __le__(self, other):
return self < other or self == other
def __gt__(self, other):
return not self <= other
def __ge__(self, other):
return not self < other
# Test
f1 = Fraction(1, 2)
f2 = Fraction(3, 4)
print(f1 + f2) # 5/4
print(f1 - f2) # -1/4
print(f1 * f2) # 3/8
print(f1 / f2) # 2/3
print(f1 + 1) # 3/2
print(1 + f1) # 3/2
print(f1 * 2) # 1/1
print(-f1) # -1/2
print(abs(Fraction(-1, 2))) # 1/2
print(f1 < f2) # True
print(f1 == Fraction(2, 4)) # TrueEnsures a class has only one instance and provides global access to it:
Classic Singleton:
class Singleton:
"""Classic singleton implementation."""
_instance = None
def __new__(cls, *args, **kwargs):
if cls._instance is None:
print("Creating the singleton instance")
cls._instance = super().__new__(cls)
return cls._instance
def __init__(self, value=None):
if not hasattr(self, 'initialized'):
print("Initializing the singleton")
self.value = value
self.initialized = True
def __str__(self):
return f"Singleton(value={self.value})"
# Test
s1 = Singleton(10)
s2 = Singleton(20)
print(s1 is s2) # True
print(s1) # Singleton(value=10)
print(s2) # Singleton(value=10) (initialized only once)Thread-Safe Singleton:
import threading
class ThreadSafeSingleton:
"""Thread-safe singleton using lock."""
_instance = None
_lock = threading.Lock()
def __new__(cls, *args, **kwargs):
# Double-checked locking
if cls._instance is None:
with cls._lock:
if cls._instance is None:
cls._instance = super().__new__(cls)
return cls._instance
def __init__(self, name=None):
if not hasattr(self, 'initialized'):
self.name = name or "Default"
self.initialized = True
# Test with multiple threads
def create_singleton(thread_id):
s = ThreadSafeSingleton(f"Thread-{thread_id}")
print(f"Thread {thread_id} got: {s.name}")
threads = []
for i in range(5):
t = threading.Thread(target=create_singleton, args=(i,))
threads.append(t)
t.start()
for t in threads:
t.join()Singleton Decorator:
def singleton(cls):
"""Decorator to make a class a singleton."""
instances = {}
def get_instance(*args, **kwargs):
if cls not in instances:
instances[cls] = cls(*args, **kwargs)
return instances[cls]
return get_instance
@singleton
class DatabaseConnection:
def __init__(self, host, port):
self.host = host
self.port = port
self.connection = None
print(f"Creating database connection to {host}:{port}")
def connect(self):
if not self.connection:
self.connection = f"Connected to {self.host}:{self.port}"
return self.connection
# Test
db1 = DatabaseConnection("localhost", 5432)
db2 = DatabaseConnection("remote", 3306) # Returns same instance
print(db1.connect())
print(db2.connect())
print(db1 is db2) # TrueMetaclass Singleton:
class SingletonMeta(type):
"""Metaclass for singleton pattern."""
_instances = {}
def __call__(cls, *args, **kwargs):
if cls not in cls._instances:
cls._instances[cls] = super().__call__(*args, **kwargs)
return cls._instances[cls]
class Logger(metaclass=SingletonMeta):
def __init__(self):
self.logs = []
def log(self, message):
self.logs.append(message)
print(f"Log: {message}")
def get_logs(self):
return self.logs.copy()
# Test
logger1 = Logger()
logger2 = Logger()
logger1.log("First message")
logger2.log("Second message")
print(logger1.get_logs()) # ['First message', 'Second message']
print(logger1 is logger2) # TrueCreates objects without specifying the exact class:
Simple Factory:
from abc import ABC, abstractmethod
# Product classes
class Vehicle(ABC):
@abstractmethod
def drive(self):
pass
@abstractmethod
def get_info(self):
pass
class Car(Vehicle):
def __init__(self, model):
self.model = model
def drive(self):
return f"Driving car: {self.model}"
def get_info(self):
return f"Car: {self.model}, 4 wheels"
class Motorcycle(Vehicle):
def __init__(self, model):
self.model = model
def drive(self):
return f"Riding motorcycle: {self.model}"
def get_info(self):
return f"Motorcycle: {self.model}, 2 wheels"
class Truck(Vehicle):
def __init__(self, model, capacity):
self.model = model
self.capacity = capacity
def drive(self):
return f"Driving truck: {self.model}"
def get_info(self):
return f"Truck: {self.model}, capacity: {self.capacity} tons"
# Simple Factory
class VehicleFactory:
@staticmethod
def create_vehicle(vehicle_type, **kwargs):
vehicles = {
'car': Car,
'motorcycle': Motorcycle,
'truck': Truck
}
vehicle_class = vehicles.get(vehicle_type.lower())
if not vehicle_class:
raise ValueError(f"Unknown vehicle type: {vehicle_type}")
return vehicle_class(**kwargs)
# Usage
factory = VehicleFactory()
car = factory.create_vehicle('car', model="Sedan")
print(car.drive())
print(car.get_info())
truck = factory.create_vehicle('truck', model="Heavy Duty", capacity=10)
print(truck.drive())
print(truck.get_info())Factory Method Pattern:
from abc import ABC, abstractmethod
# Creator abstract class
class DocumentCreator(ABC):
@abstractmethod
def create_document(self):
"""Factory method."""
pass
def create_and_process(self):
"""Template method using factory method."""
doc = self.create_document()
doc.create()
doc.open()
return doc
# Product classes
class Document(ABC):
@abstractmethod
def create(self):
pass
@abstractmethod
def open(self):
pass
@abstractmethod
def get_type(self):
pass
class PDFDocument(Document):
def create(self):
print("Creating PDF document")
def open(self):
print("Opening PDF document with Adobe Reader")
def get_type(self):
return "PDF"
class WordDocument(Document):
def create(self):
print("Creating Word document")
def open(self):
print("Opening Word document with Microsoft Word")
def get_type(self):
return "Word"
class HTMLDocument(Document):
def create(self):
print("Creating HTML document")
def open(self):
print("Opening HTML document with web browser")
def get_type(self):
return "HTML"
# Concrete creators
class PDFCreator(DocumentCreator):
def create_document(self):
return PDFDocument()
class WordCreator(DocumentCreator):
def create_document(self):
return WordDocument()
class HTMLCreator(DocumentCreator):
def create_document(self):
return HTMLDocument()
# Usage
def client_code(creator: DocumentCreator):
print(f"Working with {creator.__class__.__name__}")
doc = creator.create_and_process()
print(f"Created document type: {doc.get_type()}\n")
client_code(PDFCreator())
client_code(WordCreator())
client_code(HTMLCreator())Abstract Factory Pattern:
from abc import ABC, abstractmethod
# Abstract products
class Button(ABC):
@abstractmethod
def render(self):
pass
@abstractmethod
def click(self):
pass
class TextBox(ABC):
@abstractmethod
def render(self):
pass
@abstractmethod
def set_text(self, text):
pass
class CheckBox(ABC):
@abstractmethod
def render(self):
pass
@abstractmethod
def toggle(self):
pass
# Concrete products for Windows
class WindowsButton(Button):
def render(self):
return "Rendering Windows-style button"
def click(self):
return "Windows button clicked"
class WindowsTextBox(TextBox):
def __init__(self):
self.text = ""
def render(self):
return f"Rendering Windows-style textbox with text: {self.text}"
def set_text(self, text):
self.text = text
class WindowsCheckBox(CheckBox):
def __init__(self):
self.checked = False
def render(self):
return f"Rendering Windows-style checkbox {'✓' if self.checked else '□'}"
def toggle(self):
self.checked = not self.checked
return f"Windows checkbox toggled to {self.checked}"
# Concrete products for macOS
class MacButton(Button):
def render(self):
return "Rendering macOS-style button"
def click(self):
return "macOS button clicked"
class MacTextBox(TextBox):
def __init__(self):
self.text = ""
def render(self):
return f"Rendering macOS-style textbox with text: {self.text}"
def set_text(self, text):
self.text = text
class MacCheckBox(CheckBox):
def __init__(self):
self.checked = False
def render(self):
return f"Rendering macOS-style checkbox {'✅' if self.checked else '⬜'}"
def toggle(self):
self.checked = not self.checked
return f"macOS checkbox toggled to {self.checked}"
# Abstract factory
class GUIFactory(ABC):
@abstractmethod
def create_button(self) -> Button:
pass
@abstractmethod
def create_textbox(self) -> TextBox:
pass
@abstractmethod
def create_checkbox(self) -> CheckBox:
pass
# Concrete factories
class WindowsFactory(GUIFactory):
def create_button(self) -> Button:
return WindowsButton()
def create_textbox(self) -> TextBox:
return WindowsTextBox()
def create_checkbox(self) -> CheckBox:
return WindowsCheckBox()
class MacFactory(GUIFactory):
def create_button(self) -> Button:
return MacButton()
def create_textbox(self) -> TextBox:
return MacTextBox()
def create_checkbox(self) -> CheckBox:
return MacCheckBox()
# Client code
def create_ui(factory: GUIFactory):
button = factory.create_button()
textbox = factory.create_textbox()
checkbox = factory.create_checkbox()
textbox.set_text("Hello World")
checkbox.toggle()
print(button.render())
print(button.click())
print(textbox.render())
print(checkbox.render())
return button, textbox, checkbox
# Usage based on platform
def get_factory_for_os(os_name):
factories = {
'windows': WindowsFactory,
'mac': MacFactory,
'linux': WindowsFactory # Fallback to Windows style
}
factory_class = factories.get(os_name.lower(), WindowsFactory)
return factory_class()
# Simulate different platforms
print("=== Windows UI ===")
win_factory = get_factory_for_os('windows')
create_ui(win_factory)
print("\n=== macOS UI ===")
mac_factory = get_factory_for_os('mac')
create_ui(mac_factory)Defines a one-to-many dependency between objects:
Basic Observer:
from abc import ABC, abstractmethod
from typing import List
# Observer interface
class Observer(ABC):
@abstractmethod
def update(self, subject, *args, **kwargs):
pass
# Subject (Observable)
class Subject:
def __init__(self):
self._observers: List[Observer] = []
self._state = None
def attach(self, observer: Observer):
if observer not in self._observers:
self._observers.append(observer)
def detach(self, observer: Observer):
try:
self._observers.remove(observer)
except ValueError:
pass
def notify(self, *args, **kwargs):
for observer in self._observers:
observer.update(self, *args, **kwargs)
@property
def state(self):
return self._state
@state.setter
def state(self, value):
self._state = value
self.notify(state=value)
# Concrete Observers
class ConcreteObserverA(Observer):
def update(self, subject, *args, **kwargs):
print(f"Observer A: Subject's state changed to {kwargs.get('state')}")
print(f"Observer A: Reacting to change")
class ConcreteObserverB(Observer):
def update(self, subject, *args, **kwargs):
print(f"Observer B: Subject's state changed to {kwargs.get('state')}")
if kwargs.get('state') > 5:
print("Observer B: State is >5, taking special action")
# Usage
subject = Subject()
observer_a = ConcreteObserverA()
observer_b = ConcreteObserverB()
subject.attach(observer_a)
subject.attach(observer_b)
subject.state = 3
print("---")
subject.state = 7
print("---")
subject.detach(observer_a)
subject.state = 1Event System with Observer:
from typing import Dict, List, Callable
from enum import Enum
class EventType(Enum):
USER_CREATED = "user_created"
USER_UPDATED = "user_updated"
USER_DELETED = "user_deleted"
ORDER_PLACED = "order_placed"
PAYMENT_RECEIVED = "payment_received"
class EventSystem:
"""Central event system using observer pattern."""
def __init__(self):
self._listeners: Dict[EventType, List[Callable]] = {
event_type: [] for event_type in EventType
}
def subscribe(self, event_type: EventType, callback: Callable):
"""Subscribe to an event."""
if callback not in self._listeners[event_type]:
self._listeners[event_type].append(callback)
def unsubscribe(self, event_type: EventType, callback: Callable):
"""Unsubscribe from an event."""
if callback in self._listeners[event_type]:
self._listeners[event_type].remove(callback)
def emit(self, event_type: EventType, data=None):
"""Emit an event to all subscribers."""
print(f"\n[EventSystem] Emitting: {event_type.value}")
for callback in self._listeners[event_type]:
callback(data)
def emit_async(self, event_type: EventType, data=None):
"""Emit event asynchronously (simplified)."""
import threading
thread = threading.Thread(target=self.emit, args=(event_type, data))
thread.start()
return thread
# Event listeners
class EmailService:
def __init__(self, event_system: EventSystem):
event_system.subscribe(EventType.USER_CREATED, self.send_welcome_email)
event_system.subscribe(EventType.ORDER_PLACED, self.send_order_confirmation)
def send_welcome_email(self, user_data):
print(f"[EmailService] Sending welcome email to {user_data.get('email')}")
def send_order_confirmation(self, order_data):
print(f"[EmailService] Sending order confirmation for order #{order_data.get('order_id')}")
class AnalyticsService:
def __init__(self, event_system: EventSystem):
event_system.subscribe(EventType.USER_CREATED, self.track_user_signup)
event_system.subscribe(EventType.USER_UPDATED, self.track_user_update)
event_system.subscribe(EventType.ORDER_PLACED, self.track_order)
event_system.subscribe(EventType.PAYMENT_RECEIVED, self.track_payment)
def track_user_signup(self, user_data):
print(f"[Analytics] Tracking user signup: {user_data.get('username')}")
def track_user_update(self, user_data):
print(f"[Analytics] Tracking user update for user #{user_data.get('user_id')}")
def track_order(self, order_data):
print(f"[Analytics] Tracking order #{order_data.get('order_id')}, amount: ${order_data.get('amount')}")
def track_payment(self, payment_data):
print(f"[Analytics] Tracking payment: ${payment_data.get('amount')}")
class AuditService:
def __init__(self, event_system: EventSystem):
# Subscribe to all events for auditing
for event_type in EventType:
event_system.subscribe(event_type, self.log_event)
def log_event(self, data):
import datetime
print(f"[Audit] {datetime.datetime.now()}: Event occurred with data: {data}")
# Usage
event_system = EventSystem()
# Initialize services
email_service = EmailService(event_system)
analytics_service = AnalyticsService(event_system)
audit_service = AuditService(event_system)
# Simulate events
print("\n--- User Registration ---")
event_system.emit(EventType.USER_CREATED, {
'user_id': 123,
'username': 'john_doe',
'email': 'john@example.com'
})
print("\n--- Order Placement ---")
event_system.emit(EventType.ORDER_PLACED, {
'order_id': 1001,
'user_id': 123,
'amount': 99.99,
'items': ['laptop', 'mouse']
})
print("\n--- Payment Received ---")
event_system.emit(EventType.PAYMENT_RECEIVED, {
'payment_id': 'pay_123',
'order_id': 1001,
'amount': 99.99
})Defines a family of algorithms and makes them interchangeable:
Basic Strategy:
from abc import ABC, abstractmethod
from typing import List
# Strategy interface
class SortStrategy(ABC):
@abstractmethod
def sort(self, data: List) -> List:
pass
# Concrete strategies
class BubbleSort(SortStrategy):
def sort(self, data: List) -> List:
print("Using Bubble Sort")
arr = data.copy()
n = len(arr)
for i in range(n):
for j in range(0, n - i - 1):
if arr[j] > arr[j + 1]:
arr[j], arr[j + 1] = arr[j + 1], arr[j]
return arr
class QuickSort(SortStrategy):
def sort(self, data: List) -> List:
print("Using Quick Sort")
if len(data) <= 1:
return data
pivot = data[len(data) // 2]
left = [x for x in data if x < pivot]
middle = [x for x in data if x == pivot]
right = [x for x in data if x > pivot]
return self.sort(left) + middle + self.sort(right)
class MergeSort(SortStrategy):
def sort(self, data: List) -> List:
print("Using Merge Sort")
if len(data) <= 1:
return data
mid = len(data) // 2
left = self.sort(data[:mid])
right = self.sort(data[mid:])
return self._merge(left, right)
def _merge(self, left, right):
result = []
i = j = 0
while i < len(left) and j < len(right):
if left[i] <= right[j]:
result.append(left[i])
i += 1
else:
result.append(right[j])
j += 1
result.extend(left[i:])
result.extend(right[j:])
return result
# Context
class Sorter:
def __init__(self, strategy: SortStrategy = None):
self._strategy = strategy
@property
def strategy(self):
return self._strategy
@strategy.setter
def strategy(self, strategy: SortStrategy):
self._strategy = strategy
def sort(self, data: List) -> List:
if not self._strategy:
raise ValueError("No sorting strategy set")
return self._strategy.sort(data)
# Usage
data = [64, 34, 25, 12, 22, 11, 90]
sorter = Sorter()
# Use different strategies
sorter.strategy = BubbleSort()
print(sorter.sort(data))
sorter.strategy = QuickSort()
print(sorter.sort(data))
sorter.strategy = MergeSort()
print(sorter.sort(data))Payment Processing with Strategy:
from abc import ABC, abstractmethod
from typing import Dict
# Strategy interface
class PaymentStrategy(ABC):
@abstractmethod
def pay(self, amount: float) -> Dict:
pass
@abstractmethod
def validate(self) -> bool:
pass
# Concrete strategies
class CreditCardPayment(PaymentStrategy):
def __init__(self, card_number: str, expiry: str, cvv: str, name: str):
self.card_number = card_number
self.expiry = expiry
self.cvv = cvv
self.name = name
def validate(self) -> bool:
# Simple validation
return (len(self.card_number) == 16 and
len(self.cvv) == 3 and
self.expiry.replace('/', '').isdigit())
def pay(self, amount: float) -> Dict:
if not self.validate():
return {'success': False, 'message': 'Invalid card details'}
# Mock payment processing
return {
'success': True,
'method': 'Credit Card',
'amount': amount,
'transaction_id': f"CC-{hash(self.card_number[-4:])}",
'message': f"Charged ${amount} to card ending in {self.card_number[-4:]}"
}
class PayPalPayment(PaymentStrategy):
def __init__(self, email: str, password: str):
self.email = email
self.password = password
def validate(self) -> bool:
# Simple email validation
return '@' in self.email and len(self.password) >= 6
def pay(self, amount: float) -> Dict:
if not self.validate():
return {'success': False, 'message': 'Invalid PayPal credentials'}
return {
'success': True,
'method': 'PayPal',
'amount': amount,
'transaction_id': f"PP-{hash(self.email)}",
'message': f"Paid ${amount} via PayPal account {self.email}"
}
class CryptoPayment(PaymentStrategy):
def __init__(self, wallet_address: str, currency: str = 'BTC'):
self.wallet_address = wallet_address
self.currency = currency
def validate(self) -> bool:
# Basic wallet address validation
return len(self.wallet_address) >= 26 and self.wallet_address.startswith(('1', '3', 'bc1'))
def pay(self, amount: float) -> Dict:
if not self.validate():
return {'success': False, 'message': 'Invalid wallet address'}
# Mock crypto conversion
crypto_amount = amount / 50000 if self.currency == 'BTC' else amount / 3000
return {
'success': True,
'method': f'Cryptocurrency ({self.currency})',
'amount': amount,
'crypto_amount': crypto_amount,
'transaction_id': f"CRYPTO-{self.wallet_address[:10]}",
'message': f"Sent {crypto_amount:.6f} {self.currency} from wallet {self.wallet_address[:10]}..."
}
# Context
class PaymentProcessor:
def __init__(self):
self._payment_strategy = None
self.transactions = []
def set_payment_method(self, strategy: PaymentStrategy):
self._payment_strategy = strategy
def process_payment(self, amount: float) -> Dict:
if not self._payment_strategy:
return {'success': False, 'message': 'No payment method selected'}
result = self._payment_strategy.pay(amount)
if result['success']:
self.transactions.append(result)
return result
def get_transaction_history(self):
return self.transactions.copy()
# Usage
processor = PaymentProcessor()
# Pay with credit card
cc_payment = CreditCardPayment(
"1234567890123456",
"12/25",
"123",
"John Doe"
)
processor.set_payment_method(cc_payment)
result = processor.process_payment(99.99)
print(result['message'])
# Pay with PayPal
paypal_payment = PayPalPayment("john@example.com", "securepass123")
processor.set_payment_method(paypal_payment)
result = processor.process_payment(49.50)
print(result['message'])
# Pay with Crypto
crypto_payment = CryptoPayment("1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa", "BTC")
processor.set_payment_method(crypto_payment)
result = processor.process_payment(250.00)
print(result['message'])
# View all transactions
print("\nTransaction History:")
for t in processor.get_transaction_history():
print(f" {t['method']}: ${t['amount']} - {t['transaction_id']}")Attaches additional responsibilities to objects dynamically:
Basic Decorator:
from abc import ABC, abstractmethod
# Component interface
class Coffee(ABC):
@abstractmethod
def cost(self) -> float:
pass
@abstractmethod
def description(self) -> str:
pass
# Concrete component
class SimpleCoffee(Coffee):
def cost(self) -> float:
return 2.0
def description(self) -> str:
return "Simple coffee"
# Base decorator
class CoffeeDecorator(Coffee):
def __init__(self, coffee: Coffee):
self._coffee = coffee
def cost(self) -> float:
return self._coffee.cost()
def description(self) -> str:
return self._coffee.description()
# Concrete decorators
class MilkDecorator(CoffeeDecorator):
def cost(self) -> float:
return self._coffee.cost() + 0.5
def description(self) -> str:
return self._coffee.description() + ", milk"
class SugarDecorator(CoffeeDecorator):
def cost(self) -> float:
return self._coffee.cost() + 0.2
def description(self) -> str:
return self._coffee.description() + ", sugar"
class WhippedCreamDecorator(CoffeeDecorator):
def cost(self) -> float:
return self._coffee.cost() + 0.7
def description(self) -> str:
return self._coffee.description() + ", whipped cream"
class CaramelDecorator(CoffeeDecorator):
def cost(self) -> float:
return self._coffee.cost() + 0.6
def description(self) -> str:
return self._coffee.description() + ", caramel"
# Usage
coffee = SimpleCoffee()
print(f"{coffee.description()}: ${coffee.cost()}")
# Add milk
coffee = MilkDecorator(coffee)
print(f"{coffee.description()}: ${coffee.cost()}")
# Add sugar and whipped cream
coffee = SugarDecorator(WhippedCreamDecorator(coffee))
print(f"{coffee.description()}: ${coffee.cost()}")
# Add all toppings
coffee = CaramelDecorator(
WhippedCreamDecorator(
SugarDecorator(
MilkDecorator(
SimpleCoffee()
)
)
)
)
print(f"{coffee.description()}: ${coffee.cost()}")Functional Decorator Pattern:
from functools import wraps
import time
import logging
# Decorator pattern using Python function decorators
def log_execution(func):
"""Decorator to log function execution."""
@wraps(func)
def wrapper(*args, **kwargs):
print(f"Executing {func.__name__} with args={args}, kwargs={kwargs}")
result = func(*args, **kwargs)
print(f"{func.__name__} returned {result}")
return result
return wrapper
def timer(func):
"""Decorator to time function execution."""
@wraps(func)
def wrapper(*args, **kwargs):
start = time.time()
result = func(*args, **kwargs)
end = time.time()
print(f"{func.__name__} took {end - start:.4f} seconds")
return result
return wrapper
def retry(max_attempts=3, delay=1):
"""Decorator to retry function on failure."""
def decorator(func):
@wraps(func)
def wrapper(*args, **kwargs):
for attempt in range(max_attempts):
try:
return func(*args, **kwargs)
except Exception as e:
if attempt == max_attempts - 1:
raise
print(f"Attempt {attempt + 1} failed: {e}. Retrying...")
time.sleep(delay)
return None
return wrapper
return decorator
def cache_result(func):
"""Decorator to cache function results."""
cache = {}
@wraps(func)
def wrapper(*args, **kwargs):
# Create cache key from args and kwargs
key = str(args) + str(sorted(kwargs.items()))
if key not in cache:
cache[key] = func(*args, **kwargs)
print(f"Cache miss for {func.__name__}{args}")
else:
print(f"Cache hit for {func.__name__}{args}")
return cache[key]
# Add cache management methods
wrapper.cache = cache
wrapper.clear_cache = lambda: cache.clear()
return wrapper
# Usage
@log_execution
@timer
def slow_function(n):
time.sleep(1)
return n * 2
@retry(max_attempts=3, delay=0.5)
def unstable_function():
import random
if random.random() < 0.7:
raise ValueError("Random failure")
return "Success!"
@cache_result
def fibonacci(n):
if n <= 1:
return n
return fibonacci(n - 1) + fibonacci(n - 2)
# Test
print("--- Logging and Timing ---")
result = slow_function(5)
print("\n--- Retry Pattern ---")
for i in range(3):
try:
result = unstable_function()
print(f"Result: {result}")
except Exception as e:
print(f"Failed after retries: {e}")
print("\n--- Caching ---")
print(fibonacci(10))
print(fibonacci(10)) # Cache hit
print(f"Cache contents: {fibonacci.cache}")
fibonacci.clear_cache()
print(fibonacci(10)) # Cache miss againAllows incompatible interfaces to work together:
Class Adapter:
# Target interface (what the client expects)
class MediaPlayer:
def play(self, audio_type, filename):
pass
# Adaptee (existing interface)
class AdvancedMediaPlayer:
def play_vlc(self, filename):
return f"Playing vlc file: {filename}"
def play_mp4(self, filename):
return f"Playing mp4 file: {filename}"
def play_avi(self, filename):
return f"Playing avi file: {filename}"
# Adapter
class MediaAdapter(MediaPlayer):
def __init__(self, audio_type):
self.advanced_player = AdvancedMediaPlayer()
self.audio_type = audio_type
def play(self, audio_type, filename):
if audio_type == "vlc":
return self.advanced_player.play_vlc(filename)
elif audio_type == "mp4":
return self.advanced_player.play_mp4(filename)
elif audio_type == "avi":
return self.advanced_player.play_avi(filename)
else:
return f"Unsupported format: {audio_type}"
# Client
class AudioPlayer(MediaPlayer):
def play(self, audio_type, filename):
# Built-in support for mp3
if audio_type == "mp3":
return f"Playing mp3 file: {filename}"
# Use adapter for other formats
else:
adapter = MediaAdapter(audio_type)
return adapter.play(audio_type, filename)
# Usage
player = AudioPlayer()
print(player.play("mp3", "song.mp3"))
print(player.play("mp4", "video.mp4"))
print(player.play("vlc", "movie.vlc"))
print(player.play("avi", "clip.avi"))
print(player.play("wmv", "windows.wmv"))Object Adapter:
from typing import Dict, List
# Existing classes (legacy system)
class LegacyWeatherService:
"""Legacy service with different interface."""
def get_weather_data(self, city_code):
# Simulate API call
data = {
'NYC': {'temp_f': 72, 'humidity': 65, 'conditions': 'Sunny'},
'LA': {'temp_f': 85, 'humidity': 45, 'conditions': 'Clear'},
'CHI': {'temp_f': 68, 'humidity': 70, 'conditions': 'Cloudy'}
}
return data.get(city_code, {'temp_f': 0, 'humidity': 0, 'conditions': 'Unknown'})
class NewWeatherSystem:
"""New system with different interface."""
def get_temperature_celsius(self, city_name: str) -> float:
pass
def get_humidity_percent(self, city_name: str) -> int:
pass
def get_conditions(self, city_name: str) -> str:
pass
# Adapter (makes LegacyWeatherService work with NewWeatherSystem interface)
class WeatherAdapter(NewWeatherSystem):
"""Adapter for LegacyWeatherService."""
# City name to city code mapping
CITY_CODES = {
'New York': 'NYC',
'Los Angeles': 'LA',
'Chicago': 'CHI',
'San Francisco': 'SF'
}
def __init__(self):
self.legacy_service = LegacyWeatherService()
def _fahrenheit_to_celsius(self, f_temp):
return (f_temp - 32) * 5 / 9
def get_temperature_celsius(self, city_name: str) -> float:
city_code = self.CITY_CODES.get(city_name)
if not city_code:
return 0.0
data = self.legacy_service.get_weather_data(city_code)
return self._fahrenheit_to_celsius(data['temp_f'])
def get_humidity_percent(self, city_name: str) -> int:
city_code = self.CITY_CODES.get(city_name)
if not city_code:
return 0
data = self.legacy_service.get_weather_data(city_code)
return data['humidity']
def get_conditions(self, city_name: str) -> str:
city_code = self.CITY_CODES.get(city_name)
if not city_code:
return "Unknown"
data = self.legacy_service.get_weather_data(city_code)
return data['conditions']
# Client code expecting NewWeatherSystem
def display_weather(weather_system: NewWeatherSystem, city: str):
temp = weather_system.get_temperature_celsius(city)
humidity = weather_system.get_humidity_percent(city)
conditions = weather_system.get_conditions(city)
print(f"Weather in {city}:")
print(f" Temperature: {temp:.1f}°C")
print(f" Humidity: {humidity}%")
print(f" Conditions: {conditions}")
# Usage
adapter = WeatherAdapter()
display_weather(adapter, "New York")
display_weather(adapter, "Los Angeles")Separates the construction of a complex object from its representation:
Basic Builder:
from abc import ABC, abstractmethod
from typing import List
# Product
class Computer:
def __init__(self):
self.cpu = None
self.gpu = None
self.ram = 0
self.storage = []
self.motherboard = None
self.power_supply = None
self.case = None
self.cooling = None
self.operating_system = None
def __str__(self):
specs = [
f"Computer Specifications:",
f" CPU: {self.cpu}",
f" GPU: {self.gpu}",
f" RAM: {self.ram}GB",
f" Storage: {', '.join(self.storage)}",
f" Motherboard: {self.motherboard}",
f" Power Supply: {self.power_supply}",
f" Case: {self.case}",
f" Cooling: {self.cooling}",
f" OS: {self.operating_system}"
]
return "\n".join(specs)
# Builder interface
class ComputerBuilder(ABC):
@abstractmethod
def reset(self):
pass
@abstractmethod
def build_cpu(self, cpu):
pass
@abstractmethod
def build_gpu(self, gpu):
pass
@abstractmethod
def build_ram(self, size):
pass
@abstractmethod
def build_storage(self, storage):
pass
@abstractmethod
def build_motherboard(self, motherboard):
pass
@abstractmethod
def build_power_supply(self, power_supply):
pass
@abstractmethod
def build_case(self, case):
pass
@abstractmethod
def build_cooling(self, cooling):
pass
@abstractmethod
def build_os(self, os):
pass
@abstractmethod
def get_result(self) -> Computer:
pass
# Concrete builder
class GamingComputerBuilder(ComputerBuilder):
def __init__(self):
self.reset()
def reset(self):
self.computer = Computer()
def build_cpu(self, cpu="Intel Core i9-13900K"):
self.computer.cpu = cpu
def build_gpu(self, gpu="NVIDIA RTX 4090"):
self.computer.gpu = gpu
def build_ram(self, size=64):
self.computer.ram = size
def build_storage(self, storage=None):
if storage is None:
storage = ["2TB NVMe SSD", "4TB HDD"]
self.computer.storage = storage
def build_motherboard(self, motherboard="ASUS ROG Maximus Z790"):
self.computer.motherboard = motherboard
def build_power_supply(self, power_supply="1000W Platinum"):
self.computer.power_supply = power_supply
def build_case(self, case="Lian Li PC-O11 Dynamic"):
self.computer.case = case
def build_cooling(self, cooling="Custom Water Cooling"):
self.computer.cooling = cooling
def build_os(self, os="Windows 11 Pro"):
self.computer.operating_system = os
def get_result(self) -> Computer:
return self.computer
# Another concrete builder
class OfficeComputerBuilder(ComputerBuilder):
def __init__(self):
self.reset()
def reset(self):
self.computer = Computer()
def build_cpu(self, cpu="Intel Core i5-13400"):
self.computer.cpu = cpu
def build_gpu(self, gpu="Integrated Graphics"):
self.computer.gpu = gpu
def build_ram(self, size=16):
self.computer.ram = size
def build_storage(self, storage=None):
if storage is None:
storage = ["512GB SSD"]
self.computer.storage = storage
def build_motherboard(self, motherboard="MSI B760"):
self.computer.motherboard = motherboard
def build_power_supply(self, power_supply="500W Bronze"):
self.computer.power_supply = power_supply
def build_case(self, case="Fractal Design Define 7"):
self.computer.case = case
def build_cooling(self, cooling="Air Cooling"):
self.computer.cooling = cooling
def build_os(self, os="Windows 11 Home"):
self.computer.operating_system = os
def get_result(self) -> Computer:
return self.computer
# Director (optional, for pre-defined configurations)
class ComputerDirector:
def __init__(self, builder: ComputerBuilder):
self.builder = builder
def build_gaming_pc(self):
self.builder.reset()
self.builder.build_cpu()
self.builder.build_gpu()
self.builder.build_ram()
self.builder.build_storage()
self.builder.build_motherboard()
self.builder.build_power_supply()
self.builder.build_case()
self.builder.build_cooling()
self.builder.build_os()
return self.builder.get_result()
def build_office_pc(self):
self.builder.reset()
self.builder.build_cpu("Intel Core i5-13400")
self.builder.build_gpu("Integrated Graphics")
self.builder.build_ram(16)
self.builder.build_storage(["512GB SSD"])
self.builder.build_motherboard("MSI B760")
self.builder.build_power_supply("500W Bronze")
self.builder.build_case("Fractal Design Define 7")
self.builder.build_cooling("Air Cooling")
self.builder.build_os("Windows 11 Home")
return self.builder.get_result()
def build_custom_pc(self, specs):
self.builder.reset()
if 'cpu' in specs:
self.builder.build_cpu(specs['cpu'])
if 'gpu' in specs:
self.builder.build_gpu(specs['gpu'])
if 'ram' in specs:
self.builder.build_ram(specs['ram'])
if 'storage' in specs:
self.builder.build_storage(specs['storage'])
if 'motherboard' in specs:
self.builder.build_motherboard(specs['motherboard'])
if 'power_supply' in specs:
self.builder.build_power_supply(specs['power_supply'])
if 'case' in specs:
self.builder.build_case(specs['case'])
if 'cooling' in specs:
self.builder.build_cooling(specs['cooling'])
if 'os' in specs:
self.builder.build_os(specs['os'])
return self.builder.get_result()
# Usage
print("=== Gaming PC ===")
gaming_builder = GamingComputerBuilder()
director = ComputerDirector(gaming_builder)
gaming_pc = director.build_gaming_pc()
print(gaming_pc)
print("\n=== Office PC ===")
office_builder = OfficeComputerBuilder()
director = ComputerDirector(office_builder)
office_pc = director.build_office_pc()
print(office_pc)
print("\n=== Custom PC ===")
custom_builder = GamingComputerBuilder()
director = ComputerDirector(custom_builder)
custom_pc = director.build_custom_pc({
'cpu': "AMD Ryzen 9 7950X",
'gpu': "AMD Radeon RX 7900 XTX",
'ram': 128,
'storage': ["4TB NVMe SSD"],
'cooling': "Noctua Air Cooling",
'os': "Ubuntu 22.04"
})
print(custom_pc)Fluent Builder Pattern:
class Pizza:
def __init__(self):
self.size = None
self.crust = None
self.sauce = None
self.cheese = None
self.toppings = []
self.bake_time = 0
self.temperature = 0
def __str__(self):
toppings_str = ", ".join(self.toppings) if self.toppings else "no toppings"
return (f"Pizza: {self.size} inch, {self.crust} crust, "
f"{self.sauce} sauce, {self.cheese} cheese, "
f"toppings: {toppings_str}")
class PizzaBuilder:
def __init__(self):
self.pizza = Pizza()
def set_size(self, size):
self.pizza.size = size
return self
def set_crust(self, crust):
self.pizza.crust = crust
return self
def set_sauce(self, sauce):
self.pizza.sauce = sauce
return self
def set_cheese(self, cheese):
self.pizza.cheese = cheese
return self
def add_topping(self, topping):
self.pizza.toppings.append(topping)
return self
def add_toppings(self, *toppings):
self.pizza.toppings.extend(toppings)
return self
def set_baking(self, time, temp):
self.pizza.bake_time = time
self.pizza.temperature = temp
return self
def build(self):
# Validate required fields
if not self.pizza.size:
raise ValueError("Pizza must have a size")
if not self.pizza.crust:
raise ValueError("Pizza must have a crust type")
return self.pizza
@classmethod
def margherita(cls):
"""Pre-defined pizza recipe."""
return (cls()
.set_size(12)
.set_crust("thin")
.set_sauce("tomato")
.set_cheese("mozzarella")
.add_topping("basil")
.set_baking(15, 450)
.build())
@classmethod
def pepperoni(cls):
return (cls()
.set_size(14)
.set_crust("hand-tossed")
.set_sauce("tomato")
.set_cheese("mozzarella")
.add_topping("pepperoni")
.add_toppings("oregano", "garlic")
.set_baking(18, 425)
.build())
# Usage
print("=== Custom Pizza ===")
pizza = (PizzaBuilder()
.set_size(16)
.set_crust("stuffed")
.set_sauce("bbq")
.set_cheese("cheddar")
.add_topping("chicken")
.add_toppings("onions", "peppers", "mushrooms")
.set_baking(20, 475)
.build())
print(pizza)
print("\n=== Margherita ===")
print(PizzaBuilder.margherita())
print("\n=== Pepperoni ===")
print(PizzaBuilder.pepperoni())Model-View-Controller separates data, presentation, and control logic:
Simple MVC Example:
# Model: Manages data and business logic
class TaskModel:
def __init__(self):
self.tasks = []
self.observers = []
def add_observer(self, observer):
self.observers.append(observer)
def notify_observers(self):
for observer in self.observers:
observer.update(self)
def add_task(self, task):
self.tasks.append({"description": task, "completed": False})
self.notify_observers()
return True
def complete_task(self, index):
if 0 <= index < len(self.tasks):
self.tasks[index]["completed"] = True
self.notify_observers()
return True
return False
def delete_task(self, index):
if 0 <= index < len(self.tasks):
self.tasks.pop(index)
self.notify_observers()
return True
return False
def get_tasks(self):
return self.tasks.copy()
def get_stats(self):
total = len(self.tasks)
completed = sum(1 for t in self.tasks if t["completed"])
pending = total - completed
return {
'total': total,
'completed': completed,
'pending': pending,
'completion_rate': (completed / total * 100) if total > 0 else 0
}
# View: Handles presentation
class TaskView:
@staticmethod
def display_tasks(tasks):
if not tasks:
print("\nNo tasks available.")
return
print("\n" + "="*50)
print("TASKS")
print("="*50)
for i, task in enumerate(tasks):
status = "✓" if task["completed"] else "○"
print(f"{i+1}. [{status}] {task['description']}")
print("="*50)
@staticmethod
def display_stats(stats):
print(f"\nStats: {stats['completed']}/{stats['total']} completed "
f"({stats['completion_rate']:.1f}%)")
@staticmethod
def show_message(message):
print(f"\n>>> {message}")
@staticmethod
def show_menu():
print("\nCommands:")
print(" add <task> - Add new task")
print(" done <number> - Mark task as completed")
print(" delete <number> - Delete task")
print(" list - Show all tasks")
print(" stats - Show statistics")
print(" help - Show this menu")
print(" quit - Exit")
print()
# Controller: Handles user input and coordinates model and view
class TaskController:
def __init__(self, model, view):
self.model = model
self.view = view
self.model.add_observer(self)
def update(self, model):
# View automatically updates when model changes
self.show_tasks()
def show_tasks(self):
self.view.display_tasks(self.model.get_tasks())
def show_stats(self):
self.view.display_stats(self.model.get_stats())
def handle_command(self, command):
if not command.strip():
return True
parts = command.strip().split(maxsplit=1)
cmd = parts[0].lower()
if cmd == "quit":
self.view.show_message("Goodbye!")
return False
elif cmd == "list":
self.show_tasks()
elif cmd == "stats":
self.show_stats()
elif cmd == "help":
self.view.show_menu()
elif cmd == "add" and len(parts) > 1:
self.model.add_task(parts[1])
self.view.show_message(f"Task added: '{parts[1]}'")
elif cmd == "done" and len(parts) > 1:
try:
index = int(parts[1]) - 1
if self.model.complete_task(index):
self.view.show_message(f"Task {parts[1]} marked as completed")
else:
self.view.show_message(f"Invalid task number: {parts[1]}")
except ValueError:
self.view.show_message("Please provide a valid task number")
elif cmd == "delete" and len(parts) > 1:
try:
index = int(parts[1]) - 1
if self.model.delete_task(index):
self.view.show_message(f"Task {parts[1]} deleted")
else:
self.view.show_message(f"Invalid task number: {parts[1]}")
except ValueError:
self.view.show_message("Please provide a valid task number")
else:
self.view.show_message(f"Unknown command: {command}")
return True
def run(self):
self.view.show_message("Welcome to Task Manager!")
self.view.show_menu()
self.show_tasks()
running = True
while running:
try:
command = input("\n> ").strip()
running = self.handle_command(command)
except KeyboardInterrupt:
print("\n")
running = self.handle_command("quit")
except Exception as e:
self.view.show_message(f"Error: {e}")
# Usage
if __name__ == "__main__":
model = TaskModel()
view = TaskView()
controller = TaskController(model, view)
# Add some initial tasks
model.add_task("Learn Python MVC")
model.add_task("Build a web app")
model.add_task("Study design patterns")
# Start the application
controller.run()Web MVC Example (Conceptual):
# Model (Database interaction)
class UserModel:
def __init__(self, db_connection):
self.db = db_connection
def get_user(self, user_id):
# Simulate database query
return {
'id': user_id,
'username': 'john_doe',
'email': 'john@example.com',
'created_at': '2026-01-15'
}
def get_all_users(self):
return [
{'id': 1, 'username': 'alice', 'email': 'alice@example.com'},
{'id': 2, 'username': 'bob', 'email': 'bob@example.com'},
{'id': 3, 'username': 'charlie', 'email': 'charlie@example.com'}
]
def create_user(self, user_data):
# Insert into database
return {'id': 4, **user_data, 'created_at': '2026-02-28'}
# View (Template rendering)
class UserView:
@staticmethod
def render_user(user):
return f"""
<div class="user-profile">
<h2>{user['username']}</h2>
<p>Email: {user['email']}</p>
<p>Member since: {user['created_at']}</p>
</div>
"""
@staticmethod
def render_user_list(users):
items = ''.join([f"<li>{u['username']} - {u['email']}</li>" for u in users])
return f"""
<div class="user-list">
<h1>All Users</h1>
<ul>
{items}
</ul>
</div>
"""
@staticmethod
def render_form():
return """
<form action="/users" method="POST">
<input type="text" name="username" placeholder="Username">
<input type="email" name="email" placeholder="Email">
<button type="submit">Create User</button>
</form>
"""
# Controller (Request handling)
class UserController:
def __init__(self, model, view):
self.model = model
self.view = view
def show_user(self, request, user_id):
"""GET /users/{id}"""
user = self.model.get_user(user_id)
if user:
return {
'status': 200,
'content_type': 'text/html',
'body': self.view.render_user(user)
}
return {
'status': 404,
'body': 'User not found'
}
def list_users(self, request):
"""GET /users"""
users = self.model.get_all_users()
return {
'status': 200,
'content_type': 'text/html',
'body': self.view.render_user_list(users)
}
def create_user(self, request):
"""POST /users"""
user_data = request.get('form_data', {})
new_user = self.model.create_user(user_data)
return {
'status': 201,
'headers': {'Location': f'/users/{new_user["id"]}'},
'body': self.view.render_user(new_user)
}
def show_create_form(self, request):
"""GET /users/new"""
return {
'status': 200,
'content_type': 'text/html',
'body': self.view.render_form()
}
# Router (URL routing)
class Router:
def __init__(self):
self.routes = {}
def add_route(self, path, method, handler):
self.routes[(path, method)] = handler
def dispatch(self, request):
path = request.get('path')
method = request.get('method', 'GET')
handler = self.routes.get((path, method))
if handler:
return handler(request)
return {
'status': 404,
'body': 'Not found'
}
# Application setup
model = UserModel(db_connection=None) # Would pass real DB connection
view = UserView()
controller = UserController(model, view)
router = Router()
router.add_route('/users', 'GET', controller.list_users)
router.add_route('/users/new', 'GET', controller.show_create_form)
router.add_route('/users', 'POST', controller.create_user)
router.add_route('/users/1', 'GET', lambda req: controller.show_user(req, 1))
# Simulate requests
requests = [
{'path': '/users', 'method': 'GET'},
{'path': '/users/1', 'method': 'GET'},
{'path': '/users/new', 'method': 'GET'},
{'path': '/users', 'method': 'POST', 'form_data': {'username': 'dave', 'email': 'dave@example.com'}}
]
for req in requests:
response = router.dispatch(req)
print(f"\nRequest: {req['method']} {req['path']}")
print(f"Status: {response['status']}")
if response.get('content_type') == 'text/html':
print(f"Response body preview: {response['body'][:100]}...")Functional programming is a paradigm that treats computation as the evaluation of mathematical functions and avoids changing state and mutable data. Python, while primarily an object-oriented language, incorporates many functional programming features that enable cleaner, more predictable code.
In Python, functions are first-class citizens, meaning they can be treated like any other object:
Functions as Objects:
def greet(name):
return f"Hello, {name}!"
# Functions can be assigned to variables
say_hello = greet
print(say_hello("Alice")) # Hello, Alice!
# Functions can be stored in data structures
functions = [greet, str.upper, len]
for func in functions:
print(func("hello"))
# Functions can be passed as arguments
def apply_twice(func, arg):
return func(func(arg))
def add_exclamation(text):
return text + "!"
result = apply_twice(add_exclamation, "Hello")
print(result) # Hello!!
# Functions can be returned from other functions
def make_multiplier(factor):
def multiplier(x):
return x * factor
return multiplier
double = make_multiplier(2)
triple = make_multiplier(3)
print(double(5)) # 10
print(triple(5)) # 15Function Introspection:
def example(a, b=2, *args, **kwargs):
"""This is an example function."""
pass
# Functions have attributes
print(example.__name__) # 'example'
print(example.__doc__) # 'This is an example function.'
print(example.__defaults__) # (2,)
print(example.__code__.co_varnames) # ('a', 'b', 'args', 'kwargs')
# Using inspect module for detailed information
import inspect
print(inspect.signature(example)) # (a, b=2, *args, **kwargs)Higher-order functions either take functions as arguments or return functions:
Custom Higher-Order Functions:
def create_logger(level):
"""Create a logging function for a specific level."""
def logger(message):
print(f"[{level}] {message}")
return logger
info_logger = create_logger("INFO")
error_logger = create_logger("ERROR")
info_logger("Application started") # [INFO] Application started
error_logger("Database connection failed") # [ERROR] Database connection failed
def timed(func):
"""Decorator-like higher-order function."""
import time
def wrapper(*args, **kwargs):
start = time.time()
result = func(*args, **kwargs)
end = time.time()
print(f"{func.__name__} took {end - start:.4f} seconds")
return result
return wrapper
def slow_function():
time.sleep(1)
return "Done"
timed_slow = timed(slow_function)
print(timed_slow())
def conditional_executor(condition_func, true_func, false_func):
"""Execute different functions based on condition."""
def wrapper(*args, **kwargs):
if condition_func(*args, **kwargs):
return true_func(*args, **kwargs)
return false_func(*args, **kwargs)
return wrapper
is_even = lambda x: x % 2 == 0
process_even = lambda x: f"{x} is even, doubled: {x * 2}"
process_odd = lambda x: f"{x} is odd, squared: {x ** 2}"
process_number = conditional_executor(is_even, process_even, process_odd)
print(process_number(4)) # 4 is even, doubled: 8
print(process_number(5)) # 5 is odd, squared: 25These built-in functions are fundamental to functional programming:
map() - Apply function to every item:
numbers = [1, 2, 3, 4, 5]
# Square each number
squares = list(map(lambda x: x ** 2, numbers))
print(squares) # [1, 4, 9, 16, 25]
# Using multiple iterables
numbers1 = [1, 2, 3]
numbers2 = [10, 20, 30]
sums = list(map(lambda x, y: x + y, numbers1, numbers2))
print(sums) # [11, 22, 33]
# With built-in functions
words = ["hello", "world", "python"]
uppercase = list(map(str.upper, words))
print(uppercase) # ['HELLO', 'WORLD', 'PYTHON']
# map() is lazy - returns iterator
squares_lazy = map(lambda x: x ** 2, numbers)
print(squares_lazy) # <map object at 0x...>
print(list(squares_lazy)) # Force evaluation
# Practical example: convert temperatures
celsius = [0, 20, 30, 40]
fahrenheit = list(map(lambda c: (c * 9/5) + 32, celsius))
print(fahrenheit) # [32.0, 68.0, 86.0, 104.0]filter() - Select items based on condition:
numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
# Filter even numbers
evens = list(filter(lambda x: x % 2 == 0, numbers))
print(evens) # [2, 4, 6, 8, 10]
# Filter with None (removes falsy values)
mixed = [0, 1, False, True, "", "hello", [], [1, 2], None]
truthy = list(filter(None, mixed))
print(truthy) # [1, True, 'hello', [1, 2]]
# Filter strings by length
words = ["cat", "elephant", "dog", "butterfly", "ant"]
long_words = list(filter(lambda w: len(w) > 5, words))
print(long_words) # ['elephant', 'butterfly']
# Practical: filter valid email addresses
import re
def is_valid_email(email):
pattern = r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'
return re.match(pattern, email) is not None
emails = ["user@example.com", "invalid-email", "another@test.org", "bad@.com"]
valid_emails = list(filter(is_valid_email, emails))
print(valid_emails) # ['user@example.com', 'another@test.org']reduce() - Cumulative processing:
from functools import reduce
numbers = [1, 2, 3, 4, 5]
# Sum all numbers
total = reduce(lambda acc, x: acc + x, numbers)
print(total) # 15
# With initial value
total = reduce(lambda acc, x: acc + x, numbers, 10)
print(total) # 25
# Find maximum
maximum = reduce(lambda a, b: a if a > b else b, numbers)
print(maximum) # 5
# Compute product
product = reduce(lambda acc, x: acc * x, numbers)
print(product) # 120
# Build dictionary from list of pairs
pairs = [("a", 1), ("b", 2), ("c", 3)]
dict_from_pairs = reduce(lambda d, pair: {**d, pair[0]: pair[1]}, pairs, {})
print(dict_from_pairs) # {'a': 1, 'b': 2, 'c': 3}
# Flatten list of lists
lists = [[1, 2], [3, 4, 5], [6]]
flattened = reduce(lambda acc, lst: acc + lst, lists, [])
print(flattened) # [1, 2, 3, 4, 5, 6]
# Practical: compute factorial
def factorial(n):
return reduce(lambda acc, x: acc * x, range(1, n + 1), 1)
print(factorial(5)) # 120Combining map, filter, reduce:
from functools import reduce
# Process pipeline: filter even numbers, square them, then sum
numbers = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
result = reduce(
lambda acc, x: acc + x,
map(
lambda x: x ** 2,
filter(lambda x: x % 2 == 0, numbers)
)
)
print(result) # 4 + 16 + 36 + 64 + 100 = 220
# More readable with list comprehension
result = sum(x ** 2 for x in numbers if x % 2 == 0)
print(result) # 220
# Complex data processing
data = [
{"name": "Alice", "score": 85, "age": 30},
{"name": "Bob", "score": 92, "age": 25},
{"name": "Charlie", "score": 78, "age": 35},
{"name": "Diana", "score": 95, "age": 28}
]
# Calculate average score of people over 30
over_30 = filter(lambda p: p["age"] > 30, data)
scores = map(lambda p: p["score"], over_30)
avg_score = reduce(lambda acc, x: acc + x, scores) / len(list(filter(lambda p: p["age"] > 30, data)))
print(f"Average score (over 30): {avg_score:.1f}")Closures are functions that remember the environment in which they were created:
Basic Closures:
def outer_function(message):
"""Outer function creates a closure."""
def inner_function():
print(f"Message: {message}")
return inner_function
# Create closures
hello_closure = outer_function("Hello, World!")
goodbye_closure = outer_function("Goodbye!")
# Each remembers its own message
hello_closure() # Message: Hello, World!
goodbye_closure() # Message: Goodbye!
# Closures capture variables by reference
def counter():
count = 0
def increment():
nonlocal count
count += 1
return count
return increment
counter1 = counter()
counter2 = counter()
print(counter1()) # 1
print(counter1()) # 2
print(counter2()) # 1
print(counter1()) # 3
print(counter2()) # 2Practical Closures:
# Function factory with configuration
def create_power_function(exponent):
"""Create a function that raises numbers to a specific power."""
def power(base):
return base ** exponent
return power
square = create_power_function(2)
cube = create_power_function(3)
fourth = create_power_function(4)
print(square(5)) # 25
print(cube(5)) # 125
print(fourth(5)) # 625
# Memoization with closures
def memoize(func):
"""Create a memoized version of a function."""
cache = {}
def memoized(*args):
if args not in cache:
cache[args] = func(*args)
print(f"Cache miss: {args} -> {cache[args]}")
else:
print(f"Cache hit: {args} -> {cache[args]}")
return cache[args]
return memoized
@memoize
def fibonacci(n):
if n <= 1:
return n
return fibonacci(n - 1) + fibonacci(n - 2)
print(fibonacci(10))
# Data hiding with closures
def create_secure_counter():
"""Create a counter with protected state."""
_count = 0
def get_count():
return _count
def increment():
nonlocal _count
_count += 1
return _count
def decrement():
nonlocal _count
_count -= 1
return _count
# Return dictionary of functions
return {
'get': get_count,
'inc': increment,
'dec': decrement
}
counter = create_secure_counter()
print(counter['inc']()) # 1
print(counter['inc']()) # 2
print(counter['dec']()) # 1
print(counter['get']()) # 1
# Cannot directly access _countDecorators are a powerful application of closures that modify function behavior:
Basic Decorators:
def simple_decorator(func):
"""Basic decorator that adds behavior before and after."""
def wrapper(*args, **kwargs):
print(f"Calling {func.__name__} with args={args}, kwargs={kwargs}")
result = func(*args, **kwargs)
print(f"{func.__name__} returned {result}")
return result
return wrapper
@simple_decorator
def add(a, b):
return a + b
@simple_decorator
def multiply(a, b):
return a * b
add(3, 5)
multiply(4, 7)
# Decorators with arguments
def repeat(times):
"""Decorator that repeats a function multiple times."""
def decorator(func):
def wrapper(*args, **kwargs):
results = []
for _ in range(times):
result = func(*args, **kwargs)
results.append(result)
return results
return wrapper
return decorator
@repeat(3)
def say_hello(name):
return f"Hello, {name}!"
print(say_hello("Alice")) # ['Hello, Alice!', 'Hello, Alice!', 'Hello, Alice!']Useful Decorator Examples:
import time
import functools
import logging
# Timing decorator
def timer(func):
"""Measure and print execution time."""
@functools.wraps(func)
def wrapper(*args, **kwargs):
start = time.perf_counter()
result = func(*args, **kwargs)
end = time.perf_counter()
print(f"{func.__name__} took {end - start:.6f} seconds")
return result
return wrapper
# Retry decorator
def retry(max_attempts=3, delay=1):
"""Retry function on failure."""
def decorator(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
for attempt in range(max_attempts):
try:
return func(*args, **kwargs)
except Exception as e:
if attempt == max_attempts - 1:
raise
print(f"Attempt {attempt + 1} failed: {e}. Retrying...")
time.sleep(delay)
return None
return wrapper
return decorator
# Rate limiting decorator
def rate_limit(calls_per_second):
"""Limit function call rate."""
import time
def decorator(func):
last_called = [0.0] # Use list for nonlocal mutability
@functools.wraps(func)
def wrapper(*args, **kwargs):
elapsed = time.time() - last_called[0]
left_to_wait = 1.0 / calls_per_second - elapsed
if left_to_wait > 0:
time.sleep(left_to_wait)
result = func(*args, **kwargs)
last_called[0] = time.time()
return result
return wrapper
return decorator
# Validation decorator
def validate_args(*expected_types):
"""Validate argument types."""
def decorator(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
for i, (arg, expected) in enumerate(zip(args, expected_types)):
if not isinstance(arg, expected):
raise TypeError(f"Argument {i} must be {expected.__name__}, got {type(arg).__name__}")
return func(*args, **kwargs)
return wrapper
return decorator
# Cache decorator with TTL
def cache_ttl(seconds=60):
"""Cache function results with time-to-live."""
def decorator(func):
cache = {}
@functools.wraps(func)
def wrapper(*args, **kwargs):
key = str(args) + str(sorted(kwargs.items()))
now = time.time()
if key in cache:
result, timestamp = cache[key]
if now - timestamp < seconds:
print(f"Cache hit for {func.__name__}{args}")
return result
else:
print(f"Cache expired for {func.__name__}{args}")
result = func(*args, **kwargs)
cache[key] = (result, now)
return result
return wrapper
return decorator
# Usage examples
@timer
def slow_function():
time.sleep(1)
return "Done"
@retry(max_attempts=3, delay=0.5)
def unstable_function():
import random
if random.random() < 0.7:
raise ValueError("Random failure")
return "Success!"
@validate_args(int, int)
def divide(a, b):
return a / b
@cache_ttl(seconds=5)
def expensive_computation(n):
print(f"Computing for {n}...")
time.sleep(2)
return n * n
# Test decorators
print(slow_function())
try:
result = unstable_function()
print(f"Result: {result}")
except Exception as e:
print(f"Failed: {e}")
print(divide(10, 2))
# print(divide("10", 2)) # TypeError
print(expensive_computation(5))
print(expensive_computation(5)) # Cache hit
time.sleep(6)
print(expensive_computation(5)) # Cache expiredClass-based Decorators:
class CountCalls:
"""Decorator class that counts function calls."""
def __init__(self, func):
self.func = func
self.count = 0
def __call__(self, *args, **kwargs):
self.count += 1
print(f"Call {self.count} of {self.func.__name__}")
return self.func(*args, **kwargs)
@CountCalls
def say_hello(name):
return f"Hello, {name}!"
print(say_hello("Alice"))
print(say_hello("Bob"))
print(say_hello("Charlie"))
class CacheDecorator:
"""Class-based caching decorator."""
def __init__(self, func):
self.func = func
self.cache = {}
def __call__(self, *args, **kwargs):
key = str(args) + str(sorted(kwargs.items()))
if key not in self.cache:
self.cache[key] = self.func(*args, **kwargs)
print(f"Cache miss: {key}")
else:
print(f"Cache hit: {key}")
return self.cache[key]
def clear_cache(self):
self.cache.clear()
@CacheDecorator
def fibonacci(n):
if n <= 1:
return n
return fibonacci(n - 1) + fibonacci(n - 2)
print(fibonacci(10))
print(fibonacci(10)) # Cache hit
fibonacci.clear_cache()
print(fibonacci(10)) # Cache miss againGenerators produce values lazily, one at a time, using the yield keyword:
Basic Generators:
def simple_generator():
"""A simple generator that yields three values."""
yield 1
yield 2
yield 3
gen = simple_generator()
print(next(gen)) # 1
print(next(gen)) # 2
print(next(gen)) # 3
# print(next(gen)) # StopIteration
# Generator expression
squares = (x ** 2 for x in range(5))
print(list(squares)) # [0, 1, 4, 9, 16]
# Generator with loop
def countdown(n):
print(f"Starting countdown from {n}")
while n > 0:
yield n
n -= 1
print("Done!")
for num in countdown(5):
print(num)Generator Use Cases:
# Infinite sequences
def fibonacci():
"""Generate Fibonacci numbers indefinitely."""
a, b = 0, 1
while True:
yield a
a, b = b, a + b
fib = fibonacci()
for _ in range(10):
print(next(fib), end=" ") # 0 1 1 2 3 5 8 13 21 34
print()
# Reading large files line by line
def read_large_file(file_path):
"""Read a large file line by line without loading entire file."""
with open(file_path, 'r') as file:
for line in file:
yield line.strip()
# Process only first 10 lines
# for i, line in enumerate(read_large_file("huge_file.txt")):
# if i >= 10:
# break
# print(line)
# Generate range with step
def frange(start, stop, step):
"""Generate floating point range."""
current = start
while current < stop:
yield current
current += step
for num in frange(0, 1, 0.1):
print(f"{num:.1f}", end=" ") # 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
print()
# Data pipeline with generators
def read_numbers(file_path):
"""Read numbers from file."""
with open(file_path, 'r') as f:
for line in f:
yield float(line.strip())
def filter_positive(numbers):
"""Filter positive numbers."""
for num in numbers:
if num > 0:
yield num
def square(numbers):
"""Square each number."""
for num in numbers:
yield num ** 2
# Pipeline (lazy evaluation)
# pipeline = square(filter_positive(read_numbers("numbers.txt")))
# for result in pipeline:
# print(result)Generator Methods:
def task_manager():
"""Generator with send(), throw(), close() methods."""
print("Task manager started")
tasks = []
while True:
try:
new_task = yield tasks
if new_task:
tasks.append(new_task)
print(f"Added task: {new_task}")
except ValueError as e:
print(f"Error: {e}")
tasks.clear()
except GeneratorExit:
print("Task manager shutting down")
break
# Using the generator
manager = task_manager()
next(manager) # Start the generator
# Send values
print(manager.send("Task 1")) # ['Task 1']
print(manager.send("Task 2")) # ['Task 1', 'Task 2']
print(manager.send("Task 3")) # ['Task 1', 'Task 2', 'Task 3']
# Throw exception
manager.throw(ValueError, "Clearing all tasks")
print(manager.send("Task 4")) # ['Task 4']
# Close the generator
manager.close()yield from - Delegating to subgenerators:
def subgenerator():
"""Subgenerator yields some values."""
yield "From sub: 1"
yield "From sub: 2"
yield "From sub: 3"
def main_generator():
"""Main generator using yield from."""
yield "Main start"
yield from subgenerator() # Delegate to subgenerator
yield "Main middle"
yield from range(3, 6) # Delegate to range
yield "Main end"
for value in main_generator():
print(value)
# Nested generators for tree traversal
def flatten(nested):
"""Flatten arbitrarily nested iterables."""
for item in nested:
if isinstance(item, (list, tuple)):
yield from flatten(item)
else:
yield item
nested = [1, [2, 3, [4, 5]], 6, [7, 8]]
print(list(flatten(nested))) # [1, 2, 3, 4, 5, 6, 7, 8]Iterators are objects that implement the iterator protocol:
Iterator Protocol:
# An object is iterable if it has __iter__() method
# An iterator is an object with __iter__() and __next__() methods
class CountDown:
"""Custom iterator that counts down from n."""
def __init__(self, start):
self.start = start
self.current = start
def __iter__(self):
"""Return the iterator object itself."""
return self
def __next__(self):
"""Return next value or raise StopIteration."""
if self.current <= 0:
raise StopIteration
value = self.current
self.current -= 1
return value
# Usage
countdown = CountDown(5)
for num in countdown:
print(num, end=" ") # 5 4 3 2 1
print()
# Iterators are exhausted after one pass
print(list(countdown)) # [] - already exhausted
# Creating a fresh iterator
countdown = CountDown(3)
print(next(countdown)) # 3
print(next(countdown)) # 2
print(next(countdown)) # 1
# print(next(countdown)) # StopIterationInfinite Iterators:
class FibonacciIterator:
"""Infinite Fibonacci iterator."""
def __init__(self):
self.a = 0
self.b = 1
def __iter__(self):
return self
def __next__(self):
result = self.a
self.a, self.b = self.b, self.a + self.b
return result
fib = FibonacciIterator()
for _ in range(10):
print(next(fib), end=" ") # 0 1 1 2 3 5 8 13 21 34
print()
class CyclicIterator:
"""Iterator that cycles through a sequence indefinitely."""
def __init__(self, iterable):
self.iterable = iterable
self.iterator = iter(iterable)
def __iter__(self):
return self
def __next__(self):
try:
return next(self.iterator)
except StopIteration:
self.iterator = iter(self.iterable)
return next(self.iterator)
colors = CyclicIterator(['red', 'green', 'blue'])
for _ in range(7):
print(next(colors), end=" ") # red green blue red green blue red
print()Building Custom Iterables:
class Range:
"""Custom range implementation."""
def __init__(self, start, stop=None, step=1):
if stop is None:
start, stop = 0, start
self.start = start
self.stop = stop
self.step = step
def __iter__(self):
current = self.start
while (self.step > 0 and current < self.stop) or \
(self.step < 0 and current > self.stop):
yield current
current += self.step
def __len__(self):
if self.step > 0:
return max(0, (self.stop - self.start + self.step - 1) // self.step)
else:
return max(0, (self.start - self.stop - self.step - 1) // -self.step)
def __getitem__(self, index):
if index < 0:
index += len(self)
if index < 0 or index >= len(self):
raise IndexError("Range index out of range")
return self.start + index * self.step
r = Range(1, 10, 2)
print(list(r)) # [1, 3, 5, 7, 9]
print(len(r)) # 5
print(r[2]) # 5
print(r[-1]) # 9Itertools Integration:
import itertools
# Chain multiple iterables
combined = itertools.chain([1, 2, 3], ['a', 'b', 'c'], range(3))
print(list(combined)) # [1, 2, 3, 'a', 'b', 'c', 0, 1, 2]
# Cycle through an iterable
colors = itertools.cycle(['red', 'green', 'blue'])
for _ in range(7):
print(next(colors), end=" ") # red green blue red green blue red
print()
# Repeat a value
repeated = itertools.repeat('hello', 3)
print(list(repeated)) # ['hello', 'hello', 'hello']
# Accumulate
numbers = [1, 2, 3, 4, 5]
accumulated = itertools.accumulate(numbers)
print(list(accumulated)) # [1, 3, 6, 10, 15]
# Product (Cartesian product)
product = itertools.product([1, 2], ['a', 'b'])
print(list(product)) # [(1, 'a'), (1, 'b'), (2, 'a'), (2, 'b')]
# Permutations
perms = itertools.permutations([1, 2, 3], 2)
print(list(perms)) # [(1, 2), (1, 3), (2, 1), (2, 3), (3, 1), (3, 2)]
# Combinations
combs = itertools.combinations([1, 2, 3, 4], 2)
print(list(combs)) # [(1, 2), (1, 3), (1, 4), (2, 3), (2, 4), (3, 4)]
# Takewhile and dropwhile
numbers = [1, 3, 5, 2, 4, 6, 1, 3]
taken = itertools.takewhile(lambda x: x < 5, numbers)
print(list(taken)) # [1, 3]
dropped = itertools.dropwhile(lambda x: x < 5, numbers)
print(list(dropped)) # [2, 4, 6, 1, 3]
# Groupby
data = [('fruit', 'apple'), ('fruit', 'banana'), ('veg', 'carrot'),
('fruit', 'cherry'), ('veg', 'broccoli')]
data.sort() # groupby requires sorted data
for key, group in itertools.groupby(data, key=lambda x: x[0]):
print(f"{key}: {list(group)}")Coroutines extend generators to consume data as well as produce it:
Basic Coroutines:
def simple_coroutine():
"""Simple coroutine that receives values."""
print("Coroutine started")
while True:
value = yield
print(f"Received: {value}")
# Create and prime coroutine
coro = simple_coroutine()
next(coro) # Prime the coroutine (or coro.send(None))
# Send values
coro.send(10) # Received: 10
coro.send(20) # Received: 20
coro.send("hello") # Received: hello
# Close coroutine
coro.close()
def coroutine_with_return():
"""Coroutine that yields and receives values."""
print("Coroutine started")
items = []
while True:
value = yield items # Yield current list, receive next value
if value is None:
break
items.append(value)
return items
# Usage
coro = coroutine_with_return()
next(coro) # Prime
print(coro.send(1)) # [1]
print(coro.send(2)) # [1, 2]
print(coro.send(3)) # [1, 2, 3]
try:
coro.send(None) # Stop iteration
except StopIteration as e:
print(f"Final result: {e.value}") # Final result: [1, 2, 3]Coroutine Decorators:
def coroutine(func):
"""Decorator to prime coroutines."""
@functools.wraps(func)
def primer(*args, **kwargs):
gen = func(*args, **kwargs)
next(gen)
return gen
return primer
@coroutine
def running_average():
"""Coroutine that calculates running average."""
total = 0
count = 0
average = None
print("Average calculator started")
while True:
value = yield average
total += value
count += 1
average = total / count
avg_coro = running_average()
print(avg_coro.send(10)) # 10.0
print(avg_coro.send(20)) # 15.0
print(avg_coro.send(30)) # 20.0
@coroutine
def grep(pattern):
"""Coroutine that searches for pattern in received lines."""
print(f"Looking for {pattern}")
while True:
line = yield
if pattern in line:
print(f"Found: {line}")
g = grep("python")
g.send("I love programming")
g.send("Python is great") # Found: Python is great
g.send("Learning python is fun") # Found: Learning python is fun
g.close()Pipeline with Coroutines:
@coroutine
def producer(target):
"""Producer coroutine."""
for i in range(5):
print(f"Producing {i}")
target.send(i)
target.close()
@coroutine
def filter_even(target):
"""Filter even numbers."""
while True:
value = yield
if value % 2 == 0:
print(f"Filter passing {value}")
target.send(value)
@coroutine
def consumer():
"""Consumer coroutine."""
try:
while True:
value = yield
print(f"Consuming {value}")
except GeneratorExit:
print("Consumer done")
# Build pipeline
pipeline = producer(filter_even(consumer()))
# Producing 0
# Filter passing 0
# Consuming 0
# Producing 1
# Producing 2
# Filter passing 2
# Consuming 2
# Producing 3
# Producing 4
# Filter passing 4
# Consuming 4
# Consumer doneAdvanced Coroutine Example - Event Handler:
class EventHandler:
"""Event handling system using coroutines."""
def __init__(self):
self.handlers = {}
def register(self, event_type, coro):
"""Register a coroutine handler for an event type."""
@coroutine
def wrapper():
while True:
event_data = yield
coro.send(event_data)
self.handlers.setdefault(event_type, []).append(wrapper())
def emit(self, event_type, data):
"""Emit an event to all registered handlers."""
for handler in self.handlers.get(event_type, []):
handler.send(data)
def close(self):
"""Close all handlers."""
for handlers in self.handlers.values():
for handler in handlers:
handler.close()
# Create handlers
def log_handler():
while True:
event = yield
print(f"[LOG] {event['type']}: {event['data']}")
def email_handler():
while True:
event = yield
if event['type'] == 'user_created':
print(f"[EMAIL] Sending welcome email to {event['data']['email']}")
def analytics_handler():
count = 0
while True:
event = yield
count += 1
print(f"[ANALYTICS] Event #{count}: {event['type']}")
# Setup event system
handler = EventHandler()
handler.register('user_created', log_handler())
handler.register('user_created', email_handler())
handler.register('user_created', analytics_handler())
handler.register('user_updated', log_handler())
handler.register('user_updated', analytics_handler())
# Emit events
handler.emit('user_created', {
'type': 'user_created',
'data': {'username': 'alice', 'email': 'alice@example.com'}
})
handler.emit('user_updated', {
'type': 'user_updated',
'data': {'username': 'alice', 'changes': ['email']}
})
handler.close()The itertools module provides efficient tools for working with iterators:
Infinite Iterators:
import itertools
# count(start, step)
counter = itertools.count(10, 2)
print([next(counter) for _ in range(5)]) # [10, 12, 14, 16, 18]
# cycle(iterable)
cycler = itertools.cycle(['A', 'B', 'C'])
print([next(cycler) for _ in range(7)]) # ['A', 'B', 'C', 'A', 'B', 'C', 'A']
# repeat(elem, times)
repeater = itertools.repeat('Hello', 3)
print(list(repeater)) # ['Hello', 'Hello', 'Hello']Finite Iterators:
# accumulate
numbers = [1, 2, 3, 4, 5]
acc = itertools.accumulate(numbers) # Default sum
print(list(acc)) # [1, 3, 6, 10, 15]
# With different function
acc_mul = itertools.accumulate(numbers, lambda x, y: x * y)
print(list(acc_mul)) # [1, 2, 6, 24, 120]
# chain
combined = itertools.chain([1, 2, 3], ['a', 'b'], 'XYZ')
print(list(combined)) # [1, 2, 3, 'a', 'b', 'X', 'Y', 'Z']
# chain.from_iterable
nested = [[1, 2], [3, 4], [5, 6]]
flattened = itertools.chain.from_iterable(nested)
print(list(flattened)) # [1, 2, 3, 4, 5, 6]
# compress
data = ['A', 'B', 'C', 'D', 'E']
selectors = [1, 0, 1, 0, 1]
result = itertools.compress(data, selectors)
print(list(result)) # ['A', 'C', 'E']
# dropwhile
numbers = [1, 3, 5, 2, 4, 6]
result = itertools.dropwhile(lambda x: x < 5, numbers)
print(list(result)) # [5, 2, 4, 6]
# takewhile
result = itertools.takewhile(lambda x: x < 5, numbers)
print(list(result)) # [1, 3]
# filterfalse (opposite of filter)
result = itertools.filterfalse(lambda x: x % 2 == 0, numbers)
print(list(result)) # [1, 3, 5]
# groupby
data = [('fruit', 'apple'), ('fruit', 'banana'), ('veg', 'carrot'),
('fruit', 'cherry'), ('veg', 'broccoli')]
data.sort() # groupby requires sorted data
for key, group in itertools.groupby(data, key=lambda x: x[0]):
print(f"{key}: {list(group)}")
# islice
numbers = range(20)
result = itertools.islice(numbers, 5, 15, 3)
print(list(result)) # [5, 8, 11, 14]
# pairwise (Python 3.10+)
numbers = [1, 2, 3, 4, 5]
pairs = itertools.pairwise(numbers)
print(list(pairs)) # [(1, 2), (2, 3), (3, 4), (4, 5)]
# starmap
points = [(1, 2), (3, 4), (5, 6)]
result = itertools.starmap(lambda x, y: x + y, points)
print(list(result)) # [3, 7, 11]
# tee (create multiple independent iterators)
numbers = [1, 2, 3, 4, 5]
iter1, iter2, iter3 = itertools.tee(numbers, 3)
print(list(iter1)) # [1, 2, 3, 4, 5]
print(list(iter2)) # [1, 2, 3, 4, 5]
print(list(iter3)) # [1, 2, 3, 4, 5]
# zip_longest
a = [1, 2, 3]
b = ['a', 'b']
result = itertools.zip_longest(a, b, fillvalue='-')
print(list(result)) # [(1, 'a'), (2, 'b'), (3, '-')]Combinatoric Iterators:
# product - Cartesian product
product = itertools.product([1, 2], ['a', 'b'], ['x', 'y'])
print(list(product)) # 8 combinations
# permutations
perms = itertools.permutations([1, 2, 3], 2)
print(list(perms)) # [(1, 2), (1, 3), (2, 1), (2, 3), (3, 1), (3, 2)]
# combinations
combs = itertools.combinations([1, 2, 3, 4], 2)
print(list(combs)) # [(1, 2), (1, 3), (1, 4), (2, 3), (2, 4), (3, 4)]
# combinations_with_replacement
combs_wr = itertools.combinations_with_replacement([1, 2, 3], 2)
print(list(combs_wr)) # [(1, 1), (1, 2), (1, 3), (2, 2), (2, 3), (3, 3)]Practical itertools Examples:
# Generate all possible passwords from character set
def generate_passwords(chars, length):
"""Generate all possible passwords of given length."""
return itertools.product(chars, repeat=length)
passwords = generate_passwords('abc', 3)
print(list(passwords)) # 27 combinations
# Find all subsets of a set
def all_subsets(items):
"""Generate all subsets of items."""
for i in range(len(items) + 1):
yield from itertools.combinations(items, i)
items = [1, 2, 3]
subsets = list(all_subsets(items))
print(subsets) # [(), (1,), (2,), (3,), (1,2), (1,3), (2,3), (1,2,3)]
# Sliding window over sequence
def sliding_window(seq, n=2):
"""Generate sliding windows over sequence."""
it = iter(seq)
window = tuple(itertools.islice(it, n))
if len(window) == n:
yield window
for elem in it:
window = window[1:] + (elem,)
yield window
numbers = [1, 2, 3, 4, 5]
for window in sliding_window(numbers, 3):
print(window) # (1,2,3), (2,3,4), (3,4,5)
# Batched processing (Python 3.12+ has itertools.batched)
def batched(iterable, n):
"""Batch data into tuples of length n."""
it = iter(iterable)
while True:
batch = tuple(itertools.islice(it, n))
if not batch:
break
yield batch
for batch in batched(range(10), 3):
print(batch) # (0,1,2), (3,4,5), (6,7,8), (9,)
# Partition iterable based on condition
def partition(pred, iterable):
"""Split iterable into two based on predicate."""
t1, t2 = itertools.tee(iterable)
return list(filter(pred, t1)), list(itertools.filterfalse(pred, t2))
even, odd = partition(lambda x: x % 2 == 0, range(10))
print(even) # [0, 2, 4, 6, 8]
print(odd) # [1, 3, 5, 7, 9]
# Flatten nested structure with depth control
def flatten_depth(nested, depth=None):
"""Flatten nested structure up to specified depth."""
if depth == 0:
yield nested
elif isinstance(nested, (list, tuple)):
for item in nested:
if depth is None:
yield from flatten_depth(item)
else:
yield from flatten_depth(item, depth - 1)
else:
yield nested
nested = [1, [2, [3, [4, 5]]], 6]
print(list(flatten_depth(nested))) # [1, 2, 3, 4, 5, 6]
print(list(flatten_depth(nested, 1))) # [1, 2, [3, [4, 5]], 6]
print(list(flatten_depth(nested, 2))) # [1, 2, 3, [4, 5], 6]Modules are Python files containing definitions and statements:
Creating a Module (mymath.py):
"""My math module - provides basic mathematical operations."""
# Module-level documentation
__version__ = "1.0.0"
__author__ = "Python Master"
# Module variables
PI = 3.14159
E = 2.71828
# Functions
def add(a, b):
"""Add two numbers."""
return a + b
def subtract(a, b):
"""Subtract b from a."""
return a - b
def multiply(a, b):
"""Multiply two numbers."""
return a * b
def divide(a, b):
"""Divide a by b. Raises ValueError if b is zero."""
if b == 0:
raise ValueError("Cannot divide by zero")
return a / b
def power(a, b):
"""Raise a to the power b."""
return a ** b
def sqrt(a):
"""Calculate square root."""
return a ** 0.5
# Classes
class Calculator:
"""Simple calculator class."""
def __init__(self, name="Default"):
self.name = name
self.history = []
def calculate(self, operation, a, b=None):
"""Perform calculation and record history."""
if operation == "add":
result = add(a, b)
elif operation == "subtract":
result = subtract(a, b)
elif operation == "multiply":
result = multiply(a, b)
elif operation == "divide":
result = divide(a, b)
elif operation == "power":
result = power(a, b)
elif operation == "sqrt":
result = sqrt(a)
else:
raise ValueError(f"Unknown operation: {operation}")
self.history.append((operation, a, b, result))
return result
def clear_history(self):
"""Clear calculation history."""
self.history.clear()
def get_history(self):
"""Return calculation history."""
return self.history.copy()
# Module initialization
print(f"Initializing mymath module v{__version__}")
# Private by convention (underscore prefix)
def _internal_helper():
"""Internal helper function (not intended for external use)."""
return "This is internal"
# Code to run when module is executed directly
if __name__ == "__main__":
print("Running mymath module as script")
calc = Calculator("Test")
print(calc.calculate("add", 5, 3))
print(calc.calculate("multiply", 4, 7))
print(calc.get_history())Different Import Styles:
# Import entire module
import mymath
print(mymath.PI)
print(mymath.add(5, 3))
calc = mymath.Calculator()
# Import with alias
import mymath as mm
print(mm.PI)
print(mm.divide(10, 2))
# Import specific items
from mymath import PI, add, Calculator
print(PI)
print(add(5, 3))
calc = Calculator()
# Import with alias for specific item
from mymath import Calculator as Calc
calc = Calc()
# Import all (generally discouraged)
from mymath import *
print(PI)
print(add(5, 3))
# But this pollutes namespace
# Import submodules
import package.submodule
from package import submodule
from package.submodule import functionImport Search Path:
import sys
# View module search path
print(sys.path)
# Add directory to search path
sys.path.append('/path/to/my/modules')
sys.path.insert(0, '/path/to/preferred/modules')
# Environment variable PYTHONPATH also affects search path
# export PYTHONPATH=/custom/module/path:$PYTHONPATH
# Find module location
import mymath
print(mymath.__file__)
print(mymath.__name__)
print(mymath.__package__)Reloading Modules:
import importlib
import mymath
# After modifying mymath.py
importlib.reload(mymath)The __name__ variable helps distinguish between script execution and module import:
Module Guard Pattern:
# mymodule.py
def main():
"""Main function when run as script."""
print("Running as script")
# Script logic here
if __name__ == "__main__":
main()Practical Example:
# utility.py
import sys
import argparse
def process_file(filename):
"""Process a file."""
print(f"Processing {filename}")
# Actual processing logic
def main():
"""Command-line interface."""
parser = argparse.ArgumentParser(description="File processor")
parser.add_argument('filename', help="File to process")
parser.add_argument('--verbose', '-v', action='store_true', help="Verbose output")
args = parser.parse_args()
if args.verbose:
print(f"Verbose mode enabled")
process_file(args.filename)
if __name__ == "__main__":
main()The sys module provides access to system-specific parameters and functions:
Command Line Arguments:
import sys
# Get command line arguments
print(f"Script name: {sys.argv[0]}")
print(f"Arguments: {sys.argv[1:]}")
# Example: simple argument parser
if len(sys.argv) < 2:
print("Usage: python script.py <name>")
sys.exit(1)
name = sys.argv[1]
print(f"Hello, {name}!")System Information:
import sys
# Python version
print(f"Python version: {sys.version}")
print(f"Version info: {sys.version_info}")
print(f"Major.Minor: {sys.version_info.major}.{sys.version_info.minor}")
# Platform
print(f"Platform: {sys.platform}")
# System paths
print(f"Executable: {sys.executable}")
print(f"Module search paths:")
for path in sys.path:
print(f" {path}")
# Module information
print(f"Loaded modules: {list(sys.modules.keys())[:5]}...")Standard Streams:
import sys
# Redirect output
original_stdout = sys.stdout
with open('output.txt', 'w') as f:
sys.stdout = f
print("This goes to file")
sys.stdout = original_stdout
# Error output
sys.stderr.write("This is an error message\n")
# Input
print("Enter something: ")
data = sys.stdin.readline().strip()
print(f"You entered: {data}")System Exit:
import sys
def validate_input(value):
if not value:
print("Error: Empty input")
sys.exit(1)
return value
# try:
# value = validate_input("")
# except SystemExit as e:
# print(f"Exited with code: {e.code}")Memory and Garbage Collection:
import sys
# Get size of objects
data = [1, 2, 3, 4, 5]
print(f"Size of list: {sys.getsizeof(data)} bytes")
# Reference counting
x = []
y = x
print(f"Reference count for x: {sys.getrefcount(x) - 1}") # -1 for getrefcount's own reference
# Recursion limit
print(f"Recursion limit: {sys.getrecursionlimit()}")
sys.setrecursionlimit(2000)
print(f"New recursion limit: {sys.getrecursionlimit()}")Packages organize related modules into directories:
Package Structure:
mypackage/
__init__.py
module1.py
module2.py
subpackage/
__init__.py
submodule1.py
submodule2.py
tests/
__init__.py
test_module1.py
test_module2.py
Example Package (mypackage/init.py):
"""MyPackage - A demonstration package."""
__version__ = "1.0.0"
__author__ = "Python Master"
# Import key components for easier access
from mypackage.module1 import ClassA, function_x
from mypackage.module2 import ClassB, function_y
# Package initialization code
print(f"Initializing mypackage v{__version__}")
# Define what gets imported with "from mypackage import *"
__all__ = ['ClassA', 'ClassB', 'function_x', 'function_y']Module1 (mypackage/module1.py):
"""Module 1 of mypackage."""
class ClassA:
"""Class in module 1."""
def __init__(self, name):
self.name = name
def greet(self):
return f"Hello from {self.name} (ClassA)"
def function_x(value):
"""Function in module 1."""
return value * 2
# Module-level variables
VERSION = "1.0"
# Private by convention
_internal_var = 42Module2 (mypackage/module2.py):
"""Module 2 of mypackage."""
class ClassB:
"""Class in module 2."""
def __init__(self, value):
self.value = value
def display(self):
return f"Value: {self.value}"
def function_y(a, b):
"""Function in module 2."""
return a + bSubpackage (mypackage/subpackage/init.py):
"""Subpackage initialization."""
from mypackage.subpackage.submodule1 import HelperClass
from mypackage.subpackage.submodule2 import UtilityClass
__all__ = ['HelperClass', 'UtilityClass']Using the Package:
# Import entire package
import mypackage
print(mypackage.__version__)
obj = mypackage.ClassA("Test")
print(obj.greet())
# Import specific items
from mypackage import ClassB, function_y
obj = ClassB(42)
print(obj.display())
print(function_y(10, 20))
# Import subpackage
from mypackage.subpackage import HelperClass
helper = HelperClass()The __init__.py file controls package behavior:
Package Initialization:
# mypackage/__init__.py
# Version and metadata
__version__ = "1.2.3"
__author__ = "Your Name"
# Control what gets imported
__all__ = ['public_api', 'PublicClass']
# Package-level variables
DEFAULT_CONFIG = {
'debug': False,
'timeout': 30
}
# Import key components for convenience
from .module1 import public_api
from .module2 import PublicClass
# Package initialization code
import logging
logging.getLogger(__name__).addHandler(logging.NullHandler())
# Optional: set up package-level logger
def setup_logging(level=logging.INFO):
logger = logging.getLogger(__name__)
handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter('%(name)s - %(levelname)s - %(message)s'))
logger.addHandler(handler)
logger.setLevel(level)
# Lazy imports (import only when needed)
def get_heavy_module():
"""Dynamically import heavy module."""
from .heavy_module import HeavyClass
return HeavyClass()Different init.py Patterns:
# Simple exports
from .module import *
# Controlled exports
from .module import func1, func2, Class1
__all__ = ['func1', 'func2', 'Class1']
# Namespace package (no __init__.py needed in Python 3.3+)
# Just create directories without __init__.py
# Package as singleton
_instance = None
def get_instance():
global _instance
if _instance is None:
from .core import MainClass
_instance = MainClass()
return _instanceProject Structure for Distribution:
myproject/
README.md
LICENSE
setup.py
setup.cfg
pyproject.toml
requirements.txt
MANIFEST.in
mypackage/
__init__.py
module1.py
module2.py
tests/
__init__.py
test_module1.py
test_module2.py
docs/
conf.py
index.rst
scripts/
myscript.py
setup.py:
#!/usr/bin/env python
"""Setup script for mypackage."""
from setuptools import setup, find_packages
with open("README.md", "r", encoding="utf-8") as fh:
long_description = fh.read()
with open("requirements.txt", "r", encoding="utf-8") as fh:
requirements = [line.strip() for line in fh if line.strip() and not line.startswith("#")]
setup(
name="mypackage",
version="1.0.0",
author="Your Name",
author_email="your.email@example.com",
description="A comprehensive Python package",
long_description=long_description,
long_description_content_type="text/markdown",
url="https://github.com/yourusername/mypackage",
project_urls={
"Bug Tracker": "https://github.com/yourusername/mypackage/issues",
"Documentation": "https://mypackage.readthedocs.io/",
"Source Code": "https://github.com/yourusername/mypackage",
},
classifiers=[
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
"Development Status :: 4 - Beta",
"Intended Audience :: Developers",
"Topic :: Software Development :: Libraries :: Python Modules",
],
packages=find_packages(exclude=["tests", "tests.*", "docs"]),
python_requires=">=3.8",
install_requires=requirements,
extras_require={
"dev": [
"pytest>=7.0",
"pytest-cov>=4.0",
"black>=22.0",
"flake8>=5.0",
"mypy>=1.0",
"sphinx>=5.0",
],
"docs": [
"sphinx>=5.0",
"sphinx-rtd-theme>=1.0",
],
},
entry_points={
"console_scripts": [
"mypackage-script=mypackage.cli:main",
"my-util=mypackage.utils:cli_main",
],
"gui_scripts": [
"my-gui=mypackage.gui:main",
],
},
package_data={
"mypackage": ["data/*.dat", "config/*.ini"],
},
include_package_data=True,
zip_safe=False,
)pyproject.toml (modern alternative):
[build-system]
requires = ["setuptools>=61.0", "wheel"]
build-backend = "setuptools.build_meta"
[project]
name = "mypackage"
version = "1.0.0"
authors = [
{name = "Your Name", email = "your.email@example.com"},
]
description = "A comprehensive Python package"
readme = "README.md"
requires-python = ">=3.8"
license = {text = "MIT"}
keywords = ["example", "package", "tutorial"]
classifiers = [
"Programming Language :: Python :: 3",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
]
dependencies = [
"requests>=2.25",
"click>=8.0",
]
[project.optional-dependencies]
dev = ["pytest>=7.0", "black>=22.0", "flake8>=5.0"]
docs = ["sphinx>=5.0", "sphinx-rtd-theme>=1.0"]
[project.urls]
Homepage = "https://github.com/yourusername/mypackage"
Documentation = "https://mypackage.readthedocs.io/"
Repository = "https://github.com/yourusername/mypackage.git"
[project.scripts]
mypackage-script = "mypackage.cli:main"
my-util = "mypackage.utils:cli_main"
[tool.setuptools.packages.find]
exclude = ["tests", "tests.*", "docs"]
[tool.black]
line-length = 88
target-version = ['py38', 'py39', 'py310', 'py311']
[tool.mypy]
python_version = "3.8"
warn_return_any = true
warn_unused_configs = trueMANIFEST.in (for including non-code files):
include README.md
include LICENSE
include requirements.txt
recursive-include mypackage/data *.dat
recursive-include mypackage/config *.ini
recursive-include docs *
prune docs/_build
global-exclude *.pyc
global-exclude __pycache__
global-exclude .git
requirements.txt:
requests>=2.25,<3.0
click>=8.0
pyyaml>=5.4
# Development dependencies (commented out)
# pytest>=7.0
# black>=22.0
Building the Package:
# Install build tools
pip install build twine
# Build distribution packages
python -m build
# Check the built packages
twine check dist/*Uploading to Test PyPI:
# Upload to Test PyPI (for testing)
twine upload --repository-url https://test.pypi.org/legacy/ dist/*
# Install from Test PyPI
pip install --index-url https://test.pypi.org/simple/ mypackageUploading to PyPI:
# Upload to PyPI
twine upload dist/*
# Install from PyPI
pip install mypackage.pypirc configuration file:
[distutils]
index-servers =
pypi
testpypi
[pypi]
username = __token__
password = pypi-xxxxxxxxxxxxxxxxxxxx
[testpypi]
repository = https://test.pypi.org/legacy/
username = __token__
password = pypi-xxxxxxxxxxxxxxxxxxxx
Version Management:
# mypackage/__init__.py
__version__ = "1.0.0"
# For semantic versioning (MAJOR.MINOR.PATCH)
# MAJOR: incompatible API changes
# MINOR: add functionality (backward compatible)
# PATCH: bug fixes (backward compatible)
# Version bumping script (bump_version.py)
import re
import sys
def bump_version(version_file, part='patch'):
with open(version_file, 'r') as f:
content = f.read()
match = re.search(r"__version__\s*=\s*['\"]([^'\"]+)['\"]", content)
if not match:
raise ValueError("Version not found")
major, minor, patch = map(int, match.group(1).split('.'))
if part == 'major':
major += 1
minor = 0
patch = 0
elif part == 'minor':
minor += 1
patch = 0
else: # patch
patch += 1
new_version = f"{major}.{minor}.{patch}"
new_content = re.sub(
r"__version__\s*=\s*['\"][^'\"]+['\"]",
f"__version__ = '{new_version}'",
content
)
with open(version_file, 'w') as f:
f.write(new_content)
print(f"Version bumped to {new_version}")
if __name__ == "__main__":
bump_version("mypackage/__init__.py", sys.argv[1] if len(sys.argv) > 1 else "patch")Exception handling is a crucial aspect of writing robust Python applications. It allows programs to respond gracefully to unexpected situations rather than crashing.
The basic structure of exception handling in Python uses try, except, else, and finally blocks:
Basic Try-Except:
# Simple exception handling
try:
number = int(input("Enter a number: "))
result = 10 / number
print(f"Result: {result}")
except ValueError:
print("That's not a valid number!")
except ZeroDivisionError:
print("Cannot divide by zero!")
# Multiple exceptions in one block
try:
data = [1, 2, 3]
index = int(input("Enter index: "))
print(data[index])
except (ValueError, IndexError) as e:
print(f"Error: {e}")
# Catching all exceptions (use sparingly)
try:
risky_operation()
except Exception as e:
print(f"Something went wrong: {e}")The Else Clause:
The else block executes only if no exception occurs in the try block:
def divide_numbers(a, b):
try:
result = a / b
except ZeroDivisionError:
print("Error: Division by zero!")
return None
else:
print("Division successful!")
return result
# Using else for code that should run only when no exception occurs
try:
file = open("data.txt", "r")
except FileNotFoundError:
print("File not found!")
else:
content = file.read()
print(f"File contains: {content}")
file.close()The Finally Clause:
finally always executes, regardless of whether an exception occurred:
# Resource cleanup with finally
file = None
try:
file = open("important.txt", "r")
data = file.read()
# Process data
result = 10 / len(data) # Might raise exception
except ZeroDivisionError:
print("Data is empty!")
finally:
if file:
file.close()
print("File closed.")
# Finally always runs
def test_finally():
try:
print("In try")
return "Return from try"
finally:
print("In finally - always executes!")
print(test_finally())
# Output:
# In try
# In finally - always executes!
# Return from tryComplete Example:
def process_user_data(filename):
"""Process user data from file with comprehensive error handling."""
print(f"Processing {filename}...")
try:
# Attempt to open and read file
with open(filename, 'r') as file:
data = file.readlines()
# Process each line
users = []
for line_num, line in enumerate(data, 1):
try:
# Parse line (format: "name,age")
name, age_str = line.strip().split(',')
age = int(age_str)
if age < 0 or age > 150:
raise ValueError(f"Invalid age: {age}")
users.append({"name": name, "age": age})
except ValueError as e:
print(f"Line {line_num}: Invalid data - {e}")
except Exception as e:
print(f"Line {line_num}: Unexpected error - {e}")
except FileNotFoundError:
print(f"Error: File '{filename}' not found!")
return []
except PermissionError:
print(f"Error: No permission to read '{filename}'!")
return []
except Exception as e:
print(f"Unexpected error opening file: {e}")
return []
else:
print(f"Successfully processed {len(users)} users")
return users
finally:
print("File processing completed")
# Test the function
users = process_user_data("users.txt")
print(f"Users: {users}")You can raise exceptions using the raise statement:
Basic Raising:
def validate_age(age):
if age < 0:
raise ValueError("Age cannot be negative")
if age > 150:
raise ValueError("Age cannot exceed 150")
return age
try:
validate_age(-5)
except ValueError as e:
print(f"Validation failed: {e}")
# Raising built-in exceptions
def divide(a, b):
if b == 0:
raise ZeroDivisionError("Cannot divide by zero")
if not isinstance(a, (int, float)) or not isinstance(b, (int, float)):
raise TypeError("Arguments must be numbers")
return a / bRaising with Custom Messages:
def get_user_by_id(user_id):
"""Get user from database."""
if not isinstance(user_id, int):
raise TypeError(f"user_id must be int, got {type(user_id).__name__}")
if user_id <= 0:
raise ValueError(f"user_id must be positive, got {user_id}")
# Simulate database lookup
users = {1: "Alice", 2: "Bob", 3: "Charlie"}
if user_id not in users:
raise KeyError(f"User {user_id} not found")
return users[user_id]Raising and Re-raising:
def process_data(data):
try:
result = complex_calculation(data)
except ValueError as e:
# Log the error and re-raise with additional context
print(f"Error processing data: {e}")
raise ValueError(f"Failed to process data: {data}") from e
def complex_calculation(data):
if not data:
raise ValueError("Empty data")
return data * 2
# Example with chained exceptions
try:
process_data(None)
except ValueError as e:
print(f"Caught: {e}")
print(f"Original cause: {e.__cause__}")Creating custom exception classes for domain-specific errors:
Basic Custom Exception:
class ValidationError(Exception):
"""Base exception for validation errors."""
pass
class EmailError(ValidationError):
"""Exception raised for email validation errors."""
pass
class PasswordError(ValidationError):
"""Exception raised for password validation errors."""
pass
def validate_email(email):
if '@' not in email:
raise EmailError(f"Invalid email: missing @ symbol")
if '.' not in email.split('@')[1]:
raise EmailError(f"Invalid email: missing domain extension")
return True
def validate_password(password):
if len(password) < 8:
raise PasswordError("Password must be at least 8 characters")
if not any(c.isupper() for c in password):
raise PasswordError("Password must contain uppercase letter")
if not any(c.isdigit() for c in password):
raise PasswordError("Password must contain digit")
return True
# Usage
try:
validate_email("user@example")
validate_password("weak")
except EmailError as e:
print(f"Email error: {e}")
except PasswordError as e:
print(f"Password error: {e}")
except ValidationError as e:
print(f"Validation error: {e}")Advanced Custom Exceptions:
class DatabaseError(Exception):
"""Base exception for database operations."""
def __init__(self, message, query=None, error_code=None):
self.message = message
self.query = query
self.error_code = error_code
super().__init__(self.message)
def __str__(self):
base = self.message
if self.query:
base += f"\nQuery: {self.query}"
if self.error_code:
base += f"\nError code: {self.error_code}"
return base
class ConnectionError(DatabaseError):
"""Exception raised for database connection issues."""
pass
class QueryError(DatabaseError):
"""Exception raised for query execution issues."""
pass
class IntegrityError(DatabaseError):
"""Exception raised for data integrity violations."""
pass
def execute_query(query):
"""Simulate database query execution."""
if not query:
raise QueryError("Empty query", query=query, error_code=400)
if "DROP" in query.upper():
raise IntegrityError("Destructive operation not allowed",
query=query, error_code=1001)
if "SELECT" not in query.upper():
raise QueryError("Only SELECT queries allowed",
query=query, error_code=403)
# Simulate connection error
import random
if random.random() < 0.1:
raise ConnectionError("Database connection lost",
error_code=2001)
return f"Executed: {query}"
# Usage with comprehensive error handling
def safe_execute(query):
try:
result = execute_query(query)
print(f"Success: {result}")
except ConnectionError as e:
print(f"Connection failed: {e}")
# Retry logic could go here
except IntegrityError as e:
print(f"Integrity violation: {e}")
# Log and notify admin
except QueryError as e:
print(f"Query error: {e}")
# Fix query and retry
except DatabaseError as e:
print(f"Database error: {e}")
except Exception as e:
print(f"Unexpected error: {e}")
# Test
safe_execute("SELECT * FROM users")
safe_execute("") # QueryError
safe_execute("DROP TABLE users") # IntegrityErrorException Hierarchy Design:
class ApplicationError(Exception):
"""Base class for all application errors."""
pass
class ConfigurationError(ApplicationError):
"""Errors related to application configuration."""
pass
class DataError(ApplicationError):
"""Base class for data-related errors."""
pass
class ValidationError(DataError):
"""Data validation errors."""
pass
class ProcessingError(DataError):
"""Data processing errors."""
pass
class StorageError(ApplicationError):
"""Base class for storage-related errors."""
pass
class DatabaseError(StorageError):
"""Database operation errors."""
pass
class FileSystemError(StorageError):
"""File system operation errors."""
pass
class NetworkError(ApplicationError):
"""Network communication errors."""
pass
class APIError(NetworkError):
"""External API errors."""
def __init__(self, message, status_code=None, response=None):
super().__init__(message)
self.status_code = status_code
self.response = responseGuidelines for Effective Exception Handling:
1. Be Specific with Exceptions:
# Bad - too broad
try:
data = process_input()
except Exception:
print("Error occurred")
# Good - specific exceptions
try:
data = process_input()
except ValueError:
print("Invalid input format")
except KeyError:
print("Missing required field")
except ConnectionError:
print("Network issue, please retry")2. Don't Silently Pass Exceptions:
# Bad - silent failure
try:
risky_operation()
except Exception:
pass
# Good - log and handle appropriately
import logging
try:
risky_operation()
except Exception as e:
logging.error(f"Operation failed: {e}", exc_info=True)
# Either re-raise, return default value, or notify user
raise # Re-raise if can't handle3. Use Finally for Cleanup:
# Bad - resource leak on exception
def read_file(filename):
f = open(filename, 'r')
data = f.read() # If exception here, file stays open
f.close()
return data
# Good - use with statement or finally
def read_file(filename):
f = open(filename, 'r')
try:
data = f.read()
return data
finally:
f.close()
# Better - use context manager
def read_file(filename):
with open(filename, 'r') as f:
return f.read()4. Raise Exceptions at the Right Level:
def get_user_name(user_id):
"""Get user name from database."""
# Low-level function raises specific exceptions
if not isinstance(user_id, int):
raise TypeError("user_id must be integer")
# Database lookup might raise DatabaseError
user = db.find_user(user_id)
if not user:
raise ValueError(f"User {user_id} not found")
return user.name
def display_user_profile(user_id):
"""High-level function handles exceptions appropriately."""
try:
name = get_user_name(user_id)
print(f"User: {name}")
except TypeError:
print("Invalid user ID format")
except ValueError as e:
print(f"User error: {e}")
except DatabaseError as e:
print(f"System error, please try later: {e}")
# Log for administrators
logging.error(f"Database error for user {user_id}: {e}")5. Use Exception Chaining:
def process_order(order_data):
try:
validated_order = validate_order(order_data)
saved_order = save_to_database(validated_order)
return saved_order
except ValidationError as e:
# Add context and re-raise
raise ProcessingError(f"Order validation failed: {order_data}") from e
except DatabaseError as e:
raise ProcessingError("Failed to save order") from e
# Now the caller sees both errors
try:
process_order({"invalid": "data"})
except ProcessingError as e:
print(f"Processing failed: {e}")
print(f"Caused by: {e.__cause__}")6. Create a Consistent Error Handling Strategy:
class ErrorHandler:
"""Centralized error handling."""
@staticmethod
def handle_error(error, context=None):
"""Log error and return appropriate response."""
error_info = {
'type': type(error).__name__,
'message': str(error),
'context': context or {}
}
# Log error
logging.error(f"Error: {error_info}", exc_info=True)
# Return user-friendly message
if isinstance(error, ValidationError):
return {"status": "error", "message": f"Invalid data: {error}"}
elif isinstance(error, PermissionError):
return {"status": "error", "message": "Permission denied"}
elif isinstance(error, ConnectionError):
return {"status": "error", "message": "Service unavailable, please retry"}
else:
return {"status": "error", "message": "An unexpected error occurred"}
def api_endpoint(data):
try:
result = process_data(data)
return {"status": "success", "data": result}
except Exception as e:
return ErrorHandler.handle_error(e, {'data': data})7. Document Exceptions:
def read_config(filename):
"""
Read configuration from file.
Args:
filename (str): Path to configuration file
Returns:
dict: Configuration settings
Raises:
FileNotFoundError: If configuration file doesn't exist
PermissionError: If insufficient permissions to read file
ValueError: If configuration file has invalid format
json.JSONDecodeError: If file contains invalid JSON
"""
with open(filename, 'r') as f:
return json.load(f)8. Use Assertions for Development, Exceptions for Production:
def calculate_discount(price, discount_percent):
"""Calculate discounted price."""
# Assertions for development/debugging
assert price >= 0, "Price cannot be negative"
assert 0 <= discount_percent <= 100, "Discount must be between 0 and 100"
# Production validation
if price < 0:
raise ValueError(f"Invalid price: {price}")
if not 0 <= discount_percent <= 100:
raise ValueError(f"Invalid discount: {discount_percent}")
return price * (100 - discount_percent) / 1009. Create Exception Wrappers for External Libraries:
class ExternalServiceError(Exception):
"""Wrapper for external service errors."""
pass
def call_external_api():
"""Wrap external API calls with custom exceptions."""
try:
import requests
response = requests.get("https://api.example.com/data", timeout=5)
response.raise_for_status()
return response.json()
except requests.Timeout:
raise ExternalServiceError("API request timed out")
except requests.ConnectionError:
raise ExternalServiceError("Failed to connect to API")
except requests.HTTPError as e:
raise ExternalServiceError(f"API returned error: {e.response.status_code}")
except requests.RequestException as e:
raise ExternalServiceError(f"API request failed: {e}")10. Use Context Managers for Resource Management:
class DatabaseConnection:
"""Context manager for database connections."""
def __init__(self, connection_string):
self.connection_string = connection_string
self.connection = None
def __enter__(self):
try:
self.connection = create_connection(self.connection_string)
return self.connection
except Exception as e:
raise ConnectionError(f"Failed to connect to database: {e}")
def __exit__(self, exc_type, exc_val, exc_tb):
if self.connection:
try:
self.connection.close()
except Exception as e:
logging.error(f"Error closing connection: {e}")
# Handle any exception that occurred in the with block
if exc_type is not None:
print(f"Exception occurred: {exc_val}")
# Return False to propagate, True to suppress
return False
# Usage
try:
with DatabaseConnection("db://localhost:5432/mydb") as conn:
result = conn.query("SELECT * FROM users")
process_result(result)
except ConnectionError as e:
print(f"Database error: {e}")The logging module provides a flexible framework for emitting log messages:
Basic Logging Configuration:
import logging
# Simple configuration
logging.basicConfig(
level=logging.DEBUG,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
datefmt='%Y-%m-%d %H:%M:%S'
)
# Log messages at different levels
logging.debug("Debug message - detailed information")
logging.info("Info message - confirmation that things are working")
logging.warning("Warning message - something unexpected but not critical")
logging.error("Error message - serious problem")
logging.critical("Critical message - program may not continue")
# Logging with variables
user = "Alice"
action = "login"
logging.info(f"User {user} performed {action}")Logger Hierarchy:
# Create named loggers
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
# Create child loggers
module_logger = logging.getLogger(__name__ + ".module")
submodule_logger = logging.getLogger(__name__ + ".module.submodule")
# Configure handlers
console_handler = logging.StreamHandler()
console_handler.setLevel(logging.INFO)
file_handler = logging.FileHandler('app.log')
file_handler.setLevel(logging.DEBUG)
# Create formatters
console_format = logging.Formatter('%(name)s - %(levelname)s - %(message)s')
file_format = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
console_handler.setFormatter(console_format)
file_handler.setFormatter(file_format)
# Add handlers to logger
logger.addHandler(console_handler)
logger.addHandler(file_handler)
# Test
logger.debug("This goes to file only")
logger.info("This goes to both console and file")Advanced Logging:
import logging
import logging.handlers
import json
class JSONFormatter(logging.Formatter):
"""Custom JSON formatter for structured logging."""
def format(self, record):
log_record = {
'timestamp': self.formatTime(record),
'name': record.name,
'level': record.levelname,
'message': record.getMessage(),
'module': record.module,
'function': record.funcName,
'line': record.lineno
}
if record.exc_info:
log_record['exception'] = self.formatException(record.exc_info)
if hasattr(record, 'extra_data'):
log_record['extra'] = record.extra_data
return json.dumps(log_record)
# Configure rotating file handler
handler = logging.handlers.RotatingFileHandler(
'app.log',
maxBytes=10485760, # 10MB
backupCount=5
)
handler.setFormatter(JSONFormatter())
# Configure timed rotating file handler
timed_handler = logging.handlers.TimedRotatingFileHandler(
'app_daily.log',
when='midnight',
interval=1,
backupCount=30
)
# Configure syslog handler
syslog_handler = logging.handlers.SysLogHandler(address='/dev/log')
# Configure email handler for critical errors
smtp_handler = logging.handlers.SMTPHandler(
mailhost=('smtp.example.com', 587),
fromaddr='logger@example.com',
toaddrs=['admin@example.com'],
subject='Application Error',
credentials=('username', 'password'),
secure=()
)
smtp_handler.setLevel(logging.CRITICAL)
# Root logger configuration
logging.basicConfig(
level=logging.INFO,
handlers=[handler, timed_handler, syslog_handler, smtp_handler]
)
# Create contextual logger
class ContextAdapter(logging.LoggerAdapter):
"""Logger adapter that adds context to all messages."""
def process(self, msg, kwargs):
# Add request_id to all log messages if present
if 'request_id' in self.extra:
msg = f"[Request: {self.extra['request_id']}] {msg}"
return msg, kwargs
# Usage with context
logger = ContextAdapter(logging.getLogger(__name__), {'request_id': '12345'})
logger.info("Processing request")
logger.error("Failed to process", extra={'extra_data': {'user': 'alice', 'action': 'login'}})Logging Configuration File (logging.conf):
[loggers]
keys=root,myapp
[handlers]
keys=consoleHandler,fileHandler
[formatters]
keys=simpleFormatter,detailedFormatter
[logger_root]
level=DEBUG
handlers=consoleHandler
[logger_myapp]
level=DEBUG
handlers=consoleHandler,fileHandler
qualname=myapp
propagate=0
[handler_consoleHandler]
class=StreamHandler
level=INFO
formatter=simpleFormatter
args=(sys.stdout,)
[handler_fileHandler]
class=handlers.RotatingFileHandler
level=DEBUG
formatter=detailedFormatter
args=('app.log', 'a', 10485760, 5)
[formatter_simpleFormatter]
format=%(asctime)s - %(name)s - %(levelname)s - %(message)s
datefmt=
[formatter_detailedFormatter]
format=%(asctime)s - %(name)s - %(levelname)s - %(module)s:%(lineno)d - %(message)s
datefmt=%Y-%m-%d %H:%M:%SLoad configuration:
import logging.config
logging.config.fileConfig('logging.conf')
logger = logging.getLogger('myapp')Python's built-in debugger for interactive debugging:
Starting pdb:
# Method 1: Insert breakpoint in code
import pdb
def buggy_function(x, y):
result = x + y
pdb.set_trace() # Debugger starts here
result = result * 2
return result
# Method 2: Run script with debugger
# python -m pdb script.py
# Method 3: Post-mortem debugging after exception
try:
buggy_function(5, 3)
except Exception:
import pdb
pdb.post_mortem()
# Method 4: Python 3.7+ built-in breakpoint()
def calculate(data):
breakpoint() # Same as pdb.set_trace()
return sum(data) / len(data)pdb Commands:
# Debugger commands
"""
Basic commands:
h(elp) - Show help
q(uit) - Exit debugger
c(ontinue) - Continue execution
n(ext) - Execute next line
s(tep) - Step into function call
r(eturn) - Continue until function returns
Breakpoints:
b(reak) [file:]lineno - Set breakpoint
b(reak) function - Set breakpoint at function
cl(ear) [bpnumber] - Clear breakpoint
tbreak - Temporary breakpoint
disable/enable - Disable/enable breakpoint
Inspecting:
p(rint) expression - Print expression value
pp expression - Pretty-print expression
a(rgs) - Print arguments of current function
w(here) - Print stack trace
u(p) - Move up one stack frame
d(own) - Move down one stack frame
l(ist) - Show source code around current line
ll - Show full source for current function
Variables:
whatis expression - Show type of expression
display expression - Display expression value when changed
undisplay - Stop displaying expression
Execution:
j(ump) lineno - Jump to line (use carefully)
unt(il) - Continue until line number
condition bpnumber expression - Set breakpoint condition
"""
# Example debugging session
def factorial(n):
if n == 0:
return 1
result = n * factorial(n - 1)
return result
def process_numbers(numbers):
breakpoint() # Start debugging here
total = 0
for i, num in enumerate(numbers):
total += factorial(num)
return total / len(numbers)
# Run with debugger
numbers = [1, 2, 3, 4, 5]
result = process_numbers(numbers)
print(f"Average factorial: {result}")Advanced pdb Usage:
import pdb
class DebuggerExample:
def __init__(self):
self.data = []
def add_data(self, value):
self.data.append(value)
def process(self):
total = 0
for i, item in enumerate(self.data):
# Conditional breakpoint
if i == 3: # Break on specific iteration
pdb.set_trace()
total += item * 2
# Post-mortem debugging setup
try:
result = total / len(self.data)
except Exception:
pdb.post_mortem()
return result
# Custom debugger commands
class CustomPdb(pdb.Pdb):
def do_showdata(self, arg):
"""Show current data."""
if hasattr(self.curframe.f_locals, 'self'):
obj = self.curframe.f_locals['self']
if hasattr(obj, 'data'):
print(f"Data: {obj.data}")
return 1
# Use custom debugger
def debug_with_custom():
debugger = CustomPdb()
debugger.set_trace()
example = DebuggerExample()
for i in range(5):
example.add_data(i)
result = example.process()
print(result)Profiling helps identify performance bottlenecks:
Basic Profiling:
import cProfile
import pstats
import io
def slow_function():
total = 0
for i in range(1000000):
total += i ** 2
return total
def medium_function():
result = []
for i in range(100000):
result.append(i * 2)
return result
def fast_function():
return [i * 2 for i in range(100000)]
def main():
slow_function()
medium_function()
fast_function()
# Profile with context manager
with cProfile.Profile() as pr:
main()
# Print stats
stats = pstats.Stats(pr)
stats.sort_stats(pstats.SortKey.TIME)
stats.print_stats(10) # Top 10 by time
# Save to file
pr.dump_stats('profile_results.prof')
# Run from command line
# python -m cProfile -o output.prof script.pyAnalyzing Profile Data:
import pstats
from pstats import SortKey
# Load and analyze profile
p = pstats.Stats('profile_results.prof')
# Sort by different criteria
p.sort_stats(SortKey.CUMULATIVE) # Cumulative time
p.print_stats(20) # Top 20
p.sort_stats(SortKey.TIME) # Internal time
p.print_stats(20)
p.sort_stats(SortKey.CALLS) # Number of calls
p.print_stats(20)
# Print callers of specific function
p.print_callers('slow_function')
# Print callees
p.print_callees('main')
# Strip directories for cleaner output
p.strip_dirs()
p.sort_stats(SortKey.TIME)
p.print_stats()
# Print stats as percentage
p.sort_stats(SortKey.TIME)
p.print_stats(.5) # Only functions with >50% of timeProfiling Specific Functions:
import cProfile
import functools
def profile_decorator(func):
"""Decorator to profile a function."""
@functools.wraps(func)
def wrapper(*args, **kwargs):
profiler = cProfile.Profile()
try:
return profiler.runcall(func, *args, **kwargs)
finally:
stats = pstats.Stats(profiler)
stats.sort_stats(SortKey.TIME)
stats.print_stats()
return wrapper
@profile_decorator
def function_to_profile():
total = 0
for i in range(1000000):
total += i ** 0.5
return total
function_to_profile()
# Profile code block
code = '''
for i in range(1000000):
x = i ** 2
'''
cProfile.run(code, 'block_profile.prof')
# Profile with context
profiler = cProfile.Profile()
profiler.enable()
# Code to profile
result = slow_function()
profiler.disable()
profiler.print_stats()Visualizing Profile Data:
# Using snakeviz for visualization
# pip install snakeviz
# snakeviz profile_results.prof
# Generate call graph
import gprof2dot
import subprocess
# Convert profile to dot format
subprocess.run([
'gprof2dot', '-f', 'pstats', 'profile_results.prof',
'-o', 'profile.dot'
])
# Generate PNG
subprocess.run([
'dot', '-Tpng', 'profile.dot', '-o', 'profile.png'
])
# Using pyprof2calltree for KCachegrind
# pip install pyprof2calltree
# pyprof2calltree -i profile_results.prof -o profile.calltree
# Then open with kcachegrindMemory profiling helps identify memory leaks and optimize memory usage:
Basic Memory Profiling:
import tracemalloc
import sys
# Start tracing memory allocations
tracemalloc.start()
# Take snapshot
snapshot1 = tracemalloc.take_snapshot()
# Code to profile
large_list = [i for i in range(1000000)]
large_dict = {i: i**2 for i in range(10000)}
# Take another snapshot
snapshot2 = tracemalloc.take_snapshot()
# Compare snapshots
top_stats = snapshot2.compare_to(snapshot1, 'lineno')
print("[ Top 10 differences ]")
for stat in top_stats[:10]:
print(stat)
# Get memory usage of specific objects
print(f"List size: {sys.getsizeof(large_list)} bytes")
print(f"Dict size: {sys.getsizeof(large_dict)} bytes")
# Trace memory allocations
tracemalloc.stop()Using memory_profiler:
# pip install memory_profiler
# pip install psutil # for better performance
from memory_profiler import profile
@profile
def memory_intensive_function():
"""Function with memory profiling."""
# Create large data structures
a = [i for i in range(1000000)]
b = [i ** 2 for i in range(500000)]
c = a + b
# Create dictionary
d = {i: i ** 0.5 for i in range(100000)}
# Some processing
result = sum(c[:100000]) / len(d)
return result
result = memory_intensive_function()
print(f"Result: {result}")
# Profile line by line
# mprof run script.py
# mprof plotMemory Leak Detection:
import tracemalloc
import gc
import sys
class LeakyClass:
def __init__(self):
self.data = [i for i in range(1000)]
self.circular = self # Circular reference
def detect_leaks():
# Force garbage collection
gc.collect()
# Baseline snapshot
tracemalloc.start()
snapshot1 = tracemalloc.take_snapshot()
# Create objects that might leak
leaky_objects = []
for _ in range(100):
obj = LeakyClass()
leaky_objects.append(obj)
# Delete references
del leaky_objects
# Force garbage collection
gc.collect()
# Take another snapshot
snapshot2 = tracemalloc.take_snapshot()
# Compare
top_stats = snapshot2.compare_to(snapshot1, 'lineno')
print("Memory changes:")
for stat in top_stats[:10]:
if stat.size_diff > 0:
print(f"{stat.traceback.format()[-1]} - {stat.size_diff / 1024:.1f} KB")
# Get objects still alive
unreachable = gc.collect()
print(f"Unreachable objects collected: {unreachable}")
# Get objects by type
objects = gc.get_objects()
type_counts = {}
for obj in objects:
obj_type = type(obj).__name__
type_counts[obj_type] = type_counts.get(obj_type, 0) + 1
print("Object counts by type:")
for obj_type, count in sorted(type_counts.items(), key=lambda x: x[1], reverse=True)[:10]:
print(f" {obj_type}: {count}")
detect_leaks()Memory Usage Optimization:
import sys
import array
import numpy as np
def compare_memory_usage():
"""Compare memory usage of different data structures."""
# List of integers
list_int = list(range(1000000))
print(f"List of ints: {sys.getsizeof(list_int) / 1024 / 1024:.2f} MB")
# Array of integers
array_int = array.array('i', range(1000000))
print(f"Array of ints: {sys.getsizeof(array_int) / 1024 / 1024:.2f} MB")
# Tuple of integers
tuple_int = tuple(range(1000000))
print(f"Tuple of ints: {sys.getsizeof(tuple_int) / 1024 / 1024:.2f} MB")
# NumPy array (if available)
try:
numpy_array = np.arange(1000000, dtype=np.int32)
print(f"NumPy array (int32): {numpy_array.nbytes / 1024 / 1024:.2f} MB")
except ImportError:
pass
# Generator (lazy evaluation)
def gen():
for i in range(1000000):
yield i
gen_obj = gen()
print(f"Generator: {sys.getsizeof(gen_obj) / 1024:.2f} KB")
compare_memory_usage()
# Using __slots__ to reduce memory in classes
class WithoutSlots:
def __init__(self, x, y):
self.x = x
self.y = y
class WithSlots:
__slots__ = ['x', 'y']
def __init__(self, x, y):
self.x = x
self.y = y
# Compare memory
obj1 = WithoutSlots(1, 2)
obj2 = WithSlots(1, 2)
print(f"Without __slots__: {sys.getsizeof(obj1)} bytes")
print(f"With __slots__: {sys.getsizeof(obj2)} bytes")
# Create many instances
instances1 = [WithoutSlots(i, i) for i in range(10000)]
instances2 = [WithSlots(i, i) for i in range(10000)]
# The difference becomes significant with many instancesTechniques for optimizing Python code:
Identifying Bottlenecks:
import time
import functools
def timer_decorator(func):
"""Simple timer decorator."""
@functools.wraps(func)
def wrapper(*args, **kwargs):
start = time.perf_counter()
result = func(*args, **kwargs)
end = time.perf_counter()
print(f"{func.__name__} took {end - start:.4f} seconds")
return result
return wrapper
@timer_decorator
def slow_method():
time.sleep(1)
return "Done"
# Using timeit for micro-benchmarks
import timeit
# Time a small code snippet
time = timeit.timeit('"-".join(str(n) for n in range(100))', number=10000)
print(f"Time: {time:.4f} seconds")
# Time a function
def test_function():
return sum(range(1000))
time = timeit.timeit(test_function, number=100000)
print(f"Average: {time / 100000 * 1e6:.2f} µs per call")
# Using timeit in IPython/Jupyter
# %timeit sum(range(1000))Optimization Techniques:
# 1. Use local variables
@timer_decorator
def global_variable():
global_var = 0
for i in range(1000000):
global_var += i
return global_var
@timer_decorator
def local_variable():
local_var = 0
for i in range(1000000):
local_var += i
return local_var
# 2. Use list comprehensions instead of loops
@timer_decorator
def loop_approach():
result = []
for i in range(1000000):
result.append(i ** 2)
return result
@timer_decorator
def comprehension_approach():
return [i ** 2 for i in range(1000000)]
# 3. Use built-in functions
@timer_decorator
def manual_sum():
total = 0
for i in range(1000000):
total += i
return total
@timer_decorator
def builtin_sum():
return sum(range(1000000))
# 4. Use appropriate data structures
@timer_decorator
def list_search():
data = list(range(10000))
return [i in data for i in range(1000)]
@timer_decorator
def set_search():
data = set(range(10000))
return [i in data for i in range(1000)]
# 5. String concatenation
@timer_decorator
def string_concat():
result = ""
for i in range(10000):
result += str(i)
return len(result)
@timer_decorator
def string_join():
parts = []
for i in range(10000):
parts.append(str(i))
result = "".join(parts)
return len(result)
# 6. Use generators for large datasets
def large_dataset_generator():
for i in range(10000000):
yield i ** 2
@timer_decorator
def process_with_list():
data = [i ** 2 for i in range(10000000)]
return sum(data) / len(data)
@timer_decorator
def process_with_generator():
data = large_dataset_generator()
total = 0
count = 0
for value in data:
total += value
count += 1
return total / countCaching and Memoization:
import functools
# LRU Cache
@functools.lru_cache(maxsize=128)
def fibonacci(n):
"""Calculate nth Fibonacci number with caching."""
if n < 2:
return n
return fibonacci(n - 1) + fibonacci(n - 2)
# Compare performance
@timer_decorator
def fib_without_cache():
def fib(n):
if n < 2:
return n
return fib(n - 1) + fib(n - 2)
return fib(35)
@timer_decorator
def fib_with_cache():
return fibonacci(35)
print(fib_without_cache())
print(fib_with_cache())
print(fibonacci.cache_info()) # Cache statistics
# Custom caching
def memoize(func):
"""Simple memoization decorator."""
cache = {}
@functools.wraps(func)
def wrapper(*args):
if args not in cache:
cache[args] = func(*args)
return cache[args]
return wrapper
@memoize
def expensive_function(n):
"""Expensive computation."""
import time
time.sleep(0.1) # Simulate work
return n * n
# Usage
for i in range(5):
print(expensive_function(5)) # First call slow, others fastUsing C Extensions:
# Example using NumPy for numerical operations
import numpy as np
@timer_decorator
def pure_python_sum():
data = [i for i in range(1000000)]
return sum(data)
@timer_decorator
def numpy_sum():
data = np.arange(1000000)
return np.sum(data)
# Example using built-in optimized modules
import itertools
@timer_decorator
def manual_permutations():
result = []
for i in range(10):
for j in range(10):
for k in range(10):
if i != j and j != k and i != k:
result.append((i, j, k))
return len(result)
@timer_decorator
def itertools_permutations():
return len(list(itertools.permutations(range(10), 3)))Concurrent Processing:
import concurrent.futures
import threading
import multiprocessing
def cpu_intensive_task(n):
"""CPU-intensive task."""
return sum(i * i for i in range(n))
@timer_decorator
def sequential_processing():
results = []
for i in range(10):
results.append(cpu_intensive_task(1000000))
return results
@timer_decorator
def thread_pool_processing():
with concurrent.futures.ThreadPoolExecutor(max_workers=4) as executor:
results = list(executor.map(cpu_intensive_task, [1000000] * 10))
return results
@timer_decorator
def process_pool_processing():
with concurrent.futures.ProcessPoolExecutor(max_workers=4) as executor:
results = list(executor.map(cpu_intensive_task, [1000000] * 10))
return results
# Compare performance
print("Sequential:", sequential_processing())
print("Thread Pool:", thread_pool_processing()) # May not help with CPU-bound
print("Process Pool:", process_pool_processing()) # Better for CPU-boundProfiling and Optimization Workflow:
import cProfile
import pstats
import io
import functools
def profile_optimize(func):
"""Profile and optimize decorator."""
@functools.wraps(func)
def wrapper(*args, **kwargs):
# Profile the function
profiler = cProfile.Profile()
try:
result = profiler.runcall(func, *args, **kwargs)
return result
finally:
# Analyze profile
stats = pstats.Stats(profiler)
stats.sort_stats(pstats.SortKey.TIME)
# Save detailed profile
stats.dump_stats(f"{func.__name__}.prof")
# Print top 10 time-consuming functions
stream = io.StringIO()
stats.stream = stream
stats.print_stats(10)
print(f"Profile for {func.__name__}:")
print(stream.getvalue())
# Suggest optimizations
print("\nOptimization suggestions:")
for func_stats in stats.stats.items():
if func_stats[1][3] > 0.1: # If total time > 0.1 seconds
print(f" - {func_stats[0][2]} takes {func_stats[1][3]:.3f}s")
return wrapper
@profile_optimize
def complex_operation():
"""Complex operation to profile and optimize."""
# Phase 1
data = []
for i in range(100000):
data.append(i ** 2)
# Phase 2
processed = [x * 2 for x in data if x % 2 == 0]
# Phase 3
result = 0
for value in processed[:10000]:
result += value ** 0.5
return result
result = complex_operation()
print(f"Result: {result}")Performance Best Practices:
"""
Performance Optimization Guidelines:
1. Measure First, Optimize Later
- Use profilers to identify real bottlenecks
- Don't optimize prematurely
2. Choose Appropriate Data Structures
- Lists for ordered collections with frequent access
- Sets/Dicts for fast lookups (O(1) vs O(n))
- Deques for queue operations
- Heaps for priority queues
3. Use Built-in Functions and Libraries
- Built-ins are implemented in C and faster
- Use NumPy for numerical operations
- Use itertools for iterator operations
4. Avoid Unnecessary Operations
- Move invariant calculations out of loops
- Use local variables (faster than global)
- Avoid attribute access in loops
5. Use Generators for Large Data
- Process data lazily
- Reduce memory usage
6. Consider Concurrency
- Threads for I/O-bound tasks
- Processes for CPU-bound tasks
- Asyncio for I/O-bound with many connections
7. Cache Expensive Computations
- Use functools.lru_cache
- Implement custom caching for specific needs
8. Optimize String Operations
- Use join() instead of concatenation
- Use f-strings for formatting
9. Use List Comprehensions
- Faster than manual loops
- More readable
10. Profile and Benchmark
- Use timeit for micro-benchmarks
- Use cProfile for overall profiling
- Use memory_profiler for memory optimization
"""File handling is a fundamental aspect of programming, allowing applications to persist data, read configurations, and process external data sources.
Python provides simple and powerful file handling capabilities:
Basic File Operations:
# Writing to a file
with open('example.txt', 'w') as file:
file.write('Hello, World!\n')
file.write('This is a second line.\n')
file.write('And a third line.\n')
# Reading entire file
with open('example.txt', 'r') as file:
content = file.read()
print(content)
# Reading line by line
with open('example.txt', 'r') as file:
for line in file:
print(line.strip()) # strip() removes newline
# Reading all lines into a list
with open('example.txt', 'r') as file:
lines = file.readlines()
print(lines) # ['Hello, World!\n', 'This is a second line.\n', 'And a third line.\n']
# Appending to file
with open('example.txt', 'a') as file:
file.write('This line is appended.\n')File Modes:
"""
File modes:
'r' - Read (default)
'w' - Write (creates new file or truncates existing)
'x' - Exclusive creation (fails if file exists)
'a' - Append (writes to end of file)
'b' - Binary mode
't' - Text mode (default)
'+' - Read and write
Combinations:
'rb' - Read binary
'wb' - Write binary
'ab' - Append binary
'r+' - Read and write (file must exist)
'w+' - Read and write (creates new or truncates)
'a+' - Read and append
"""
# Examples of different modes
try:
with open('new_file.txt', 'x') as file:
file.write('This file is created exclusively')
except FileExistsError:
print('File already exists')
# Read and write mode
with open('example.txt', 'r+') as file:
content = file.read()
file.seek(0) # Go back to beginning
file.write('MODIFIED: ' + content)Working with File Positions:
with open('example.txt', 'r+') as file:
# Get current position
position = file.tell()
print(f"Current position: {position}")
# Read first 10 characters
data = file.read(10)
print(f"Read: {data}")
print(f"New position: {file.tell()}")
# Seek to specific position
file.seek(0) # Go to beginning
print(f"After seek(0): {file.tell()}")
file.seek(10, 1) # Seek 10 bytes from current position
print(f"After seek(10, 1): {file.tell()}")
file.seek(-5, 2) # Seek 5 bytes from end
print(f"After seek(-5, 2): {file.tell()}")Context Managers and Custom File Handlers:
class ManagedFile:
"""Custom context manager for file handling."""
def __init__(self, filename, mode):
self.filename = filename
self.mode = mode
self.file = None
def __enter__(self):
print(f"Opening {self.filename}")
self.file = open(self.filename, self.mode)
return self.file
def __exit__(self, exc_type, exc_val, exc_tb):
print(f"Closing {self.filename}")
if self.file:
self.file.close()
# Handle any exceptions
if exc_type:
print(f"An exception occurred: {exc_val}")
return False # Propagate exceptions
# Usage
with ManagedFile('example.txt', 'r') as file:
content = file.read()
print(content)Working with binary data:
Reading and Writing Binary Files:
# Writing binary data
data = bytes(range(256))
with open('binary.dat', 'wb') as file:
file.write(data)
# Reading binary data
with open('binary.dat', 'rb') as file:
binary_data = file.read()
print(f"Read {len(binary_data)} bytes")
print(f"First 10 bytes: {binary_data[:10]}")
# Working with bytearray (mutable)
with open('binary.dat', 'r+b') as file:
# Read first 10 bytes
data = bytearray(file.read(10))
print(f"Original: {data}")
# Modify bytes
for i in range(len(data)):
data[i] = data[i] ^ 0xFF # XOR operation
# Write back
file.seek(0)
file.write(data)Structured Binary Data:
import struct
# Packing data into binary format
# Format strings:
# 'i' - integer (4 bytes)
# 'f' - float (4 bytes)
# 'd' - double (8 bytes)
# 's' - string (bytes)
# 'h' - short (2 bytes)
# '?' - bool (1 byte)
# Packing
values = (42, 3.14159, True, b'Hello')
packed_data = struct.pack('ifd5s', *values)
print(f"Packed {len(packed_data)} bytes: {packed_data}")
# Writing packed data
with open('structured.dat', 'wb') as file:
file.write(packed_data)
# Reading and unpacking
with open('structured.dat', 'rb') as file:
data = file.read()
unpacked = struct.unpack('ifd5s', data)
print(f"Unpacked: {unpacked}")
# Working with multiple records
records = [
(1, 3.14, b'first'),
(2, 2.718, b'second'),
(3, 1.618, b'third')
]
# Write records
with open('records.dat', 'wb') as file:
for record in records:
packed = struct.pack('if6s', *record)
file.write(packed)
# Read records
with open('records.dat', 'rb') as file:
record_size = struct.calcsize('if6s')
while True:
data = file.read(record_size)
if not data:
break
record = struct.unpack('if6s', data)
# Convert bytes to string, strip null bytes
record = (record[0], record[1], record[2].decode().strip('\x00'))
print(f"Record: {record}")JavaScript Object Notation - human-readable data interchange format:
Basic JSON Operations:
import json
# Python data to JSON string
data = {
'name': 'Alice',
'age': 30,
'city': 'New York',
'hobbies': ['reading', 'cycling', 'photography'],
'is_student': False,
'grades': None,
'address': {
'street': '123 Main St',
'zipcode': '10001'
}
}
# Serialize to JSON string
json_string = json.dumps(data, indent=2)
print(json_string)
# Write to file
with open('data.json', 'w') as file:
json.dump(data, file, indent=2)
# Read from file
with open('data.json', 'r') as file:
loaded_data = json.load(file)
print(loaded_data['name'])
print(loaded_data['hobbies'])
# Parse JSON string
json_str = '{"name": "Bob", "age": 25, "city": "Boston"}'
parsed = json.loads(json_str)
print(parsed['name'])Advanced JSON Handling:
import json
from datetime import datetime, date
# Custom JSON encoder for dates and custom objects
class CustomJSONEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, (datetime, date)):
return obj.isoformat()
if isinstance(obj, complex):
return {'real': obj.real, 'imag': obj.imag}
if hasattr(obj, '__dict__'):
return obj.__dict__
return super().default(obj)
# Custom JSON decoder
def custom_decoder(dct):
if 'real' in dct and 'imag' in dct:
return complex(dct['real'], dct['imag'])
if 'isoformat' in dct:
return datetime.fromisoformat(dct['isoformat'])
return dct
# Complex data with custom types
data = {
'name': 'Alice',
'created_at': datetime.now(),
'updated_at': date.today(),
'complex_number': 3 + 4j,
'items': [1, 2, 3]
}
# Serialize with custom encoder
json_str = json.dumps(data, cls=CustomJSONEncoder, indent=2)
print(json_str)
# Deserialize with custom decoder
decoded = json.loads(json_str, object_hook=custom_decoder)
print(decoded)
# Handling JSON lines format (each line is a JSON object)
import jsonlines # pip install jsonlines
# Writing JSON lines
data = [
{'name': 'Alice', 'age': 30},
{'name': 'Bob', 'age': 25},
{'name': 'Charlie', 'age': 35}
]
with jsonlines.open('data.jsonl', 'w') as writer:
writer.write_all(data)
# Reading JSON lines
with jsonlines.open('data.jsonl') as reader:
for obj in reader:
print(obj)Comma-Separated Values - common format for tabular data:
Basic CSV Operations:
import csv
# Writing CSV file
data = [
['Name', 'Age', 'City'],
['Alice', 30, 'New York'],
['Bob', 25, 'Los Angeles'],
['Charlie', 35, 'Chicago']
]
with open('people.csv', 'w', newline='') as file:
writer = csv.writer(file)
writer.writerows(data)
# Reading CSV file
with open('people.csv', 'r') as file:
reader = csv.reader(file)
for row in reader:
print(row)
# Using DictReader and DictWriter
data = [
{'Name': 'Alice', 'Age': 30, 'City': 'New York'},
{'Name': 'Bob', 'Age': 25, 'City': 'Los Angeles'},
{'Name': 'Charlie', 'Age': 35, 'City': 'Chicago'}
]
with open('people_dict.csv', 'w', newline='') as file:
fieldnames = ['Name', 'Age', 'City']
writer = csv.DictWriter(file, fieldnames=fieldnames)
writer.writeheader()
writer.writerows(data)
with open('people_dict.csv', 'r') as file:
reader = csv.DictReader(file)
for row in reader:
print(f"{row['Name']} is {row['Age']} years old and lives in {row['City']}")Advanced CSV Handling:
import csv
# Custom dialect
csv.register_dialect('pipes', delimiter='|', quoting=csv.QUOTE_MINIMAL)
# Writing with custom dialect
data = [
['Name', 'Age', 'City'],
['Alice', 30, 'New York'],
['Bob', 25, 'Los Angeles']
]
with open('people_pipes.csv', 'w', newline='') as file:
writer = csv.writer(file, dialect='pipes')
writer.writerows(data)
# Reading with custom dialect
with open('people_pipes.csv', 'r') as file:
reader = csv.reader(file, dialect='pipes')
for row in reader:
print(row)
# Handling different quoting options
data = [
['Name', 'Description', 'Price'],
['Product A', 'This is a "great" product', 10.99],
['Product B', 'Contains, commas, in text', 20.50]
]
with open('products.csv', 'w', newline='') as file:
writer = csv.writer(file, quoting=csv.QUOTE_ALL)
writer.writerows(data)
with open('products.csv', 'r') as file:
reader = csv.reader(file)
for row in reader:
print(row)
# Large CSV processing (streaming)
def process_large_csv(filename, chunk_size=1000):
"""Process large CSV file in chunks."""
with open(filename, 'r') as file:
reader = csv.DictReader(file)
chunk = []
for i, row in enumerate(reader):
chunk.append(row)
if (i + 1) % chunk_size == 0:
# Process chunk
yield chunk
chunk = []
if chunk:
yield chunk
# Example usage
# for chunk in process_large_csv('huge_data.csv'):
# process_chunk(chunk)
# CSV with different delimiters
tsv_data = "Name\tAge\tCity\nAlice\t30\tNew York\nBob\t25\tLos Angeles"
with open('data.tsv', 'w') as file:
file.write(tsv_data)
with open('data.tsv', 'r') as file:
reader = csv.reader(file, delimiter='\t')
for row in reader:
print(row)YAML Ain't Markup Language - human-friendly data serialization:
Basic YAML Operations:
import yaml # pip install pyyaml
# Python data to YAML
data = {
'name': 'Alice',
'age': 30,
'address': {
'street': '123 Main St',
'city': 'New York',
'zipcode': 10001
},
'hobbies': ['reading', 'cycling', 'photography'],
'is_student': False,
'grades': None
}
# Convert to YAML string
yaml_str = yaml.dump(data, default_flow_style=False)
print(yaml_str)
# Write to file
with open('data.yaml', 'w') as file:
yaml.dump(data, file, default_flow_style=False)
# Read from file
with open('data.yaml', 'r') as file:
loaded_data = yaml.safe_load(file)
print(loaded_data['name'])
print(loaded_data['address']['city'])
# Parse YAML string
yaml_text = """
name: Bob
age: 25
address:
street: 456 Oak Ave
city: Boston
zipcode: 02108
hobbies:
- swimming
- hiking
"""
parsed = yaml.safe_load(yaml_text)
print(parsed)Advanced YAML Features:
import yaml
from datetime import datetime
# Custom YAML representer and constructor
class Person:
def __init__(self, name, age):
self.name = name
self.age = age
def __repr__(self):
return f"Person(name={self.name}, age={self.age})"
# Representer for custom class
def person_representer(dumper, data):
return dumper.represent_mapping('!Person', {
'name': data.name,
'age': data.age
})
# Constructor for custom class
def person_constructor(loader, node):
values = loader.construct_mapping(node)
return Person(values['name'], values['age'])
# Register custom handlers
yaml.add_representer(Person, person_representer)
yaml.add_constructor('!Person', person_constructor)
# YAML with custom types
data = {
'person': Person('Alice', 30),
'created_at': datetime.now(),
'tags': ['python', 'yaml', 'tutorial']
}
# Dump with custom types
yaml_str = yaml.dump(data, default_flow_style=False)
print(yaml_str)
# Load with custom types
loaded = yaml.safe_load(yaml_str)
print(loaded)
# YAML with anchors and aliases (reusing data)
yaml_with_refs = """
defaults: &defaults
adapter: postgresql
host: localhost
development:
database: myapp_dev
<<: *defaults
test:
database: myapp_test
<<: *defaults
production:
database: myapp_prod
host: production.db.example.com
<<: *defaults
"""
config = yaml.safe_load(yaml_with_refs)
print(config['development']['host']) # localhost
print(config['production']['host']) # production.db.example.com
# Multi-document YAML
multi_doc = """
---
name: Document 1
content: First document
---
name: Document 2
content: Second document
---
name: Document 3
content: Third document
"""
documents = list(yaml.safe_load_all(multi_doc))
for doc in documents:
print(doc['name'])Extensible Markup Language - structured data format:
Basic XML Operations:
import xml.etree.ElementTree as ET
# Creating XML
root = ET.Element('data')
person1 = ET.SubElement(root, 'person')
person1.set('id', '1')
name1 = ET.SubElement(person1, 'name')
name1.text = 'Alice'
age1 = ET.SubElement(person1, 'age')
age1.text = '30'
city1 = ET.SubElement(person1, 'city')
city1.text = 'New York'
person2 = ET.SubElement(root, 'person')
person2.set('id', '2')
name2 = ET.SubElement(person2, 'name')
name2.text = 'Bob'
age2 = ET.SubElement(person2, 'age')
age2.text = '25'
city2 = ET.SubElement(person2, 'city')
city2.text = 'Los Angeles'
# Write to file
tree = ET.ElementTree(root)
tree.write('people.xml', encoding='utf-8', xml_declaration=True)
# Read XML file
tree = ET.parse('people.xml')
root = tree.getroot()
# Iterate through elements
for person in root.findall('person'):
person_id = person.get('id')
name = person.find('name').text
age = person.find('age').text
city = person.find('city').text
print(f"Person {person_id}: {name}, {age}, {city}")
# Find specific elements
alice = root.find(".//person[name='Alice']")
if alice is not None:
print(f"Found Alice: {alice.find('city').text}")
# Modify XML
for person in root.findall('person'):
age = int(person.find('age').text)
person.find('age').text = str(age + 1) # Birthday!
tree.write('people_updated.xml')Advanced XML Handling:
import xml.etree.ElementTree as ET
import xml.dom.minidom as minidom
# Pretty print XML
def prettify(elem):
"""Return pretty-printed XML string."""
rough_string = ET.tostring(elem, 'utf-8')
reparsed = minidom.parseString(rough_string)
return reparsed.toprettyxml(indent=" ")
# Create XML with namespaces
root = ET.Element('{http://example.com/ns}root')
root.set('{http://www.w3.org/2001/XMLSchema-instance}schemaLocation',
'http://example.com/ns schema.xsd')
child = ET.SubElement(root, '{http://example.com/ns}child')
child.text = 'Content with namespace'
print(prettify(root))
# Parse XML with namespaces
xml_with_ns = '''<?xml version="1.0"?>
<root xmlns="http://example.com/ns"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<child>Content</child>
</root>
'''
root = ET.fromstring(xml_with_ns)
# Need to use full namespace in find
ns = {'ns': 'http://example.com/ns'}
child = root.find('ns:child', ns)
print(child.text)
# XPath queries
xml_data = '''
<library>
<book category="fiction">
<title>The Great Gatsby</title>
<author>F. Scott Fitzgerald</author>
<year>1925</year>
</book>
<book category="fiction">
<title>1984</title>
<author>George Orwell</author>
<year>1949</year>
</book>
<book category="non-fiction">
<title>A Brief History of Time</title>
<author>Stephen Hawking</author>
<year>1988</year>
</book>
</library>
'''
root = ET.fromstring(xml_data)
# Find all titles
titles = root.findall('.//title')
for title in titles:
print(title.text)
# Find books from fiction category
fiction_books = root.findall(".//book[@category='fiction']")
for book in fiction_books:
print(book.find('title').text)
# Find books published after 1940
recent_books = root.findall(".//book[year > 1940]")
for book in recent_books:
print(book.find('title').text)
# Using iterparse for large XML files
def process_large_xml(filename):
"""Process large XML file incrementally."""
context = ET.iterparse(filename, events=('start', 'end'))
for event, elem in context:
if event == 'end' and elem.tag == 'book':
# Process book element
title = elem.find('title').text
year = elem.find('year').text
print(f"Book: {title} ({year})")
# Clear element to free memory
elem.clear()
# Clean up ancestors
while elem.getprevious() is not None:
del elem.getparent()[0]
# process_large_xml('large_library.xml')XML with lxml (more powerful):
# pip install lxml
from lxml import etree
# Create XML with lxml
root = etree.Element('root')
child = etree.SubElement(root, 'child')
child.text = 'Content'
child.set('attr', 'value')
# XPath with lxml (more powerful)
xml = '''
<root>
<items>
<item id="1">First</item>
<item id="2">Second</item>
<item id="3">Third</item>
</items>
</root>
'''
root = etree.fromstring(xml)
# XPath expressions
items = root.xpath('//item')
print([item.text for item in items])
items_with_id = root.xpath('//item[@id="2"]')
print(items_with_id[0].text)
# XPath with functions
item_count = root.xpath('count(//item)')
print(f"Number of items: {item_count}")
# HTML parsing with lxml
html = '''
<html>
<body>
<div class="content">
<h1>Title</h1>
<p>Paragraph 1</p>
<p>Paragraph 2</p>
</div>
</body>
</html>
'''
tree = etree.HTML(html)
paragraphs = tree.xpath('//p')
for p in paragraphs:
print(p.text)
# Find by class
content = tree.xpath('//div[@class="content"]')
if content:
print(etree.tostring(content[0], pretty_print=True).decode())SQLite is a lightweight, file-based database that comes built into Python:
Basic SQLite Operations:
import sqlite3
from contextlib import closing
# Connect to database (creates file if not exists)
conn = sqlite3.connect('example.db')
# Create a cursor
cursor = conn.cursor()
# Create table
cursor.execute('''
CREATE TABLE IF NOT EXISTS users (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name TEXT NOT NULL,
email TEXT UNIQUE NOT NULL,
age INTEGER,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
)
''')
# Insert data
cursor.execute('''
INSERT INTO users (name, email, age)
VALUES (?, ?, ?)
''', ('Alice', 'alice@example.com', 30))
# Insert multiple records
users = [
('Bob', 'bob@example.com', 25),
('Charlie', 'charlie@example.com', 35),
('Diana', 'diana@example.com', 28)
]
cursor.executemany('''
INSERT INTO users (name, email, age)
VALUES (?, ?, ?)
''', users)
# Commit changes
conn.commit()
# Query data
cursor.execute('SELECT * FROM users')
rows = cursor.fetchall()
for row in rows:
print(row)
# Query with parameters
cursor.execute('SELECT * FROM users WHERE age > ?', (25,))
older_users = cursor.fetchall()
print("\nUsers older than 25:")
for user in older_users:
print(user)
# Fetch one row
cursor.execute('SELECT * FROM users WHERE email = ?', ('alice@example.com',))
alice = cursor.fetchone()
print(f"\nFound: {alice}")
# Update data
cursor.execute('''
UPDATE users SET age = ? WHERE name = ?
''', (31, 'Alice'))
conn.commit()
# Delete data
cursor.execute('DELETE FROM users WHERE name = ?', ('Charlie',))
conn.commit()
# Close connection
conn.close()Using Context Managers:
import sqlite3
from contextlib import closing
# Using connection as context manager
with sqlite3.connect('example.db') as conn:
cursor = conn.cursor()
cursor.execute('SELECT * FROM users')
rows = cursor.fetchall()
for row in rows:
print(row)
# Auto-commit on success, rollback on exception
# Using closing for cursor
with sqlite3.connect('example.db') as conn:
with closing(conn.cursor()) as cursor:
cursor.execute('SELECT * FROM users')
while True:
row = cursor.fetchone()
if row is None:
break
print(row)Row Factories and Custom Types:
import sqlite3
# Use Row factory for dictionary-like access
conn = sqlite3.connect('example.db')
conn.row_factory = sqlite3.Row
cursor = conn.cursor()
cursor.execute('SELECT * FROM users')
row = cursor.fetchone()
print(f"Name: {row['name']}, Email: {row['email']}")
# Custom adapter and converter for Python types
import json
from datetime import datetime
def adapt_datetime(dt):
return dt.isoformat()
def convert_datetime(s):
return datetime.fromisoformat(s.decode())
# Register adapters and converters
sqlite3.register_adapter(datetime, adapt_datetime)
sqlite3.register_converter("timestamp", convert_datetime)
# Connect with detection of types
conn = sqlite3.connect(':memory:', detect_types=sqlite3.PARSE_DECLTYPES)
# Create table with timestamp
conn.execute('''
CREATE TABLE events (
id INTEGER PRIMARY KEY,
name TEXT,
event_time timestamp
)
''')
# Insert datetime
now = datetime.now()
conn.execute('INSERT INTO events (name, event_time) VALUES (?, ?)',
('test', now))
# Retrieve datetime
cursor = conn.execute('SELECT * FROM events')
event_id, name, event_time = cursor.fetchone()
print(f"Event: {name}, Time: {event_time}, Type: {type(event_time)}")SQLite in Memory:
import sqlite3
# In-memory database (fast, temporary)
conn = sqlite3.connect(':memory:')
# Create schema and populate
conn.executescript('''
CREATE TABLE employees (
id INTEGER PRIMARY KEY,
name TEXT,
department TEXT,
salary REAL
);
INSERT INTO employees (name, department, salary) VALUES
('Alice', 'Engineering', 85000),
('Bob', 'Marketing', 65000),
('Charlie', 'Engineering', 95000),
('Diana', 'Sales', 75000);
''')
# Aggregate queries
cursor = conn.execute('''
SELECT department, AVG(salary) as avg_salary, COUNT(*) as count
FROM employees
GROUP BY department
''')
for row in cursor:
print(f"Dept: {row[0]}, Avg Salary: ${row[1]:.2f}, Count: {row[2]}")
# Subqueries
cursor = conn.execute('''
SELECT name, salary
FROM employees
WHERE salary > (SELECT AVG(salary) FROM employees)
''')
print("\nEmployees above average salary:")
for row in cursor:
print(f" {row[0]}: ${row[1]}")PostgreSQL is a powerful, open-source object-relational database:
Connecting to PostgreSQL:
# First install: pip install psycopg2 or psycopg2-binary
import psycopg2
from psycopg2 import sql, extras
# Connection parameters
conn_params = {
'host': 'localhost',
'port': 5432,
'database': 'mydb',
'user': 'myuser',
'password': 'mypassword'
}
# Connect to PostgreSQL
try:
conn = psycopg2.connect(**conn_params)
cursor = conn.cursor()
# Create table
cursor.execute('''
CREATE TABLE IF NOT EXISTS products (
id SERIAL PRIMARY KEY,
name VARCHAR(100) NOT NULL,
price DECIMAL(10, 2),
quantity INTEGER DEFAULT 0,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
)
''')
conn.commit()
# Insert data
cursor.execute('''
INSERT INTO products (name, price, quantity)
VALUES (%s, %s, %s)
''', ('Laptop', 999.99, 10))
# Insert multiple with executemany
products = [
('Mouse', 25.50, 50),
('Keyboard', 75.00, 30),
('Monitor', 299.99, 15)
]
cursor.executemany('''
INSERT INTO products (name, price, quantity)
VALUES (%s, %s, %s)
''', products)
conn.commit()
# Query with parameters
cursor.execute('''
SELECT * FROM products WHERE price > %s
''', (100,))
expensive = cursor.fetchall()
for product in expensive:
print(product)
# Use DictCursor for dictionary results
dict_cursor = conn.cursor(cursor_factory=psycopg2.extras.DictCursor)
dict_cursor.execute('SELECT * FROM products')
for row in dict_cursor:
print(f"Product: {row['name']}, Price: ${row['price']}")
except psycopg2.Error as e:
print(f"Database error: {e}")
conn.rollback()
finally:
if conn:
conn.close()Advanced PostgreSQL Features:
import psycopg2
from psycopg2 import pool
from contextlib import contextmanager
# Connection pooling
connection_pool = psycopg2.pool.SimpleConnectionPool(
1, # min connections
20, # max connections
host='localhost',
database='mydb',
user='myuser',
password='mypassword'
)
@contextmanager
def get_connection():
"""Context manager for getting connections from pool."""
conn = connection_pool.getconn()
try:
yield conn
conn.commit()
except Exception:
conn.rollback()
raise
finally:
connection_pool.putconn(conn)
# Use connection pool
with get_connection() as conn:
with conn.cursor() as cursor:
cursor.execute('SELECT * FROM products')
print(cursor.fetchall())
# JSON data type
with get_connection() as conn:
with conn.cursor() as cursor:
# Create table with JSON
cursor.execute('''
CREATE TABLE IF NOT EXISTS user_preferences (
user_id INTEGER PRIMARY KEY,
preferences JSONB
)
''')
# Insert JSON data
prefs = {
'theme': 'dark',
'notifications': True,
'language': 'en',
'favorites': ['python', 'postgresql', 'json']
}
cursor.execute('''
INSERT INTO user_preferences (user_id, preferences)
VALUES (%s, %s)
ON CONFLICT (user_id) DO UPDATE
SET preferences = EXCLUDED.preferences
''', (1, psycopg2.extras.Json(prefs)))
# Query JSON data
cursor.execute('''
SELECT preferences->>'theme' as theme,
preferences->'favorites' as favorites
FROM user_preferences
WHERE user_id = %s
''', (1,))
result = cursor.fetchone()
print(f"Theme: {result[0]}")
print(f"Favorites: {result[1]}")
# Transactions and Savepoints
with get_connection() as conn:
with conn.cursor() as cursor:
try:
# Start transaction
cursor.execute('BEGIN')
# Insert main order
cursor.execute('''
INSERT INTO orders (customer_id, total)
VALUES (%s, %s) RETURNING id
''', (1, 1250.00))
order_id = cursor.fetchone()[0]
# Create savepoint
cursor.execute('SAVEPOINT before_items')
try:
# Insert order items
items = [
(order_id, 'Laptop', 1, 999.99),
(order_id, 'Mouse', 2, 25.50)
]
for item in items:
cursor.execute('''
INSERT INTO order_items (order_id, product, quantity, price)
VALUES (%s, %s, %s, %s)
''', item)
# Commit transaction
cursor.execute('COMMIT')
except Exception as e:
# Rollback to savepoint, then continue transaction
cursor.execute('ROLLBACK TO SAVEPOINT before_items')
cursor.execute('COMMIT')
print(f"Error inserting items, but order created: {e}")
except Exception as e:
cursor.execute('ROLLBACK')
print(f"Transaction failed: {e}")MySQL is another popular relational database:
Basic MySQL Operations:
# First install: pip install mysql-connector-python
import mysql.connector
from mysql.connector import Error
# Connection configuration
config = {
'host': 'localhost',
'database': 'mydb',
'user': 'myuser',
'password': 'mypassword',
'use_pure': True
}
try:
# Connect to MySQL
conn = mysql.connector.connect(**config)
cursor = conn.cursor()
# Create table
cursor.execute('''
CREATE TABLE IF NOT EXISTS employees (
id INT AUTO_INCREMENT PRIMARY KEY,
name VARCHAR(100) NOT NULL,
position VARCHAR(50),
salary DECIMAL(10, 2),
hire_date DATE
)
''')
# Insert data
insert_query = '''
INSERT INTO employees (name, position, salary, hire_date)
VALUES (%s, %s, %s, %s)
'''
employee_data = [
('Alice Smith', 'Software Engineer', 85000, '2023-01-15'),
('Bob Johnson', 'Product Manager', 95000, '2023-02-20'),
('Carol Williams', 'Data Analyst', 70000, '2023-03-10')
]
cursor.executemany(insert_query, employee_data)
conn.commit()
print(f"Inserted {cursor.rowcount} rows")
# Query with filtering
cursor.execute('''
SELECT name, position, salary
FROM employees
WHERE salary > %s
''', (80000,))
for (name, position, salary) in cursor:
print(f"{name} - {position}: ${salary:.2f}")
# Update data
cursor.execute('''
UPDATE employees
SET salary = salary * 1.1
WHERE position = %s
''', ('Software Engineer',))
conn.commit()
print(f"Updated {cursor.rowcount} rows")
# Delete data
cursor.execute('DELETE FROM employees WHERE hire_date < %s', ('2023-02-01',))
conn.commit()
print(f"Deleted {cursor.rowcount} rows")
except Error as e:
print(f"MySQL Error: {e}")
if conn:
conn.rollback()
finally:
if conn.is_connected():
cursor.close()
conn.close()Advanced MySQL Features:
import mysql.connector
from mysql.connector import pooling
# Connection pooling
pool = mysql.connector.pooling.MySQLConnectionPool(
pool_name="mypool",
pool_size=5,
**config
)
# Get connection from pool
conn = pool.get_connection()
cursor = conn.cursor()
# Use dictionary cursor
cursor = conn.cursor(dictionary=True)
cursor.execute('SELECT * FROM employees')
rows = cursor.fetchall()
for row in rows:
print(f"{row['name']} - {row['position']}")
# Prepared statements
cursor = conn.cursor(prepared=True)
cursor.execute('SELECT * FROM employees WHERE salary > %s', (80000,))
print(cursor.fetchall())
# Stored procedures
# Create stored procedure
cursor.execute('''
CREATE PROCEDURE GetEmployeesByPosition(IN pos VARCHAR(50))
BEGIN
SELECT * FROM employees WHERE position = pos;
END
''')
# Call stored procedure
cursor.callproc('GetEmployeesByPosition', ['Software Engineer'])
for result in cursor.stored_results():
print(result.fetchall())
# Batch insert with ON DUPLICATE KEY UPDATE
employees = [
('Alice Smith', 'Senior Engineer', 95000, '2023-01-15'),
('David Brown', 'Software Engineer', 82000, '2023-04-01')
]
insert_query = '''
INSERT INTO employees (name, position, salary, hire_date)
VALUES (%s, %s, %s, %s)
ON DUPLICATE KEY UPDATE
position = VALUES(position),
salary = VALUES(salary)
'''
cursor.executemany(insert_query, employees)
conn.commit()
print(f"{cursor.rowcount} rows affected")MongoDB is a NoSQL document database:
Basic MongoDB Operations:
# First install: pip install pymongo
from pymongo import MongoClient
from pymongo.errors import ConnectionFailure, DuplicateKeyError
from datetime import datetime
import pprint
# Connect to MongoDB
client = MongoClient('mongodb://localhost:27017/')
# Select database
db = client['mydatabase']
# Select collection (like a table)
collection = db['users']
# Insert single document
user = {
'name': 'Alice',
'email': 'alice@example.com',
'age': 30,
'address': {
'street': '123 Main St',
'city': 'New York',
'zip': '10001'
},
'hobbies': ['reading', 'cycling'],
'created_at': datetime.now()
}
result = collection.insert_one(user)
print(f"Inserted ID: {result.inserted_id}")
# Insert multiple documents
users = [
{
'name': 'Bob',
'email': 'bob@example.com',
'age': 25,
'address': {'city': 'Los Angeles'},
'hobbies': ['gaming', 'hiking']
},
{
'name': 'Charlie',
'email': 'charlie@example.com',
'age': 35,
'address': {'city': 'Chicago'},
'hobbies': ['photography', 'cooking']
}
]
result = collection.insert_many(users)
print(f"Inserted IDs: {result.inserted_ids}")
# Find documents
# Find one
alice = collection.find_one({'name': 'Alice'})
pprint.pprint(alice)
# Find many
for user in collection.find({'age': {'$gt': 28}}):
print(f"{user['name']} - {user['age']}")
# Find with projection (select specific fields)
for user in collection.find(
{'age': {'$lt': 30}},
{'name': 1, 'email': 1, '_id': 0}
):
print(user)
# Count documents
count = collection.count_documents({'age': {'$gte': 25}})
print(f"Users aged 25+: {count}")
# Update documents
# Update one
collection.update_one(
{'name': 'Alice'},
{'$set': {'age': 31, 'hobbies': ['reading', 'cycling', 'yoga']}}
)
# Update many
collection.update_many(
{'age': {'$lt': 30}},
{'$inc': {'age': 1}} # Birthday increment
)
# Delete documents
# Delete one
collection.delete_one({'name': 'Charlie'})
# Delete many
collection.delete_many({'age': {'$gt': 40}})Advanced MongoDB Features:
from pymongo import MongoClient, ASCENDING, DESCENDING
from pymongo import IndexModel, TEXT
import pprint
client = MongoClient('mongodb://localhost:27017/')
db = client['mydatabase']
collection = db['products']
# Create indexes
collection.create_index([('name', ASCENDING)])
collection.create_index([('price', DESCENDING)])
collection.create_index([('category', ASCENDING), ('price', DESCENDING)])
collection.create_index([('description', TEXT)]) # Text index for search
# Compound index with unique constraint
collection.create_index(
[('sku', ASCENDING)],
unique=True
)
# Aggregation pipeline
pipeline = [
# Match stage - filter documents
{'$match': {'category': 'Electronics'}},
# Group stage - aggregate
{'$group': {
'_id': '$subcategory',
'avg_price': {'$avg': '$price'},
'min_price': {'$min': '$price'},
'max_price': {'$max': '$price'},
'count': {'$sum': 1}
}},
# Sort stage
{'$sort': {'avg_price': -1}},
# Limit stage
{'$limit': 5}
]
results = collection.aggregate(pipeline)
for result in results:
print(result)
# More complex aggregation with lookup (join)
orders = db['orders']
products = db['products']
# Sample data
orders.insert_many([
{'user_id': 1, 'product_id': 101, 'quantity': 2, 'date': datetime.now()},
{'user_id': 2, 'product_id': 102, 'quantity': 1, 'date': datetime.now()}
])
products.insert_many([
{'_id': 101, 'name': 'Laptop', 'price': 999.99},
{'_id': 102, 'name': 'Mouse', 'price': 25.50}
])
# Join orders with products
pipeline = [
{
'$lookup': {
'from': 'products',
'localField': 'product_id',
'foreignField': '_id',
'as': 'product'
}
},
{
'$unwind': '$product'
},
{
'$project': {
'user_id': 1,
'quantity': 1,
'product_name': '$product.name',
'total_price': {'$multiply': ['$quantity', '$product.price']}
}
}
]
order_details = orders.aggregate(pipeline)
for detail in order_details:
print(detail)
# Text search
collection.create_index([('description', TEXT)])
results = collection.find(
{'$text': {'$search': 'laptop computer'}},
{'score': {'$meta': 'textScore'}}
).sort([('score', {'$meta': 'textScore'})])
for doc in results:
print(f"Score: {doc['score']}, Title: {doc.get('title')}")
# Geospatial queries
locations = db['locations']
# Create 2dsphere index
locations.create_index([('location', '2dsphere')])
# Insert locations
locations.insert_many([
{'name': 'Central Park', 'location': {'type': 'Point', 'coordinates': [-73.97, 40.77]}},
{'name': 'Empire State', 'location': {'type': 'Point', 'coordinates': [-73.98, 40.75]}},
{'name': 'Brooklyn Bridge', 'location': {'type': 'Point', 'coordinates': [-74.00, 40.70]}}
])
# Find nearby locations
nearby = locations.find({
'location': {
'$near': {
'$geometry': {'type': 'Point', 'coordinates': [-73.98, 40.76]},
'$maxDistance': 5000 # 5 km
}
}
})
for loc in nearby:
print(f"{loc['name']} is nearby")SQLAlchemy provides an Object-Relational Mapping layer:
Basic SQLAlchemy Setup:
# First install: pip install sqlalchemy
from sqlalchemy import create_engine, Column, Integer, String, Float, DateTime, ForeignKey
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker, relationship
from datetime import datetime
# Create engine
engine = create_engine('sqlite:///mydatabase.db', echo=True)
# Create base class for declarative models
Base = declarative_base()
# Define models
class User(Base):
__tablename__ = 'users'
id = Column(Integer, primary_key=True)
name = Column(String(50), nullable=False)
email = Column(String(100), unique=True, nullable=False)
age = Column(Integer)
created_at = Column(DateTime, default=datetime.now)
# Relationship
orders = relationship('Order', back_populates='user')
def __repr__(self):
return f"<User(name='{self.name}', email='{self.email}')>"
class Product(Base):
__tablename__ = 'products'
id = Column(Integer, primary_key=True)
name = Column(String(100), nullable=False)
price = Column(Float, nullable=False)
stock = Column(Integer, default=0)
# Relationship
order_items = relationship('OrderItem', back_populates='product')
def __repr__(self):
return f"<Product(name='{self.name}', price={self.price})>"
class Order(Base):
__tablename__ = 'orders'
id = Column(Integer, primary_key=True)
user_id = Column(Integer, ForeignKey('users.id'))
order_date = Column(DateTime, default=datetime.now)
total = Column(Float, default=0)
# Relationships
user = relationship('User', back_populates='orders')
items = relationship('OrderItem', back_populates='order')
def __repr__(self):
return f"<Order(id={self.id}, user_id={self.user_id}, total={self.total})>"
class OrderItem(Base):
__tablename__ = 'order_items'
id = Column(Integer, primary_key=True)
order_id = Column(Integer, ForeignKey('orders.id'))
product_id = Column(Integer, ForeignKey('products.id'))
quantity = Column(Integer, nullable=False)
price = Column(Float, nullable=False) # Price at time of order
# Relationships
order = relationship('Order', back_populates='items')
product = relationship('Product', back_populates='order_items')
def __repr__(self):
return f"<OrderItem(product_id={self.product_id}, quantity={self.quantity})>"
# Create tables
Base.metadata.create_all(engine)
# Create session
Session = sessionmaker(bind=engine)
session = Session()CRUD Operations with SQLAlchemy:
# CREATE
# Create new users
user1 = User(name='Alice', email='alice@example.com', age=30)
user2 = User(name='Bob', email='bob@example.com', age=25)
user3 = User(name='Charlie', email='charlie@example.com', age=35)
# Add to session
session.add(user1)
session.add_all([user2, user3])
# Commit transaction
session.commit()
# Create products
products = [
Product(name='Laptop', price=999.99, stock=10),
Product(name='Mouse', price=25.50, stock=50),
Product(name='Keyboard', price=75.00, stock=30),
Product(name='Monitor', price=299.99, stock=15)
]
session.add_all(products)
session.commit()
# READ
# Get all users
all_users = session.query(User).all()
for user in all_users:
print(user)
# Filter users
young_users = session.query(User).filter(User.age < 30).all()
print("\nYoung users:")
for user in young_users:
print(f"{user.name}: {user.age}")
# Get single user by ID
user = session.query(User).get(1)
print(f"\nUser 1: {user}")
# Filter with multiple conditions
active_users = session.query(User).filter(
User.age >= 25,
User.age <= 35
).all()
print("\nUsers aged 25-35:")
for user in active_users:
print(f"{user.name}: {user.age}")
# Order by
users_by_age = session.query(User).order_by(User.age.desc()).all()
print("\nUsers by age (descending):")
for user in users_by_age:
print(f"{user.name}: {user.age}")
# UPDATE
# Update single user
user = session.query(User).filter_by(name='Alice').first()
user.age = 31
session.commit()
# Update multiple
session.query(User).filter(User.age < 30).update({'age': User.age + 1})
session.commit()
# DELETE
# Delete single user
user = session.query(User).filter_by(name='Charlie').first()
if user:
session.delete(user)
session.commit()
# Delete multiple
session.query(User).filter(User.age > 40).delete()
session.commit()Relationships and Joins:
# Create an order
user = session.query(User).filter_by(name='Alice').first()
product1 = session.query(Product).filter_by(name='Laptop').first()
product2 = session.query(Product).filter_by(name='Mouse').first()
# Create order
order = Order(user=user)
session.add(order)
session.flush() # Get order ID
# Add order items
item1 = OrderItem(order=order, product=product1, quantity=1, price=product1.price)
item2 = OrderItem(order=order, product=product2, quantity=2, price=product2.price)
session.add_all([item1, item2])
# Update order total
order.total = sum(item.price * item.quantity for item in [item1, item2])
session.commit()
# Query with joins
# Explicit join
orders = session.query(Order).join(User).filter(User.name == 'Alice').all()
print("\nAlice's orders:")
for order in orders:
print(f"Order {order.id}: ${order.total}")
# Join with eager loading
orders = session.query(Order).options(
joinedload(Order.items).joinedload(OrderItem.product)
).all()
for order in orders:
print(f"\nOrder {order.id} - Total: ${order.total}")
for item in order.items:
print(f" {item.quantity}x {item.product.name} @ ${item.price}")
# Aggregate queries
from sqlalchemy import func
# Count users by age group
result = session.query(
User.age,
func.count(User.id).label('count')
).group_by(User.age).all()
print("\nUsers by age:")
for age, count in result:
print(f"Age {age}: {count} users")
# Total sales by product
result = session.query(
Product.name,
func.sum(OrderItem.quantity).label('total_sold'),
func.sum(OrderItem.price * OrderItem.quantity).label('revenue')
).join(OrderItem).group_by(Product.id).all()
print("\nProduct sales:")
for name, total_sold, revenue in result:
print(f"{name}: sold {total_sold}, revenue ${revenue:.2f}")Advanced SQLAlchemy Features:
from sqlalchemy import and_, or_, not_, func, text
from sqlalchemy.orm import aliased
# Complex queries with multiple conditions
users = session.query(User).filter(
and_(
User.age >= 25,
User.age <= 35,
or_(
User.name.like('A%'),
User.name.like('B%')
)
)
).all()
# Subqueries
subquery = session.query(
Order.user_id,
func.sum(Order.total).label('total_spent')
).group_by(Order.user_id).subquery()
users_with_spending = session.query(
User,
subquery.c.total_spent
).outerjoin(subquery, User.id == subquery.c.user_id).all()
print("\nUsers with spending:")
for user, spent in users_with_spending:
print(f"{user.name}: ${spent if spent else 0}")
# Using aliases for self-joins
UserAlias = aliased(User)
users = session.query(User, UserAlias).join(
UserAlias, User.id < UserAlias.id
).filter(
User.age == UserAlias.age
).all()
print("\nUsers with same age:")
for user1, user2 in users:
print(f"{user1.name} and {user2.name} are both {user1.age}")
# Raw SQL queries
result = session.execute(
text("SELECT * FROM users WHERE age > :age"),
{'age': 25}
)
for row in result:
print(row)
# Transactions and rollbacks
try:
# Start transaction
session.begin_nested()
user = User(name='David', email='david@example.com', age=28)
session.add(user)
# This will fail (duplicate email)
user2 = User(name='David2', email='david@example.com', age=29)
session.add(user2)
session.commit()
except Exception as e:
print(f"Error: {e}")
session.rollback()
print("Transaction rolled back")
# Bulk operations
# Bulk insert
users_data = [
{'name': f'User{i}', 'email': f'user{i}@example.com', 'age': 20 + i}
for i in range(10)
]
session.bulk_insert_mappings(User, users_data)
session.commit()
# Bulk update
session.bulk_update_mappings(User, [
{'id': 1, 'age': 32},
{'id': 2, 'age': 26}
])
session.commit()Transaction management ensures data integrity:
Transaction Concepts:
import sqlite3
from contextlib import contextmanager
@contextmanager
def transaction(connection):
"""Context manager for database transactions."""
try:
yield connection
connection.commit()
print("Transaction committed")
except Exception as e:
connection.rollback()
print(f"Transaction rolled back: {e}")
raise
# Example with SQLite
conn = sqlite3.connect('bank.db')
conn.execute('''
CREATE TABLE IF NOT EXISTS accounts (
id INTEGER PRIMARY KEY,
name TEXT,
balance REAL
)
''')
# Insert sample data
conn.execute("INSERT OR IGNORE INTO accounts (id, name, balance) VALUES (1, 'Alice', 1000)")
conn.execute("INSERT OR IGNORE INTO accounts (id, name, balance) VALUES (2, 'Bob', 500)")
conn.commit()
def transfer_funds(conn, from_id, to_id, amount):
"""Transfer funds between accounts with transaction."""
with transaction(conn):
# Check sender balance
cursor = conn.execute(
"SELECT balance FROM accounts WHERE id = ?",
(from_id,)
)
balance = cursor.fetchone()[0]
if balance < amount:
raise ValueError(f"Insufficient funds: available {balance}, needed {amount}")
# Withdraw from sender
conn.execute(
"UPDATE accounts SET balance = balance - ? WHERE id = ?",
(amount, from_id)
)
# Deposit to receiver
conn.execute(
"UPDATE accounts SET balance = balance + ? WHERE id = ?",
(amount, to_id)
)
# Log transaction
conn.execute('''
INSERT INTO transactions (from_id, to_id, amount, timestamp)
VALUES (?, ?, ?, datetime('now'))
''', (from_id, to_id, amount))
# Test successful transfer
try:
transfer_funds(conn, 1, 2, 200)
print("Transfer successful")
except ValueError as e:
print(f"Transfer failed: {e}")
# Test failed transfer
try:
transfer_funds(conn, 1, 2, 2000) # Insufficient funds
except ValueError as e:
print(f"Transfer failed: {e}")
# Check final balances
cursor = conn.execute("SELECT * FROM accounts")
for row in cursor:
print(f"Account {row[0]}: {row[1]} - ${row[2]}")ACID Properties with PostgreSQL:
import psycopg2
from psycopg2 import extensions, sql
# Set isolation level
conn = psycopg2.connect(
host='localhost',
database='testdb',
user='testuser',
password='testpass'
)
# Set isolation level to SERIALIZABLE for highest consistency
conn.set_isolation_level(extensions.ISOLATION_LEVEL_SERIALIZABLE)
def perform_transfer(conn, from_acct, to_acct, amount):
"""Perform transfer with proper transaction handling."""
cursor = conn.cursor()
try:
# Start transaction
cursor.execute("BEGIN")
# Lock accounts in consistent order to prevent deadlock
accounts = sorted([from_acct, to_acct])
for acct in accounts:
cursor.execute(
"SELECT * FROM accounts WHERE id = %s FOR UPDATE",
(acct,)
)
# Check balance
cursor.execute(
"SELECT balance FROM accounts WHERE id = %s",
(from_acct,)
)
balance = cursor.fetchone()[0]
if balance < amount:
raise ValueError("Insufficient funds")
# Perform transfer
cursor.execute(
"UPDATE accounts SET balance = balance - %s WHERE id = %s",
(amount, from_acct)
)
cursor.execute(
"UPDATE accounts SET balance = balance + %s WHERE id = %s",
(amount, to_acct)
)
# Log transaction
cursor.execute('''
INSERT INTO transaction_log (from_id, to_id, amount, status)
VALUES (%s, %s, %s, 'completed')
''', (from_acct, to_acct, amount))
# Commit transaction
conn.commit()
print("Transfer completed successfully")
except Exception as e:
conn.rollback()
print(f"Transfer failed, rolled back: {e}")
# Log failure
cursor.execute('''
INSERT INTO transaction_log (from_id, to_id, amount, status)
VALUES (%s, %s, %s, 'failed')
''', (from_acct, to_acct, amount))
conn.commit()
raise
finally:
cursor.close()
# Test concurrent transfers
import threading
import time
def concurrent_transfer(conn_params, from_id, to_id, amount):
"""Perform transfer in separate thread."""
conn = psycopg2.connect(**conn_params)
try:
perform_transfer(conn, from_id, to_id, amount)
finally:
conn.close()
# Create multiple threads for concurrent transfers
threads = []
for i in range(5):
t = threading.Thread(
target=concurrent_transfer,
args=(conn_params, 1, 2, 100)
)
threads.append(t)
t.start()
for t in threads:
t.join()Multithreading allows multiple threads of execution to run within a single process, sharing the same memory space. This is particularly useful for I/O-bound operations where threads can wait for I/O while others execute.
The threading module provides a high-level interface for working with threads:
Basic Thread Creation:
import threading
import time
def worker(name, delay):
"""Simple worker function that runs in a thread."""
print(f"Worker {name} starting")
for i in range(5):
time.sleep(delay)
print(f"Worker {name}: step {i}")
print(f"Worker {name} finished")
# Create threads
thread1 = threading.Thread(target=worker, args=("A", 0.5))
thread2 = threading.Thread(target=worker, args=("B", 0.3))
# Start threads
thread1.start()
thread2.start()
# Wait for threads to complete
thread1.join()
thread2.join()
print("All threads completed")Thread Class Subclassing:
import threading
import time
class WorkerThread(threading.Thread):
"""Custom thread class."""
def __init__(self, name, delay):
super().__init__()
self.name = name
self.delay = delay
self.result = None
def run(self):
"""Called when thread starts."""
print(f"Thread {self.name} starting")
total = 0
for i in range(5):
time.sleep(self.delay)
total += i
print(f"Thread {self.name}: computed {total}")
self.result = total
print(f"Thread {self.name} finished")
# Create and start threads
threads = []
for i in range(3):
thread = WorkerThread(f"Worker-{i}", 0.2)
threads.append(thread)
thread.start()
# Wait for all threads
for thread in threads:
thread.join()
print(f"Thread {thread.name} result: {thread.result}")Thread Identification and Properties:
import threading
import time
def thread_info():
"""Display current thread information."""
current = threading.current_thread()
print(f"Thread: {current.name}")
print(f" ID: {current.ident}")
print(f" Alive: {current.is_alive()}")
print(f" Daemon: {current.daemon}")
print(f" Native ID: {threading.get_native_id()}")
def worker():
thread_info()
time.sleep(1)
# Main thread info
print("Main thread:")
thread_info()
# Create daemon thread
daemon_thread = threading.Thread(target=worker, name="DaemonWorker", daemon=True)
daemon_thread.start()
# Regular thread
regular_thread = threading.Thread(target=worker, name="RegularWorker")
regular_thread.start()
regular_thread.join()
# Daemon thread will be terminated when main thread exitsThread Pools with concurrent.futures:
import concurrent.futures
import time
import urllib.request
def download_url(url):
"""Download a URL and return its length."""
print(f"Downloading {url}")
with urllib.request.urlopen(url) as response:
content = response.read()
return len(content)
# Using ThreadPoolExecutor
urls = [
'http://www.example.com',
'http://www.python.org',
'http://www.github.com',
'http://www.stackoverflow.com'
]
with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
# Submit individual tasks
futures = [executor.submit(download_url, url) for url in urls]
# Process results as they complete
for future in concurrent.futures.as_completed(futures):
try:
result = future.result(timeout=5)
print(f"Downloaded {result} bytes")
except Exception as e:
print(f"Error: {e}")
# Map method
with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
results = executor.map(download_url, urls)
for url, size in zip(urls, results):
print(f"{url}: {size} bytes")The Global Interpreter Lock (GIL) is a mutex that protects access to Python objects, preventing multiple threads from executing Python bytecode simultaneously:
GIL Demonstration:
import threading
import time
import sys
def cpu_intensive_task(name):
"""CPU-intensive task that demonstrates GIL limitations."""
print(f"Task {name} starting")
count = 0
for i in range(50_000_000):
count += i
print(f"Task {name} finished: {count}")
def io_bound_task(name):
"""I/O-bound task that releases the GIL."""
print(f"I/O Task {name} starting")
time.sleep(2) # Simulates I/O operation (releases GIL)
print(f"I/O Task {name} finished")
# CPU-bound tasks - GIL prevents parallel execution
print("CPU-bound tasks (GIL limited):")
start = time.time()
threads = []
for i in range(4):
t = threading.Thread(target=cpu_intensive_task, args=(f"CPU-{i}",))
threads.append(t)
t.start()
for t in threads:
t.join()
cpu_time = time.time() - start
print(f"CPU-bound time: {cpu_time:.2f} seconds\n")
# I/O-bound tasks - GIL is released during I/O
print("I/O-bound tasks (GIL released):")
start = time.time()
threads = []
for i in range(4):
t = threading.Thread(target=io_bound_task, args=(f"IO-{i}",))
threads.append(t)
t.start()
for t in threads:
t.join()
io_time = time.time() - start
print(f"I/O-bound time: {io_time:.2f} seconds")Working Around the GIL:
import multiprocessing
import threading
import time
import math
# Using multiprocessing for CPU-bound tasks
def cpu_bound_work(n):
"""CPU-intensive calculation."""
return sum(math.sqrt(i) for i in range(n))
def parallel_process():
"""Use multiprocessing for true parallelism."""
numbers = [10_000_000, 10_000_000, 10_000_000, 10_000_000]
start = time.time()
with multiprocessing.Pool(processes=4) as pool:
results = pool.map(cpu_bound_work, numbers)
process_time = time.time() - start
print(f"Multiprocessing time: {process_time:.2f} seconds")
return results
def parallel_thread():
"""Use threading (GIL-limited)."""
numbers = [10_000_000, 10_000_000, 10_000_000, 10_000_000]
results = []
def worker(n, results_list):
results_list.append(cpu_bound_work(n))
threads = []
start = time.time()
for n in numbers:
t = threading.Thread(target=worker, args=(n, results))
threads.append(t)
t.start()
for t in threads:
t.join()
thread_time = time.time() - start
print(f"Threading time: {thread_time:.2f} seconds")
return results
# Compare approaches
print("Multiprocessing (true parallelism):")
process_results = parallel_process()
print("\nThreading (GIL limited):")
thread_results = parallel_thread()
# Using C extensions that release GIL
import numpy as np
def numpy_parallel():
"""NumPy operations release GIL."""
size = 10_000_000
arrays = [np.random.rand(size) for _ in range(4)]
start = time.time()
def process_array(arr):
return np.mean(arr)
with concurrent.futures.ThreadPoolExecutor(max_workers=4) as executor:
results = list(executor.map(process_array, arrays))
numpy_time = time.time() - start
print(f"NumPy (releases GIL): {numpy_time:.2f} seconds")
return results
numpy_parallel()Ensuring thread safety when multiple threads access shared data:
Race Conditions:
import threading
# Race condition example
counter = 0
def increment():
global counter
for _ in range(100000):
current = counter
# Simulate context switch
counter = current + 1
# Create threads
threads = []
for _ in range(5):
t = threading.Thread(target=increment)
threads.append(t)
t.start()
for t in threads:
t.join()
print(f"Expected: 500000, Actual: {counter}") # Will be less due to race conditionsUsing Locks:
import threading
# Thread-safe counter with lock
class Counter:
def __init__(self):
self.value = 0
self.lock = threading.Lock()
def increment(self):
with self.lock: # Acquire lock automatically
self.value += 1
def decrement(self):
with self.lock:
self.value -= 1
def get_value(self):
with self.lock:
return self.value
counter = Counter()
def worker():
for _ in range(100000):
counter.increment()
threads = []
for _ in range(5):
t = threading.Thread(target=worker)
threads.append(t)
t.start()
for t in threads:
t.join()
print(f"Thread-safe counter: {counter.get_value()}") # Correct: 500000RLock (Reentrant Lock):
import threading
class ReentrantExample:
def __init__(self):
self.lock = threading.RLock()
self.data = {}
def update(self, key, value):
with self.lock:
# This method can call other methods that use the same lock
self._validate(key, value)
self.data[key] = value
def _validate(self, key, value):
with self.lock: # Same thread can acquire RLock again
if key in self.data:
print(f"Updating existing key: {key}")
if value < 0:
raise ValueError("Value cannot be negative")
# Without RLock, this would deadlock
example = ReentrantExample()
example.update("count", 42)Semaphores:
import threading
import time
import random
class ConnectionPool:
"""Thread-safe connection pool using Semaphore."""
def __init__(self, max_connections):
self.semaphore = threading.Semaphore(max_connections)
self.connections = []
self.lock = threading.Lock()
def get_connection(self):
"""Get a connection from the pool."""
self.semaphore.acquire()
with self.lock:
conn = f"Connection-{len(self.connections)}"
self.connections.append(conn)
print(f"Acquired {conn}")
return conn
def release_connection(self, conn):
"""Release connection back to pool."""
with self.lock:
print(f"Released {conn}")
self.semaphore.release()
def worker(pool, worker_id):
"""Simulate worker using database connection."""
conn = pool.get_connection()
print(f"Worker {worker_id} using {conn}")
time.sleep(random.uniform(0.5, 2)) # Simulate work
pool.release_connection(conn)
# Create connection pool with max 3 connections
pool = ConnectionPool(3)
# Create 10 workers
threads = []
for i in range(10):
t = threading.Thread(target=worker, args=(pool, i))
threads.append(t)
t.start()
for t in threads:
t.join()Thread-Safe Queue:
import threading
import queue
import time
import random
# Producer-Consumer pattern with Queue
def producer(q, items):
"""Produce items and put them in queue."""
for item in items:
print(f"Producing {item}")
time.sleep(random.uniform(0.1, 0.5))
q.put(item)
# Signal completion
q.put(None)
def consumer(q, name):
"""Consume items from queue."""
while True:
item = q.get()
if item is None:
q.put(None) # Pass termination signal to other consumers
break
print(f"Consumer {name} processing {item}")
time.sleep(random.uniform(0.2, 0.8))
q.task_done()
# Create thread-safe queue
q = queue.Queue(maxsize=5)
# Create producer
items = [f"Task-{i}" for i in range(10)]
producer_thread = threading.Thread(target=producer, args=(q, items))
# Create consumers
consumers = []
for i in range(3):
t = threading.Thread(target=consumer, args=(q, i))
consumers.append(t)
t.start()
producer_thread.start()
producer_thread.join()
# Wait for all consumers to finish
for t in consumers:
t.join()Thread-Local Data:
import threading
import random
# Thread-local storage
thread_local = threading.local()
def set_session(user_id):
"""Set session data for current thread."""
thread_local.user_id = user_id
thread_local.session_id = f"session_{user_id}_{random.randint(1000, 9999)}"
def process_request():
"""Process request using thread-local data."""
try:
user_id = thread_local.user_id
session = thread_local.session_id
print(f"Thread {threading.current_thread().name}: User {user_id}, Session {session}")
except AttributeError:
print(f"Thread {threading.current_thread().name}: No session data")
def worker(user_id):
"""Worker thread function."""
set_session(user_id)
process_request()
# Each thread has its own session data
# Create threads with different user IDs
threads = []
for i in range(5):
t = threading.Thread(target=worker, args=(100 + i,), name=f"Worker-{i}")
threads.append(t)
t.start()
for t in threads:
t.join()Condition Variables:
import threading
import time
import random
class TaskQueue:
"""Task queue with condition variable for coordination."""
def __init__(self):
self.tasks = []
self.condition = threading.Condition()
def add_task(self, task):
"""Add task to queue."""
with self.condition:
self.tasks.append(task)
print(f"Added task: {task}")
self.condition.notify() # Notify one waiting thread
def get_task(self):
"""Get task from queue (block if empty)."""
with self.condition:
while not self.tasks:
print(f"Thread {threading.current_thread().name} waiting for task")
self.condition.wait() # Release lock and wait
task = self.tasks.pop(0)
print(f"Thread {threading.current_thread().name} got task: {task}")
return task
def producer(queue, tasks):
"""Producer thread."""
for task in tasks:
time.sleep(random.uniform(0.5, 1.5))
queue.add_task(task)
def consumer(queue):
"""Consumer thread."""
while True:
task = queue.get_task()
if task == "STOP":
break
# Process task
time.sleep(random.uniform(1, 2))
print(f"Completed: {task}")
# Create queue
queue = TaskQueue()
# Create producer
tasks = [f"Task-{i}" for i in range(5)] + ["STOP"]
producer_thread = threading.Thread(target=producer, args=(queue, tasks))
# Create consumers
consumers = []
for i in range(3):
t = threading.Thread(target=consumer, args=(queue,), name=f"Consumer-{i}")
consumers.append(t)
t.start()
producer_thread.start()
producer_thread.join()
for t in consumers:
t.join()Barrier for Synchronization:
import threading
import time
def worker(barrier, name):
"""Worker that waits at barrier."""
print(f"Worker {name} starting phase 1")
time.sleep(name * 0.5) # Simulate work
print(f"Worker {name} waiting at barrier")
# Wait for all threads to reach barrier
barrier.wait()
print(f"Worker {name} starting phase 2")
time.sleep(1)
print(f"Worker {name} finished")
# Create barrier for 3 threads
barrier = threading.Barrier(3, timeout=10)
# Create threads with different work times
threads = []
for i in range(3):
t = threading.Thread(target=worker, args=(barrier, i))
threads.append(t)
t.start()
for t in threads:
t.join()
print("All threads completed both phases")Event Objects:
import threading
import time
class DataProcessor:
def __init__(self):
self.data_ready = threading.Event()
self.data = None
def prepare_data(self):
"""Prepare data in background."""
print("Preparing data...")
time.sleep(3) # Simulate data preparation
self.data = [1, 2, 3, 4, 5]
print("Data ready!")
self.data_ready.set() # Signal data is ready
def process_data(self):
"""Process data (waits for it to be ready)."""
print("Waiting for data...")
self.data_ready.wait() # Wait for signal
print(f"Processing data: {self.data}")
return sum(self.data)
# Create processor
processor = DataProcessor()
# Create threads
preparer = threading.Thread(target=processor.prepare_data)
processor_thread = threading.Thread(target=processor.process_data)
# Start threads
processor_thread.start()
preparer.start()
preparer.join()
processor_thread.join()Multiprocessing bypasses the GIL by using separate processes with their own memory space, enabling true parallelism for CPU-bound tasks.
Basic Process Creation:
import multiprocessing
import os
import time
def worker(name, delay):
"""Worker function that runs in separate process."""
print(f"Worker {name} starting (PID: {os.getpid()})")
time.sleep(delay)
print(f"Worker {name} finished")
return f"Result from {name}"
if __name__ == "__main__":
# Create processes
processes = []
for i in range(4):
p = multiprocessing.Process(target=worker, args=(f"Process-{i}", i))
processes.append(p)
p.start()
# Wait for all processes
for p in processes:
p.join()
print(f"Process {p.pid} finished")
print("All processes completed")Process Class Subclassing:
import multiprocessing
import time
class WorkerProcess(multiprocessing.Process):
def __init__(self, name, workload):
super().__init__()
self.name = name
self.workload = workload
self.result = None
def run(self):
"""Called when process starts."""
print(f"Process {self.name} starting (PID: {self.pid})")
total = 0
for i in range(self.workload):
total += i ** 2
self.result = total
print(f"Process {self.name} finished")
if __name__ == "__main__":
processes = []
for i in range(4):
p = WorkerProcess(f"Worker-{i}", 10_000_000)
processes.append(p)
p.start()
for p in processes:
p.join()
print(f"Process {p.name} result: {p.result}")Process pools manage a pool of worker processes for parallel execution:
Basic Pool Usage:
import multiprocessing
import time
import math
def is_prime(n):
"""Check if number is prime."""
if n < 2:
return False
for i in range(2, int(math.sqrt(n)) + 1):
if n % i == 0:
return False
return True
def check_prime(n):
"""Check prime and return result."""
result = is_prime(n)
return (n, result)
if __name__ == "__main__":
numbers = [i for i in range(10_000_000, 10_000_100)]
# Sequential processing
start = time.time()
sequential_results = [check_prime(n) for n in numbers]
sequential_time = time.time() - start
print(f"Sequential time: {sequential_time:.2f} seconds")
# Parallel processing with Pool
start = time.time()
with multiprocessing.Pool(processes=4) as pool:
parallel_results = pool.map(check_prime, numbers)
parallel_time = time.time() - start
print(f"Parallel time: {parallel_time:.2f} seconds")
print(f"Speedup: {sequential_time / parallel_time:.2f}x")
# Show some results
for n, is_p in parallel_results[:10]:
print(f"{n}: {'prime' if is_p else 'not prime'}")Pool Methods:
import multiprocessing
import time
def square(x):
"""Simple square function."""
time.sleep(0.1) # Simulate work
return x * x
def cube(x):
"""Cube function."""
time.sleep(0.1)
return x ** 3
if __name__ == "__main__":
with multiprocessing.Pool(processes=4) as pool:
numbers = list(range(20))
# map - block until all results ready
squares = pool.map(square, numbers)
print(f"Map results: {squares[:5]}...")
# map_async - non-blocking
result_async = pool.map_async(cube, numbers)
print("Waiting for async results...")
cubes = result_async.get(timeout=5)
print(f"Async results: {cubes[:5]}...")
# apply - run single function
result = pool.apply(square, (10,))
print(f"Apply result: {result}")
# apply_async - non-blocking single function
result_async = pool.apply_async(cube, (10,))
result = result_async.get()
print(f"Apply async result: {result}")
# starmap - for multiple arguments
pairs = [(i, i+1) for i in range(5)]
def multiply(a, b):
return a * b
products = pool.starmap(multiply, pairs)
print(f"Starmap results: {products}")
# imap - lazy iterator (preserves order)
for result in pool.imap(square, range(10)):
print(f"IMap result: {result}")
# imap_unordered - results as they complete
results = []
for result in pool.imap_unordered(square, range(10)):
results.append(result)
print(f"Unordered result: {result}")Parallel Data Processing:
import multiprocessing
import time
from functools import partial
def process_chunk(chunk, multiplier):
"""Process a chunk of data."""
return [x * multiplier for x in chunk]
def parallel_map_reduce():
"""Parallel map-reduce example."""
# Generate large dataset
data = list(range(10_000_000))
# Split into chunks for parallel processing
num_chunks = multiprocessing.cpu_count()
chunk_size = len(data) // num_chunks
chunks = [data[i:i + chunk_size] for i in range(0, len(data), chunk_size)]
# Parallel map
start = time.time()
with multiprocessing.Pool() as pool:
# Process chunks in parallel
processed_chunks = pool.map(partial(process_chunk, multiplier=2), chunks)
# Reduce - combine results
result = []
for chunk in processed_chunks:
result.extend(chunk)
parallel_time = time.time() - start
print(f"Parallel processing time: {parallel_time:.2f} seconds")
print(f"Result size: {len(result)}")
# Sequential for comparison
start = time.time()
sequential_result = [x * 2 for x in data]
sequential_time = time.time() - start
print(f"Sequential time: {sequential_time:.2f} seconds")
print(f"Speedup: {sequential_time / parallel_time:.2f}x")
if __name__ == "__main__":
parallel_map_reduce()Processes have separate memory spaces, so sharing data requires special mechanisms:
Value and Array:
import multiprocessing
import time
def increment_counter(counter, lock, increments):
"""Increment shared counter."""
for _ in range(increments):
with lock:
counter.value += 1
def fill_array(arr, lock, start, end):
"""Fill portion of shared array."""
with lock:
for i in range(start, end):
arr[i] = i * i
if __name__ == "__main__":
# Shared Value
counter = multiprocessing.Value('i', 0) # 'i' for integer
lock = multiprocessing.Lock()
processes = []
for i in range(4):
p = multiprocessing.Process(target=increment_counter,
args=(counter, lock, 100000))
processes.append(p)
p.start()
for p in processes:
p.join()
print(f"Final counter value: {counter.value}") # Should be 400000
# Shared Array
arr = multiprocessing.Array('i', 100) # Array of 100 integers
processes = []
chunk_size = 25
for i in range(0, 100, chunk_size):
p = multiprocessing.Process(target=fill_array,
args=(arr, lock, i, i + chunk_size))
processes.append(p)
p.start()
for p in processes:
p.join()
print(f"Array first 10 elements: {arr[:10]}")Manager for Complex Data:
import multiprocessing
from multiprocessing import Manager
import time
def worker_process(shared_dict, shared_list, lock, worker_id):
"""Worker process that manipulates shared data."""
with lock:
# Add to shared dictionary
shared_dict[f"worker_{worker_id}"] = worker_id * 10
# Add to shared list
shared_list.append(f"Result from worker {worker_id}")
# Simulate work
time.sleep(worker_id)
return f"Worker {worker_id} done"
if __name__ == "__main__":
# Create manager for shared complex data
with Manager() as manager:
# Create shared structures
shared_dict = manager.dict()
shared_list = manager.list()
lock = manager.Lock()
# Create processes
processes = []
for i in range(5):
p = multiprocessing.Process(target=worker_process,
args=(shared_dict, shared_list, lock, i))
processes.append(p)
p.start()
# Wait for completion
for p in processes:
p.join()
# Print results
print("Shared dictionary:")
for key, value in shared_dict.items():
print(f" {key}: {value}")
print("\nShared list:")
for item in shared_list:
print(f" {item}")Shared Memory with NumPy:
import multiprocessing
import numpy as np
import ctypes
def worker_process(shared_array, shape, dtype, worker_id):
"""Process that works on shared numpy array."""
# Create numpy array from shared memory
arr = np.frombuffer(shared_array.get_obj(), dtype=dtype).reshape(shape)
# Modify a portion of the array
chunk_size = shape[0] // 4
start = worker_id * chunk_size
end = (worker_id + 1) * chunk_size if worker_id < 3 else shape[0]
arr[start:end] += worker_id * 10
print(f"Worker {worker_id} processed rows {start}:{end}")
if __name__ == "__main__":
# Create shared array
shape = (100, 100)
dtype = np.float64
size = int(np.prod(shape))
# Create shared memory
shared_array = multiprocessing.Array(ctypes.c_double, size)
# Create numpy array from shared memory
arr = np.frombuffer(shared_array.get_obj(), dtype=dtype).reshape(shape)
arr.fill(0) # Initialize
# Create processes
processes = []
for i in range(4):
p = multiprocessing.Process(target=worker_process,
args=(shared_array, shape, dtype, i))
processes.append(p)
p.start()
for p in processes:
p.join()
# Check results
print(f"Array statistics: min={arr.min()}, max={arr.max()}, mean={arr.mean():.2f}")Process Pools with Shared Memory:
import multiprocessing
from functools import partial
import numpy as np
def process_row(row_data, multiplier):
"""Process a single row of data."""
row, row_index = row_data
return row_index, row * multiplier
def parallel_matrix_operation():
"""Parallel matrix operation using shared memory."""
# Create large matrix
rows, cols = 1000, 1000
matrix = np.random.rand(rows, cols)
# Split into rows for parallel processing
rows_data = [(matrix[i], i) for i in range(rows)]
with multiprocessing.Pool(processes=multiprocessing.cpu_count()) as pool:
# Process rows in parallel
results = pool.map(partial(process_row, multiplier=2), rows_data)
# Reconstruct matrix
result_matrix = np.zeros_like(matrix)
for row_index, processed_row in results:
result_matrix[row_index] = processed_row
# Verify
print(f"Original matrix: {matrix[0, :5]}")
print(f"Processed matrix: {result_matrix[0, :5]}")
print(f"All close: {np.allclose(result_matrix, matrix * 2)}")
if __name__ == "__main__":
parallel_matrix_operation()Asynchronous programming allows handling many concurrent operations efficiently, especially for I/O-bound tasks.
Event Loop and Coroutines:
import asyncio
import time
async def say_hello(name, delay):
"""Async function (coroutine)."""
await asyncio.sleep(delay)
print(f"Hello, {name}!")
return f"Greeted {name}"
async def main():
"""Main coroutine."""
print("Starting async tasks")
# Run coroutines sequentially
await say_hello("Alice", 2)
await say_hello("Bob", 1)
print("Sequential done")
# Run coroutines concurrently
task1 = asyncio.create_task(say_hello("Charlie", 2))
task2 = asyncio.create_task(say_hello("Diana", 1))
# Wait for both
await task1
await task2
print("Concurrent done")
# Gather results
results = await asyncio.gather(
say_hello("Eve", 2),
say_hello("Frank", 1),
say_hello("Grace", 1.5)
)
print(f"Gather results: {results}")
# Run the async program
asyncio.run(main())Async Context Managers:
import asyncio
class AsyncResource:
"""Async context manager example."""
async def __aenter__(self):
print("Acquiring resource")
await asyncio.sleep(1)
self.resource = "Resource acquired"
return self
async def __aexit__(self, exc_type, exc_val, exc_tb):
print("Releasing resource")
await asyncio.sleep(1)
self.resource = None
async def use(self):
"""Use the resource."""
print(f"Using: {self.resource}")
await asyncio.sleep(0.5)
async def main():
# Using async context manager
async with AsyncResource() as resource:
await resource.use()
await resource.use()
print("Context exited")
asyncio.run(main())Understanding Async/Await:
import asyncio
import random
async def fetch_data(url):
"""Simulate fetching data from URL."""
print(f"Fetching {url}...")
delay = random.uniform(0.5, 2)
await asyncio.sleep(delay)
data = f"Data from {url} (took {delay:.2f}s)"
print(f"Completed {url}")
return data
async def process_data(data):
"""Simulate processing data."""
print(f"Processing: {data[:20]}...")
await asyncio.sleep(0.5)
return f"Processed: {data}"
async def main():
urls = [
"http://api1.example.com",
"http://api2.example.com",
"http://api3.example.com",
"http://api4.example.com"
]
# Fetch all URLs concurrently
fetch_tasks = [asyncio.create_task(fetch_data(url)) for url in urls]
fetched_data = await asyncio.gather(*fetch_tasks)
# Process results
process_tasks = [asyncio.create_task(process_data(data)) for data in fetched_data]
results = await asyncio.gather(*process_tasks)
for result in results:
print(result)
asyncio.run(main())Async Generators:
import asyncio
async def async_range(start, stop, delay=0.5):
"""Async generator that yields numbers with delay."""
for i in range(start, stop):
await asyncio.sleep(delay)
yield i
async def fibonacci(limit):
"""Async Fibonacci generator."""
a, b = 0, 1
while a < limit:
await asyncio.sleep(0.2)
yield a
a, b = b, a + b
async def main():
# Using async for loop
print("Async range:")
async for num in async_range(0, 5):
print(f" {num}")
print("\nFibonacci sequence:")
async for num in fibonacci(20):
print(f" {num}")
# Collect async generator results
fib_numbers = [num async for num in fibonacci(30)]
print(f"\nFibonacci list: {fib_numbers}")
asyncio.run(main())Event Loop Management:
import asyncio
import threading
import time
def print_event_loop_info():
"""Print current event loop information."""
try:
loop = asyncio.get_running_loop()
print(f"Event loop running: {loop}")
print(f" Thread: {threading.current_thread().name}")
print(f" Debug mode: {loop.get_debug()}")
except RuntimeError:
print("No running event loop")
async def task(name, duration):
"""Async task."""
print(f"Task {name} starting")
await asyncio.sleep(duration)
print(f"Task {name} finished")
return f"Result from {name}"
async def main():
print_event_loop_info()
# Create multiple tasks
tasks = [
asyncio.create_task(task("A", 2)),
asyncio.create_task(task("B", 1)),
asyncio.create_task(task("C", 1.5))
]
# Get current time
loop = asyncio.get_running_loop()
start = loop.time()
# Wait for first task to complete
done, pending = await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED)
print(f"\nFirst completed after {loop.time() - start:.2f}s")
# Cancel remaining tasks
for task in pending:
task.cancel()
# Wait for cancellations
await asyncio.gather(*pending, return_exceptions=True)
print_event_loop_info()
asyncio.run(main())Custom Event Loop Policy:
import asyncio
import threading
class ThreadSafeEventLoopPolicy(asyncio.DefaultEventLoopPolicy):
"""Custom event loop policy that creates event loops per thread."""
def get_event_loop(self):
"""Get or create event loop for current thread."""
try:
return super().get_event_loop()
except RuntimeError:
loop = self.new_event_loop()
self.set_event_loop(loop)
return loop
async def worker_task(name):
"""Worker task."""
print(f"Worker {name} running in {threading.current_thread().name}")
await asyncio.sleep(1)
return f"Worker {name} done"
def run_worker(worker_id):
"""Run worker in separate thread with its own event loop."""
# Set custom policy for this thread
asyncio.set_event_loop_policy(ThreadSafeEventLoopPolicy())
async def main():
result = await worker_task(worker_id)
print(result)
asyncio.run(main())
# Create threads with their own event loops
threads = []
for i in range(3):
t = threading.Thread(target=run_worker, args=(i,))
threads.append(t)
t.start()
for t in threads:
t.join()Async HTTP client/server library:
Async HTTP Client:
import asyncio
import aiohttp
import time
async def fetch_url(session, url):
"""Fetch a URL asynchronously."""
try:
async with session.get(url) as response:
# Read response text
text = await response.text()
return {
'url': url,
'status': response.status,
'size': len(text),
'encoding': response.get_encoding()
}
except Exception as e:
return {
'url': url,
'error': str(e)
}
async def fetch_multiple_urls():
"""Fetch multiple URLs concurrently."""
urls = [
'http://python.org',
'http://github.com',
'http://stackoverflow.com',
'http://example.com',
'http://google.com'
]
# Create session with custom settings
timeout = aiohttp.ClientTimeout(total=10)
connector = aiohttp.TCPConnector(limit=10, ttl_dns_cache=300)
async with aiohttp.ClientSession(
timeout=timeout,
connector=connector,
headers={'User-Agent': 'AsyncBot/1.0'}
) as session:
# Create tasks for all URLs
tasks = [fetch_url(session, url) for url in urls]
# Gather results
results = await asyncio.gather(*tasks)
# Process results
for result in results:
if 'error' in result:
print(f"❌ {result['url']}: {result['error']}")
else:
print(f"✅ {result['url']}: {result['status']} - {result['size']} bytes")
async def main():
start = time.time()
await fetch_multiple_urls()
elapsed = time.time() - start
print(f"\nTotal time: {elapsed:.2f} seconds")
asyncio.run(main())Async File Downloader:
import asyncio
import aiohttp
import aiofiles
import os
from urllib.parse import urlparse
async def download_file(session, url, output_dir="downloads"):
"""Download a file asynchronously."""
# Extract filename from URL
filename = os.path.basename(urlparse(url).path)
if not filename:
filename = "index.html"
filepath = os.path.join(output_dir, filename)
try:
async with session.get(url) as response:
if response.status == 200:
# Stream download to file
async with aiofiles.open(filepath, 'wb') as f:
async for chunk in response.content.iter_chunked(8192):
await f.write(chunk)
size = os.path.getsize(filepath)
print(f"Downloaded {filename}: {size} bytes")
return filepath
else:
print(f"Failed to download {url}: {response.status}")
return None
except Exception as e:
print(f"Error downloading {url}: {e}")
return None
async def download_files(urls, max_concurrent=5):
"""Download multiple files concurrently."""
# Create download directory
os.makedirs("downloads", exist_ok=True)
# Configure connection limits
connector = aiohttp.TCPConnector(limit=max_concurrent, limit_per_host=2)
async with aiohttp.ClientSession(connector=connector) as session:
# Create tasks with semaphore to limit concurrency
semaphore = asyncio.Semaphore(max_concurrent)
async def bounded_download(url):
async with semaphore:
return await download_file(session, url)
tasks = [bounded_download(url) for url in urls]
results = await asyncio.gather(*tasks)
return [r for r in results if r is not None]
async def main():
urls = [
'http://python.org/',
'http://github.com/',
'http://google.com/',
'http://stackoverflow.com/',
'http://example.com/',
'http://wikipedia.org/'
]
start = time.time()
downloaded = await download_files(urls, max_concurrent=3)
elapsed = time.time() - start
print(f"\nDownloaded {len(downloaded)} files in {elapsed:.2f} seconds")
for filepath in downloaded:
print(f" - {filepath}")
asyncio.run(main())Async Web Server with aiohttp:
from aiohttp import web
import asyncio
import json
# Request handlers
async def handle_root(request):
"""Root endpoint."""
return web.Response(
text="<h1>Async Web Server</h1><p>Welcome!</p>",
content_type='text/html'
)
async def handle_api(request):
"""API endpoint returning JSON."""
data = {
'status': 'success',
'message': 'Hello from async API',
'method': request.method,
'path': request.path,
'headers': dict(request.headers)
}
return web.json_response(data)
async def handle_greet(request):
"""Greeting endpoint with path parameter."""
name = request.match_info.get('name', 'Anonymous')
return web.json_response({
'greeting': f'Hello, {name}!',
'timestamp': asyncio.get_event_loop().time()
})
async def handle_echo(request):
"""Echo POST data."""
try:
data = await request.json()
return web.json_response({
'echo': data,
'received': True
})
except json.JSONDecodeError:
return web.json_response(
{'error': 'Invalid JSON'},
status=400
)
async def handle_websocket(request):
"""WebSocket endpoint."""
ws = web.WebSocketResponse()
await ws.prepare(request)
print("WebSocket connection opened")
async for msg in ws:
if msg.type == web.WSMsgType.TEXT:
print(f"Received: {msg.data}")
# Echo back
await ws.send_str(f"Echo: {msg.data}")
elif msg.type == web.WSMsgType.ERROR:
print(f"WebSocket error: {ws.exception()}")
print("WebSocket connection closed")
return ws
# Middleware
@web.middleware
async def middleware(request, handler):
"""Logging middleware."""
print(f"Request: {request.method} {request.path}")
start = asyncio.get_event_loop().time()
try:
response = await handler(request)
elapsed = asyncio.get_event_loop().time() - start
print(f"Response: {response.status} - {elapsed:.3f}s")
return response
except web.HTTPException as ex:
elapsed = asyncio.get_event_loop().time() - start
print(f"HTTP Exception: {ex.status} - {elapsed:.3f}s")
raise
# Background tasks
async def background_task(app):
"""Background task that runs with the server."""
print("Background task started")
try:
while True:
await asyncio.sleep(5)
print("Background task: server still running...")
except asyncio.CancelledError:
print("Background task stopped")
async def start_background_tasks(app):
app['background_task'] = asyncio.create_task(background_task(app))
async def cleanup_background_tasks(app):
app['background_task'].cancel()
await app['background_task']
# Create application
app = web.Application(middlewares=[middleware])
# Add routes
app.router.add_get('/', handle_root)
app.router.add_get('/api', handle_api)
app.router.add_get('/greet/{name}', handle_greet)
app.router.add_post('/echo', handle_echo)
app.router.add_get('/ws', handle_websocket)
# Add static files route
app.router.add_static('/static/', path='static/', name='static')
# Setup background tasks
app.on_startup.append(start_background_tasks)
app.on_cleanup.append(cleanup_background_tasks)
if __name__ == "__main__":
print("Starting async web server on http://localhost:8080")
web.run_app(app, host='localhost', port=8080)WebSocket Client:
import asyncio
import aiohttp
import json
async def websocket_client():
"""WebSocket client example."""
uri = "ws://localhost:8080/ws"
async with aiohttp.ClientSession() as session:
async with session.ws_connect(uri) as ws:
print(f"Connected to {uri}")
# Send messages
messages = ["Hello", "How are you?", "Goodbye"]
for msg in messages:
print(f"Sending: {msg}")
await ws.send_str(msg)
# Receive response
response = await ws.receive()
if response.type == aiohttp.WSMsgType.TEXT:
print(f"Received: {response.data}")
elif response.type == aiohttp.WSMsgType.ERROR:
break
await asyncio.sleep(1)
print("Closing connection")
await ws.close()
async def main():
try:
await websocket_client()
except ConnectionRefusedError:
print("Could not connect to WebSocket server")
asyncio.run(main())Async Rate Limiting:
import asyncio
import aiohttp
import time
from collections import deque
class RateLimiter:
"""Rate limiter for async requests."""
def __init__(self, max_calls, period):
self.max_calls = max_calls
self.period = period
self.calls = deque()
async def acquire(self):
"""Acquire permission to make a request."""
now = time.time()
# Remove old calls
while self.calls and self.calls[0] < now - self.period:
self.calls.popleft()
if len(self.calls) >= self.max_calls:
# Wait until oldest call expires
sleep_time = self.calls[0] + self.period - now
print(f"Rate limited, waiting {sleep_time:.2f}s")
await asyncio.sleep(sleep_time)
self.calls.append(now)
async def rate_limited_fetch(session, url, limiter):
"""Fetch with rate limiting."""
await limiter.acquire()
async with session.get(url) as response:
return await response.text()
async def main():
limiter = RateLimiter(max_calls=2, period=1) # 2 calls per second
urls = [f"http://example.com/page/{i}" for i in range(10)]
async with aiohttp.ClientSession() as session:
tasks = [rate_limited_fetch(session, url, limiter) for url in urls]
results = await asyncio.gather(*tasks, return_exceptions=True)
for i, result in enumerate(results):
if isinstance(result, Exception):
print(f"Task {i} failed: {result}")
else:
print(f"Task {i} succeeded: {len(result)} bytes")
asyncio.run(main())Async Producer-Consumer Pattern:
import asyncio
import random
class AsyncQueue:
"""Async producer-consumer example."""
def __init__(self, maxsize=10):
self.queue = asyncio.Queue(maxsize=maxsize)
self.producer_done = False
async def producer(self, producer_id):
"""Producer that generates items."""
for i in range(5):
item = f"Item-{producer_id}-{i}"
await self.queue.put(item)
print(f"Producer {producer_id} produced: {item}")
await asyncio.sleep(random.uniform(0.1, 0.5))
print(f"Producer {producer_id} done")
async def consumer(self, consumer_id):
"""Consumer that processes items."""
while not (self.producer_done and self.queue.empty()):
try:
# Wait for item with timeout
item = await asyncio.wait_for(self.queue.get(), timeout=0.5)
print(f"Consumer {consumer_id} processing: {item}")
await asyncio.sleep(random.uniform(0.2, 0.8))
self.queue.task_done()
except asyncio.TimeoutError:
if self.producer_done:
break
print(f"Consumer {consumer_id} finished")
async def main():
queue = AsyncQueue(maxsize=3)
# Create producers
producers = [asyncio.create_task(queue.producer(i)) for i in range(3)]
# Create consumers
consumers = [asyncio.create_task(queue.consumer(i)) for i in range(2)]
# Wait for producers
await asyncio.gather(*producers)
queue.producer_done = True
# Wait for queue to empty
await queue.queue.join()
# Cancel consumers
for consumer in consumers:
consumer.cancel()
await asyncio.gather(*consumers, return_exceptions=True)
print("All done!")
asyncio.run(main())Networking is fundamental to modern applications, enabling communication between systems, services, and clients across networks.
Sockets provide the foundation for network communication, allowing processes to communicate over TCP/IP.
TCP Server:
import socket
import threading
import sys
class TCPServer:
"""Simple TCP server that echoes messages back to clients."""
def __init__(self, host='localhost', port=8888):
self.host = host
self.port = port
self.server_socket = None
self.running = False
self.clients = []
def start(self):
"""Start the TCP server."""
try:
# Create TCP socket
self.server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.server_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
# Bind to address
self.server_socket.bind((self.host, self.port))
self.server_socket.listen(5)
self.running = True
print(f"TCP Server started on {self.host}:{self.port}")
# Accept connections
while self.running:
try:
client_socket, client_address = self.server_socket.accept()
print(f"New connection from {client_address}")
# Handle client in new thread
client_thread = threading.Thread(
target=self.handle_client,
args=(client_socket, client_address)
)
client_thread.daemon = True
client_thread.start()
except socket.timeout:
continue
except Exception as e:
if self.running:
print(f"Error accepting connection: {e}")
except Exception as e:
print(f"Server error: {e}")
finally:
self.stop()
def handle_client(self, client_socket, client_address):
"""Handle individual client connection."""
self.clients.append(client_socket)
try:
while self.running:
# Receive data
data = client_socket.recv(1024)
if not data:
break
message = data.decode('utf-8')
print(f"Received from {client_address}: {message}")
# Echo back
response = f"Echo: {message}".encode('utf-8')
client_socket.send(response)
except ConnectionResetError:
print(f"Client {client_address} disconnected abruptly")
except Exception as e:
print(f"Error handling client {client_address}: {e}")
finally:
self.clients.remove(client_socket)
client_socket.close()
print(f"Connection closed: {client_address}")
def stop(self):
"""Stop the server."""
self.running = False
# Close all client connections
for client in self.clients:
try:
client.close()
except:
pass
if self.server_socket:
self.server_socket.close()
print("Server stopped")
# TCP Client
class TCPClient:
"""Simple TCP client."""
def __init__(self, host='localhost', port=8888):
self.host = host
self.port = port
self.socket = None
def connect(self):
"""Connect to server."""
self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.socket.connect((self.host, self.port))
print(f"Connected to {self.host}:{self.port}")
def send_message(self, message):
"""Send message to server."""
if not self.socket:
raise Exception("Not connected")
self.socket.send(message.encode('utf-8'))
response = self.socket.recv(1024).decode('utf-8')
return response
def close(self):
"""Close connection."""
if self.socket:
self.socket.close()
self.socket = None
# Usage example
def run_tcp_example():
"""Run TCP server and client example."""
import time
# Start server in background thread
server = TCPServer()
server_thread = threading.Thread(target=server.start)
server_thread.daemon = True
server_thread.start()
time.sleep(1) # Wait for server to start
# Create client
client = TCPClient()
try:
client.connect()
# Send messages
messages = ["Hello", "How are you?", "Goodbye"]
for msg in messages:
response = client.send_message(msg)
print(f"Server response: {response}")
time.sleep(1)
finally:
client.close()
server.stop()
if __name__ == "__main__":
run_tcp_example()UDP Server and Client:
import socket
import threading
class UDPServer:
"""Simple UDP server."""
def __init__(self, host='localhost', port=8889):
self.host = host
self.port = port
self.socket = None
self.running = False
def start(self):
"""Start UDP server."""
try:
# Create UDP socket
self.socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
self.socket.bind((self.host, self.port))
self.running = True
print(f"UDP Server started on {self.host}:{self.port}")
while self.running:
# Receive data
data, client_address = self.socket.recvfrom(1024)
message = data.decode('utf-8')
print(f"Received from {client_address}: {message}")
# Send response
response = f"UDP Echo: {message}".encode('utf-8')
self.socket.sendto(response, client_address)
except Exception as e:
print(f"UDP Server error: {e}")
finally:
self.stop()
def stop(self):
self.running = False
if self.socket:
self.socket.close()
class UDPClient:
"""Simple UDP client."""
def __init__(self, host='localhost', port=8889):
self.host = host
self.port = port
self.socket = None
def connect(self):
"""Create UDP socket (no actual connection)."""
self.socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
# Set timeout to avoid blocking forever
self.socket.settimeout(5)
def send_message(self, message):
"""Send message and wait for response."""
if not self.socket:
raise Exception("Socket not created")
self.socket.sendto(message.encode('utf-8'), (self.host, self.port))
try:
data, _ = self.socket.recvfrom(1024)
return data.decode('utf-8')
except socket.timeout:
return "No response (timeout)"
def close(self):
if self.socket:
self.socket.close()
self.socket = None
# Usage
def run_udp_example():
"""Run UDP server and client example."""
import time
# Start server
server = UDPServer()
server_thread = threading.Thread(target=server.start)
server_thread.daemon = True
server_thread.start()
time.sleep(1)
# Create client
client = UDPClient()
try:
client.connect()
# Send messages (UDP is connectionless)
messages = ["Hello UDP", "This is message 2", "Goodbye UDP"]
for msg in messages:
response = client.send_message(msg)
print(f"Server response: {response}")
time.sleep(1)
finally:
client.close()
server.stop()Non-blocking Sockets:
import socket
import select
import sys
import time
class NonBlockingServer:
"""Non-blocking server using select."""
def __init__(self, host='localhost', port=8890):
self.host = host
self.port = port
self.server_socket = None
self.sockets_list = []
self.clients = {}
def start(self):
"""Start non-blocking server."""
# Create server socket
self.server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.server_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self.server_socket.setblocking(False)
self.server_socket.bind((self.host, self.port))
self.server_socket.listen(5)
self.sockets_list = [self.server_socket]
print(f"Non-blocking server started on {self.host}:{self.port}")
try:
while True:
# Use select to monitor sockets
read_sockets, _, exception_sockets = select.select(
self.sockets_list, [], self.sockets_list, 1
)
# Handle readable sockets
for notified_socket in read_sockets:
if notified_socket == self.server_socket:
# New connection
self.accept_connection()
else:
# Existing client sent data
self.handle_client(notified_socket)
# Handle exceptional sockets
for notified_socket in exception_sockets:
self.remove_client(notified_socket)
except KeyboardInterrupt:
print("\nServer shutting down...")
finally:
self.cleanup()
def accept_connection(self):
"""Accept new connection."""
client_socket, client_address = self.server_socket.accept()
client_socket.setblocking(False)
self.sockets_list.append(client_socket)
self.clients[client_socket] = client_address
print(f"New connection from {client_address}")
def handle_client(self, client_socket):
"""Handle data from client."""
try:
data = client_socket.recv(1024)
if not data:
self.remove_client(client_socket)
return
message = data.decode('utf-8')
client_address = self.clients[client_socket]
print(f"Received from {client_address}: {message}")
# Echo back
response = f"Echo: {message}".encode('utf-8')
client_socket.send(response)
except Exception as e:
print(f"Error handling client: {e}")
self.remove_client(client_socket)
def remove_client(self, client_socket):
"""Remove client connection."""
if client_socket in self.sockets_list:
self.sockets_list.remove(client_socket)
if client_socket in self.clients:
address = self.clients[client_socket]
print(f"Client disconnected: {address}")
del self.clients[client_socket]
client_socket.close()
def cleanup(self):
"""Clean up all connections."""
for socket in self.sockets_list:
socket.close()
self.sockets_list.clear()
self.clients.clear()
# Usage
def run_nonblocking_example():
"""Run non-blocking server example."""
server = NonBlockingServer()
try:
server.start()
except KeyboardInterrupt:
print("\nServer stopped")Socket Options and Advanced Features:
import socket
import struct
def demonstrate_socket_options():
"""Demonstrate various socket options."""
# Create socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# Get default options
print("Default socket options:")
print(f" SO_SNDBUF: {sock.getsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF)}")
print(f" SO_RCVBUF: {sock.getsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF)}")
print(f" SO_REUSEADDR: {sock.getsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR)}")
print(f" SO_KEEPALIVE: {sock.getsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE)}")
# Set options
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
# Set TCP_NODELAY (disable Nagle's algorithm)
sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
# Set send/receive buffer sizes
sock.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, 65536)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 65536)
# Set timeout
sock.settimeout(5)
# Get updated options
print("\nUpdated socket options:")
print(f" SO_REUSEADDR: {sock.getsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR)}")
print(f" SO_KEEPALIVE: {sock.getsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE)}")
print(f" TCP_NODELAY: {sock.getsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY)}")
# Get socket name and peer (if connected)
try:
sock.bind(('localhost', 0)) # Bind to random port
print(f"\nBound to: {sock.getsockname()}")
except:
pass
sock.close()
# Broadcast UDP
def udp_broadcast():
"""Demonstrate UDP broadcasting."""
# Sender
sender = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sender.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
# Send broadcast message
message = "Broadcast message".encode('utf-8')
sender.sendto(message, ('<broadcast>', 9999))
print("Broadcast message sent")
sender.close()
# Receiver
receiver = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
receiver.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
receiver.bind(('', 9999))
print("Listening for broadcast...")
receiver.settimeout(3)
try:
data, addr = receiver.recvfrom(1024)
print(f"Received broadcast from {addr}: {data.decode('utf-8')}")
except socket.timeout:
print("No broadcast received")
finally:
receiver.close()Making HTTP requests to web services and APIs:
Basic HTTP Requests with urllib:
import urllib.request
import urllib.parse
import json
from urllib.error import URLError, HTTPError
def basic_http_examples():
"""Demonstrate basic HTTP requests with urllib."""
# GET request
try:
response = urllib.request.urlopen('http://httpbin.org/get', timeout=5)
print(f"GET Status: {response.status}")
print(f"GET Headers: {response.getheaders()[:5]}")
data = response.read().decode('utf-8')
print(f"GET Data (first 200 chars): {data[:200]}")
except HTTPError as e:
print(f"HTTP Error: {e.code} - {e.reason}")
except URLError as e:
print(f"URL Error: {e.reason}")
# POST request with data
data = urllib.parse.urlencode({'key1': 'value1', 'key2': 'value2'}).encode('utf-8')
req = urllib.request.Request('http://httpbin.org/post', data=data, method='POST')
req.add_header('Content-Type', 'application/x-www-form-urlencoded')
try:
with urllib.request.urlopen(req) as response:
print(f"\nPOST Status: {response.status}")
result = json.loads(response.read().decode('utf-8'))
print(f"POST Response: {json.dumps(result, indent=2)[:200]}...")
except Exception as e:
print(f"POST Error: {e}")
# Custom headers
req = urllib.request.Request('http://httpbin.org/headers')
req.add_header('User-Agent', 'CustomBot/1.0')
req.add_header('Accept', 'application/json')
req.add_header('X-Custom-Header', 'CustomValue')
with urllib.request.urlopen(req) as response:
data = json.loads(response.read().decode('utf-8'))
print(f"\nCustom headers: {json.dumps(data, indent=2)[:200]}...")HTTPX Library (modern, async-capable HTTP client):
# First install: pip install httpx
import httpx
import asyncio
def sync_httpx_examples():
"""Synchronous HTTPX examples."""
with httpx.Client() as client:
# GET request
response = client.get('https://httpbin.org/get', params={'key': 'value'})
print(f"GET: {response.status_code}")
print(f"GET JSON: {response.json()['args']}")
# POST request
response = client.post(
'https://httpbin.org/post',
json={'name': 'John', 'age': 30}
)
print(f"POST: {response.json()['json']}")
# With headers
response = client.get(
'https://httpbin.org/headers',
headers={'User-Agent': 'HTTPX/1.0', 'X-Custom': 'Value'}
)
print(f"Headers: {response.json()['headers']['X-Custom']}")
# File upload
files = {'file': ('test.txt', b'Hello World', 'text/plain')}
response = client.post('https://httpbin.org/post', files=files)
print(f"Upload: {response.json()['files']['file']}")
# Download file
response = client.get('https://httpbin.org/image/png')
with open('image.png', 'wb') as f:
f.write(response.content)
print("Downloaded image")
async def async_httpx_examples():
"""Asynchronous HTTPX examples."""
async with httpx.AsyncClient() as client:
# Multiple concurrent requests
urls = [
'https://httpbin.org/get',
'https://httpbin.org/headers',
'https://httpbin.org/user-agent'
]
tasks = [client.get(url) for url in urls]
responses = await asyncio.gather(*tasks)
for i, response in enumerate(responses):
print(f"URL {i+1}: {response.status_code}")
# Streaming response
async with client.stream('GET', 'https://httpbin.org/stream/5') as response:
async for line in response.aiter_lines():
if line:
print(f"Stream line: {json.loads(line)['id']}")
# Run async example
# asyncio.run(async_httpx_examples())Building and consuming RESTful APIs:
REST API Client:
import requests
import json
from typing import Dict, List, Optional
class RESTClient:
"""Generic REST API client."""
def __init__(self, base_url: str, api_key: Optional[str] = None):
self.base_url = base_url.rstrip('/')
self.session = requests.Session()
self.session.headers.update({
'Content-Type': 'application/json',
'Accept': 'application/json'
})
if api_key:
self.session.headers.update({'Authorization': f'Bearer {api_key}'})
def get(self, endpoint: str, params: Optional[Dict] = None) -> Dict:
"""GET request."""
url = f"{self.base_url}/{endpoint.lstrip('/')}"
response = self.session.get(url, params=params)
response.raise_for_status()
return response.json()
def post(self, endpoint: str, data: Optional[Dict] = None) -> Dict:
"""POST request."""
url = f"{self.base_url}/{endpoint.lstrip('/')}"
response = self.session.post(url, json=data)
response.raise_for_status()
return response.json()
def put(self, endpoint: str, data: Optional[Dict] = None) -> Dict:
"""PUT request."""
url = f"{self.base_url}/{endpoint.lstrip('/')}"
response = self.session.put(url, json=data)
response.raise_for_status()
return response.json()
def delete(self, endpoint: str) -> Dict:
"""DELETE request."""
url = f"{self.base_url}/{endpoint.lstrip('/')}"
response = self.session.delete(url)
response.raise_for_status()
return response.json()
def patch(self, endpoint: str, data: Optional[Dict] = None) -> Dict:
"""PATCH request."""
url = f"{self.base_url}/{endpoint.lstrip('/')}"
response = self.session.patch(url, json=data)
response.raise_for_status()
return response.json()
# Example: JSONPlaceholder API
class JSONPlaceholderAPI(RESTClient):
"""JSONPlaceholder API client."""
def __init__(self):
super().__init__('https://jsonplaceholder.typicode.com')
def get_posts(self, user_id: Optional[int] = None) -> List[Dict]:
"""Get all posts, optionally filtered by user."""
params = {'userId': user_id} if user_id else None
return self.get('posts', params=params)
def get_post(self, post_id: int) -> Dict:
"""Get single post."""
return self.get(f'posts/{post_id}')
def create_post(self, title: str, body: str, user_id: int) -> Dict:
"""Create new post."""
data = {'title': title, 'body': body, 'userId': user_id}
return self.post('posts', data)
def update_post(self, post_id: int, **kwargs) -> Dict:
"""Update post."""
return self.put(f'posts/{post_id}', kwargs)
def delete_post(self, post_id: int) -> Dict:
"""Delete post."""
return self.delete(f'posts/{post_id}')
def get_comments(self, post_id: int) -> List[Dict]:
"""Get comments for a post."""
return self.get(f'posts/{post_id}/comments')
def get_users(self) -> List[Dict]:
"""Get all users."""
return self.get('users')
# Usage
def rest_api_example():
"""Demonstrate REST API usage."""
api = JSONPlaceholderAPI()
# GET requests
print("=== GET Requests ===")
posts = api.get_posts(user_id=1)
print(f"User 1 has {len(posts)} posts")
post = api.get_post(1)
print(f"Post 1 title: {post['title']}")
# POST request
print("\n=== POST Request ===")
new_post = api.create_post(
title="My New Post",
body="This is the content of my post",
user_id=1
)
print(f"Created post with ID: {new_post['id']}")
# PUT request
print("\n=== PUT Request ===")
updated = api.update_post(1, title="Updated Title")
print(f"Updated post title: {updated['title']}")
# GET with relationship
print("\n=== Related Data ===")
comments = api.get_comments(1)
print(f"Post 1 has {len(comments)} comments")
users = api.get_users()
print(f"Total users: {len(users)}")
# Error handling
print("\n=== Error Handling ===")
try:
api.get_post(999999)
except requests.exceptions.HTTPError as e:
print(f"Error: {e.response.status_code} - {e.response.reason}")
# rest_api_example()Building a REST API with Flask:
from flask import Flask, request, jsonify, abort
from flask_cors import CORS
from functools import wraps
import jwt
import datetime
app = Flask(__name__)
CORS(app)
app.config['SECRET_KEY'] = 'your-secret-key-here'
# In-memory database
tasks = [
{'id': 1, 'title': 'Learn Python', 'completed': False},
{'id': 2, 'title': 'Build REST API', 'completed': False},
{'id': 3, 'title': 'Write documentation', 'completed': True}
]
next_id = 4
# Authentication decorator
def token_required(f):
@wraps(f)
def decorated(*args, **kwargs):
token = request.headers.get('Authorization')
if not token:
return jsonify({'message': 'Token is missing'}), 401
try:
# Remove 'Bearer ' prefix if present
if token.startswith('Bearer '):
token = token[7:]
data = jwt.decode(token, app.config['SECRET_KEY'], algorithms=['HS256'])
current_user = data['user']
except:
return jsonify({'message': 'Token is invalid'}), 401
return f(current_user, *args, **kwargs)
return decorated
# Routes
@app.route('/')
def index():
"""API root."""
return jsonify({
'name': 'Task API',
'version': '1.0',
'endpoints': {
'GET /tasks': 'Get all tasks',
'GET /tasks/<id>': 'Get specific task',
'POST /tasks': 'Create new task',
'PUT /tasks/<id>': 'Update task',
'DELETE /tasks/<id>': 'Delete task',
'GET /stats': 'Get task statistics',
'POST /auth/login': 'Get authentication token'
}
})
@app.route('/tasks', methods=['GET'])
def get_tasks():
"""Get all tasks with optional filtering."""
# Query parameters
completed = request.args.get('completed')
search = request.args.get('search')
filtered_tasks = tasks
if completed is not None:
completed_bool = completed.lower() == 'true'
filtered_tasks = [t for t in filtered_tasks if t['completed'] == completed_bool]
if search:
filtered_tasks = [t for t in filtered_tasks if search.lower() in t['title'].lower()]
return jsonify({
'tasks': filtered_tasks,
'count': len(filtered_tasks),
'total': len(tasks)
})
@app.route('/tasks/<int:task_id>', methods=['GET'])
def get_task(task_id):
"""Get specific task."""
task = next((t for t in tasks if t['id'] == task_id), None)
if task is None:
abort(404, description=f"Task {task_id} not found")
return jsonify(task)
@app.route('/tasks', methods=['POST'])
@token_required
def create_task(current_user):
"""Create new task."""
data = request.get_json()
if not data or 'title' not in data:
return jsonify({'error': 'Title is required'}), 400
global next_id
new_task = {
'id': next_id,
'title': data['title'],
'completed': data.get('completed', False),
'created_by': current_user
}
tasks.append(new_task)
next_id += 1
return jsonify(new_task), 201
@app.route('/tasks/<int:task_id>', methods=['PUT'])
def update_task(task_id):
"""Update task."""
task = next((t for t in tasks if t['id'] == task_id), None)
if task is None:
abort(404)
data = request.get_json()
if 'title' in data:
task['title'] = data['title']
if 'completed' in data:
task['completed'] = data['completed']
return jsonify(task)
@app.route('/tasks/<int:task_id>', methods=['DELETE'])
def delete_task(task_id):
"""Delete task."""
global tasks
task = next((t for t in tasks if t['id'] == task_id), None)
if task is None:
abort(404)
tasks = [t for t in tasks if t['id'] != task_id]
return jsonify({'message': 'Task deleted', 'deleted': task})
@app.route('/stats', methods=['GET'])
def get_stats():
"""Get task statistics."""
total = len(tasks)
completed = len([t for t in tasks if t['completed']])
pending = total - completed
return jsonify({
'total': total,
'completed': completed,
'pending': pending,
'completion_rate': (completed / total * 100) if total > 0 else 0
})
@app.route('/auth/login', methods=['POST'])
def login():
"""Get authentication token."""
auth = request.get_json()
if not auth or not auth.get('username') or not auth.get('password'):
return jsonify({'message': 'Username and password required'}), 401
# Simple authentication (in production, use proper user database)
if auth['username'] == 'admin' and auth['password'] == 'secret':
token = jwt.encode({
'user': auth['username'],
'exp': datetime.datetime.utcnow() + datetime.timedelta(hours=24)
}, app.config['SECRET_KEY'], algorithm='HS256')
return jsonify({'token': token})
return jsonify({'message': 'Invalid credentials'}), 401
# Error handlers
@app.errorhandler(404)
def not_found(error):
return jsonify({'error': str(error)}), 404
@app.errorhandler(400)
def bad_request(error):
return jsonify({'error': str(error)}), 400
@app.errorhandler(500)
def internal_error(error):
return jsonify({'error': 'Internal server error'}), 500
if __name__ == '__main__':
app.run(debug=True, port=5000)REST API Client for the Flask API:
import requests
import json
class TaskAPIClient:
"""Client for Task API."""
def __init__(self, base_url='http://localhost:5000', token=None):
self.base_url = base_url
self.session = requests.Session()
if token:
self.session.headers.update({'Authorization': f'Bearer {token}'})
def login(self, username, password):
"""Authenticate and get token."""
response = self.session.post(
f"{self.base_url}/auth/login",
json={'username': username, 'password': password}
)
response.raise_for_status()
token = response.json()['token']
self.session.headers.update({'Authorization': f'Bearer {token}'})
return token
def get_tasks(self, completed=None, search=None):
"""Get all tasks."""
params = {}
if completed is not None:
params['completed'] = str(completed).lower()
if search:
params['search'] = search
response = self.session.get(f"{self.base_url}/tasks", params=params)
response.raise_for_status()
return response.json()
def get_task(self, task_id):
"""Get specific task."""
response = self.session.get(f"{self.base_url}/tasks/{task_id}")
response.raise_for_status()
return response.json()
def create_task(self, title, completed=False):
"""Create new task."""
response = self.session.post(
f"{self.base_url}/tasks",
json={'title': title, 'completed': completed}
)
response.raise_for_status()
return response.json()
def update_task(self, task_id, **kwargs):
"""Update task."""
response = self.session.put(
f"{self.base_url}/tasks/{task_id}",
json=kwargs
)
response.raise_for_status()
return response.json()
def delete_task(self, task_id):
"""Delete task."""
response = self.session.delete(f"{self.base_url}/tasks/{task_id}")
response.raise_for_status()
return response.json()
def get_stats(self):
"""Get task statistics."""
response = self.session.get(f"{self.base_url}/stats")
response.raise_for_status()
return response.json()
# Usage example
def task_api_example():
"""Demonstrate Task API client usage."""
client = TaskAPIClient()
# Login
print("=== Login ===")
try:
token = client.login('admin', 'secret')
print(f"Got token: {token[:20]}...")
except requests.exceptions.HTTPError as e:
print(f"Login failed: {e}")
return
# Get all tasks
print("\n=== All Tasks ===")
result = client.get_tasks()
for task in result['tasks']:
print(f"[{'✓' if task['completed'] else ' '}] {task['id']}: {task['title']}")
# Filter tasks
print("\n=== Pending Tasks ===")
pending = client.get_tasks(completed=False)
for task in pending['tasks']:
print(f" {task['title']}")
# Search
print("\n=== Search 'Python' ===")
search = client.get_tasks(search='Python')
for task in search['tasks']:
print(f" {task['title']}")
# Create task
print("\n=== Create Task ===")
new_task = client.create_task("Test API Client", completed=False)
print(f"Created task {new_task['id']}: {new_task['title']}")
# Update task
print("\n=== Update Task ===")
updated = client.update_task(new_task['id'], completed=True)
print(f"Task {updated['id']} completed: {updated['completed']}")
# Get stats
print("\n=== Statistics ===")
stats = client.get_stats()
print(f"Total: {stats['total']}")
print(f"Completed: {stats['completed']}")
print(f"Completion rate: {stats['completion_rate']:.1f}%")
# Delete task
print("\n=== Delete Task ===")
deleted = client.delete_task(new_task['id'])
print(f"Deleted: {deleted['deleted']['title']}")
# task_api_example()Extracting data from websites:
Basic Web Scraping with BeautifulSoup:
# First install: pip install beautifulsoup4 requests lxml
import requests
from bs4 import BeautifulSoup
import re
from urllib.parse import urljoin, urlparse
import time
class WebScraper:
"""Basic web scraper with polite crawling."""
def __init__(self, delay=1):
self.delay = delay
self.session = requests.Session()
self.session.headers.update({
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
})
self.visited = set()
def get_soup(self, url):
"""Get BeautifulSoup object for URL."""
time.sleep(self.delay) # Be polite
try:
response = self.session.get(url, timeout=10)
response.raise_for_status()
response.encoding = response.apparent_encoding
return BeautifulSoup(response.text, 'lxml')
except Exception as e:
print(f"Error fetching {url}: {e}")
return None
def extract_links(self, url, base_domain=None):
"""Extract all links from page."""
soup = self.get_soup(url)
if not soup:
return []
links = []
for a in soup.find_all('a', href=True):
href = a['href']
full_url = urljoin(url, href)
if base_domain:
parsed = urlparse(full_url)
if parsed.netloc and parsed.netloc != base_domain:
continue
links.append({
'url': full_url,
'text': a.get_text(strip=True),
'title': a.get('title', '')
})
return links
def extract_text(self, url):
"""Extract main text content from page."""
soup = self.get_soup(url)
if not soup:
return ""
# Remove script and style elements
for script in soup(['script', 'style', 'nav', 'footer']):
script.decompose()
# Get text
text = soup.get_text()
# Clean up whitespace
lines = (line.strip() for line in text.splitlines())
chunks = (phrase.strip() for line in lines for phrase in line.split(" "))
text = ' '.join(chunk for chunk in chunks if chunk)
return text
def extract_meta(self, url):
"""Extract meta information from page."""
soup = self.get_soup(url)
if not soup:
return {}
meta = {
'title': soup.title.string if soup.title else None,
'h1': [h1.get_text(strip=True) for h1 in soup.find_all('h1')],
'meta_tags': {}
}
# Meta tags
for tag in soup.find_all('meta'):
if tag.get('name'):
meta['meta_tags'][tag['name']] = tag.get('content', '')
elif tag.get('property'):
meta['meta_tags'][tag['property']] = tag.get('content', '')
return meta
def extract_images(self, url):
"""Extract image information."""
soup = self.get_soup(url)
if not soup:
return []
images = []
for img in soup.find_all('img'):
src = img.get('src')
if src:
full_url = urljoin(url, src)
images.append({
'url': full_url,
'alt': img.get('alt', ''),
'title': img.get('title', ''),
'width': img.get('width'),
'height': img.get('height')
})
return images
# Example usage
def scraping_example():
"""Demonstrate web scraping."""
scraper = WebScraper(delay=1)
url = 'https://example.com'
print(f"=== Scraping {url} ===")
# Extract meta information
meta = scraper.extract_meta(url)
print(f"Title: {meta['title']}")
print(f"H1: {meta['h1']}")
# Extract links
links = scraper.extract_links(url)
print(f"\nFound {len(links)} links:")
for link in links[:5]:
print(f" {link['text']}: {link['url']}")
# Extract images
images = scraper.extract_images(url)
print(f"\nFound {len(images)} images:")
for img in images[:3]:
print(f" {img['alt']}: {img['url']}")
# Extract text
text = scraper.extract_text(url)
print(f"\nText length: {len(text)} characters")
print(f"Preview: {text[:200]}...")
# scraping_example()Advanced Scraping with Selenium:
# First install: pip install selenium webdriver-manager
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
import time
class SeleniumScraper:
"""Advanced scraper using Selenium for JavaScript-heavy sites."""
def __init__(self, headless=True):
options = webdriver.ChromeOptions()
if headless:
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
self.driver = webdriver.Chrome(
service=Service(ChromeDriverManager().install()),
options=options
)
self.wait = WebDriverWait(self.driver, 10)
def close(self):
"""Close browser."""
self.driver.quit()
def get_page(self, url):
"""Navigate to URL and wait for page load."""
self.driver.get(url)
self.wait.until(EC.presence_of_element_located((By.TAG_NAME, "body")))
def click_element(self, by, value):
"""Click element after waiting for it."""
element = self.wait.until(EC.element_to_be_clickable((by, value)))
element.click()
return element
def get_text(self, by, value):
"""Get text from element."""
try:
element = self.wait.until(EC.presence_of_element_located((by, value)))
return element.text
except:
return None
def get_elements_text(self, by, value):
"""Get text from multiple elements."""
elements = self.driver.find_elements(by, value)
return [el.text for el in elements if el.text]
def scroll_to_bottom(self):
"""Scroll to bottom of page (for infinite scroll)."""
last_height = self.driver.execute_script("return document.body.scrollHeight")
while True:
self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(2)
new_height = self.driver.execute_script("return document.body.scrollHeight")
if new_height == last_height:
break
last_height = new_height
def take_screenshot(self, filename):
"""Take screenshot."""
self.driver.save_screenshot(filename)
def execute_script(self, script, *args):
"""Execute JavaScript."""
return self.driver.execute_script(script, *args)
# Example: Scraping a dynamic site
def selenium_example():
"""Demonstrate Selenium scraping."""
scraper = SeleniumScraper(headless=False)
try:
# Navigate to page
scraper.get_page('https://example.com')
# Get page title
title = scraper.get_text(By.TAG_NAME, 'h1')
print(f"Page title: {title}")
# Get all paragraphs
paragraphs = scraper.get_elements_text(By.TAG_NAME, 'p')
for i, p in enumerate(paragraphs[:3]):
print(f"P{i+1}: {p[:100]}...")
# Click a link
scraper.click_element(By.LINK_TEXT, 'More information...')
# Wait for new page
time.sleep(2)
# Take screenshot
scraper.take_screenshot('screenshot.png')
print("Screenshot saved")
finally:
scraper.close()
# selenium_example()Real-time bidirectional communication:
WebSocket Server with websockets:
# First install: pip install websockets
import asyncio
import websockets
import json
import random
from datetime import datetime
class WebSocketServer:
"""WebSocket server for real-time communication."""
def __init__(self, host='localhost', port=8765):
self.host = host
self.port = port
self.clients = set()
self.user_names = {}
async def register(self, websocket):
"""Register new client."""
self.clients.add(websocket)
print(f"New client connected. Total: {len(self.clients)}")
async def unregister(self, websocket):
"""Unregister client."""
self.clients.remove(websocket)
if websocket in self.user_names:
name = self.user_names[websocket]
del self.user_names[websocket]
await self.broadcast(json.dumps({
'type': 'leave',
'user': name,
'message': f"{name} left the chat"
}))
print(f"Client disconnected. Total: {len(self.clients)}")
async def broadcast(self, message):
"""Broadcast message to all clients."""
if self.clients:
await asyncio.gather(
*[client.send(message) for client in self.clients],
return_exceptions=True
)
async def handle_message(self, websocket, message):
"""Handle incoming message."""
try:
data = json.loads(message)
msg_type = data.get('type', 'message')
if msg_type == 'join':
# Client joining with username
name = data.get('name', f"User{random.randint(1000, 9999)}")
self.user_names[websocket] = name
await self.broadcast(json.dumps({
'type': 'join',
'user': name,
'message': f"{name} joined the chat",
'timestamp': datetime.now().isoformat()
}))
elif msg_type == 'message':
# Regular chat message
name = self.user_names.get(websocket, 'Anonymous')
await self.broadcast(json.dumps({
'type': 'message',
'user': name,
'message': data.get('message', ''),
'timestamp': datetime.now().isoformat()
}))
elif msg_type == 'typing':
# Typing indicator
name = self.user_names.get(websocket, 'Anonymous')
await self.broadcast(json.dumps({
'type': 'typing',
'user': name,
'is_typing': data.get('is_typing', False)
}))
elif msg_type == 'ping':
# Ping-pong for connection testing
await websocket.send(json.dumps({
'type': 'pong',
'timestamp': datetime.now().isoformat()
}))
except json.JSONDecodeError:
print(f"Invalid JSON: {message}")
except Exception as e:
print(f"Error handling message: {e}")
async def handler(self, websocket, path):
"""Handle WebSocket connection."""
await self.register(websocket)
try:
async for message in websocket:
await self.handle_message(websocket, message)
except websockets.exceptions.ConnectionClosed:
pass
finally:
await self.unregister(websocket)
async def start(self):
"""Start WebSocket server."""
async with websockets.serve(self.handler, self.host, self.port):
print(f"WebSocket server started on ws://{self.host}:{self.port}")
await asyncio.Future() # Run forever
def run(self):
"""Run server."""
asyncio.run(self.start())
# WebSocket Client
class WebSocketClient:
"""WebSocket client."""
def __init__(self, uri="ws://localhost:8765"):
self.uri = uri
self.websocket = None
self.callbacks = {}
def on(self, event_type, callback):
"""Register event callback."""
self.callbacks[event_type] = callback
async def connect(self, name):
"""Connect to server."""
self.websocket = await websockets.connect(self.uri)
# Send join message
await self.websocket.send(json.dumps({
'type': 'join',
'name': name
}))
# Start listening
asyncio.create_task(self.listen())
async def listen(self):
"""Listen for messages."""
try:
async for message in self.websocket:
data = json.loads(message)
event_type = data.get('type')
if event_type in self.callbacks:
await self.callbacks[event_type](data)
except websockets.exceptions.ConnectionClosed:
print("Connection closed")
async def send_message(self, message):
"""Send chat message."""
if self.websocket:
await self.websocket.send(json.dumps({
'type': 'message',
'message': message
}))
async def send_typing(self, is_typing):
"""Send typing indicator."""
if self.websocket:
await self.websocket.send(json.dumps({
'type': 'typing',
'is_typing': is_typing
}))
async def ping(self):
"""Send ping."""
if self.websocket:
await self.websocket.send(json.dumps({
'type': 'ping'
}))
async def close(self):
"""Close connection."""
if self.websocket:
await self.websocket.close()
# Example chat client
async def chat_client():
"""Run chat client example."""
client = WebSocketClient()
# Register callbacks
def on_join(data):
print(f"\n*** {data['message']} ***")
def on_leave(data):
print(f"\n*** {data['message']} ***")
def on_message(data):
print(f"\n[{data['timestamp'][11:19]}] {data['user']}: {data['message']}")
def on_typing(data):
if data['is_typing']:
print(f"\n{data['user']} is typing...", end='', flush=True)
client.on('join', on_join)
client.on('leave', on_leave)
client.on('message', on_message)
client.on('typing', on_typing)
# Connect
name = input("Enter your name: ")
await client.connect(name)
# Main loop
try:
while True:
message = await asyncio.get_event_loop().run_in_executor(
None, input, "\nYou: "
)
if message.lower() == '/quit':
break
elif message.lower() == '/ping':
await client.ping()
else:
await client.send_message(message)
# Simulate typing
await client.send_typing(False)
finally:
await client.close()
# Run server (in one terminal)
# server = WebSocketServer()
# server.run()
# Run client (in another terminal)
# asyncio.run(chat_client())Fundamental cryptographic operations:
Hashing with hashlib:
import hashlib
import os
class HashExamples:
"""Demonstrate hashing techniques."""
@staticmethod
def hash_string(text, algorithm='sha256'):
"""Hash a string using specified algorithm."""
hash_func = hashlib.new(algorithm)
hash_func.update(text.encode('utf-8'))
return hash_func.hexdigest()
@staticmethod
def hash_file(filename, algorithm='sha256', chunk_size=4096):
"""Hash a file."""
hash_func = hashlib.new(algorithm)
with open(filename, 'rb') as f:
for chunk in iter(lambda: f.read(chunk_size), b''):
hash_func.update(chunk)
return hash_func.hexdigest()
@staticmethod
def hash_with_salt(password):
"""Hash password with random salt."""
# Generate random salt
salt = os.urandom(32)
# Hash password with salt
key = hashlib.pbkdf2_hmac(
'sha256',
password.encode('utf-8'),
salt,
100000 # Number of iterations
)
return salt + key # Store salt with hash
@staticmethod
def verify_password(password, stored):
"""Verify password against stored hash."""
salt = stored[:32] # Extract salt
key = stored[32:] # Extract hash
new_key = hashlib.pbkdf2_hmac(
'sha256',
password.encode('utf-8'),
salt,
100000
)
return new_key == key
@staticmethod
def compare_files(file1, file2):
"""Compare files using hash."""
return HashExamples.hash_file(file1) == HashExamples.hash_file(file2)
# Usage
def hashing_example():
"""Demonstrate hashing."""
# Hash string
text = "Hello, World!"
sha256_hash = HashExamples.hash_string(text)
md5_hash = HashExamples.hash_string(text, 'md5')
print(f"Original: {text}")
print(f"SHA-256: {sha256_hash}")
print(f"MD5: {md5_hash}")
# Password hashing
password = "my_secret_password"
print(f"\nOriginal password: {password}")
stored = HashExamples.hash_with_salt(password)
print(f"Stored hash (with salt): {stored.hex()[:50]}...")
# Verify correct password
is_valid = HashExamples.verify_password(password, stored)
print(f"Correct password valid: {is_valid}")
# Verify wrong password
is_valid = HashExamples.verify_password("wrong_password", stored)
print(f"Wrong password valid: {is_valid}")
# File hashing
with open('test.txt', 'w') as f:
f.write('This is a test file')
file_hash = HashExamples.hash_file('test.txt')
print(f"\nFile hash: {file_hash}")
# hashing_example()Message authentication codes:
import hmac
import hashlib
import base64
class HMACExamples:
"""Demonstrate HMAC usage."""
@staticmethod
def create_hmac(key, message):
"""Create HMAC for message."""
h = hmac.new(key.encode('utf-8'), message.encode('utf-8'), hashlib.sha256)
return h.hexdigest()
@staticmethod
def verify_hmac(key, message, signature):
"""Verify HMAC signature."""
expected = HMACExamples.create_hmac(key, message)
return hmac.compare_digest(expected, signature)
@staticmethod
def create_secure_token(data, key):
"""Create secure token with HMAC."""
# Convert data to string
if isinstance(data, dict):
import json
message = json.dumps(data, sort_keys=True)
else:
message = str(data)
# Create signature
signature = HMACExamples.create_hmac(key, message)
# Combine data and signature
token = base64.urlsafe_b64encode(
f"{message}:{signature}".encode('utf-8')
).decode('utf-8')
return token
@staticmethod
def verify_secure_token(token, key):
"""Verify secure token."""
try:
# Decode token
decoded = base64.urlsafe_b64decode(token.encode('utf-8')).decode('utf-8')
message, signature = decoded.rsplit(':', 1)
# Verify signature
if HMACExamples.verify_hmac(key, message, signature):
# Try to parse as JSON
try:
import json
return json.loads(message)
except:
return message
else:
return None
except Exception as e:
print(f"Token verification failed: {e}")
return None
# Usage
def hmac_example():
"""Demonstrate HMAC."""
key = "my_secret_key"
# Simple HMAC
message = "Important message"
signature = HMACExamples.create_hmac(key, message)
print(f"Message: {message}")
print(f"HMAC: {signature}")
# Verify
is_valid = HMACExamples.verify_hmac(key, message, signature)
print(f"Signature valid: {is_valid}")
# Tampered message
is_valid = HMACExamples.verify_hmac(key, "Different message", signature)
print(f"Tampered valid: {is_valid}")
# Secure tokens
data = {
'user_id': 123,
'username': 'alice',
'role': 'admin',
'expires': '2026-12-31'
}
token = HMACExamples.create_secure_token(data, key)
print(f"\nSecure token: {token[:50]}...")
# Verify token
verified = HMACExamples.verify_secure_token(token, key)
print(f"Verified data: {verified}")
# Tampered token
tampered = token[:-5] + "XXXXX"
verified = HMACExamples.verify_secure_token(tampered, key)
print(f"Tampered verification: {verified}")
# hmac_example()Symmetric and asymmetric encryption:
Symmetric Encryption with cryptography:
# First install: pip install cryptography
from cryptography.fernet import Fernet
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.kdf.pbkdf2 import PBKDF2HMAC
from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes
from cryptography.hazmat.primitives.asymmetric import rsa, padding
from cryptography.hazmat.primitives import serialization
import base64
import os
class SymmetricEncryption:
"""Symmetric encryption examples."""
@staticmethod
def generate_key():
"""Generate a Fernet key."""
return Fernet.generate_key()
@staticmethod
def encrypt_file(key, input_file, output_file):
"""Encrypt file."""
f = Fernet(key)
with open(input_file, 'rb') as f_in:
data = f_in.read()
encrypted = f.encrypt(data)
with open(output_file, 'wb') as f_out:
f_out.write(encrypted)
@staticmethod
def decrypt_file(key, input_file, output_file):
"""Decrypt file."""
f = Fernet(key)
with open(input_file, 'rb') as f_in:
encrypted = f_in.read()
decrypted = f.decrypt(encrypted)
with open(output_file, 'wb') as f_out:
f_out.write(decrypted)
@staticmethod
def encrypt_message(key, message):
"""Encrypt message."""
f = Fernet(key)
return f.encrypt(message.encode('utf-8'))
@staticmethod
def decrypt_message(key, encrypted):
"""Decrypt message."""
f = Fernet(key)
return f.decrypt(encrypted).decode('utf-8')
@staticmethod
def derive_key_from_password(password, salt=None):
"""Derive encryption key from password."""
if salt is None:
salt = os.urandom(16)
kdf = PBKDF2HMAC(
algorithm=hashes.SHA256(),
length=32,
salt=salt,
iterations=100000,
)
key = base64.urlsafe_b64encode(kdf.derive(password.encode('utf-8')))
return key, salt
# Usage
def symmetric_example():
"""Demonstrate symmetric encryption."""
# Generate key
key = SymmetricEncryption.generate_key()
print(f"Encryption key: {key.decode()[:20]}...")
# Encrypt message
message = "This is a secret message"
encrypted = SymmetricEncryption.encrypt_message(key, message)
print(f"\nOriginal: {message}")
print(f"Encrypted: {encrypted[:50]}...")
# Decrypt message
decrypted = SymmetricEncryption.decrypt_message(key, encrypted)
print(f"Decrypted: {decrypted}")
# File encryption
with open('plain.txt', 'w') as f:
f.write("This is sensitive file content")
SymmetricEncryption.encrypt_file(key, 'plain.txt', 'encrypted.bin')
print("\nFile encrypted: encrypted.bin")
SymmetricEncryption.decrypt_file(key, 'encrypted.bin', 'decrypted.txt')
print("File decrypted: decrypted.txt")
# Password-based encryption
password = "user_password"
key_from_password, salt = SymmetricEncryption.derive_key_from_password(password)
print(f"\nKey derived from password: {key_from_password.decode()[:20]}...")
encrypted2 = SymmetricEncryption.encrypt_message(key_from_password, message)
decrypted2 = SymmetricEncryption.decrypt_message(key_from_password, encrypted2)
print(f"Password-based decryption: {decrypted2}")
# symmetric_example()Asymmetric Encryption:
from cryptography.hazmat.primitives.asymmetric import rsa, padding
from cryptography.hazmat.primitives import hashes, serialization
from cryptography.hazmat.primitives.asymmetric import utils
class AsymmetricEncryption:
"""Asymmetric encryption examples."""
@staticmethod
def generate_key_pair():
"""Generate RSA key pair."""
private_key = rsa.generate_private_key(
public_exponent=65537,
key_size=2048
)
public_key = private_key.public_key()
return private_key, public_key
@staticmethod
def encrypt_message(public_key, message):
"""Encrypt message with public key."""
encrypted = public_key.encrypt(
message.encode('utf-8'),
padding.OAEP(
mgf=padding.MGF1(algorithm=hashes.SHA256()),
algorithm=hashes.SHA256(),
label=None
)
)
return encrypted
@staticmethod
def decrypt_message(private_key, encrypted):
"""Decrypt message with private key."""
decrypted = private_key.decrypt(
encrypted,
padding.OAEP(
mgf=padding.MGF1(algorithm=hashes.SHA256()),
algorithm=hashes.SHA256(),
label=None
)
)
return decrypted.decode('utf-8')
@staticmethod
def sign_message(private_key, message):
"""Sign message with private key."""
signature = private_key.sign(
message.encode('utf-8'),
padding.PSS(
mgf=padding.MGF1(hashes.SHA256()),
salt_length=padding.PSS.MAX_LENGTH
),
hashes.SHA256()
)
return signature
@staticmethod
def verify_signature(public_key, message, signature):
"""Verify signature with public key."""
try:
public_key.verify(
signature,
message.encode('utf-8'),
padding.PSS(
mgf=padding.MGF1(hashes.SHA256()),
salt_length=padding.PSS.MAX_LENGTH
),
hashes.SHA256()
)
return True
except:
return False
@staticmethod
def save_key(key, filename, password=None):
"""Save key to file."""
if isinstance(key, rsa.RSAPrivateKey):
# Save private key
if password:
encryption = serialization.BestAvailableEncryption(password.encode('utf-8'))
else:
encryption = serialization.NoEncryption()
pem = key.private_bytes(
encoding=serialization.Encoding.PEM,
format=serialization.PrivateFormat.PKCS8,
encryption_algorithm=encryption
)
else:
# Save public key
pem = key.public_bytes(
encoding=serialization.Encoding.PEM,
format=serialization.PublicFormat.SubjectPublicKeyInfo
)
with open(filename, 'wb') as f:
f.write(pem)
@staticmethod
def load_private_key(filename, password=None):
"""Load private key from file."""
with open(filename, 'rb') as f:
pem_data = f.read()
if password:
password = password.encode('utf-8')
return serialization.load_pem_private_key(pem_data, password=password)
@staticmethod
def load_public_key(filename):
"""Load public key from file."""
with open(filename, 'rb') as f:
pem_data = f.read()
return serialization.load_pem_public_key(pem_data)
# Usage
def asymmetric_example():
"""Demonstrate asymmetric encryption."""
# Generate key pair
private_key, public_key = AsymmetricEncryption.generate_key_pair()
print("RSA key pair generated")
# Encrypt with public key, decrypt with private
message = "Secret message for asymmetric encryption"
encrypted = AsymmetricEncryption.encrypt_message(public_key, message)
decrypted = AsymmetricEncryption.decrypt_message(private_key, encrypted)
print(f"\nOriginal: {message}")
print(f"Encrypted (hex): {encrypted.hex()[:50]}...")
print(f"Decrypted: {decrypted}")
# Digital signatures
message2 = "Important document content"
signature = AsymmetricEncryption.sign_message(private_key, message2)
print(f"\nMessage: {message2}")
print(f"Signature (hex): {signature.hex()[:50]}...")
# Verify signature
is_valid = AsymmetricEncryption.verify_signature(public_key, message2, signature)
print(f"Signature valid: {is_valid}")
# Tampered message
is_valid = AsymmetricEncryption.verify_signature(public_key, "Different message", signature)
print(f"Tampered valid: {is_valid}")
# Save keys to files
AsymmetricEncryption.save_key(private_key, 'private_key.pem', password='keypass')
AsymmetricEncryption.save_key(public_key, 'public_key.pem')
print("\nKeys saved to files")
# Load keys from files
loaded_private = AsymmetricEncryption.load_private_key('private_key.pem', password='keypass')
loaded_public = AsymmetricEncryption.load_public_key('public_key.pem')
# Test loaded keys
encrypted2 = AsymmetricEncryption.encrypt_message(loaded_public, "Test")
decrypted2 = AsymmetricEncryption.decrypt_message(loaded_private, encrypted2)
print(f"Loaded keys test: {decrypted2}")
# asymmetric_example()Best practices for writing secure Python code:
Input Validation:
import re
from typing import Any, Optional
import html
class SecureInput:
"""Secure input handling."""
@staticmethod
def validate_email(email: str) -> bool:
"""Validate email address."""
pattern = r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'
return bool(re.match(pattern, email))
@staticmethod
def sanitize_html(text: str) -> str:
"""Sanitize HTML to prevent XSS."""
return html.escape(text)
@staticmethod
def validate_integer(value: str, min_val: Optional[int] = None,
max_val: Optional[int] = None) -> Optional[int]:
"""Validate integer input."""
try:
num = int(value)
if min_val is not None and num < min_val:
return None
if max_val is not None and num > max_val:
return None
return num
except ValueError:
return None
@staticmethod
def validate_filename(filename: str) -> bool:
"""Validate filename to prevent path traversal."""
# Check for path traversal attempts
if '..' in filename or '/' in filename or '\\' in filename:
return False
# Check for dangerous extensions
dangerous = ['.exe', '.sh', '.bat', '.cmd', '.ps1']
if any(filename.lower().endswith(ext) for ext in dangerous):
return False
return True
@staticmethod
def validate_url(url: str) -> bool:
"""Validate URL."""
pattern = r'^https?://[^\s/$.?#].[^\s]*$'
return bool(re.match(pattern, url))
class SecureDatabase:
"""Secure database operations."""
@staticmethod
def safe_query(conn, query: str, params: tuple) -> Any:
"""Execute query with parameterization."""
# NEVER do this:
# cursor.execute(f"SELECT * FROM users WHERE name = '{user_input}'")
# ALWAYS do this:
cursor = conn.cursor()
cursor.execute(query, params)
return cursor.fetchall()
@staticmethod
def escape_identifier(identifier: str) -> str:
"""Escape database identifiers."""
# Simple escaping - in practice, use database-specific quoting
return identifier.replace('`', '``').replace('"', '""')
class SecureFileOperations:
"""Secure file operations."""
@staticmethod
def safe_path(base_dir: str, filename: str) -> str:
"""Create safe file path."""
import os
# Normalize path
full_path = os.path.normpath(os.path.join(base_dir, filename))
# Check if path is within base directory
if not full_path.startswith(os.path.abspath(base_dir)):
raise ValueError("Path traversal attempt detected")
return full_path
@staticmethod
def safe_read_file(base_dir: str, filename: str) -> Optional[str]:
"""Safely read file."""
try:
filepath = SecureFileOperations.safe_path(base_dir, filename)
with open(filepath, 'r', encoding='utf-8') as f:
return f.read()
except (ValueError, IOError) as e:
print(f"Error reading file: {e}")
return None
# Usage examples
def secure_coding_example():
"""Demonstrate secure coding practices."""
# Input validation
email = "user@example.com"
invalid_email = "not-an-email"
print(f"Email '{email}' valid: {SecureInput.validate_email(email)}")
print(f"Email '{invalid_email}' valid: {SecureInput.validate_email(invalid_email)}")
# HTML sanitization
user_input = "<script>alert('XSS')</script>Hello"
sanitized = SecureInput.sanitize_html(user_input)
print(f"\nOriginal: {user_input}")
print(f"Sanitized: {sanitized}")
# Integer validation
age = SecureInput.validate_integer("25", min_val=0, max_val=150)
invalid_age = SecureInput.validate_integer("200", min_val=0, max_val=150)
print(f"\nAge 25 valid: {age}")
print(f"Age 200 valid: {invalid_age}")
# Filename validation
safe_file = "document.pdf"
unsafe_file = "../../etc/passwd"
print(f"\n'{safe_file}' safe: {SecureInput.validate_filename(safe_file)}")
print(f"'{unsafe_file}' safe: {SecureInput.validate_filename(unsafe_file)}")
# URL validation
safe_url = "https://example.com/page?param=value"
unsafe_url = "javascript:alert('XSS')"
print(f"\nURL '{safe_url}' valid: {SecureInput.validate_url(safe_url)}")
print(f"URL '{unsafe_url}' valid: {SecureInput.validate_url(unsafe_url)}")
# Safe file paths
import tempfile
import os
base_dir = tempfile.mkdtemp()
try:
# Create test file
test_file = os.path.join(base_dir, "test.txt")
with open(test_file, 'w') as f:
f.write("Sensitive data")
# Safe read
content = SecureFileOperations.safe_read_file(base_dir, "test.txt")
print(f"\nSafe read successful: {content}")
# Attempt path traversal
try:
SecureFileOperations.safe_read_file(base_dir, "../test.txt")
except ValueError as e:
print(f"Path traversal blocked: {e}")
finally:
import shutil
shutil.rmtree(base_dir)
# secure_coding_example()Authentication and Authorization:
import hashlib
import secrets
import time
from functools import wraps
from typing import Optional, Dict, Any
import jwt
class AuthSystem:
"""Simple authentication system."""
def __init__(self, secret_key: str):
self.secret_key = secret_key
self.users = {} # In production, use database
self.sessions = {}
def hash_password(self, password: str, salt: Optional[bytes] = None) -> tuple:
"""Hash password with salt."""
if salt is None:
salt = secrets.token_bytes(32)
key = hashlib.pbkdf2_hmac(
'sha256',
password.encode('utf-8'),
salt,
100000
)
return salt, key
def register(self, username: str, password: str) -> bool:
"""Register new user."""
if username in self.users:
return False
salt, key = self.hash_password(password)
self.users[username] = {
'salt': salt,
'key': key,
'created_at': time.time(),
'roles': ['user']
}
return True
def authenticate(self, username: str, password: str) -> Optional[Dict]:
"""Authenticate user."""
if username not in self.users:
return None
user = self.users[username]
salt = user['salt']
stored_key = user['key']
_, key = self.hash_password(password, salt)
if secrets.compare_digest(key, stored_key):
# Create session
session_token = secrets.token_urlsafe(32)
session_data = {
'username': username,
'created_at': time.time(),
'expires_at': time.time() + 3600 # 1 hour
}
self.sessions[session_token] = session_data
return {
'session_token': session_token,
'user': username,
'roles': user['roles']
}
return None
def validate_session(self, session_token: str) -> Optional[Dict]:
"""Validate session token."""
if session_token not in self.sessions:
return None
session = self.sessions[session_token]
if session['expires_at'] < time.time():
del self.sessions[session_token]
return None
return session
def logout(self, session_token: str):
"""Logout user."""
if session_token in self.sessions:
del self.sessions[session_token]
def create_jwt(self, username: str, expires_in: int = 3600) -> str:
"""Create JWT token."""
payload = {
'username': username,
'exp': time.time() + expires_in,
'iat': time.time(),
'roles': self.users[username]['roles']
}
return jwt.encode(payload, self.secret_key, algorithm='HS256')
def verify_jwt(self, token: str) -> Optional[Dict]:
"""Verify JWT token."""
try:
payload = jwt.decode(token, self.secret_key, algorithms=['HS256'])
return payload
except jwt.InvalidTokenError:
return None
# Decorator for authentication
def require_auth(auth_system: AuthSystem):
"""Decorator to require authentication."""
def decorator(func):
@wraps(func)
def wrapper(*args, **kwargs):
# In real app, get token from request headers
session_token = kwargs.get('session_token')
if not session_token:
return {'error': 'Authentication required'}, 401
session = auth_system.validate_session(session_token)
if not session:
return {'error': 'Invalid or expired session'}, 401
# Add user to kwargs
kwargs['user'] = session['username']
return func(*args, **kwargs)
return wrapper
return decorator
def require_role(role: str):
"""Decorator to require specific role."""
def decorator(func):
@wraps(func)
def wrapper(*args, **kwargs):
user = kwargs.get('user')
auth = kwargs.get('auth_system')
if not user or not auth:
return {'error': 'Authentication required'}, 401
user_data = auth.users.get(user)
if not user_data or role not in user_data['roles']:
return {'error': f'Role {role} required'}, 403
return func(*args, **kwargs)
return wrapper
return decorator
# Usage
def auth_example():
"""Demonstrate authentication system."""
auth = AuthSystem("my_secret_key_123")
# Register users
auth.register("alice", "password123")
auth.register("bob", "secure_pass")
# Add admin role to alice
auth.users["alice"]["roles"].append("admin")
# Authenticate
print("=== Authentication ===")
result = auth.authenticate("alice", "password123")
if result:
print(f"Login successful: {result}")
session_token = result['session_token']
else:
print("Login failed")
return
# Wrong password
result = auth.authenticate("alice", "wrongpass")
print(f"Wrong password: {result}")
# Validate session
print("\n=== Session Validation ===")
session = auth.validate_session(session_token)
print(f"Session valid: {session is not None}")
print(f"Session data: {session}")
# JWT tokens
print("\n=== JWT Tokens ===")
jwt_token = auth.create_jwt("alice")
print(f"JWT: {jwt_token[:50]}...")
verified = auth.verify_jwt(jwt_token)
print(f"Verified: {verified['username']}")
# Logout
auth.logout(session_token)
session = auth.validate_session(session_token)
print(f"\nAfter logout - session valid: {session is not None}")
# auth_example()Secure Configuration Management:
import os
import json
import base64
from cryptography.fernet import Fernet
from pathlib import Path
class SecureConfig:
"""Secure configuration management."""
def __init__(self, config_file: str, key_file: Optional[str] = None):
self.config_file = Path(config_file)
self.key_file = Path(key_file) if key_file else None
self.config = {}
self._load_config()
def _generate_key(self) -> bytes:
"""Generate encryption key."""
return Fernet.generate_key()
def _load_key(self) -> bytes:
"""Load or create encryption key."""
if self.key_file and self.key_file.exists():
with open(self.key_file, 'rb') as f:
return f.read()
else:
key = self._generate_key()
if self.key_file:
# Save key with restricted permissions
with open(self.key_file, 'wb') as f:
f.write(key)
os.chmod(self.key_file, 0o600) # Read/write for owner only
return key
def _load_config(self):
"""Load configuration from file."""
if self.config_file.exists():
with open(self.config_file, 'r') as f:
self.config = json.load(f)
def _save_config(self):
"""Save configuration to file."""
with open(self.config_file, 'w') as f:
json.dump(self.config, f, indent=2)
os.chmod(self.config_file, 0o600)
def get(self, key: str, default=None):
"""Get configuration value."""
return self.config.get(key, default)
def set(self, key: str, value: Any, encrypt: bool = False):
"""Set configuration value."""
if encrypt:
value = self.encrypt_value(str(value))
self.config[key] = value
self._save_config()
def encrypt_value(self, value: str) -> str:
"""Encrypt sensitive value."""
key = self._load_key()
f = Fernet(key)
encrypted = f.encrypt(value.encode('utf-8'))
return base64.urlsafe_b64encode(encrypted).decode('utf-8')
def decrypt_value(self, encrypted_value: str) -> str:
"""Decrypt sensitive value."""
key = self._load_key()
f = Fernet(key)
encrypted = base64.urlsafe_b64decode(encrypted_value.encode('utf-8'))
decrypted = f.decrypt(encrypted)
return decrypted.decode('utf-8')
def get_sensitive(self, key: str, default=None) -> Optional[str]:
"""Get and decrypt sensitive value."""
value = self.config.get(key)
if value is None:
return default
try:
return self.decrypt_value(value)
except:
return None
@staticmethod
def get_env_var(key: str, default=None) -> Optional[str]:
"""Get environment variable."""
return os.environ.get(key, default)
@staticmethod
def set_env_var(key: str, value: str):
"""Set environment variable."""
os.environ[key] = value
# Usage
def config_example():
"""Demonstrate secure configuration."""
# Create secure config
config = SecureConfig('config.json', 'config.key')
# Store regular config
config.set('app_name', 'MyApp')
config.set('debug', False)
config.set('port', 8080)
# Store sensitive data (encrypted)
config.set('database_password', 'supersecret123', encrypt=True)
config.set('api_key', 'sk-1234567890abcdef', encrypt=True)
# Retrieve values
print(f"App name: {config.get('app_name')}")
print(f"Debug: {config.get('debug')}")
print(f"Port: {config.get('port')}")
# Retrieve sensitive values
db_password = config.get_sensitive('database_password')
print(f"DB Password: {db_password}")
api_key = config.get_sensitive('api_key')
print(f"API Key: {api_key}")
# Environment variables
SecureConfig.set_env_var('DATABASE_URL', 'postgresql://localhost:5432/mydb')
db_url = SecureConfig.get_env_var('DATABASE_URL')
print(f"Database URL from env: {db_url}")
# config_example()Understanding HTTP is essential for web development. Here's a comprehensive look at HTTP fundamentals:
HTTP Request Structure:
import http.client
import urllib.parse
from typing import Dict, Optional
class HTTPRequest:
"""Representation of an HTTP request."""
def __init__(self, method: str, path: str, headers: Dict = None, body: str = None):
self.method = method.upper()
self.path = path
self.headers = headers or {}
self.body = body
self.version = "HTTP/1.1"
def __str__(self):
"""String representation of HTTP request."""
lines = [f"{self.method} {self.path} {self.version}"]
for key, value in self.headers.items():
lines.append(f"{key}: {value}")
if self.body:
lines.append("")
lines.append(self.body)
return "\r\n".join(lines)
@classmethod
def parse(cls, raw_request: str):
"""Parse raw HTTP request string."""
lines = raw_request.split("\r\n")
# Parse request line
request_line = lines[0].split()
method, path, version = request_line
# Parse headers
headers = {}
i = 1
while i < len(lines) and lines[i]:
key, value = lines[i].split(": ", 1)
headers[key] = value
i += 1
# Parse body
body = None
if i + 1 < len(lines):
body = "\r\n".join(lines[i+1:])
return cls(method, path, headers, body)
# HTTP Response
class HTTPResponse:
"""Representation of an HTTP response."""
def __init__(self, status_code: int, status_text: str, headers: Dict = None, body: str = None):
self.status_code = status_code
self.status_text = status_text
self.headers = headers or {}
self.body = body
self.version = "HTTP/1.1"
def __str__(self):
"""String representation of HTTP response."""
lines = [f"{self.version} {self.status_code} {self.status_text}"]
for key, value in self.headers.items():
lines.append(f"{key}: {value}")
if self.body:
lines.append("")
lines.append(self.body)
return "\r\n".join(lines)
@classmethod
def parse(cls, raw_response: str):
"""Parse raw HTTP response string."""
lines = raw_response.split("\r\n")
# Parse status line
status_line = lines[0].split()
version, status_code, status_text = status_line[0], int(status_line[1]), " ".join(status_line[2:])
# Parse headers
headers = {}
i = 1
while i < len(lines) and lines[i]:
key, value = lines[i].split(": ", 1)
headers[key] = value
i += 1
# Parse body
body = None
if i + 1 < len(lines):
body = "\r\n".join(lines[i+1:])
return cls(status_code, status_text, headers, body)
# HTTP Status Codes
HTTP_STATUS_CODES = {
# 1xx: Informational
100: "Continue",
101: "Switching Protocols",
# 2xx: Success
200: "OK",
201: "Created",
202: "Accepted",
204: "No Content",
# 3xx: Redirection
301: "Moved Permanently",
302: "Found",
304: "Not Modified",
307: "Temporary Redirect",
308: "Permanent Redirect",
# 4xx: Client Error
400: "Bad Request",
401: "Unauthorized",
403: "Forbidden",
404: "Not Found",
405: "Method Not Allowed",
408: "Request Timeout",
409: "Conflict",
429: "Too Many Requests",
# 5xx: Server Error
500: "Internal Server Error",
501: "Not Implemented",
502: "Bad Gateway",
503: "Service Unavailable",
504: "Gateway Timeout"
}
def get_status_text(code: int) -> str:
"""Get status text for HTTP status code."""
return HTTP_STATUS_CODES.get(code, "Unknown")
# HTTP Methods
HTTP_METHODS = ["GET", "POST", "PUT", "DELETE", "PATCH", "HEAD", "OPTIONS"]
# Common Headers
COMMON_HEADERS = {
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Encoding": "gzip, deflate, br",
"Accept-Language": "en-US,en;q=0.5",
"Cache-Control": "no-cache",
"Connection": "keep-alive",
"Content-Type": "application/json",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
"Authorization": "Bearer <token>"
}
# Example usage
def http_example():
"""Demonstrate HTTP concepts."""
# Create a request
request = HTTPRequest(
method="POST",
path="/api/users",
headers={
"Host": "example.com",
"Content-Type": "application/json",
"Authorization": "Bearer token123"
},
body='{"name": "John", "age": 30}'
)
print("=== HTTP Request ===")
print(request)
# Parse a request
raw_request = "GET /index.html HTTP/1.1\r\nHost: example.com\r\nUser-Agent: Test\r\n\r\n"
parsed = HTTPRequest.parse(raw_request)
print(f"\nParsed request: {parsed.method} {parsed.path}")
# Create a response
response = HTTPResponse(
status_code=200,
status_text="OK",
headers={
"Content-Type": "text/html",
"Content-Length": "13"
},
body="Hello, World!"
)
print("\n=== HTTP Response ===")
print(response)
# Show common status codes
print("\n=== Common Status Codes ===")
for code in [200, 404, 500]:
print(f"{code}: {get_status_text(code)}")
# http_example()HTTP Client Implementation:
import socket
import ssl
from urllib.parse import urlparse
class SimpleHTTPClient:
"""Simple HTTP client implementation."""
def __init__(self):
self.connection = None
self.socket = None
def request(self, method: str, url: str, headers: Dict = None, body: str = None) -> HTTPResponse:
"""Make HTTP request."""
parsed = urlparse(url)
host = parsed.hostname
port = parsed.port or (443 if parsed.scheme == 'https' else 80)
path = parsed.path or '/'
if parsed.query:
path += '?' + parsed.query
# Create socket
if parsed.scheme == 'https':
context = ssl.create_default_context()
self.socket = context.wrap_socket(socket.socket(socket.AF_INET), server_hostname=host)
else:
self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.socket.connect((host, port))
# Build request
request_headers = headers or {}
request_headers.setdefault('Host', host)
request_headers.setdefault('Connection', 'close')
if body:
request_headers.setdefault('Content-Length', str(len(body)))
request_headers.setdefault('Content-Type', 'text/plain')
request = HTTPRequest(method, path, request_headers, body)
# Send request
self.socket.send(str(request).encode('utf-8'))
# Receive response
response_data = self.socket.recv(4096).decode('utf-8')
# Parse response
response = HTTPResponse.parse(response_data)
self.socket.close()
return response
def get(self, url: str, headers: Dict = None) -> HTTPResponse:
"""Make GET request."""
return self.request('GET', url, headers)
def post(self, url: str, data: str = None, headers: Dict = None) -> HTTPResponse:
"""Make POST request."""
return self.request('POST', url, headers, data)
# Usage
def http_client_example():
"""Demonstrate HTTP client."""
client = SimpleHTTPClient()
try:
# GET request
response = client.get('http://example.com')
print(f"Status: {response.status_code} {response.status_text}")
print(f"Headers: {dict(response.headers)}")
print(f"Body preview: {response.body[:100]}...")
except Exception as e:
print(f"Error: {e}")
# http_client_example()REST (Representational State Transfer) architectural style:
from typing import Dict, List, Optional, Any
import json
from datetime import datetime
class Resource:
"""Base class for REST resources."""
def __init__(self, id: Optional[int] = None):
self.id = id
self.created_at = datetime.now().isoformat()
self.updated_at = self.created_at
def to_dict(self) -> Dict:
"""Convert resource to dictionary."""
return {
'id': self.id,
'created_at': self.created_at,
'updated_at': self.updated_at
}
def update(self, data: Dict):
"""Update resource with data."""
for key, value in data.items():
if hasattr(self, key):
setattr(self, key, value)
self.updated_at = datetime.now().isoformat()
class User(Resource):
"""User resource."""
def __init__(self, id: Optional[int] = None, name: str = "", email: str = ""):
super().__init__(id)
self.name = name
self.email = email
def to_dict(self) -> Dict:
data = super().to_dict()
data.update({
'name': self.name,
'email': self.email
})
return data
class Post(Resource):
"""Post resource."""
def __init__(self, id: Optional[int] = None, title: str = "", content: str = "", user_id: int = None):
super().__init__(id)
self.title = title
self.content = content
self.user_id = user_id
def to_dict(self) -> Dict:
data = super().to_dict()
data.update({
'title': self.title,
'content': self.content,
'user_id': self.user_id
})
return data
class RESTAPI:
"""Simple REST API implementation."""
def __init__(self):
self.users: Dict[int, User] = {}
self.posts: Dict[int, Post] = {}
self.next_user_id = 1
self.next_post_id = 1
# User endpoints
def get_users(self) -> List[Dict]:
"""GET /users - List all users."""
return [user.to_dict() for user in self.users.values()]
def get_user(self, user_id: int) -> Optional[Dict]:
"""GET /users/{id} - Get specific user."""
user = self.users.get(user_id)
return user.to_dict() if user else None
def create_user(self, data: Dict) -> Dict:
"""POST /users - Create new user."""
user = User(
id=self.next_user_id,
name=data.get('name', ''),
email=data.get('email', '')
)
self.users[self.next_user_id] = user
self.next_user_id += 1
return user.to_dict()
def update_user(self, user_id: int, data: Dict) -> Optional[Dict]:
"""PUT /users/{id} - Update user."""
user = self.users.get(user_id)
if user:
user.update(data)
return user.to_dict()
return None
def delete_user(self, user_id: int) -> bool:
"""DELETE /users/{id} - Delete user."""
if user_id in self.users:
# Also delete user's posts
posts_to_delete = [pid for pid, post in self.posts.items() if post.user_id == user_id]
for pid in posts_to_delete:
del self.posts[pid]
del self.users[user_id]
return True
return False
# Post endpoints
def get_posts(self, user_id: Optional[int] = None) -> List[Dict]:
"""GET /posts - List all posts, optionally filtered by user."""
posts = self.posts.values()
if user_id:
posts = [p for p in posts if p.user_id == user_id]
return [post.to_dict() for post in posts]
def get_post(self, post_id: int) -> Optional[Dict]:
"""GET /posts/{id} - Get specific post."""
post = self.posts.get(post_id)
return post.to_dict() if post else None
def create_post(self, data: Dict) -> Optional[Dict]:
"""POST /posts - Create new post."""
user_id = data.get('user_id')
if user_id not in self.users:
return None
post = Post(
id=self.next_post_id,
title=data.get('title', ''),
content=data.get('content', ''),
user_id=user_id
)
self.posts[self.next_post_id] = post
self.next_post_id += 1
return post.to_dict()
def update_post(self, post_id: int, data: Dict) -> Optional[Dict]:
"""PUT /posts/{id} - Update post."""
post = self.posts.get(post_id)
if post:
post.update(data)
return post.to_dict()
return None
def delete_post(self, post_id: int) -> bool:
"""DELETE /posts/{id} - Delete post."""
if post_id in self.posts:
del self.posts[post_id]
return True
return False
# HATEOAS links
def add_links(self, resource: Dict, resource_type: str) -> Dict:
"""Add HATEOAS links to resource."""
resource['_links'] = {
'self': {'href': f'/{resource_type}s/{resource["id"]}'},
'collection': {'href': f'/{resource_type}s'}
}
if resource_type == 'post' and 'user_id' in resource:
resource['_links']['author'] = {'href': f'/users/{resource["user_id"]}'}
return resource
# Usage
def rest_example():
"""Demonstrate REST API concepts."""
api = RESTAPI()
# Create users
print("=== Create Users ===")
user1 = api.create_user({'name': 'Alice', 'email': 'alice@example.com'})
user2 = api.create_user({'name': 'Bob', 'email': 'bob@example.com'})
print(f"Created: {user1}")
print(f"Created: {user2}")
# Create posts
print("\n=== Create Posts ===")
post1 = api.create_post({'title': 'First Post', 'content': 'Hello World', 'user_id': user1['id']})
post2 = api.create_post({'title': 'Second Post', 'content': 'More content', 'user_id': user1['id']})
post3 = api.create_post({'title': 'Bob\'s Post', 'content': 'From Bob', 'user_id': user2['id']})
print(f"Created: {post1['title']}")
print(f"Created: {post2['title']}")
print(f"Created: {post3['title']}")
# List users
print("\n=== All Users ===")
for user in api.get_users():
print(f"User {user['id']}: {user['name']} - {user['email']}")
# List posts
print("\n=== All Posts ===")
for post in api.get_posts():
print(f"Post {post['id']}: {post['title']} by user {post['user_id']}")
# Filter posts by user
print("\n=== Alice's Posts ===")
for post in api.get_posts(user_id=user1['id']):
print(f" {post['title']}")
# Update user
print("\n=== Update User ===")
updated = api.update_user(user1['id'], {'email': 'alice@newdomain.com'})
print(f"Updated: {updated}")
# Get specific post
print("\n=== Get Post ===")
post = api.get_post(post1['id'])
print(f"Post {post['id']}: {post['title']}")
# Add HATEOAS links
print("\n=== Resource with Links ===")
linked_post = api.add_links(post, 'post')
print(json.dumps(linked_post, indent=2))
# Delete post
print("\n=== Delete Post ===")
api.delete_post(post2['id'])
print(f"Posts after deletion: {len(api.get_posts())}")
# rest_example()Flask is a lightweight WSGI web application framework:
Basic Flask Application:
# First install: pip install flask
from flask import Flask, request, jsonify, abort, redirect, url_for, render_template
import json
app = Flask(__name__)
# Basic routing
@app.route('/')
def home():
return '<h1>Welcome to Flask</h1><p>This is the home page.</p>'
@app.route('/about')
def about():
return '<h1>About</h1><p>This is a Flask application.</p>'
# Dynamic routes
@app.route('/user/<username>')
def profile(username):
return f'<h1>Profile</h1><p>Welcome, {username}!</p>'
@app.route('/post/<int:post_id>')
def show_post(post_id):
return f'<h1>Post {post_id}</h1>'
@app.route('/path/<path:subpath>')
def show_subpath(subpath):
return f'<h1>Subpath: {subpath}</h1>'
# Multiple methods
@app.route('/login', methods=['GET', 'POST'])
def login():
if request.method == 'POST':
return 'Processing login...'
else:
return '''
<form method="post">
<input type="text" name="username" placeholder="Username">
<input type="password" name="password" placeholder="Password">
<button type="submit">Login</button>
</form>
'''
# URL building
@app.route('/redirect-example')
def redirect_example():
return redirect(url_for('profile', username='guest'))
# JSON response
@app.route('/api/data')
def get_data():
data = {
'name': 'Flask API',
'version': '1.0',
'endpoints': ['/', '/about', '/user/<name>']
}
return jsonify(data)
# HTTP status codes
@app.route('/not-found')
def not_found():
abort(404)
@app.route('/created')
def created():
return 'Resource created', 201
# Error handlers
@app.errorhandler(404)
def page_not_found(error):
return jsonify({'error': 'Page not found'}), 404
@app.errorhandler(500)
def internal_error(error):
return jsonify({'error': 'Internal server error'}), 500
# Request data
@app.route('/search')
def search():
query = request.args.get('q', '')
page = request.args.get('page', 1, type=int)
return f'Searching for "{query}" on page {page}'
@app.route('/submit', methods=['POST'])
def submit():
# JSON data
if request.is_json:
data = request.get_json()
return jsonify({'received': data})
# Form data
elif request.form:
return jsonify({'form': dict(request.form)})
# Query parameters
else:
return jsonify({'args': dict(request.args)})
# Request headers
@app.route('/headers')
def headers():
user_agent = request.headers.get('User-Agent')
accept = request.headers.get('Accept')
return jsonify({
'user_agent': user_agent,
'accept': accept,
'all_headers': dict(request.headers)
})
# Response customization
@app.route('/custom-response')
def custom_response():
response = jsonify({'message': 'Custom response'})
response.status_code = 201
response.headers['X-Custom-Header'] = 'CustomValue'
response.set_cookie('session_id', 'abc123', max_age=3600)
return responseAdvanced Routing:
from flask import Flask, request, url_for
from werkzeug.routing import BaseConverter
app = Flask(__name__)
# Custom URL converters
class ListConverter(BaseConverter):
"""Convert comma-separated list to Python list."""
def to_python(self, value):
return value.split(',')
def to_url(self, values):
return ','.join(str(v) for v in values)
class RegexConverter(BaseConverter):
"""Match URLs using regular expressions."""
def __init__(self, url_map, *items):
super().__init__(url_map)
self.regex = items[0]
# Register custom converters
app.url_map.converters['list'] = ListConverter
app.url_map.converters['regex'] = RegexConverter
@app.route('/items/<list:names>')
def show_items(names):
return f'Items: {", ".join(names)}'
@app.route('/<regex("[a-z]{3}-[0-9]{3}"):code>')
def show_code(code):
return f'Code: {code}'
# Blueprints for modular applications
from flask import Blueprint
# Create blueprint
admin_bp = Blueprint('admin', __name__, url_prefix='/admin')
@admin_bp.route('/')
def admin_index():
return 'Admin Dashboard'
@admin_bp.route('/users')
def admin_users():
return 'User Management'
@admin_bp.route('/settings')
def admin_settings():
return 'Admin Settings'
# Register blueprint
app.register_blueprint(admin_bp)
# Another blueprint for API
api_bp = Blueprint('api', __name__, url_prefix='/api/v1')
@api_bp.route('/status')
def api_status():
return {'status': 'ok', 'version': '1.0'}
@api_bp.route('/users/<int:user_id>')
def api_user(user_id):
return {'user_id': user_id, 'name': f'User {user_id}'}
app.register_blueprint(api_bp)
# Route with conditions
@app.route('/blog', defaults={'page': 1})
@app.route('/blog/page/<int:page>')
def blog(page):
return f'Blog page {page}'
# HTTP method dispatching
class MethodView:
"""Class-based view with method dispatching."""
def dispatch_request(self, *args, **kwargs):
method = request.method.lower()
if hasattr(self, method):
return getattr(self, method)(*args, **kwargs)
return 'Method not allowed', 405
class UserAPI(MethodView):
def get(self, user_id=None):
if user_id:
return {'user': f'User {user_id}'}
return {'users': ['User1', 'User2', 'User3']}
def post(self):
data = request.get_json()
return {'created': data, 'id': 4}, 201
def put(self, user_id):
data = request.get_json()
return {'updated': data, 'id': user_id}
def delete(self, user_id):
return {'deleted': user_id}
# Register class-based view
user_view = UserAPI.as_view('user_api')
app.add_url_rule('/api/users', defaults={'user_id': None}, view_func=user_view, methods=['GET'])
app.add_url_rule('/api/users', view_func=user_view, methods=['POST'])
app.add_url_rule('/api/users/<int:user_id>', view_func=user_view, methods=['GET', 'PUT', 'DELETE'])Jinja2 Templates:
from flask import Flask, render_template, request, session, g
import datetime
app = Flask(__name__)
app.secret_key = 'your-secret-key-here'
# Template filters
@app.template_filter('format_date')
def format_date_filter(date, format='%Y-%m-%d'):
"""Custom template filter."""
return date.strftime(format)
@app.template_filter('pluralize')
def pluralize_filter(count, singular, plural=None):
"""Pluralize word based on count."""
if count == 1:
return f"1 {singular}"
return f"{count} {plural or singular + 's'}"
# Template context processor
@app.context_processor
def inject_current_year():
"""Inject current year into all templates."""
return {'current_year': datetime.datetime.now().year}
@app.context_processor
def inject_app_info():
"""Inject application info."""
return {
'app_name': 'Flask Demo',
'app_version': '1.0.0'
}
# Routes with templates
@app.route('/')
def index():
return render_template('index.html',
title='Home',
message='Welcome to Flask Templates!')
@app.route('/user/<name>')
def user_profile(name):
user = {
'name': name,
'email': f'{name.lower()}@example.com',
'join_date': datetime.datetime(2023, 1, 15),
'is_admin': name.lower() == 'admin',
'skills': ['Python', 'Flask', 'HTML', 'CSS'] if name == 'Alice' else ['JavaScript', 'React'],
'posts': [
{'title': 'First Post', 'date': datetime.datetime(2024, 1, 1), 'comments': 5},
{'title': 'Second Post', 'date': datetime.datetime(2024, 1, 15), 'comments': 3},
{'title': 'Third Post', 'date': datetime.datetime(2024, 2, 1), 'comments': 8}
] if name == 'Alice' else []
}
return render_template('profile.html', user=user)
@app.route('/loop-example')
def loop_example():
items = [
{'name': 'Item 1', 'price': 10.99, 'in_stock': True},
{'name': 'Item 2', 'price': 24.99, 'in_stock': False},
{'name': 'Item 3', 'price': 5.49, 'in_stock': True},
{'name': 'Item 4', 'price': 99.99, 'in_stock': True},
{'name': 'Item 5', 'price': 15.00, 'in_stock': False}
]
return render_template('loop.html', items=items)
@app.route('/form', methods=['GET', 'POST'])
def form_example():
if request.method == 'POST':
name = request.form.get('name')
email = request.form.get('email')
message = request.form.get('message')
# Store in session
session['last_submission'] = {
'name': name,
'email': email,
'timestamp': datetime.datetime.now().isoformat()
}
return render_template('form_result.html',
name=name,
email=email,
message=message)
return render_template('form.html')
@app.route('/inheritance')
def inheritance_example():
return render_template('child.html',
page_title='Template Inheritance',
content='This content comes from the child template.')
# Template files (create templates directory with these files)
"""
templates/base.html:
<!DOCTYPE html>
<html>
<head>
<title>{% block title %}{{ app_name }}{% endblock %}</title>
<link rel="stylesheet" href="{{ url_for('static', filename='style.css') }}">
{% block head %}{% endblock %}
</head>
<body>
<header>
<nav>
<a href="{{ url_for('index') }}">Home</a>
<a href="{{ url_for('form_example') }}">Form</a>
<a href="{{ url_for('loop_example') }}">Loop Demo</a>
</nav>
</header>
<main>
{% with messages = get_flashed_messages() %}
{% if messages %}
<ul class="flashes">
{% for message in messages %}
<li>{{ message }}</li>
{% endfor %}
</ul>
{% endif %}
{% endwith %}
{% block content %}{% endblock %}
</main>
<footer>
<p>© {{ current_year }} {{ app_name }} v{{ app_version }}</p>
</footer>
</body>
</html>
templates/index.html:
{% extends "base.html" %}
{% block title %}Home - {{ super() }}{% endblock %}
{% block content %}
<h1>{{ message }}</h1>
<p>This is the home page using template inheritance.</p>
<h2>Features</h2>
<ul>
<li>Template inheritance</li>
<li>Custom filters</li>
<li>Context processors</li>
<li>Form handling</li>
<li>Session management</li>
</ul>
{% endblock %}
templates/profile.html:
{% extends "base.html" %}
{% block title %}Profile - {{ user.name }}{% endblock %}
{% block content %}
<h1>{{ user.name }}'s Profile</h1>
<div class="user-info">
<p><strong>Email:</strong> {{ user.email }}</p>
<p><strong>Member since:</strong> {{ user.join_date|format_date('%B %d, %Y') }}</p>
<p><strong>Status:</strong>
{% if user.is_admin %}
<span class="admin-badge">Administrator</span>
{% else %}
Regular User
{% endif %}
</p>
</div>
<h2>Skills</h2>
{% if user.skills %}
<ul>
{% for skill in user.skills %}
<li>{{ skill }}</li>
{% endfor %}
</ul>
{% else %}
<p>No skills listed.</p>
{% endif %}
<h2>Recent Posts</h2>
{% if user.posts %}
<table>
<thead>
<tr>
<th>Title</th>
<th>Date</th>
<th>Comments</th>
</tr>
</thead>
<tbody>
{% for post in user.posts %}
<tr>
<td>{{ post.title }}</td>
<td>{{ post.date|format_date('%Y-%m-%d') }}</td>
<td>{{ post.comments }}</td>
</tr>
{% endfor %}
</tbody>
</table>
<p>Total posts: {{ user.posts|length|pluralize('post') }}</p>
{% else %}
<p>No posts yet.</p>
{% endif %}
{% endblock %}
templates/loop.html:
{% extends "base.html" %}
{% block title %}Loop Demo{% endblock %}
{% block content %}
<h1>Loop Demonstration</h1>
<h2>Items</h2>
<table border="1">
<thead>
<tr>
<th>Index</th>
<th>Name</th>
<th>Price</th>
<th>Status</th>
<th>First?</th>
<th>Last?</th>
</tr>
</thead>
<tbody>
{% for item in items %}
<tr style="background-color: {{ '#ccffcc' if item.in_stock else '#ffcccc' }}">
<td>{{ loop.index }}</td>
<td>{{ item.name }}</td>
<td>${{ "%.2f"|format(item.price) }}</td>
<td>{{ 'In Stock' if item.in_stock else 'Out of Stock' }}</td>
<td>{{ 'Yes' if loop.first else 'No' }}</td>
<td>{{ 'Yes' if loop.last else 'No' }}</td>
</tr>
{% else %}
<tr>
<td colspan="6">No items found.</td>
</tr>
{% endfor %}
</tbody>
</table>
<h2>Loop Variables</h2>
<ul>
<li><code>loop.index</code>: Current iteration (1-indexed)</li>
<li><code>loop.index0</code>: Current iteration (0-indexed)</li>
<li><code>loop.first</code>: True if first iteration</li>
<li><code>loop.last</code>: True if last iteration</li>
<li><code>loop.length</code>: Total number of items</li>
<li><code>loop.cycle</code>: Cycle between values</li>
</ul>
<h3>Cycle Example</h3>
<div style="display: flex; gap: 10px;">
{% for i in range(5) %}
<div style="width: 50px; height: 50px; background-color: {{ loop.cycle('red', 'green', 'blue') }};"></div>
{% endfor %}
</div>
{% endblock %}
templates/form.html:
{% extends "base.html" %}
{% block title %}Form Example{% endblock %}
{% block content %}
<h1>Contact Form</h1>
<form method="post">
<div>
<label for="name">Name:</label>
<input type="text" id="name" name="name" required>
</div>
<div>
<label for="email">Email:</label>
<input type="email" id="email" name="email" required>
</div>
<div>
<label for="message">Message:</label>
<textarea id="message" name="message" rows="5" required></textarea>
</div>
<button type="submit">Send Message</button>
</form>
{% if session.last_submission %}
<div class="last-submission">
<h3>Last Submission</h3>
<p>Name: {{ session.last_submission.name }}</p>
<p>Time: {{ session.last_submission.timestamp }}</p>
</div>
{% endif %}
{% endblock %}
templates/form_result.html:
{% extends "base.html" %}
{% block title %}Form Submitted{% endblock %}
{% block content %}
<h1>Thank You, {{ name }}!</h1>
<p>We received your message:</p>
<blockquote>{{ message }}</blockquote>
<p>A confirmation email will be sent to {{ email }}.</p>
<p><a href="{{ url_for('form_example') }}">Submit another message</a></p>
{% endblock %}
templates/child.html:
{% extends "base.html" %}
{% block title %}{{ page_title }} - {{ super() }}{% endblock %}
{% block head %}
<style>
.highlight {
background-color: yellow;
padding: 10px;
border-radius: 5px;
}
</style>
{% endblock %}
{% block content %}
<h1>{{ page_title }}</h1>
<div class="highlight">
{{ content }}
</div>
<h2>Super Block Example</h2>
<p>This content is from the child template, but we can also include the parent's content:</p>
<div style="border: 1px solid #ccc; padding: 10px;">
<h3>Parent's content:</h3>
{{ super() }}
</div>
{% endblock %}
"""Building RESTful APIs:
from flask import Flask, request, jsonify, url_for, abort
from flask.views import MethodView
from functools import wraps
import jwt
import datetime
from typing import Dict, List, Optional
app = Flask(__name__)
app.config['SECRET_KEY'] = 'your-secret-key-here'
app.config['TOKEN_EXPIRATION'] = 3600 # 1 hour
# In-memory database
class Database:
def __init__(self):
self.users = {}
self.posts = {}
self.next_user_id = 1
self.next_post_id = 1
def get_user(self, user_id: int) -> Optional[Dict]:
return self.users.get(user_id)
def get_user_by_username(self, username: str) -> Optional[Dict]:
for user in self.users.values():
if user['username'] == username:
return user
return None
def get_users(self) -> List[Dict]:
return list(self.users.values())
def create_user(self, user_data: Dict) -> Dict:
user_id = self.next_user_id
user = {
'id': user_id,
'username': user_data['username'],
'email': user_data['email'],
'password': user_data['password'], # In real app, hash this!
'created_at': datetime.datetime.now().isoformat()
}
self.users[user_id] = user
self.next_user_id += 1
return user
def update_user(self, user_id: int, user_data: Dict) -> Optional[Dict]:
if user_id in self.users:
self.users[user_id].update(user_data)
return self.users[user_id]
return None
def delete_user(self, user_id: int) -> bool:
if user_id in self.users:
del self.users[user_id]
return True
return False
def get_posts(self, user_id: Optional[int] = None) -> List[Dict]:
posts = list(self.posts.values())
if user_id:
posts = [p for p in posts if p['user_id'] == user_id]
return posts
def get_post(self, post_id: int) -> Optional[Dict]:
return self.posts.get(post_id)
def create_post(self, post_data: Dict) -> Optional[Dict]:
user_id = post_data.get('user_id')
if user_id not in self.users:
return None
post_id = self.next_post_id
post = {
'id': post_id,
'title': post_data['title'],
'content': post_data['content'],
'user_id': user_id,
'created_at': datetime.datetime.now().isoformat(),
'updated_at': datetime.datetime.now().isoformat()
}
self.posts[post_id] = post
self.next_post_id += 1
return post
def update_post(self, post_id: int, post_data: Dict) -> Optional[Dict]:
if post_id in self.posts:
self.posts[post_id].update(post_data)
self.posts[post_id]['updated_at'] = datetime.datetime.now().isoformat()
return self.posts[post_id]
return None
def delete_post(self, post_id: int) -> bool:
if post_id in self.posts:
del self.posts[post_id]
return True
return False
db = Database()
# Authentication decorator
def token_required(f):
@wraps(f)
def decorated(*args, **kwargs):
token = request.headers.get('Authorization')
if not token:
return jsonify({'error': 'Token is missing'}), 401
try:
# Remove 'Bearer ' prefix if present
if token.startswith('Bearer '):
token = token[7:]
data = jwt.decode(token, app.config['SECRET_KEY'], algorithms=['HS256'])
current_user = db.get_user(data['user_id'])
if not current_user:
return jsonify({'error': 'User not found'}), 401
except jwt.ExpiredSignatureError:
return jsonify({'error': 'Token has expired'}), 401
except jwt.InvalidTokenError:
return jsonify({'error': 'Invalid token'}), 401
return f(current_user, *args, **kwargs)
return decorated
# Authentication endpoints
@app.route('/api/auth/register', methods=['POST'])
def register():
"""Register new user."""
data = request.get_json()
if not data or not data.get('username') or not data.get('password') or not data.get('email'):
return jsonify({'error': 'Username, email, and password required'}), 400
# Check if user exists
if db.get_user_by_username(data['username']):
return jsonify({'error': 'Username already exists'}), 409
# Create user
user = db.create_user({
'username': data['username'],
'email': data['email'],
'password': data['password'] # In production, hash this!
})
# Generate token
token = jwt.encode({
'user_id': user['id'],
'username': user['username'],
'exp': datetime.datetime.utcnow() + datetime.timedelta(seconds=app.config['TOKEN_EXPIRATION'])
}, app.config['SECRET_KEY'], algorithm='HS256')
return jsonify({
'message': 'User created successfully',
'token': token,
'user': {
'id': user['id'],
'username': user['username'],
'email': user['email']
}
}), 201
@app.route('/api/auth/login', methods=['POST'])
def login():
"""Login user."""
data = request.get_json()
if not data or not data.get('username') or not data.get('password'):
return jsonify({'error': 'Username and password required'}), 400
user = db.get_user_by_username(data['username'])
# Check password (in production, use proper password verification)
if not user or user['password'] != data['password']:
return jsonify({'error': 'Invalid credentials'}), 401
# Generate token
token = jwt.encode({
'user_id': user['id'],
'username': user['username'],
'exp': datetime.datetime.utcnow() + datetime.timedelta(seconds=app.config['TOKEN_EXPIRATION'])
}, app.config['SECRET_KEY'], algorithm='HS256')
return jsonify({
'message': 'Login successful',
'token': token,
'user': {
'id': user['id'],
'username': user['username'],
'email': user['email']
}
})
@app.route('/api/auth/refresh', methods=['POST'])
@token_required
def refresh_token(current_user):
"""Refresh authentication token."""
new_token = jwt.encode({
'user_id': current_user['id'],
'username': current_user['username'],
'exp': datetime.datetime.utcnow() + datetime.timedelta(seconds=app.config['TOKEN_EXPIRATION'])
}, app.config['SECRET_KEY'], algorithm='HS256')
return jsonify({'token': new_token})
# User endpoints
@app.route('/api/users', methods=['GET'])
@token_required
def get_users(current_user):
"""Get all users."""
users = db.get_users()
return jsonify([{
'id': u['id'],
'username': u['username'],
'email': u['email'],
'created_at': u['created_at']
} for u in users])
@app.route('/api/users/<int:user_id>', methods=['GET'])
@token_required
def get_user(current_user, user_id):
"""Get specific user."""
user = db.get_user(user_id)
if not user:
return jsonify({'error': 'User not found'}), 404
return jsonify({
'id': user['id'],
'username': user['username'],
'email': user['email'],
'created_at': user['created_at']
})
@app.route('/api/users/<int:user_id>', methods=['PUT'])
@token_required
def update_user(current_user, user_id):
"""Update user."""
# Only allow users to update their own profile
if current_user['id'] != user_id:
return jsonify({'error': 'Cannot update other users'}), 403
data = request.get_json()
allowed_fields = ['email'] # Don't allow username/password updates via this endpoint
update_data = {}
for field in allowed_fields:
if field in data:
update_data[field] = data[field]
updated_user = db.update_user(user_id, update_data)
if not updated_user:
return jsonify({'error': 'User not found'}), 404
return jsonify({
'id': updated_user['id'],
'username': updated_user['username'],
'email': updated_user['email']
})
@app.route('/api/users/<int:user_id>', methods=['DELETE'])
@token_required
def delete_user(current_user, user_id):
"""Delete user."""
# Only allow users to delete their own account
if current_user['id'] != user_id:
return jsonify({'error': 'Cannot delete other users'}), 403
if db.delete_user(user_id):
return jsonify({'message': 'User deleted successfully'})
return jsonify({'error': 'User not found'}), 404
# Post endpoints
@app.route('/api/posts', methods=['GET'])
def get_posts():
"""Get all posts."""
user_id = request.args.get('user_id', type=int)
posts = db.get_posts(user_id)
# Add author information
result = []
for post in posts:
author = db.get_user(post['user_id'])
post_data = {
'id': post['id'],
'title': post['title'],
'content': post['content'],
'created_at': post['created_at'],
'updated_at': post['updated_at'],
'author': {
'id': author['id'],
'username': author['username']
} if author else None
}
result.append(post_data)
return jsonify({
'posts': result,
'count': len(result)
})
@app.route('/api/posts/<int:post_id>', methods=['GET'])
def get_post(post_id):
"""Get specific post."""
post = db.get_post(post_id)
if not post:
return jsonify({'error': 'Post not found'}), 404
author = db.get_user(post['user_id'])
return jsonify({
'id': post['id'],
'title': post['title'],
'content': post['content'],
'created_at': post['created_at'],
'updated_at': post['updated_at'],
'author': {
'id': author['id'],
'username': author['username']
} if author else None
})
@app.route('/api/posts', methods=['POST'])
@token_required
def create_post(current_user):
"""Create new post."""
data = request.get_json()
if not data or not data.get('title') or not data.get('content'):
return jsonify({'error': 'Title and content required'}), 400
post = db.create_post({
'title': data['title'],
'content': data['content'],
'user_id': current_user['id']
})
if not post:
return jsonify({'error': 'Could not create post'}), 500
return jsonify({
'message': 'Post created successfully',
'post': {
'id': post['id'],
'title': post['title'],
'content': post['content']
}
}), 201
@app.route('/api/posts/<int:post_id>', methods=['PUT'])
@token_required
def update_post(current_user, post_id):
"""Update post."""
post = db.get_post(post_id)
if not post:
return jsonify({'error': 'Post not found'}), 404
# Check ownership
if post['user_id'] != current_user['id']:
return jsonify({'error': 'Cannot update other users\' posts'}), 403
data = request.get_json()
update_data = {}
if 'title' in data:
update_data['title'] = data['title']
if 'content' in data:
update_data['content'] = data['content']
updated_post = db.update_post(post_id, update_data)
return jsonify({
'message': 'Post updated successfully',
'post': {
'id': updated_post['id'],
'title': updated_post['title'],
'content': updated_post['content'],
'updated_at': updated_post['updated_at']
}
})
@app.route('/api/posts/<int:post_id>', methods=['DELETE'])
@token_required
def delete_post(current_user, post_id):
"""Delete post."""
post = db.get_post(post_id)
if not post:
return jsonify({'error': 'Post not found'}), 404
# Check ownership
if post['user_id'] != current_user['id']:
return jsonify({'error': 'Cannot delete other users\' posts'}), 403
if db.delete_post(post_id):
return jsonify({'message': 'Post deleted successfully'})
return jsonify({'error': 'Could not delete post'}), 500
# API documentation
@app.route('/api')
def api_docs():
"""API documentation."""
return jsonify({
'name': 'Flask REST API',
'version': '1.0',
'description': 'Example REST API built with Flask',
'endpoints': {
'Authentication': {
'POST /api/auth/register': 'Register new user',
'POST /api/auth/login': 'Login user',
'POST /api/auth/refresh': 'Refresh token (auth required)'
},
'Users': {
'GET /api/users': 'Get all users (auth required)',
'GET /api/users/<id>': 'Get specific user (auth required)',
'PUT /api/users/<id>': 'Update user (auth required, own user only)',
'DELETE /api/users/<id>': 'Delete user (auth required, own user only)'
},
'Posts': {
'GET /api/posts': 'Get all posts',
'GET /api/posts/<id>': 'Get specific post',
'POST /api/posts': 'Create post (auth required)',
'PUT /api/posts/<id>': 'Update post (auth required, own posts only)',
'DELETE /api/posts/<id>': 'Delete post (auth required, own posts only)'
}
}
})
if __name__ == '__main__':
app.run(debug=True, port=5000)Comprehensive Authentication System:
from flask import Flask, request, jsonify, session, redirect, url_for, render_template, flash
from flask_bcrypt import Bcrypt
from flask_login import LoginManager, UserMixin, login_user, logout_user, login_required, current_user
from functools import wraps
import secrets
from datetime import datetime, timedelta
import re
app = Flask(__name__)
app.config['SECRET_KEY'] = secrets.token_hex(32)
app.config['REMEMBER_COOKIE_DURATION'] = timedelta(days=30)
app.config['SESSION_PROTECTION'] = 'strong'
bcrypt = Bcrypt(app)
login_manager = LoginManager(app)
login_manager.login_view = 'login'
login_manager.login_message = 'Please log in to access this page.'
login_manager.login_message_category = 'info'
# In-memory user database (use real database in production)
users = {}
password_reset_tokens = {}
email_verification_tokens = {}
class User(UserMixin):
"""User class for Flask-Login."""
def __init__(self, id, username, email, password_hash, is_active=True, is_admin=False):
self.id = id
self.username = username
self.email = email
self.password_hash = password_hash
self.is_active = is_active
self.is_admin = is_admin
self.created_at = datetime.now()
self.last_login = None
def get_id(self):
return str(self.id)
def verify_password(self, password):
return bcrypt.check_password_hash(self.password_hash, password)
@login_manager.user_loader
def load_user(user_id):
"""Load user by ID."""
return users.get(int(user_id))
# Decorator for admin-only routes
def admin_required(f):
@wraps(f)
def decorated_function(*args, **kwargs):
if not current_user.is_authenticated or not current_user.is_admin:
flash('You need admin privileges to access this page.', 'danger')
return redirect(url_for('index'))
return f(*args, **kwargs)
return decorated_function
# Routes
@app.route('/')
def index():
return render_template('index.html')
@app.route('/register', methods=['GET', 'POST'])
def register():
"""User registration."""
if request.method == 'POST':
username = request.form.get('username')
email = request.form.get('email')
password = request.form.get('password')
confirm_password = request.form.get('confirm_password')
# Validation
errors = []
if not username or len(username) < 3:
errors.append('Username must be at least 3 characters long.')
if not re.match(r'^[a-zA-Z0-9_]+$', username):
errors.append('Username can only contain letters, numbers, and underscores.')
if not re.match(r'^[^@]+@[^@]+\.[^@]+$', email):
errors.append('Invalid email address.')
if len(password) < 8:
errors.append('Password must be at least 8 characters long.')
if not re.search(r'[A-Z]', password):
errors.append('Password must contain at least one uppercase letter.')
if not re.search(r'[a-z]', password):
errors.append('Password must contain at least one lowercase letter.')
if not re.search(r'[0-9]', password):
errors.append('Password must contain at least one number.')
if password != confirm_password:
errors.append('Passwords do not match.')
# Check if user exists
for user in users.values():
if user.username == username:
errors.append('Username already taken.')
break
for user in users.values():
if user.email == email:
errors.append('Email already registered.')
break
if errors:
for error in errors:
flash(error, 'danger')
return render_template('register.html')
# Create user
user_id = len(users) + 1
password_hash = bcrypt.generate_password_hash(password).decode('utf-8')
# First user is admin
is_admin = (user_id == 1)
user = User(user_id, username, email, password_hash, is_active=False, is_admin=is_admin)
users[user_id] = user
# Generate email verification token
token = secrets.token_urlsafe(32)
email_verification_tokens[token] = {
'user_id': user_id,
'expires': datetime.now() + timedelta(hours=24)
}
# In production, send email with verification link
verify_url = url_for('verify_email', token=token, _external=True)
print(f"Verification email would be sent to {email}: {verify_url}")
flash('Registration successful! Please check your email to verify your account.', 'success')
return redirect(url_for('login'))
return render_template('register.html')
@app.route('/verify-email/<token>')
def verify_email(token):
"""Verify email address."""
token_data = email_verification_tokens.get(token)
if not token_data or token_data['expires'] < datetime.now():
flash('Invalid or expired verification token.', 'danger')
return redirect(url_for('login'))
user = users.get(token_data['user_id'])
if user:
user.is_active = True
del email_verification_tokens[token]
flash('Email verified successfully! You can now log in.', 'success')
else:
flash('User not found.', 'danger')
return redirect(url_for('login'))
@app.route('/login', methods=['GET', 'POST'])
def login():
"""User login."""
if request.method == 'POST':
username = request.form.get('username')
password = request.form.get('password')
remember = request.form.get('remember', False)
# Find user by username or email
user = None
for u in users.values():
if u.username == username or u.email == username:
user = u
break
if user and user.verify_password(password):
if not user.is_active:
flash('Please verify your email before logging in.', 'warning')
return render_template('login.html')
login_user(user, remember=remember)
user.last_login = datetime.now()
flash(f'Welcome back, {user.username}!', 'success')
next_page = request.args.get('next')
return redirect(next_page) if next_page else redirect(url_for('profile'))
else:
flash('Invalid username or password.', 'danger')
return render_template('login.html')
@app.route('/logout')
@login_required
def logout():
"""User logout."""
logout_user()
flash('You have been logged out.', 'info')
return redirect(url_for('index'))
@app.route('/profile')
@login_required
def profile():
"""User profile."""
return render_template('profile.html', user=current_user)
@app.route('/profile/edit', methods=['GET', 'POST'])
@login_required
def edit_profile():
"""Edit user profile."""
if request.method == 'POST':
email = request.form.get('email')
# Validate email
if not re.match(r'^[^@]+@[^@]+\.[^@]+$', email):
flash('Invalid email address.', 'danger')
return render_template('edit_profile.html')
# Check if email is taken by another user
for user in users.values():
if user.email == email and user.id != current_user.id:
flash('Email already registered by another user.', 'danger')
return render_template('edit_profile.html')
current_user.email = email
flash('Profile updated successfully!', 'success')
return redirect(url_for('profile'))
return render_template('edit_profile.html')
@app.route('/change-password', methods=['GET', 'POST'])
@login_required
def change_password():
"""Change user password."""
if request.method == 'POST':
current_password = request.form.get('current_password')
new_password = request.form.get('new_password')
confirm_password = request.form.get('confirm_password')
# Verify current password
if not current_user.verify_password(current_password):
flash('Current password is incorrect.', 'danger')
return render_template('change_password.html')
# Validate new password
errors = []
if len(new_password) < 8:
errors.append('Password must be at least 8 characters long.')
if not re.search(r'[A-Z]', new_password):
errors.append('Password must contain at least one uppercase letter.')
if not re.search(r'[a-z]', new_password):
errors.append('Password must contain at least one lowercase letter.')
if not re.search(r'[0-9]', new_password):
errors.append('Password must contain at least one number.')
if new_password != confirm_password:
errors.append('Passwords do not match.')
if errors:
for error in errors:
flash(error, 'danger')
return render_template('change_password.html')
# Update password
current_user.password_hash = bcrypt.generate_password_hash(new_password).decode('utf-8')
flash('Password changed successfully!', 'success')
return redirect(url_for('profile'))
return render_template('change_password.html')
@app.route('/forgot-password', methods=['GET', 'POST'])
def forgot_password():
"""Request password reset."""
if request.method == 'POST':
email = request.form.get('email')
# Find user by email
user = None
for u in users.values():
if u.email == email:
user = u
break
if user:
# Generate reset token
token = secrets.token_urlsafe(32)
password_reset_tokens[token] = {
'user_id': user.id,
'expires': datetime.now() + timedelta(hours=1)
}
# In production, send email with reset link
reset_url = url_for('reset_password', token=token, _external=True)
print(f"Password reset email would be sent to {email}: {reset_url}")
flash('Password reset instructions have been sent to your email.', 'info')
else:
# Don't reveal that email doesn't exist
flash('If that email is registered, you will receive reset instructions.', 'info')
return redirect(url_for('login'))
return render_template('forgot_password.html')
@app.route('/reset-password/<token>', methods=['GET', 'POST'])
def reset_password(token):
"""Reset password with token."""
token_data = password_reset_tokens.get(token)
if not token_data or token_data['expires'] < datetime.now():
flash('Invalid or expired reset token.', 'danger')
return redirect(url_for('forgot_password'))
if request.method == 'POST':
new_password = request.form.get('new_password')
confirm_password = request.form.get('confirm_password')
# Validate password
errors = []
if len(new_password) < 8:
errors.append('Password must be at least 8 characters long.')
if not re.search(r'[A-Z]', new_password):
errors.append('Password must contain at least one uppercase letter.')
if not re.search(r'[a-z]', new_password):
errors.append('Password must contain at least one lowercase letter.')
if not re.search(r'[0-9]', new_password):
errors.append('Password must contain at least one number.')
if new_password != confirm_password:
errors.append('Passwords do not match.')
if errors:
for error in errors:
flash(error, 'danger')
return render_template('reset_password.html', token=token)
# Update password
user = users.get(token_data['user_id'])
if user:
user.password_hash = bcrypt.generate_password_hash(new_password).decode('utf-8')
del password_reset_tokens[token]
flash('Password reset successfully! You can now log in.', 'success')
else:
flash('User not found.', 'danger')
return redirect(url_for('login'))
return render_template('reset_password.html', token=token)
@app.route('/admin')
@login_required
@admin_required
def admin_panel():
"""Admin panel."""
return render_template('admin.html', users=users.values())
@app.route('/admin/users/<int:user_id>/toggle-active')
@login_required
@admin_required
def toggle_user_active(user_id):
"""Toggle user active status (admin only)."""
user = users.get(user_id)
if user and user.id != current_user.id: # Can't deactivate yourself
user.is_active = not user.is_active
status = 'activated' if user.is_active else 'deactivated'
flash(f'User {user.username} has been {status}.', 'success')
elif user and user.id == current_user.id:
flash('You cannot deactivate your own account.', 'danger')
else:
flash('User not found.', 'danger')
return redirect(url_for('admin_panel'))
@app.route('/admin/users/<int:user_id>/delete')
@login_required
@admin_required
def delete_user(user_id):
"""Delete user (admin only)."""
user = users.get(user_id)
if user and user.id != current_user.id: # Can't delete yourself
del users[user_id]
flash(f'User {user.username} has been deleted.', 'success')
elif user and user.id == current_user.id:
flash('You cannot delete your own account.', 'danger')
else:
flash('User not found.', 'danger')
return redirect(url_for('admin_panel'))
# Templates (create these in templates directory)
"""
templates/base.html:
<!DOCTYPE html>
<html>
<head>
<title>{% block title %}Flask Auth{% endblock %}</title>
<link href="https://cdn.jsdelivr.net/npm/bootstrap@5.1.3/dist/css/bootstrap.min.css" rel="stylesheet">
</head>
<body>
<nav class="navbar navbar-expand-lg navbar-dark bg-dark">
<div class="container">
<a class="navbar-brand" href="{{ url_for('index') }}">Flask Auth</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarNav">
<span class="navbar-toggler-icon"></span>
</button>
<div class="collapse navbar-collapse" id="navbarNav">
<ul class="navbar-nav ms-auto">
{% if current_user.is_authenticated %}
<li class="nav-item">
<a class="nav-link" href="{{ url_for('profile') }}">Profile</a>
</li>
{% if current_user.is_admin %}
<li class="nav-item">
<a class="nav-link" href="{{ url_for('admin_panel') }}">Admin</a>
</li>
{% endif %}
<li class="nav-item">
<a class="nav-link" href="{{ url_for('logout') }}">Logout</a>
</li>
{% else %}
<li class="nav-item">
<a class="nav-link" href="{{ url_for('login') }}">Login</a>
</li>
<li class="nav-item">
<a class="nav-link" href="{{ url_for('register') }}">Register</a>
</li>
{% endif %}
</ul>
</div>
</div>
</nav>
<div class="container mt-4">
{% with messages = get_flashed_messages(with_categories=true) %}
{% if messages %}
{% for category, message in messages %}
<div class="alert alert-{{ category }} alert-dismissible fade show" role="alert">
{{ message }}
<button type="button" class="btn-close" data-bs-dismiss="alert"></button>
</div>
{% endfor %}
{% endif %}
{% endwith %}
{% block content %}{% endblock %}
</div>
<script src="https://cdn.jsdelivr.net/npm/bootstrap@5.1.3/dist/js/bootstrap.bundle.min.js"></script>
</body>
</html>
templates/index.html:
{% extends "base.html" %}
{% block title %}Home{% endblock %}
{% block content %}
<div class="jumbotron">
<h1 class="display-4">Welcome to Flask Authentication</h1>
<p class="lead">A complete authentication system built with Flask.</p>
<hr class="my-4">
<p>Features include:</p>
<ul>
<li>User registration with email verification</li>
<li>Secure password hashing with bcrypt</li>
<li>Login with remember me functionality</li>
<li>Password reset via email</li>
<li>Profile management</li>
<li>Admin panel for user management</li>
<li>Session protection</li>
</ul>
{% if not current_user.is_authenticated %}
<a class="btn btn-primary btn-lg" href="{{ url_for('register') }}" role="button">Register</a>
<a class="btn btn-success btn-lg" href="{{ url_for('login') }}" role="button">Login</a>
{% endif %}
</div>
{% endblock %}
templates/register.html:
{% extends "base.html" %}
{% block title %}Register{% endblock %}
{% block content %}
<div class="row justify-content-center">
<div class="col-md-6">
<div class="card">
<div class="card-header">
<h3>Register</h3>
</div>
<div class="card-body">
<form method="post">
<div class="mb-3">
<label for="username" class="form-label">Username</label>
<input type="text" class="form-control" id="username" name="username" required>
<div class="form-text">Letters, numbers, and underscores only (min. 3 characters).</div>
</div>
<div class="mb-3">
<label for="email" class="form-label">Email</label>
<input type="email" class="form-control" id="email" name="email" required>
</div>
<div class="mb-3">
<label for="password" class="form-label">Password</label>
<input type="password" class="form-control" id="password" name="password" required>
<div class="form-text">
At least 8 characters with uppercase, lowercase, and number.
</div>
</div>
<div class="mb-3">
<label for="confirm_password" class="form-label">Confirm Password</label>
<input type="password" class="form-control" id="confirm_password" name="confirm_password" required>
</div>
<button type="submit" class="btn btn-primary">Register</button>
</form>
</div>
<div class="card-footer text-center">
Already have an account? <a href="{{ url_for('login') }}">Login here</a>
</div>
</div>
</div>
</div>
{% endblock %}
templates/login.html:
{% extends "base.html" %}
{% block title %}Login{% endblock %}
{% block content %}
<div class="row justify-content-center">
<div class="col-md-6">
<div class="card">
<div class="card-header">
<h3>Login</h3>
</div>
<div class="card-body">
<form method="post">
<div class="mb-3">
<label for="username" class="form-label">Username or Email</label>
<input type="text" class="form-control" id="username" name="username" required>
</div>
<div class="mb-3">
<label for="password" class="form-label">Password</label>
<input type="password" class="form-control" id="password" name="password" required>
</div>
<div class="mb-3 form-check">
<input type="checkbox" class="form-check-input" id="remember" name="remember">
<label class="form-check-label" for="remember">Remember me</label>
</div>
<button type="submit" class="btn btn-primary">Login</button>
</form>
</div>
<div class="card-footer text-center">
<a href="{{ url_for('forgot_password') }}">Forgot Password?</a><br>
Don't have an account? <a href="{{ url_for('register') }}">Register here</a>
</div>
</div>
</div>
</div>
{% endblock %}
templates/profile.html:
{% extends "base.html" %}
{% block title %}Profile{% endblock %}
{% block content %}
<div class="row">
<div class="col-md-4">
<div class="card">
<div class="card-header">
<h4>Profile</h4>
</div>
<div class="card-body">
<p><strong>Username:</strong> {{ user.username }}</p>
<p><strong>Email:</strong> {{ user.email }}</p>
<p><strong>Member since:</strong> {{ user.created_at.strftime('%Y-%m-%d') }}</p>
<p><strong>Last login:</strong>
{{ user.last_login.strftime('%Y-%m-%d %H:%M') if user.last_login else 'Never' }}
</p>
<p><strong>Role:</strong>
{% if user.is_admin %}
<span class="badge bg-danger">Administrator</span>
{% else %}
<span class="badge bg-secondary">User</span>
{% endif %}
</p>
</div>
<div class="card-footer">
<a href="{{ url_for('edit_profile') }}" class="btn btn-primary btn-sm">Edit Profile</a>
<a href="{{ url_for('change_password') }}" class="btn btn-warning btn-sm">Change Password</a>
</div>
</div>
</div>
</div>
{% endblock %}
templates/edit_profile.html:
{% extends "base.html" %}
{% block title %}Edit Profile{% endblock %}
{% block content %}
<div class="row justify-content-center">
<div class="col-md-6">
<div class="card">
<div class="card-header">
<h3>Edit Profile</h3>
</div>
<div class="card-body">
<form method="post">
<div class="mb-3">
<label for="email" class="form-label">Email</label>
<input type="email" class="form-control" id="email" name="email"
value="{{ current_user.email }}" required>
</div>
<button type="submit" class="btn btn-primary">Update Profile</button>
<a href="{{ url_for('profile') }}" class="btn btn-secondary">Cancel</a>
</form>
</div>
</div>
</div>
</div>
{% endblock %}
templates/change_password.html:
{% extends "base.html" %}
{% block title %}Change Password{% endblock %}
{% block content %}
<div class="row justify-content-center">
<div class="col-md-6">
<div class="card">
<div class="card-header">
<h3>Change Password</h3>
</div>
<div class="card-body">
<form method="post">
<div class="mb-3">
<label for="current_password" class="form-label">Current Password</label>
<input type="password" class="form-control" id="current_password"
name="current_password" required>
</div>
<div class="mb-3">
<label for="new_password" class="form-label">New Password</label>
<input type="password" class="form-control" id="new_password"
name="new_password" required>
<div class="form-text">
At least 8 characters with uppercase, lowercase, and number.
</div>
</div>
<div class="mb-3">
<label for="confirm_password" class="form-label">Confirm New Password</label>
<input type="password" class="form-control" id="confirm_password"
name="confirm_password" required>
</div>
<button type="submit" class="btn btn-primary">Change Password</button>
<a href="{{ url_for('profile') }}" class="btn btn-secondary">Cancel</a>
</form>
</div>
</div>
</div>
</div>
{% endblock %}
templates/forgot_password.html:
{% extends "base.html" %}
{% block title %}Forgot Password{% endblock %}
{% block content %}
<div class="row justify-content-center">
<div class="col-md-6">
<div class="card">
<div class="card-header">
<h3>Forgot Password</h3>
</div>
<div class="card-body">
<p>Enter your email address and we'll send you instructions to reset your password.</p>
<form method="post">
<div class="mb-3">
<label for="email" class="form-label">Email</label>
<input type="email" class="form-control" id="email" name="email" required>
</div>
<button type="submit" class="btn btn-primary">Send Reset Instructions</button>
<a href="{{ url_for('login') }}" class="btn btn-secondary">Back to Login</a>
</form>
</div>
</div>
</div>
</div>
{% endblock %}
templates/reset_password.html:
{% extends "base.html" %}
{% block title %}Reset Password{% endblock %}
{% block content %}
<div class="row justify-content-center">
<div class="col-md-6">
<div class="card">
<div class="card-header">
<h3>Reset Password</h3>
</div>
<div class="card-body">
<form method="post">
<div class="mb-3">
<label for="new_password" class="form-label">New Password</label>
<input type="password" class="form-control" id="new_password"
name="new_password" required>
<div class="form-text">
At least 8 characters with uppercase, lowercase, and number.
</div>
</div>
<div class="mb-3">
<label for="confirm_password" class="form-label">Confirm New Password</label>
<input type="password" class="form-control" id="confirm_password"
name="confirm_password" required>
</div>
<input type="hidden" name="token" value="{{ token }}">
<button type="submit" class="btn btn-primary">Reset Password</button>
</form>
</div>
</div>
</div>
</div>
{% endblock %}
templates/admin.html:
{% extends "base.html" %}
{% block title %}Admin Panel{% endblock %}
{% block content %}
<h1>Admin Panel</h1>
<p>Welcome, {{ current_user.username }}. Here you can manage users.</p>
<div class="card">
<div class="card-header">
<h4>Users</h4>
</div>
<div class="card-body">
<table class="table table-striped">
<thead>
<tr>
<th>ID</th>
<th>Username</th>
<th>Email</th>
<th>Joined</th>
<th>Last Login</th>
<th>Status</th>
<th>Role</th>
<th>Actions</th>
</tr>
</thead>
<tbody>
{% for user in users %}
<tr>
<td>{{ user.id }}</td>
<td>{{ user.username }}</td>
<td>{{ user.email }}</td>
<td>{{ user.created_at.strftime('%Y-%m-%d') }}</td>
<td>{{ user.last_login.strftime('%Y-%m-%d %H:%M') if user.last_login else 'Never' }}</td>
<td>
{% if user.is_active %}
<span class="badge bg-success">Active</span>
{% else %}
<span class="badge bg-danger">Inactive</span>
{% endif %}
</td>
<td>
{% if user.is_admin %}
<span class="badge bg-danger">Admin</span>
{% else %}
<span class="badge bg-secondary">User</span>
{% endif %}
</td>
<td>
{% if user.id != current_user.id %}
<a href="{{ url_for('toggle_user_active', user_id=user.id) }}"
class="btn btn-sm btn-warning">
{{ 'Deactivate' if user.is_active else 'Activate' }}
</a>
<a href="{{ url_for('delete_user', user_id=user.id) }}"
class="btn btn-sm btn-danger"
onclick="return confirm('Are you sure you want to delete this user?')">
Delete
</a>
{% else %}
<span class="text-muted">(You)</span>
{% endif %}
</td>
</tr>
{% endfor %}
</tbody>
</table>
</div>
</div>
{% endblock %}
"""
if __name__ == '__main__':
app.run(debug=True, port=5000)Django is a high-level Python web framework that encourages rapid development and clean, pragmatic design. It follows the "batteries-included" philosophy, providing built-in solutions for common web development tasks.
Creating a Django Project:
# Install Django
pip install django
# Create project
django-admin startproject myproject
cd myproject
# Create app
python manage.py startapp blog
python manage.py startapp usersProject Structure:
myproject/
manage.py
myproject/
__init__.py
settings.py
urls.py
asgi.py
wsgi.py
blog/
__init__.py
admin.py
apps.py
models.py
views.py
urls.py
templates/
blog/
static/
blog/
users/
__init__.py
admin.py
apps.py
models.py
views.py
urls.py
templates/
users/
templates/
base.html
static/
css/
js/
images/
Settings Configuration:
# myproject/settings.py
import os
from pathlib import Path
# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent
# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = 'django-insecure-your-secret-key-here'
# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = True
ALLOWED_HOSTS = ['localhost', '127.0.0.1']
# Application definition
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
# Third party apps
'rest_framework',
'crispy_forms',
'crispy_bootstrap5',
# Local apps
'blog.apps.BlogConfig',
'users.apps.UsersConfig',
]
MIDDLEWARE = [
'django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
]
ROOT_URLCONF = 'myproject.urls'
TEMPLATES = [
{
'BACKEND': 'django.template.backends.django.DjangoTemplates',
'DIRS': [BASE_DIR / 'templates'],
'APP_DIRS': True,
'OPTIONS': {
'context_processors': [
'django.template.context_processors.debug',
'django.template.context_processors.request',
'django.contrib.auth.context_processors.auth',
'django.contrib.messages.context_processors.messages',
],
},
},
]
WSGI_APPLICATION = 'myproject.wsgi.application'
# Database
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': BASE_DIR / 'db.sqlite3',
}
}
# Password validation
AUTH_PASSWORD_VALIDATORS = [
{
'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
},
]
# Internationalization
LANGUAGE_CODE = 'en-us'
TIME_ZONE = 'UTC'
USE_I18N = True
USE_TZ = True
# Static files (CSS, JavaScript, Images)
STATIC_URL = 'static/'
STATICFILES_DIRS = [BASE_DIR / 'static']
STATIC_ROOT = BASE_DIR / 'staticfiles'
# Media files
MEDIA_URL = 'media/'
MEDIA_ROOT = BASE_DIR / 'media'
# Default primary key field type
DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'
# Login/Logout URLs
LOGIN_URL = 'login'
LOGIN_REDIRECT_URL = 'blog-home'
LOGOUT_REDIRECT_URL = 'blog-home'
# Crispy Forms
CRISPY_ALLOWED_TEMPLATE_PACKS = 'bootstrap5'
CRISPY_TEMPLATE_PACK = 'bootstrap5'
# Email backend for development
EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'Blog Models:
# blog/models.py
from django.db import models
from django.contrib.auth.models import User
from django.utils import timezone
from django.urls import reverse
from taggit.managers import TaggableManager
import markdown
class Category(models.Model):
"""Blog post category."""
name = models.CharField(max_length=100, unique=True)
slug = models.SlugField(max_length=100, unique=True)
description = models.TextField(blank=True)
created_at = models.DateTimeField(auto_now_add=True)
class Meta:
verbose_name_plural = "Categories"
ordering = ['name']
def __str__(self):
return self.name
def get_absolute_url(self):
return reverse('blog:category', args=[self.slug])
class Post(models.Model):
"""Blog post model."""
STATUS_CHOICES = (
('draft', 'Draft'),
('published', 'Published'),
)
title = models.CharField(max_length=200)
slug = models.SlugField(max_length=200, unique_for_date='published_at')
author = models.ForeignKey(User, on_delete=models.CASCADE, related_name='blog_posts')
category = models.ForeignKey(Category, on_delete=models.SET_NULL, null=True, related_name='posts')
tags = TaggableManager(blank=True)
content = models.TextField()
excerpt = models.TextField(max_length=500, blank=True,
help_text="Brief description of the post")
featured_image = models.ImageField(upload_to='blog/%Y/%m/%d/', blank=True, null=True)
status = models.CharField(max_length=10, choices=STATUS_CHOICES, default='draft')
views_count = models.PositiveIntegerField(default=0)
likes = models.ManyToManyField(User, related_name='liked_posts', blank=True)
created_at = models.DateTimeField(auto_now_add=True)
updated_at = models.DateTimeField(auto_now=True)
published_at = models.DateTimeField(null=True, blank=True)
class Meta:
ordering = ['-published_at', '-created_at']
indexes = [
models.Index(fields=['-published_at']),
models.Index(fields=['status', 'published_at']),
]
def __str__(self):
return self.title
def save(self, *args, **kwargs):
if self.status == 'published' and not self.published_at:
self.published_at = timezone.now()
super().save(*args, **kwargs)
def get_absolute_url(self):
return reverse('blog:post_detail', args=[
self.published_at.year,
self.published_at.month,
self.published_at.day,
self.slug
])
def get_html_content(self):
"""Convert markdown content to HTML."""
md = markdown.Markdown(extensions=['extra', 'codehilite'])
return md.convert(self.content)
def total_likes(self):
return self.likes.count()
def increment_views(self):
self.views_count += 1
self.save(update_fields=['views_count'])
class Comment(models.Model):
"""Blog post comments."""
post = models.ForeignKey(Post, on_delete=models.CASCADE, related_name='comments')
name = models.CharField(max_length=100)
email = models.EmailField()
website = models.URLField(blank=True)
content = models.TextField()
is_active = models.BooleanField(default=True)
is_spam = models.BooleanField(default=False)
parent = models.ForeignKey('self', on_delete=models.CASCADE, null=True, blank=True,
related_name='replies')
created_at = models.DateTimeField(auto_now_add=True)
updated_at = models.DateTimeField(auto_now=True)
class Meta:
ordering = ['created_at']
def __str__(self):
return f'Comment by {self.name} on {self.post}'
def get_replies(self):
return self.replies.filter(is_active=True, is_spam=False)User Profile Model:
# users/models.py
from django.db import models
from django.contrib.auth.models import User
from django.db.models.signals import post_save
from django.dispatch import receiver
from PIL import Image
class Profile(models.Model):
"""Extended user profile."""
user = models.OneToOneField(User, on_delete=models.CASCADE, related_name='profile')
bio = models.TextField(max_length=500, blank=True)
location = models.CharField(max_length=100, blank=True)
birth_date = models.DateField(null=True, blank=True)
avatar = models.ImageField(upload_to='avatars/', default='avatars/default.png')
website = models.URLField(blank=True)
github = models.URLField(blank=True)
twitter = models.URLField(blank=True)
linkedin = models.URLField(blank=True)
is_public = models.BooleanField(default=True)
email_notifications = models.BooleanField(default=True)
created_at = models.DateTimeField(auto_now_add=True)
updated_at = models.DateTimeField(auto_now=True)
def __str__(self):
return f'{self.user.username} Profile'
def save(self, *args, **kwargs):
super().save(*args, **kwargs)
# Resize avatar image
if self.avatar and hasattr(self.avatar, 'path'):
img = Image.open(self.avatar.path)
if img.height > 300 or img.width > 300:
output_size = (300, 300)
img.thumbnail(output_size)
img.save(self.avatar.path)
@receiver(post_save, sender=User)
def create_user_profile(sender, instance, created, **kwargs):
"""Create profile when user is created."""
if created:
Profile.objects.create(user=instance)
@receiver(post_save, sender=User)
def save_user_profile(sender, instance, **kwargs):
"""Save profile when user is saved."""
instance.profile.save()Blog Views:
# blog/views.py
from django.shortcuts import render, get_object_or_404, redirect
from django.contrib.auth.decorators import login_required
from django.contrib import messages
from django.core.paginator import Paginator, EmptyPage, PageNotAnInteger
from django.db.models import Q, Count
from django.utils import timezone
from django.views.generic import ListView, DetailView, CreateView, UpdateView, DeleteView
from django.contrib.auth.mixins import LoginRequiredMixin, UserPassesTestMixin
from django.urls import reverse_lazy
from taggit.models import Tag
from .models import Post, Category, Comment
from .forms import PostForm, CommentForm
class PostListView(ListView):
"""List all published posts."""
model = Post
template_name = 'blog/post_list.html'
context_object_name = 'posts'
paginate_by = 10
def get_queryset(self):
return Post.objects.filter(
status='published',
published_at__lte=timezone.now()
).select_related('author', 'category').prefetch_related('tags')
class PostDetailView(DetailView):
"""Display single post."""
model = Post
template_name = 'blog/post_detail.html'
def get_queryset(self):
return Post.objects.filter(
status='published',
published_at__lte=timezone.now()
).select_related('author', 'category').prefetch_related('tags', 'comments')
def get_context_data(self, **kwargs):
context = super().get_context_data(**kwargs)
context['comment_form'] = CommentForm()
# Increment view count
self.object.increment_views()
# Get related posts
context['related_posts'] = Post.objects.filter(
category=self.object.category,
status='published'
).exclude(id=self.object.id)[:5]
return context
class CategoryPostListView(ListView):
"""List posts in a category."""
model = Post
template_name = 'blog/category_posts.html'
context_object_name = 'posts'
paginate_by = 10
def get_queryset(self):
self.category = get_object_or_404(Category, slug=self.kwargs['slug'])
return Post.objects.filter(
category=self.category,
status='published',
published_at__lte=timezone.now()
).select_related('author')
def get_context_data(self, **kwargs):
context = super().get_context_data(**kwargs)
context['category'] = self.category
return context
class TagPostListView(ListView):
"""List posts with a specific tag."""
model = Post
template_name = 'blog/tag_posts.html'
context_object_name = 'posts'
paginate_by = 10
def get_queryset(self):
self.tag = get_object_or_404(Tag, slug=self.kwargs['slug'])
return Post.objects.filter(
tags__slug=self.tag.slug,
status='published',
published_at__lte=timezone.now()
).select_related('author').distinct()
def get_context_data(self, **kwargs):
context = super().get_context_data(**kwargs)
context['tag'] = self.tag
return context
class PostCreateView(LoginRequiredMixin, CreateView):
"""Create new post."""
model = Post
form_class = PostForm
template_name = 'blog/post_form.html'
def form_valid(self, form):
form.instance.author = self.request.user
messages.success(self.request, 'Post created successfully!')
return super().form_valid(form)
class PostUpdateView(LoginRequiredMixin, UserPassesTestMixin, UpdateView):
"""Update existing post."""
model = Post
form_class = PostForm
template_name = 'blog/post_form.html'
def form_valid(self, form):
messages.success(self.request, 'Post updated successfully!')
return super().form_valid(form)
def test_func(self):
post = self.get_object()
return self.request.user == post.author or self.request.user.is_superuser
class PostDeleteView(LoginRequiredMixin, UserPassesTestMixin, DeleteView):
"""Delete post."""
model = Post
success_url = reverse_lazy('blog:post_list')
template_name = 'blog/post_confirm_delete.html'
def test_func(self):
post = self.get_object()
return self.request.user == post.author or self.request.user.is_superuser
def delete(self, request, *args, **kwargs):
messages.success(self.request, 'Post deleted successfully!')
return super().delete(request, *args, **kwargs)
def search_posts(request):
"""Search posts by title, content, or tags."""
query = request.GET.get('q', '')
if query:
posts = Post.objects.filter(
Q(status='published'),
Q(published_at__lte=timezone.now()),
Q(title__icontains=query) |
Q(content__icontains=query) |
Q(tags__name__icontains=query)
).distinct().select_related('author', 'category')
else:
posts = Post.objects.none()
paginator = Paginator(posts, 10)
page = request.GET.get('page')
try:
posts = paginator.page(page)
except PageNotAnInteger:
posts = paginator.page(1)
except EmptyPage:
posts = paginator.page(paginator.num_pages)
return render(request, 'blog/search_results.html', {
'posts': posts,
'query': query
})
@login_required
def like_post(request, pk):
"""Like or unlike a post."""
post = get_object_or_404(Post, pk=pk)
if request.user in post.likes.all():
post.likes.remove(request.user)
liked = False
else:
post.likes.add(request.user)
liked = True
return JsonResponse({
'liked': liked,
'total_likes': post.total_likes()
})
def add_comment(request, pk):
"""Add comment to post."""
post = get_object_or_404(Post, pk=pk)
if request.method == 'POST':
form = CommentForm(request.POST)
if form.is_valid():
comment = form.save(commit=False)
comment.post = post
comment.save()
messages.success(request, 'Your comment has been added.')
return redirect('blog:post_detail', slug=post.slug)User Views:
# users/views.py
from django.shortcuts import render, redirect, get_object_or_404
from django.contrib.auth import login, authenticate, logout
from django.contrib.auth.decorators import login_required
from django.contrib import messages
from django.contrib.auth.models import User
from django.contrib.auth.views import LoginView, PasswordChangeView
from django.urls import reverse_lazy
from django.views.generic import CreateView, UpdateView, DetailView
from django.contrib.auth.mixins import LoginRequiredMixin
from .forms import UserRegisterForm, UserUpdateForm, ProfileUpdateForm, LoginForm
from .models import Profile
from blog.models import Post
class RegisterView(CreateView):
"""User registration view."""
form_class = UserRegisterForm
template_name = 'users/register.html'
success_url = reverse_lazy('login')
def form_valid(self, form):
response = super().form_valid(form)
messages.success(self.request, 'Account created! You can now log in.')
return response
class CustomLoginView(LoginView):
"""Custom login view."""
form_class = LoginForm
template_name = 'users/login.html'
def form_valid(self, form):
messages.success(self.request, 'Login successful!')
return super().form_valid(form)
def form_invalid(self, form):
messages.error(self.request, 'Invalid username or password.')
return super().form_invalid(form)
@login_required
def logout_view(request):
"""Custom logout view."""
logout(request)
messages.info(request, 'You have been logged out.')
return redirect('blog:post_list')
@login_required
def profile(request, username=None):
"""User profile view."""
if username:
user = get_object_or_404(User, username=username)
else:
user = request.user
posts = Post.objects.filter(
author=user,
status='published'
).order_by('-published_at')[:5]
context = {
'profile_user': user,
'posts': posts,
'is_own_profile': user == request.user
}
return render(request, 'users/profile.html', context)
class ProfileUpdateView(LoginRequiredMixin, UpdateView):
"""Update user profile."""
model = Profile
form_class = ProfileUpdateForm
template_name = 'users/profile_edit.html'
def get_object(self, queryset=None):
return self.request.user.profile
def get_success_url(self):
return reverse_lazy('users:profile', args=[self.request.user.username])
def form_valid(self, form):
messages.success(self.request, 'Profile updated successfully!')
return super().form_valid(form)
@login_required
def update_profile(request):
"""Combined user and profile update view."""
if request.method == 'POST':
u_form = UserUpdateForm(request.POST, instance=request.user)
p_form = ProfileUpdateForm(request.POST, request.FILES, instance=request.user.profile)
if u_form.is_valid() and p_form.is_valid():
u_form.save()
p_form.save()
messages.success(request, 'Your profile has been updated!')
return redirect('users:profile')
else:
u_form = UserUpdateForm(instance=request.user)
p_form = ProfileUpdateForm(instance=request.user.profile)
context = {
'u_form': u_form,
'p_form': p_form
}
return render(request, 'users/profile_edit.html', context)Base Template:
<!-- templates/base.html -->
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>{% block title %}Django Blog{% endblock %}</title>
<!-- Bootstrap 5 -->
<link href="https://cdn.jsdelivr.net/npm/bootstrap@5.1.3/dist/css/bootstrap.min.css" rel="stylesheet">
<!-- Font Awesome -->
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.0.0/css/all.min.css">
<!-- Custom CSS -->
{% load static %}
<link rel="stylesheet" href="{% static 'css/style.css' %}">
{% block extra_css %}{% endblock %}
</head>
<body>
<!-- Navigation -->
<nav class="navbar navbar-expand-lg navbar-dark bg-dark">
<div class="container">
<a class="navbar-brand" href="{% url 'blog:post_list' %}">
<i class="fas fa-blog"></i> Django Blog
</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarNav">
<span class="navbar-toggler-icon"></span>
</button>
<div class="collapse navbar-collapse" id="navbarNav">
<ul class="navbar-nav me-auto">
<li class="nav-item">
<a class="nav-link" href="{% url 'blog:post_list' %}">Home</a>
</li>
<li class="nav-item">
<a class="nav-link" href="{% url 'blog:post_list' %}">Blog</a>
</li>
<li class="nav-item">
<a class="nav-link" href="#about">About</a>
</li>
<li class="nav-item">
<a class="nav-link" href="#contact">Contact</a>
</li>
</ul>
<!-- Search Form -->
<form class="d-flex me-3" action="{% url 'blog:search' %}" method="GET">
<input class="form-control me-2" type="search" name="q" placeholder="Search posts..."
value="{{ request.GET.q }}">
<button class="btn btn-outline-light" type="submit">
<i class="fas fa-search"></i>
</button>
</form>
<!-- User Menu -->
<ul class="navbar-nav">
{% if user.is_authenticated %}
<li class="nav-item dropdown">
<a class="nav-link dropdown-toggle" href="#" id="userDropdown"
role="button" data-bs-toggle="dropdown">
<i class="fas fa-user"></i> {{ user.username }}
</a>
<ul class="dropdown-menu dropdown-menu-end">
<li>
<a class="dropdown-item" href="{% url 'users:profile' %}">
<i class="fas fa-id-card"></i> Profile
</a>
</li>
<li>
<a class="dropdown-item" href="{% url 'blog:post_create' %}">
<i class="fas fa-plus-circle"></i> New Post
</a>
</li>
{% if user.is_staff %}
<li>
<hr class="dropdown-divider">
</li>
<li>
<a class="dropdown-item" href="{% url 'admin:index' %}">
<i class="fas fa-cog"></i> Admin
</a>
</li>
{% endif %}
<li>
<hr class="dropdown-divider">
</li>
<li>
<a class="dropdown-item" href="{% url 'users:logout' %}">
<i class="fas fa-sign-out-alt"></i> Logout
</a>
</li>
</ul>
</li>
{% else %}
<li class="nav-item">
<a class="nav-link" href="{% url 'users:login' %}">
<i class="fas fa-sign-in-alt"></i> Login
</a>
</li>
<li class="nav-item">
<a class="nav-link" href="{% url 'users:register' %}">
<i class="fas fa-user-plus"></i> Register
</a>
</li>
{% endif %}
</ul>
</div>
</div>
</nav>
<!-- Messages -->
{% if messages %}
<div class="container mt-3">
{% for message in messages %}
<div class="alert alert-{{ message.tags }} alert-dismissible fade show" role="alert">
{{ message }}
<button type="button" class="btn-close" data-bs-dismiss="alert"></button>
</div>
{% endfor %}
</div>
{% endif %}
<!-- Main Content -->
<main>
{% block content %}{% endblock %}
</main>
<!-- Footer -->
<footer class="bg-dark text-white mt-5 py-4">
<div class="container">
<div class="row">
<div class="col-md-6">
<h5>Django Blog</h5>
<p>A comprehensive blog application built with Django.</p>
</div>
<div class="col-md-3">
<h5>Links</h5>
<ul class="list-unstyled">
<li><a href="#" class="text-white-50">Home</a></li>
<li><a href="#" class="text-white-50">About</a></li>
<li><a href="#" class="text-white-50">Contact</a></li>
<li><a href="#" class="text-white-50">Privacy Policy</a></li>
</ul>
</div>
<div class="col-md-3">
<h5>Follow Us</h5>
<ul class="list-unstyled">
<li><a href="#" class="text-white-50"><i class="fab fa-twitter"></i> Twitter</a></li>
<li><a href="#" class="text-white-50"><i class="fab fa-facebook"></i> Facebook</a></li>
<li><a href="#" class="text-white-50"><i class="fab fa-github"></i> GitHub</a></li>
<li><a href="#" class="text-white-50"><i class="fab fa-linkedin"></i> LinkedIn</a></li>
</ul>
</div>
</div>
<hr class="bg-secondary">
<div class="text-center">
<p>© {% now "Y" %} Django Blog. All rights reserved.</p>
</div>
</div>
</footer>
<!-- Bootstrap JS -->
<script src="https://cdn.jsdelivr.net/npm/bootstrap@5.1.3/dist/js/bootstrap.bundle.min.js"></script>
<!-- jQuery -->
<script src="https://code.jquery.com/jquery-3.6.0.min.js"></script>
<!-- Custom JS -->
<script src="{% static 'js/main.js' %}"></script>
{% block extra_js %}{% endblock %}
</body>
</html>Post List Template:
<!-- blog/templates/blog/post_list.html -->
{% extends 'base.html' %}
{% load static %}
{% block title %}Blog Posts{% endblock %}
{% block content %}
<div class="container mt-4">
<div class="row">
<!-- Main Content -->
<div class="col-md-8">
<h1 class="mb-4">Latest Posts</h1>
{% for post in posts %}
<article class="card mb-4">
{% if post.featured_image %}
<img src="{{ post.featured_image.url }}" class="card-img-top" alt="{{ post.title }}">
{% endif %}
<div class="card-body">
<h2 class="card-title">
<a href="{{ post.get_absolute_url }}" class="text-decoration-none">
{{ post.title }}
</a>
</h2>
<p class="card-text text-muted">
<i class="fas fa-user"></i>
<a href="{% url 'users:profile' post.author.username %}">
{{ post.author.username }}
</a>
| <i class="fas fa-calendar"></i> {{ post.published_at|date:"F j, Y" }}
| <i class="fas fa-folder"></i>
<a href="{% url 'blog:category' post.category.slug %}">
{{ post.category.name }}
</a>
| <i class="fas fa-eye"></i> {{ post.views_count }} views
| <i class="fas fa-heart"></i> {{ post.total_likes }} likes
</p>
{% if post.excerpt %}
<p class="card-text">{{ post.excerpt }}</p>
{% else %}
<p class="card-text">{{ post.content|truncatewords:50 }}</p>
{% endif %}
<a href="{{ post.get_absolute_url }}" class="btn btn-primary">
Read More <i class="fas fa-arrow-right"></i>
</a>
</div>
<div class="card-footer text-muted">
<i class="fas fa-tags"></i> Tags:
{% for tag in post.tags.all %}
<a href="{% url 'blog:tag' tag.slug %}" class="badge bg-secondary text-decoration-none">
{{ tag.name }}
</a>
{% endfor %}
</div>
</article>
{% empty %}
<div class="alert alert-info">
No posts found.
</div>
{% endfor %}
<!-- Pagination -->
{% if is_paginated %}
<nav aria-label="Page navigation">
<ul class="pagination justify-content-center">
{% if page_obj.has_previous %}
<li class="page-item">
<a class="page-link" href="?page=1">
<i class="fas fa-angle-double-left"></i>
</a>
</li>
<li class="page-item">
<a class="page-link" href="?page={{ page_obj.previous_page_number }}">
<i class="fas fa-angle-left"></i>
</a>
</li>
{% endif %}
{% for num in page_obj.paginator.page_range %}
{% if page_obj.number == num %}
<li class="page-item active">
<span class="page-link">{{ num }}</span>
</li>
{% elif num > page_obj.number|add:'-3' and num < page_obj.number|add:'3' %}
<li class="page-item">
<a class="page-link" href="?page={{ num }}">{{ num }}</a>
</li>
{% endif %}
{% endfor %}
{% if page_obj.has_next %}
<li class="page-item">
<a class="page-link" href="?page={{ page_obj.next_page_number }}">
<i class="fas fa-angle-right"></i>
</a>
</li>
<li class="page-item">
<a class="page-link" href="?page={{ page_obj.paginator.num_pages }}">
<i class="fas fa-angle-double-right"></i>
</a>
</li>
{% endif %}
</ul>
</nav>
{% endif %}
</div>
<!-- Sidebar -->
<div class="col-md-4">
<!-- Categories Widget -->
<div class="card mb-4">
<div class="card-header">
<h5><i class="fas fa-folder"></i> Categories</h5>
</div>
<div class="card-body">
<ul class="list-unstyled">
{% for category in categories %}
<li class="mb-2">
<a href="{% url 'blog:category' category.slug %}"
class="text-decoration-none">
{{ category.name }}
<span class="badge bg-secondary float-end">
{{ category.posts.count }}
</span>
</a>
</li>
{% endfor %}
</ul>
</div>
</div>
<!-- Tags Widget -->
<div class="card mb-4">
<div class="card-header">
<h5><i class="fas fa-tags"></i> Popular Tags</h5>
</div>
<div class="card-body">
{% for tag in popular_tags %}
<a href="{% url 'blog:tag' tag.slug %}"
class="badge bg-primary text-decoration-none me-1 mb-1"
style="font-size: {{ tag.size }}px;">
{{ tag.name }}
</a>
{% endfor %}
</div>
</div>
<!-- Recent Posts Widget -->
<div class="card mb-4">
<div class="card-header">
<h5><i class="fas fa-clock"></i> Recent Posts</h5>
</div>
<div class="card-body">
<ul class="list-unstyled">
{% for post in recent_posts %}
<li class="mb-2">
<a href="{{ post.get_absolute_url }}" class="text-decoration-none">
{{ post.title }}
</a>
<small class="text-muted d-block">
{{ post.published_at|date:"M j, Y" }}
</small>
</li>
{% endfor %}
</ul>
</div>
</div>
</div>
</div>
</div>
{% endblock %}Admin Configuration:
# blog/admin.py
from django.contrib import admin
from django.utils.html import format_html
from django.urls import reverse
from .models import Category, Post, Comment
@admin.register(Category)
class CategoryAdmin(admin.ModelAdmin):
list_display = ['name', 'slug', 'post_count', 'created_at']
prepopulated_fields = {'slug': ('name',)}
search_fields = ['name', 'description']
def post_count(self, obj):
return obj.posts.count()
post_count.short_description = 'Posts'
class CommentInline(admin.TabularInline):
model = Comment
extra = 0
readonly_fields = ['name', 'email', 'content', 'created_at']
@admin.register(Post)
class PostAdmin(admin.ModelAdmin):
list_display = ['title', 'author', 'category', 'status',
'published_at', 'views_count', 'like_count', 'comment_count']
list_filter = ['status', 'category', 'author', 'created_at']
search_fields = ['title', 'content']
prepopulated_fields = {'slug': ('title',)}
readonly_fields = ['views_count', 'created_at', 'updated_at']
fieldsets = (
('Basic Information', {
'fields': ('title', 'slug', 'author', 'category')
}),
('Content', {
'fields': ('content', 'excerpt', 'featured_image')
}),
('Tags', {
'fields': ('tags',)
}),
('Status', {
'fields': ('status', 'published_at')
}),
('Statistics', {
'fields': ('views_count', 'likes'),
'classes': ('collapse',)
}),
('Timestamps', {
'fields': ('created_at', 'updated_at'),
'classes': ('collapse',)
}),
)
inlines = [CommentInline]
def like_count(self, obj):
return obj.likes.count()
like_count.short_description = 'Likes'
def comment_count(self, obj):
return obj.comments.count()
comment_count.short_description = 'Comments'
def save_model(self, request, obj, form, change):
if not obj.pk:
obj.author = request.user
super().save_model(request, obj, form, change)
@admin.register(Comment)
class CommentAdmin(admin.ModelAdmin):
list_display = ['name', 'post', 'content_preview', 'is_active', 'is_spam', 'created_at']
list_filter = ['is_active', 'is_spam', 'created_at']
search_fields = ['name', 'email', 'content']
actions = ['mark_as_spam', 'mark_as_ham', 'approve_comments']
def content_preview(self, obj):
return obj.content[:50] + '...' if len(obj.content) > 50 else obj.content
content_preview.short_description = 'Content'
def mark_as_spam(self, request, queryset):
queryset.update(is_spam=True, is_active=False)
mark_as_spam.short_description = "Mark selected as spam"
def mark_as_ham(self, request, queryset):
queryset.update(is_spam=False, is_active=True)
mark_as_ham.short_description = "Mark selected as ham"
def approve_comments(self, request, queryset):
queryset.update(is_active=True)
approve_comments.short_description = "Approve selected comments"
# users/admin.py
from django.contrib import admin
from django.contrib.auth.admin import UserAdmin
from django.contrib.auth.models import User
from .models import Profile
class ProfileInline(admin.StackedInline):
model = Profile
can_delete = False
verbose_name_plural = 'Profile'
class CustomUserAdmin(UserAdmin):
inlines = [ProfileInline]
list_display = ['username', 'email', 'first_name', 'last_name',
'is_staff', 'date_joined', 'last_login']
list_filter = ['is_staff', 'is_superuser', 'is_active', 'groups']
fieldsets = UserAdmin.fieldsets + (
('Profile Information', {
'fields': ('profile__bio', 'profile__location', 'profile__birth_date'),
}),
)
# Unregister default UserAdmin and register custom
admin.site.unregister(User)
admin.site.register(User, CustomUserAdmin)
@admin.register(Profile)
class ProfileAdmin(admin.ModelAdmin):
list_display = ['user', 'location', 'is_public', 'created_at']
list_filter = ['is_public', 'created_at']
search_fields = ['user__username', 'user__email', 'bio']
readonly_fields = ['created_at', 'updated_at']
fieldsets = (
('User', {
'fields': ('user',)
}),
('Profile Information', {
'fields': ('bio', 'location', 'birth_date', 'avatar')
}),
('Social Links', {
'fields': ('website', 'github', 'twitter', 'linkedin')
}),
('Preferences', {
'fields': ('is_public', 'email_notifications')
}),
('Timestamps', {
'fields': ('created_at', 'updated_at'),
'classes': ('collapse',)
}),
)Django REST Framework Integration:
# First install: pip install djangorestframework
# blog/serializers.py
from rest_framework import serializers
from django.contrib.auth.models import User
from .models import Post, Category, Comment
class UserSerializer(serializers.ModelSerializer):
class Meta:
model = User
fields = ['id', 'username', 'email', 'first_name', 'last_name']
class CategorySerializer(serializers.ModelSerializer):
post_count = serializers.IntegerField(source='posts.count', read_only=True)
class Meta:
model = Category
fields = ['id', 'name', 'slug', 'description', 'post_count', 'created_at']
class CommentSerializer(serializers.ModelSerializer):
author_name = serializers.CharField(source='name', read_only=True)
class Meta:
model = Comment
fields = ['id', 'name', 'email', 'website', 'content',
'is_active', 'created_at', 'author_name']
read_only_fields = ['is_active', 'created_at']
class PostSerializer(serializers.ModelSerializer):
author = UserSerializer(read_only=True)
category = CategorySerializer(read_only=True)
category_id = serializers.PrimaryKeyRelatedField(
queryset=Category.objects.all(), source='category', write_only=True
)
comments = CommentSerializer(many=True, read_only=True)
tags = serializers.ListField(
child=serializers.CharField(), write_only=True, required=False
)
tag_list = serializers.SerializerMethodField()
class Meta:
model = Post
fields = ['id', 'title', 'slug', 'author', 'category', 'category_id',
'content', 'excerpt', 'featured_image', 'status',
'views_count', 'likes_count', 'tags', 'tag_list',
'comments', 'created_at', 'updated_at', 'published_at']
read_only_fields = ['slug', 'views_count', 'likes_count',
'created_at', 'updated_at']
def get_tag_list(self, obj):
return [tag.name for tag in obj.tags.all()]
def get_likes_count(self, obj):
return obj.likes.count()
def create(self, validated_data):
tags = validated_data.pop('tags', [])
post = Post.objects.create(**validated_data)
for tag_name in tags:
post.tags.add(tag_name)
return post
def update(self, instance, validated_data):
tags = validated_data.pop('tags', None)
for attr, value in validated_data.items():
setattr(instance, attr, value)
instance.save()
if tags is not None:
instance.tags.clear()
for tag_name in tags:
instance.tags.add(tag_name)
return instance
# blog/views_api.py
from rest_framework import generics, permissions, filters, status
from rest_framework.decorators import api_view, permission_classes
from rest_framework.response import Response
from rest_framework.pagination import PageNumberPagination
from django_filters.rest_framework import DjangoFilterBackend
from django.db.models import Q
from .models import Post, Category, Comment
from .serializers import PostSerializer, CategorySerializer, CommentSerializer
class StandardResultsSetPagination(PageNumberPagination):
page_size = 10
page_size_query_param = 'page_size'
max_page_size = 100
class PostListAPIView(generics.ListCreateAPIView):
queryset = Post.objects.filter(status='published')
serializer_class = PostSerializer
pagination_class = StandardResultsSetPagination
filter_backends = [DjangoFilterBackend, filters.SearchFilter, filters.OrderingFilter]
filterset_fields = ['category__slug', 'author__username']
search_fields = ['title', 'content', 'tags__name']
ordering_fields = ['published_at', 'views_count', 'title']
ordering = ['-published_at']
def perform_create(self, serializer):
serializer.save(author=self.request.user)
class PostDetailAPIView(generics.RetrieveUpdateDestroyAPIView):
queryset = Post.objects.all()
serializer_class = PostSerializer
permission_classes = [permissions.IsAuthenticatedOrReadOnly]
def retrieve(self, request, *args, **kwargs):
instance = self.get_object()
instance.increment_views()
serializer = self.get_serializer(instance)
return Response(serializer.data)
def perform_update(self, serializer):
if self.get_object().author != self.request.user:
raise permissions.PermissionDenied("You can only edit your own posts")
serializer.save()
def perform_destroy(self, instance):
if instance.author != self.request.user:
raise permissions.PermissionDenied("You can only delete your own posts")
instance.delete()
class CategoryListAPIView(generics.ListAPIView):
queryset = Category.objects.all()
serializer_class = CategorySerializer
pagination_class = None
class PostByCategoryAPIView(generics.ListAPIView):
serializer_class = PostSerializer
pagination_class = StandardResultsSetPagination
def get_queryset(self):
category_slug = self.kwargs['slug']
return Post.objects.filter(
category__slug=category_slug,
status='published'
)
class CommentListAPIView(generics.ListCreateAPIView):
serializer_class = CommentSerializer
permission_classes = [permissions.AllowAny]
def get_queryset(self):
post_id = self.kwargs['post_id']
return Comment.objects.filter(
post_id=post_id,
is_active=True,
is_spam=False
)
def perform_create(self, serializer):
post_id = self.kwargs['post_id']
serializer.save(post_id=post_id)
@api_view(['POST'])
@permission_classes([permissions.IsAuthenticated])
def like_post_api(request, pk):
"""Like or unlike a post via API."""
try:
post = Post.objects.get(pk=pk)
except Post.DoesNotExist:
return Response({'error': 'Post not found'}, status=status.HTTP_404_NOT_FOUND)
if request.user in post.likes.all():
post.likes.remove(request.user)
liked = False
else:
post.likes.add(request.user)
liked = True
return Response({
'liked': liked,
'total_likes': post.likes.count()
})
# blog/urls_api.py
from django.urls import path
from . import views_api
urlpatterns = [
path('posts/', views_api.PostListAPIView.as_view(), name='api_post_list'),
path('posts/<int:pk>/', views_api.PostDetailAPIView.as_view(), name='api_post_detail'),
path('posts/<int:pk>/like/', views_api.like_post_api, name='api_post_like'),
path('categories/', views_api.CategoryListAPIView.as_view(), name='api_category_list'),
path('categories/<slug:slug>/posts/', views_api.PostByCategoryAPIView.as_view(), name='api_category_posts'),
path('posts/<int:post_id>/comments/', views_api.CommentListAPIView.as_view(), name='api_comment_list'),
]
# myproject/urls.py
from django.contrib import admin
from django.urls import path, include
from django.conf import settings
from django.conf.urls.static import static
urlpatterns = [
path('admin/', admin.site.urls),
path('', include('blog.urls')),
path('users/', include('users.urls')),
path('api/', include('blog.urls_api')),
path('api-auth/', include('rest_framework.urls')),
]
if settings.DEBUG:
urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)
urlpatterns += static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)Django REST Framework Authentication:
# users/serializers.py
from rest_framework import serializers
from django.contrib.auth.models import User
from django.contrib.auth.password_validation import validate_password
from .models import Profile
class UserSerializer(serializers.ModelSerializer):
class Meta:
model = User
fields = ['id', 'username', 'email', 'first_name', 'last_name']
class ProfileSerializer(serializers.ModelSerializer):
user = UserSerializer(read_only=True)
class Meta:
model = Profile
fields = ['user', 'bio', 'location', 'birth_date', 'avatar',
'website', 'github', 'twitter', 'linkedin',
'is_public', 'created_at']
class RegisterSerializer(serializers.ModelSerializer):
password = serializers.CharField(write_only=True, required=True, validators=[validate_password])
password2 = serializers.CharField(write_only=True, required=True)
class Meta:
model = User
fields = ['username', 'password', 'password2', 'email', 'first_name', 'last_name']
def validate(self, attrs):
if attrs['password'] != attrs['password2']:
raise serializers.ValidationError({"password": "Password fields didn't match."})
return attrs
def create(self, validated_data):
validated_data.pop('password2')
user = User.objects.create_user(**validated_data)
return user
class ChangePasswordSerializer(serializers.Serializer):
old_password = serializers.CharField(required=True)
new_password = serializers.CharField(required=True, validators=[validate_password])
def validate_old_password(self, value):
user = self.context['request'].user
if not user.check_password(value):
raise serializers.ValidationError("Old password is incorrect")
return value
# users/views_api.py
from rest_framework import generics, permissions, status
from rest_framework.response import Response
from rest_framework.decorators import api_view, permission_classes
from rest_framework.authtoken.models import Token
from rest_framework.authtoken.views import ObtainAuthToken
from django.contrib.auth.models import User
from .serializers import UserSerializer, ProfileSerializer, RegisterSerializer, ChangePasswordSerializer
class RegisterAPIView(generics.CreateAPIView):
serializer_class = RegisterSerializer
permission_classes = [permissions.AllowAny]
def post(self, request, *args, **kwargs):
serializer = self.get_serializer(data=request.data)
serializer.is_valid(raise_exception=True)
user = serializer.save()
token, created = Token.objects.get_or_create(user=user)
return Response({
'user': UserSerializer(user).data,
'token': token.key
}, status=status.HTTP_201_CREATED)
class LoginAPIView(ObtainAuthToken):
def post(self, request, *args, **kwargs):
serializer = self.serializer_class(data=request.data,
context={'request': request})
serializer.is_valid(raise_exception=True)
user = serializer.validated_data['user']
token, created = Token.objects.get_or_create(user=user)
return Response({
'user': UserSerializer(user).data,
'token': token.key
})
class LogoutAPIView(APIView):
permission_classes = [permissions.IsAuthenticated]
def post(self, request):
request.user.auth_token.delete()
return Response({'message': 'Successfully logged out'})
class ProfileAPIView(generics.RetrieveUpdateAPIView):
serializer_class = ProfileSerializer
permission_classes = [permissions.IsAuthenticated]
def get_object(self):
return self.request.user.profile
class ChangePasswordAPIView(generics.UpdateAPIView):
serializer_class = ChangePasswordSerializer
permission_classes = [permissions.IsAuthenticated]
def get_object(self):
return self.request.user
def update(self, request, *args, **kwargs):
serializer = self.get_serializer(data=request.data)
serializer.is_valid(raise_exception=True)
user = self.get_object()
user.set_password(serializer.validated_data['new_password'])
user.save()
return Response({'message': 'Password changed successfully'})
@api_view(['GET'])
@permission_classes([permissions.IsAuthenticated])
def user_list(request):
users = User.objects.all()
serializer = UserSerializer(users, many=True)
return Response(serializer.data)
# users/urls_api.py
from django.urls import path
from . import views_api
urlpatterns = [
path('register/', views_api.RegisterAPIView.as_view(), name='api_register'),
path('login/', views_api.LoginAPIView.as_view(), name='api_login'),
path('logout/', views_api.LogoutAPIView.as_view(), name='api_logout'),
path('profile/', views_api.ProfileAPIView.as_view(), name='api_profile'),
path('change-password/', views_api.ChangePasswordAPIView.as_view(), name='api_change_password'),
path('users/', views_api.user_list, name='api_user_list'),
]Django REST Framework Permissions:
# blog/permissions.py
from rest_framework import permissions
class IsAuthorOrReadOnly(permissions.BasePermission):
"""
Custom permission to only allow authors of an object to edit it.
"""
def has_object_permission(self, request, view, obj):
# Read permissions are allowed to any request
if request.method in permissions.SAFE_METHODS:
return True
# Write permissions are only allowed to the author
return obj.author == request.user
class IsCommentAuthorOrReadOnly(permissions.BasePermission):
"""
Custom permission for comments - allows author to edit/delete,
and post author to delete inappropriate comments.
"""
def has_object_permission(self, request, view, obj):
if request.method in permissions.SAFE_METHODS:
return True
# Allow comment author to edit/delete
if hasattr(obj, 'user') and obj.user == request.user:
return True
# Allow post author to delete comments on their post
if request.method == 'DELETE' and obj.post.author == request.user:
return True
return FalseTesting is crucial for ensuring code quality, preventing regressions, and maintaining confidence in your codebase.
Python's built-in testing framework:
Basic Unit Tests:
import unittest
import math
from typing import List, Optional
# Code to test
class Calculator:
"""Simple calculator class."""
def add(self, a: float, b: float) -> float:
return a + b
def subtract(self, a: float, b: float) -> float:
return a - b
def multiply(self, a: float, b: float) -> float:
return a * b
def divide(self, a: float, b: float) -> float:
if b == 0:
raise ValueError("Cannot divide by zero")
return a / b
def power(self, a: float, b: float) -> float:
return a ** b
def sqrt(self, a: float) -> float:
if a < 0:
raise ValueError("Cannot calculate square root of negative number")
return math.sqrt(a)
class StringUtils:
"""String utility functions."""
@staticmethod
def reverse(s: str) -> str:
return s[::-1]
@staticmethod
def is_palindrome(s: str) -> bool:
s = s.lower().replace(' ', '')
return s == s[::-1]
@staticmethod
def count_vowels(s: str) -> int:
vowels = 'aeiou'
return sum(1 for char in s.lower() if char in vowels)
@staticmethod
def to_camel_case(s: str) -> str:
words = s.split('_')
return words[0] + ''.join(w.capitalize() for w in words[1:])
# Test cases
class TestCalculator(unittest.TestCase):
"""Test cases for Calculator class."""
def setUp(self):
"""Set up test fixtures."""
self.calc = Calculator()
print("\nSetting up calculator test")
def tearDown(self):
"""Clean up after tests."""
print("Cleaning up calculator test")
def test_add(self):
"""Test addition."""
self.assertEqual(self.calc.add(2, 3), 5)
self.assertEqual(self.calc.add(-1, 1), 0)
self.assertEqual(self.calc.add(0, 0), 0)
self.assertEqual(self.calc.add(2.5, 3.7), 6.2)
def test_subtract(self):
"""Test subtraction."""
self.assertEqual(self.calc.subtract(5, 3), 2)
self.assertEqual(self.calc.subtract(1, 1), 0)
self.assertEqual(self.calc.subtract(0, 5), -5)
self.assertEqual(self.calc.subtract(10.5, 3.2), 7.3)
def test_multiply(self):
"""Test multiplication."""
self.assertEqual(self.calc.multiply(3, 4), 12)
self.assertEqual(self.calc.multiply(-2, 3), -6)
self.assertEqual(self.calc.multiply(0, 5), 0)
self.assertEqual(self.calc.multiply(2.5, 2), 5.0)
def test_divide(self):
"""Test division."""
self.assertEqual(self.calc.divide(10, 2), 5)
self.assertEqual(self.calc.divide(7, 2), 3.5)
self.assertEqual(self.calc.divide(-6, 3), -2)
with self.assertRaises(ValueError) as context:
self.calc.divide(5, 0)
self.assertEqual(str(context.exception), "Cannot divide by zero")
def test_power(self):
"""Test power operation."""
self.assertEqual(self.calc.power(2, 3), 8)
self.assertEqual(self.calc.power(5, 0), 1)
self.assertEqual(self.calc.power(4, 0.5), 2)
def test_sqrt(self):
"""Test square root."""
self.assertEqual(self.calc.sqrt(9), 3)
self.assertEqual(self.calc.sqrt(2), math.sqrt(2))
with self.assertRaises(ValueError):
self.calc.sqrt(-1)
class TestStringUtils(unittest.TestCase):
"""Test cases for StringUtils."""
@classmethod
def setUpClass(cls):
"""Set up once for all tests."""
print("\nSetting up StringUtils test class")
cls.utils = StringUtils()
@classmethod
def tearDownClass(cls):
"""Clean up after all tests."""
print("Tearing down StringUtils test class")
def test_reverse(self):
"""Test string reversal."""
self.assertEqual(self.utils.reverse("hello"), "olleh")
self.assertEqual(self.utils.reverse(""), "")
self.assertEqual(self.utils.reverse("a"), "a")
self.assertEqual(self.utils.reverse("racecar"), "racecar")
def test_is_palindrome(self):
"""Test palindrome detection."""
self.assertTrue(self.utils.is_palindrome("racecar"))
self.assertTrue(self.utils.is_palindrome("A man a plan a canal Panama"))
self.assertTrue(self.utils.is_palindrome("madam"))
self.assertFalse(self.utils.is_palindrome("hello"))
self.assertTrue(self.utils.is_palindrome(""))
def test_count_vowels(self):
"""Test vowel counting."""
self.assertEqual(self.utils.count_vowels("hello"), 2)
self.assertEqual(self.utils.count_vowels("AEIOU"), 5)
self.assertEqual(self.utils.count_vowels("bcdfg"), 0)
self.assertEqual(self.utils.count_vowels(""), 0)
def test_to_camel_case(self):
"""Test snake_case to camelCase conversion."""
self.assertEqual(self.utils.to_camel_case("hello_world"), "helloWorld")
self.assertEqual(self.utils.to_camel_case("my_long_variable_name"), "myLongVariableName")
self.assertEqual(self.utils.to_camel_case("alreadyCamel"), "alreadyCamel")
self.assertEqual(self.utils.to_camel_case(""), "")
# Running tests
if __name__ == '__main__':
unittest.main(argv=[''], verbosity=2, exit=False)Test Discovery and Organization:
import unittest
import sys
import os
# Add project root to path
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
# Test suites
def suite():
"""Create test suite."""
suite = unittest.TestSuite()
# Add test cases
suite.addTest(TestCalculator('test_add'))
suite.addTest(TestCalculator('test_subtract'))
suite.addTest(TestCalculator('test_multiply'))
suite.addTest(TestCalculator('test_divide'))
# Add entire test classes
suite.addTest(unittest.makeSuite(TestStringUtils))
return suite
# Test discovery
def discover_tests():
"""Discover and run all tests in a directory."""
loader = unittest.TestLoader()
start_dir = os.path.dirname(__file__)
suite = loader.discover(start_dir, pattern='test_*.py')
return suite
# Custom test runner with output formatting
class CustomTestRunner(unittest.TextTestRunner):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.stream = unittest.runner._WritelnDecorator(sys.stderr)
def run(self, test):
result = super().run(test)
print(f"\nTests run: {result.testsRun}")
print(f"Failures: {len(result.failures)}")
print(f"Errors: {len(result.errors)}")
print(f"Skipped: {len(result.skipped)}")
return result
# Parameterized tests
class TestMathOperations(unittest.TestCase):
def test_addition_with_multiple_inputs(self):
test_cases = [
(2, 3, 5),
(-1, 1, 0),
(0, 0, 0),
(2.5, 3.5, 6.0),
(100, 200, 300)
]
for a, b, expected in test_cases:
with self.subTest(a=a, b=b):
self.assertEqual(a + b, expected)
# Skipping tests
class TestSkipping(unittest.TestCase):
@unittest.skip("Demonstrating skipping")
def test_skipped(self):
self.fail("This won't run")
@unittest.skipIf(sys.version_info < (3, 8), "Requires Python 3.8+")
def test_version_dependent(self):
print("Running version-dependent test")
@unittest.skipUnless(sys.platform.startswith("linux"), "Linux only")
def test_linux_only(self):
print("Running Linux-specific test")
def test_maybe_skipped(self):
if not hasattr(math, 'comb'):
self.skipTest("math.comb not available")
self.assertEqual(math.comb(5, 2), 10)
# Expected failures
class TestExpectedFailures(unittest.TestCase):
@unittest.expectedFailure
def test_known_failure(self):
self.assertEqual(1, 2) # This will fail but test passes
def test_unreliable(self):
import random
if random.random() < 0.5:
self.fail("Random failure")Mocking with unittest.mock:
from unittest.mock import Mock, patch, MagicMock, PropertyMock, call
import unittest
import requests
# Class to test
class UserService:
"""Service for user operations."""
def __init__(self, api_url):
self.api_url = api_url
def get_user(self, user_id):
response = requests.get(f"{self.api_url}/users/{user_id}")
if response.status_code == 200:
return response.json()
return None
def create_user(self, user_data):
response = requests.post(f"{self.api_url}/users", json=user_data)
return response.status_code == 201
def update_user(self, user_id, user_data):
response = requests.put(f"{self.api_url}/users/{user_id}", json=user_data)
return response.status_code == 200
def delete_user(self, user_id):
response = requests.delete(f"{self.api_url}/users/{user_id}")
return response.status_code == 204
def get_all_users(self):
response = requests.get(f"{self.api_url}/users")
if response.status_code == 200:
return response.json()
return []
class TestUserService(unittest.TestCase):
"""Test UserService with mocking."""
def setUp(self):
self.service = UserService("https://api.example.com")
@patch('requests.get')
def test_get_user_success(self, mock_get):
# Configure mock
mock_response = Mock()
mock_response.status_code = 200
mock_response.json.return_value = {'id': 1, 'name': 'Alice'}
mock_get.return_value = mock_response
# Call method
result = self.service.get_user(1)
# Assertions
self.assertEqual(result, {'id': 1, 'name': 'Alice'})
mock_get.assert_called_once_with("https://api.example.com/users/1")
@patch('requests.get')
def test_get_user_not_found(self, mock_get):
# Configure mock
mock_response = Mock()
mock_response.status_code = 404
mock_get.return_value = mock_response
# Call method
result = self.service.get_user(999)
# Assertions
self.assertIsNone(result)
mock_get.assert_called_once_with("https://api.example.com/users/999")
@patch('requests.post')
def test_create_user_success(self, mock_post):
# Configure mock
mock_response = Mock()
mock_response.status_code = 201
mock_post.return_value = mock_response
# Call method
user_data = {'name': 'Bob', 'email': 'bob@example.com'}
result = self.service.create_user(user_data)
# Assertions
self.assertTrue(result)
mock_post.assert_called_once_with(
"https://api.example.com/users",
json=user_data
)
@patch('requests.post')
def test_create_user_failure(self, mock_post):
# Configure mock
mock_response = Mock()
mock_response.status_code = 400
mock_post.return_value = mock_response
# Call method
result = self.service.create_user({'invalid': 'data'})
# Assertions
self.assertFalse(result)
@patch('requests.get')
def test_get_all_users_with_pagination(self, mock_get):
# Configure mock to return different values on consecutive calls
mock_response1 = Mock()
mock_response1.status_code = 200
mock_response1.json.return_value = [{'id': 1}, {'id': 2}]
mock_response2 = Mock()
mock_response2.status_code = 200
mock_response2.json.return_value = [{'id': 3}, {'id': 4}]
mock_get.side_effect = [mock_response1, mock_response2]
# Call method multiple times
result1 = self.service.get_all_users()
result2 = self.service.get_all_users()
self.assertEqual(len(result1), 2)
self.assertEqual(len(result2), 2)
self.assertEqual(mock_get.call_count, 2)
@patch('requests.get')
def test_network_error(self, mock_get):
# Simulate network error
mock_get.side_effect = requests.exceptions.ConnectionError("Network error")
with self.assertRaises(requests.exceptions.ConnectionError):
self.service.get_user(1)
# Mocking with side effects and exceptions
class TestAdvancedMocking(unittest.TestCase):
def test_mock_side_effect(self):
mock = Mock()
mock.side_effect = [1, 2, 3, ValueError("Error")]
self.assertEqual(mock(), 1)
self.assertEqual(mock(), 2)
self.assertEqual(mock(), 3)
with self.assertRaises(ValueError):
mock()
def test_mock_call_count(self):
mock = Mock()
mock(1)
mock(2, a=3)
mock(3, b=4)
self.assertEqual(mock.call_count, 3)
mock.assert_has_calls([
call(1),
call(2, a=3),
call(3, b=4)
])
def test_mock_property(self):
mock = Mock()
type(mock).name = PropertyMock(return_value="Test")
self.assertEqual(mock.name, "Test")
def test_patch_object(self):
class TestClass:
def method(self):
return "original"
with patch.object(TestClass, 'method', return_value="mocked"):
obj = TestClass()
self.assertEqual(obj.method(), "mocked")
obj = TestClass()
self.assertEqual(obj.method(), "original")pytest is a more powerful and feature-rich testing framework:
# First install: pip install pytest pytest-cov pytest-mock
import pytest
from typing import List
import sys
import os
# Add project root to path
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
# Code to test
class ShoppingCart:
"""Shopping cart implementation."""
def __init__(self):
self.items = []
self.discount = 0.0
def add_item(self, name: str, price: float, quantity: int = 1):
if quantity <= 0:
raise ValueError("Quantity must be positive")
if price < 0:
raise ValueError("Price cannot be negative")
self.items.append({
'name': name,
'price': price,
'quantity': quantity
})
def remove_item(self, name: str):
self.items = [item for item in self.items if item['name'] != name]
def total(self) -> float:
subtotal = sum(item['price'] * item['quantity'] for item in self.items)
return subtotal * (1 - self.discount)
def apply_discount(self, discount_percent: float):
if not 0 <= discount_percent <= 100:
raise ValueError("Discount must be between 0 and 100")
self.discount = discount_percent / 100
def item_count(self) -> int:
return len(self.items)
def clear(self):
self.items.clear()
self.discount = 0.0
# Basic pytest tests
def test_shopping_cart_initialization():
"""Test cart initialization."""
cart = ShoppingCart()
assert cart.items == []
assert cart.discount == 0.0
assert cart.total() == 0.0
def test_add_item():
"""Test adding items to cart."""
cart = ShoppingCart()
cart.add_item("Apple", 0.5, 3)
cart.add_item("Banana", 0.3, 2)
assert len(cart.items) == 2
assert cart.total() == (0.5 * 3) + (0.3 * 2)
def test_add_item_negative_quantity():
"""Test adding item with negative quantity."""
cart = ShoppingCart()
with pytest.raises(ValueError, match="Quantity must be positive"):
cart.add_item("Apple", 0.5, -1)
def test_remove_item():
"""Test removing items from cart."""
cart = ShoppingCart()
cart.add_item("Apple", 0.5)
cart.add_item("Banana", 0.3)
cart.remove_item("Apple")
assert len(cart.items) == 1
assert cart.items[0]['name'] == "Banana"
def test_apply_discount():
"""Test applying discount."""
cart = ShoppingCart()
cart.add_item("Laptop", 1000)
cart.apply_discount(10)
assert cart.discount == 0.1
assert cart.total() == 900
def test_apply_invalid_discount():
"""Test applying invalid discount."""
cart = ShoppingCart()
with pytest.raises(ValueError, match="Discount must be between 0 and 100"):
cart.apply_discount(150)
# Using fixtures
@pytest.fixture
def empty_cart():
"""Fixture providing empty cart."""
return ShoppingCart()
@pytest.fixture
def filled_cart():
"""Fixture providing cart with items."""
cart = ShoppingCart()
cart.add_item("Apple", 0.5, 3)
cart.add_item("Banana", 0.3, 2)
cart.add_item("Orange", 0.4, 1)
return cart
@pytest.fixture
def sample_items():
"""Fixture providing sample item data."""
return [
{'name': 'Apple', 'price': 0.5, 'quantity': 3},
{'name': 'Banana', 'price': 0.3, 'quantity': 2},
{'name': 'Orange', 'price': 0.4, 'quantity': 1}
]
def test_empty_cart_total(empty_cart):
"""Test empty cart total."""
assert empty_cart.total() == 0.0
assert empty_cart.item_count() == 0
def test_filled_cart_total(filled_cart):
"""Test filled cart total."""
expected = (0.5 * 3) + (0.3 * 2) + (0.4 * 1)
assert filled_cart.total() == expected
def test_filled_cart_item_count(filled_cart):
"""Test item count."""
assert filled_cart.item_count() == 3
def test_clear_cart(filled_cart):
"""Test clearing cart."""
filled_cart.clear()
assert filled_cart.item_count() == 0
assert filled_cart.total() == 0.0
assert filled_cart.discount == 0.0
# Parameterized tests
@pytest.mark.parametrize("price,quantity,expected", [
(10, 2, 20),
(5.5, 3, 16.5),
(0, 5, 0),
(100, 0, 0),
])
def test_calculate_item_total(price, quantity, expected):
"""Test item total calculation."""
cart = ShoppingCart()
cart.add_item("Test", price, quantity)
assert cart.total() == expected
@pytest.mark.parametrize("discount,expected_multiplier", [
(0, 1.0),
(10, 0.9),
(25, 0.75),
(50, 0.5),
(100, 0.0),
])
def test_discount_multiplier(discount, expected_multiplier):
"""Test discount multiplier calculation."""
cart = ShoppingCart()
cart.apply_discount(discount)
assert cart.discount == expected_multiplier
# Grouping tests
class TestShoppingCartAdvanced:
"""Advanced test cases grouped in class."""
@pytest.mark.parametrize("items,expected_total", [
([('Apple', 1.0, 2)], 2.0),
([('Apple', 1.0, 2), ('Banana', 2.0, 1)], 4.0),
([('Apple', 1.5, 3), ('Banana', 2.5, 2), ('Orange', 3.5, 1)], 13.0),
])
def test_multiple_items_total(self, items, expected_total):
cart = ShoppingCart()
for name, price, quantity in items:
cart.add_item(name, price, quantity)
assert cart.total() == expected_total
def test_discount_with_multiple_items(self):
cart = ShoppingCart()
cart.add_item("Item1", 100)
cart.add_item("Item2", 200)
cart.apply_discount(20)
assert cart.total() == 240 # (100 + 200) * 0.8
# Mocking with pytest
def test_get_user_with_mock(mocker):
"""Test with pytest-mock."""
mock_response = mocker.Mock()
mock_response.status_code = 200
mock_response.json.return_value = {'id': 1, 'name': 'Alice'}
mock_get = mocker.patch('requests.get')
mock_get.return_value = mock_response
service = UserService("https://api.example.com")
result = service.get_user(1)
assert result == {'id': 1, 'name': 'Alice'}
mock_get.assert_called_once_with("https://api.example.com/users/1")
# Fixtures with scope
@pytest.fixture(scope="session")
def db_connection():
"""Create database connection for entire test session."""
print("\nSetting up database connection")
connection = {"connected": True}
yield connection
print("\nClosing database connection")
@pytest.fixture(scope="module")
def test_data():
"""Create test data for module."""
print("\nCreating test data")
data = [1, 2, 3, 4, 5]
yield data
print("\nCleaning up test data")
def test_with_db_connection(db_connection):
assert db_connection["connected"] is True
def test_with_test_data(test_data):
assert sum(test_data) == 15
# Markers for categorizing tests
@pytest.mark.slow
def test_slow_operation():
"""Slow test marked for selective running."""
import time
time.sleep(2)
assert True
@pytest.mark.integration
def test_integration_with_external_api():
"""Integration test marked for selective running."""
# Test with actual API (or mock in unit tests)
pass
@pytest.mark.smoke
def test_critical_functionality():
"""Smoke test for critical features."""
cart = ShoppingCart()
cart.add_item("Critical", 100)
assert cart.total() == 100
# Running specific markers
# pytest -m slow
# pytest -m "not slow"
# pytest -m "smoke or integration"
# Temporary files and directories
def test_write_to_file(tmp_path):
"""Test file operations with temporary directory."""
d = tmp_path / "sub"
d.mkdir()
f = d / "test.txt"
f.write_text("Hello, World!")
assert f.read_text() == "Hello, World!"
assert len(list(tmp_path.iterdir())) == 1
# Exception testing
def test_divide_by_zero():
with pytest.raises(ZeroDivisionError):
1 / 0
def test_exception_with_message():
with pytest.raises(ValueError, match=".*invalid.*"):
raise ValueError("invalid value")
# Approximate comparisons
def test_floating_point():
assert 0.1 + 0.2 == pytest.approx(0.3, rel=1e-9)
# Custom assertions
def pytest_assertrepr_compare(op, left, right):
"""Custom assertion representation."""
if isinstance(left, ShoppingCart) and isinstance(right, ShoppingCart) and op == "==":
return [
"Comparing ShoppingCart instances:",
f" Left items: {left.items}",
f" Right items: {right.items}",
]
# Configuration in pytest.ini
"""
[pytest]
minversion = 6.0
testpaths = tests
python_files = test_*.py
python_classes = Test*
python_functions = test_*
markers =
slow: marks tests as slow
integration: marks tests as integration tests
smoke: marks tests as smoke tests
addopts = -v --tb=short --strict-markers
"""Advanced mocking techniques:
from unittest.mock import Mock, patch, MagicMock, PropertyMock, call, create_autospec
import pytest
import requests
from datetime import datetime
# Complex class to mock
class DataProcessor:
"""Complex data processing class."""
def __init__(self, config):
self.config = config
self.data = []
self.processed = []
def load_data(self, source):
"""Load data from source."""
# Complex implementation
pass
def process_item(self, item):
"""Process single item."""
# Complex implementation
return item * 2
def save_results(self, results):
"""Save results to database."""
# Database operation
pass
def run_pipeline(self, source):
"""Run complete processing pipeline."""
self.load_data(source)
for item in self.data:
result = self.process_item(item)
self.processed.append(result)
self.save_results(self.processed)
return len(self.processed)
# Mocking with autospec
def test_create_autospec():
"""Test using autospec to maintain signature."""
mock_processor = create_autospec(DataProcessor)
# Valid call
mock_processor.run_pipeline("test")
# Invalid call would raise TypeError
with pytest.raises(TypeError):
mock_processor.run_pipeline("test", "extra_arg")
# Mocking properties
class TemperatureSensor:
@property
def temperature(self):
# Read from hardware
return 25.5
@property
def status(self):
return "OK"
def test_sensor_mocking():
"""Test mocking properties."""
mock_sensor = Mock(spec=TemperatureSensor)
# Mock properties
type(mock_sensor).temperature = PropertyMock(return_value=30.0)
type(mock_sensor).status = PropertyMock(return_value="ERROR")
assert mock_sensor.temperature == 30.0
assert mock_sensor.status == "ERROR"
# Mocking context managers
class DatabaseConnection:
def __enter__(self):
print("Opening connection")
return self
def __exit__(self, *args):
print("Closing connection")
def query(self, sql):
return ["result1", "result2"]
def test_context_manager_mock():
"""Test mocking context manager."""
mock_db = MagicMock(spec=DatabaseConnection)
mock_db.query.return_value = ["mocked_result"]
# Mock __enter__ to return the mock itself
mock_db.__enter__.return_value = mock_db
with mock_db as db:
results = db.query("SELECT * FROM users")
assert results == ["mocked_result"]
mock_db.__enter__.assert_called_once()
mock_db.__exit__.assert_called_once()
# Mocking decorators
def retry_on_failure(max_retries=3):
"""Decorator for retrying failed operations."""
def decorator(func):
def wrapper(*args, **kwargs):
for attempt in range(max_retries):
try:
return func(*args, **kwargs)
except Exception as e:
if attempt == max_retries - 1:
raise
return None
return wrapper
return decorator
class APIClient:
@retry_on_failure(max_retries=3)
def call_api(self):
# Actual API call
response = requests.get("https://api.example.com/data")
return response.json()
def test_decorator_mocking():
"""Test mocking function with decorator."""
mock_client = MagicMock(spec=APIClient)
mock_client.call_api.side_effect = [Exception("First"), Exception("Second"), {"data": "success"}]
# This would normally retry, but our mock controls the behavior
with patch.object(APIClient, 'call_api', mock_client.call_api):
client = APIClient()
try:
result = client.call_api()
except Exception:
pass
# Verify call count
assert mock_client.call_api.call_count == 3
# Mocking async functions
import asyncio
class AsyncService:
async def fetch_data(self):
await asyncio.sleep(1)
return {"data": "important"}
async def process(self):
data = await self.fetch_data()
return data["data"].upper()
@pytest.mark.asyncio
async def test_async_mocking():
"""Test mocking async functions."""
mock_service = MagicMock(spec=AsyncService)
# Mock async method
mock_fetch = asyncio.Future()
mock_fetch.set_result({"data": "mocked"})
mock_service.fetch_data.return_value = mock_fetch
# Mock process to return directly
mock_service.process.return_value = "MOCKED"
result = await mock_service.process()
assert result == "MOCKED"
# Mocking datetime
from datetime import datetime, date
def test_datetime_mocking():
"""Test mocking datetime."""
with patch('datetime.datetime') as mock_datetime:
mock_datetime.now.return_value = datetime(2024, 1, 15, 12, 0, 0)
now = datetime.now()
assert now.year == 2024
assert now.month == 1
assert now.day == 15
# Mocking environment variables
import os
def test_environment_mocking():
"""Test mocking environment variables."""
with patch.dict('os.environ', {'API_KEY': 'test_key', 'DEBUG': 'true'}):
assert os.environ.get('API_KEY') == 'test_key'
assert os.environ.get('DEBUG') == 'true'
# Outside context, original env restored
assert os.environ.get('API_KEY') is None
# Mocking multiple methods
class ServiceManager:
def __init__(self):
self.services = []
def start_service(self, name):
# Complex startup
return True
def stop_service(self, name):
# Complex shutdown
return True
def restart_all(self):
for service in self.services:
self.stop_service(service)
self.start_service(service)
def test_multiple_methods_mocking():
"""Test mocking multiple methods simultaneously."""
manager = ServiceManager()
manager.services = ["web", "db", "cache"]
with patch.multiple(
ServiceManager,
start_service=Mock(return_value=True),
stop_service=Mock(return_value=True)
):
manager.restart_all()
assert manager.stop_service.call_count == 3
assert manager.start_service.call_count == 3
# Verify call order
expected_calls = [call("web"), call("db"), call("cache")]
manager.stop_service.assert_has_calls(expected_calls)
manager.start_service.assert_has_calls(expected_calls)
# Mocking with side effects and callbacks
def test_callback_mocking():
"""Test mocking with callbacks."""
def callback_handler(mock, *args, **kwargs):
print(f"Mock called with {args}, {kwargs}")
return "handled"
mock = Mock()
mock.side_effect = callback_handler
result = mock(1, 2, test="value")
assert result == "handled"
# Mocking and patching class attributes
class Config:
settings = {
'debug': False,
'timeout': 30,
'retries': 3
}
@classmethod
def get(cls, key):
return cls.settings.get(key)
def test_class_attribute_mocking():
"""Test mocking class attributes."""
with patch.object(Config, 'settings', {'debug': True, 'timeout': 60}):
assert Config.get('debug') is True
assert Config.get('timeout') == 60
assert Config.get('retries') is None
# Outside context
assert Config.get('debug') is FalseTest-Driven Development workflow with examples:
import unittest
from typing import List, Optional
# Step 1: Write tests first
class TestTodoList(unittest.TestCase):
"""Test cases for TodoList (written first)."""
def test_create_empty_todo_list(self):
"""Test creating an empty todo list."""
todo = TodoList()
self.assertEqual(len(todo), 0)
self.assertEqual(todo.get_all(), [])
def test_add_task(self):
"""Test adding a task to the list."""
todo = TodoList()
task_id = todo.add_task("Buy groceries")
self.assertEqual(len(todo), 1)
task = todo.get_task(task_id)
self.assertEqual(task['description'], "Buy groceries")
self.assertFalse(task['completed'])
self.assertIn('id', task)
self.assertIn('created_at', task)
def test_add_task_with_empty_description(self):
"""Test adding task with empty description raises error."""
todo = TodoList()
with self.assertRaises(ValueError):
todo.add_task("")
with self.assertRaises(ValueError):
todo.add_task(" ")
def test_complete_task(self):
"""Test marking a task as completed."""
todo = TodoList()
task_id = todo.add_task("Write code")
result = todo.complete_task(task_id)
self.assertTrue(result)
task = todo.get_task(task_id)
self.assertTrue(task['completed'])
self.assertIsNotNone(task['completed_at'])
def test_complete_nonexistent_task(self):
"""Test completing a task that doesn't exist."""
todo = TodoList()
result = todo.complete_task(999)
self.assertFalse(result)
def test_delete_task(self):
"""Test deleting a task."""
todo = TodoList()
task_id = todo.add_task("Temporary task")
self.assertEqual(len(todo), 1)
result = todo.delete_task(task_id)
self.assertTrue(result)
self.assertEqual(len(todo), 0)
def test_delete_nonexistent_task(self):
"""Test deleting a task that doesn't exist."""
todo = TodoList()
result = todo.delete_task(999)
self.assertFalse(result)
def test_get_completed_tasks(self):
"""Test getting only completed tasks."""
todo = TodoList()
todo.add_task("Task 1")
task2_id = todo.add_task("Task 2")
todo.add_task("Task 3")
todo.complete_task(task2_id)
completed = todo.get_completed()
self.assertEqual(len(completed), 1)
self.assertEqual(completed[0]['description'], "Task 2")
def test_get_pending_tasks(self):
"""Test getting only pending tasks."""
todo = TodoList()
todo.add_task("Task 1")
task2_id = todo.add_task("Task 2")
todo.add_task("Task 3")
todo.complete_task(task2_id)
pending = todo.get_pending()
self.assertEqual(len(pending), 2)
self.assertEqual(pending[0]['description'], "Task 1")
self.assertEqual(pending[1]['description'], "Task 3")
def test_update_task(self):
"""Test updating task description."""
todo = TodoList()
task_id = todo.add_task("Old description")
result = todo.update_task(task_id, "New description")
self.assertTrue(result)
task = todo.get_task(task_id)
self.assertEqual(task['description'], "New description")
def test_clear_all_tasks(self):
"""Test clearing all tasks."""
todo = TodoList()
todo.add_task("Task 1")
todo.add_task("Task 2")
todo.add_task("Task 3")
self.assertEqual(len(todo), 3)
todo.clear()
self.assertEqual(len(todo), 0)
def test_search_tasks(self):
"""Test searching tasks by keyword."""
todo = TodoList()
todo.add_task("Buy milk")
todo.add_task("Buy bread")
todo.add_task("Call mom")
todo.add_task("Write report")
results = todo.search("buy")
self.assertEqual(len(results), 2)
for task in results:
self.assertIn("buy", task['description'].lower())
# Step 2: Implement the class to make tests pass
import time
import re
from typing import List, Dict, Optional
class TodoList:
"""Todo list implementation (written after tests)."""
def __init__(self):
self._tasks = {}
self._next_id = 1
def __len__(self) -> int:
return len(self._tasks)
def add_task(self, description: str) -> int:
"""Add a new task."""
if not description or not description.strip():
raise ValueError("Task description cannot be empty")
task_id = self._next_id
self._next_id += 1
self._tasks[task_id] = {
'id': task_id,
'description': description.strip(),
'completed': False,
'created_at': time.time(),
'completed_at': None
}
return task_id
def get_task(self, task_id: int) -> Optional[Dict]:
"""Get task by ID."""
return self._tasks.get(task_id)
def get_all(self) -> List[Dict]:
"""Get all tasks."""
return list(self._tasks.values())
def complete_task(self, task_id: int) -> bool:
"""Mark task as completed."""
if task_id not in self._tasks:
return False
self._tasks[task_id]['completed'] = True
self._tasks[task_id]['completed_at'] = time.time()
return True
def delete_task(self, task_id: int) -> bool:
"""Delete a task."""
if task_id not in self._tasks:
return False
del self._tasks[task_id]
return True
def get_completed(self) -> List[Dict]:
"""Get completed tasks."""
return [t for t in self._tasks.values() if t['completed']]
def get_pending(self) -> List[Dict]:
"""Get pending tasks."""
return [t for t in self._tasks.values() if not t['completed']]
def update_task(self, task_id: int, description: str) -> bool:
"""Update task description."""
if task_id not in self._tasks:
return False
if not description or not description.strip():
raise ValueError("Task description cannot be empty")
self._tasks[task_id]['description'] = description.strip()
return True
def clear(self):
"""Clear all tasks."""
self._tasks.clear()
self._next_id = 1
def search(self, keyword: str) -> List[Dict]:
"""Search tasks by keyword."""
keyword = keyword.lower()
return [
t for t in self._tasks.values()
if keyword in t['description'].lower()
]
# Step 3: Refactor and add more features with tests
class TestTodoListAdvanced(unittest.TestCase):
"""Additional tests for new features."""
def setUp(self):
self.todo = TodoList()
def test_priority_levels(self):
"""Test task priority."""
task_id = self.todo.add_task("High priority task")
self.todo.set_priority(task_id, 3)
task = self.todo.get_task(task_id)
self.assertEqual(task['priority'], 3)
def test_due_dates(self):
"""Test task due dates."""
import datetime
due_date = datetime.datetime.now() + datetime.timedelta(days=7)
task_id = self.todo.add_task("Task with due date")
self.todo.set_due_date(task_id, due_date)
task = self.todo.get_task(task_id)
self.assertEqual(task['due_date'], due_date)
def test_overdue_tasks(self):
"""Test finding overdue tasks."""
import datetime
# Add tasks with different due dates
task1_id = self.todo.add_task("Past due")
self.todo.set_due_date(task1_id, datetime.datetime.now() - datetime.timedelta(days=1))
task2_id = self.todo.add_task("Future due")
self.todo.set_due_date(task2_id, datetime.datetime.now() + datetime.timedelta(days=1))
self.todo.add_task("No due date")
overdue = self.todo.get_overdue()
self.assertEqual(len(overdue), 1)
self.assertEqual(overdue[0]['description'], "Past due")
def test_task_categories(self):
"""Test task categorization."""
task_id = self.todo.add_task("Buy milk")
self.todo.add_category(task_id, "shopping")
task_id2 = self.todo.add_task("Write code")
self.todo.add_category(task_id2, "work")
shopping_tasks = self.todo.get_by_category("shopping")
self.assertEqual(len(shopping_tasks), 1)
self.assertEqual(shopping_tasks[0]['description'], "Buy milk")
def test_completion_percentage(self):
"""Test completion percentage calculation."""
for i in range(10):
self.todo.add_task(f"Task {i}")
# Complete 3 tasks
for i, task_id in enumerate(self.todo.get_all()):
if i < 3:
self.todo.complete_task(task_id['id'])
self.assertEqual(self.todo.completion_percentage(), 30)
# TDD workflow demonstration
def tdd_workflow_example():
"""Demonstrate TDD workflow."""
print("TDD Workflow:")
print("1. Write a failing test")
print("2. Write minimal code to pass the test")
print("3. Refactor if needed")
print("4. Repeat")
# Run tests
suite = unittest.TestLoader().loadTestsFromTestCase(TestTodoList)
runner = unittest.TextTestRunner(verbosity=2)
result = runner.run(suite)
print(f"\nTests run: {result.testsRun}")
print(f"Failures: {len(result.failures)}")
print(f"Errors: {len(result.errors)}")
# if __name__ == '__main__':
# tdd_workflow_example()Measuring test coverage:
# First install: pip install coverage pytest-cov
"""
Coverage configuration (.coveragerc):
[run]
source = myproject
omit = */tests/*,*/migrations/*,*/admin.py
[report]
exclude_lines =
pragma: no cover
def __repr__
raise AssertionError
raise NotImplementedError
if 0:
if __name__ == .__main__.:
[html]
directory = coverage_html_report
"""
import coverage
import pytest
def run_coverage_report():
"""Run tests with coverage and generate report."""
# Start coverage
cov = coverage.Coverage()
cov.start()
# Run tests
pytest.main(['-v', 'tests/'])
# Stop coverage
cov.stop()
cov.save()
# Generate reports
print("\n=== Coverage Report ===")
cov.report()
# Generate HTML report
cov.html_report(directory='coverage_html')
print("\nHTML report generated in 'coverage_html' directory")
# Generate XML report for CI
cov.xml_report(outfile='coverage.xml')
print("XML report generated in 'coverage.xml'")
# Command line usage
"""
# Run with coverage
coverage run -m pytest
coverage report -m
coverage html
# Run with pytest-cov
pytest --cov=myproject tests/
pytest --cov=myproject --cov-report=html tests/
pytest --cov=myproject --cov-report=xml tests/
# Combine coverage from multiple test runs
coverage combine
coverage report
"""
# Example: Ensuring 100% coverage
def function_to_test(x):
"""Function that needs full coverage."""
if x < 0:
return "negative"
elif x == 0:
return "zero"
elif x < 10:
return "small"
elif x < 100:
return "medium"
else:
return "large"
# Complete test for full coverage
def test_complete_coverage():
"""Test all branches for 100% coverage."""
assert function_to_test(-5) == "negative"
assert function_to_test(0) == "zero"
assert function_to_test(5) == "small"
assert function_to_test(50) == "medium"
assert function_to_test(200) == "large"
# Coverage configuration in pyproject.toml
"""
[tool.coverage.run]
source = ["myproject"]
omit = ["*/tests/*", "*/migrations/*", "*/admin.py"]
[tool.coverage.report]
exclude_lines = [
"pragma: no cover",
"def __repr__",
"raise AssertionError",
"raise NotImplementedError",
"if 0:",
"if __name__ == .__main__.:",
"@abstractmethod",
]
"""
# Integration with CI/CD
"""
# GitHub Actions workflow
name: Tests
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: '3.11'
- name: Install dependencies
run: |
pip install -r requirements.txt
pip install pytest pytest-cov
- name: Run tests with coverage
run: pytest --cov=myproject --cov-report=xml
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v3
with:
file: ./coverage.xml
flags: unittests
name: codecov-umbrella
"""Continuous Integration and Deployment with GitHub Actions:
# .github/workflows/ci.yml
name: CI/CD Pipeline
on:
push:
branches: [ main, develop ]
pull_request:
branches: [ main ]
release:
types: [ published ]
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ['3.9', '3.10', '3.11']
django-version: ['3.2', '4.0', '4.1']
steps:
- uses: actions/checkout@v3
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- name: Cache pip packages
uses: actions/cache@v3
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('requirements.txt') }}
restore-keys: |
${{ runner.os }}-pip-
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
pip install Django==${{ matrix.django-version }}
pip install pytest pytest-cov flake8 black mypy
- name: Lint with flake8
run: |
flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
flake8 . --count --exit-zero --max-complexity=10 --statistics
- name: Check formatting with black
run: black --check .
- name: Type check with mypy
run: mypy myproject
- name: Run tests with pytest
run: |
pytest --cov=myproject --cov-report=xml --cov-report=html
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v3
with:
file: ./coverage.xml
flags: unittests
name: codecov-umbrella
fail_ci_if_error: false
security:
runs-on: ubuntu-latest
needs: test
steps:
- uses: actions/checkout@v3
- name: Run Bandit security scan
run: |
pip install bandit
bandit -r myproject -f json -o bandit-report.json
- name: Run Safety check
run: |
pip install safety
safety check -r requirements.txt --full-report
- name: Upload security reports
uses: actions/upload-artifact@v3
with:
name: security-reports
path: |
bandit-report.json
safety-report.txt
docker-build:
runs-on: ubuntu-latest
needs: [test, security]
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
steps:
- uses: actions/checkout@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
- name: Login to DockerHub
uses: docker/login-action@v2
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Build and push Docker image
uses: docker/build-push-action@v4
with:
context: .
push: true
tags: |
${{ secrets.DOCKER_USERNAME }}/myapp:latest
${{ secrets.DOCKER_USERNAME }}/myapp:${{ github.sha }}
cache-from: type=registry,ref=${{ secrets.DOCKER_USERNAME }}/myapp:buildcache
cache-to: type=registry,ref=${{ secrets.DOCKER_USERNAME }}/myapp:buildcache,mode=max
deploy-staging:
runs-on: ubuntu-latest
needs: docker-build
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
environment: staging
steps:
- uses: actions/checkout@v3
- name: Deploy to staging
env:
DEPLOY_KEY: ${{ secrets.DEPLOY_KEY }}
STAGING_HOST: ${{ secrets.STAGING_HOST }}
run: |
echo "$DEPLOY_KEY" > deploy_key
chmod 600 deploy_key
ssh -i deploy_key -o StrictHostKeyChecking=no user@$STAGING_HOST "
cd /app &&
docker-compose pull &&
docker-compose up -d
"
- name: Run smoke tests
run: |
curl -f http://${{ secrets.STAGING_HOST }}/health || exit 1
deploy-production:
runs-on: ubuntu-latest
needs: deploy-staging
if: github.event_name == 'release'
environment: production
steps:
- uses: actions/checkout@v3
- name: Deploy to production
env:
DEPLOY_KEY: ${{ secrets.DEPLOY_KEY }}
PRODUCTION_HOST: ${{ secrets.PRODUCTION_HOST }}
run: |
echo "$DEPLOY_KEY" > deploy_key
chmod 600 deploy_key
ssh -i deploy_key -o StrictHostKeyChecking=no user@$PRODUCTION_HOST "
cd /app &&
docker-compose pull &&
docker-compose up -d &&
docker system prune -f
"
- name: Run health checks
run: |
for i in {1..30}; do
if curl -f http://${{ secrets.PRODUCTION_HOST }}/health; then
exit 0
fi
sleep 10
done
exit 1
- name: Notify deployment success
uses: rtCamp/action-slack-notify@v2
env:
SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
SLACK_MESSAGE: "Production deployment successful :rocket:"
SLACK_COLOR: goodPython Package Publishing:
# .github/workflows/publish.yml
name: Publish to PyPI
on:
release:
types: [created]
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.11'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install build twine
- name: Build package
run: python -m build
- name: Publish to PyPI
env:
TWINE_USERNAME: __token__
TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
run: twine upload dist/*
- name: Publish to TestPyPI
env:
TWINE_USERNAME: __token__
TWINE_PASSWORD: ${{ secrets.TEST_PYPI_API_TOKEN }}
run: |
twine upload --repository-url https://test.pypi.org/legacy/ dist/* || trueContainerization with Docker:
# Dockerfile
FROM python:3.11-slim as builder
# Set environment variables
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
PIP_NO_CACHE_DIR=1 \
PIP_DISABLE_PIP_VERSION_CHECK=1
WORKDIR /app
# Install system dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
gcc \
libpq-dev \
&& rm -rf /var/lib/apt/lists/*
# Install Python dependencies
COPY requirements.txt .
RUN pip wheel --no-cache-dir --no-deps --wheel-dir /app/wheels -r requirements.txt
# Final stage
FROM python:3.11-slim
WORKDIR /app
# Install runtime dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
libpq-dev \
curl \
&& rm -rf /var/lib/apt/lists/*
# Copy wheels from builder
COPY --from=builder /app/wheels /wheels
COPY --from=builder /app/requirements.txt .
# Install Python packages
RUN pip install --no-cache /wheels/*
# Copy application code
COPY . .
# Create non-root user
RUN useradd -m -u 1000 appuser && chown -R appuser:appuser /app
USER appuser
# Health check
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
CMD curl -f http://localhost:8000/health || exit 1
# Run application
CMD ["gunicorn", "--bind", "0.0.0.0:8000", "--workers", "4", "--threads", "2", "myproject.wsgi:application"]Docker Compose:
# docker-compose.yml
version: '3.8'
services:
db:
image: postgres:15-alpine
container_name: myapp-db
environment:
POSTGRES_DB: myapp
POSTGRES_USER: myapp
POSTGRES_PASSWORD: ${DB_PASSWORD:-secret}
volumes:
- postgres_data:/var/lib/postgresql/data
ports:
- "5432:5432"
healthcheck:
test: ["CMD-SHELL", "pg_isready -U myapp"]
interval: 10s
timeout: 5s
retries: 5
redis:
image: redis:7-alpine
container_name: myapp-redis
ports:
- "6379:6379"
volumes:
- redis_data:/data
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 10s
timeout: 5s
retries: 5
web:
build: .
container_name: myapp-web
environment:
DATABASE_URL: postgresql://myapp:${DB_PASSWORD:-secret}@db:5432/myapp
REDIS_URL: redis://redis:6379/0
SECRET_KEY: ${SECRET_KEY:-django-insecure-dev-key}
DEBUG: ${DEBUG:-false}
ports:
- "8000:8000"
depends_on:
db:
condition: service_healthy
redis:
condition: service_healthy
volumes:
- static_volume:/app/staticfiles
- media_volume:/app/media
command: >
sh -c "
python manage.py migrate &&
python manage.py collectstatic --noinput &&
gunicorn --bind 0.0.0.0:8000 --workers 4 --threads 2 myproject.wsgi:application
"
nginx:
image: nginx:alpine
container_name: myapp-nginx
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf:ro
- static_volume:/static:ro
- media_volume:/media:ro
ports:
- "80:80"
- "443:443"
depends_on:
- web
celery:
build: .
container_name: myapp-celery
environment:
DATABASE_URL: postgresql://myapp:${DB_PASSWORD:-secret}@db:5432/myapp
REDIS_URL: redis://redis:6379/0
SECRET_KEY: ${SECRET_KEY:-django-insecure-dev-key}
depends_on:
db:
condition: service_healthy
redis:
condition: service_healthy
command: celery -A myproject worker --loglevel=info
celery-beat:
build: .
container_name: myapp-celery-beat
environment:
DATABASE_URL: postgresql://myapp:${DB_PASSWORD:-secret}@db:5432/myapp
REDIS_URL: redis://redis:6379/0
SECRET_KEY: ${SECRET_KEY:-django-insecure-dev-key}
depends_on:
db:
condition: service_healthy
redis:
condition: service_healthy
command: celery -A myproject beat --loglevel=info
volumes:
postgres_data:
redis_data:
static_volume:
media_volume:Nginx Configuration:
# nginx.conf
events {
worker_connections 1024;
}
http {
upstream django {
server web:8000;
}
server {
listen 80;
server_name example.com;
return 301 https://$server_name$request_uri;
}
server {
listen 443 ssl http2;
server_name example.com;
ssl_certificate /etc/nginx/ssl/cert.pem;
ssl_certificate_key /etc/nginx/ssl/key.pem;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers HIGH:!aNULL:!MD5;
location /static/ {
alias /static/;
expires 30d;
add_header Cache-Control "public, immutable";
}
location /media/ {
alias /media/;
expires 30d;
add_header Cache-Control "public, immutable";
}
location / {
proxy_pass http://django;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_redirect off;
}
location /health {
access_log off;
return 200 "healthy\n";
add_header Content-Type text/plain;
}
}
}Orchestration with Kubernetes:
# k8s/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
namespace: production
labels:
app: myapp
environment: production
spec:
replicas: 3
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
selector:
matchLabels:
app: myapp
template:
metadata:
labels:
app: myapp
spec:
containers:
- name: web
image: ${DOCKER_USERNAME}/myapp:latest
imagePullPolicy: Always
ports:
- containerPort: 8000
name: http
env:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: myapp-secrets
key: database-url
- name: SECRET_KEY
valueFrom:
secretKeyRef:
name: myapp-secrets
key: secret-key
- name: REDIS_URL
value: redis://redis-service:6379/0
resources:
requests:
memory: "256Mi"
cpu: "250m"
limits:
memory: "512Mi"
cpu: "500m"
livenessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 5
periodSeconds: 5
volumeMounts:
- name: static-volume
mountPath: /app/staticfiles
- name: media-volume
mountPath: /app/media
volumes:
- name: static-volume
persistentVolumeClaim:
claimName: static-pvc
- name: media-volume
persistentVolumeClaim:
claimName: media-pvc
---
# k8s/service.yaml
apiVersion: v1
kind: Service
metadata:
name: myapp-service
namespace: production
spec:
selector:
app: myapp
ports:
- port: 80
targetPort: 8000
name: http
type: ClusterIP
---
# k8s/ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: myapp-ingress
namespace: production
annotations:
kubernetes.io/ingress.class: nginx
cert-manager.io/cluster-issuer: letsencrypt-prod
nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
tls:
- hosts:
- myapp.example.com
secretName: myapp-tls
rules:
- host: myapp.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: myapp-service
port:
number: 80
---
# k8s/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: myapp-config
namespace: production
data:
DEBUG: "false"
ALLOWED_HOSTS: myapp.example.com,localhost
TIME_ZONE: UTC
LOG_LEVEL: INFO
---
# k8s/secrets.yaml
apiVersion: v1
kind: Secret
metadata:
name: myapp-secrets
namespace: production
type: Opaque
data:
database-url: <base64-encoded-database-url>
secret-key: <base64-encoded-secret-key>
redis-password: <base64-encoded-redis-password>
---
# k8s/persistent-volumes.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: static-pvc
namespace: production
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 10Gi
storageClassName: nfs-client
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: media-pvc
namespace: production
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 50Gi
storageClassName: nfs-client
---
# k8s/hpa.yaml (Horizontal Pod Autoscaler)
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: myapp-hpa
namespace: production
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: myapp
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
- type: Resource
resource:
name: memory
target:
type: Utilization
averageUtilization: 80Helm Chart:
# helm/myapp/Chart.yaml
apiVersion: v2
name: myapp
description: A Helm chart for Kubernetes
type: application
version: 0.1.0
appVersion: "1.0.0"
dependencies:
- name: postgresql
version: 12.1.0
repository: https://charts.bitnami.com/bitnami
condition: postgresql.enabled
- name: redis
version: 17.3.0
repository: https://charts.bitnami.com/bitnami
condition: redis.enabled
# helm/myapp/values.yaml
replicaCount: 3
image:
repository: myregistry/myapp
tag: latest
pullPolicy: Always
nameOverride: ""
fullnameOverride: ""
service:
type: ClusterIP
port: 80
ingress:
enabled: true
className: nginx
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
hosts:
- host: myapp.example.com
paths:
- path: /
pathType: Prefix
tls:
- secretName: myapp-tls
hosts:
- myapp.example.com
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 250m
memory: 256Mi
autoscaling:
enabled: true
minReplicas: 2
maxReplicas: 10
targetCPUUtilizationPercentage: 70
targetMemoryUtilizationPercentage: 80
env:
DEBUG: "false"
ALLOWED_HOSTS: myapp.example.com
secrets:
databaseUrl: ""
secretKey: ""
postgresql:
enabled: true
auth:
database: myapp
username: myapp
password: ""
primary:
persistence:
size: 10Gi
redis:
enabled: true
auth:
enabled: false
master:
persistence:
size: 5Gi
# helm/myapp/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "myapp.fullname" . }}
labels:
{{- include "myapp.labels" . | nindent 4 }}
spec:
{{- if not .Values.autoscaling.enabled }}
replicas: {{ .Values.replicaCount }}
{{- end }}
selector:
matchLabels:
{{- include "myapp.selectorLabels" . | nindent 6 }}
template:
metadata:
labels:
{{- include "myapp.selectorLabels" . | nindent 8 }}
spec:
containers:
- name: {{ .Chart.Name }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
ports:
- containerPort: 8000
name: http
env:
- name: DATABASE_URL
valueFrom:
secretKeyRef:
name: {{ include "myapp.fullname" . }}
key: database-url
- name: SECRET_KEY
valueFrom:
secretKeyRef:
name: {{ include "myapp.fullname" . }}
key: secret-key
{{- range $key, $value := .Values.env }}
- name: {{ $key }}
value: {{ $value | quote }}
{{- end }}
resources:
{{- toYaml .Values.resources | nindent 10 }}Blue-Green Deployment:
# deploy/blue_green.py
import subprocess
import time
import sys
import requests
class BlueGreenDeployment:
"""Blue-Green deployment strategy."""
def __init__(self, app_name, k8s_namespace):
self.app_name = app_name
self.namespace = k8s_namespace
self.blue_service = f"{app_name}-blue"
self.green_service = f"{app_name}-green"
self.active_service = None
def get_current_active(self):
"""Determine which environment is currently active."""
try:
# Check blue service endpoints
result = subprocess.run(
["kubectl", "get", "svc", self.blue_service,
"-n", self.namespace, "-o", "jsonpath={.spec.selector.version}"],
capture_output=True, text=True
)
if result.returncode == 0 and result.stdout:
return "blue"
# Check green service
result = subprocess.run(
["kubectl", "get", "svc", self.green_service,
"-n", self.namespace, "-o", "jsonpath={.spec.selector.version}"],
capture_output=True, text=True
)
if result.returncode == 0 and result.stdout:
return "green"
except:
pass
return None
def deploy_new_version(self, image_tag):
"""Deploy new version to inactive environment."""
# Determine which environment is inactive
current = self.get_current_active()
target = "green" if current == "blue" else "blue"
print(f"Deploying new version to {target} environment...")
# Update deployment image
deploy_name = f"{self.app_name}-{target}"
subprocess.run([
"kubectl", "set", "image", f"deployment/{deploy_name}",
f"{self.app_name}={image_tag}", "-n", self.namespace
], check=True)
# Wait for rollout to complete
subprocess.run([
"kubectl", "rollout", "status", f"deployment/{deploy_name}",
"-n", self.namespace, "--timeout=300s"
], check=True)
return target
def test_environment(self, environment):
"""Test the deployed environment."""
# Get service endpoint
result = subprocess.run([
"kubectl", "get", "svc", f"{self.app_name}-{environment}",
"-n", self.namespace, "-o", "jsonpath={.status.loadBalancer.ingress[0].ip}"
], capture_output=True, text=True)
endpoint = result.stdout.strip()
if not endpoint:
# Try cluster IP with port-forward
endpoint = "localhost:8000"
subprocess.Popen([
"kubectl", "port-forward", f"svc/{self.app_name}-{environment}",
"8000:80", "-n", self.namespace
])
time.sleep(5)
# Run health checks
try:
response = requests.get(f"http://{endpoint}/health", timeout=10)
if response.status_code == 200:
print(f"✅ {environment} environment health check passed")
# Run smoke tests
response = requests.get(f"http://{endpoint}/api/version")
if response.status_code == 200:
print(f"✅ {environment} smoke tests passed")
return True
except Exception as e:
print(f"❌ {environment} tests failed: {e}")
return False
def switch_traffic(self, to_environment):
"""Switch traffic to specified environment."""
# Update main service selector
subprocess.run([
"kubectl", "patch", "svc", self.app_name,
"-n", self.namespace, "-p",
f'{{"spec":{{"selector":{{"version":"{to_environment}"}}}}}}'
], check=True)
print(f"✅ Switched traffic to {to_environment} environment")
self.active_service = to_environment
def rollback(self):
"""Rollback to previous version."""
current = self.active_service
previous = "blue" if current == "green" else "green"
print(f"Rolling back to {previous}...")
self.switch_traffic(previous)
def deploy(self, image_tag, auto_switch=True):
"""Execute complete blue-green deployment."""
try:
# Deploy new version
target = self.deploy_new_version(image_tag)
# Test new environment
if not self.test_environment(target):
raise Exception(f"Tests failed for {target} environment")
if auto_switch:
# Switch traffic
self.switch_traffic(target)
# Keep old environment running for rollback
print("Deployment successful! Old environment preserved for rollback.")
except Exception as e:
print(f"Deployment failed: {e}")
if self.active_service:
print("Traffic remains on stable environment")
sys.exit(1)
# Usage
if __name__ == "__main__":
deployer = BlueGreenDeployment("myapp", "production")
deployer.deploy("myapp:v2.0.0")Canary Deployment:
# deploy/canary.py
import yaml
import subprocess
import time
import requests
class CanaryDeployment:
"""Canary deployment with gradual traffic shift."""
def __init__(self, app_name, namespace, canary_percentages=None):
self.app_name = app_name
self.namespace = namespace
self.canary_percentages = canary_percentages or [5, 10, 25, 50, 100]
def create_canary(self, image_tag):
"""Create canary deployment."""
# Create canary deployment
canary_deploy = {
"apiVersion": "apps/v1",
"kind": "Deployment",
"metadata": {
"name": f"{self.app_name}-canary",
"namespace": self.namespace
},
"spec": {
"replicas": 1,
"selector": {
"matchLabels": {
"app": self.app_name,
"version": "canary"
}
},
"template": {
"metadata": {
"labels": {
"app": self.app_name,
"version": "canary"
}
},
"spec": {
"containers": [{
"name": self.app_name,
"image": image_tag,
"ports": [{"containerPort": 8000}]
}]
}
}
}
}
# Apply canary deployment
with open("/tmp/canary.yaml", "w") as f:
yaml.dump(canary_deploy, f)
subprocess.run([
"kubectl", "apply", "-f", "/tmp/canary.yaml"
], check=True)
# Wait for canary to be ready
subprocess.run([
"kubectl", "rollout", "status", f"deployment/{self.app_name}-canary",
"-n", self.namespace, "--timeout=300s"
], check=True)
def update_service(self, percentage):
"""Update service to route percentage of traffic to canary."""
# Using Istio VirtualService for traffic splitting
vs = {
"apiVersion": "networking.istio.io/v1beta1",
"kind": "VirtualService",
"metadata": {
"name": self.app_name,
"namespace": self.namespace
},
"spec": {
"hosts": [self.app_name],
"http": [{
"route": [
{
"destination": {
"host": self.app_name,
"subset": "stable"
},
"weight": 100 - percentage
},
{
"destination": {
"host": self.app_name,
"subset": "canary"
},
"weight": percentage
}
]
}]
}
}
with open("/tmp/vs.yaml", "w") as f:
yaml.dump(vs, f)
subprocess.run([
"kubectl", "apply", "-f", "/tmp/vs.yaml"
], check=True)
print(f"Routing {percentage}% traffic to canary")
def monitor_canary(self, duration=300):
"""Monitor canary for errors during traffic shift."""
start_time = time.time()
error_threshold = 0.01 # 1% error rate
# Get canary pod
result = subprocess.run([
"kubectl", "get", "pods", "-l", "version=canary",
"-n", self.namespace, "-o", "name"
], capture_output=True, text=True)
canary_pod = result.stdout.strip()
while time.time() - start_time < duration:
# Check canary logs for errors
result = subprocess.run([
"kubectl", "logs", canary_pod, "--tail=100",
"-n", self.namespace
], capture_output=True, text=True)
logs = result.stdout
error_count = logs.count("ERROR") + logs.count("Exception")
total_lines = len(logs.split('\n'))
if total_lines > 0:
error_rate = error_count / total_lines
if error_rate > error_threshold:
print(f"⚠️ High error rate detected: {error_rate:.2%}")
return False
# Check metrics via API
try:
response = requests.get(
f"http://{self.app_name}/metrics",
timeout=5,
headers={"Host": self.app_name}
)
if response.status_code != 200:
print(f"⚠️ Metrics endpoint returned {response.status_code}")
except:
pass
time.sleep(30)
return True
def promote_canary(self):
"""Promote canary to stable."""
# Scale down canary
subprocess.run([
"kubectl", "scale", f"deployment/{self.app_name}-canary",
"--replicas=0", "-n", self.namespace
], check=True)
# Update stable deployment with new version
subprocess.run([
"kubectl", "set", "image", f"deployment/{self.app_name}",
f"{self.app_name}={self.image_tag}", "-n", self.namespace
], check=True)
# Wait for rollout
subprocess.run([
"kubectl", "rollout", "status", f"deployment/{self.app_name}",
"-n", self.namespace, "--timeout=300s"
], check=True)
# Reset traffic to 100% stable
self.update_service(0)
# Delete canary resources
subprocess.run([
"kubectl", "delete", f"deployment/{self.app_name}-canary",
"-n", self.namespace
], check=True)
print("✅ Canary promoted to stable")
def rollback(self):
"""Rollback canary deployment."""
# Delete canary
subprocess.run([
"kubectl", "delete", f"deployment/{self.app_name}-canary",
"-n", self.namespace, "--ignore-not-found"
], check=True)
# Reset traffic to 100% stable
self.update_service(0)
print("↩️ Canary rolled back")
def deploy(self, image_tag):
"""Execute complete canary deployment."""
self.image_tag = image_tag
try:
# Create canary
print("Creating canary deployment...")
self.create_canary(image_tag)
# Gradual traffic shift
for percentage in self.canary_percentages:
print(f"\n--- Shifting {percentage}% traffic ---")
# Update traffic split
self.update_service(percentage)
# Monitor for issues
if not self.monitor_canary(duration=120):
print("❌ Issues detected, rolling back...")
self.rollback()
return False
print(f"✅ {percentage}% traffic stable")
# Promote canary
self.promote_canary()
return True
except Exception as e:
print(f"❌ Deployment failed: {e}")
self.rollback()
return False
# Usage
if __name__ == "__main__":
deployer = CanaryDeployment("myapp", "production")
deployer.deploy("myapp:v2.0.0")