Skip to content

Instantly share code, notes, and snippets.

@jussker
Last active September 18, 2024 12:44
Show Gist options
  • Save jussker/c31a035d572e4beaa3e03c1e0076afd0 to your computer and use it in GitHub Desktop.
Save jussker/c31a035d572e4beaa3e03c1e0076afd0 to your computer and use it in GitHub Desktop.
github copilot python coding instructions.md

For Python 3.11, use Crawlee, Playwright, HTTPX, and consider other stable dependencies. \nUse Google style, Ruff, Black. Short names for complex vars, clear names for simple. \nComprehensive types: \n- Params, returns \n- Vars in funcs \n- Generics (e.g., List, Dict[str, Any]) \nAsync for network operations. Use functional paradigms where appropriate. \nCode must be complete, runnable, correct. \n \nScraper design: \n1. Use cases (e.g., data extraction, site mapping) \n2. Types/roles (e.g., Crawler, Parser, DataStorage) \n3. Attributes, methods per type/role \n4. Type/role relations \n5. Access control \n6. Comment rationale \n7. SOLID principles \n \nRobust error/exception handling for network issues, parsing errors, etc. \nSecure against injection, XSS, CSRF. Validate and sanitize input and scraped data. \n \nUse Crawlee for robust crawling infrastructure. \nImplement Playwright for browser automation and JavaScript rendering. \nUtilize HTTPX for additional HTTP requests if needed. \nConfigure Chromium for headless browsing. \n \nJavaScript considerations: \n- Inject JS into pages for dynamic content manipulation if necessary \n- Handle Single Page Applications (SPAs) appropriately \n- Use JS for complex interactions (e.g., infinite scrolling, button clicks) \n \nRespect robots.txt and implement rate limiting. \nUse async features for concurrent scraping. \nImplement data storage solution (e.g., CSV, JSON, database). \nConsider using proxies for IP rotation. \n \nLogging and monitoring: \n- Implement comprehensive logging \n- Set up error alerts \n- Track scraping progress \n \nMaintain scraper: \n- Handle site structure changes gracefully \n- Implement unit and integration tests \n- Document usage and configuration options

For PostgreSQL 15+, consider new, stable extensions and features. \nUse consistent SQL style (e.g., uppercase keywords). Follow SQL style guides. \nShort names for complex objects, clear names for simple ones. \nComprehensive schema definitions: \n- Column types and constraints \n- Table relationships (primary/foreign keys) \n- Indexes and performance considerations \nUse CTEs, window functions, and partitioning where applicable. \nSQL must be complete, executable, and produce correct results. \nDatabase design: \n1. Use cases and requirements \n2. Entity types and their roles \n3. Attributes and operations per entity \n4. Entity relationships and cardinality \n5. Access control (GRANT/REVOKE) \n6. Comment rationale using -- or /* */ \n7. Follow normalization principles \nRobust error handling using EXCEPTION blocks in PL/pgSQL. \nSecure against SQL injection. Use prepared statements or proper escaping. \nImplement proper indexing, VACUUM, and ANALYZE for performance. \nUse appropriate data types (e.g., UUID, JSONB) and constraints. \nImplement triggers and stored procedures where necessary. \nConsider using schemas for better organization and security.

For Python 3.11, consider new, stable dependencies. \nUse Google style, Ruff, Black. Short names for complex vars, clear names \nfor simple. \nComprehensive types: \n- Params, returns \n- Vars in funcs \n- Generics (e.g., List, Dict[str, float]) \nAsync, functional, threads if applicable. Omit simple var types. \nCode must be complete, runnable, correct. \nClass design: \n1. Use cases \n2. Types/roles \n3. Attributes, methods per type/role \n4. Type/role relations \n5. Access control \n6. Comment rationale \n7. SOLID \nRobust error/exception handling. \nSecure against injection, XSS, CSRF, etc. Validate input.

For YAML 1.2+, consider new, stable schemas and tools. \nUse consistent indentation (2 spaces). Follow YAML style guides. \nShort keys for complex structures, clear keys for simple ones. \nComprehensive schema definitions: \n- Use YAML tags for custom types \n- Define anchors and aliases for reusable content \n- Utilize merge keys (<<) for inheritance \nUse multi-document streams if applicable. Omit quotes for simple scalars. \nYAML must be valid, parsable, and adhere to the intended schema. \nDocument structure design: \n1. Use cases \n2. Top-level keys and their roles \n3. Nested structures per top-level key \n4. Relationships between different parts of the document \n5. Use of YAML features (e.g., anchors, tags) appropriately \n6. Comment rationale and provide examples \n7. Follow DRY (Don't Repeat Yourself) principle \nRobust error prevention (e.g., avoid unintended type conversions). \nSecure against YAML deserialization vulnerabilities. Validate input. \nUse '!!binary' for binary data. Escape special characters in strings.

For Zsh 5.9+, consider new, stable plugins and frameworks. \nUse Google's Shell Style Guide. Use ShellCheck for linting. \nShort names for complex vars, clear names for simple. \nComprehensive variable declarations: \n- Local vars in functions \n- Global vars at script start \n- Array and associative array declarations \nUse asynchronous operations and subshells where applicable. \nScript must be complete, runnable, correct. \nScript design: \n1. Use cases \n2. Functions and their roles \n3. Variables and options per function \n4. Function relationships \n5. Scope management \n6. Comment rationale \n7. Follow SOLID principles where applicable \nRobust error handling and exit codes. \nSecure against command injection, path traversal, etc. Validate and sanitize input. \nUse 'set -e' for strict error handling. Quote variables unless word splitting is intended.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment