Skip to content

Instantly share code, notes, and snippets.

View jdnichollsc's full-sized avatar
🏠
Working from home

J.D Nicholls jdnichollsc

🏠
Working from home
View GitHub Profile
@jdnichollsc
jdnichollsc / docker-compose.yml
Created February 15, 2025 07:47
Enable Redis Streams, TimeSeries and Cluster mode with Docker
version: '3.6'
# Define name templates
x-name-templates:
project-name: &project-name ${COMPOSE_PROJECT_NAME:-projectx}
service-names:
redis-node-0: &name-redis-0 ${COMPOSE_PROJECT_NAME:-projectx}-redis-node-0
redis-node-1: &name-redis-1 ${COMPOSE_PROJECT_NAME:-projectx}-redis-node-1
redis-node-2: &name-redis-2 ${COMPOSE_PROJECT_NAME:-projectx}-redis-node-2
redis-node-3: &name-redis-3 ${COMPOSE_PROJECT_NAME:-projectx}-redis-node-3
@jdnichollsc
jdnichollsc / Dockerfile.postgres
Last active February 12, 2025 15:20
Async Data ingestion with Hugging Face datasets
# Use official Postgres image with multi-arch support
FROM postgres:17-bullseye
# Install required packages and build pgvector
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
# SSL certificates
ca-certificates \
# PostGIS
postgis \
@jdnichollsc
jdnichollsc / prompt_1.md
Created February 12, 2025 04:13
AI prompts for documentation

Analyze the entire codebase of this software project, including package.json scripts, dependencies, existing READMEs, and the current folder structure.

Then, generate a structured and developer-friendly documentation set within a /docs folder at the root of the repository. Ensure modularity by splitting documentation into separate markdown files, including:

  1. /docs/README.md - High-level project overview, purpose, and quick start guide.
  2. /docs/architecture.md - System architecture, service interactions, and outline high level designs with all important components using diagrams in Mermaid format with Material Design colors and high-contrast text.
  3. /docs/folder-structure.md - Explanation of the project directory structure and file purposes.
  4. /docs/development-guide.md - Setup instructions, key dependencies, and important scripts.
  5. /docs/operations.md - Deployment guide, CI/CD details, and configuration options.
  6. /docs/api-reference.md - API endpoints, request/response stru
@jdnichollsc
jdnichollsc / docker-compose.yml
Created February 11, 2025 18:48
Redis cluster mode enabled with Docker compose for local development
version: '3.6'
# Define name templates
x-name-templates:
project-name: &project-name ${COMPOSE_PROJECT_NAME:-solana}
service-names:
redis-node-0: &name-redis-0 ${COMPOSE_PROJECT_NAME:-solana}-redis-node-0
redis-node-1: &name-redis-1 ${COMPOSE_PROJECT_NAME:-solana}-redis-node-1
redis-node-2: &name-redis-2 ${COMPOSE_PROJECT_NAME:-solana}-redis-node-2
redis-node-3: &name-redis-3 ${COMPOSE_PROJECT_NAME:-solana}-redis-node-3
@jdnichollsc
jdnichollsc / Dockerfile.postgres
Last active February 12, 2025 15:21
Data ingestion with Hugging Face datasets
# Use official Postgres image with multi-arch support
FROM postgres:17-bullseye
# Install required packages and build pgvector
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
# SSL certificates
ca-certificates \
# PostGIS
postgis \
@jdnichollsc
jdnichollsc / data_ingest.py
Last active February 10, 2025 21:51
Data ingestion using Ray actors with Hugging Face datasets
"""
Fast image ingestion tool for Wolt Food dataset.
This script efficiently processes the Wolt Food CLIP-ViT-B-32 dataset,
uploading images to S3 and storing metadata and embeddings in PostgreSQL.
Optimized for processing 100k+ records using:
- Parallel image downloads and S3 uploads with connection pooling
- PostgreSQL COPY for bulk inserts
- Ray actors for efficient resource management
"""
@jdnichollsc
jdnichollsc / RandomNumber.sol
Last active March 18, 2024 15:16
Create a Random Number with Chainlink
// SPDX-License-Identifier: MIT
pragma solidity 0.8.21;
import {VRFCoordinatorV2Interface} from "@chainlink/contracts/src/v0.8/interfaces/VRFCoordinatorV2Interface.sol";
import {VRFConsumerBaseV2} from "@chainlink/contracts/src/v0.8/vrf/VRFConsumerBaseV2.sol";
import {ConfirmedOwner} from "@chainlink/contracts/src/v0.8/shared/access/ConfirmedOwner.sol";
// Fuji Testnet: 0xb9477bfbCAe729643b8251404Fa6E4356bB3d9A3
contract RandomNumber is VRFConsumerBaseV2 {
VRFCoordinatorV2Interface COORDINATOR;
@jdnichollsc
jdnichollsc / useMarketplace.ts
Created July 1, 2023 00:50
Custom hook for infinite scrolling using React Query
import { useInfiniteQuery } from '@tanstack/react-query';
import { MarketplaceItem } from '../models';
import { getMarketplaceItems } from '../services';
export type UseMarketplaceItemsProps = {
query: string;
initialData?: MarketplaceItem[];
from?: number;
size?: number;
@jdnichollsc
jdnichollsc / alternative_for_own_packages.js
Last active June 23, 2023 02:13
Fix RemixJS issues; "process is not defined" & "Buffer is not defined" - discussion related https://github.com/remix-run/remix/discussions/4906
// This is optional, only for building your own npm packages using those polyfills
// This is not useful to fix issues from external packages like web3 dependencies
import { Buffer } from "buffer"
import * as process from "process"
globalThis.Buffer = Buffer as unknown as BufferConstructor;
globalThis.process = process as unknown as NodeJS.Process;
// So only use this alternative if you don't want to use the below patch fix for RemixJS!
@jdnichollsc
jdnichollsc / user-signature.dto.ts
Created August 29, 2022 21:29
Verify that a request is signed by the owner of the public key (NestJS, ExpressJS and Solana)
import { ApiProperty } from '@nestjs/swagger';
export class UserSignatureDto {
@ApiProperty({ description: 'Wallet base58-encoded public key' })
walletPK: string;
@ApiProperty({ description: 'base58-encoded signature of a unique message' })
signature: string;
}