cnolanminich

Asset Granularity Demo

This demo project showcases three different approaches to modeling ETL operations as Dagster assets, each with different trade-offs for visibility, control, and simplicity.

Quick Start

# Activate virtual environment
source .venv/bin/activate

cnolanminich

The Two-Step Approach to Dynamic Dagster+ Deployments

Creating a new persistent deployment requires coordinating two systems: the Dagster+ API and your Kubernetes agent configuration.

1. Dagster+ API (Create the Deployment Resource)

The deployment must exist in Dagster+ as a logical entity. This is done via the GraphQL API:

import httpx

cnolanminich

Per-Asset Concurrency with Dagster Pools

This document explains how to limit concurrent executions of individual assets using Dagster's pool-based concurrency system.

Overview

By default, Dagster allows unlimited concurrent executions of the same asset across multiple runs. This can cause issues when:

• An asset writes to a shared resource (database table, file, API)
• An asset has rate limits or resource constraints
• You want to prevent duplicate work

cnolanminich

#!/usr/bin/env python3
# /// script
# requires-python = ">=3.10"
# dependencies = [
#     "requests",
# ]
# ///
"""
Query Dagster's GraphQL API to get monthly credits (ops executed + asset materializations).

cnolanminich

Sigma Data Models Implementation Plan

Executive Summary

This document outlines a detailed plan to add support for Sigma's new Data Models API to the dagster-sigma library. Data Models are replacing the deprecated datasets (end-of-life: June 2, 2026) and provide enhanced semantic layer capabilities including metrics, relationships, column/row-level security, and more.

Timeline: ~5-8 days development + testing Priority: Medium-High (18 month deprecation timeline for datasets) Complexity: Moderate

cnolanminich

mutation {
  createSecret(
    secretName: "MY_API_KEY"
    secretValue: "abc123"
    scopes: { fullDeploymentScope: true }
    locationNames: ["prod"]
  ) {
    __typename
    ... on Error {
      message

cnolanminich

Dagster Demo Generator - GitHub Copilot Guide

This document provides comprehensive instructions for GitHub Copilot to replicate the Dagster demo generation workflow implemented in this repository.

Architecture Overview

The Dagster Demo Generator automates the creation of demonstration Dagster projects using a multi-tiered workflow:

1. Command Entry Point: /create-dagster-demo slash command (.claude/commands/create-dagster-demo-2.md)
2. Skills System: Three specialized skills that handle different aspects of Dagster project creation

cnolanminich

from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.utils.dates import days_ago
from airflow.settings import Session
from airflow.models import DagRun
from datetime import timedelta
import pandas as pd
import numpy as np

# Constants for flexibility

cnolanminich

pipeline {
    agent any
    
    environment {
        // Dagster Cloud Configuration
        DAGSTER_CLOUD_ORGANIZATION = 'your-org-name'  // Replace with your Dagster+ org name
        DAGSTER_CLOUD_API_TOKEN = credentials('dagster-cloud-api-token')  // Jenkins credential ID
        DAGSTER_BUILD_STATEDIR = "${WORKSPACE}/build_state"
        
        // AWS ECR Configuration

cnolanminich

with daily as(
select 
    ds::date as reporting_date,
    sum(standard_credits) as daily_standard_credits,
    sum(high_velocity_credits) as daily_high_velo_credits,
    daily_standard_credits + daily_high_velo_credits as total_credits_daily
from purina.product.usage_metrics_daily 
where usage_metrics_daily.organization_id = {org_id here} 
group by ds::date
)