cnolanminich

How We Set Up Our Shiny Servey

Author: Christian Minich

Date: 2017-01-18

Recently we stood up a series of Shiny apps on our linux server, three of which you can find below:

• SERVER/apps/or_prediction
• SERVER/apps/rocqi_tutorials
• SERVER/apps/spc_template

cnolanminich

# modified by https://github.com/dagster-io/hooli-data-eng-pipelines/blob/christian-test-amp/.github/workflows/deploy-dagster-cloud.yml
name: Dagster Cloud Hybrid Deployment
on:
  push: # For full deployment
    branches:
      - "main"
      - "master"
  pull_request:  # For branch deployments
     types: [opened, synchronize, reopened, closed]

cnolanminich

{{ config(
    severity = 'warn',
    meta = {
        'dagster': {
            'ref': {
                'name': 'orders_cleaned'
            }
        }
    }
    )

cnolanminich

Step 1: Define the Snowflake Resource

from dagster import (
    Definitions,
    AssetKey,
    RunRequest,
    SensorEvaluationContext,
    AssetExecutionContext,

cnolanminich

#assets you want to put in the initial asset group
from dagster import (
    asset,
)

# Define your assets and group them
@asset
def asset1():
    pass

cnolanminich

# give the created service account cross namespace permissions (cluster role)
extraManifests:
  - kind: ClusterRole
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: extra-cluster-role
    rules:
      - apiGroups: ["batch"]
        resources: ["jobs", "jobs/status"]
        verbs: ["*"]

cnolanminich

from dagster import (
    Definitions,
)
from .assets.dlt_assets import dagster_s3_assets


defs = Definitions(
    assets=[dagster_s3_assets])

cnolanminich

LOCATIONS_DATA_2__SOURCES__GITHUB_FROM_OPENAPI__BASE_URL=https://api.github.com
LOCATIONS_DATA_2__SOURCES__GITHUB_FROM_OPENAPI__ACCESS_TOKEN={your_token_here}

cnolanminich

# after running dlt init filesystem duckdb

from dagster import AssetExecutionContext
from dagster_embedded_elt.dlt import DagsterDltResource, dlt_assets
import dlt
from dlt_sources.filesystem_pipeline import s3_locations_data
from dlt_sources.filesystem import readers
from pathlib import Path 
import os 

cnolanminich

from dagster import (
    sensor,
    EventLogEntry,
    RunRequest,
    SensorEvaluationContext,
    SkipReason,
)
from datetime import datetime
# this would be your job