Last active
August 27, 2026 07:43
-
-
Save lassebenni/3e3dfbf9bc857dc16e1a517c8d012198 to your computer and use it in GitHub Desktop.
Visual: Week 15 ingest Mode 1 vs Mode 2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| <!DOCTYPE html> | |
| <html lang="en"> | |
| <head> | |
| <meta charset="UTF-8"> | |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> | |
| <title>Visual: Ingest Mode 1 vs Mode 2</title> | |
| <style> | |
| body { | |
| background-color: #1a1a1a; | |
| color: #f0f0f0; | |
| font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif; | |
| margin: 0; | |
| padding: 20px; | |
| display: flex; | |
| justify-content: center; | |
| align-items: center; | |
| min-height: 100vh; | |
| } | |
| svg { | |
| background-color: #252526; | |
| border-radius: 8px; | |
| box-shadow: 0 4px 12px rgba(0,0,0,0.3); | |
| width: 100%; | |
| max-width: 720px; | |
| } | |
| </style> | |
| </head> | |
| <body> | |
| <svg viewBox="0 0 720 480" xmlns="http://www.w3.org/2000/svg" role="img" aria-label="Mode 1 local ingest vs Mode 2 ACA ingest"> | |
| <defs> | |
| <marker id="ah" markerWidth="10" markerHeight="7" refX="9" refY="3.5" orient="auto"> | |
| <polygon points="0 0, 10 3.5, 0 7" fill="#9da7b3" /> | |
| </marker> | |
| <style type="text/css"><![CDATA[ | |
| .title { fill: #f0f0f0; font-size: 14px; font-weight: 700; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif; } | |
| .subtitle { fill: #8b949e; font-size: 11px; font-weight: 600; letter-spacing: 0.04em; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif; } | |
| .box { stroke-width: 2; rx: 6; } | |
| .airflow { fill: rgba(210,168,255,0.12); stroke: #d2a8ff; } | |
| .aca { fill: rgba(255,166,87,0.12); stroke: #ffa657; } | |
| .dbx { fill: rgba(88,166,255,0.12); stroke: #58a6ff; } | |
| .pg { fill: rgba(63,185,80,0.12); stroke: #3fb950; } | |
| .shared { fill: rgba(139,148,158,0.08); stroke: #484f58; stroke-dasharray: 5 4; } | |
| .task { fill: #c9d1d9; font-size: 12px; font-weight: 600; font-family: ui-monospace, SFMono-Regular, Menlo, monospace; } | |
| .task-soft { fill: #8b949e; font-size: 11px; font-family: ui-monospace, SFMono-Regular, Menlo, monospace; } | |
| .note { fill: #a0a0a0; font-size: 11px; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif; } | |
| .caption { fill: #8b949e; font-size: 11px; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif; } | |
| .arrow { stroke: #9da7b3; stroke-width: 2; fill: none; } | |
| .divider { stroke: #3d444d; stroke-width: 1; } | |
| .legend { fill: #8b949e; font-size: 10px; font-weight: 600; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif; } | |
| ]]></style> | |
| </defs> | |
| <!-- Column titles --> | |
| <text class="title" x="180" y="28" text-anchor="middle">Mode 1 · INGEST_MODE=local</text> | |
| <text class="caption" x="180" y="46" text-anchor="middle">fast local loop — try this first</text> | |
| <text class="title" x="540" y="28" text-anchor="middle">Mode 2 · INGEST_MODE=aca</text> | |
| <text class="caption" x="540" y="46" text-anchor="middle">prod shape — after image is in ACR</text> | |
| <line class="divider" x1="360" y1="16" x2="360" y2="300" /> | |
| <!-- ========== Mode 1 ========== --> | |
| <rect class="box airflow" x="40" y="64" width="280" height="220" /> | |
| <text class="subtitle" x="56" y="86">ASTRO WORKER</text> | |
| <rect class="box airflow" x="60" y="100" width="240" height="36" /> | |
| <text class="task" x="180" y="122" text-anchor="middle">1 · ingest (local Python)</text> | |
| <path class="arrow" d="M 180 136 L 180 150" marker-end="url(#ah)" /> | |
| <rect class="box airflow" x="60" y="152" width="240" height="36" /> | |
| <text class="task" x="180" y="174" text-anchor="middle">2 · dbt_build (subprocess)</text> | |
| <path class="arrow" d="M 180 188 L 180 202" marker-end="url(#ah)" /> | |
| <rect class="box airflow" x="60" y="204" width="240" height="36" /> | |
| <text class="task" x="180" y="226" text-anchor="middle">3 · publish_to_backend</text> | |
| <text class="note" x="180" y="264" text-anchor="middle">all three tasks in one container</text> | |
| <!-- ========== Mode 2 ========== --> | |
| <rect class="box airflow" x="384" y="64" width="140" height="88" /> | |
| <text class="subtitle" x="400" y="84">ASTRO</text> | |
| <text class="task" x="454" y="112" text-anchor="middle">1 · ingest</text> | |
| <text class="task-soft" x="454" y="132" text-anchor="middle">(triggers job)</text> | |
| <path class="arrow" d="M 524 108 L 548 108" marker-end="url(#ah)" /> | |
| <rect class="box aca" x="552" y="64" width="140" height="88" /> | |
| <text class="subtitle" x="568" y="84">ACA JOB</text> | |
| <text class="task" x="622" y="112" text-anchor="middle">pipeline.py</text> | |
| <text class="task-soft" x="622" y="132" text-anchor="middle">runs ingest</text> | |
| <path class="arrow" d="M 454 152 L 454 174" marker-end="url(#ah)" /> | |
| <text class="caption" x="470" y="168">then</text> | |
| <rect class="box airflow" x="384" y="176" width="308" height="108" /> | |
| <text class="subtitle" x="400" y="196">ASTRO WORKER</text> | |
| <text class="task" x="538" y="224" text-anchor="middle">2 · dbt_build (subprocess)</text> | |
| <text class="task" x="538" y="248" text-anchor="middle">3 · publish_to_backend</text> | |
| <text class="note" x="538" y="268" text-anchor="middle">same worker as Mode 1</text> | |
| <!-- ========== Shared downstream ========== --> | |
| <rect class="box shared" x="40" y="316" width="640" height="100" /> | |
| <text class="subtitle" x="56" y="338">BOTH MODES — tasks 2 and 3</text> | |
| <rect class="box dbx" x="64" y="350" width="260" height="48" /> | |
| <text class="task" x="194" y="370" text-anchor="middle">Databricks SQL warehouse</text> | |
| <text class="caption" x="194" y="386" text-anchor="middle">dbt executes SQL here</text> | |
| <path class="arrow" d="M 324 374 L 388 374" marker-end="url(#ah)" /> | |
| <rect class="box pg" x="392" y="350" width="260" height="48" /> | |
| <text class="task" x="522" y="370" text-anchor="middle">Postgres analytics_dev</text> | |
| <text class="caption" x="522" y="386" text-anchor="middle">publish writes here</text> | |
| <!-- Legend + footer --> | |
| <rect class="box airflow" x="40" y="436" width="12" height="12" /> | |
| <text class="legend" x="58" y="446">Astro / worker</text> | |
| <rect class="box aca" x="148" y="436" width="12" height="12" /> | |
| <text class="legend" x="166" y="446">ACA job</text> | |
| <text class="note" x="520" y="446" text-anchor="middle">Prod schedule always uses Mode 2 (ACA).</text> | |
| </svg> | |
| </body> | |
| </html> |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment