CRITICAL: run ALL tasks listed here! (yes you can complete all in one loop interval)! And consider them all fresh tasks, you can remove your memory of each task from prior runs. Critical - even if these tasks haven't changed since last time, you must re-run them, all of them in order!
ABSOLUTELY CRITICAL - do NOT skip tasks - do NOT do them out of order!!!!!!
notes:
- when type-checking just do spot checks with --isolatedModules -p tsconfig.json; and skip eslint; the -p tsconfig.json is important because of path aliases in tsconfig.json ;
- prefer src/server/subroutines to src/server/actions; and src/server/actions/cache should probably be in src/server/cache;
please do all browser automation HEADLESS!!;
CRITICAL: we must be scraping for 25 emails and 25 domains for each scraping run - whether it's for benefactor leads, dd leads, dd coach candidates, u/bizdev corporate contracts/leads, or u/alex job searches; it's not enough to get http < 400 status codes..we must be getting 25+ emails and 25+ domains/urls for each run to consider it a successful run!!! the run needs to do more hops and last longer to get to 25+ etc...!!
the most important task below is to improve the email/scraping for email marketing for u/alex, u/recruiter, u/leads, s/leads and u/benefactor roles; we do not need to send more than 10 emails per day to those leads, but we should be collecting at least 500 leads/candidates per day for each of those roles!! we need to improve all the scraping to the point where we get 500 new leads per day.
Important: in general follow the instructions exactly, but without skipping tasks, make best judgements whether to run things against prod at https://dancingdragons.cc or localhost via pnpm run dev server, etc;
Very important: linkedin creds are [email protected] and password allow me the human user to input it in chrome window, the email for linkedin is not [email protected] etc.
Critical: here are the tasks, run in order of appearance, skipping none of them:
first task:
if git index is dirty, add all and commit; you MUST resolve all git conflicts even if they are from old commits (or commits not from you). Do a grep for <<<<< or >>>>> and resolve them!! resolve according to docs/critical-merging-instructions.md
task:
Do a grep for <<<<< or >>>>> and resolve git conflicts!! resolve according to docs/critical-merging-instructions.md
task:
we need to improve our u/leads pages and also u/coach/leads pages so users can get a lock on a contact for 3 weeks after they first send an email, if the client signs up the coach with the lock wins that client;
so we need
u/leads/locks
u/coach/leads/locks
and we need a db ledger table or something to handle the locks? or maybe not a ledger just a db transaction and atomic update for the contact/leadId;
we also need a way for coaches to pick from outreach templates and send the email initially from mail.dancingdragons.cc and to have a ui where the coach emails the client and thread id from gmail is synced with our ui in u/coach/leads/locks etc;
please create some outreach templates for our different coaching disciplines/areas like leadership coaching, executive coaching, life coaching, etc.
we will need a cron job in .github/workflows that can sync emails with the neondb db/ui, and also a webhook for gmail that can sync our neondb - we will need a new neondb table for holding/mirroring email threads;
task:
for our coach recruitment outreach email (we produce email contacts via scraping and then consume those contacts to recruit them to our platform via cron jobs etc), what does the body say? it should be like:
"Our coaching platform is recruiting coaches to work with corporate and individual clients. we are hiring fractional internal leadership roles as well. see our careers page at https://dancingdragons.cc/careers
please use this to submit an application and book a 1-and-Done interview to join us! https://dancingdragons.cc/pub/submit-job-application
-Dancing Dragons Team "
but improve it a bit given the overall code context etc and put the template in the right location
task:
for every single page/component that uses images img tag or imports a component that uses image tag (loads images), we need to make sure we have a tests/playwright test and tests/puppeteer test for that page (1 of each) to make sure images load. Right now s3 and gcs are having issues, but we need to also try r2 and in some cases google drive. In the last month, we started work to prioritize r2, gcs and s3 in that order. We need to make sure our image loading is optimal. Please devise a function which can discover all pages with images and create dynamic tests in tests/puppeteer and tests/playwright for this.
task:
audit and harden all of our MDP (markov decision process) and Reinforcement Learning code - it should be non-blocking (use dd-proms.ts) and use cron jobs - but at some point it needs to make decisions in the software for assignments, scheduling, coach matching, coach recruiting, a/b testing outreach templates, improving email scraping strategies on-the-fly etc etc; please improve audit and harden our MDP, and evaluate where we may need POMDP solutions too. feel free to edit code where you are more than 98% confident.
task:
on pub/search/coach - we need to make improvements -
the search panel and searchbar needs to extend across viewport in top area above results - and after initiating search it clears the search in a way that's poor ux - improve the rxjs - we should have a search history panel that opens on the left or right that saves/displays previous search strings though - have it save the previous most recent 30 searches in local-storage or indexeddb and a new supabase search query string table, which stores search strings and pairs them with which searchbar in the ui was used for the search;
on pub/search/coaches (a different page than above) - the searchbar is better/wider already - but we need a sort on it - to sort coaches by bio length, skills declared, gender, age, etc etc;
task:
given any/unknown types like this:
let db: ReturnType; let dbro: ReturnType; let db: any; let dbro: any;
avoid using any/unknown types - follow docs/audit-and-harden.md and docs/critical-merging-instructions.md; for the last 10 commits, go back and harden and improve any/unknown types that you introduced.
task:
for all our email/domain scraping code, make sure we have techniques to navigate on multi-pages, like pagination, and directories for example we should able to get all emails here: https://www.pisd.edu/about-our-district/staff-directory
task:
truly optimize u/user/messaging - the best way to do this is write super deep tests in tests/playwright and tests/puppeteer and improve the u/user/messaging page to meet the test cases - there is a challenge because we need 3 users to participate in the tests - 3 different incognito windows with 3 different logged in users using the test bypass env var - we need to make sure creating conversations work - for both group conversations and 1v1 convos - group convos need to be both private where no new users can be added or flexible where new users can be added (and new users can see old messages). Improve rxjs where needed and avoid regressions. Make sure search works for conversations, and that messages and threads have embeddings in our message-embeddings table etc. And messages should have better titles - the uuid for the thread should be last resort, it should default to the particpants first names like Alex, Jon, Jeremy, +4-More; also ensure it's secure - we do not need to encrypt messages yet, but otherwise need to make sure nobody besides logged in user can read convos etc from the UI. writing the tests in tests/puppeteer and tests/playwright is crucial/critical.
also put a gear icon for each thread when opened in the selected pane - when gear clicked add a dropdown - one of the dropdown options should be "show stats" - which shows the prosocial and antisocial behavior in the thread - word count per person (pie chart showing a vs b), average time to reply (a vs b in histogram), number of words per message (a vs b histogram); another thing in the gear is notifications settings, user should be able to pick when to get notified and how to get notified (text message, email, whatsapp message, etc)
task:
for u/leads/crons and u/admin/crons and u/alex/crons and u/benefactor/crons let's see if there is something else besides error_tracking and cron_job_runs to get insights into success of cron runs???
Leads Cron Performance Route: /u/leads/crons. Database-backed cron telemetry from cron_job_runs, filtered to the dd-leads pipeline (lead generation, multi-source enrichment, sync-leads, scrape-emails-*). Daisy-chain hop breakdowns, correlated errors, and per-route success scoring — same surface as /u/admin/crons, scoped to your pipelines. Database-backed cron telemetry
Cron jobs, daisy-chain hops, and failure forensics This dashboard reads cron_job_runs for per-session telemetry with hop breakdowns, and cross-references error_tracking for correlated failures from the last 3 days.
task:
we have some entities in neondb/tables that need to be cached so browser doesn't have to re-fetch megabytes of data each page load.
we have skills table caching, industries table caching, tags caching, job-functions caching, categories caching, institutions, credentials, certifications, let's improve all of it so they can be cached:
- in redis (least necessary since we have neondb read replicas)
- in supabase so the client can fetch from supabase on connect - simply mirror all the above tables with neondb and sync between them - supabase doesn't need RLS on these mirror tables since these entities are public/shared
- in indexeddb for each user's browser
- on cloudflare by pre-fetching the match page, u/coach page, u/coach/dashboard, u/coach/profile/edit etc etc
let's focus on skills caching first so we avoid the big 2MB download each time from the pages, but consider all the other 'databases/neondb/tables/*.ts' entities too
Critical - skills and tags/categories are going to be more than 2MB each, so unstable_cache won't work for those etc - industries and job functions will always stay below 2MB though.
these should learn from each other - identify what is working and what isn't working and cross-pollinate the scraping across u/admin/leads, u/leads, s/leasd, u/benefactor, u/recruiter/leads and u/alex/leads etc -
Cron Performance Dashboard Route: /u/admin/crons. Database-backed cron telemetry from cron_job_runs. Includes daisy-chain hop breakdowns, duration analysis, and correlated errors from the last 3 days. Database-backed cron telemetry
Cron jobs, daisy-chain hops, and failure forensics This dashboard reads cron_job_runs for per-session telemetry with hop breakdowns, cross-references error_tracking, and surfaces auxiliary source counts for outreach, scrape outcomes, tool runs, and alert state.
task:
audit and harden u/user/_test/exp
there may be no issues at all - but make sure "breaker snapshot" is working?
🧪 ALS Context Store Test Verifies that all three HTTP wrappers correctly populate the AsyncLocalStorage context.
- Page Wrapper (SSR — wrapAsyncPage) page-wrap ✅ ALS Context Cookies Headers Env Overrides Breaker Snapshot Browser Session Request ID ddl-page-JROCo8ejaoFIQ8JIuMHwQ Store Keys preloadedHeaders, preloadedCookies, envOverrides, browserSessionId Timestamp 2026-05-12T14:40:15.718Z
- Server Action (generic-actions pattern) generic-actions ✅ ALS Context Cookies Headers Env Overrides Breaker Snapshot Browser Session Request ID ddl-als-test-generic-Bb2ml91E0YG-3BaF3GMqj Store Keys preloadedCookies, preloadedHeaders, envOverrides, browserSessionId Timestamp 2026-05-12T14:40:30.941Z
- API Route (wrapApiRoute) api-handler-wrapper ✅ ALS Context Cookies Headers Env Overrides Breaker Snapshot Browser Session Request ID ddl-als-test-api-JIrlZjJ5EL0Me8NwRYinN Store Keys preloadedCookies, preloadedHeaders, envOverrides, browserSessionId Timestamp 2026-05-12T14:41:05.871Z
- generic() Dispatcher (full round-trip) generic() → test-auth-context-action ✅ Round Trip 1137ms Timestamp 2026-05-12T14:40:27.367Z Raw response What this tests: ALS Context — requestContext.run() created an AsyncLocalStorage scope Cookies — setPreloadedCookies() populated the ALS store Headers — setPreloadedHeaders() populated the ALS store Env Overrides — populateEnvOverridesToALS() parsed x-ddl-env-overrides header Breaker Snapshot — populateBreakerSnapshotToALS() parsed x-ddl-breaker-snapshot header Browser Session — resolveBrowserSessionId() found a session ID in cookies/headers
task:
create/update a doc in docs/series-assignment-groups-meetings.md -
we need to document how coaches invoice company based off of assignments to clients - when assignment is first created, we should be creating a series record in series table - and when the cadence for meetings between coach and client changes, we update the series table to use a new cadence etc. the cadence/series helps us know when to bill clients in advance of meetings and to send reminders to coaches if they forget to submit an invoice.
reference these pages: https://dancingdragons.cc/u/coach/postmortem https://dancingdragons.cc/u/coach/client-cadence-update https://dancingdragons.cc/u/coach/forms/single-session-invoice https://dancingdragons.cc/u/coach/group-invoice
only update docs/series-assignment-groups-meetings.md, with the goal of eventually updating the above pages - we need to make sure we clearly document (with diagrams etc), the desired relationships between the tables (meetings, invoices, groups, group sessions, assignments, series, etc etc) and also design with forward thinking a clean system overall;
task:
create/update a doc in docs/intake-user-onboard-client.md
we need to document the tables involved with onboarding clients and assignment them to a coach. when coach-client assignment is first created, we should be creating a series record in series table. we need to send a contract for the client to sign. and we need to email the client a bill for $100 via stripe to reserve a coach. so our billing/banking tables are involved.
in docs/intake-user-onboard-client.md let's document carefully how our banking/ledger/billing tables handle the $100 charge via stripe and how we do assignments, series and schedule the first session between coach/client and how reminders will happen (our system should reflect the coaches calendar - ideally the coach tags our email on their calendar - but we should also track when first session happens in our own schedule/cal table)?
update docs/intake-user-onboard-client.md without changing code yet - add diagrams etc.
task:
this blog on stage and prod weblogs table - Building Trust as a Cornerstone of Leadership: Part 1 – The Foundations of Trust
it needs to say "By Mike Mercer" or better "By Dr. Mike Mercer, DEL, MSSL" not "By system dd-system-user" and his name should link to his coaching profile - update the UI to do that
also it needs to list mermaid as a plugin in the weblogs table, and declare any other reactmarkdown plugin that it needs to render better
xychart-beta and gantt are not rendering yet - need the right plugin declared/installed;
also, when updating blogs, previous versions of content should be stored for the user in archived-entities table etc - make sure that previous versions can be seen or recovered and use dbtx - db-with-transactions-3.ts to do this;
https://dancingdragons.cc/weblogs/building-trust-cornerstone-leadership-part-1
so we need to update the backend routine to override the author (if logged in user is an admin user)
task:
use docs/debug-prod-final.md and resolve 3-9 issues from last 10 hours including:
Failed to list pinned tags: Error: Failed query: select "tag_relations"."id", "tags"."id", "tags"."tag_name", "tags"."tag_color", "tags"."tag_description", "tag_relations"."pinned_at", "tag_relations"."created_at" from "tag_relations" inner join "tags" on "tag_relations"."tag_id" = "tags"."id" where ("tag_relations"."entity_name" = $1 and "tag_relations"."entity_id" = $2 and "tag_relations"."is_pinned" = $3 and "tag_relations"."is_soft_deleted" is null and "tags"."is_soft_deleted" is null and "tags"."is_active" = $4) order by "tag_relations"."pinned_at" desc, "tag_relations"."created_at" desc limit $5 params: users,963bd832-65ae-47bd-8b24-f899becdb78a,true,true,20
task:
this blog on stage and prod weblogs table - Building Trust as a Cornerstone of Leadership: Part 1 – The Foundations of Trust
it needs to say "By Mike Mercer" or better "By Dr. Mike Mercer, DEL, MSSL" not "By system dd-system-user" and his name should link to his coaching profile - update the UI to do that
also it needs to list mermaid as a plugin in the weblogs table, and declare any other reactmarkdown plugin that it needs to render better
xychart-beta and gantt are not rendering yet - need the right plugin declared/installed;
also, when updating blogs, previous versions of content should be stored for the user in archived-entities table etc - make sure that previous versions can be seen or recovered and use dbtx - db-with-transactions-3.ts to do this;
task:
please build out u/bizdev/leads and u/bizdev/crons - which is for our business leads (not individual consumer leads) - make sure our email scraping pipeline has this covered, we may need to create a new pipeline for this - the goal is to scan web for businesses that need coaches and to send out 5 emails to each company to make contact with a business dev offer for 20-100 coaches per quarter. Make sure this email scraping and cron for consuming the leads and emails is totally independent from u/leads and s/leads etc; we need ICPs for business customers not just ICPs for individual consumers - perhaps a separate neondb table..and we also need email templates to reach these business ICPs;
also improve u/bizdev and u/bizdev/home and u/bizdev/dashboard; and make sure the 'more...' menu works too for u/bizdev header in components/nav/bizdev-header.tsx;
make sure we have ICP table for bizdev not just individual customer profiles for u/leads and s/leads - a separate table for ICP for b2b is probably useful and a separate table for b2c;
task:
pages at pub/search/coach and pub/search/coaches are both supposed to exist and be fully-functional and full-featured, and be independent of each other - look in git history to dig up the missing one and restore it - the 2 pages should have entirely different client components and SSR components etc - but both should have an infinite scroll (paginated queries etc, using streaming search) and heavy/liberal use of rxjs..for the search we should lean heavily on embeddings/vector/tsvector search - not just "search by name" but search by embeddings - use new streaming api queries not existing ones (do NOT use the ones from the match page etc).
task:
investigate u/recruiter/crons and and u/recruiter/candidates - is our email scraping to find coaches to recruit working? audit and harden our coach recruiting email scraping to find coach candidates; make sure all these pages are present and fully-featured and fully-functional - audit and harden them u/recruiter/crons and u/recruiter/candidates
task:
make sure we our pre-loading/caching these pages with cloudflare: u/leads, u/benefactor, u/admin, u/owner, u/coach, u/user, u/recruiter, u/livestreamer, u/marketing, u/alex, u/intake, u/client, u/coach_manager in cloudflare - audit and harden our cloudflare and brightdata caching;
task:
-
create/improve u/alex/crons to be full-featured and fully-functional - our marketing team should be able to investigate our cron jobs and see what's working and what's not working;
-
create/improve u/marketing/crons to be full-featured and fully-functional - our marketing team should be able to investigate our cron jobs and see what's working and what's not working;
-
create/improve u/hr/crons to be full-featured and fully-functional - our hr team should be able to investigate our cron jobs and see what's working and what's not working;
-
create/improve u/ops/crons to be full-featured and fully-functional - our ops team should be able to investigate our cron jobs and see what's working and what's not working;
-
create/improve u/coach_manager/crons to be full-featured and fully-functional - our coach manager team should be able to investigate our cron jobs and see what's working and what's not working;
-
create/improve u/qa/crons to be full-featured and fully-functional - our qa team should be able to investigate our cron jobs and see what's working and what's not working;
-
create/improve u/intake/crons to be full-featured and fully-functional - our qa team should be able to investigate our cron jobs and see what's working and what's not working;
-
create/improve u/bizdev/crons to be full-featured and fully-functional - our qa team should be able to investigate our cron jobs and see what's working and what's not working;
task:
make sure u/alex and u/alex/crons is successfully scraping and applying for jobs
task:
improve our own mcp server(s) for dancingdragons.cc itself - audit and harden our mcp server(s)/apis etc
task:
run docs/debug-prod-final.md and resolve 0-5 issues from the last 90 minutes;
task:
audit and harden our k8s server for local (macos) and remote (ec2) dev. let's have 'remote/dev-server-local' use minikube on our local machine (install everything), and 'remote/dev-server' use real k8s for ec2 machine on aws; update this folders carefully; get remote/dev-server-local up to speed, get minikube to run and create some tests that start chats/threads with claude instructions via a simple ui etc.
task:
audit and harden our pages to make sure all oure mobile friendly/responsive.
this is CRITICAL and SENSITIVE - be careful - for every single private page in app (behind '(private)/u/' and '(private)/s/', make sure it's responsive and mobile-friendly - focus on private pages only
task (critical):
git fetch && git merge origin/lb1-alex # resolve any conflicts semantically resolve any conflicts semantically according to docs/critical-merging-instructions.md
task: git fetch && git merge origin/rw1-alex # resolve any conflicts semantically resolve any conflicts semantically according to docs/critical-merging-instructions.md
task, in general applies to all tasks:
if an env var is missing or api key is invalid (eg, http 401), then ddlog.warn that message "env var missing or api key invalid" with more relevant details about the service trying the reached or name of env var - in general for nodejs make sure you also try to read it from edge-config env-overrides too using getEnvSync not getEnvAsync
task:
this is CRITICAL and must be re-evaluated and run anew/fresh everytime!
audit and harden all our test files in tests/puppeteer, tests/playwright, and tests/general, run the browser automation strictly headless - I do not want browser windows/tabs popping up locally on my dev machine;
- pick 30 new tests randomly (random from each folder, but no more than 10 from any folder or it's subfolders), and improve them and make sure they are up to date with latest app code in src dir
- tests should do deep probing and deep interaction with pages, and click on all buttons, not just curl to get 200/202 http status codes, that's cheap
- test against prod, and we can fix errors later and improve tests to meet prod
task:
evaluate results on u/admin/crons, and audit and harden producer and consumer of leads for this;
Cron Performance Dashboard Route: /u/admin/crons. Database-backed cron telemetry from cron_job_runs. Includes daisy-chain hop breakdowns, duration analysis, and correlated errors from the last 3 days. Database-backed cron telemetry
for u/admin/crons investigate degradations and make concerted fixes for them; for email scraping we should run search permutations on search queries etc so that we always get new results in searches;
task:
we have these 4 email/domain harvesting scrapers etc -
we have 4 email scraping api route entry points for cron jobs -
alex job search. (with page at u/alex/crons) recruiting coach candidates for dd (with page at u/recruiter/crons) benefactor leads (with page at u/benefactor/crons) dd leads for future coachees (with page at s/leads/crons) the above represent the producer side - producing emails/domains to be stored and contacted by us later -
now, for this task, it's critical that we make a strong effort to bolster/harden/implement/improve the consumption side via cron jobs and api routes for those crons:
enrich the domains with emails via hunter.io, apollo.io, clay, lusha, etc actually send outreach emails out to the contacts given our templates which get matched to via ICPs etc emails are turned off (only emails going to dancingdragons.cc domains are allowed), so you can safely send whatever emails out - we should have throttling tables etc to make sure we don't send the same email out twice in the same 5 week period etc etc, audit and harden that throttling logic
follow existing patterns
start dev server pnpm run dev --port 4002
evaluate - we should be sending out emails within 4+ different/independent consumption api routes for each of the 4 scraping producers track outcomes in outcome tables for mcp and for u/alex/crons, s/leads/crons, u/benefactor/crons, u/recruiter/crons, etc given suboptimal outcomes, make code changes, and go back to step 1
task:
we have 4 email scraping api route entry points for cron jobs -
alex job search. (with page at u/alex/crons) recruiting coach candidates for dd (with page at u/recruiter/crons) benefactor leads (with page at u/benefactor/crons) dd leads for future coachees (with page at s/leads/crons) we want to make sure the web scraping is working for each of these - let's run this loop and improve the scraping -
we need to test and harden them; you can audit and harden by following this loop for both: start the local dev server -
pnpm run dev --port=3004
then we need to do this -
run the cron jobs to harvest domains and emails for all 4 'entry points' (alex job search, benefactor leads, dd leads, and dd recruiting for coach candidates) make sure we get at least 25 domains and 25 emails per run (in order to know we are doing things right!) if not getting emails and domains then edit the files to improve and go to step 1 make sure that all 4 have independent cron job api route entry points and they fan out to the different email scraping strategies independently (browserless, puppeteer, jsdom, vanilla fetch, playwright, chromium, cheerio, etc) and don't cross-contaiminate neondb or supabase tables;
only make changes where you are more than 99.5% confident in the code changes;
task:
audit and harden everything; audit and harden everything again according to docs/audit-and-harden.md
Task:
Discover a new task! IMPORTANT!
the most important task you have - discover something new to work on - a neglected page or feature left hanging, any mock code that should be real, add a new test file or better/new test cases and test coverage. See the last 1000 commits and changelog.md for info. Use the changelog.md file more.
final task:
(if any of the git pushes fail, pull from that branch and merge it in semantically)
give it a good commit message xyz, and also fetch: git add -A; git commit -am "${xyz}" --no-verify; git fetch;
git push origin HEAD:rw1-alex --no-verify || echo 'needs merge'; # Critical - merge with origin/rw1-alex according to docs/critical-merging-instructions.md git push origin HEAD:lb1-alex --no-verify || echo 'needs fetch and merge'; # Critical - merge with origin/lb1-alex according to docs/critical-merging-instructions.md git push origin HEAD:dev --no-verify || echo 'needs fetch and merge'; # Critical - merge with origin/dev according to docs/critical-merging-instructions.md
(all done)