CRITICAL: Run ALL tasks listed here (yes, you can complete all in one loop interval). Consider them all fresh tasks — remove memory of each task from prior runs. Even if tasks haven't changed since last time, re-run them all in order. Follow docs/critical-merging-instructions.md throughout.
ABSOLUTELY CRITICAL — do NOT skip tasks — do NOT do them out of order!
- Type-checking: do spot checks with --isolatedModules -p tsconfig.json; skip eslint. The -p tsconfig.json flag is important because of path aliases.
- All browser automation must be HEADLESS — no browser windows/tabs popping up on local dev machine.
- The most important ongoing goal: improve email/domain scraping for u/alex, u/recruiter, u/leads, s/leads, and u/benefactor. We should be collecting at least 500 leads/candidates per day for each role. We do not need to send more than 10 emails per day to those leads.
- Prefer src/server/subroutines over src/server/actions; src/server/actions/cache should probably be in src/server/cache.
- Follow instructions exactly, but use best judgement about whether to run against prod at https://dancingdragons.cc or localhost via pnpm run dev.
- LinkedIn creds: [email protected] — allow the human user to input the password in a Chrome window. The email for LinkedIn is NOT [email protected].
- If an env var is missing or an API key is invalid (e.g. HTTP 401), ddlog.warn with "env var missing or api key invalid" plus relevant details. Also try to read from edge-config env-overrides using getEnvSync (not getEnvAsync).
- Scraping success threshold: 25+ emails AND 25+ domains per run. The run needs to do more hops and last longer to hit that target.
If git index is dirty, add all and commit. You MUST resolve all git conflicts even if they are from old commits or commits not from you. Grep for <<<<< or >>>>> and resolve all conflicts according to docs/critical-merging-instructions.md.
make sure we have the right webhooks to respond to emails from [email protected] and [email protected] and [email protected] etc - we have a webhook for [email protected] already, right? it ties into our gmail thread syncing - but when people respond to gmail threads we need to be notified so we can automatically respond etc etc.
dancingdragons.cc should have it's own mcp endpoints - make sure they are built-out etc
Do another grep for <<<<< or >>>>> and resolve any remaining git conflicts according to docs/critical-merging-instructions.md.
For all email/domain scraping code, ensure we have techniques to navigate multi-page sites including pagination and directory listings. Examples of targets we must be able to fully scrape:
- https://www.pisd.edu/about-our-district/staff-directory
- https://www.bentley.edu/directory-index
- https://transparentnevada.com/directory/
Audit and harden all MDP (Markov Decision Process) and Reinforcement Learning code. It must be non-blocking (use dd-proms.ts) and use cron jobs. It needs to make decisions for: assignments, scheduling, coach matching, coach recruiting, A/B testing outreach templates, and improving email scraping strategies on-the-fly. Evaluate where POMDP solutions are needed. Feel free to edit code where you are more than 98% confident.
For every single page/component that uses img tags or imports a component that loads images, ensure there is one tests/playwright test AND one tests/puppeteer test to verify images load. S3 and GCS are having issues; also try R2 and in some cases Google Drive. Priority order: R2 > GCS > S3. Devise a function that discovers all pages with images and creates dynamic tests in tests/puppeteer and tests/playwright.
Truly optimize u/user/messaging by writing deep tests in tests/playwright and tests/puppeteer. Use 3 different incognito windows with 3 different logged-in users via the test bypass env var. Test both group conversations (private/locked and flexible/open) and 1v1 convos. New users added to flexible group convos should see old messages. Improve RxJS where needed and avoid regressions. Ensure search works for conversations. Messages and threads must have embeddings in the message-embeddings table. Thread titles should default to participant first names (e.g. "Alex, Jon, Jeremy, +4 More") — UUID is last resort. Ensure security: only the logged-in user can read their own convos from the UI.
Add a gear icon per thread (when thread is open in selected pane). When clicked, show a dropdown with:
- Show Stats — prosocial/antisocial behavior: word count per person (pie chart), average time to reply (histogram), number of words per message (histogram).
- Notification Settings — user picks when and how to be notified (text message, email, WhatsApp, etc.).
Clean up and improve visuals on blogs. Mermaid diagrams need to render better and ideally have expandable/collapsible interactive sections. Example: https://dancingdragons.cc/weblogs/building-trust-cornerstone-leadership-part-1
If mermaid diagrams won't render well, use a different plugin for reactmarkdown and a different charts/graphs tool. xychart-beta and gantt are not rendering yet — need the right plugin declared/installed. Make blogs interactive so users can click on parts and interact. Pick 5 data-driven blogs and improve them without losing any content/text.
Improve u/leads pages and u/coach/leads pages so users can get a lock on a contact for 3 weeks after they first send an email. If the client signs up, the coach with the lock wins that client.
Create pages:
- u/leads/locks
- u/coach/leads/locks
Implement a db ledger table or db transaction with atomic update on the contact/leadId to handle the locks. Also build a UI where the coach can pick from outreach templates and send the initial email from mail.dancingdragons.cc. A thread ID from Gmail should be synced with our UI in u/coach/leads/locks.
We need a cron job in .github/workflows to sync emails with the neondb db/ui, and a Gmail webhook to sync with neondb. We need a new neondb table for holding/mirroring email threads.
For our coach recruitment outreach email (produced via scraping, consumed by cron jobs to recruit coaches), create and place the template in the right location. The body should say something like:
"Our coaching platform is recruiting coaches to work with corporate and individual clients. We are hiring fractional internal leadership roles as well. See our careers page at https://dancingdragons.cc/careers
Please use this to submit an application and book a 1-and-Done interview to join us! https://dancingdragons.cc/pub/submit-job-application
-Dancing Dragons Team"
Improve the language a bit given the overall code context. Also create outreach templates for different coaching disciplines: leadership coaching, executive coaching, life coaching, etc.
For any/unknown types like the following patterns:
let db: ReturnType;
let dbro: ReturnType;
let db: any;
let dbro: any;
Avoid using any/unknown types. Follow docs/audit-and-harden.md and docs/critical-merging-instructions.md. Go back through the last 10 commits and harden any/unknown types that were introduced.
For u/leads/crons, u/admin/crons, u/alex/crons, and u/benefactor/crons — evaluate whether there is something besides error_tracking and cron_job_runs to get deeper insights into cron run success. The dashboard reads cron_job_runs for per-session telemetry with hop breakdowns and cross-references error_tracking for correlated failures from the last 3 days.
On pub/search/coach:
- The search panel and searchbar needs to extend across the viewport in the top area above results.
- After initiating a search it clears in a way that's poor UX — improve the RxJS.
- Add a search history panel that opens on the left or right, saves the previous 30 searches in localStorage or IndexedDB, and also stores them in a new Supabase search query string table paired with which searchbar was used.
On pub/search/coaches (a different page):
- The searchbar is better/wider already, but add a sort: by bio length, skills declared, gender, age, etc.
Both pages must exist independently and be fully functional. Look in git history to restore the missing one if needed. Both should have entirely different client components and SSR components. Both need infinite scroll (paginated queries using streaming search) and heavy use of RxJS. Search should lean heavily on embeddings/vector/tsvector — not just search by name. Use new streaming API queries, not the ones from the match page.
Several neondb/tables entities need to be cached so the browser doesn't re-fetch megabytes of data each page load. Entities: skills, industries, job functions, tags, categories, institutions, credentials, certifications.
Cache in the following layers:
- Redis (least necessary since we have neondb read replicas)
- Supabase — mirror all the above tables and sync between them. No RLS needed on these mirror tables since entities are public/shared.
- IndexedDB for each user's browser.
- Cloudflare by pre-fetching the match page, u/coach, u/coach/dashboard, u/coach/profile/edit, etc.
Focus on skills caching first to avoid the 2MB download on each page load. Note: skills and tags/categories will be more than 2MB each so unstable_cache won't work for those. Industries and job functions will stay below 2MB.
For the blog "Building Trust as a Cornerstone of Leadership: Part 1 – The Foundations of Trust" on stage and prod (weblogs table):
- The author attribution must say "By Dr. Mike Mercer, DEL, MSSL" not "By system dd-system-user". His name should link to his coaching profile.
- Update the backend routine to allow an admin user to override the author.
- List mermaid as a plugin in the weblogs table and declare any other reactmarkdown plugins needed to render better.
- When updating blogs, previous versions of content must be stored in the archived-entities table. Previous versions must be viewable and recoverable. Use dbtx (db-with-transactions-3.ts) to do this.
URL: https://dancingdragons.cc/weblogs/building-trust-cornerstone-leadership-part-1
These pipelines should learn from each other. Identify what is working and what isn't working, and cross-pollinate scraping improvements across:
- u/admin/leads
- u/leads
- s/leads
- u/benefactor
- u/recruiter/leads
- u/alex/leads
Use docs/debug-prod-final.md and resolve 3–9 issues from the last 10 hours, including this known error:
Failed to list pinned tags: Error: Failed query: select "tag_relations"."id", "tags"."id", "tags"."tag_name", "tags"."tag_color", "tags"."tag_description", "tag_relations"."pinned_at", "tag_relations"."created_at" from "tag_relations" inner join "tags" on "tag_relations"."tag_id" = "tags"."id" where ("tag_relations"."entity_name" = $1 and "tag_relations"."entity_id" = $2 and "tag_relations"."is_pinned" = $3 and "tag_relations"."is_soft_deleted" is null and "tags"."is_soft_deleted" is null and "tags"."is_active" = $4) order by "tag_relations"."pinned_at" desc, "tag_relations"."created_at" desc limit $5 params: users,963bd832-65ae-47bd-8b24-f899becdb78a,true,true,20
Build out u/bizdev/leads and u/bizdev/crons for business leads (not individual consumer leads). Create a new scraping pipeline to scan the web for businesses that need coaches and send out 5 emails to each company for a business dev offer of 20–100 coaches per quarter.
This scraping and cron pipeline must be totally independent from u/leads and s/leads. Create email templates for business ICPs. Create a separate neondb ICP table for B2B (separate from B2C individual customer profiles). Improve u/bizdev, u/bizdev/home, and u/bizdev/dashboard. Make sure the "more..." menu works in components/nav/bizdev-header.tsx.
Investigate u/recruiter/crons and u/recruiter/candidates. Is our email scraping to find coaches to recruit working? Audit and harden coach recruiting email scraping. Make sure all pages are present, fully featured, and fully functional.
Make sure we are pre-loading and caching these pages with Cloudflare:
- u/leads, u/benefactor, u/admin, u/owner, u/coach, u/user, u/recruiter, u/livestreamer, u/marketing, u/alex, u/intake, u/client, u/coach_manager
Audit and harden Cloudflare and BrightData caching.
Create or improve the following cron pages so each team can investigate what is working and what is not:
- u/alex/crons
- u/marketing/crons
- u/hr/crons
- u/ops/crons
- u/coach_manager/crons
- u/qa/crons
- u/intake/crons
- u/bizdev/crons
All must be full-featured and fully functional with database-backed cron telemetry.
Make sure u/alex and u/alex/crons is successfully scraping and applying for jobs. Current telemetry shows all zeros — this must be fixed. Too many 0's across formsFound, targetsStaged, formsSubmitted, etc.
Audit and harden our own MCP server(s) for dancingdragons.cc. There are three MCP surfaces:
- src/tools/mcp/dd-integrations-mcp-server.ts — 124 lines, 3 tools (Lusha enrich, Apollo search/enrich). Orphaned: registered nowhere, strict subset of dd-lead-stack. Should be deleted.
- src/tools/mcp/dd-lead-stack-mcp-server.ts — 29 KB, ~12 providers. Registered in mcp.json but dropped from .mcp.json.
- src/app/api/mcp/route.ts — HTTP MCP at /api/mcp, name dd-entity-embeddings, gated by CRON_SECRET Bearer auth, 2 tools (search_embeddings, list_entity_kinds).
Recommended plan: delete the orphan (#1), leave dd-lead-stack (#2) as-is but confirm whether dropping from .mcp.json was intentional, and harden the HTTP route (#3): separate auth secret from CRON_SECRET, allowlist enum params, redact error.data, derive entity kinds from DB, add rate limiting, use structured JSON-RPC tool errors.
Audit and harden u/user/_test/exp. Verify that "breaker snapshot" is working across all three wrappers:
- Page Wrapper (SSR — wrapAsyncPage)
- Server Action (generic-actions pattern)
- API Route (wrapApiRoute)
All three should pass: ALS Context, Cookies, Headers, Env Overrides, Breaker Snapshot, Browser Session.
Create or update docs/series-assignment-groups-meetings.md. Document how coaches invoice the company based on assignments to clients:
- When an assignment is first created, a series record should be created in the series table.
- When the cadence for meetings changes, update the series table to use a new cadence.
- The cadence/series helps determine when to bill clients in advance and when to send reminders to coaches who forget to submit an invoice.
Reference: https://dancingdragons.cc/u/coach/postmortem, https://dancingdragons.cc/u/coach/client-cadence-update, https://dancingdragons.cc/u/coach/forms/single-session-invoice, https://dancingdragons.cc/u/coach/group-invoice
Only update the doc — do not update any code yet. Include diagrams showing desired relationships between tables: meetings, invoices, groups, group sessions, assignments, series, etc.
Create or update docs/intake-user-onboard-client.md. Document the tables involved in onboarding clients and assigning them to a coach:
- When a coach-client assignment is first created, create a series record in the series table.
- Send a contract for the client to sign.
- Email the client a bill for $100 via Stripe to reserve a coach.
- Document how banking/ledger/billing tables handle the $100 Stripe charge.
- Document how assignments and series work and how the first session is scheduled.
- Document how reminders work — ideally the coach tags our email on their calendar, but we should also track when the first session happens in our own schedule/cal table.
Only update the doc — do not change code yet. Add diagrams.
For u/benefactor and u/benefactor/leads — when sending outreach emails, make a best effort to tailor each email to the contact/lead. Leverage templates interpolated with data from their website, Yelp reviews, Google Business, Facebook, or any public API. Identify shortcomings in their marketing and offer to shore those up.
Draft multiple templates for each of the following, designed to be inputted into Gemini AI for per-contact personalization:
Assigned to Elijah Gizzarelli:
- Handyman service
- Locksmithing
- Roofing
Assigned to Ekaterina Muntyan:
- Dentists
- Photographers
- House painters
Assigned to Vinayak Pandey:
- Pest control
- Security staff
- DJs
Audit and harden u/benefactor/emails, u/benefactor/contacts, and u/benefactor/hubspot. The benefactor_contacts data model needs to be fully wired up. Add a HubSpot sync button to all relevant u/benefactor pages. Also add u/leads/hubspot and s/leads/hubspot pages with HubSpot sync buttons.
(ignore this, outmoded task)
For every single private page behind (private)/u/* and (private)/s/*, make sure it is responsive and mobile-friendly. Focus only on private pages. Be careful and avoid regressions.
Audit and harden the cron job for Google Search Console indexing of public pages at service/[slug], coaches/[id], and weblogs/[slug]:
- Make /api/cron/index-pages the canonical daily SEO discovery job.
- Add Vercel cron scheduling at 0 8,9 * * * (UTC) — the route already guards for 4am New York.
- Always submit/refresh https://dancingdragons.cc/sitemap.xml through the Search Console Sitemaps API.
- Inspect a rotating sample of priority URLs from /service, /coaches, and /weblogs.
- Only use the Indexing API behind an explicit feature flag for truly eligible structured-data URLs.
Also vastly improve service/[slug] pages:
- Add neondb/tables/service-locations-tables.ts (fields: id, locationType, city, region, country, countryCode, lat, lon, canonicalKey, baseSlug, publicSlug, shortId, isActive, audit fields).
- Add neondb/tables/services-offered-tables.ts (fields: id, name, slug, description, longDescription, aliases, seo, icon, status, sortOrder, audit fields).
Slug patterns:
- /service/financial-coaching
- /service/financial-coaching/delaware-a1b2c3 (territory always gets stable nanoid(6) suffix)
- /service/delaware-a1b2c3
Keep all existing patterns working. Update sitemap generation to emit canonical DB-backed service and service-location URLs. If the cross-product grows beyond sitemap limits, split into sitemap indexes.
Fix the following issues observed in u/leads/crons telemetry:
- lead-generation-pipeline repeatedly failing with "No prospects found — pipeline ending early" for segments: bizdev, ics, leadership, icp.
- browser-scrape-emails consistently degraded (3/10) with 0 emails scraped, 0 URLs visited, 3 errors per run.
- scrape-emails getting HTTP 508 Loop Detected.
- scrape-emails-multi-tool completing with emailsFound: 0 and urlsVisited: 0.
- 19 zombie sessions in the last 3 days.
For email scraping daisy chains: keep running until we get at least 25 emails and 25 domains, or until 10 minutes has elapsed, whichever comes first. Make at least 1 hop if it is a daisy-chained cron. Run search permutations on queries so we always get new results.
Fix the following issues observed in u/admin/crons telemetry (read from neondb prod directly since the page may be pin-protected):
- alex-job-apply-consumer consistently degraded (3/10) with formsFound: 0, targetsStaged: 0, formsSubmitted: 0 across every 30-minute interval.
- alex-auto-apply showing skipped (5/10) with profileConfigSkipped: true — fix the profile config so it is not skipping.
- coach-recruitment degraded (3/10): sent 0, errors 6, failed 6, skipped 14 out of 20 discovered.
- alex-job-search degraded (3/10): 8 error count, 1 email found direct, captcha hit on ycombinator.com/bing.
- Same lead-generation-pipeline and browser-scrape-emails failures as in u/leads/crons.
- 19 zombies.
Run all crons in series against prod. There is a script for this in src/tools or scripts. Then run all crons in series against localhost:3009 with dev server at pnpm run dev --port 3009. Kill the local dev server when done.
Fully test the u/journal page:
- Share 1–3 journal entries with [email protected] and [email protected].
- Also share with a group of those same 2 users.
- Test u/user/messaging: start a conversation with those users, and use the same group for sharing and messaging.
- Use both tests/playwright and tests/puppeteer.
- Create meaningful tests with deep navigation.
- Reference generic tests in tests/general, tests/puppeteer, and tests/playwright and audit/harden those too.
The 4 email scraping producer entry points are:
- Alex job search (page: u/alex/crons)
- Recruiting coach candidates for DD (page: u/recruiter/crons)
- Benefactor leads (page: u/benefactor/crons)
- DD leads for future coachees (page: s/leads/crons)
For the consumption side, bolster and harden:
- Enrich domains with emails via hunter.io, apollo.io, clay, lusha, etc.
- Actually send outreach emails to contacts matched via ICPs and templates.
Note: emails are turned off except to dancingdragons.cc domains — you can safely send whatever. Audit and harden throttling logic so we never send the same email twice within a 5-week period.
Start dev server: pnpm run dev --port 4002
Loop:
- Evaluate — we should have 4+ independent consumption API routes (one per producer).
- Track outcomes in outcome tables for u/alex/crons, s/leads/crons, u/benefactor/crons, u/recruiter/crons.
- Given suboptimal outcomes, make code changes and go back to step 1.
Start the local dev server: pnpm run dev --port=3004
Loop:
- Run the cron jobs to harvest domains and emails for all 4 entry points.
- Confirm we get at least 25 domains and 25 emails per run.
- If not, edit the files to improve and go back to step 1.
Ensure all 4 have independent cron API route entry points that fan out to different scraping strategies independently (browserless, puppeteer, jsdom, vanilla fetch, playwright, chromium, cheerio, etc.). They must not cross-contaminate neondb or supabase tables.
Only make changes where you are more than 99.5% confident.
Audit and harden everything. Audit and harden everything again according to docs/audit-and-harden.md.
Run docs/debug-prod-final.md and resolve 1–3 issues from the last 90 minutes.
git fetch && git merge origin/lb1-alex # resolve conflicts semantically per docs/critical-merging-instructions.md
git fetch && git merge origin/rw1-alex # resolve conflicts semantically per docs/critical-merging-instructions.md
the most important task you have here - discover something new to work on - a neglected page or feature left hanging, any mock code that should be real, add a new test file or better/new test cases and test coverage. See the last 1000 commits and changelog.md for info. Use the changelog.md file more.
Give it a good commit message (xyz), then:
git add -A; git commit -am "${xyz}" --no-verify;
git fetch;
git push origin HEAD:rw1-alex --no-verify || echo 'needs merge'; # merge with origin/rw1-alex per docs/critical-merging-instructions.md
git push origin HEAD:lb1-alex --no-verify || echo 'needs fetch and merge'; # merge with origin/lb1-alex per docs/critical-merging-instructions.md
git push origin HEAD:dev --no-verify || echo 'needs fetch and merge'; # merge with origin/dev per docs/critical-merging-instructions.md
(all done)