SKILL.md

name	bfc-knowledge-base
description	Knowledge base for BFC (Batched Financial Calculator) for cost basis runs in accounting. Use when working with cost basis runs, BFC workflows, queue routing, job scheduling, concurrency control, or debugging stuck jobs.
allowed-tools	Read, Grep, Glob, Bash, Task

About this skill

Rule of Thumb

• Use it at my local first, before releasing to the team.
• Define the context for cost basis running, which should be universal for any implementation, no matter it's RQ or RFC.
• Document the design and principle for BFC, eg. one user can only have one BFC job at a time.
• Don't go into details too much, which might be changed frequently, eg. maintaining a codemap. It'll get outdated eventually.
• Create and extract more concrete and self-contained sub skill if possible.

When to Use

• Working with cost basis runs
• Working with BFC job enqueueing or scheduling
• Debugging stuck or failed BFC jobs
• Modifying queue routing logic
• Adding new queue types or user tiers
• Deduping BFC jobs
• Running BFC jobs through a fixed schedule
• Understanding concurrency control (Redis locks, bfc_pending_jobs)
• Investigating cost basis computation issues
• Performance optimization

BFC Knowledge Base

BFC is a Temporal-based workflow system that handles cost basis accounting for cryptocurrency transactions. It computes realized gains/losses for tax reporting by processing user transaction data in batches.

Historical Context

BFC replaced the legacy RQ-based cost basis jobs. Key migration benefits:

Aspect	RQ (Legacy)	BFC (Current)
Max transactions	~200k reliable	2.5M+ supported
Timeouts	2-hour limits	Configurable per activity
Observability	Limited	Temporal UI + Datadog
Incremental	Limited	Full support with recomp dates
Resumability	None	GCS caching enables resume

Note: BFC doesn't guarantee faster execution—RQ ran synchronously which could be faster for small users. BFC's value is reliability, observability, and handling whale accounts.

Read BFC Migration for details over migration process. For Temporal fundamentals at CoinTracker, see Introduction to Temporal at CoinTracker.

Designs

• One user can only have one BFC job at a time. It doesn't make sense to run multiple/concurrent cost basis jobs for a single user. A user's runs are tracked through bfc_pending_jobs for dedup purpose.
• Eventual consistency. A failed BFC run will eventually be retried in the following requests.

API

class CostBasisRunService:
    def request_compute(self, context: CostBasisJobContext):
        """Main entry point for triggering BFC runs."""
        if self.should_use_new_bfc_queue(context.caller_source):
            self._enqueue_bfc_scheduler_workflow(context=context)

class CostBasisJobContext:
    recomputation_date: date | None = None  # None = full sync (0001-01-01)
    caller_source: CostBasisJobCallerSource = CostBasisJobCallerSource.SYSTEM_JOB
    bypass_closed_period: bool = False

Sync Types:

• recomp_date=None → Full sync (translates to 0001-01-01)
• recomp_date=2025-01-01 → Incremental sync from that date

Local Enqueue

For local development, use the flask enqueue-cost-basis-from-gist CLI command:

flask enqueue-cost-basis-from-gist \
  --user-id-gist-url=<gist_raw_url> \
  --recomputation-date=2025-05-01 \
  --diff-context-source=SYNC

• --user-id-gist-url: URL to a raw gist containing the user ID(s) to enqueue
• --recomputation-date: Date to recompute from (omit for full sync)
• --diff-context-source: Context source for the run (e.g., SYNC)

Production Enqueue via GitHub Actions

To enqueue cost basis runs on production at a fixed cadence, use the trigger-add-bfc-recomputation-queue-from-gist GitHub Action. It enqueues users from a gist at a rate of 100 users per minute, useful for bulk re-runs without overwhelming the system.

BFC Scheduler

• manages bfc_pending_jobs to ensure one BFC run job at a time and dedup through pending jobs. When a job is about to be enqueued,
  • if there is no job running, create a bfc_pending_jobs and enqueue
  • if there is a job running,
    • if the user has enabled scheduled cost basis run, update the bfc_pending_job
    • if the job is running healthy, update the bfc_pending_job
    • if the job is in terminated status, cleanup, create a bfc_pending_jobs and enqueue
• builds the actual input to the BFC workflow
• manages the knowledge of queueing logic through BFCQueueManager

8 queues based on user type + load tier:

User Type	Low Queue (<50k flows)	High Queue (≥50k flows)
EMBEDDED	bfc-embedded-low-queue	bfc-embedded-high-queue
WHALE	bfc-whale-low-queue	bfc-whale-high-queue
PAID	bfc-paid-low-queue	bfc-paid-high-queue
CONSUMER	bfc-consumer-low-queue	bfc-consumer-high-queue

User Type Priority: EMBEDDED > WHALE > PAID > CONSUMER

Important: "low" / "high" refers to user transaction volume, NOT priority.

• Low queues = users with <50k flows (covers 0-1k, 1k-5k, 5k-20k txn SLA buckets). These handle ~95% of all BFC runs and are the SLA-critical queues.
• High queues = users with ≥50k flows (whale-tier accounts). Fewer runs, longer execution times, no strict SLA.

The low queues are intentionally provisioned with headroom to handle peak traffic bursts. Idle workers and available task slots are expected and by design — the goal is to keep schedule_to_start_latency low so that P95 end-to-end latency meets SLA targets. Each BFC workflow runs multiple activities sequentially, so per-activity queue wait time compounds across the workflow.

BFC Orchestrator

To run BFC in a scheduled way. Check Scheduled Cost Basis.

Currently, it's only enabled for Mysten.

BFC Workflow Activities

update_cost_basis_state

Mark the status of a cost basis run, STARTED, FINISHED, FAILED

build_transaction_derived_data

go through all user transactions and build metadata can be queried by later steps, eg. calculated balances, transaction_counts_by table and etc.

build_transaction_links

go through all user transactions, trying to figure out the transaction links between transactions. There are four sub activities, considering there are limits on returning at most 2MB data of a Temporal activity, intermediate data is sent to GCS in between.

• build_hash_based_transaction_links_activity
• build_estimate_based_transfer_activity
• build_bridge_detection_activity
• build_link_writer_activity
  • the end results is from this activity, that writes data to transaction_links
  • generates the earliest_link_diff_date

It doesn't support incremental now. An incremental run (with recomp date set) will load all transaction in this activity. This has been our main memory comsuption.

Read [How does transfer detection work today?] (https://www.notion.so/cointracker/How-does-transfer-detection-work-today-2e0a6ee5e05f804f97abc614e9b8786e?source=copy_link) for details.

enrich_workflow_context

Figure out the actual recomp date used in the downstream activities, by getting the minimum of (initial_recomputation_date, earliest_link_diff_date) and adjusting with closed_period_date

build_and_cache_accounting_transactions_in_batches

• Raw transactions are converted to accounting transactions, then cached for cost basis computation. The workflow processes raw transactions and produces accounting-ready transactions for tax calculations.
  • Transactions = raw transaction data from exchanges/wallets (entries)
  • Accounting Transactions = processed/translated transactions ready for tax calculations
  • Applies pricing
  • Handles transaction links (transfers, trades)
  • Generates tax events and classifications
  • Creates callouts for missing data or edge cases
  • Validates and normalizes flows
• Batch size 50k, configured through [Statsig gate] (https://console.statsig.com/3z6qtN2XavYB6weM8SXCOr/dynamic_configs/batched_financial_calculator_runtime_property_config)
• Time-based processing: starts from the actual recomp date and advances day by day. Tracks latest_processed_transaction_id to resume
• Generates accounting transactions in batches and store in GCS

compute_cost_basis_in_batches

• This is the key activity. Taking batched accounting transactions as the input and calculating cost basis through CostBasisCalculationService.compute_cost_basis
  • Loads pre-cached accounting transactions from GCS
  • Processes each transaction chronologically to generate tax lots
  • Applies country-specific cost basis methods (FIFO/LIFO/HIFO/ACB/Share Pooling)
  • Uses incremental computation for efficiency (only recomputes from recomputation date)
  • Persists lots, callouts, and wash sale data to the database
• Batch size 50k, configured through [Statsig gate] (https://console.statsig.com/3z6qtN2XavYB6weM8SXCOr/dynamic_configs/batched_financial_calculator_runtime_property_config)
• It generates many key data, eg. incremental_available_lots, incremental_created_lots and incremental_disposed_lots , tax_summary_snapshots, wallet_summary, and callouts etc

build_and_save_tax_summary_in_batches

• Batch size 50k, configured through Statsig gate
• Fetches accounting transactions from GCS cache
• Builds transaction counts by year and wallet
• Fetches disposed lots for the date range
• Fetches created/available lots (method depends on cost basis method)
• Fetches callouts for the transactions
• Builds incremental tax summaries using IncrementalTaxSummaryBuilderService
• Generates and saves accounting transaction statistics

start_portfolio_workflow

• Child workflow for portfolio/performance calculations
• Runs synchronously (failures block BFC completion)

start_enterprise_workflow only for enterprise user

• Child workflow for enterprise features
• Runs with ABANDON parent close policy (failures don't affect BFC)
• Ideally, enterprise workflow should be running fire-and-forget and get decoupled from BFC. However, there is a dependency that user lots data are locked for the whole BFC running time. Meaning enterprise workflow has to wait for the end of BFC to start running, otherwise there will be an exception when reading from lots data. It's a naive implementation, that works when BFC are fast. Now it becomes a big concern and have other services coupled to BFC running.
  • Sanjiv has made effort through Enterprise Domain Temporal Architecture. Now enterprise workers are running their own queues and workers to scale and get managed separately. We agree that mentally we can consider Enterprise as decoupled by ignoring failures from Enterprise workflow and exclude Enterprise workflow duration from total BFC ones, but the data dependency is the elephant in the room.
  • Accounting has made some effect. An RFC is created targeting the root cause in Versioning BFC Workflow

start_dedup_workflow

enqueue the pending job

instrument_bfc_workflow_result

send instrumentations to DD and BigQuery

Data flow

Batch Processing

• Batch size 50k, configured through Statsig gate
• Each batch processes up to 50K transactions
• Batches are processed sequentially in a while loop
• GCS cache enables resumeability across batches

Runtime

Worker Configuration

• One worker = one process = one thread (concurrency controlled by worker count)
• Grace shutdown: 3hrs in app, 4hrs in k8s (on SIGTERM, finish current task)
• DB connections: 3500 open connections limit with PGBouncer (bumped from 2000); DB itself has a higher limit as headroom. With explicit transactions enabled on all workers, one worker holds one connection at a time.

For workers with label pod=accounting (example), we have a dedicated node pool. We are using n2-highmem-64 machines (64 CPU and 512 GB mem). We are running 4 nodes at a time, with a maximum of 30 (limit is set here).

Node-Level Resource Management (Requests vs Limits)

Each BFC pod has two resource dimensions in Kubernetes:

Field	K8s Concept	_low workers	_high workers	Meaning
gke_min_memory	Request	2560Mi (2.5 GB)	6Gi	Guaranteed minimum, used for scheduling
memory	Limit	4Gi	64Gi	Maximum allowed per pod
gke_min_cpu	Request	200m (0.2 CPU)	200m (0.2 CPU)	Guaranteed minimum
cpu	Limit	3	3	Maximum allowed per pod

The Kubernetes scheduler places pods on nodes based on requests, not limits. CPU request is intentionally kept low (200m) so that memory is the sole driver of pod placement — CPU throttling is harmless (pod just runs slower), while memory OOM kills are catastrophic (pod terminated, job fails).

Two OOM paths:

1. cgroup limit OOM (Kubernetes): pod exceeds its own limit (e.g., _low uses >4 GB) — killed by kubelet
2. node-level OOM (Linux kernel): node runs out of physical memory because too many pods spike at once — kernel kills pods that are furthest above their request, even if they're well below their limit

Scaling Ceiling & Bottlenecks

Current scheduling footprint (609 total BFC pods):

Group	Pods	Request/pod	Total Request
_low (5 groups)	456	2.5 Gi	1,140 Gi
_high (4 groups)	96	6 Gi	576 Gi
Other (scheduler, orchestrator, load_testing)	57	~1.5 Gi	86 Gi
Total	609		~1,802 Gi

• Steady state: ~4 nodes
• During rolling deploy (2x pods due to 4hr graceful shutdown): ~8 nodes

Theoretical scaling ceiling (constrained by rolling deploy — old and new pods coexist):

• Deploy-safe capacity: 14,400 Gi (30 nodes × 480 Gi usable) ÷ 2 = 7,200 Gi
• Scale _low only: 2,880 pods (6.3x current)
• Scale _high only: 1,200 pods (12.5x current)
• Scale current mix proportionally: ~2,400 pods (4x current)

Bottlenecks (in order of concern):

1. DB connections (PGBouncer) — 3,500 limit shared company-wide (web, API, all workers). BFC currently uses ~609/3,500 (17%). Practical BFC ceiling: ~1,500–2,000 workers before starving other services.
2. DB throughput — more concurrent BFC runs = more query load on Postgres. Already a concern at current scale.
3. Node pool (30 nodes) — ~2,400 workers deploy-safe. DB connections bottleneck first.

BFC Worker Groups

Worker naming pattern: worker_temporal_bfc_{user_type}_{priority}

9 queue worker groups serving BFC workloads:

Worker Group	Queue	Segment	Notes
embedded_low	bfc-embedded-low-queue	Embedded	Low tier (<50k flows) — SLA-critical
embedded_low_2	bfc-embedded-low-queue	Embedded	Second deployment for extra capacity
embedded_high	bfc-embedded-high-queue	Embedded	High tier (>=50k flows)
whale_low	bfc-whale-low-queue	Whale	Low tier — SLA-critical
whale_high	bfc-whale-high-queue	Whale	High tier
paid_low	bfc-paid-low-queue	Paid	Low tier — SLA-critical
paid_high	bfc-paid-high-queue	Paid	High tier
consumer_low	bfc-consumer-low-queue	Regular	Low tier — SLA-critical
consumer_high	bfc-consumer-high-queue	Regular	High tier

Infrastructure workers (excluded from reporting): scheduler, orchestrator, load_testing

Segment mapping for reporting: embedded → Embedded, whale → Whale, paid → Paid, consumer → Regular

Parsing Worker Counts from YAML

Worker configuration lives in deployment-config/workers.yml. Each entry has:

- worker: worker_temporal_bfc_{name}
  memory: 4Gi        # per-pod memory limit (_low) or 64Gi (_high)
  cpu: 3             # per-pod CPU limit
  gke_replicas: 128  # actual pod count (this is the worker count)
  gke_min_memory: 2560Mi  # per-pod memory request (scheduling)
  gke_min_cpu: 200m       # per-pod CPU request (scheduling)

Parsing algorithm:

1. Find all entries matching worker_temporal_bfc_*
2. Skip infrastructure workers (scheduler, orchestrator, load_testing)
3. Extract gke_replicas, memory, and cpu for each remaining entry
4. Map worker name substring to segment using WORKER_SEGMENT_MAP

Best practices:

• Use git show origin/master:deployment-config/workers.yml to get production state
• Don't hardcode values — always parse dynamically from the YAML
• Use gke_replicas (not base_scale_instances) as the actual worker count — BFC workers deploy to GKE only

Reference implementation: parse_worker_counts() in bfc-weekly-report/post_to_slack.py

Activity Configuration

Dynamically configured through Statsig - temporal_batched_financial_calculator_workflow_config.

DB

Postgres DB with PGBouncer

• Reads and writes are from primary DB
• PGBouncer connection limit: 3500 (bumped from 2000), shared company-wide (web, API, all workers — not just BFC). DB itself has a higher limit as headroom, but PGBouncer is the current bottleneck. With explicit transactions enabled on all workers, one worker holds one connection at a time. BFC currently uses ~609 of 3,500 connections (~17%).

Monitoring

• Datadog - BFC Dashboard which doesn't support per-user data. It's used as realtime tracking and monitoring for alerts.
• Per-user BFC runs data are instrumented to BigQuery, check How to query BFC runs. Will add the HEX dashboard after we build it

Worker Capacity / Provisioning Metrics

Three key Temporal metrics for assessing whether BFC workers are right-sized. Queryable via Datadog MCP or Metric Explorer:

Metric	What it measures	Over-provisioned signal	Under-provisioned signal
temporal.activity_schedule_to_start_latency{task_queue:*bfc*} by {task_queue}	Time an activity waits in queue before a worker picks it up	Near-zero avg consistently	High avg or large max spikes
temporal.activity_poll_no_task{task_queue:*bfc*} by {task_queue} (as_count)	Workers polling the queue but finding no work (idle polls)	Very high counts	Very low counts
temporal.worker_task_slots_available{task_queue:*bfc*,worker_type:activityworker} by {task_queue}	Unused worker capacity (available slots)	Consistently high	Near zero

Interpretation guidance:

• Low queues are SLA-critical. Idle polls and available slots are expected — they represent headroom for burst handling. Do NOT interpret idle workers on low queues as over-provisioning without also checking schedule_to_start_latency during peak periods.
• Latency compounds across activities. A BFC workflow runs ~5-6 sequential activities. If each waits 1-2s at peak, the cumulative queue wait can be 5-12s — significant for users targeting <20s P95 SLA.
• Evaluate max latency, not just avg. Average latency may look fine while P95/max spikes blow SLA during bursts.

Performance

According to this query for user txns count distributions,

• 98% of users have less than 10k txns
• 99% of users have less than 20k (actually 16.5k) txns
• 99.9% of users have less than 100k txns

Our goal for 2025-2026 tax season, p95 end to end latency

• users with <1K txns: 19.62s
• users with <5K txns: 30.71s
• users with <20K txns: 58.00s

Known Limitations

1. build_transaction_links: No incremental support—always loads all transactions (main memory consumer)

2. Enterprise workflow coupling: Has data dependency on lots when BFC jobs is running. Mentally decouple by ignoring enterprise failures, but the data dependency remains. RFC - Versioning Solution for BFC Data Availability

3. DB throughput: Many concurrent BFC runs can cause DB downgrade. See load testing docs