Concurrent Data Pipeline with Backpressure
Implement a high-throughput data processor in Go that simulates the FireGroup analytics workload:
// ProcessData ingests items from a source channel, transforms them via a CPU-intensive
// operation (simulate with time.Sleep), and sends to a sink. Requirements:
// 1. Process up to N items concurrently (worker pool), but limit total concurrent processing
// to prevent OOM under load (backpressure)
// 2. Implement graceful shutdown on context cancellation: finish in-flight items,
// reject new items, close output channel properly
// 3. Preserve order of completion (not necessarily input order, but output must indicate
// which input it corresponds to)
// 4. Handle panics in worker goroutines without crashing the processor
// 5. Optimize for high throughput (minimize allocations, reuse worker goroutines)
type Item struct {
ID int
Value string
}
func ProcessData(ctx context.Context, input <-chan Item, concurrency int) <-chan Result {
// Implement this
}
type Result struct {
ID int
Output string
Error error
}