This is a project-based Zig guide. It explains the language and standard-library
features used by cl365-lead-enrichment, a command-line data
pipeline that:
- downloads CSV files from Google Drive;
- normalizes US phone numbers;
- deduplicates those numbers with a disk-backed bitmap;
- queries a DNC provider under a rate limit;
- builds an indexed local lookup;
- joins DNC data back into compressed enriched CSV files.
The codebase targets Zig 0.16.0. Zig evolves quickly, so examples written for older versions may use different process, file, networking, or I/O APIs.
The examples below are shortened to emphasize one idea. The project source is the authoritative complete implementation.
This is not a toy program. It processes tens of gigabytes and millions of phone numbers while remaining restartable. That forces the code to address several important systems-programming concerns:
- explicit memory ownership;
- bounded memory use;
- streaming rather than loading entire datasets;
- binary representation and bit manipulation;
- memory-mapped files;
- HTTP clients and a local HTTP test server;
- JSON and CSV parsing;
- gzip compression;
- concurrency, cancellation, clocks, and rate limiting;
- durable checkpoints and atomic file replacement;
- precise error handling and diagnostics.
Those concerns fit Zig well because the language makes resource ownership, errors, and data representation visible in the source.
| Area | Representative modules | Main Zig concepts |
|---|---|---|
| CLI | src/main.zig |
process initialization, argument iteration, error handling |
| Configuration | src/config/env_file.zig |
allocators, string parsing, error sets, maps |
| Drive API | src/drive/*.zig |
HTTP, JSON, URI encoding, retries, resumable transfers |
| Phone normalization | src/phones/normalize.zig |
arrays, slices, optionals, integer arithmetic |
| Deduplication | src/phones/bitmap.zig, src/phone_dedupe.zig |
mmap, bitsets, binary files, endianness |
| CSV transformation | src/phones/csv_transform.zig |
streaming parsers, callbacks, borrowed data |
| DNC client | src/dnc/*.zig |
HTTP POST, tagged unions, arenas, diagnostics |
| Scheduling | src/dnc/scheduler.zig, src/phone_scrub.zig |
clocks, rate limits, std.Io.Select, cancellation |
| DNC storage | src/dnc/storage.zig |
gzip, SHA-256, atomic output |
| Local lookup | src/enrich/lookup.zig |
multiple mmaps, offsets, binary search, checkpoints |
| Final enrichment | src/enrich/csv_transform.zig |
generic contexts, joins, gzip output |
| Build and tests | build.zig, build.zig.zon |
modules, dependencies, build steps, test discovery |
Typical development commands are:
zig fmt build.zig src
zig build
zig build test
zig build -Doptimize=ReleaseSafe run -- helpThe -- separates arguments understood by zig build from arguments passed to
the executable.
Zig has four standard optimization modes:
Debug: safety checks and convenient debugging;ReleaseSafe: optimization with safety checks;ReleaseFast: maximum speed, fewer safety checks;ReleaseSmall: optimize for binary size.
For a long-running data pipeline, ReleaseSafe is a useful default: it is much
faster than Debug but still detects many invalid operations.
const creates an immutable binding and var creates a mutable binding:
const std = @import("std");
const max_attempts: usize = 8;
const buffer_size = 256 * 1024; // comptime-known integer
var completed_files: u64 = 0;
completed_files += 1;Zig infers types when the initializer contains enough information. Add explicit types at API boundaries and whenever they clarify representation.
Integer literals may contain underscores:
const ten_digit_min: u64 = 1_000_000_000;
const requests_per_minute: u32 = 100;There are no implicit integer conversions. Use an explicit cast when moving between widths:
const index: usize = 42;
const stored: u64 = @intCast(index);
const widened: u128 = @as(u128, stored);@intCast infers the destination type from context. @as supplies a type
explicitly. The project often widens arithmetic to u128 before multiplying so
that large byte and time calculations cannot overflow a u64 intermediate.
A Zig array owns a fixed number of elements:
var digits: [11]u8 = undefined;
var digest: [32]u8 = undefined;A slice is a pointer plus a length. It views memory owned elsewhere:
const all_digits: []const u8 = digits[0..digit_count];
const mutable_prefix: []u8 = digits[0..3];The const in []const u8 applies to the bytes, not merely to the slice
variable. Zig strings are normally UTF-8 byte slices, not a special string type:
const name: []const u8 = "US_S01_001-1.csv";Important lifetime rule: a slice does not keep its backing storage alive. For example, a CSV callback may receive fields borrowed from the parser's temporary buffer. If a field must survive after the callback returns, duplicate it:
const owned = try allocator.dupe(u8, borrowed_field);
defer allocator.free(owned);The DNC response parser uses exactly this distinction: parser rows are borrowed, then retained fields are duplicated into an arena.
Most of this project uses ordinary slices. When interoperating with operating
system or C APIs, Zig may use sentinel-terminated types such as [:0]const u8.
Do not add a sentinel unless the called API requires it.
Structs group data and behavior:
const Totals = struct {
files_processed: u64 = 0,
files_skipped: u64 = 0,
rows: u64 = 0,
valid: u64 = 0,
invalid: u64 = 0,
fn add(self: *Totals, rows: u64, valid: u64, invalid: u64) void {
self.rows += rows;
self.valid += valid;
self.invalid += invalid;
}
};
var totals: Totals = .{};
totals.add(100, 98, 2);.{} uses the expected type and fills fields with their defaults. A method is
just a namespaced function whose first argument is conventionally self.
Configuration structs benefit from defaults:
const Config = struct {
input_dir: []const u8 = "data/csv_original",
output_dir: []const u8 = "data/work/csv_normalized",
rebuild: bool = false,
};The provider response maps one-byte result codes to semantic enum values:
const ResultCode = enum {
malformed,
clean_landline,
blocked,
do_not_call,
invalid,
clean_wireless,
clean_voip,
unknown,
};
fn decodeResultCode(value: []const u8) ResultCode {
if (value.len != 1) return .unknown;
return switch (value[0]) {
'M' => .malformed,
'C' => .clean_landline,
'B' => .blocked,
'D' => .do_not_call,
'I' => .invalid,
'W' => .clean_wireless,
'Y' => .clean_voip,
else => .unknown,
};
}An enum may specify its backing integer type. This is useful when enum values are array indices:
const Field = enum(usize) {
phone,
result_code,
reason,
};
const fields: [3][]const u8 = .{ "3125550100", "C", "" };
const phone = fields[@intFromEnum(Field.phone)];switch is an expression: every branch produces the result value. Exhaustive
checking means adding an enum case can reveal all switches that need updating.
An enum says which state exists. A tagged union says which state exists and stores different data for each state:
const RequestOutcome = union(enum) {
success: Batch,
retryable: RetryInfo,
permanent_failure: FailureInfo,
};
switch (outcome) {
.success => |*batch| batch.deinit(),
.retryable => |info| scheduleRetry(info.delay),
.permanent_failure => |info| logFailure(info),
}The tag and payload cannot disagree. This is safer than a struct containing
several nullable fields such as batch, retry, and failure.
The project also uses tagged unions to race asynchronous operations:
const Race = union(enum) {
scrub: ClientError!Batch,
timeout: void,
};The selected tag tells the caller whether the HTTP operation or timeout won.
?T holds either a T or null. Phone normalization naturally returns an
optional because invalid input is expected data, not a system failure:
fn normalizePhone(raw: []const u8) ?u64 {
var value: u64 = 0;
var digits: usize = 0;
for (raw) |byte| {
if (byte < '0' or byte > '9') continue;
if (digits == 10) return null;
value = value * 10 + (byte - '0');
digits += 1;
}
return if (digits == 10) value else null;
}Unwrap with if capture:
if (normalizePhone(raw)) |phone| {
try writer.print("{d}", .{phone});
} else {
try writer.writeAll("inalid"); // project-required spelling
}Use orelse for a fallback or early return:
const equals_index = std.mem.indexOfScalar(u8, line, '=') orelse continue;
const token = maybe_token orelse return error.MissingAccessToken;Use .? only when absence is an invariant violation and a panic is appropriate.
Zig errors are values. An error set lists expected failure categories:
const ConfigError = error{
MissingLogin,
InvalidTimeout,
OutOfMemory,
};E!T means a function either returns T or an error from E:
fn parseTimeout(text: []const u8) ConfigError!u64 {
return std.fmt.parseInt(u64, text, 10) catch error.InvalidTimeout;
}try returns an error immediately and otherwise unwraps the success value:
const timeout = try parseTimeout(text);catch handles or transforms an error:
const timeout = parseTimeout(text) catch |err| {
std.debug.print("bad timeout: {s}\n", .{@errorName(err)});
return err;
};anyerror!T is convenient at broad boundaries, but precise error sets are useful
inside modules because they document behavior and let callers distinguish retry,
configuration, parsing, and permanent failures.
The DNC code deliberately separates invalid provider data from HTTP failures. That prevents malformed data from being retried forever as if it were a network problem.
defer runs when the current scope exits for any reason:
const data = try std.Io.Dir.cwd().readFileAlloc(
io,
path,
allocator,
.limited(4 * 1024 * 1024),
);
defer allocator.free(data);errdefer runs only if the scope returns an error. It is essential when building
an owning value in stages:
fn cloneFile(allocator: std.mem.Allocator, source: File) !File {
const id = try allocator.dupe(u8, source.id);
errdefer allocator.free(id);
const name = try allocator.dupe(u8, source.name);
errdefer allocator.free(name);
return .{ .id = id, .name = name };
}After successful return, neither errdefer runs; ownership transfers to the
returned struct. If the second allocation fails, the first allocation is freed.
For resources with several fields, define deinit:
const OwnedFile = struct {
id: []u8,
name: []u8,
fn deinit(self: *OwnedFile, allocator: std.mem.Allocator) void {
allocator.free(self.id);
allocator.free(self.name);
self.* = undefined;
}
};Assigning undefined after cleanup helps expose accidental reuse during debug
builds. It is a convention, not a garbage collector.
Zig does not hide allocation behind ordinary language operations. Functions that
allocate receive a std.mem.Allocator:
fn markerPath(
allocator: std.mem.Allocator,
state_dir: []const u8,
relative_path: []const u8,
) ![]u8 {
const marker = try std.fmt.allocPrint(
allocator,
"{s}.done",
.{relative_path},
);
defer allocator.free(marker);
return std.fs.path.join(allocator, &.{ state_dir, marker });
}The returned slice is owned by the caller. The intermediate formatted string is freed locally.
The executable receives a process initializer containing the allocator and I/O implementation:
pub fn main(init: std.process.Init) !void {
try run(init.gpa, init.io);
}Passing both values downward makes allocation and I/O dependencies visible and testable.
An arena is excellent when many values have the same lifetime. The DNC response parser duplicates every retained CSV field into one arena:
var arena = std.heap.ArenaAllocator.init(parent_allocator);
errdefer arena.deinit();
const result_allocator = arena.allocator();
const owned_field = try result_allocator.dupe(u8, borrowed_field);
return .{
.arena = arena,
.rows = rows,
};The batch releases all rows and fields at once by calling arena.deinit().
Arenas trade individual frees for simple bulk lifetime management.
Use std.testing.allocator in unit tests. The test runner checks for leaks:
test "owned copy is released" {
const allocator = std.testing.allocator;
const copy = try allocator.dupe(u8, "hello");
defer allocator.free(copy);
try std.testing.expectEqualStrings("hello", copy);
}In Zig 0.16, this project commonly uses an unmanaged std.ArrayList initialized
with .empty. Allocation calls receive the allocator:
var files: std.ArrayList(OwnedFile) = .empty;
defer {
for (files.items) |*file| file.deinit(allocator);
files.deinit(allocator);
}
try files.append(allocator, owned_file);Turn the list into an owned slice when returning it:
const owned: []OwnedFile = try files.toOwnedSlice(allocator);After toOwnedSlice, the caller owns the slice. The list no longer owns the same
buffer.
For maps and sets, the same ownership questions apply: who owns the keys, who owns the values, and which allocator releases the container storage? In this project, a 6.4-billion-bit bitmap is much more memory-efficient than a hash set of millions of 64-bit phone numbers.
Zig generics are ordinary functions evaluated with compile-time-known types. The enrichment transformer accepts any lookup object that provides the expected method:
fn enrichFile(
allocator: std.mem.Allocator,
lookup: anytype,
input: []const u8,
) !void {
const Context = TransformContext(@TypeOf(lookup));
var context: Context = .{
.allocator = allocator,
.lookup = lookup,
};
try parseRows(Context, &context, input);
}
fn TransformContext(comptime LookupType: type) type {
return struct {
allocator: std.mem.Allocator,
lookup: LookupType,
fn onPhone(self: *@This(), phone: u64) !void {
if (try self.lookup.find(phone)) |record| {
// Write joined data.
_ = record;
}
}
};
}There is no runtime interface object here. Zig specializes the function for the
concrete lookup type and checks at compilation that find exists with a usable
signature.
This also enables lightweight test doubles:
const FakeLookup = struct {
fn find(_: @This(), phone: u64) !?[]const u8 {
return if (phone == 3_125_550_100) "C,landline" else null;
}
};Use comptime when behavior or layout truly depends on a type or value known at
compilation. Ordinary runtime values should stay ordinary runtime values.
Each Zig file is a module namespace:
const std = @import("std");
const normalize = @import("phones/normalize.zig");
const csv = @import("csv_parser");The first two imports resolve built-in or relative modules. csv_parser is a
named module connected by build.zig.
Symbols are private unless declared pub:
pub const bitmap_bit_count: u64 = 6_400_000_000;
pub fn normalize(raw: []const u8) ?u64 {
// ...
}
fn internalHelper() void {}It is usually better to divide code by responsibility than to create one very large source file. This project separates Drive API modeling, transfer/resume, progress, DNC scheduling, response parsing, storage, and lookup building.
A shortened version of the project's build structure looks like this:
const std = @import("std");
pub fn build(b: *std.Build) void {
const target = b.standardTargetOptions(.{});
const optimize = b.standardOptimizeOption(.{});
const csv_dep = b.dependency("zig_csv_parser", .{
.target = target,
.optimize = optimize,
});
const app = b.createModule(.{
.root_source_file = b.path("src/main.zig"),
.target = target,
.optimize = optimize,
.imports = &.{
.{
.name = "csv_parser",
.module = csv_dep.module("csv_parser"),
},
},
});
const exe = b.addExecutable(.{
.name = "leads",
.root_module = app,
});
b.installArtifact(exe);
const run = b.addRunArtifact(exe);
if (b.args) |args| run.addArgs(args);
b.step("run", "Run leads").dependOn(&run.step);
const tests = b.addTest(.{ .root_module = app });
const run_tests = b.addRunArtifact(tests);
b.step("test", "Run tests").dependOn(&run_tests.step);
}The actual build.zig should be consulted for exact names. The key ideas are:
- target and optimization mode are user-selectable;
- dependencies become named imports;
- the executable and tests can reuse the same root module;
- named build steps provide
zig build runandzig build test.
The package manifest records package identity, minimum Zig version, dependencies, and included paths. A dependency is pinned with a URL and content hash:
.dependencies = .{
.zig_csv_parser = .{
.url = "git+https://example.invalid/repository#commit",
.hash = "...",
},
},Pinning makes builds reproducible. Update both the revision and hash deliberately.
Zig 0.16 supplies process resources through std.process.Init:
pub fn main(init: std.process.Init) !void {
var args = try std.process.Args.Iterator.initAllocator(
init.minimal.args,
init.gpa,
);
defer args.deinit();
_ = args.next(); // executable name
const command = args.next() orelse {
printUsage();
return;
};
if (std.mem.eql(u8, command, "download")) {
try download.run(init, &args);
} else if (std.mem.eql(u8, command, "extract")) {
try extract.run(init, &args);
} else {
return error.UnknownCommand;
}
}A small project can parse options manually. Recognize each flag, consume its value, validate it immediately, and reject unknown options. That keeps invalid configuration from reaching the long-running part of the pipeline.
At the top-level boundary, std.process.fatal is useful for an error that should
produce a message and nonzero exit:
runCommand(init, &args) catch |err| {
std.process.fatal("command failed: {s}", .{@errorName(err)});
};Many older Zig examples call file operations without an I/O parameter. This
project uses Zig 0.16 APIs where operations receive a std.Io capability:
fn loadConfig(
allocator: std.mem.Allocator,
io: std.Io,
path: []const u8,
) ![]u8 {
return std.Io.Dir.cwd().readFileAlloc(
io,
path,
allocator,
.limited(4 * 1024 * 1024),
);
}Passing std.Io supports the standard library's concurrency and testable I/O
design. Store it in long-lived objects that perform repeated operations:
const Progress = struct {
io: std.Io,
total_files: u64,
completed_files: u64 = 0,
};Never allow a remote server or malformed file to make an unbounded allocation:
const body = try response.reader.allocRemaining(
allocator,
.limited(max_response_bytes + 1),
);
defer allocator.free(body);
if (body.len > max_response_bytes) return error.ResponseTooLarge;The same principle appears in configuration files, provider rows, CSV rows, and HTTP responses.
Large files should be streamed through bounded buffers:
var input_buffer: [256 * 1024]u8 = undefined;
var output_buffer: [256 * 1024]u8 = undefined;
var input_reader = input_file.reader(io, &input_buffer);
var output_writer = output_file.writer(io, &output_buffer);
try input_reader.interface.streamRemaining(&output_writer.interface);
try output_writer.interface.flush();
try output_file.sync(io);Buffering reduces system-call overhead. flush moves bytes from the userspace
buffer to the file abstraction. sync asks the operating system to make file
data durable. They solve different problems.
For append/resume behavior, seek to a validated position before writing:
try partial_file.seekTo(io, existing_bytes);Do not assume a partial file is valid merely because it exists. The Drive downloader verifies the server's range response and local metadata before continuing.
src/drive/progress.zig wraps a destination writer and counts bytes without
changing the transfer code. The central pattern is:
const ProgressWriter = struct {
destination: *std.Io.Writer,
progress: *Progress,
interface: std.Io.Writer,
fn drain(
writer: *std.Io.Writer,
data: []const []const u8,
splat: usize,
) std.Io.Writer.Error!usize {
const self: *ProgressWriter = @alignCast(
@fieldParentPtr("interface", writer),
);
const consumed = try self.destination.writeSplatHeader(
writer.buffered(),
data,
splat,
);
self.progress.addBytes(consumed);
return writer.consume(consumed);
}
};@fieldParentPtr recovers the containing ProgressWriter from its embedded
interface field. @alignCast confirms the required alignment. This is Zig's
explicit version of implementing a writer interface with a vtable.
The benefit is composition:
HTTP response reader -> progress-counting writer -> buffered file writer
The HTTP copy loop knows nothing about terminal progress.
The pipeline inventories input files before processing them. Typical operations include:
var directory = try std.Io.Dir.cwd().openDir(io, root, .{ .iterate = true });
defer directory.close(io);
var walker = try directory.walk(allocator);
defer walker.deinit();
while (try walker.next(io)) |entry| {
if (entry.kind != .file) continue;
if (!std.mem.endsWith(u8, entry.path, ".csv")) continue;
// Duplicate entry.path if it must outlive this iteration.
}Build paths portably with std.fs.path.join:
const output_path = try std.fs.path.join(
allocator,
&.{ output_dir, relative_path },
);
defer allocator.free(output_path);Sort an inventory before hashing or processing it:
std.mem.sort(InputFile, files.items, {}, struct {
fn lessThan(_: void, left: InputFile, right: InputFile) bool {
return std.mem.lessThan(u8, left.relative_path, right.relative_path);
}
}.lessThan);Deterministic ordering makes fingerprints, logs, and output reproducible across runs even if the file system returns directory entries in a different order.
Long-running tools must survive interruption. The project follows a common transaction-like pattern:
- write to
name.part; - flush the buffered writer;
- sync the file;
- close it;
- rename it to the final name;
- write a completion marker containing input identity and statistics.
A simplified helper:
fn writeAtomic(
allocator: std.mem.Allocator,
io: std.Io,
final_path: []const u8,
data: []const u8,
) !void {
const partial_path = try std.fmt.allocPrint(
allocator,
"{s}.part",
.{final_path},
);
defer allocator.free(partial_path);
const file = try std.Io.Dir.cwd().createFile(io, partial_path, .{});
defer file.close(io);
var buffer: [64 * 1024]u8 = undefined;
var file_writer = file.writer(io, &buffer);
try file_writer.interface.writeAll(data);
try file_writer.interface.flush();
try file.sync(io);
try std.Io.Dir.cwd().rename(
partial_path,
std.Io.Dir.cwd(),
final_path,
io,
);
}The exact close ownership and rename options should follow the project's
tested helpers; this shortened example illustrates the ordering.
A marker can record:
- format version;
- inventory fingerprint;
- input size and modification time;
- output row counts;
- checksum or expected size.
On restart, the tool skips work only when the marker and output agree with the current input. This turns restartability into an explicit protocol rather than a guess.
The extractor periodically syncs its bitmap and then writes markers for a group of completed files. The ordering matters: a marker must never claim completion for bits that have not been made durable.
Configuration files contain KEY=value lines. Useful standard-library tools are:
var lines = std.mem.splitScalar(u8, contents, '\n');
while (lines.next()) |raw_line| {
const line = std.mem.trim(u8, raw_line, " \t\r");
if (line.len == 0 or line[0] == '#') continue;
const equals = std.mem.indexOfScalar(u8, line, '=') orelse
return error.InvalidLine;
const key = std.mem.trim(u8, line[0..equals], " \t");
const value = std.mem.trim(u8, line[equals + 1 ..], " \t");
_ = key;
_ = value;
}Other frequently useful operations:
std.mem.eql(u8, left, right)
std.mem.startsWith(u8, value, "Bearer ")
std.mem.endsWith(u8, path, ".csv")
std.mem.indexOf(u8, body, "reason=")
std.mem.indexOfScalar(u8, field, 0)Formatting options cover stack buffers, allocated strings, and writers:
var buffer: [64]u8 = undefined;
const text = try std.fmt.bufPrint(&buffer, "rows={d}", .{rows});
const owned = try std.fmt.allocPrint(allocator, "{s}.done", .{path});
defer allocator.free(owned);
try writer.print("phone={d}\n", .{phone});Prefer bufPrint for small bounded values and allocPrint when the required size
is naturally dynamic.
src/phones/normalize.zig demonstrates a valuable design: isolate complicated
validation in a pure function with no allocation and no I/O.
A simplified normalizer:
fn normalizeUsPhone(raw: []const u8) ?u64 {
var digits: [11]u8 = undefined;
var count: usize = 0;
for (raw) |byte| {
switch (byte) {
'0'...'9' => {
if (count == digits.len) return null;
digits[count] = byte;
count += 1;
},
' ', '-', '(', ')', '.', '+' => {},
else => return null,
}
}
const ten = switch (count) {
10 => digits[0..10],
11 => if (digits[0] == '1') digits[1..11] else return null,
else => return null,
};
// The real implementation also validates NANP area/exchange rules.
var value: u64 = 0;
for (ten) |digit| value = value * 10 + (digit - '0');
return value;
}Benefits of pure normalization:
- very fast per row;
- easy exhaustive unit tests;
- no allocator failure path;
- reusable by extraction, DNC parsing, and enrichment;
- invalid input represented separately from operational errors.
There are ten billion possible ten-digit strings, but many are invalid under the North American Numbering Plan. The project maps valid area/exchange/subscriber parts into a compact continuous range and uses one bit per possible number.
Core bitset operations look like this:
fn setBit(bytes: []u8, bit_index: u64) bool {
const byte_index: usize = @intCast(bit_index / 8);
const shift: u3 = @intCast(bit_index % 8);
const mask: u8 = @as(u8, 1) << shift;
const was_set = (bytes[byte_index] & mask) != 0;
bytes[byte_index] |= mask;
return !was_set;
}
fn isSet(bytes: []const u8, bit_index: u64) bool {
const byte_index: usize = @intCast(bit_index / 8);
const shift: u3 = @intCast(bit_index % 8);
return (bytes[byte_index] & (@as(u8, 1) << shift)) != 0;
}The shift amount for an 8-bit value is a u3, because only values 0 through 7
are meaningful. Zig's precise integer widths make representation constraints
visible.
One bit per domain value is deterministic and dramatically smaller than a hash table entry containing a phone, hash metadata, and spare capacity. The tradeoff is that the full domain-sized file exists even when relatively few bits are set.
The phone bitmap and enrichment lookup use std.posix.mmap. A memory mapping
lets ordinary loads and stores access a file-backed region:
const Mapping = struct {
io: std.Io,
file: std.Io.File,
bytes: []align(std.heap.page_size_min) u8,
fn deinit(self: *Mapping) void {
std.posix.munmap(self.bytes);
self.file.close(self.io);
self.* = undefined;
}
};The real implementation:
- opens or creates the file;
- validates or sets its exact length;
- maps it with appropriate protection and sharing flags;
- uses
errdeferso a failure after opening does not leak the file; - calls
msyncat checkpoint boundaries; - unmaps before closing.
Mapped slices have page alignment in their type:
[]align(std.heap.page_size_min) u8This is more precise than []u8 and documents a requirement imposed by the OS.
Writes to a shared mapping eventually reach the file, but restart guarantees require explicit synchronization and carefully ordered checkpoint markers. Also validate file lengths before mapping; a truncated backing file can cause a fault when a mapped page is accessed.
Deduplicated phones and lookup offsets are stored as fixed-width integers. A portable binary format chooses a byte order explicitly:
fn writeU64Le(writer: *std.Io.Writer, value: u64) !void {
var bytes: [8]u8 = undefined;
std.mem.writeInt(u64, &bytes, value, .little);
try writer.writeAll(&bytes);
}
fn readU64Le(bytes: *const [8]u8) u64 {
return std.mem.readInt(u64, bytes, .little);
}Never serialize a native struct by dumping its memory unless the format is explicitly machine-local and you accept padding, alignment, endianness, and Zig layout changes. Stable files should contain explicit fields, version metadata, and exact integer encodings.
The local DNC lookup combines:
- a sorted array of phones;
- an offset array;
- a variable-length record blob;
- a small exchange-range index;
- metadata describing version and counts.
The offset pair offsets[i]..offsets[i + 1] identifies record i without parsing
all earlier records.
Because deduplicated phones are sorted, lookup uses binary search:
fn findPhone(phones: []const u64, needle: u64) ?usize {
var low: usize = 0;
var high: usize = phones.len;
while (low < high) {
const middle = low + (high - low) / 2;
const value = phones[middle];
if (value < needle) {
low = middle + 1;
} else {
high = middle;
}
}
return if (low < phones.len and phones[low] == needle) low else null;
}The actual lookup first narrows the search to a phone exchange range. This is a small secondary index: a cheap prefix calculation yields a smaller interval, then binary search finds the exact phone.
This design gives predictable memory use and avoids loading a huge hash table.
The project uses SHA-256 for two related but distinct purposes:
- content checksums for result chunks;
- deterministic fingerprints of an input inventory and its metadata.
One-shot hashing:
var digest: [std.crypto.hash.sha2.Sha256.digest_length]u8 = undefined;
std.crypto.hash.sha2.Sha256.hash(data, &digest, .{});
const hex = std.fmt.bytesToHex(digest, .lower);
try writer.print("sha256={s}\n", .{&hex});Incremental hashing is preferable for large streams:
var hasher = std.crypto.hash.sha2.Sha256.init(.{});
hasher.update(chunk);
// Repeat for every chunk.
hasher.final(&digest);An inventory fingerprint should hash an unambiguous encoding: include path lengths or separators, file sizes, modification times, and a stable sorted order. Otherwise different inventories could accidentally produce the same byte stream before hashing.
CSV is deceptively complex: commas and newlines may occur inside quoted fields, quotes are escaped by doubling, files may start with a BOM, and rows may be malformed. Splitting each line on commas is not a CSV parser.
The project uses the zig_csv_parser dependency in streaming mode:
var parser = try csv.StreamingBorrowedParser.init(allocator, .{
.allow_bom = true,
.skip_empty_rows = true,
.ragged_row_policy = .error_on_ragged,
.expected_fields = expected_fields,
.max_row_bytes = 64 * 1024,
});
defer parser.deinit();
while (true) {
const read_length = try reader.readSliceShort(&buffer);
if (read_length == 0) break;
try parser.feed(
buffer[0..read_length],
Context,
&context,
Context.onRow,
);
}
try parser.finish(Context, &context, Context.onRow);The feed API belongs to the pinned parser dependency; recheck
src/dnc/response.zig and src/enrich/csv_transform.zig when upgrading it.
The callback receives a borrowed row:
fn onRow(self: *Context, row: csv.StreamingBorrowedRow) !void {
if (row.len() != self.expected_fields) return error.WrongFieldCount;
const phone_field = try row.field(0);
// Consume now, or duplicate before returning.
_ = phone_field;
}A field must be quoted if it contains a comma, quote, carriage return, or line feed. Embedded quotes become two quotes:
fn writeCsvField(writer: *std.Io.Writer, field: []const u8) !void {
const needs_quotes = std.mem.indexOfAny(u8, field, ",\"\r\n") != null;
if (!needs_quotes) return writer.writeAll(field);
try writer.writeByte('"');
for (field) |byte| {
if (byte == '"') try writer.writeByte('"');
try writer.writeByte(byte);
}
try writer.writeByte('"');
}The normalization stage writes CL365_Phone as the first field and preserves
the original row bytes where possible. That avoids reformatting every original
column unnecessarily.
Google Drive returns JSON. Zig can parse directly into a type matching the fields the application needs:
const FileList = struct {
files: []const RemoteFile = &.{},
nextPageToken: ?[]const u8 = null,
};
const RemoteFile = struct {
id: []const u8,
name: []const u8,
mimeType: []const u8,
size: ?[]const u8 = null,
md5Checksum: ?[]const u8 = null,
};
var parsed = try std.json.parseFromSlice(
FileList,
allocator,
body,
.{ .ignore_unknown_fields = true },
);
defer parsed.deinit();The parsed strings are owned by or borrow from the parse result depending on options and API details. In this project, remote file models that outlive the parse result explicitly duplicate their fields.
Ignoring unknown fields makes the client tolerant of additional server fields, while the typed struct still requires and validates the data the application uses.
For a bounded request payload, an allocating writer is convenient:
var output: std.Io.Writer.Allocating = .init(allocator);
errdefer output.deinit();
try output.writer.writeAll("{\"phones\":[");
for (phones, 0..) |phone, index| {
if (index != 0) try output.writer.writeByte(',');
try output.writer.print("\"{d}\"", .{phone});
}
try output.writer.writeAll("]}");
const body = try output.toOwnedSlice();
defer allocator.free(body);For arbitrary strings, use a JSON writer or correct escaping rather than manual concatenation. Numeric phone strings need a much smaller escape surface.
The Drive API uses convenient fetch-style requests for metadata. Large file downloads use a lower-level request so response data can stream directly to disk.
Conceptual POST flow:
var client: std.http.Client = .{ .allocator = allocator, .io = io };
defer client.deinit();
const uri = try std.Uri.parse(endpoint);
var request = try client.request(.POST, uri, .{
.keep_alive = false,
.headers = .{
.content_type = .{ .override = "application/json" },
},
.extra_headers = &.{
.{ .name = "loginId", .value = login_id },
},
});
defer request.deinit();
request.transfer_encoding = .{ .content_length = request_body.len };
var body_writer = try request.sendBodyUnflushed(&.{});
try body_writer.writer.writeAll(request_body);
try body_writer.end();
try request.connection.?.flush();
var response = try request.receiveHead(&.{});
var transfer_buffer: [4096]u8 = undefined;
const body = try response.reader(&transfer_buffer).allocRemaining(
allocator,
.limited(max_response_bytes),
);
defer allocator.free(body);Consult src/dnc/client.zig for the exact Zig 0.16 request calls and buffer setup.
Do not reduce all non-2xx responses to one error. The client classifies them:
- success: parse and store the response;
- rate-limited or temporary server error: retry after a delay;
- authentication/configuration error: fail with a useful message;
- permanent request error: record it without endless retries;
- provider-specific automation block: stop a download batch and cool down.
Capture a bounded response-body prefix in diagnostics. It is often the only clue why a provider rejected a request, but never log credentials.
Drive search expressions and query parameters must be encoded. Encode values,
not an entire already-structured URL. Keep reserved separators such as ?, &,
and = under the URL builder's control.
True partial-file resumption is a protocol between local state and the server:
- inspect the
.partfile length; - send
Range: bytes=N-; - require an HTTP partial-content response;
- parse
Content-Rangeand ensure its start equalsN; - seek the local file to
N; - append the response while updating progress;
- validate final size and, where available, checksum;
- sync and atomically rename.
If the server ignores Range and returns the complete object, appending it would
corrupt the file. The correct response is to restart safely or return an error,
not to assume.
Remote identity also matters. A partial file for one Drive object revision should not be resumed against unrelated content with the same local filename.
DNC chunks and final enriched CSV files are gzip-compressed with Zig's standard flate implementation:
var compression_buffer: [std.compress.flate.max_window_len]u8 = undefined;
var compressor = try std.compress.flate.Compress.init(
&file_writer.interface,
&compression_buffer,
.gzip,
.fastest,
);
try compressor.writer.writeAll(csv_bytes);
try compressor.finish();
try file_writer.interface.flush();
try file.sync(io);finish is required: it emits the final compressed blocks and gzip trailer.
Flushing only the outer file writer is not a substitute.
For decompression, wrap the compressed reader and stream or copy the decompressed bytes into a bounded destination. The lookup builder rejects a chunk that exceeds its configured decompressed maximum.
Compression level is a pipeline tradeoff. .fastest spends less CPU and still
reduces CSV size substantially; that often matters more than squeezing out the
smallest possible archive.
Rate limiting must use a monotonic clock, not wall-clock time. Wall time can jump because of synchronization or administrator changes.
The project uses the awake monotonic clock:
const now = std.Io.Clock.awake.now(io);
const elapsed_ns = started.durationTo(now).toNanoseconds();At 100 requests per minute, the average start interval is:
const minute_ns: u64 = 60 * std.time.ns_per_s;
const interval_ns = minute_ns / 100; // 600 msA start scheduler reserves request slots. Concurrency does not remove the rate limit; it hides response latency while start times remain controlled.
Use sleep through the I/O capability:
const deadline = std.Io.Clock.Timestamp.now(io, .awake).addDuration(.{
.clock = .awake,
.raw = .fromNanoseconds(wait_ns),
});
try deadline.wait(io);The scheduler also uses saturating arithmetic such as +|= where an overflowing
statistic should clamp rather than wrap:
total_latency_ns +|= sample_latency_ns;Retries should not hammer a service at a fixed cadence. Exponential backoff grows the delay after repeated failures:
fn backoffNs(attempt: u5) u64 {
const capped = @min(attempt, 6);
return std.time.ns_per_s * (@as(u64, 1) << capped);
}Add random jitter so many workers or processes do not retry simultaneously:
var random_value: u32 = undefined;
io.random(@ptrCast(&random_value));
const extra_ns: u64 = random_value % (std.time.ns_per_s + 1);
const delay_ns = base_delay_ns +| extra_ns;Honor a valid Retry-After response when the provider supplies one. Cap both
maximum delay and maximum resource usage even if the higher-level policy retries
indefinitely.
Retry only errors likely to improve with time. Bad credentials and malformed requests need correction, not patience.
The DNC scrubber keeps several requests in flight and reacts to whichever one finishes first. A simplified shape is:
const Completion = union(enum) {
request: RequestResult,
};
var result_buffer: [hard_max_concurrency]Completion = undefined;
var select = std.Io.Select(Completion).init(io, &result_buffer);
errdefer drainRequestSelect(&select);
while (have_work or in_flight != 0) {
while (have_work and in_flight < concurrency_limit) {
try rate_limiter.waitForStart();
select.concurrent(.request, performRequest, .{
client,
owned_batch,
});
in_flight += 1;
}
const completed = try select.await();
switch (completed) {
.request => |result| {
in_flight -= 1;
try handleResult(result);
},
}
}See src/phone_scrub.zig for the exact API and fixed result-storage setup.
Important ownership rule: data passed to a concurrent operation must remain alive until that operation finishes or is cancelled and drained. The scrubber transfers ownership of the phone batch into the request operation and releases it only after receiving the result.
If a timeout wins a race, the HTTP operation may still own a response or allocated batch. The client cancels or drains the losing operation and deinitializes any late successful value. Otherwise timeouts would leak memory and connections.
An unlimited number of in-flight requests can exhaust:
- sockets and file descriptors;
- memory for request and response buffers;
- provider connection limits;
- useful work when latency suddenly increases.
The project estimates latency and adjusts a bounded concurrency window to stay near the allowed request-start rate. Rate and concurrency are separate controls.
Interactive terminals and log files need different behavior. The downloader checks whether stderr is a TTY:
const live = std.Io.File.stderr().isTty(io) catch false;For a TTY, it refreshes a single line several times per second using carriage return and ANSI clear-line sequences. For redirected logs, it emits a normal line only every few seconds.
This distinction avoids producing enormous logs while preserving responsive interactive progress.
Speed calculations widen the multiplication:
const bytes_per_second: u64 = @intCast(@min(
@as(u128, bytes) * std.time.ns_per_s / @as(u128, elapsed_ns),
std.math.maxInt(u64),
));Subtraction can saturate at zero with -|, useful when counters may be reset
between attempts:
const attempt_bytes = current_bytes -| attempt_start_bytes;src/config/env_file.zig loads a bounded .env file, parses owned key/value
pairs, and lets the process environment override file values. This precedence is
useful for production:
command environment > .env file > built-in default
Do not commit API logins or access tokens. Keep secrets out of:
- source code;
- generated metadata;
- command diagnostics;
- HTTP response dumps;
- test fixtures committed to Git.
An owning environment map must duplicate strings if its source buffer will be freed. Parsing slices directly from one loaded buffer is also valid if the owning configuration keeps that buffer alive for at least as long as all slices.
The most important lesson is architectural rather than syntactic: each stage has a bounded working set and a durable boundary.
Drive files
-> normalized CSV files + phone bitmap
-> sorted deduplicated phone file
-> gzip DNC result chunks
-> memory-mapped lookup files
-> gzip enriched CSV files
Every source row must already be read to add CL365_Phone. Setting the phone's
bitmap bit during the same pass makes deduplication nearly free and avoids a
second scan of tens of gigabytes.
They preserve original rows while adding a stable join key. Final enrichment no longer needs to rediscover which original column contains the phone or repeat normalization rules.
Network calls are expensive, slow, and failure-prone. Durable result chunks decouple them from local enrichment, which can then be rerun without billing or network access.
The raw response chunks are convenient append-only recovery artifacts. The memory-mapped sorted lookup is convenient for millions of random joins. One format does not need to serve both purposes.
Returning only error.InvalidCsv is insufficient for an unattended long run.
The DNC response parser fills a diagnostic struct:
const Diagnostic = struct {
parse_error: ?ParseError = null,
csv_error_row: ?usize = null,
csv_error_column: ?usize = null,
csv_error_byte_offset: ?usize = null,
field_count_value: ?usize = null,
result_code_byte: ?u8 = null,
phone_digits: ?usize = null,
};The function still returns a typed error, while the diagnostic adds context for logs and debugging. This avoids building human-readable text deep inside parser logic and lets callers choose how much safe detail to expose.
test "normalizes common phone forms" {
const cases = [_]struct {
input: []const u8,
expected: ?u64,
}{
.{ .input = "(312) 555-0100", .expected = 3_125_550_100 },
.{ .input = "+1 312 555 0100", .expected = 3_125_550_100 },
.{ .input = "not a phone", .expected = null },
};
for (cases) |case| {
try std.testing.expectEqual(case.expected, normalizeUsPhone(case.input));
}
}try std.testing.expectError(
error.InvalidTimeout,
parseTimeout("tomorrow"),
);Use a temporary directory for file-format and restart tests. Write a fixture,
open or build the real object, assert contents, then let the test fixture clean
up. Avoid tests that depend on the production data/ directory.
src/dnc/client.zig starts a loopback server with the standard HTTP server API,
runs it concurrently, and points the real client at it. This verifies:
- request method and headers;
- request body framing;
- response parsing;
- status classification;
- timeout and cleanup behavior.
It is much more reliable than calling the real paid provider from a unit test.
The generic enrichment transformer receives a fake lookup in tests. This checks CSV output ordering and missing/invalid paths without constructing the large production index.
Zig discovers tests in analyzed modules. A root test can ensure imported module tests are included:
test {
_ = @import("phones/normalize.zig");
_ = @import("dnc/scheduler.zig");
_ = @import("enrich/lookup.zig");
}Use std.debug.assert for programmer invariants:
std.debug.assert(bit_index < bitmap_bit_count);Use returned errors for invalid external input or operational failures:
if (file_size != expected_size) return error.InvalidBitmapSize;An assertion says, “correct code cannot reach this state.” An error says, “the world may legitimately provide this bad state, and the caller must handle it.”
Validate at boundaries:
- CLI values when parsed;
- file version and size when opened;
- CSV field count per row;
- HTTP status before parsing the body as success;
- range start before appending;
- checksums before accepting final output;
- lookup offsets before slicing mapped data.
Early validation keeps internal code simpler because it can rely on established invariants.
| Syntax | Meaning | Project use |
|---|---|---|
| `+ | /+ |
=` |
| `- | ` | saturating subtraction |
@min, @max |
choose numeric bound | capped backoff/concurrency |
@intCast |
checked integer conversion under safe modes | indices and sizes |
@as |
explicit type coercion | widen before arithmetic |
@intFromEnum |
enum tag to integer | response field index |
@TypeOf |
obtain expression type | generic transform context |
@This |
current containing type | callbacks and fake objects |
@fieldParentPtr |
recover outer struct from field pointer | writer adapter |
@alignCast |
assert/restore pointer alignment | writer adapter, mmap |
@errorName |
error value to static name | CLI diagnostics |
Safety checks depend on optimization mode. Do not use unchecked arithmetic merely because production uses a release build; choose wrapping or saturating operators only where their semantics match the domain.
Use a streaming reader and streaming CSV parser. Bound row and response sizes.
Estimate memory first. A domain bitmap is smaller and has predictable access.
Duplicate fields that outlive a parser callback or parse result.
Use an errdefer immediately after each successful acquisition.
Durability order is part of correctness. Sync data, then publish its marker.
Validate 206 Partial Content and Content-Range before appending.
Classify authentication, malformed requests, throttling, transient server errors, and permanent failures separately.
Rate controls how often work starts. Concurrency controls how much work can be in flight. Use both.
finish writes the final gzip state. A mere outer flush can leave an invalid
archive.
Sort before fingerprinting or generating deterministic output.
Write fixed-width fields with explicit endianness and version metadata.
Read these modules in order:
src/phones/normalize.zig— pure functions, optionals, arrays, tests.src/config/env_file.zig— ownership, parsing, errors, cleanup.src/main.zig— process entry and command dispatch.src/phones/bitmap.zig— file-backed data and bit operations.src/phone_extract.zig— inventories, checkpointing, large-data orchestration.src/drive/model.zig— owning structs and clone/deinit discipline.src/drive/api.zig— HTTP metadata calls, JSON, pagination, backoff.src/drive/transfer.zig— streaming downloads and resumption.src/drive/progress.zig— clocks and custom writer interfaces.src/dnc/response.zig— arena ownership and CSV diagnostics.src/dnc/client.zig— HTTP POST, timeout races, local server tests.src/dnc/scheduler.zig— rate and adaptive concurrency arithmetic.src/phone_scrub.zig— completion-driven orchestration.src/dnc/storage.zig— compression, hashes, and atomic files.src/enrich/lookup.zig— binary indexes, mmap, checkpoints.src/enrich/csv_transform.zig— comptime generic callbacks and joins.src/phone_enrich.zig— final pipeline orchestration and gzip output.
For each file, identify:
- which values it owns;
- which slices are borrowed;
- which functions can allocate;
- every
deferanderrdeferpair; - how external data is bounded and validated;
- what remains valid after interruption.
- Add table-driven tests for every punctuation form accepted by phone normalization and every invalid NANP area/exchange prefix.
- Write a tiny Zig program that creates a one-megabit mapped bitmap, sets random bits, syncs it, reopens it, and verifies them.
- Extend a marker format with an output SHA-256 and reject mismatches on resume.
- Build a local HTTP server test that returns
429withRetry-After, then assert the client classifies it as retryable. - Implement a writer adapter that hashes all bytes while forwarding them to a file, similar to the progress writer.
- Generate a small sorted phone/offset/blob lookup and query it with binary search.
- Interrupt an extraction test after a checkpoint, restart it, and assert that completed rows are neither lost nor double-counted.
- Compare bitmap memory and estimated hash-table memory for 1 million, 10 million, and 100 million unique phones.
- Add a new optional DNC response field while keeping older lookup versions readable, forcing an explicit file-format migration decision.
- Replace a manual JSON fragment with the standard JSON writer and test quote, slash, control-character, and Unicode escaping.
This quick index connects imports to their role in the project:
| Module or type | What it provides here |
|---|---|
std.mem |
slices, equality/search, sorting, integer encoding |
std.mem.Allocator |
explicit dynamic-memory ownership |
std.heap.ArenaAllocator |
batch-lifetime DNC response storage |
std.ArrayList |
growable inventories and parsed rows |
std.fmt |
integer parsing, formatted paths, output, hex encoding |
std.process |
entry initialization, arguments, environment, fatal exit |
std.Io |
filesystem, streams, clocks, sleep, randomness, concurrency |
std.Io.Reader / Writer |
bounded streaming and composable adapters |
std.Io.Dir / File |
traversal, metadata, seek, sync, rename |
std.fs.path |
portable path joins and parent-directory extraction |
std.posix |
mmap, msync, and munmap |
std.http |
Drive and DNC clients plus loopback test server |
std.Uri |
checked endpoint parsing |
std.json |
typed Drive response parsing |
std.compress.flate |
gzip result and enriched-output compression |
std.crypto.hash.sha2 |
SHA-256 content and inventory fingerprints |
std.Io.Clock |
monotonic timestamps, deadlines, and durations |
std.Io.Select |
completion-driven requests and timeout races |
std.math |
checked arithmetic, integer bounds, min/max helpers |
std.ascii |
case-insensitive HTTP header comparisons |
std.debug |
assertions and immediate terminal progress |
std.log |
categorized operational messages |
std.testing |
assertions, leak-checking allocator, temporary fixtures |
zig_csv_parser |
bounded streaming CSV parser and row diagnostics |
std.debug.print is appropriate for the downloader's actively rewritten progress
line. Prefer std.log.info, warn, and err for ordinary operational messages
that may be redirected and filtered.
The most useful way to think about Zig in this project is:
- a slice is a view, so always ask who owns the bytes;
- an allocator is a visible dependency, so always ask who frees the result;
- an error is a typed value, so decide which layer handles it;
deferprotects an acquired resource anderrdeferprotects partial construction;- a tagged union makes state and its payload agree;
comptimespecializes code without hiding runtime allocation;- streaming and bounded reads turn dataset size into elapsed time rather than memory consumption;
- durable files require ordering, synchronization, identity checks, and versions;
- concurrency requires explicit ownership and cleanup for both winners and losers;
- data representation is an algorithmic choice: bitmaps, sorted arrays, offsets, and mmap make this workload practical.
That combination—explicit resources, precise types, and deliberately designed on-disk state—is the core of both Zig and this pipeline.