ETL is a useful test of whether a language's data model survives beyond syntax examples.
A real pipeline rarely starts with one tidy JSON array. Inputs arrive as CSV, directories, and HTTP responses. Fields need normalization, records become ineligible, remote enrichment fails, and outputs must serve both humans and downstream programs.
DataFlow ETL reads a customer CSV and a directory of event JSON, normalizes fields, excludes inactive customers, enriches profiles concurrently, applies a minimum-spend rule, aggregates departments, and atomically writes JSON plus CSV.
Turn each source into explicit data
CSV ingestion owns whitespace cleanup, lowercasing, boolean conversion, and integer parsing:
return path_join(input_root, "customers.csv")
|> read_lines
|> parse_csv({ header: true })
|> map { row ->
return {
id: trim(row.id),
email: lower(trim(row.email)),
active: lower(trim(row.active)) == "true",
spend: to_int(row.spend)
}
}
|> collect
Events take a different path through files("*.json"). I did not build a universal loader: the formats differ, but both source functions return Maps the business pipeline can understand.
Enrich concurrently without erasing failure
Inactive customers are filtered before HTTP work. Eligible customers enter parallel(4); each request has a three-second timeout and one retry.
let profile = attempt {
http.get(join([api_base, "profiles", customer.id], "/"))
|> timeout(3s)
|> retry({ count: 1, backoff: 50ms })
|> send
|> response_body
|> parse_json
}
A failed enrichment does not make the customer disappear. The output retains the original fields and adds enriched: false plus an error. The report is successful only when every qualified customer was enriched. That is more honest than silently dropping records and gives an operator something actionable to retry.
Streams express relationships; arrays mark boundaries
Qualification is a short pipeline:
customers
|> stream
|> where { customer -> customer.spend >= minimum_spend }
|> sort_by({ order: "desc" }) { customer -> customer.spend }
|> collect
Department totals use group_by and sum. I keep data in a Stream while it is being transformed, then collect where concurrency, reuse, or report ownership needs a concrete array. That makes lifetime visible and avoids consuming a single-pass Stream twice.
Two outputs, one business path
JSON retains customers, events, aggregates, and source statistics for diagnosis. CSV projects only seven stable downstream columns. Both writes are atomic, so a failed run does not leave half of a new file over the previous good result.
The deterministic test verifies three qualified customers, two event files, and two departments, along with email cleanup, spend ordering, remote region/tier values, and both output formats.
sh practical-projects/dataflow-etl/self-test.sh
The lesson from this flow
ETL complexity usually sits at boundaries rather than inside one map: when cleaning occurs, whether failures remain visible, how remote calls are bounded, and whether output replacement is safe.
That is also where HHY earns its place. Pipes keep direction clear, Streams express filtering and grouping, attempt turns failure into data, and the runtime provides shared timeout and atomic-effect semantics. The result is not merely SQL-like syntax. It is a pipeline that can run, fail, explain itself, and run again.