“Collect Hong Kong film companies” sounds like a small crawler exercise. In practice, HTTP quickly gives way to research questions: Is a search result actually a company? What should tests do when external data changes? Should one failed page discard the batch? How does output retain provenance?
This project uses the official MediaWiki API. It searches for pages related to “香港電影公司”, concurrently fetches page IDs, canonical URLs, timestamps, and introductions, applies an explainable company filter, then atomically writes CSV and JSON.
Search and detail are separate stages
Search discovers candidates; it is not final data. The flow retrieves query.search, deduplicates by pageid, and requests details through bounded parallel work.
return search_results
|> stream
|> group_by { result -> result.pageid }
|> map { group -> group.values[0] }
|> parallel(parallelism) { result ->
let detail = attempt { /* MediaWiki detail request */ }
// return success or structured failure
}
|> collect
Requests have a ten-second timeout, two retries, and 250 ms backoff. attempt isolates each page, so a timeout on page seven does not erase six completed results.
Introductions are capped at 600 characters. This is not merely bandwidth thrift; it bounds the value carried back from each worker. Unbounded external responses can turn concurrency throughput into memory pressure.
The filter admits that it is a heuristic
The final rule requires the introduction to mention Hong Kong, film, and a company, then sorts by title. It is explainable and testable, but it is not a corporate registry.
On August 26, 2026, the real run retained seven of ten candidates, including individual companies and the Hong Kong film-company list. I would rather state that boundary explicitly than present search output as an authoritative directory.
CSV keeps the canonical Wikipedia URL, page ID, timestamp, and introduction so a reader can verify the source. Republishing introductions also requires attention to the licenses and attribution identified by their pages.
Dynamic research and regression testing are different evidence
Wikipedia changes and networks fluctuate. A test that insists there are still seven results tomorrow mostly tests the internet, not the program.
self-test.sh therefore starts a local MediaWiki fixture. Five fixed candidates pass through the same deduplication, concurrent fetch, and semantic filter and must produce three companies. Assertions cover CSV fields, ordering, URLs, and JSON statistics.
sh practical-projects/hong-kong-film-companies/self-test.sh
The live run answers “what can I find now?” The fixture answers “does the program still honor its contract?” Both matter, but they should not be confused.
One compatibility detail
The Chinese search term is percent-encoded in configuration because HHY 1.1.1 still has a limitation around non-ASCII query Map values. ASCII detail parameters continue to use the structured query Map. I prefer documenting that boundary over hiding it behind an implicit workaround.
This is what practical projects add to a language roadmap: real APIs, real characters, and real failure modes turn a feature list into engineering work.