Data collection is a natural HHY workload.
A collector combines seeds, HTTP, retries, timeouts, concurrency, parsing, normalization, deduplication, failure records, and persistence. It brings HHY's Flow, Stream, Effect, and resource-boundary ideas into one real task.
I did not want to begin with “Scrapy for HHY.”
Scrapy carries mature URL scheduling, middleware, downloading, item pipelines, caching, robots.txt, and an extension ecosystem. Playwright adds browser processes, JavaScript, page lifecycle, and interactive state. Promising all of that in v0.1 would expand the language runtime, extension protocol, and framework API simultaneously, making it difficult to know which layer was actually stable.
I chose a narrower definition:
HHY Collector Framework is a Flow-first, auditable, resource-bounded data collector for APIs and static documents.
The first boundary: collector, not browser
The current version targets JSON APIs, CSV/JSON datasets, and static HTML. It does not execute JavaScript, handle login challenges, bypass robots.txt, authentication, or anti-bot controls, and it does not promise arbitrary response streams.
That is an architectural boundary rather than a deferred-feature disclaimer. HHY currently buffers an HTTP response body under max_http_body; parallel(n) is bounded and ordered by default. Those behaviors give documentation pages and ordinary APIs clear resource ceilings. They are not the right model for video, archives, enormous NDJSON, or browser-rendered applications.
A later Browser Worker can remain external. HHY can own plans, resources, data, and failures while a browser owns page execution. The runtime does not need to become a browser.
Do not disguise a DOM parser as Regex
HTTP was not the missing capability. Structured HTML was.
split, replace, and PCRE2 are useful for regular text, but they should not become a general HTML extraction layer. Real pages need error-tolerant parsing, nesting, entities, attributes, and selector semantics.
I implemented the official html process extension using Lexbor's HTML5 parser and CSS selectors. A schema looks like this:
{
"root_selector": "section.card",
"max_results": 10,
"schema": {
"title": {
"selector": "h2",
"value": "text",
"required": true
},
"anchor": {
"selector": "h2",
"value": "attr",
"name": "id"
}
}
}
The engine sends HTML, a root selector, the schema, and a result limit in one call:
html.extract(
fetched.value,
root_selector,
schema,
{ max_results: max_results }
)
The extension returns an ordinary List<Map> that can rejoin an HHY Stream.
This API is deliberate. Process Extension Protocol values are Null, Bool, numbers, String, List, and Map. It cannot carry an opaque DOM handle or Stream. A browser-like parse → node handle → select → text interface would first require new protocol and lifetime semantics.
For v0.1, “HTML + Selector + Schema → List<Map>” keeps the DOM inside the extension process. It gives up arbitrary traversal in exchange for simple ownership, explicit serialization, and an immediately testable application API. Real projects can tell us whether a Node logical value eventually belongs in Core.
The engine has one main path
crawler.hhy owns arguments, configuration, statistics, and three atomic outputs. lib/engine.hhy owns execution:
export fn crawl(config) {
let user_agent = config.user_agent
let root_selector = config.root_selector
let schema = config.schema
let max_results = config.max_results
return config.seeds
|> stream
|> distinct
|> parallel(config.parallelism) { url ->
crawl_seed(url, user_agent, root_selector, schema, max_results)
}
|> collect
}
Seeds are deduplicated before entering configured bounded concurrency. HTTP uses an identifiable User-Agent, a ten-second timeout, two retries, and 500 ms backoff. There are no unlimited workers or hidden global pools.
Each seed has two attempt boundaries: fetch and HTML extraction.
let fetched = attempt { fetch_page(url, user_agent) }
if fetched.ok != true {
return { ok: false, url: url, records: [], error: fetched.error.message }
}
let parsed = attempt {
html.extract(fetched.value, root_selector, schema, { max_results: max_results })
}
if parsed.ok != true {
return { ok: false, url: url, records: [], error: parsed.error.message }
}
Network and schema failures become normal results without erasing peer seeds. Every successful record receives source_url; collected data without provenance is difficult to audit, update, or correct.
Three outputs carry three responsibilities
The program writes:
records.jsonfor extracted records;report.jsonfor requested, successful, failed, and record counts;failures.jsonfor failed URLs and structured errors.
All three use atomic writes. Successful records remain useful when some pages fail, and failure details do not disappear into stderr. At the same time, report.ok becomes false and the process exits 1, so automation cannot mistake partial output for complete success.
This is better suited to collection than stopping at the first failure, and more honest than always returning zero.
Configuration is the first Spider language
The default job fetches https://hhylang.dev/zh/learn/cli-reference and extracts main article h2 text plus IDs:
{
"project": "HHY Documentation Crawler",
"seeds": ["https://hhylang.dev/zh/learn/cli-reference"],
"parallelism": 2,
"user_agent": "HHY-Collector/1.0 (+https://hhylang.dev)",
"root_selector": "main article h2",
"max_results": 100,
"schema": {
"title": { "selector": "", "value": "text" },
"anchor": { "selector": "", "value": "attr", "name": "id" }
}
}
An empty selector reads the record root. Fields support text, html, and attr, plus all. max_results crosses the configuration and extension boundary, preventing a mistaken selector from returning thousands of nodes to the runtime.
The Spider is currently pure data. I deliberately avoided function references in JSON because it cannot reliably express executable closures. Pagination, follow requests, and normalize hooks should eventually come from .hhy Spider modules rather than code strings inside configuration.
Extension installation should stay project-local
init.sh installs the official html extension into the project's .hhy-extensions. It is idempotent and leaves the user's global extension home untouched.
make
./practical-projects/my-crawler/init.sh
./practical-projects/my-crawler/run.sh
Execution explicitly sets HHY_EXTENSION_HOME, allowing the project to carry its expected extension version rather than depending on something globally installed on one developer's machine.
Regression tests should not depend on the public internet
The live task proves that HHY can collect a real documentation site. The self-test proves the code contract.
self-test.sh starts a local fixture, creates configuration dynamically, selects two section.card records, and asserts titles First item and Second item, anchors one and two, local provenance URLs, two report records, and an empty failure list.
./practical-projects/my-crawler/self-test.sh
A public redesign should not turn the framework regression suite red. Conversely, a passing fixture does not prove that a live selector still matches. Smoke tests and deterministic tests provide different evidence.
It is not yet the complete framework I imagined
The implementation proves the minimum loop—Seed → Fetch → Extract → Provenance → Output—but does not yet include:
- pagination and follow requests;
- URL canonicalization and relative resolution;
- per-domain admission and robots.txt;
- request fingerprints and a visited set;
- incremental checkpoints and failure replay;
- conditional requests, cache, and content hashes;
- JSON, CSV, and database storage adapters;
- unordered completion such as
parallel(..., { ordered: false }).
Most of these can be written in HHY without immediately changing the runtime. Real Spiders should drive priority. I would add pagination, request identity, checkpoints, and domain admission first, then measure whether ordered parallelism is a genuine throughput bottleneck.
What the implementation reinforced
A framework's value does not come from having the most concepts. It comes from making failure, resources, and effects visible.
HHY Collector Framework currently has one engine, one declarative schema, and one HTML extension. Yet every request has a timeout, every batch has bounded concurrency, every extraction has a result ceiling, every record has provenance, every run has a failure audit, and every file is written atomically.
Those boundaries matter more than starting with dozens of middleware names.
I did not begin by cloning Scrapy because HHY does not need to imitate another ecosystem to justify itself. It should first prove that a collection flow can remain direct while staying dependable around networks, parsing, and filesystem effects.