ahref

Open source · Rust · CLI. This case examines published source revision 06879810. Commands refer to the crawler, not the earlier HTML parsing library.

From one page's links to site structure

ahref began as a small link parser. I rebuilt it as a CLI that crawls internal HTML pages and saves a directed graph. The problem had changed: a list of href values cannot explain how a visitor reaches an article, which pages exist only in a sitemap, or which links lead to an observed HTTP error.

I designed crawling, the observation model, metrics and exports. The result needed to support repeated analysis and other tools. Network collection therefore ends at a saved graph, independent of its visualization. Page history and comparisons with search traffic remain responsibilities of SEO Version Control and SEO Agent.

Find a page known only from a sitemap

For my own website, I can start at the sitemap, bound the crawl and save a graph. Subsequent commands work with the local file:

ahref crawl https://opatsay.com/sitemap.xml --max-pages 200 --concurrency 2 --output graph.json
ahref analyze graph.json --output analyzed.json
ahref export analyzed.json --format dot --output site.dot

While preparing this case, I checked the workflow on a separate three-page HTTP fixture. The homepage linked to /guide, which linked back; /orphan appeared only in the sitemap. The crawler fetched three pages and recorded two hyperlinks. The orphan had depth: null, zero incoming degree and an orphan-candidate classification. Offline analysis and DOT export succeeded with the same saved file.

This was a synthetic check, not an opatsay.com audit. On a real site, the next step is to inspect the candidate and decide whether a contextual link belongs there. Sitemap membership alone does not make a page reachable through navigation.

Three choices that determine what a report means

Separate crawl depth from homepage distance

A sitemap seeds the queue, but its URLs do not automatically receive depth zero in the report. Final depth is recalculated from the homepage using links and redirects. Otherwise an orphan would appear as accessible as the homepage. With an incomplete crawl, missing incoming links remain an observation rather than proof about the entire site.

Keep links, redirects and failures distinct

The graph preserves individual hyperlink occurrences with anchors and rel; degree and PageRank use unique neighbors. A redirect is separate metadata, not an invented hyperlink. A broken destination requires an observed HTTP error status. Timeouts, robots-blocked pages and pending URLs have different states so downstream automation can distinguish why information is incomplete.

Bound network work while retaining a usable result

The page budget includes errors and redirect requests. Concurrency and decoded body sizes are bounded; each redirect destination must pass scope and robots checks. HTML bodies are discarded after parsing, leaving graph records and metadata. A successful CLI exit means that a report was produced, not that every known page was fetched successfully.

A result that can be checked

On revision 06879810, cargo test --locked --test crawl ran ten integration tests; all passed. They start a loopback fixture and check graph behavior, request bounds, failures, sitemaps and exports. This is reproducible behavioral evidence without public websites, not a performance benchmark.

The repository exposes tests for sitemap orphan depth, redirects and terminal errors, machine-readable stdout, and the graph schema. Saving JSON and separating human output onto stderr makes the crawler usable within another collector.

Analysis boundaries

The crawler does not execute JavaScript, check external-link status or discover sitemaps automatically. Supply the sitemap URL when it matters. Internal PageRank describes the observed graph, includes nofollow links and does not reproduce Google's ranking. Query order and values, HTTP versus HTTPS and trailing slashes retain separate identities. That may increase graph size, but avoids merging potentially different resources.

Build the current crawler

Use Rust 1.90 or newer. Installing from the checkout avoids confusing the crawler with older parser-only packages:

git clone https://github.com/tenqz/ahref.git
cd ahref
cargo install --path . --locked
ahref --version

Start with a modest budget on your own site, save JSON and inspect warnings before drawing conclusions. The ahref article develops the analysis workflow; the architecture document explains queues, memory and the graph model.