The first ahref solved a narrow problem: extract link tags and URLs from HTML. A list of links on one page cannot explain a whole website, though. Which articles are known only through a sitemap? How many steps separate a page from the homepage? Which pages receive links from different parts of the site? I rebuilt ahref as a Rust CLI crawler that produces a directed graph.
Nodes represent normalized URLs. Edges represent observed a href links, preserving anchor text and rel. Once saved, the graph can be analyzed again without making network requests. The crawler needs no account, API key, or LLM connection.
Build the current implementation
git clone https://github.com/tenqz/ahref.git
cd ahref
cargo install --path . --locked
ahref --versionThe checkout requires Rust 1.90 or newer. Installing from source here is deliberate: older published ahref versions were HTML parsing helpers. A local 1.0.0 build does not by itself establish that the same version is published to a package registry. These CLI commands describe the rebuilt crawler.
Crawl once and save the graph
ahref crawl https://example.com/sitemap.xml --max-pages 1000 --concurrency 3 --output graph.json
ahref analyze graph.json
ahref export graph.json --format graphml --output site.graphmlInput can be a page URL, a sitemap, a sitemap index, or a compressed .xml.gz sitemap. With sitemap input, its URLs and the origin homepage seed the crawl. Links in fetched HTML can discover additional pages. Sitemap discovery is not automatic, so provide one explicitly when investigating orphan candidates.
--max-pages limits page request work, including redirects and HTTP errors. --concurrency caps simultaneous requests. Robots rules are respected by default. Starting with modest concurrency makes an initial crawl easier on a small server.
Read the graph as an observation
Incoming degree counts distinct source pages. Repeated links remain visible as separate occurrences, but they do not become distinct neighbors. Ten copies of a link on one page should not look like ten independent sources of navigation.
Depth describes a path from the homepage through observed links. Being listed in a sitemap does not make a page depth zero. If no path from the homepage is found, depth remains null. A sitemap URL without an incoming link from another internal URL is an orphan candidate. That finding needs particular care when the crawl stops at its budget.
Internal PageRank describes link distribution within this graph. It uses unique internal neighbors and includes nofollow hyperlinks. It is not Google’s score and does not predict a search position. It is useful as a structural measure alongside depth and the role of each page.
Do not manufacture certainty from failed requests
Broken-link findings require an observed HTTP status of 400 or greater, directly or through a checked redirect chain. Timeouts, unvisited pages, and external links are not automatically broken. Redirects have separate metadata rather than being invented as HTML links.
A dead end must be a successfully fetched HTML page without outgoing internal hyperlinks. A blocked or unfetched URL does not satisfy that definition. These distinctions keep a report from presenting unknown states as confident findings.
Export without another crawl
JSON is useful for a custom pipeline, GraphML for Gephi, and DOT for Graphviz. analyze recomputes metrics from the stored graph. export changes its representation. Both work offline, so a different visualization does not require revisiting the website.
The crawler does not execute JavaScript or check external link status. It does not automatically merge a www hostname with the apex domain. HTTP and HTTPS, trailing slashes, and query parameters retain their distinctions because a server may treat those URLs differently.
Combine structure with search evidence
ahref answers structural questions. GSC MCP and Webmaster MCP answer questions about search-system reports. ahref itself does not maintain history, integrate GSC, or generate SEO recommendations. A separate collector can preserve graphs and compare them with dated search observations.
That boundary keeps the crawler reusable. A person can inspect its output in Gephi, a script can compare nodes, and an agent can read a derived report. They all start from the same recorded links instead of a recommendation whose evidence has disappeared.