Guides HTML Processing
Extract links from a webpage using HTMLRewriter
Extract links from a webpage
Bun's HTMLRewriter API extracts links from HTML. Chain CSS selectors to match the elements, text, and attributes you want to process, then pass .transform a Response, ArrayBuffer, or string.
async function extractLinks(url: string) {
const links = new Set<string>();
const response = await fetch(url);
const rewriter = new HTMLRewriter().on("a[href]", {
element(el) {
const href = el.getAttribute("href");
if (href) {
links.add(href);
}
},
});
// Wait for the response to be processed
await rewriter.transform(response).blob();
console.log([...links]); // ["https://bun.com", "/docs", ...]
}
// Extract all links from the Bun website
await extractLinks("https://bun.com");Convert relative URLs to absolute
When scraping websites, you often want to convert relative URLs (like /docs) to absolute URLs:
async function extractLinksFromURL(url: string) {
const response = await fetch(url);
const links = new Set<string>();
const rewriter = new HTMLRewriter().on("a[href]", {
element(el) {
const href = el.getAttribute("href");
if (href) {
// Convert relative URLs to absolute
try {
const absoluteURL = new URL(href, url).href;
links.add(absoluteURL);
} catch {
links.add(href);
}
}
},
});
// Wait for the response to be processed
await rewriter.transform(response).blob();
return [...links];
}
const websiteLinks = await extractLinksFromURL("https://example.com");See HTMLRewriter.