Back to problems

Web URL Crawler at Scale

Algorithm · Snowflake · Hard

Scalable Web URL Crawler Coding Software Engineer Problem Overview The following helper interfaces are provided: fetch_page(url) -> str retrieves the contents of a page. parse_urls(content) -> list[str] extracts the links found in that content. Create a traversal approach and explain how it should behave when requests fail. The interview is divided into three stages: Visit every reachable URL. Explain or build an approach suitable for very large crawls. Make the crawler…

Checking your access…