System Design · Google · Hard
Design a Web Crawler Hard · Topics · Company Tags · Hints Your task is to design a distributed web crawler that can fetch and process tens of billions of documents, revisit them on a regular schedule, and feed parsed content into a search engine. The crawler only needs to handle plain HTTP responses; executing JavaScript is out of scope. The core pipeline forms a closed loop: a URL frontier schedules downloads; a fetcher retrieves pages; a parser pulls out text and outbound…
Checking your access…