Back to problems

Web Crawler Implementation

Object-Oriented Programming · Anthropic · Medium

Problem Write a basic web crawler in Python. It receives one starting URL, retrieves that page, and recursively visits accessible pages linked from it. For every fetched page, obtain the text inside its element and identify every link that belongs to the starting domain. The crawler must accept a configurable depth limit and must not request the same URL more than once. Return the collected page information, including each page's URL, title, and extracted internal links.…

Checking your access…