Design Soft 404 Detection

1. Framing the Problem Search engines constantly run into pages that pretend to be valid when they are not. The server answers with HTTP 200, meaning success, but the page itself says "item not found," shows an empty product list, or renders a generic placeholder. These soft 404s waste crawl capacity and fill the index with pages that contain no useful content. The task is to build a classifier that detects them during the crawl, before they enter the index. The scope covers…