Crawler, Typeahead or Ranking: Which Search Design Will You Get?

As of 2026-10-19, TrueInterview's bank holds 25 candidate-reported search and crawling design questions from 16 big-tech companies. The most-linked titles are Design Distributed Web Crawler, with 10 candidate write-ups, and Design Typehead Suggestion, with 6. Our take: search design is three interviews that share a name. A crawler pipeline, a typeahead serving index and a search ranking stack need different first diagrams, and the company you are interviewing at tells you which one to prepare more reliably than a generic design-Google-Search outline does.

Disclosure: TrueInterview is an interview-preparation product and publishes this article. Facts about other products come from their public pages on the dates listed under Sources.

Which search, typeahead and crawler questions do candidates report?

The titles below lead the write-up links, and the list is current: 22 of the 25 questions were last reported in the 12 months before 2026-10-19. If you do not yet know your company's lean, learn the crawler and typeahead rows first, because they carry the most links. Otherwise use the company table further down to choose your first row.

Read Design Distributed Web Crawler as a pipeline problem: a frontier of URLs waiting to be fetched, a pool of fetchers, a check for URLs already seen and a check for duplicate content. The decisions to prepare for at senior level are how the frontier stays polite to each host and how it survives a crash, and the two crawler sections below take them in turn.

Design Typehead Suggestion is the serving-side counterpart, a prefix index that has to answer on every keystroke. Prepare to defend how suggestions are re-ranked when popularity shifts and how you split the hot short prefixes that every user types first.

Facebook Search (Mini Elasticsearch) names its architecture in the title. Treat it as an inverted index with a query path: documents tokenised into posting lists, posting lists sharded, each query fanned out and the ranked partial results merged. The trade-offs to defend are how quickly a new post becomes searchable and whether you partition the index by document or by term.

The ranking rows and the Uber Eats row are different problems again, and they get their own section below. These are candidate-reported questions reconstructed for practice, not any company's official question list. To see the reports behind a title, open the candidate write-ups for your target company.

Is your prompt a crawler, a typeahead, an index or a ranking question?

Classify the prompt from its title, then check whether it carries an ML framing. By subtype the 25 questions split into 12 ML system design, 12 distributed systems and 1 frontend system design question, so an ML framing can apply even when the title reads like infrastructure. Each class below starts from a different diagram, so pick the class before the first box.

By problem family, assigned from the titles, 12 questions are search, autocomplete and crawling, 4 are maps, ride-hailing and delivery, 3 are ads, recommendation and ranking ML, and the rest sit in smaller groups such as news feed and LLM infrastructure. The table maps the reported titles to the core you should draw first and the decisions to rehearse.

Our take: the usual advice to prepare one search-engine walkthrough under-serves this family. A crawl, index and serve outline answers the first three rows and says almost nothing about the last two, which cover the 4 maps and delivery titles and the 3 ranking-ML titles counted above. If the title names ranking, personalisation or availability, a crawler diagram spends your opening minutes on the wrong system.

Which track should you prepare for your target company?

Choose the track from your company's filings first. Meta and Amazon each file 3 search, autocomplete and crawling questions, while Airbnb files 2 ranking-ML questions and Uber files none in the crawling family. Spend your first week on the track your company files, and treat the rest of the family as stretch work.

The three columns do not cover every filed question, so a company's row can sum to less than its total in the paragraph below; the remainder sits in smaller families outside this choice.

The cross-company titles sharpen the split. Design A Personalized Search Ranking System is filed under Instacart, Airbnb, DoorDash, Robinhood and Uber. Design Distributed Web Crawler is filed under OpenAI, Databricks, Lyft, Meta, Microsoft and Pinterest, and Design Typehead Suggestion under LinkedIn, Amazon, Google, Meta and Pinterest. Those two are the designs to carry from one infrastructure-leaning loop to the next.

Keep the counts in proportion. Airbnb and Meta file 5 questions each in this family, Amazon 4, and Uber, Pinterest and Google 3 each. A company's lean is a direction for your first week of preparation, not a forecast of the prompt you will get.

Decision rule (our take): at Meta, Amazon or Google, prepare the infrastructure track first, with the crawler, typeahead and an inverted-index search such as Facebook Search (Mini Elasticsearch). At Airbnb, Instacart, DoorDash or Uber, prepare personalised ranking and geo search first. For OpenAI, Lyft, Pinterest, Microsoft or Databricks, make the distributed crawler your first full design. In every case, give at least half of your practice time to requirements and follow-up trade-offs.

How do you scope a distributed web crawler?

Ask what the crawl is for before you size anything. Google's reported title is Web Crawler at Search Scale, last reported onsite in 2026-08, while Hello Interview's breakdown assumes the company "needs this data to be available for training 5 days after you start crawling". Those two purposes share components and optimise for different things, so name yours first.

Hello Interview's breakdown says that, whatever the use case, the interview "is likely to focus on the crawling task". ByteByteGo's chapter says "It is unlikely for anyone to design a massive web crawler within the interview duration". ByteByteGo also warns that even for a crawler, you and your interviewer "might not have the same assumptions". Read together, the guides treat scoping as part of the answer, so budget real minutes for it.

What the purpose changes (our take). A search-index crawl lives on freshness: the frontier needs a recrawl priority that brings back pages which change, and dedup protects index quality. A training-data crawl lives on throughput against a deadline: the frontier can run closer to a single pass, dedup protects the dataset, and text extraction matters more than recrawl. Say which one you are building and let it set the frontier's ordering.

The components themselves are table stakes. ByteByteGo calls the store of URLs to download the URL Frontier and says you can treat it as a first-in-first-out queue. Hello Interview's breakdown says the queue could use "Kafka, Redis, or SQS". ByteByteGo names a Bloom filter and a hash table as common techniques for the URL Seen? check, and recommends comparing page hash values to detect duplicate content. On choosing seed URLs, ByteByteGo's chapter says "Just think out loud".

Our take on the seen-URL check: at web scale, pick the Bloom filter and accept that a false positive skips a page, because the memory saving decides whether the check fits on each crawler node. Choose an exact hash store only when a skipped page has a cost you can name in the requirements.

Treat the guides' scale figures as teaching assumptions. Hello Interview assumes 10B pages at an average of 2MB per page. ByteByteGo assumes 1 billion pages downloaded a month and 30 PB of storage to keep five years of content. Neither set is a reported interview requirement, so ask your interviewer for the real target. Hello Interview's breakdown also lists JavaScript-rendered dynamic content as out of scope; name your own exclusions the same way so the interviewer knows what you chose to leave out.

Which crawler requirements should you state before drawing boxes?

State politeness and fault tolerance in the first requirements pass. Hello Interview lists politeness, meaning adhering to robots.txt and not overloading servers, and fault tolerance, meaning resuming without losing progress, as core non-functional requirements. Each one shapes the frontier, so stating them late means redrawing the design you have already defended.

ByteByteGo's chapter lists the same pair in its own words. It says a polite crawler "should not make too many requests to a website within a short time interval". ByteByteGo also says a good crawler must handle bad HTML, unresponsive servers, crashes and malicious links.

Google's documentation says a robots.txt file tells crawlers which URLs they can access, and that it is used "mainly to avoid overloading your site with requests". Google also says robots.txt instructions cannot enforce crawler behaviour and that obeying them is up to the crawler. That puts the obligation inside your design: fetch and cache each host's robots.txt, and check it before every fetch. Google adds that it might still "find and index a disallowed URL if it is linked from other places on the web". If your crawl feeds an index, decide how a linked but disallowed URL is stored without its content.

How to make both requirements concrete (our take): give the frontier one queue per host with a next-allowed-fetch time, so politeness becomes a scheduling rule rather than a sleep inside the fetcher. Persist the frontier and the seen-URL set, so a crashed worker's URLs return to the queue and a restart resumes from the last checkpoint instead of the seed list. Key stored pages on a URL hash, so a retried fetch overwrites its earlier copy instead of duplicating it.

Trie, inverted index or prefix cache: how should you build typeahead?

Start from a trie with precomputed top-k lists, then spend most of the round on freshness and skew. Databricks files Design a Typeahead Search System, last reported onsite in 2026-09. In our view the follow-ups, rather than the trie, decide this round.

PracHub's guide sets the order of work: "Start with matching semantics, define what top-k means, and then allocate the latency budget". PracHub's guide labels its 100 ms p99 target a teaching assumption, so ask for the real budget instead of quoting that figure.

The baseline in PracHub's guide stores the best k completions at each prefix node, so lookup reads a bounded list after traversal. PracHub's guide also says reaching the prefix subtree "does not by itself identify the best suggestions". PracHub adds that per-prefix lists grow storage because one suggestion sits in several ancestor lists, and recommends term IDs instead of full strings.

Which structure wins (our take): for a query-completion box, build the trie offline and serve a flattened prefix-to-top-k map from memory. The trie is the build tool and the map is the serving index. Reach for an inverted index with a suggester when the prompt needs matches inside a phrase or typo tolerance, and accept a slower path for it. Elasticsearch's reference says its term suggester works from edit distance. Elasticsearch also recommends low max_errors values, warning that otherwise "the time spend in suggest calls might exceed the time spend in query execution".

The follow-up to rehearse first is lowering a score. PracHub's guide says "Increasing one term's score is often easier than decreasing the current winner: a short top-k list may lack the next replacement". PracHub's defensible baseline rebuilds an immutable ranked index from a durable event stream and periodically swaps a versioned snapshot. Explain both halves out loud: a decay or a removal can leave a prefix's list short, and the snapshot rebuild is how you refill it without locking readers.

Then rehearse the skew follow-up. PracHub's guide says partitioning by prefix narrows routing "but short popular prefixes create skew". Our answer is to replicate and cache the shortest prefixes on every serving node, since they are few and hot, and to partition only the longer prefixes.

bugfree.ai's Medium post, which describes one author's outline rather than a reported interview, treats autocomplete as a read and search submission as a write that "updates the query count and helps rebuild the search Trie for future queries". Keep those two paths apart in your diagram: the submission stream feeds the rebuild, and the keystroke path never writes.

Two further points are worth raising (our take). On the client, PracHub's guide notes that debouncing "reduces request volume but does not establish response order", and recommends a monotonically increasing sequence number for each input generation so stale responses lose. On eligibility, PracHub's guide warns that falling back to unfiltered public results is inappropriate when access permissions determine eligibility. Raise that point yourself for any workspace or enterprise search box.

When is a search question really an ML ranking or geo question?

When the title names ranking, personalisation, availability or a location. The ranking family includes Search / Ranking / Experimentation (Amazon, Pinterest), Design A Personalized Search Ranking System (Instacart, Airbnb) and Airbnb's Home Page — Search + Availability + Ranking. Answer those as retrieval-then-ranking designs, and settle recall before you discuss the model.

PracHub's autocomplete guide states the order plainly: "Increasing the ranking model's sophistication does not help when retrieval has discarded the winner". The same rule carries over to search ranking. Design the candidate generator first, with its recall target, then the ranker, then the experiment that shows the ranker helped.

The decisions to defend (our take): which ranking features must be fresh at query time and which can come from a batch job, how offline metrics map to the online test, and where hard constraints apply. In a search-plus-availability prompt, availability is a hard filter, so apply it before ranking instead of asking the model to learn it.

The geo titles are a separate class. The maps, ride-hailing and delivery family includes Uber Eats Search, Meta's Design a location-based radius top-K search, Uber's Design a Maps Address Search Bar and Airbnb's Query System — Time + Geo Filtered User Activity. Uber Eats Search was last reported onsite in 2026-05. For these, draw a spatial index, such as geohash cells or a quadtree, that narrows candidates by distance, then apply text relevance and business filters. Expect the follow-ups (our take) to land on dense cells in city centres and on how you merge distance with relevance in one score.

Search also turns up inside other design prompts. Oracle's Video Upload + Search with LLM was reported in a phone screen in 2026-02. In our view, search there is one component of a larger system, and a short retrieval sketch is enough while the upload pipeline takes most of the time.

Worked example: how would you prepare a Meta onsite?

Build the plan from Meta's own filings. Meta files 5 questions in this family; 3 are search, autocomplete and crawling, 1 is maps and delivery, none are ranking ML, and the remaining one sits in another family. That points to the infrastructure track, with one inverted-index design, the crawler and typeahead as the core and one geo prompt as the stretch.

Start with Facebook Search (Mini Elasticsearch), last reported onsite in 2026-08 and linked only from senior-or-above write-ups. Sketch the inverted index, the sharding scheme and the query fan-out, and defend how a new post becomes searchable within the freshness target you set in requirements.

Next, the crawler, which is filed under Meta among its six companies. Open by asking what the crawl is for, state politeness and fault tolerance as requirements, and sketch the frontier, the fetcher pool, the seen-URL check and page-hash dedup from the crawler sections above.

Then typeahead, which Meta files as Design Typehead Suggestion. Work from matching semantics to top-k, then defend score decay and the split of hot short prefixes. Keep Design a location-based radius top-K search as the stretch: it is filed under Meta and has a staff-plus write-up.

Compare an Airbnb loop. Airbnb files 2 ranking-ML questions in this family, and its dated onsite example is Home Page — Search + Availability + Ranking, last reported in 2026-03. The same three weeks would go to a ranking stack with availability filtering, plus one geo design, and the crawler would drop to a single review session.

Should you expect these in the phone screen or the onsite?

Expect the onsite. Of the 25 questions, 21 have been reported in an onsite round and 6 in a phone screen, and a question can appear in both. Of the 21 onsite questions, 11 fall in the search, autocomplete and crawling family. Practise these designs at onsite length, with a full requirements pass, rather than as a quick sketch for a screen.

Our take: the onsite placement changes how you practise. A phone screen rewards a fast, correct sketch, while an onsite design round leaves the interviewer time to push on the second and third decision. That is where the crawler's politeness and the typeahead's score updates come in. Rehearse each design against a timer with a written list of follow-ups, and answer them out loud.

What changes at senior and staff level?

The level signal is strong. By candidates' self-reported levels, write-ups link these questions 31 times from senior, staff-plus or manager candidates and 5 times from junior, new-grad or intern candidates. If you are senior or above, expect to be judged on requirements and trade-offs, since a correct component list is only the baseline here.

The pattern holds title by title. Design Distributed Web Crawler has 8 senior-or-above links, 1 of them staff-plus or manager, against 2 junior-or-below links. Design Typehead Suggestion has 5 senior-or-above links, 1 of them staff-plus or manager, against 1. Staff-plus or manager write-ups also link Design a location-based radius top-K search.

The interviewer-authored guides name the failure. Hello Interview's author says defining good non-functional requirements matters "especially for more senior candidates". The same Hello Interview author describes candidates who "rush through the requirements" and then stall after a simple design.

Our take: at senior and staff level, depth on requirements and trade-offs is worth more than another memorised title. A senior answer states the crawl's purpose, politeness and fault tolerance before drawing, and explains how the design shifts with scale and freshness. A staff answer adds the operating model: how you would roll out a new ranked snapshot, how you would detect a politeness regression against one host, and what you would cut if the deadline halved.

Down-leveling note (our view): in this family the level signal sits in the design round's follow-ups. A candidate who draws a clean crawler but cannot say how the frontier resumes after a crash, or a clean trie but cannot lower a winner's score, reads as mid-level even with a correct diagram. If you are targeting staff, rehearse the purpose question and the trade-off defence until both come out unprompted.

How we counted

Counts come from TrueInterview's question bank as of 2026-10-19 and cover 25 candidate-reported search and crawling design questions at 16 big-tech companies. A question can be filed under several companies and reported in more than one round, so company and round tallies overlap. Families are assigned from question titles, level splits count write-up links by the level each candidate reported, and a link count measures what candidates wrote up, not how often a company asks a question. Scale and latency figures from Hello Interview, ByteByteGo and PracHub are teaching assumptions, not reported requirements.

FAQ

Is the web crawler a coding question or a system design question?

Both exist, and they belong to different rounds. TrueInterview's catalog lists an Anthropic Web Crawler as a coding question reported in phone-screen and onsite rounds, last reported in 2026-05 at junior and senior levels. TrueInterview's catalog tags it async, concurrency, Python and threading. Design Distributed Web Crawler is the system design version covered above. If a recruiter mentions a web crawler, ask whether the round is coding or design, because the coding version wants working concurrent code and the design version wants an architecture.

Does robots.txt stop a crawler from fetching a page?

No. Beyond the point above that the file cannot enforce behaviour, Google's documentation says robots.txt rules may not be supported by all search engines. Google also says "Different crawlers interpret syntax differently". In an interview, treat obeying robots.txt as your crawler's job: fetch and cache the file per host, refresh it on a schedule, and check it before every fetch. Google's robots.txt introduction was last updated on 2025-12-10, so it is a current reference to cite.

Should you use Elasticsearch's suggesters for autocomplete in a design interview?

Use the term and phrase suggesters when the prompt needs typo tolerance or phrase correction, because those features target correction rather than prefix completion. Elasticsearch's reference says its phrase suggester selects entire corrected phrases weighted by n-gram language models. Elasticsearch also says the phrase suggester "requires special mapping up front to work", and that suggest across indices with different analysis chains is not supported. Our take: for a plain prefix box, the in-memory prefix map from the typeahead section is the simpler design to defend.

When should you do back-of-the-envelope estimates for a crawler?

When a decision depends on them. Hello Interview's breakdown says that for a crawler, scale "will have a large impact on the database design and the overall architecture". Hello Interview's author says to delay the calculations "until they are necessary for solving a specific problem". Ask for the page count and the deadline during requirements, then compute storage or fetch throughput at the moment you choose the page store or size the fetcher pool.

Which search design should you practise first if you do not know your company's lean?

The distributed crawler. Design Distributed Web Crawler is linked from 10 candidate write-ups, 8 of them by senior-or-above candidates, the highest count in this family's table above. Its purpose question, politeness rules and crash-safe frontier also give you the requirements habits that the typeahead and inverted-index prompts reward. Once it is solid, add typeahead, then one ranking or geo variant.

Your practice plan

A three-week plan for someone working full time:

1. First week, first session. Look up your target company's filings in the question bank, sort them into search and crawling, maps and delivery, and ranking ML using the company table above, and pick the infrastructure or ranking track.
2. First week, remaining sessions. Write one full design for the most-linked title in your track: Design Distributed Web Crawler for infrastructure, or Search / Ranking / Experimentation for ranking.
3. Second week, first half. Infrastructure track: take the crawler end to end, then rewrite its opening requirements twice, once for Web Crawler at Search Scale and once for a training-data crawl, rehearsing politeness, crash recovery and page-hash dedup aloud. Ranking track: design Design A Personalized Search Ranking System retrieval first, then the ranker, then the experiment.
4. Second week, second half. Infrastructure track: design Facebook Search (Mini Elasticsearch) as an inverted index with sharding and query fan-out, and state how quickly a new document becomes searchable. Ranking track: design Home Page search with availability and ranking with availability applied as a hard filter before ranking.
5. Third week, first half. Infrastructure track: work Design Typehead Suggestion, then Design a Typeahead Search System, rehearsing the follow-ups out loud: lowering a current winner's score, swapping a versioned snapshot and splitting hot short prefixes. Ranking track: design Uber Eats Search as a spatial index plus relevance and filters.
6. Third week, second half. Infrastructure track: do one ranking or geo variant from your company's filings at full onsite length. Ranking track: give the distributed crawler a single review session, stating its purpose, politeness and crash recovery, then re-run the ranking design you narrated least cleanly.
7. Before the onsite. TrueInterview says it does not sell mock interviews with human engineers and suggests booking one or two elsewhere before an onsite. Spend that session on the design from your track that you trust least.

Sources

• TrueInterview question bank — search and crawling design at big-tech companies — counted 2026-10-19
• Design a Web Crawler | Hello Interview System Design in a Hurry — checked 2026-10-19
• ByteByteGo | Technical Interview Prep — checked 2026-10-19
• Robots.txt Introduction and Guide | Google Search Central | Documentation | Google for Developers — checked 2026-10-19
• Search Autocomplete System Design Interview: Tries, Ranking, and Low Latency | PracHub Knowledge Hub — checked 2026-10-19
• Suggester examples | Elasticsearch Reference — checked 2026-10-19
• Medium — checked 2026-10-19
• Coding Interview Questions by Company · TrueInterview — checked 2026-10-19
• Interview prep compared: your application, end to end · TrueInterview — checked 2026-10-19

Last reviewed: 2026-10-19.