Design a recursive distributed file crawler

Design a distributed system that walks a file hierarchy starting from a given root directory. A client can call an HTTP/gRPC endpoint to initiate a crawl job; traversal happens asynchronously in background workers rather than in the request path. While processing one directory, a worker may split its direct children into smaller work units and push new jobs into a queue. Those queued jobs can themselves enqueue more jobs, enabling the crawl to expand recursively without…