Explain handling very large datasets

Walk through a project where you ingested and processed a dataset of at least 500 million rows or 1 TB from start to finish. Cover the storage formats and partitioning, memory and compute constraints, schema evolution, data quality checks, indexing strategies, and the tools you selected (for example, Spark SQL, Pandas, or BigQuery) and why. Report the before and after runtimes and costs, along with one code-level optimization you made (such as vectorization, predicate pushdown, window functions, or bucketing). If you were restricted to a single machine with 32 GB of RAM, how would your approach change?

Overview: This question assesses a candidate's ability to ingest and process very large datasets, spanning storage formats and partitioning, memory and compute constraints, schema evolution, data quality checks, indexing strategies, SQL/Python tool selection, and code-level performance optimizations.