Back to problems

Implement robust word counts and min/max

Algorithm · Amazon · Hard

Suppose a 50 GB UTF-8 text corpus is stored on local disk. Write a Python program that reads the file incrementally, never holding the entire contents in memory at once. The program should count word frequencies case-insensitively after normalizing Unicode (such as stripping accents), removing punctuation, and making e-mail equivalent to email. It must also define and follow explicit tokenization rules for hyphens, apostrophes, and emojis. A supplied stopword list with…

Checking your access…