Longest-Match Tokenizer

You are given a dictionary vocab that maps strings to integer token IDs. The dictionary always contains the special key "UNK" whose value is -1; this key is a sentinel for unknown characters and must never be used as a matchable token. Implement tokenize(text, vocab) that converts text into a list of token IDs using the following greedy longest-match rule: Start at index 0. At the current position, look at all prefixes of the remaining text. Choose the longest prefix that is…