String Tokenization

Implement the function tokenize(s, vocab), where s is the string to segment and vocab is an array of entries in the form " : ". Each is a token, and is the identifier returned when that token is matched. At each index of s, inspect every vocabulary key that starts at that index. If at least one key matches, choose the longest matching key, append its to the result, and move past the matched characters. If no key matches, append the current character as a literal and move…