2 Commits

Author	SHA1	Message	Date
Awni Hannun	8fe9539af7	Fix detokenizer space match for quote (#1072) ... * fix + test * remove transformer flax/torch warning * format	2024-10-27 15:06:07 -07:00
Awni Hannun	8dca1a2f60	Tokenizer updates + tests (#1024) ... * tokenizer updates + tests * nit * add can_trim_prompt_cache * nits	2024-10-14 10:48:46 -07:00