4 Commits

Author	SHA1	Message	Date
Awni Hannun	6fd1f70f73	fix spm decoder multi-byte (#1092)	2024-11-05 06:06:26 -08:00
Awni Hannun	0f799947d0	fix (#1079)	2024-11-01 16:30:32 -07:00
Awni Hannun	8fe9539af7	Fix detokenizer space match for quote (#1072) ... * fix + test * remove transformer flax/torch warning * format	2024-10-27 15:06:07 -07:00
Awni Hannun	8dca1a2f60	Tokenizer updates + tests (#1024) ... * tokenizer updates + tests * nit * add can_trim_prompt_cache * nits	2024-10-14 10:48:46 -07:00