Hello guys !
I have made a Byte Level Tokenizer to convert big corpus of text into tokens to train. The compression is about ~1.3x because i convert every values to 16bits ( float ) to make the training easier.
Any help is welcome !
https://github.com/StarlightShade15/ByteLevelTokenizer
0 likes · 1 comments