tiktoken_ruby
An unofficial Ruby wrapper for Tiktoken, a BPE tokenizer written by and used by OpenAI. It can be used to count the number of tokens in text before sending it to OpenAI APIs.
0
users added this gem as favorite
RubyGems
tiktoken_ruby
GitHub Repository
IAPark/tiktoken_ruby
GitHub Stars
166
RubyGems Downloads
6,191,695
Current Version
0.0.15.1
License
MIT
Number of issues
1