chore: bump tokenizers from 0.21.4 to 0.23.1 - #6

chore: bump tokenizers from 0.21.4 to 0.23.1#6

dependabot[bot] wants to merge 1 commit into

masterAyyankhan101/Transformer-In-Rust:masterfrom

dependabot/cargo/tokenizers-0.23.1Ayyankhan101/Transformer-In-Rust:dependabot/cargo/tokenizers-0.23.1Copy head branch name to clipboard