Commits

Branch selector

 main

User selector

All users

All time

Commit History

Commits on Jan 13, 2026

• 同步包装器

  Huiyicc

  committed
  e450f25

  Copy full SHA for e450f25

Commits on Jan 12, 2026

• 同步项目

  Huiyicc

  committed
  fee663d

  Copy full SHA for fee663d

• Merge remote-tracking branch 'upstream/main'

  Show description for ce763aa
  Huiyicc

  committed
  ce763aa

  Copy full SHA for ce763aa

• 更新依赖

  Huiyicc

  committed
  3567204

  Copy full SHA for 3567204

Commits on Aug 8, 2025

• [Fix] Make raw pointer deref explicit to satisfy dangerous_implicit_autorefs lint (#87)

  Show description for 55d53aa
  MasterJH5574

  authored
  55d53aa

  Copy full SHA for 55d53aa

Commits on Jul 29, 2025

• add windows on arm support (#86)

  Darce-One

  authored
  b7f7763

  Copy full SHA for b7f7763

Commits on Jun 25, 2025

• [Rust] Bump huggingface tokenizer to 0.21.2 (#81)

  Show description for f777109
  MasterJH5574

  authored
  f777109

  Copy full SHA for f777109

Commits on Jun 24, 2025

• feat: fix aHashmap typebinding error to tokenizer (#79)

  joeldushouyu

  authored
  e896487

  Copy full SHA for e896487

Commits on Jun 17, 2025

• Avoid loading `added_tokens` when this variable is empty (#77)

  Show description for 5f68f07
  hebangwen

  authored
  5f68f07

  Copy full SHA for 5f68f07

Commits on May 31, 2025

• [Rust] Bump huggingface tokenizer to 0.21.1 (#73)

  Show description for aae1209
  MasterJH5574

  authored
  aae1209

  Copy full SHA for aae1209

Commits on May 17, 2025

• Add build option for mt/md (#1) (#70)

  jhlee525

  authored
  4fbe996

  Copy full SHA for 4fbe996

Commits on May 15, 2025

• Fixed compilation for aarch64 (#66)

  Show description for ae3d206
  NaveenMittal0

  authored
  ae3d206

  Copy full SHA for ae3d206

Commits on May 3, 2025

• [Web] Bump web-tokenizer to 0.1.6 (#67)

  Show description for db3f72c
  CharlieFRuan

  authored
  db3f72c

  Copy full SHA for db3f72c

Commits on Apr 15, 2025

• add support for Xcode and MacOS (#65)

  Darce-One

  authored
  dc7ea3a

  Copy full SHA for dc7ea3a

Commits on Feb 24, 2025

• Add a way to force the macos target in case it is set (#52)

  ykhrustalev

  authored
  832b872

  Copy full SHA for 832b872

• updating tokenizer version from 0.20.0 to 0.21.0 (#62)

  NaveenMittal0

  authored
  f290eb5

  Copy full SHA for f290eb5

• Fix for loading tokenizers using non-utf8 strings (#55)

  Show description for c290994
  ThomasProg

  authored
  c290994

  Copy full SHA for c290994

Commits on Feb 6, 2025

• Bump CMake minimum to v3.19 (#59)

  afuller-TT

  authored
  0621f84

  Copy full SHA for 0621f84

Commits on Dec 10, 2024

• 更新安卓构建

  Huiyicc

  committed
  383bc91

  Copy full SHA for 383bc91

Commits on Dec 9, 2024

• 同步linux编译

  Show description for 0da0415
  Huiyicc

  committed
  0da0415

  Copy full SHA for 0da0415

Commits on Dec 7, 2024

• fix:android build

  Huiyicc

  committed
  b321fd3

  Copy full SHA for b321fd3

Commits on Nov 16, 2024

• 更新encode_ex

  Huiyicc

  committed
  6b452c8

  Copy full SHA for 6b452c8

Commits on Nov 15, 2024

• [Rust] Bump huggingface tokenizer to 0.20.0 (#49)

  Show description for 4bb7533
  MasterJH5574

  authored
  4bb7533

  Copy full SHA for 4bb7533

Commits on Nov 14, 2024

• [CMake] Enable SentencePiece tokenizer by default (#47)

  Show description for 528d695
  MasterJH5574

  authored
  528d695

  Copy full SHA for 528d695

Commits on Aug 12, 2024

• [Web] Bump version to 0.1.5

  CharlieFRuan

  committed
  5de6f65

  Copy full SHA for 5de6f65

• [Web] Set TOKENIZERS_PARALLELISM to false for HFTokenizer (#42)

  Show description for d5dd7f4
  CharlieFRuan

  authored
  d5dd7f4

  Copy full SHA for d5dd7f4

Commits on Jun 27, 2024

• [CMake] Support disable SentencePiece tokenizer (#38)

  Show description for c0fab1e
  MasterJH5574

  authored
  c0fab1e

  Copy full SHA for c0fab1e

Commits on Jun 3, 2024

• [Web] Bump version to 0.1.4

  CharlieFRuan

  committed
  eec9d68

  Copy full SHA for eec9d68

Commits on Jun 2, 2024

• [Rust] Bump HuggingFace tokenizer version (#36)

  Show description for a52e33e
  MasterJH5574

  authored
  a52e33e

  Copy full SHA for a52e33e

Commits on May 29, 2024

• Remove incorrect final for non-virtual func (#34)

  vinx13

  authored
  2db668a

  Copy full SHA for 2db668a

• Added EncodeBatch interface (#33)

  vinx13

  authored
  55170e8

  Copy full SHA for 55170e8

Commits on Apr 5, 2024

• Set value of TOKENIZERS_CPP_CARGO_TARGET based on ANDROID_ABI (#28)

  yuxuanchiadm

  authored
  07376b9

  Copy full SHA for 07376b9

Commits on Apr 4, 2024

• [HFTokenizer] Allow use of "add_special_tokens" for HFTokenizer.Encode and "skip_special_tokens" for HFTokenizer.Decode methods. (#27)

  Abhishek8394

  authored
  35bad0e

  Copy full SHA for 35bad0e

Commits on Mar 30, 2024

• [Web] Expose getVocabSize and idToToken to web, bump version to 0.1.3 (#25)

  CharlieFRuan

  authored
  7466de5

  Copy full SHA for 7466de5

Commits on Dec 19, 2023

• Add support for querying vocabulary from tokenizer (#22)

  Ubospica

  authored
  27dbe17

  Copy full SHA for 27dbe17