Commits

Breadcrumbs

History for

onmain

User selector

All users

All time

Commit History

Commits on Jan 13, 2026

• 同步包装器

  Huiyicc

  committed
  e450f25

  Copy full SHA for e450f25

Commits on Nov 16, 2024

• 更新encode_ex

  Huiyicc

  committed
  6b452c8

  Copy full SHA for 6b452c8

Commits on Aug 12, 2024

• [Web] Set TOKENIZERS_PARALLELISM to false for HFTokenizer (#42)

  Show description for d5dd7f4
  CharlieFRuan

  authored
  d5dd7f4

  Copy full SHA for d5dd7f4

Commits on Jun 27, 2024

• [CMake] Support disable SentencePiece tokenizer (#38)

  Show description for c0fab1e
  MasterJH5574

  authored
  c0fab1e

  Copy full SHA for c0fab1e

Commits on May 29, 2024

• Remove incorrect final for non-virtual func (#34)

  vinx13

  authored
  2db668a

  Copy full SHA for 2db668a

• Added EncodeBatch interface (#33)

  vinx13

  authored
  55170e8

  Copy full SHA for 55170e8

Commits on Apr 4, 2024

• [HFTokenizer] Allow use of "add_special_tokens" for HFTokenizer.Encode and "skip_special_tokens" for HFTokenizer.Decode methods. (#27)

  Abhishek8394

  authored
  35bad0e

  Copy full SHA for 35bad0e

Commits on Dec 19, 2023

• Add support for querying vocabulary from tokenizer (#22)

  Ubospica

  authored
  27dbe17

  Copy full SHA for 27dbe17

Commits on Nov 16, 2023

• fix rwkv world tokenzier (#19)

  Show description for 4456fa8
  BBuf

  authored
  4456fa8

  Copy full SHA for 4456fa8

Commits on Sep 1, 2023

• add rwkv world tokenizer (#14)

  Show description for eec72a6
  BBuf

  authored
  eec72a6

  Copy full SHA for eec72a6

Commits on May 20, 2023

• Add examples

  tqchen

  committed
  5703f8d

  Copy full SHA for 5703f8d

Commits on May 17, 2023

• Initial checkin of tokenizer cpp

  Show description for a3fd734
  

  6 peoplecommitted
  a3fd734

  Copy full SHA for a3fd734

Loading