noun ·Rare ·Advanced level

Definitions

Noun

1. 1
   The act or process of tokenizing. countable, uncountable
2. 2
   Something tokenized. countable, uncountable

   "This was an unlikely tokenization of the input string."

Example

More examples

"This was an unlikely tokenization of the input string."

Etymology

From token + -ization.

Data sourced from Wiktionary, WordNet, CMU, and other open linguistic databases. Updated March 2026.