Tokenizer

noun

noun ·Rare ·Advanced level

Definitions

Noun
  1. 1
    A system that parses an input stream into its component tokens.

Etymology

From tokenize + -er.

More for "tokenizer"