tokenizer

English

Etymology

From tokenize +‎ -er.

Noun

tokenizer (plural tokenizers)

  1. (computing) A system that parses an input stream into its component tokens.
This article is issued from Wiktionary. The text is available under Creative Commons Attribution-Share Alike 4.0 unless otherwise noted. Additional terms may apply for the media files.