tokenization

Tokenization in Natural Language Processing (NLP) is the process of partitioning natural language text into smaller units, which are then manipulated by RNN or other artificial neural network models.