Text Processors

class textprocessors.Windower(tokens, n=1, beginmarker='<begin>', endmarker='<end>')

Moves a sliding window over a list of tokens, returning all windows

textprocessors.crude_tokenizer(line)

This is a very crude tokenizer

textprocessors.strip_accents(s, encoding='utf-8')