neuralmonkey.writers.plain_text_writer module

neuralmonkey.writers.plain_text_writer.T2TWriter(path: str, data: Iterator[List[str]]) → None

neuralmonkey.writers.plain_text_writer.UtfPlainTextWriter(path: str, data: Iterator[List[str]]) → None

neuralmonkey.writers.plain_text_writer.t2t_detokenize(data: Iterator[List[str]]) → Iterator[str]

Detokenize text tokenized by t2t_tokenized_text_reader.

Method is inspired by tensor2tensor tokenizer.decode method: https://github.com/tensorflow/tensor2tensor/blob/v1.5.5/tensor2tensor/data_generators/tokenizer.py

neuralmonkey.writers.plain_text_writer.t2t_tokenized_text_writer(encoding: str = 'utf-8') → Callable[[str, Any], NoneType]

Get a writer that is reversed to the t2t_tokenized_text_reader.

neuralmonkey.writers.plain_text_writer.text_writer(encoding: str = 'utf-8') → Callable[[str, Any], NoneType]

neuralmonkey.writers.plain_text_writer.tokenized_text_writer(encoding: str = 'utf-8') → Callable[[str, Any], NoneType]

Get a writer that is reversed to the tokenized_text_reader.