neuralmonkey.writers.plain_text_writer module¶
-
neuralmonkey.writers.plain_text_writer.
T2TWriter
(path: str, data: Iterator[List[str]]) → None¶
-
neuralmonkey.writers.plain_text_writer.
UtfPlainTextWriter
(path: str, data: Iterator[List[str]]) → None¶
-
neuralmonkey.writers.plain_text_writer.
t2t_detokenize
(data: Iterator[List[str]]) → Iterator[str]¶ Detokenize text tokenized by t2t_tokenized_text_reader.
Method is inspired by tensor2tensor tokenizer.decode method: https://github.com/tensorflow/tensor2tensor/blob/v1.5.5/tensor2tensor/data_generators/tokenizer.py
-
neuralmonkey.writers.plain_text_writer.
t2t_tokenized_text_writer
(encoding: str = 'utf-8') → Callable[[str, Any], NoneType]¶ Get a writer that is reversed to the t2t_tokenized_text_reader.
-
neuralmonkey.writers.plain_text_writer.
text_writer
(encoding: str = 'utf-8') → Callable[[str, Any], NoneType]¶
-
neuralmonkey.writers.plain_text_writer.
tokenized_text_writer
(encoding: str = 'utf-8') → Callable[[str, Any], NoneType]¶ Get a writer that is reversed to the tokenized_text_reader.