neuralmonkey.processors.bpe module

class neuralmonkey.processors.bpe.BPEPostprocessor(separator: str = '@@') → None

Bases: object

__init__(separator: str = '@@') → None

Initialize self. See help(type(self)) for accurate signature.

decode(sentence: List[str]) → List[str]

class neuralmonkey.processors.bpe.BPEPreprocessor(merge_file: str, separator: str = '@@', encoding: str = 'utf-8') → None

Bases: object

Wrapper class for Byte-Pair Encoding.

Paper: https://arxiv.org/abs/1508.07909 Code: https://github.com/rsennrich/subword-nmt

__init__(merge_file: str, separator: str = '@@', encoding: str = 'utf-8') → None

Initialize self. See help(type(self)) for accurate signature.