neuralmonkey.processors package

Submodules

neuralmonkey.processors.alignment module

class neuralmonkey.processors.alignment.WordAlignmentPreprocessor(source_len, target_len, dtype=<class 'numpy.float32'>, normalize=True, zero_based=True)

Bases: object

A preprocessor for word alignments in a text format.

One of the following formats is expected:

s1-t1 s2-t2 ...

s1:1/w1 s2:t2/w2 ...

where each s and t is the index of a word in the source and target sentence, respectively, and w is the corresponding weight. If the weight is not given, it is assumend to be 1. The separators - and : are interchangeable.

The output of the preprocessor is an alignment matrix of the fixed shape (target_len, source_len) for each sentence.

neuralmonkey.processors.bpe module

class neuralmonkey.processors.bpe.BPEPostprocessor(separator: str = '@@') → None

Bases: object

decode(sentence: typing.List[str]) → typing.List[str]

class neuralmonkey.processors.bpe.BPEPreprocessor(merge_file: str, separator: str = '@@') → None

Bases: object

Wrapper class for Byte-Pair Encoding.

Paper: https://arxiv.org/abs/1508.07909 Code: https://github.com/rsennrich/subword-nmt

neuralmonkey.processors.editops module

class neuralmonkey.processors.editops.Postprocess(source_id: str, edits_id: str) → None

Bases: object

Proprocessor applying edit operations on a series.

class neuralmonkey.processors.editops.Preprocess(source_id: str, target_id: str) → None

Bases: object

Preprocessor transorming two series into series of edit operations.

neuralmonkey.processors.editops.convert_to_edits(source: typing.List[str], target: typing.List[str]) → typing.List[str]

neuralmonkey.processors.editops.reconstruct(source: typing.List[str], edits: typing.List[str]) → typing.List[str]

neuralmonkey.processors.german module

class neuralmonkey.processors.german.GermanPostprocessor(compounding=True, contracting=True, pronouns=True)

Bases: object

decode(sentence)

class neuralmonkey.processors.german.GermanPreprocessor(compounding=True, contracting=True, pronouns=True)

Bases: object

neuralmonkey.processors.helpers module

neuralmonkey.processors.helpers.pipeline(processors: typing.List[typing.Callable]) → typing.Callable

Concatenate processors.

neuralmonkey.processors.helpers.postprocess_char_based(sentence: typing.List[str]) → typing.List[str]

neuralmonkey.processors.helpers.preprocess_char_based(sentence: typing.List[str]) → typing.List[str]

neuralmonkey.processors.helpers.untruecase(sentences: typing.List[typing.List[str]]) → typing.Generator[[typing.List[str], NoneType], NoneType]

Module contents