neuralmonkey.evaluators.multeval module

class neuralmonkey.evaluators.multeval.MultEvalWrapper(wrapper: str, name: str = 'MultEval', encoding: str = 'utf-8', metric: str = 'bleu', language: str = 'en') → None

Bases: neuralmonkey.evaluators.evaluator.Evaluator

Wrapper for mult-eval’s reference BLEU and METEOR scorer.

__init__(wrapper: str, name: str = 'MultEval', encoding: str = 'utf-8', metric: str = 'bleu', language: str = 'en') → None

Initialize the wrapper.

Parameters:

• wrapper – Path to multeval.sh script
• name – Name of the evaluator
• encoding – Encoding of input files
• language – Language of hypotheses and references
• metric – Evaluation metric “bleu”, “ter”, “meteor”

score_batch(hypotheses: List[List[str]], references: List[List[str]]) → float

Score a batch of hyp/ref pairs.

The default implementation of this method calls score_instance for each instance in the batch and returns the average score.

Parameters:

• hypotheses – List of model predictions.
• references – List of golden outputs.

Returns:

A float.

serialize_to_bytes(sentences: List[List[str]]) → bytes