neuralmonkey.evaluators package

Submodules

neuralmonkey.evaluators.accuracy module

class neuralmonkey.evaluators.accuracy.Accuracy(name='Accuracy')

Bases: object

static compare_scores(score1, score2)

neuralmonkey.evaluators.bleu module

class neuralmonkey.evaluators.bleu.BLEUEvaluator(n=4, deduplicate=False, name=None)

Bases: object

static bleu(hypotheses, references, ngrams=4, case_sensitive=True)

Computes BLEU on a corpus with multiple references using uniform weights. Default is to use smoothing as in reference implementation on: https://github.com/ufal/qtleap/blob/master/cuni_train/bin/mteval-v13a.pl#L831-L873

Parameters:
  • hypotheses – List of hypotheses
  • references – LIst of references. There can be more than one reference.
  • ngram – Maximum order of n-grams. Default 4.
  • case_sensitive – Perform case-sensitive computation. Default True.
static compare_scores(score1, score2)
static effective_reference_length(hypotheses, references_list)

Computes the effective reference corpus length (based on best match length)

Parameters:
  • hypotheses – List of output sentences as lists of words
  • references – List of lists of references (as lists of words)
static merge_max_counters(counters: typing.List[collections.Counter]) → collections.Counter

Merge counters using maximum values

static minimum_reference_length(hypotheses, references_list)

Computes the effective reference corpus length (based on the shortest reference sentence length)

Parameters:
  • hypotheses – List of output sentences as lists of words
  • references – List of lists of references (as lists of words)
static modified_ngram_precision(hypotheses: typing.List[typing.List[str]], references_list: typing.List[typing.List[typing.List[str]]], n: int, case_sensitive: bool) → typing.Tuple[float, int]

Computes the modified n-gram precision on a list of sentences

Parameters:
  • hypothesis – List of output sentences as lists of words
  • references – List of lists of reference sentences (as lists of words)
  • n – n-gram order
  • case_sensitive – Whether to perform case-sensitive computation
static ngram_counts(sentence: typing.List[str], n: int, lowercase: bool, delimiter=' ') → collections.Counter

Get n-grams from a sentence

Parameters:
  • sentence – Sentence as a list of words
  • n – n-gram order
  • lowercase – Convert ngrams to lowercase
  • delimiter – delimiter to use to create counter entries

neuralmonkey.evaluators.bleu_ref module

class neuralmonkey.evaluators.bleu_ref.BLEUReferenceImplWrapper(wrapper, name='BLEU', encoding='utf-8')

Bases: object

Wrapper for TectoMT’s wrapper for reference NIST and BLEU scorer

serialize_to_bytes(sentences)

neuralmonkey.evaluators.edit_distance module

class neuralmonkey.evaluators.edit_distance.EditDistanceEvaluator(name='Edit distance')

Bases: object

static compare_scores(score1, score2)
static ratio(str1, str2)

neuralmonkey.evaluators.perplexity module

class neuralmonkey.evaluators.perplexity.Perplexity(name='Perplexity')

Bases: object

neuralmonkey.evaluators.ter module

Module contents