Results table for comparison with published results

[mberr]

For the paper we need a table which shows our obtained results in comparison to the previously published results.

I suggest the following format, here a excerpt from the FB15k table

		MR	MRR (%)	AMR (%)	Hits@1 (%)	Hits@3 (%)	Hits@5 (%)	Hits@10 (%)
ComplEx	[Troullion2016]o	-	69.2	-	59.9	75.9	-	84.0
	best	171.16 ± 16.05	19.13 ± 0.45	2.46 ± 0.28	10.06 ± 0.42	20.82 ± 0.67	27.59 ± 0.94	38.03 ± 1.02
	...
ConvE	[Dettmers2018]n	51	65.7
	best	50.76 ± 0.40	59.56 ± 0.06

with the following legend:

• With ^b / ^a / ^w / ⁿ we denote that the best / average / worst / non-deterministic rank
• [Troullion2016] should be a proper reference

[mberr]

@cthoyt @mali-git Let's discuss the format here

[mberr]

I would add a \midrule before a new interaction model starts.

[mberr]

As far as I saw we already have a method which collects a dataframe containing the combination of best/avg/worst results for each model. Thus, we would need to create a multi-index ['model', 'rank_type'], and add for each interaction model the entry

# published_results is a dictionary of the loaded results for one dataset
# example
assert published_results[('trouillon2016', 'complex')] == {
    'rank_type': 'best',
    'mean_reciprocal_rank': 0.692,
    'hits_at_1': 0.599,
    'hits_at_3': 0.759,
    'hits_at_10': 0.840,
}

for reference_key, model_name, metrics in published_results.items():
    rank_type_key = metrics.pop('rank_type')[0]
    reference = r'\cite{' + str(reference_key) + '}^{' + rank_type_key + '}'
    for metric_name, value in metrics.items():
        df[(model_name, reference), metric_name] = metric_value

[mberr]

I am not sure what is the best way to get the JSON configs from pykeen master into this repository 😅

[mali-git]

I am not sure what is the best way to get the JSON configs from pykeen master into this repository 😅

The old configs are contained in reproducibility/configs. We could replace those.

[mberr]

Done in #18

[mali-git]

Thanks!