microsoft / openkp Goto Github PK

The evaluation script expects 2 files. Can someone explain the format and use case of each?

Shouldn't this be:
precision.append(100*true_positive/min(float(i + 1), len(candidates)))

Which script among these can help with the prediction and what is the structure of input files expected?
MakeOpenKP.py
makepredsform.py

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

Hi, when we're doing the case study using the evaluation script, we got some confusion about the results.

1. We find that you have the normalize_answer function. Several rules are set to normalize the candidates and references. However, we found these methods are not applied to the answers: in the return statement, these separate rules are not even called. This caused at least the punctuation problem: no punctuations are included in your raw text field of the dataset, but some punctuations exist in your keyphrases which makes it impossible for the models to predict the "exactly matched keyphrases".
   Refer to:
   

   OpenKP/evaluate.py

   Lines 7 to 17 in 6e1d770

   	def normalize_answer(s):
   	def remove_articles(text):
   	return re.sub(r'\b(a|an|the)\b', ' ', text)
   	def white_space_fix(text):
   	return ' '.join(text.split())
   	def remove_punc(text):
   	exclude = set(string.punctuation)
   	return ''.join(ch for ch in text if ch not in exclude)
   	def lower(text):
   	return text.lower()
   	return ' '.join([lower(x) for x in s]).rstrip()

2. We notice that you filter keyphrases beginning with an empty string "". However, there're some(quite a few) keypharses in the dataset which are not exactly empty like ["", "Middle", "East", "&", "Jewish", "World"] in line 6109 of dev.jsonl. So is there a reason to discard these kind of keyphrases?
   Refer to:
   

   OpenKP/evaluate.py

   Lines 19 to 25 in 6e1d770

   	def remove_empty(a_list):
   	new_list = []
   	for i in a_list:
   	if len(i) > 0:
   	if len(i[0]) >0:
   	new_list.append(normalize_answer(i))
   	return new_list

I want to get the OpenKP dataset, but I can't find it in https://microsoft.github.io/msmarco/ or msmarco.org. Is it removed?

Can you please share your code for generating the VDOM?

Hi MARCO team,

I appreciate your effort in creating this great dataset and maintaining the leaderboard. Since the leaderboard for OpenKP has been retired, I wonder if there is any plan releasing the test dataset, so that the research community can continue studies with it.

Thank you,
Rui Meng

I am unable to find the data to download. Can you please point me to the location?

Hi OpenKP Team,

I tried to visit http://www.msmarco.org/dataset.aspx, but it seems not to work at this moment.

Thanks,
Jiaying