Comments (9)
Hi @jojonki ,
Thanks for pointing out the problem. I just checked my data splits and found it has different format from the original aida data for the train split (speparated by ',' instead of '\t'). Because the data split part and preprocessing part are done by another author of the paper and I don't know how she splits the aida data. I'll check with her and get back to you.
from entqa.
Sure, you can use the pretrained models on your custom input text. Since it's trained on AIDA, I'm not sure if it perform very well on your custom text. Check our GERBIL performance for reference.
from entqa.
Hi @hezongfeng ,
We split the "AIDA-YAGO2-dataset.tsv into those three files by ourselves according to the README.txt file in the downloaded aida-yago2-datasets.zip. In the README.txt file, they keep the ordering among the documents as in the original CoNLL data: TRAIN: '1 EU' to '946 SOCCER' , TESTA: '947testa CRICKET' to '1162testa Dhaka', TESTB: '1163testb SOCCER' to '1393testb SOCCER'. So the records from '1 EU' to '946 SOCCER' is 'aida-yago2-dataset-train.tsv, '947testa CRICKET' to '1162testa Dhaka' is 'aida-yago2-dataset-val.tsv' and '1163testb SOCCER' to '1393testb SOCCER' is 'aida-yago2-dataset-test.tsv'.
from entqa.
Thank you for your nice work!
I have similar questions.
First, why did you use ,
delimiter to split the train file? I am assuming all the files are split by \t
, not ,
.
https://github.com/WenzhengZhang/EntQA/blob/main/preprocess_data.py#L23
So, I modified your code to only use \t
. Though, I faced out-of-index
error at https://github.com/WenzhengZhang/EntQA/blob/main/preprocess_data.py#L241
Do you have some special manipulations to CoNLL data?
from entqa.
Regarding to my second mention (out-of-index
error), I noticed csvreader
sometimes read multiple lines at once.
So, I modified process_raw_aida
like this.
# for data in csvreader:
for data in f:
data = data.strip().split("\t")
from entqa.
Hello!
Is it possible for someone to share a sample of the preprocessed dataset? Just couple of entites would help a lot for studying the code. Thanks!
from entqa.
Hi, is there anyone that managed to split the AIDA dataset correctly in order to have the preprocesing script to work ?
from entqa.
Hi,
Unfortunately, I'm not able to contact that author since she left. Therefore, I updated the readme.md file and public all the preprocessed data. You can download the preprocessed data here.
from entqa.
Thank you for the files :) the preprocessed_kilt is not included but removing the aida part and with the downloaded kilt file I guess it is possible to obtain it with preprocessed_data.py.
Is it possible to use your pretrained models on a custom input text ?
from entqa.
Related Issues (10)
- some issues to reproduce the Gerbil result HOT 2
- ValueError np.load(ars.cands_embeds_path)
- Probability of predicted entities decreases with increase in number of candidates HOT 1
- Some issues in data preprocessing
- The file in gerbil-SpotWrapNifWS4Test/repository
- 可以发一个安装并运行EntQA的视频么
- Cannot reproduce the results HOT 3
- evaluate on GERBIL
- Download files are missing
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from entqa.