abbr.tsv has two entries for ICML:

Proc. of ICML | Proceedings of the .* International Conference on Machine Learning
Proc. of ICML | Machine Learning, Proceedings of the .* International Conference

but they are not both loaded, because in load_abbr_tsv() a dictionary is used such that the second entry overwrites the first one:

ls = line.split("|")
if len(ls) == 2:
    abbr_dict[ls[0].strip()] = ls[1].strip()

I see two solutions here: either don't load the file into a dictionary (but just a list of tuples), or allow specifying or regex patterns (i.e., (pattern1|pattern2)) which would require using a different character than | to separate the left- and right-hand sides in abbr.tsv.

With this feature, users do not need to upgrade via pip if the code doesn't change at all.

It's a great tools, but when I try to transfer my .bib file, which is generated by an application BibDesk, the references are filtered, here is a minimal example of my bib file.

@inproceedings{zhang2019heterogeneous,
        author = {Zhang, Chuxu and Song, Dongjin and Huang, Chao and Swami, Ananthram and Chawla, Nitesh V},
        booktitle = {Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery \& Data Mining},
        date-added = {2021-04-03 01:39:20 +0800},
        date-modified = {2021-04-03 01:44:13 +0800},
        keywords = {Recommender system, Graph Neural Network},
        pages = {793--803},
        title = {Heterogeneous graph neural network},
        year = {2019},
        Bdsk-Url-1 = {https://doi.org/10.1145/3292500.3330961}}

I think this is due to load_bib_file. The last line of this reference contains {, so load_bib_file skipped this reference.

However, in the BibtexParser, this kind of bib file can be recognized.

The ImageNet paper has its last author listed as Li Fei-Fei, which is how she publishes in general, both on the paper and in the IEEE metadata; their .bib has her as Li Fei-Fei in the author.

The DBLP record lists her as Li Fei{-}Fei.

And yet rebiber/data/cvpr2009.bib.json has her as Fei{-}Fei Li, and so running either through rebiber incorrectly changes it to that ordering.

The same is true for most (but not all) of her papers in the database. No idea why this would be, since DBLP consistently has her as Li Fei{-}Fei.

cc @pranav-ust

I have multiple bib files for several research fields and hope convert their information in one-click. I've written a bat file to automatically execute bib files in work directory:

@echo off
for %%i in (*.bib) do echo "%%i"
for %%i in (*.bib) do rebiber -i %%i -o Pub%%i
pause
exit

But a build-in command would be easier to use. Would you like to add this?

Hello, I find that some papers accepted by some conferences (e.g. AAAI 2020) cannot be indexed. The reason might be that we can only download the first 1000 entries when the accepted papers are more than 1,000 from DBLP. Is there any way to address such problem? Thanks very much!

I can work on this soon. The main one is MICCAI.

I come across two issues here:

• Somehow the tool transforms my comments (ones starting with %) in bib file into @comment{} and places them at the head of the file;
• All the bibs are by default organized in an alphabetic manner, is there a way (option) I can remain the order of bibs (and thus keep the comments where they are) .

Great tool by the way :)

Hi Yuchen,
It seems you try to ignore the month field in a bib entry in is_contain_var() and build_json(). Can you please explain why is that necessary?
You also ignore '@string' entry. Why not just let bibtexparser parse the entire bib file?
Thank you!

Hi,

Thanks for the great tool! I faced an issue where an entry was deleted after using rebiber though. It's this one:

@article{loon,
  title={Autonomous navigation of stratospheric balloons using reinforcement learning.},
  author={Marc G. Bellemare and Salvatore Candido and P. S. Castro and J. Gong and Marlos C. Machado and Subhodeep Moitra and Sameera S. Ponda and Ziyu Wang},
  journal={Nature},
  year={2020},
  volume={588 7836},
  pages={
          77-82
        }
}

Do you have any idea what could be wrong?

rebiber --update fails on windows, because a series of linux commands such as wget

When trying to run the code in README.

This will be ignored.

@software{spacy,
  author = {Honnibal, Matthew and Montani, Ines and Van Landeghem, Sofie and Boyd, Adriane},
  title = {{spaCy: Industrial-strength Natural Language Processing in Python}},
  year = 2020,
  publisher = {Zenodo},
  doi = {10.5281/zenodo.1212303},
  url = {https://doi.org/10.5281/zenodo.1212303}
}

Nice tool!
It seems that currently it doesn't handle @string in BibTeX. Any plan to add this feature?

Example:

@string{emnlp = "Empirical Methods in Natural Language Processing (EMNLP)"}

@inproceedings{li2020efficient,
 title={Efficient One-Pass End-to-End Entity Linking for Questions},
 author={Li, Belinda Z. and Min, Sewon and Iyer, Srinivasan and Mehdad, Yashar and Yih, Wen-tau},
 booktitle=emnlp,
 year={2020}
}

I found that the booktitle of many papers in DBLP has too many names and information.

For example：

@inproceedings{seo-etal-2016-bidirectional,
 author = {Min Joon Seo and
Aniruddha Kembhavi and
Ali Farhadi and
Hannaneh Hajishirzi},
 bibsource = {dblp computer science bibliography, https://dblp.org},
 biburl = {https://dblp.org/rec/conf/iclr/SeoKFH17.bib},
 booktitle = {5th International Conference on Learning Representations, {ICLR} 2017,
Toulon, France, April 24-26, 2017, Conference Track Proceedings},
 publisher = {OpenReview.net},
 timestamp = {Thu, 25 Jul 2019 01:00:00 +0200},
 title = {Bidirectional Attention Flow for Machine Comprehension},
 url = {https://openreview.net/forum?id=HJ0UKP9ge},
 year = {2017}
}

The booktitle here contains the full name and abbreviation of ICLR, as well as their location.
Can you keep only the first one of this information?

For example:
booktitle =“5th International Conference on Learning Representations”

zotero是个常见的开源文献管理软件，可以导出为bib格式
或许可以基于rebiber写一个zotero插件，批量把zotero中的条目的arkiv引用更新

Hi! It's a great tool. I have downloaded some other conferences of personal interest, and wanted to share them with others. There are some gaps in downloads here and there (specifically, for multi-volume conferences), but it's more complete than the one present in the repo. I have included more years as well.

The data is >400 MB uncompressed, and 30Mb compressed. I leave a link here for those who may find it useful.
http://tsitsul.in/data/confdata.zip

Big thanks to the people behind rebiber. It is a very helpful tool.

I noticed some peculiarities with the matching heuristic (see example below).
TLDR: rebiber turned the entries for the Deep Learning book (Goodfellow) and the Deep Learning nature article (LeCun, Bengio, Hinton) into entries referring to a SIGKDD paper (Salakuthdinov) with the same title--without any warning.

Based on this behaviour, I assume the matching is done only based on title.

I suggest to check additionally for at least one common author. Or (less invasive): to emit an adequate warning when there is not a single same-author-name in a new entry compared to the original entry.

Input

@article{deeplearning,
  title = {Deep Learning},
  author = {LeCun, Yann and Bengio, Yoshua and Hinton, Geoffrey},
  year = {2015},
  month = may,
  journal = {Nature},
  volume = {521},
  number = {7553},
  pages = {436--444},
  publisher = {{Nature Publishing Group}},
  issn = {1476-4687},
  doi = {10.1038/nature14539},
  copyright = {2015 Nature Publishing Group, a division of Macmillan Publishers Limited. All Rights Reserved.},
}

@book{goodfellow_deep_2016,
	series = {Adaptive computation and machine learning},
	title = {Deep {Learning}},
	isbn = {978-0-262-03561-3},
	url = {http://www.deeplearningbook.org/},
	publisher = {MIT Press},
	author = {Goodfellow, Ian J. and Bengio, Yoshua and Courville, Aaron C.},
	year = {2016},
}

Cmd: rebiber -i main.bib -o main.bib -r editor -d True -s True

Output

@inproceedings{deeplearning,
 author = {Ruslan Salakhutdinov},
 bibsource = {dblp computer science bibliography, https://dblp.org},
 biburl = {https://dblp.org/rec/conf/kdd/Salakhutdinov14.bib},
 booktitle = {Proc. of KDD},
 doi = {10.1145/2623330.2630809},
 pages = {1973},
 publisher = {{ACM}},
 timestamp = {Tue, 06 Nov 2018 00:00:00 +0100},
 title = {Deep learning},
 url = {https://doi.org/10.1145/2623330.2630809},
 year = {2014}
}

@inproceedings{goodfellow_deep_2016,
 author = {Ruslan Salakhutdinov},
 bibsource = {dblp computer science bibliography, https://dblp.org},
 biburl = {https://dblp.org/rec/conf/kdd/Salakhutdinov14.bib},
 booktitle = {Proc. of KDD},
 doi = {10.1145/2623330.2630809},
 pages = {1973},
 publisher = {{ACM}},
 timestamp = {Tue, 06 Nov 2018 00:00:00 +0100},
 title = {Deep learning},
 url = {https://doi.org/10.1145/2623330.2630809},
 year = {2014}
}

The command line output of rebiber only state

• Converted. ID: deeplearning ; Title: Deep Learning
• Converted. ID: goodfellow_deep_2016 ; Title: Deep {Learning}

Desired output

a) Do not have the entries replaced, as there is not a single author name in common

or b) Emit a major warning on the command line, when replacing without a single author name in common

so that one can choose to turn on/off some entries such as date/year/URL/DOI stuff.

Thank you for your wonderful tool ! Saved me a lot of time !

I used the following scripts to extract bib tex file from dplb using the API if that can be useful :

confs=['ecml', 'wacv']#['eccv', 'iccv', 'bmvc','cvpr', 'accv', 'neurips','miccai', 'ecml']
years=list(range(2000, 2024))

for conf in confs : 
    for year in years : 
        cites = ''
        for step in range(5) :
            print(step, conf, year)
            s=f'https://dblp.org/search/publ/api?q=conf/{conf}/{year}&h=1000&f={step*1000}&format=bib'
            cs = requests.get(s).text
            if cs == '' : 
                print('stop')
                break
            cites+= cs
        with open(f'/rebiber/rebiber/raw_data/{conf}{year}.bib', 'w') as f : 
            f.write(cites)

For journals you can use

journals=['ijcv']#'pami'
editions=range(100, 132)
  
s=f'https://dblp.org/search/publ/api?q=toc:db/journals/{journal}/{journal}{edition}.bht:&h=1000&f={step*1000}&format=bib'

I also attached a zip with the classical computer vision conferences / journals.
CV-bib-files.zip

Although a scripting approach is provided, would you consider providing a Python API ？

for example

import rebiber
str = '@article{lin2020birds,
	title={Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Models},
	author={Lin, Bill Yuchen and Lee, Seyeon and Khanna, Rahul and Ren, Xiang},
	journal={arXiv preprint arXiv:2005.00683},
	year={2020}
}'
res = rebiber.trans(str)
print(res)

For @article and @book entries, a double-closed braces at the end will generate an extra @comment block in the output bib file.

Input example:

@article{Ando2005,
	Acmid = {1194905},
	Author = {Ando, Rie Kubota and Zhang, Tong},
	Issn = {1532-4435},
	Pages = {1817--1853},
	Publisher = {JMLR.org},
	Title = {A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data},
	Volume = {6},
	Year = {2005}}

Output:

@comment{}}

@article{Ando2005,
 acmid = {1194905},
 author = {Ando, Rie Kubota and Zhang, Tong},
 issn = {1532-4435},
 issue_date = {12/1/2005},
 journal = {Journal of Machine Learning Research},
 numpages = {37},
 pages = {1817--1853},
 publisher = {JMLR.org},
 title = {A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data},
 volume = {6},
 year = {2005}
}

Try this one in Hugging Face , it will generate an empty output.

@inproceedings{LocalAlgorithmFinding2013zhu,
title = {A Local Algorithm for Finding Well-Connected Clusters},
booktitle = {International {{Conference}} on {{Machine Learning}}},
author = {Zhu, Zeyuan Allen and Lattanzi, Silvio and Mirrokni, Vahab},
year = {2013},
pages = {396--404},
publisher = {{PMLR}}
}

Work in progress and suggestions are welcome!

The LREC Sign Language workshop has this website -
https://www.sign-lang.uni-hamburg.de/lrec/index.html

Which links to two bib files:
without abstracts: https://www.sign-lang.uni-hamburg.de/lrec/sign-lang_lrec.bib
with abstracts: https://www.sign-lang.uni-hamburg.de/lrec/sign-lang_lrec_a.bib

While one can add them manually to this repo, I was wondering if there is a setting somewhere to just put this link, and whenever someone runs an "update" script it will re-fetch the bib file and process it?

Hi,

Is there any way to auto-add DOIs from the BibTeX files? This would be highly useful. I see a similar thing in one of the forks but idk if this is useful since it's only for arxiv I think.

thanks !!

Try this one in Hugging Face , it will generate an empty output.

@inproceedings{LocalAlgorithmFinding2013zhu,
title = {A Local Algorithm for Finding Well-Connected Clusters},
booktitle = {International {{Conference}} on {{Machine Learning}}},
author = {Zhu, Zeyuan Allen and Lattanzi, Silvio and Mirrokni, Vahab},
year = {2013},
pages = {396--404},
publisher = {{PMLR}}
}