pip install -e ".[dev]"
For the data in data/arxiv_tables_2308_high_quality
, there are three relevant files:
tables.jsonl
: a list of 30 papers. Each line is ajson
object with the following fields:tabid
: a unique hash for the table that comes from the latex-processing pipelinetable
: a python dictionary dump of the table created frompd.DataFrame.to_dict()
. Can be loaded back into pandas withpd.DataFrame(table['table'])
. The index of the data frame is the thecorpus_id
of the full text of the paper (which can be accessed atdata/full_texts/{corpus_id}.jsonl
).row_bib_map
: a list ofjson
objects, where each represents a citation from the table. (This shouldn't be needed anymore, but I'm leaving it in in case it's useful for now. Will probably be deleted soon.) Each object contains:- the
corpus_id
of the full text of the paper - the
row
in the table thatcorpus_id
corresponds to - the
type
of the citation -"ref"
means it's an external reference and"ours"
means that the paper containing the table is represented in the given row. - the
bib_hash_or_arxiv_id
associated with that corpus_id.arxiv_id
is used when thetype
is"ours"
and thebib_hash
is used when it's"ref"
. Thebib_hash
comes from the latex-processing pipeline and it's a function of the citation text and the citing paper's arxiv_id.
- the
papers.jsonl
: a list of all the papers needed for generating the tables. Each paper entry contains:tabids
: a list of ids for the tables the paper is cited in.corpus_id
, the corpus id of a paper. If the full text is available, it can be accessed atdata/full_texts/{corpus_id}.jsonl
. (For the small subset, all of these texts should be available.)title
: the title of the paper from s2.paper_id
: the s2 id of the paper.
dataset.jsonl
: contains a superset of the information intables.jsonl
(it additionally includes an html version of the table from which the json format was derived, some artefacts from the html->json conversion, and some additional metadata about the papers and tables)
These are loaded in using the configs/full_texts
config, with the data.FullTexts
class performing the loading part.
python paper_comparison/run.py data=full_texts data.path=data/arxiv_tables_2308_high_quality endtoend=debug endtoend.name=oracle
(Setting endtoend.name=oracle
just has the system output the gold tables.)
See scripts/data_processing
When running, you need to pass some input example, and either a baseline or a schematizer and populator.
To run an end-to-end model on a single example:
python paper_comparison/run.py data.path=data/debug_abstracts endtoend=debug
To run a schematizer-populator combination on a single example:
python paper_comparison/run.py data.path=data/debug_abstracts schematizer=followup_questions populator=qa
After running, the results path will be output to the terminal. This path contains three files:
tables.jsonl
: a file with one table per linemetrics.jsonl
: a file with table metricscommands.txt
: a list of commands that were run to obtain the results in the directory. Most recent at the bottom.
Currently all data is in sub-directories of the data
path. These directories are passed to the data.path
field at the command line. In these directories, there are one or two files: one (usually called papers.jsonl
) has one paper json object per line. For example, one line will have the following fields:
{
"tabids": ["<Table IDs. All papers that will be compared should have the same Table ID>. This is a list because papers can"],
"paperid": "<Paper ID. A string used as the name of the row in the table. It's used to link tables to papers."
<other information eg abstracts, title, authors, etc.>
}
If this data is being used for training or supervised evaluation, there will also be gold tables in this directory (in a file usually called tables.jsonl
). Each line in the table has the following structure:
{
"tabid" : "<Table ID",
"table": {
"row1": {"paperid_1": "value", "paperid_2": "value"},
"row2": {"paperid_1": "value", "paperid_2": "value"},
...
}
}
Each of these files can be overridden individually by setting the data.papers_path
or data.tables_path
at the command line. E.g.:
python paper_comparison/run.py data.papers_path=data/debug_abstracts/papers.jsonl data.tables_path=data/debug_abstracts/tables.jsonl endtoend=debug
(Not implemented yet)
For training, train one of endtoend
, schematizer
, or populator
. The relevant configs must have training details in the training
field of the config.
Training an end-to-end model:
python paper_comparison/train.py train_data=<data_config> val_data=<data_config> endtoend=<endtoend_config> trainer=<train_config>
Training a schematizer model:
python paper_comparison/train.py train_data=<data_config> val_data=<data_config> schematizer=<schematizer_config>
Training a populator model requires specifying a schematizer to use as well:
python paper_comparison/train.py train_data=<data_config> val_data=<data_config> schematizer=<schematizer_config> populator=<populator>
loaders
: Code for loading in tables (in json format) and papers (using papermage) [Not implemented yet]endtoend.py
: Transform collections of papers directly into tables. Used for baselines.schematizer.py
: Transform collections of papers into schemapopulator.py
: Tansforms papers + schema into tableeval.py
: Evaluates tablesutils.py
: Various experiment utilities.