Giter Club home page Giter Club logo

gittables's Introduction

GitTables

Code for extracting, parsing and annotating tables from GitTables (https://gittables.github.io), a corpus of 1.7M tables extracted from GitHub.

Quick links

Purpose

The code in this repository resemble the procedures for:

  • Extracting CSV files from GitHub based on query topics from WordNet.
  • Parsing CSV files to Pandas tables.
  • Annotating the tables with syntactic and semantic matching.
  • Writing the table and annotation metadata to Parquet files.

Installation

Before running any of the code, a few steps need to be executed:

  • From the root directory, install the gittables package using pip install ..
  • Install the dependencies in your environment with e.g. pip using pip install -r requirements.txt.
  • Add your personal GitHub username and token to the settings.toml file.
  • In case you run into issues with the FastTtext download (see scripts/table_annotation.py) you should download the proper FastText model yourself here (i.e. the binary file from crawl-300d-2M-subword.zip). Make sure the file is (re)named to cc.en.300.bin and is placed in the scripts/ directory.

Usage

The pipeline consists of two main stages, of which the main scripts are stored in scripts/, run these scripts from the root directory. Log files of the extraction and annotation process are written to the logs/ directory.

Warning: running the code as-is is time consuming as it builds many queries for extracting many files.

Extracting CSV files

The CSV files can be extracted by running python scripts/file_extraction.py.

This step will use the GitHub code search API and request module to extract CSV files based on topics from WordNet (WordNet will be downloaded automatically).

In each topic directory within the table_collection directory, you will find the raw CSV files and tables in csv_files/.

If you want to get tables for a custom list of topics, you can modify the file_extraction.py script by setting the custom_topics argument of the set_topics method to a specified list of query topics, e.g. ['apple', 'pie', 'nut']. This list will then be used to build the table collection repository, instead of the topics from WordNet.

Parsing and annotating tables

When the CSV files for (some of) the topics are extracted, these files can be parsed to a table and annotated with column semantics by running python scripts/table_annotation.py.

The ontologies used for the annotation are written to the ontologies/ directory for future reference. The ontologies used for constructing GitTables 1.7M can be downloaded from our website https://gittables.github.io.

Issues and contributions

Contributions to speed up the processes are appreciated. If you run into issues or have question, please file them through GitHub here.

gittables's People

Contributors

madelonhulsebos avatar tdoehmen avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

Forkers

tdoehmen

gittables's Issues

syntactic processing

In table_annotator.py on line 632, we process the original column name to match the name against the ontologies of DBpedia and Schema. The original column names are processed using the code below:

cleaned_table_columns = [
                re.sub(r"[_-]", " ", " ".join(
                    re.findall("[0-9,a-z,.,\"#!$%\^&\*;:{}=\-_`~()\n\t\d]+|[A-Z](?:[A-Z]*(?![a-z])|[a-z]*)", col)
                )).lower() for col in table_columns.copy()
            ]

I wonder if the first " " inside the re.sub() call, currently a space, should be converted to "", an empty string. Because we already match the _- in the regex inside findall, which in turn means the _ or _ is replaced by a space using " ".join(). This join keeps the matched _ or - in the string, which in turn means the _ or - is replaced by another " " using the re.sub(r"[_-]", " ", ...).

For example:
"Team-Name" would be converted into "team name", 2 spaces between 'team' and 'name'. Is this desired behaviour, am I missing something? Or is this a bug?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.