Giter Club home page Giter Club logo

syncairtablepostgres's Introduction

Sync Airtable to Postgres

This is a utility to sync tables from Airtable to Postgres

Requirements

Docker

Docker Compose

Installation

Docker

On a command line run

curl -fsSL https://test.docker.com -o test-docker.sh

sh test-docker.sh

Docker Compose

On a command line run

sudo wget --output-document=/usr/local/bin/docker-compose "https://github.com/docker/compose/releases/download/$(wget --quiet --output-document=- https://api.github.com/repos/docker/compose/releases/latest | grep --perl-regexp --only-matching '"tag_name": "\K.*?(?=")')/run.sh"

sudo chmod +x /usr/local/bin/docker-compose

sudo wget --output-document=/etc/bash_completion.d/docker-compose "https://raw.githubusercontent.com/docker/compose/$(docker-compose version --short)/contrib/completion/bash/docker-compose"

Usage

0. Clone the Repostiory

Run the following on your command line

git clone https://github.com/sjain07/SyncAirtablePostgres.git
cd SyncAirtablePostgres

1. Edit db-params.json

Add your database connection parameters

2. Add the list of tables you want to sync and your api key to airtable.json

You have to add the BaseId and Table Name

To get Base Id:

a. Go to Airtable API

b. Click on the Base of your choice, it will take you to a url looking like:

https://airtable.com/appnUr44xd9unezyt/api/docs#curl/introduction

appnUr44xd9unezyt is your BaseId and within that BaseId pick the table you want (like Epics) to sync

c. add to airtable.json

"bases": {
    "appnUr44xd9unezyt" : ["Epics"]
}

d. add your Airtable API Key to airtable.json

https://support.airtable.com/hc/en-us/articles/219046777-How-do-I-get-my-API-key-

3. Run with Docker Compose

docker-compose up -d --build

It will sync your Airtable Table to the Postgres DB that you specified every 5 minutes.

Use Cases

We at Nanonets use it to Sync data about Scrum, OKRs, Failures, Errors, CRM that we store in Airtable. We also support our customers who want data written back to Airtable from Invoices they send to us for processing Invoice Processing

Limitations

  1. Currently only supports numbers and dates.
  2. Aggressively changes field and table names to avoid Postgres keywords, this could be much lighter

To DO

  1. Add support for JSON, Lists, Boolean etc. Currently only supports numbers and dates.
  2. Autodetect Bases and Tables (Requires Airtable Meta API Access)

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

License

MIT

syncairtablepostgres's People

Contributors

sjain07 avatar

Stargazers

Andhi Jeannot avatar stoneAge avatar Leon Lowitzki avatar

Watchers

James Cloos avatar  avatar

Forkers

isabella232

syncairtablepostgres's Issues

Add id column

Looking through the python I see the record_id variable. Is it possible to add an "id" column with the record_id value? Seems like this would be a very key component in differentiating rows. Thanks!

Limitations

I am very interested in using this docker image. When you say it only supports numbers and dates does that mean that if you have columns with strings or varchar format that it ignores them?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.