arro / mongo-airtable Goto Github PK
View Code? Open in Web Editor NEWKeep Airtable tables synced with local MongoDB collections
License: MIT License
Keep Airtable tables synced with local MongoDB collections
License: MIT License
I'm considering to use your tool to pull our entire Airtable base to our Staging webserver's MongoDB, from which I will then generate pages for our website. We then deploy all this static content as complete stand-alone ZIP to our Live web server.
I already use the official Airtable API JS client, but with your tool I hope to avoid problems with the Airtable API limitations (5 requests per second).
This won't be a big improvement to using the Airtable API directly, because I can delay the API requests and we are OK with waiting 10-30min. However, I'm wondering if at some point I can have the MongoDB directly on our Production and threfore skip generating the static pages on Staging server and then read/write dynamic content (e.g. Orders, User profiles) directly from the MongoDB, which in background will be synced by your tool every 10min or so.
Are you using it for scenarios like this?
Or only to sync our offline copy of the base, like for local development for example?
Asking because I've read this:
The problem is that sometimes you don't have internet access, so pulling and saving data using their API isn't feasible.
Thanks for your input!
I tried mongo-airtable to pull my base to mongoDB, my Airtable base has 12 tables and more than 6500 records...
it took 190 seconds to finish pulling and inserting
I tried tp modify some functions to make it work async, all tables start pulling at once but as soon as any table finish pulling it will start inserting into mongodb directly ... this improved the performance and took 92 seconds instead of 190 seconds
this still too much as I am intending to fire this syncing process every 5 minutes intervals or less to always keep mongodb synchronized with airtable
here is my edits to reach 92 secs... added this function to airtable.js:
export async function initialPullAndInsert(config) {
for (const table_to_sync of config.sync) {
pullTable({
...table_to_sync,
auth_key: config.auth.airtable
}).then(() => {
console.log(`end sync of ${table_to_sync.primary}`)
const last_pulled_filename = path.resolve(`${__dirname}/../build/last-pulled.txt`);
writeFile(last_pulled_filename, moment().format('ddd MMM D YYYY h:mm A ZZ'), 'utf-8').then(() => {
initialInsertTable(table_to_sync);
})
})
}
}
and added this to mongodb.js:
export async function initialInsertTable(table_to_sync) {
const { primary, collection, database } = table_to_sync;
await putIntoDB({
primary,
collection,
database
})
}
and in pull.js I called await initialPullAndInsert(config);
When I delete any record in airtable and then call "npm run update", it's not deleted in mongodb
also if I delete a field in airtable and call "npm run update", the field is not deleted inside mongodb
Is it possible to not pull hidden fields from an airtable view? This would allow one collection to be used for more API calls and act more dynamically.
Hi
I ran the tests and all 6 passed. When I run "npm run push", I get the follow error. I tried both simple and complex mongodb collections
(node:8198) UnhandledPromiseRejectionWarning: TypeError: (0 , _mongodb.lookForChanges) is not a function
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.