Giter Club home page Giter Club logo

integrify's Introduction

πš’πš—πšπšŽπšπš›πš’πšπš’

Build & Test Code Coverage Status npm package Mentioned in Awesome Firebase Firebase Open Source

🀝 Enforce referential and data integrity in Cloud Firestore using triggers

Introductory blog post

Usage

// index.js

const { integrify } = require('integrify');

const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp();
const db = admin.firestore();

integrify({ config: { functions, db } });

// Automatically replicate attributes from source to target
module.exports.replicateMasterToDetail = integrify({
  rule: 'REPLICATE_ATTRIBUTES',
  source: {
    collection: 'master',
  },
  targets: [
    {
      collection: 'detail1',
      foreignKey: 'masterId',
      attributeMapping: {
        masterField1: 'detail1Field1',
        masterField2: 'detail1Field2',
      },
    },
    {
      collection: 'detail2',
      foreignKey: 'masterId',
      attributeMapping: {
        masterField1: 'detail2Field1',
        masterField3: 'detail2Field3',
      },

      // Optional:
      isCollectionGroup: true, // Replicate into collection group, see more below
    },
  ],

  // Optional:
  hooks: {
    pre: (change, context) => {
      // Code to execute before replicating attributes
      // See: https://firebase.google.com/docs/functions/firestore-events
    },
  },
});

// Automatically delete stale references
module.exports.deleteReferencesToMaster = integrify({
  rule: 'DELETE_REFERENCES',
  source: {
    collection: 'master',
  },
  targets: [
    {
      collection: 'detail1',
      foreignKey: 'masterId',

      // Optional:
      isCollectionGroup: true, // Delete from collection group, see more below
    },
  ],

  // Optional:
  hooks: {
    pre: (snap, context) => {
      // Code to execute before deleting references
      // See: https://firebase.google.com/docs/functions/firestore-events
    },
  },
});

// Automatically maintain count
module.exports.maintainFavoritesCount = integrify({
  rule: 'MAINTAIN_COUNT',
  source: {
    collection: 'favorites',
    foreignKey: 'articleId',
  },
  target: {
    collection: 'articles',
    attribute: 'favoritesCount',
  },
});

Deploy to Firebase by executing:

$ firebase deploy --only functions

Rules File

Alternately, rules can be specified in a file named integrify.rules.js.

// index.js

const { integrify } = require('integrify');

const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp();
const db = admin.firestore();

integrify({ config: { functions, db } });

// Rules will be loaded from "integrify.rules.js"
module.exports = integrify();
// integrify.rules.js

module.exports = [
  {
    rule: 'REPLICATE_ATTRIBUTES',
    name: 'replicateMasterToDetail',
    // ...
  },
  // ...
];

Collection Groups (isCollectionGroup)

Firestore allows searching over multiple collections (a.k.a. collection group) with the same name at any level in the database. This is called a collection group query.

Integrify allows you to replicate tracked master attributes into (optionally) collection groups linked by a foreign key using the isCollectionGroup parameter (see above) in the REPLICATE_ATTRIBUTES rule. Similarly, you can delete references in a collection group (instead of just a collection) using the isCollectionGroup in the DELETE_REFERENCES rule.

Note: You need to first create the appropriate index to be able to use Collection Group Queries. The first time you attempt to use it, Firebase will throw an error message with a link which when clicked will prompt you to create the appropriate index. For example:

The query requires a COLLECTION_GROUP_ASC index for collection detail1 and field masterId. You can create it here: https://console.firebase.google.com/project/integrify-dev/database/firestore/indexes/single_field?create_exemption=ClNwcm9qZWNxxxxxx3RlcklkEAE

For more help, see here.

integrify's People

Contributors

anishkny avatar azure-pipelines[bot] avatar dependabot-preview[bot] avatar dependabot[bot] avatar depfu[bot] avatar greenkeeper[bot] avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

integrify's Issues

An in-range update of ava is breaking the build 🚨


🚨 Reminder! Less than one month left to migrate your repositories over to Snyk before Greenkeeper says goodbye on June 3rd! πŸ’œ πŸššπŸ’¨ πŸ’š

Find out how to migrate to Snyk at greenkeeper.io


The devDependency ava was updated from 3.8.1 to 3.8.2.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

ava is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details

Release Notes for 3.8.2
  • Fix bad dependency fd92b4a
  • Use configured depth limit for diffs in assertion failures, thanks @bunysae! a5385a4

v3.8.1...v3.8.2

Commits

The new version differs by 6 commits.

  • dace976 3.8.2
  • 05c3158 Prep for release
  • fd92b4a Correctly use picomatch dependency
  • bec7c9e Configure import/no-unresolved linter rule
  • c8f31a3 Test AVA using AVA
  • a5385a4 Use configured depth limit for diffs in assertion failures

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

[REPLICATE_ATTRIBUTES] should check if tracked value has changed before updating

In the code whether the tracked attributes value has changed or not is not checked. This results in writes even if there's no change to tracked attributes.

// Check if atleast one of the attributes to be replicated was changed
      let relevantUpdate = false;
      Object.keys(newValue).forEach(changedAttribute => {
        if (trackedMasterAttributes[changedAttribute]) {
          relevantUpdate = true;
        }
      });

We should be comparing old values of tracked attributes to new values, like

// Check if atleast one of the attributes to be replicated was changed
      let relevantUpdate = false;
      Object.keys(newValue).forEach(changedAttribute => {
        if (trackedMasterAttributes[changedAttribute] && newValue[changedAttribute] !== oldValue[changedAttribute]) {
          relevantUpdate = true;
        }
      });

Target collection params in query

Would it be possible to use the parameters from the source collection in the target collection? I was thinking of being able to use the parameters in the source to the target collection.

source: {
    collection: 'master/{someId}/detail',
  },
  targets: [
    {
      collection: 'details/{someId}/more_detail',
      foreignKey: 'masterId',
    },
  ],

Very basic example but the idea to use the someId in the target collection query.

An in-range update of firebase-admin is breaking the build 🚨


🚨 Reminder! Less than one month left to migrate your repositories over to Snyk before Greenkeeper says goodbye on June 3rd! πŸ’œ πŸššπŸ’¨ πŸ’š

Find out how to migrate to Snyk at greenkeeper.io


The devDependency firebase-admin was updated from 8.11.0 to 8.12.0.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

firebase-admin is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details

Release Notes for Firebase Admin Node.js SDK v8.12.0

New Features

  • feat(auth): Add bulk get/delete methods (#726)

Miscellaneous

  • [chore] Release 8.12.0 (#878)
  • Bump jquery from 3.4.1 to 3.5.0 (#873)
  • Fixed lint (#868)
  • Refines UserRecord.customClaims type. (#866)
  • Generate camelcase doc paths for machineLearning module (#863)
  • Fix typo in release.yml (#862)
FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.


Your Greenkeeper Bot 🌴

Throw better error for incorrect config

I get this error:

TypeError: Cannot read property 'collection' of undefined
    at /srv/node_modules/integrify/lib/rules/replicateAttributes.js:61:18
    at Array.forEach (<anonymous>)
    at /srv/node_modules/integrify/lib/rules/replicateAttributes.js:47:22
    at cloudFunctionNewSignature (/srv/node_modules/firebase-functions/lib/cloud-functions.js:105:23)
    at /worker/worker.js:825:24
    at <anonymous>
    at process._tickDomainCallback (internal/process/next_tick.js:229:7)

I have the following Firebase Cloud Function running in Node 8 environment with Typescript code:

import * as functions from 'firebase-functions';
import * as admin from 'firebase-admin';
const { integrify } = require('integrify'); 

admin.initializeApp();
const firestore = admin.firestore();
integrify({ config: { functions, firestore } });

exports.inventory_item_replicate = integrify({
    rule: 'REPLICATE_ATTRIBUTES',
    source: {
      collection: 'inventory_item',
    },
    targets: [
      {
        collection: 'cart',
        foreignKey: 'id',
        attributeMapping: { 
            'barcode': 'barcode',
            'category': 'category', 
            'image': 'image', 
            'name': 'name', 
            'price': 'price',
            'stock': 'stock',
            'type': 'type',
        },
      },
    ],
  }); 

The replication function seems to fire correctly after an update on the original collection, as I can see from the logs, and the error is logged directly afterwards:

integrify: Detected update in [inventory_item], id [QmNKvERKZI3HFEyKz3Ft], 
new value: { ...
...
...
...
integrify: On collection [cart], applying update: { ....

Question: 'REPLICATE_ATTRIBUTES' for > 10.000 records

Hi,

I hope this is the right "forum" for this question; if not, please feel free to point me to the correct one and I'll post this question there.

I need to replicate attributes from one collection to another collection; however, the number of records that are affected are > 10.000. I was having problems before with Firebase and lots of concurrent writes to the same collection. I was wondering if this library is addressing this problem; from reading the source, it seems all update queries are triggered in parallel.
Do you have any thoughts or plans for this?

Support using in TypeScript (TS2769: No overload matches this call.)

@anishkny It appears to be working at least in the emulator suite so far.

TypeScript is complaining, however, about the overloads generated for the integrify function.

It seems the Rule interface only specifies the actual rule field (and an optional name).

Example:

export const replUserAttrs = integrify({
	rule: 'REPLICATE_ATTRIBUTES',
	source: {
		collection: 'users',
	},
	targets: [
		{
			collection: 'profiles',
			foreignKey: 'userId',
			attributeMapping: {
				role: 'role',
			},
		},
	],
});

Error:

TS2769: No overload matches this call. Β Β 
Overload 1 of 3, '(config: Config): null', gave the following error. Β Β Β Β 
    Argument of type '{ rule: string; source: { collection: string; }; targets: { collection: string; foreignKey: string; attributeMapping: { role: string; }; }[]; }' is not assignable to parameter of type 'Config'. 
    Object literal may only specify known properties, and 'rule' does not exist in type 'Config'. Β Β 
Overload 2 of 3, '(rule: Rule): IntegrifyFunction', gave the following error. Β Β Β Β 
    Argument of type '{ rule: "REPLICATE_ATTRIBUTES"; source: { collection: string; }; targets: { collection: string; foreignKey: string; attributeMapping: { role: string; }; }[]; }' is not assignable to parameter of type 'Rule'. Β Β Β Β Β Β 
    Object literal may only specify known properties, and 'source' does not exist in type 'Rule'.

Fix:

I see you extend the interface for each type of rule, you might have to invert the definition. i.e:

type Rule = { name?: string } & (ReplicateAttributesRule | MaintainCountRule | DeleteReferencesRule);

Then add each rule field to each interface itself. I think this will let TypeScript infer which type of rule you're using too and provide type hints for that specific interface which is nice.

Here's a TypeScript Playground of what I'm talking about

Originally posted by @dhkatz in #116 (comment)

replicate to nested destination

Hell @anishkny,

Love this architecture and approach, using it already on my latest project!

Question - is it possible (or hard to add) the ability to target a nested property for replication?

My use-case is a "users" collection that is essentially private, and a "workspaces" collection that has an array of memberIds.

I'd like to use integrify to update a memberInfo object on workspaces, keyed by uid, with just the user's name and avatar.

So the source is a users collection and target looks something like:

workspaces: [{
   id: "some guid",
   memberIds: ["firebaseUid1","firebaseUid2"], <--- this is currently how users are added/removed to workspaces
   memberInfoCache: {  <--- I'd like to add meta info cache to avoid complicated permissions on accessing /users/
       "firebaseUid1": { <--- the foreign key
           "photoURL": "replicated from users/firebaseUid1/photoURL",
           "name": "replicated from users/firebaseUid1/name"
       },
       ...
    }
}]

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.