arrowheadapps / strapi-connector-firestore Goto Github PK

Higher-complexity relations (many-many, etc.) and components have not been tested yet and may not be working.

For models that do not allow non-native queries (and search), allow a field to be elected for native search.

A search on such a model will fun a query for == on the given field, a primitive kind of search using native native Firestore queries only.

Inequality queries are now natively supported and don't require manual calculations. Upgrade the Firestore dependency and migrate these query types to the native implementation.

https://firebase.googleblog.com/2020/09/cloud-firestore-not-equal-queries.html

I'm not able to npm develop the strapi.

I got an error : Unknown dialect undefined

const path = require("path");

module.exports = ({ env }) => ({
  defaultConnection: "default",
  connections: {
    default: {
      connector: "firestore",
      settings: {
        projectId: "foo-3ef77",
      },
      options: {
        // Connect to a local running Firestore emulator
        // when running in development mode
        useEmulator: env("NODE_ENV") == "development",
      },
    },
  },
});

Currently, components (repeatable) and dynamic zone are stored directly in the document. This does not provide good support for running queries on fields within those components.

• Add an option flattenComponents (global setting and per-model override) wich defaults to true current behaviour.
• On the per-model option, allow to specify a test for selecting which components are flattened.

When component flattening is disabled, store the components as individual documents within a sub-collection of the master document.

While developing with Firebase Emulators, everything is going smoothly. Currently my project development is almost done and I'm working on the deployment, The loading is extremely slow. It took 1.5min to load after signed in. I've tried with my local Strapi with remote firebase and the result is also slow.

Strapi: 3.6.5
strapi-connector-firestore: 3.0.0-alpha.46

Allow a model to be configured as a subcollection of a parent model, by adding a parentModel option.

{
  "options": {
    "parentModel": "users"
  },
  "attributes": {
    "owner": {
      "model": "users"
    }
  }
}

If there is a one-way relation to the parent model, then that reltion will become the partent document of the subcollection. If there is no one-way relation to the parent model, then one will be implicitly created with the name "parent".

If the parent relation is changed, then the document will be deleted from the old subcollection and added to the new subcollection.

Commit 4a1cfa5 introduced use of the non-privileged user in docker install and run, both to allow use of patch-package without unsafe permissions (user needs to be set before install), and as security best-practice (user can be set after install).

While this works for local Docker build, it does not work for remote cloud build using gcloud as reported by @zirho. I could just move the user statement to after the install command but I want to understand why it works for local build but not remote build...

Error:

When using component that have media field always throw error that File model is not found

[2021-07-12T06:43:09.283Z] error Something went wrong in the model `Application::benefit.benefit` with the attribute `benefitItems$meta.icon`
[2021-07-12T06:43:09.285Z] error Error: The collection `File`, used in the attribute `benefitItems$meta.icon` in the model Application::benefit.benefit, is missing from the models
    at Object.getNature (/Users/skyshi/code/node/iris-cms/node_modules/strapi-utils/lib/models.js:79:15)
    at Object.defineAssociations (/Users/skyshi/code/node/iris-cms/node_modules/strapi-utils/lib/models.js:373:26)
    at mountModel (/Users/skyshi/code/node/iris-cms/node_modules/strapi-connector-firestore/lib/model.js:155:22)
    at Object.mountModels (/Users/skyshi/code/node/iris-cms/node_modules/strapi-connector-firestore/lib/model.js:54:21)
    at /Users/skyshi/code/node/iris-cms/node_modules/strapi-connector-firestore/lib/index.js:88:21
    at async Promise.all (index 0)
    at async Object.initialize (/Users/skyshi/code/node/iris-cms/node_modules/strapi-connector-firestore/lib/index.js:55:9)
    at async Object.initialize (/Users/skyshi/code/node/iris-cms/node_modules/strapi-database/lib/connector-registry.js:30:9)
    at async DatabaseManager.initialize (/Users/skyshi/code/node/iris-cms/node_modules/strapi-database/lib/database-manager.js:43:5)
    at async Strapi.load (/Users/skyshi/code/node/iris-cms/node_modules/strapi/lib/Strapi.js:354:5)

dependencies in package.json

    {
    "lodash": "^4.17.21",
    "strapi": "3.6.5",
    "strapi-admin": "3.6.5",
    "strapi-connector-firestore": "^3.0.0-alpha.35",
    "strapi-plugin-content-manager": "3.6.5",
    "strapi-plugin-content-type-builder": "3.6.5",
    "strapi-plugin-email": "3.6.5",
    "strapi-plugin-upload": "3.6.5",
    "strapi-plugin-users-permissions": "3.6.5",
    "strapi-provider-upload-google-cloud-storage": "^3.6.3",
    "strapi-utils": "3.6.5"
  }

How to reproduce

Create content type that have component inside it, and the component have media field.

The Firestore Node.js SDK verision 4.13.0 made read-only transactions availeble. This should be utilised where appropriate to avoid acquiring locks on documents.

Because components are embedded directly in the parent document, no lifecycle hooks are executed for those components.

Also ensure that lifecycle hooks are executed with #17.

Also, confirm that components can even have lifecycle hooks? The default configuration provides only the model JSON.

Migrate to TypeScript for better development and type safety

Hi @brettwillis
Thanks for this project. I have installed this package and changed ./config/database.js accordingly. What are the next steps to run locally and deploy. Can I deploy my strapi thing to firebase or should I use heroku or something like that?

Relations are stored as references in Firestore database. In some situations (when accessing the database natively, not via the Strapi API) it may be convenient to have certain values from the target document stored alongside the entry.

In such a way, key data from the relation would be immediately available without need to fetch the target document.

When trying to edit a role in Users & Permissions Plugin the page briefly loads, then I get "an error occurred" modal and this error in the console:

TypeError: Cannot read property 'reduce' of undefined at Object.getRole (/Users/davide/Sviluppo/App/node_modules/strapi-plugin-users-permissions/services/UsersPermissions.js:200:42) at async Object.getRole (/Users/davide/Sviluppo/App/node_modules/strapi-plugin-users-permissions/controllers/UsersPermissions.js:92:18) at async /Users/davide/Sviluppo/App/node_modules/strapi/lib/middlewares/router/utils/routerChecker.js:79:22 at async /Users/davide/Sviluppo/App/node_modules/strapi-utils/lib/policy.js:68:5 at async /Users/davide/Sviluppo/App/node_modules/strapi/lib/middlewares/parser/index.js:48:23 at async /Users/davide/Sviluppo/Appnode_modules/strapi/lib/middlewares/xss/index.js:26:9 [2020-11-23T18:20:17.225Z] debug GET /users-permissions/roles/2xmxFW0T9DJK0Y6dCYmU (661 ms) 500

Strapi 3.3.3
strapi-connector-firestore ^3.0.0-alpha.22

https://strapi.io/documentation/v3.x/content-api/parameters.html#deep-filtering

In cases where only one single document is fetched by ID in a query (find, update, or delete, where an ID is given), then the request can be optimised by using a field mask so Firstore returns only the single entry in the document (less data, less processing).

Hey, @brettwillis
Thank you a lot for moving forward firestore support for Strapi.
I've found several bugs and wanted to know, how can I help you to solve them.

Firstly, Image from StrapiUploadPlugin always returns array of objects. But the desired behavior — to return array, if Image is type of Multiple and an object if it is of type Single.

Secondly, I think it's problem with parsing nested objects.
Here is a structure:

• Dynamic zone
  Component

  • Repeatable Component

  • Images Array

Instead of returning array of images objects it return array of references.

Here is a fragment from response.

{
    Footer: [
        {
            __component: "shared.footer-default",
            Column: [
                {
                    Images: [
                        "upload_file/3By5RQEedQIGpJeYhnfn",
                        "upload_file/3By5R234525ssdfYhn31"
                    ],
                    Title: "INFORMATION"
                },
                {
                    Images: [
                        "upload_file/3By5RQEedQIGpJeYhnfn",
                        "upload_file/3By5R234525ssdfYhn31"
                    ],
                    Title: "PAYMENT METHODS"
                },
                {
                    Images: [
                        "upload_file/3By5RQEedQIGpJeYhnfn",
                        "upload_file/3By5R234525ssdfYhn31"
                    ],
                    Title: "GUARANTEE"
                },
                {}
            ],
        }
    ],
}

I bootstrapped this codebase from the official strapi-connector-mongoose package at version 3.0.0, because I figured that, because Mongoose is also a NoSQL database, that package would be the best starting point.

https://github.com/strapi/strapi/tree/v3.0.0/packages/strapi-connector-mongoose

Simple operations and relations seem to be working, but most features (particularly components and more complicated relations) have either been only lightly tested, or not tested at all.

To do: see project.

Contributions and pull requests welcome, because I have limited capacity to complete and maintain this package.

See also strapi/strapi#530 and strapi/strapi#5529.

Hi Brett,

I tried for a day, but the deployment still did not work for me.
If you could give me some advice when you have time, it would be very much appreciated.

The deployment itself seems to be working, but both the front-end and back-end content is not being served, and the default FirebaseHosting screen is displayed. Please see the attached images.
No errors are shown in the browser console for FrontEnd.
As for the Backend, the default screen shows "Error loading the Firebase SDK, check the console".

Here is what I did.
// Front end

• ran SET NODE_ENV=production
  (ref: https://stackoverflow.com/a/11928210/13868142)
• ran npm run build
• in firebase.json, change
• "hosting": {"public" : @"public"}
  to
  "hosting": { "public" : "build"}
• ran firebase deploy --only hosting

// Backend

• enabled permissions as stated below
  https://cloud.google.com/build/docs/deploying-builds/deploy-cloud-run#before_you_begin
• in package.json, deleted "strapi-connector-firestore": "../... /" and ran strapi-connector-firestore
• in Dockerfile, commented out User node (I had to do this in order to avoid "Error: EACCES: permission denied, mkdir '/usr/src/app/node_modules'")
• from the cloud run console, add environment variable ADMIN_JWT_SECRET=JWT_token
• ran npm run deploy:backend => Successfully deployed but the browser shows "Error loading the Firebase SDK, check the console."
• thought I need to install firebase sdk, so ran npm install --save firebase under the cloud-run-and-hosting folder
• ran npm run deploy:backend again => still have the same error

Thank you in advance.

When deleting a media, the media files are deleted in storage, but the related firestore document is not removed. It seems it occurs even when the image has no dependency attached to it (i.e is not used anywhere in Strapi)

After manually deleting the firestore docs, it works fine.

The Strapi-side error is:
Error: Operations are not supported on component collections. This connector embeds components directly into the parent document.

Thank you for this amazing work ! I was so excited when I saw someone finally built a Strapi Firestore connector 🚀

I'm trying to run examples/cloud-run-and-hosting locally, but stuck on login screen. Any help would be appreciated.

What I did:

• Run npm install on root and examples/cloud-run-and-hosting
• Changed the projectId in cloud-run-and-hosting/.firebaserc
• Run firebase init on /cloud-run-and-hosting
• Change the projectId in /cloud-run-and-hosting/config/database.js
• Created /cloud-run-and-hosting/.env
  GCP_PROJECT: my-firebase-project
ADMIN_JWT_SECRET=jwt-token (created by running node -e "console.log(require('crypto').randomBytes(64).toString('base64'))" # (all users))
• Added the jwt token to cloud-run-and-hosting/server.js
• When I launched the emulator locally, port 8080 was taken. I changed the port in firebase.json to 8081. Along with this, change the port of server.js to 8082 => Could not start up with error
• Changed the port under root/index.ts, root/index.js, and root/firestore.js to 8081
• The emulator and Strapi application started up successfully
• When I tried to visit the login page (http://localhost:8082/admin/auth/login), I was redirected. I tried to register with my existing credentials then reload
• Login page appeared. I tried to login => 500 error and could not log in

Am I missing something?
I'm also stuck on deploying the example project to the remote, but I'll try some more and if it doesn't work, I'll ask a separate question later.

Related: #11

The Strapi admin font end is designed to count the size of a collection when displaying the list of collection entries (it uses this to determine the number of pages).

A single such query can be disasterous for Firestore cost/quota usage, let alone many such queries during normal usage of the admin front end, because

• Firestore will acrue a read operation for every document that is counted
• If the collection being displayed is large (i.e. the a collection of users, which could be hundreds of thousands or even millions), then this will result in hundreds of thousands or millions of reads billed everytime the list is viewed

Consider also that the /count API could be used or abused by authorized users outside the admin fton end.

Also, this doesn't apply only to the count() API, but also find() and search() where the filters would return a large number of results.

Proposal 1 (fallback limiting)

As a fallback in place of any better solution, I propose to add a maxQuerySize option to the settings (and also allow it to be overridden per model) which will enforce a maximum limit on all queries (including count()).

• This is a fallback to prevent out-of-control quota usage for a single query
• It wouldn't prevent a large amount of quota usage for for multiple queries
• It would cause inconsistent/unexpected behaviour when the user actually wants to reliably count the exact collection size (arguably a bad idea anyway - a counter should be maintained instead)

The documentation would recommend always setting the lowest appropriate limit.

Should there be a default limit? If so, what should it be?

Proposal 2 (disable counting)

Apply a PR or patch to the Strapi font-end so it doesn't try to count collections at all. This would drastically reduce usage.

Perhaps we include a proposed patch in the examples or documetation so that people can apply it if they wish. But a PR to Strapi with to make counting behaviour configurable would be far better.

Alternatively we could add an option to disable the count() API entirely in this connector, which would cause unexpected results in the font-end.

This in itself doesn't limit out-of-control usage for find() and search() APIs, so should be used in conjunction with proposal 1.

Proposal 3 (automatically maintain a counter)

For every collection maintain a counter of the number of documents (see https://stackoverflow.com/a/49407570/1513557). This would only work for counting the total number of documents without any filter applied an would immediately become useless when a filter is applied.

Error 👇

TypeError: Cannot set property privateAttributes of #<Object> which has only a getter at Function.assign (<anonymous>) at mountModel (E:\WorkSpace\JS\COMPANY PROJECTS\kinderact\node_modules\strapi-connector-firestore\lib\model.js:166:26) at Object.mountModels (E:\WorkSpace\JS\COMPANY PROJECTS\kinderact\node_modules\strapi-connector-firestore\lib\model.js:56:21) at E:\WorkSpace\JS\COMPANY PROJECTS\kinderact\node_modules\strapi-connector-firestore\lib\index.js:82:21 at async Promise.all (index 0) at async Object.initialize (E:\WorkSpace\JS\COMPANY PROJECTS\kinderact\node_modules\strapi-connector-firestore\lib\index.js:49:9) at async Object.initialize (E:\WorkSpace\JS\COMPANY PROJECTS\kinderact\node_modules\strapi-database\lib\connector-registry.js:30:9) at async DatabaseManager.initialize (E:\WorkSpace\JS\COMPANY PROJECTS\kinderact\node_modules\strapi-database\lib\database-manager.js:43:5) at async Strapi.load (E:\WorkSpace\JS\COMPANY PROJECTS\kinderact\node_modules\strapi\lib\Strapi.js:354:5) at async Strapi.start (E:\WorkSpace\JS\COMPANY PROJECTS\kinderact\node_modules\strapi\lib\Strapi.js:196:9)

The FieldOperation (similar to the native FieldValue) class is currently a private API, so this is low priority. But we may wish to make FieldOperation a public API, at which point this will be a problem.

If the user runs a direct transaction (model.runTransaction(...)) and uses FieldOperation to atomically update a relation attribute, then the RelationHandler will crash - it cannot yet handle FieldOperation instances.

See:

Currently, search is implemented in one of three ways

• Not allowed (default)
• Prefix-like search on a single attribute
• Full manual search (fetching queries in chunks, or natuarally for flattened collections)

This proposal is to introduce a seachDelegate option which will be used to delagate search to an external service (e.g. Algolia).

This delegate can also be used for count queries, where counting may be costly in Firestore.

When performing a filter under "Collection Types" i get an internal error. Under headers i can see the right filter being used. The fields is of type Int:

Filter

Header

calendar-date?page=1&pageSize=10&_sort=recId:ASC&_where[0][calendarDay]=1

NOTE: querystring query works fine see examle below:

api/calendar-date?calendarDay=${calendarDay}

HTTP Response

{"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred"}

Header

UI Error message

Hi Guys,

I'm using the following media library - https://www.npmjs.com/package/strapi-provider-upload-google-cloud-storage

Uploading files works and are also available to the public via the bucket URL. But under media library, no images display, however, the page recognizes that there is content available by displaying the page count underneath.

I am running cloudrun and under Cloud firestore, i can see a populated table "upload_file".

Any help is much appreciated.

Currently BigInt's are stored in Firestore as strings, which means that comparison operators are not robust.

For example, in string comparison: '123' < '1221' is (incorrectly) false.

I'd like to use realtime functionality of Firestore but couldn't figure out how to fetch data directly from the database without using API endpoints. Is it possible?

Considering that Firestore charges based on read and write operations, there may be some opportunity to reduce usage costs and increase performance.

Proposal:

• Add a documentise option for each model (or an documentiseAllExcept option)
• Documentisation combines all documents in a collection/table into an array of objects in a single document

This proposal would be effective for models that are:

• Bounded (i.e. finite number of documents, like the collections for Strapi configuration and permissions/roles, but not collections like users which may be unbounded) so that they fit within the document size limit
• Queried often, or queried in their entirety (reduce many document read operations into a single read operation)

Currently, a write operation is performed immediately on startup to ensure the existence of the document containing flattened collections. Because a read operation is cheaper than a write operation, it is better to

• Read first to avoid a write operation on every subsequent startup
• Perhaps do this lazily when an entity is first created, rather than on startup

Currently there is a discrepancy between the result that is returned from query API calls and the data that is written to Firestore.

• This is because coversions and coercions are performed on the way to Firestore but not performed on the returned data
• This includes adding IDs to components
• On update operations, the returned data does not reflect the entire document, only the (potentially partial) input data

Proposal:

• Migrate type coercion into a coerceToModel() function which operates on incoming data
• Lightweight transform in document converter, which handles conversion of custom types to Firestore-compatible type
• Lightweight transform in document converter, which rebuilds certain custom types from Firestore-native
• No need to coerce/convert to API result, as it is handled by JSON.stringify() with toJSON instance method

To be determined:

• Should we write to Firestore and then read back the result for API response
  • Pros: Less processing
  • Cons: converters operate on partial data, write->read is non atomic, the data could have changed in Firestore before it is read back
• Or should we read from Firstore, combine the data manually, then write the modified data to Firestore and also return it
  • Pros: Full data in converters, enables existence check for flat collections, read->write can be an atomic operation
  • Cons: More processing, manual merging

I run export GOOGLE_APPLICATION_CREDENTIALS='./config/Timetracking inner test-97f38b6dda77.json' && strapi develop

and I got:

warn Your application doesn't have a super admin user. Bootstrap function in admin failed error Error: 3 INVALID_ARGUMENT: inequality filter property and first sort order must be the same: roles and __key__ at Object.callErrorFromStatus (/path/node_modules/@grpc/grpc-js/build/src/call.js:31:26)

"dependencies": { "knex": "<0.20.0", "sqlite3": "^5.0.0", "strapi": "3.2.5", "strapi-admin": "3.2.5", "strapi-connector-bookshelf": "3.2.5", "strapi-connector-firestore": "^3.0.0-alpha.21", "strapi-plugin-content-manager": "3.2.5", "strapi-plugin-content-type-builder": "3.2.5", "strapi-plugin-email": "3.2.5", "strapi-plugin-upload": "3.2.5", "strapi-plugin-users-permissions": "3.2.5", "strapi-utils": "3.2.5"

Newly added in Strapi 3.1.x
See https://strapi.io/documentation/v3.x/content-api/parameters.html#complexe-queries

When I am running strapi app in development mode then it's creating(adding new permissions) in users-permissions_role collection.