formkiq / formkiq-core Goto Github PK

On POST /documents/{documentId}/tags, the following results in a 500 - Internal Service Error:
{"tags":[{"key": "author", "value": "William Shakespeare"},{"key": "author", "value": "Kevin Bacon"}]}

Workaround is to submit the key once and the values in an array:
{"tags":[{"key": "author", "values": ["William Shakespeare", "Kevin Bacon"]}]}

It's possible to allow this request with the same key, but my preference would be to return a clear error message stating that the key can only be included once in the body.

Suggestion:
400 - "Tag key can only be included once in body; please use 'values' to assign multiple tag values to that key"

Document Content-Length gets set to null when using PATCH /documents/{documentId}

AWS has added a Lambda extension for retrieving SSM parameters / secets that could be used instead of the SsmService currently being used.

https://aws.amazon.com/about-aws/whats-new/2022/10/aws-parameters-secrets-lambda-extension

There is an "insertedDate", but it would be good to also provide the "lastModifiedDate".

Steps to reproduce:

1. Upload a PDF file -> POST /documents/upload
2. Go to versions; expected result is to have only one version (current)
3. Click "Upload new PDF" and select a new version of the file -> GET /documents/{id}/upload
4. Document uploads but no new version is created, i.e., GET /documents/{id}/versions responds with empty array for documents

If two documents exist with the same path, deleting one of those documents will succeed, but a new folder will be created with the following naming pattern:
{PATH} ({DOCUMENT_ID).{FILE_EXTENSION}

Currently, if I pass in a number of tags with the body payload, I get results that match any of those key/value pairs, even if only one of those tags matched. This creates a result set that is more broad that I had expected. It would be nice if all given tags were required to match. This would allow the eqOr property to be a true OR while all the given tags are AND'd together. In OpenSearch/elasticsearch syntax, this might look something like the following.

Assume I'm sending in two tags. The first "userid" has an "eq" property of 100, and a second tag "itemid" is using the "eqOr" property with a value of ["4", "7", "13"].

bool: {
    must: [
        {
            bool: {
                should: [{
                    key: "userid",
                    value: "100"
                }]
            }
        },
        {
            bool: {
                should: [
                    {
                        key: "itemid",
                        value: "4"
                    },
                    {
                        key: "itemid",
                        value: "7"
                    },
                    {
                        key: "itemid",
                        value: "13"
                    }
                ]
            }
        }
    ]
}

This query requires that any returned result includes a userid tag with value 100, and an itemid tag with a value of 4, or 7, or 13.

curl "https://ojjn7xmet4.execute-api.us-east-2.amazonaws.com/documents/5c1109ce-6d89-4267-9edd-99a4e99597bf?siteId=default" -H "Authorization: {AUTH}" -d "{\"path\":\"user25.json\",\"tags\":[{\"key\":\"goat\",\"value\":\"text\"}]}"

Took an existing document, renamed it, but created a second entry in the DynamoDB results with the same documentId.

Allow "modifiable" field on metadata to allow making metadata unchangeable

There's a number of full text search engines that could be support in addition to OpenSearhc

• https://github.com/valeriansaliou/sonic
• https://jakejscott.com/full-text-search-for-dynamodb-using-lambda-efs-tantivy-and-rust

There is a need to allow custom document meta data in addition to Tags. Document meta data would be different to document tags in the following ways:

• Meta Data is versioned (and stored in a different DynamoDB Table)
• Meta Data attributes can have a "type" -> TBD on what kind of types there would be but things like TIMESTAMP, TEXT (max 1000 characters), Date, etc
• would need an endpoint like POST /documents/{documentId}/versionRestore/{versionId} to restore to a previous version
• In addition would need an endpoint to DELETE version like /documents/{documentId}/versions/{versionId}

Rules around deleting a versions would be controlled by #68

In order to create a UI to select a folder as a destination for moving a document or folder, it would be more efficient to be able to retrieve only folders in POST /search, rather than retrieving documents as well and needing to filter.

e.g.
{"query":{"meta":{"folder":""}},"responseFields":{"tags":["sysFavoritedBy","sysSharedWith","sysDeletedBy","english","marketing","path","Test","untagged"]}}

{"query":{"meta":{"folder":"Version/"}},"responseFields":{"tags":["sysFavoritedBy","sysSharedWith","sysDeletedBy","english","marketing","path","Test","untagged"]}}

If you use the Document POST API to create a new document record, but do not use the resulting URL to actually upload the file to S3 (or if this operation fails), the original Document record cannot be deleted using the DELETE Document API. The response is "Document {GUID} not found."

When replacing a document file using GET /documents/{id}/upload and sending the file via PUT to the signed URL, no new version is created for the updated document.

If an object fails to be uploaded using GET documents/upload, the document still exists in DynamoDB as metadata; we need to determine the best way of removing or ignoring these metadata entries after some time (a sort-of timeout on the upload), as this leads to unexpected results when retrieving the most recent documents.

Using either gives the following message: Invalid version id specified

In the /versions, add a "modules" string array to list which modules are enabled in the formkiq installation

There is a new enterprise antivirus module that needs support to be added in actions.

Patch does not update or add tags added to the payload.

We are able to rename documents with PATCH /documents/{id} by specifying a new path, but there is currently no way to rename folder index items. Ideally, we would have an endpoint such as this: /indices/folder/rename that accepts the full path parameter and a folder name (no path) and renames the folder.

Conversely, we could allow a new full path that could function as a "mv" command, but this functionality is already available using POST /indices/folder/move

Since "path" is stored on the document level as well. Potentially would have to update all the documents "path" as well.

If you use the content url that results from the Get Document Content or Get Document URL endpoints to download the file, the filename will always be the GUID representing the document ID. There doesn't seem to be a good way to force the downloaded file's name to match the original "path" or filename parameter.

java.io.IOException: unable to update Fulltext
at com.formkiq.stacks.api.handler.DocumentTagsRequestHandler.updateFulltextIfInstalled(DocumentTagsRequestHandler.java:286)
at com.formkiq.stacks.api.handler.DocumentTagsRequestHandler.post(DocumentTagsRequestHandler.java:218)
at com.formkiq.aws.services.lambda.AbstractRestApiRequestHandler.callHandlerMethod(AbstractRestApiRequestHandler.java:166)
at com.formkiq.aws.services.lambda.AbstractRestApiRequestHandler.processRequest(AbstractRestApiRequestHandler.java:411)
at com.formkiq.aws.services.lambda.AbstractRestApiRequestHandler.processApiGatewayRequest(AbstractRestApiRequestHandler.java:349)
at com.formkiq.aws.services.lambda.AbstractRestApiRequestHandler.handleRequest(AbstractRestApiRequestHandler.java:286)

It would be great if Documents could have permissions based on the https://www.openpolicyagent.org/

Document "locking" could also be implemented using this.

If a document fails to process, it should be moved to a dead-letter S3 bucket to allow for reprocessing or tracking which documents failed to load.

When the CloudFormation Stack is removed, the Console Installer doesn't delete the console

Since we have a Delete Folder for removing a specific folder from its index with an index key, is it possible to have a DELETE for the tag index items, referenced by tag key?

The documentation claims the "limit" query param's default value is "null", but this endpoint only returns a maximum of 10 tags in the absence of the limit param.

When submitting a PUT to change version using version key, the response comes back ok, but retrieving the document without a version does not retrieve the version that had been specified as current using this endpoint.

PUT /documents/8c5f826f-3bec-4079-a244-78489e5fa1e3/versions?siteId=default

{"versionKey":"document%232022-11-20T20%3A00%3A00%2B0000%23v3"}

Response 200:
{"message":"Set Document to version"}

When I pass in a key of "Test" it deletes all tags that start key starts with "Test".
When I pass a key containing a space it throws a 500 error

When attempting to GET a non-existent Tag Schema, a 500 is returned:
{"message":"Internal Server Error"}

Upon submitting a DELETE request for a Tag Schema, the message does not mention that the deletion has occurred:
{"tagSchemaId":"1d65ce16-c82e-43c7-b107-d54ab1267e68"}

Currently, S3 signed URL defaults to content-disposition:attachment;filename=BLAH, and this does not always allow for rendering of files within the browser.

If we had a parameter such as ?inline=true on GET /documents/{id}/url that provided content-disposition:inline;filename=BLAH, this may allow us to render the documents inline (as the content type allows) more easily

for GET /documents/{documentId}/ocr rename the contentUrl to contentUrls which is an array of strings.

• Add Running Status
• Add Start / Stop Timestamps for each Step
• Add Number of action Retries before failure

Observed with tags property.

Example:

PATCH /documents/{id}
{
  "tags": [
    {
      "key": "author",
      "value": "Bacon"
    }
  ]
}

An upload URL is returned (to allow document itself to be updated), but the metadata (at least not tags) is not updated.

The DocumentUpdate update all document fields when it only needs to update Content-Type, Content-Length, and Checksum. If another process updates document data before the DocumentUpdate runs it could be overwritten.

POST /documents/{documentId}/action - adds document action but the action is never processed.

Not able to create create AWS cloudformation stack with latest AWS SAM CLI release.
Somehow gave error for S3 bucket creation.
Tried with Linux and Mac versions.
Once I took earlier version - Feb 2021, it started to work.

As folders hold a list of documents within, moving a document from one folder to another would require an API call to remove the document from the source folder and add to the destination folder.

Example endpoint:

POST /documents/{id}/move
{"sourceFolder": "Client ABC", "destinationFolder": "Client ABC/Client ABC Workspace"}

NOTE: if we want to use move as the resource, we can also include siteId in future, allowing a move between folders, sites, or both, e.g., sourceSiteId

Example URL: /documents/8c5f826f-3bec-4079-a244-78489e5fa1e3/url?versionKey=document%25232022-11-20T19%253A21%253A51%252B0000%2523v2&siteId=default&inline=false

As each version in this test has its version marked within the file (among other differences), it's clear that this call is only retrieving the most current document.

I am able to reliably get results from the POST document search endpoint when providing a body parameter, but as soon as I add the "limit" query parameter, it returns with a 500 error.

Furthermore, the documentation says that the limit parameter has a default of "null", but the results are always limited to 10 in the absence of the limit query param.

After deleting the "path" tag for a document, added a new "path":

{"key":"path","value":"fw4.pdf"}

Response is 200 with a success message:

{message: "Created Tag 'path'."}

But when retrieving all tags for the document, tag "path" is missing.

The welcome email could use some sprucing up and better branding.

The FormKiQ version installed is displayed in the CloudFormation Outputs, but it would be nice if it was also displayed in the CloudFormation template description, so it would be easier to see the FK version of each stack.