Add CDN field to the S3 configuration about pocketbase HOT 14 CLOSED

@danibjor No, I don't think it is disk related. The issue you are experiencing is only with the realtime events because they are fired on success SaveRecord(), before the actual files upload. The problematic part of the code is https://github.com/pocketbase/pocketbase/blob/master/forms/record_upsert.go#L746-L772.

Tomorrow I'll investigate it in more details and I'll try to submit a fix.

from pocketbase.

Or a optional URL rewrite rule config, e.g.: https://mycdn.com/{BUCKET}/{KEY}

from pocketbase.

This unfortunately ended up more troublesome than I initially anticipated.

I've implemented locally a slightly modified version of @dizys's PR#28, but I've stumbled on several issues.

Redirecting to a CDN endpoint will require the objects to be public, but unfortunatelly not all S3 providers have options to control globally the visibility of an entire bucket or prefix. One workaround I found was to upload each individual blob with special write options specifying the ACL public-read but even this is not guaranteed to be supported by every S3 provider (it will also conflict with the planned private/protected files support in #215, but I guess we can ignore it for this specific case).

Another bigger and blocker issue that I've stumbled on is that the uploaded files may not be always immediately available after upload when accessing the CDN endpoint and we'll get 40x when retrieving the file or when just sending a HEAD request to check for the file existence.
I've tested it with AWS CloudFront and Digital Ocean Spaces CDN and both have the above issue. This is not isolated only for the main uploaded file, but also affects the lazy thumbs generation.

To summarize - I'm not sure if implemententing it as it is will be a good idea since it is not reliable enough and it is not guaranteed to work with all S3 compatible vendors.

Depending on the users demand, I may revisit it again in the future, but for now I'll close the issue and the recommended workaround to reduce the number of S3 API calls would be to enable /api/files/* cache on a reverse proxy level.

from pocketbase.

@danibjor The issue with the realtime events firing before the file upload should be fixed in https://github.com/pocketbase/pocketbase/releases/tag/v0.14.1.

from pocketbase.

@danibjor The issue with the realtime events firing before the file upload should be fixed in https://github.com/pocketbase/pocketbase/releases/tag/v0.14.1.

Did a quick test, seems to have resolved the race conditions that I saw earlier on.

from pocketbase.

@jgb-solutions The first "folder" from the object prefix is the collection id and it will differ for each collection (_pb_users_auth_ is the id of the default users collection)

from pocketbase.

As a workaround, here's an idea: put some reverse proxy in front which will rewrite the returned URL to that of the CDN.
Depending on how different is the object's CDN URL from the original so it can be easily derived, you might want to give it a try.
Or maybe some middleware like min.io can handle this.
Anyway, the feature would be nice to have.

from pocketbase.

Files not being available is also an issue on local file storage.

Scenario I’ve tested locally. Add subscription on a collection. Upload new record that contains file to this collection. When subscription event receives client, fetch record again with expanded relations, then render on screen. Most of the time image returns 404, but a 2-300ms delay before fetching the record solves it for my case.

from pocketbase.

@danibjor Hm, this is because the record model is saved before uploading the files (the realtime events are triggered right after the record model save) to minimize the transaction locking times and to allow developers to specify a custom record id within a OnModelBeforeCreate hook.

I've added to the roadmap to consider refactoring the record save operations order.

from pocketbase.

It’s probably an edge case I’ve stumbled into, but something to be aware of. These are 5-10MB photos, so shuffling bytes on the disk takes the time it take. And practically no latency between the client and server don’t help on this issues as well.

from pocketbase.

hey @ganigeorgiev I'm trying to rewrite the url based on the filename. But in my cloud provider I see the uploaded file has 2 parent folders, parent-with-id-I-dont-know/record-id/filename.mp3

Any idea how to get this first folder name with the ID I don't know? I thought it was the current logged in user's ID but that's not the case. Note that in an earlier version it used to name that parent folder _pb_users_auth_.

Thanks!

from pocketbase.

@ganigeorgiev ok thanks! I'm starting to deeply learn and integrate this software into my stack. It's great! Keep improving it.

Quick question while we are at it: is it possible to bypass the server entirely and directly upload in the cloud? That's what I was thinking of doing for an existing app. Upload to cloud client side and save the filename in pocketbase.

from pocketbase.

Fire an event off s3 for any file crud. Then Pocketbase knows when the file is there and can toggle the visibility ?

from pocketbase.

@gedw99 can you elaborate on that?

from pocketbase.