Consider MongoDB and GridFS for meta, thumbnails, transcoded video etc about mediacenterjs HOT 12 CLOSED

Hej there,

Thanks for the very interesting idea. I'm curious, currently images are stored on the file system but these could be stored in the database like you said, but would that provide any speed improvements do you think? The reason images are stored on the file system is because it is very easy to access that data from the client.

The main reason I have chosen for a sqlite3 database is the fact that it will run out of the box on the Raspberry Pi.

I'd love to hear your thoughts.

from mediacenterjs.

There would be some speed improvements and architectural advantages. There would also be equally as many challenges. I hadn't considered the consequences of requiring MongoDB. It wouldn't be viable on machines with low resources like a Rasberry Pi. Some form of document-store database like Mongo would surely simplify and make certain things more efficient though. For one, having all of the META info and associated images stored in one document makes keeping track of things very simple. Secondly, Mongo would store a lot of things in RAM that could potentially cause a lot of file system chatter otherwise. See http://docs.mongodb.org/manual/core/query-plans/ for another aspect of Mongo that would be beneficial - the built-in query optimizer.

Any how, there are probably enough cons to weigh out the pros of this idea, but I'll put some more thought into it. I typically try to avoid using Node for applications that serve a lot of static content. To me, media collection meta info and associated imagery would ideally be stored in some kind of NoSQL database like Mongo.

from mediacenterjs.

Just came across this: http://www.tingodb.com/

I'm going to look into it when I find time and return to this conversation with some ideas.

from mediacenterjs.

Also https://github.com/louischatriot/nedb

from mediacenterjs.

And http://dailyjs.com/2013/04/19/leveldb-and-node-1/

from mediacenterjs.

http://leveldb.googlecode.com/svn/trunk/doc/benchmark.html

from mediacenterjs.

I really like nedb! It has the same zero dependency policy as dblite and seems to be a lot faster. I will definitely look into it. Thanks a lot!

from mediacenterjs.

I really like nedb as well. I haven't looked into it a lot, but I know that leveldb allows binary data stores which would be able to handle the images. Nedb uses Mongo's syntax which is very helpful, though.

from mediacenterjs.

Hej Philososaur,

Just so you know; I'll continue developing with the current db setup for now (I want to reach the Beta as soon as possible and I have still a lot to do). But I will definitely try Nedb and port it over once I familiarized myself and get a clear picture of the performance improvements :) Thanks for the great tip!

Jan

from mediacenterjs.

I have started to test out nedb in the branch nedb-test :)
At the moment it looks pretty good

from mediacenterjs.

nedb would be a decent idea... I don't think MongoDB and GridFS will gain you that much over the file-system on a single machine... Yes, you get indexing, but with sane file structures that's predictable from disk io.

That said, having things in a single set of files has the advantage of less file fragmentation and easier cleanup. There's also leveldb/levelup/leveldown, but that has some binary requirements, and limited to a single process (kind of a non-issue for this workflow)

from mediacenterjs.

I believe the currently the setup with sequilize and sqlite works great for this project.

from mediacenterjs.