[Request] Use MySQL for data storage over FlatFile about factions HOT 19 CLOSED

There might be some advantages to use a mysql db BUT performance is not one of them. The current system caches all data as native java objects, hashmaps, arrays etc. This would probably be faster than using a mysql db. Correct me if I'm wrong.

To edit the json files should not be to hard, php has build in json_encode / decode.
The issue is that the plugin will overwrite the changes as it does not care about what's on disc. It cashes it all in memory and occasionally backs up / overwrites all content on the disc.

The upside with using a mysql db would be the possibility to edit the data from other applications concurrently... This would be nice I guess... but since I don't need this myself I will probably not code it.

from factions.

This is something the server I contribute on behalf of would appreciate having too. Once the current build has stabilised I may look into it.

from factions.

Are you familiar with the "migrations" concept for mysql databases?
Are you familiar with data concurrency vs speed and memory optimization?
In such case we should really talk on skype.
Implementing sql storage for the Factions plugin is no small task and must be taken very seriously and be done extremely well for me to be willing to pull that request :)

I agree that it might be a good Idea but we must do some research together first in such case :)

from factions.

I am not an expert on databases but I am aware of the issues with using a database with factions.

Caching data from the database seems to be the first hurdle to me. I don't want to be issuing a query for every block placement.

from factions.

Exactly :) And if we cache... how would the system be different from what we are using now? :P
The concurrency issues would be the same wouldn't they?

from factions.

There would be a local cache for reading data but when something is written, such as land being claimed the change would be pushed to database immediately. This way another application could read from the database for live statistics.

However, if that application wished to change something in the database the caching would then prevent the change propagating through to factions immediately, which results in data discrepancies.

Having the server refresh the cache on regular basis could ease that. It would allow the server admin to trade between performance and concurrency issues with the database. If the admin has another application changing stuff in the database and a high performance machine they can set it to be very often such as every 3-5 minutes. If the admin has nothing else interfering with the database it could be every 30-45 minutes.

Another solution could be to have a flag in the database to indicate when another application has changed something and factions refresh the cache and unsets that flag automatically.

from factions.

I will most likely implement this feature on my own for select features (Factions.json).

I am implementing a faction aware economy (not iConomy) system and my web services need SQL support. If it proves too difficult then I will create query methods in the primary bukkit plugin class so I may use JSONAPI and fetch data that way. I would normally just symlink the json file into my webdir but these services are located on different boxes T.T

from factions.

My point was more to improve load times and reduce the time it takes to reboot the server. Currently rebooting the server takes almost a minute and about 20-sec of that is the loading of the factions plugin im guessing its because it has to read in the information from the files and create the java objects. You can still create the java objects just have them persist to mysql so the load times and save times are not as server impacting. If you do this it allows you to specify a sql server to store this information taking more load off the minecraft server itself. I hope someone is able to implement this. Would be an excellent addition to an AMAZING plugin.

from factions.

I have done some database optimisations in the past for high-load websites..
database should be fast enough if you put your indexes correct..
database is always faster than saving it to files, and won't get easy corrupted..

If you really want to keep caching with memory,then i guess that the next best option is make some kind of API that a webservice can connect to? soap, json, xml api..
you can open an port for example 8001 or something.. to access the API..

this will be a bit of work.. but it will help a lot :) and the databases can be in control of the plugin :)

Little note, if data-files get to large.. or to much data to cache.. i guess you will have to go to database eventually..
you can always do delayed inserts, so the plugin doesn't suffer delay from the database when writing new data..
and keep the most used/recent data cached, make a sort of rolling cache.. not that hard to code :)

from factions.

Now that i'm thinking about it.. what about an sort of Minecraft management plugin.. that hooks in the most known/common plugins.. that supports an API and a webinterface to manage your data/plugins..
and with hooking into the plugins you don't have to care what data-saving they have.. you just use their methodes/functions

I'm not sure if there is such a project out there yet? but in my opinion.. it should be very awesome to have ^^

sadly enough my java-skills are a bit rusty.. i'm more a php/python guy.. but it shouldn't be to hard in java..
soap would work.. java and php can speak soap to each other :)

anyone up for this?

from factions.

after a bit of looking arround, i found this..
my guess is, that this will hook into the internal API's of plugins and make them via web-service available for web-development :)
https://github.com/webbukkit/webbukkit

from factions.

I'll try to do implement data storage in DBs but I don't promise anything. I have idea how to do it and it would useful for my server.

from factions.

I am using MySQL for a custom RPG (Heroes like) plugin for my server. It is very reliable and fast - and it allows me to use the data on the website (The server lies in the home-directory of the server user, i would have to allow PHP that directory first, then read and parse the files).
And the best thing: It is running async to the main thread which makes almost no lag at all.

from factions.

But are you using local copy of records and next you push changes to DB or just operate only on DB??

from factions.

I have an async task every 3 minutes saving the local data to the database (period can be set in config). I am working with the "local" copies, yes. The only thing I am working on at the moment (literally this moment) is the async loading of the player login data - should be not too hard.

Edit: Done :)

from factions.

And how about conflicts between DB and local copy?? I am checking if i think right.

from factions.

For me the current system is good enough - I won't have any problems with DB and local copy conflicts, because I will never change some things in the database.
You are, however, right about the fact that this should be checked and resolved, if there are some conflicts.

from factions.

You right when you don't want to change something in DB but I am creating this system to manage factions by DB. I think I will override data in local copy if data in DB has changed.

It would be nice someone from factions accept my idea.

from factions.

My solution is the home grown system MStore. It will be used for Factions V2.0.0

from factions.