Load performance problems about rustbreak HOT 15 CLOSED

Did some measurements again (apparently missed an option last time). It looks to me like all the time really is consumed by yaml_rust, which is quite disappointing, tbh.

But maybe it's something you cannot get around with yaml: it's possible that, given a document consisting of a sequence in block representation, a parser has to perform an (expensive) look-ahaed over the entire file in order to ensure that it really is the root element (and the sequence is not, for example, the key of a mapping). However, I did not perform a thorough analysis.

Since this appears to be an issue of the yaml-dependency rather than something that can be fixed in rustbreak, this issue should be closed imo (as "wont fix").

from rustbreak.

Heya! Thanks for asking that, to be honest I don't know.

How complicated is your Yaml file? How quick is your Disk? Is it in RAM? There are so many possible causes for the slowness that I am not sure what could be going on.

from rustbreak.

@TheNeikos Im saving in a file. Disk not SDD. Yes it's complicated.

from rustbreak.

Maybe try gather some data using perf or a similar tool?

from rustbreak.

@neithernut I dont have time and resource to do such advanced tests at the moment.

from rustbreak.

I would not consider "using a profiler" advanced. However, if you can't do measurements yourself, maybe you can provide some more data. Is is possible for you to share the file you try to load? If not, can you give some hints regarding the structure (e.g. maximum/median nesting, number of entities per level)?

from rustbreak.

@neithernut will share when get to my work pc.

from rustbreak.

Have you considered switching the format to bincode, it will be read much faster..

from rustbreak.

@Boscop will try at some point, as u pointed it out!
@neithernut im sorry but my large Yaml us gone. As I did a logic workaround and made it smaller.. and it was in gitignore of the project, so :/. What I remember is that it had a huge Array inside Vec<u8>

from rustbreak.

Ok. Maybe I'll find some time to try to reproduce the issue over the weekend.

from rustbreak.

I've managed to "reproduce" the issue with the following reproducer:

extern crate rustbreak;

use std::env;
use std::fs::OpenOptions;

fn dummy_vec(mut len: usize) -> Vec<usize> {
    let mut res = Vec::new();
    while len > 0 {
        res.push(len);
        len = len - 1;
    }
    res
}

fn main() {
    let mut args = env::args().skip(1);

    let file = args.next().map(|p| OpenOptions::new().read(true).write(true).create(true).open(p)).unwrap().unwrap();
    let db = rustbreak::FileDatabase::<Vec<usize>, rustbreak::deser::Yaml>::from_file(file, Default::default()).unwrap();

    if let Some(numstr) = args.next() {
        db.put_data(dummy_vec(numstr.parse().unwrap()), true).unwrap();
        db.save().unwrap();
    } else {
        db.load().unwrap();
    }
}

I did not manage to get pointers out of perf, yet. I'll have a closer look in the near future.

from rustbreak.

@neithernut maybe u can complain in yaml_rust and see what they explain?

from rustbreak.

As I already expressed, it might be a problem which is inherent to yaml-parsers. I can investigate in the near future, but I doubt it's something which yaml_rust can fix.

You are really better of using another format.

from rustbreak.

@neithernut ok tnx.

from rustbreak.

Thank you @neithernut for taking a look into this!

from rustbreak.