zimwriterfs replacement about mwoffliner HOT 4 CLOSED

I did not test, but it looks like great work!
What was your final goal for writing this?

You might have seen that we implemented a (beta) javascript ZIM reader in https://github.com/kiwix/kiwix-html5, that works inside a desktop browser.
We consider splitting the UI and the backend, to have a standalone javascript ZIM reader. Maybe your code could be integrated inside this library, to have both the reader and the writer? Some code can probably be in common.

from mwoffliner.

@Vadp Hi, nice, I have tested it and it seems to work fine. The ZIM file I have created looks good and the size it almost the same like with zimwriterfs. The big difference is the speed, in my test it's around 3x time slower than with zimwriterfs (that it somehow a bit strange).

On a meta level a few remarks:

• zimwriterfs is a generic tool but we use it mainly in mwoffliner which is also written in for nodejs
• mwoffliner should ideally create the ZIM file on the fly using something in javascript. So the code you have written might be reused or https://github.com/cscott/node-libzim (which might be faster)
• Finally we would be interested to have an ultimate portable solution in pure js (for example to build chrome/firefox extension)... and there it would be awesome to integrate your writting function to jszim (kiwix-html5), this might be an opportunity to spit properly the jszim from kiwix-html5.

I do not know what were your motivation to create zimmer, but maybe you might be interested in continuing on that way? Let us know. Anyway congratulation for zimmer (and having added it to openzim.org).

from mwoffliner.

Hi @kelson42.

I wrote zimmer simply because zimwriterfs is not available on ubuntu and my sole intention was to have a more or less reliable method of building ZIMs for my own needs. I've also tried to make this code to fit somewhat with mwoffliner's workflow.

Apparently I wanted zimmer to be able to process the whole Wikipedia, so to keep memory footprint under control it uses sqilte to store temporary indexes. That's might be one of the reasons for it's relative slowness (although I've tried to take some measures). Another possible culprit if the compressor library, albeit this library apparently is able to utilise multiple cores.

Anyway, for myself I've achieved what I needed, but if there is some interest from the mwoffliner developers then I'd be happy to modify zimmer provide some interface so to allow mwoffliner to build ZIM on fly. Perhaps it's slowness wouldn't be that much of an issue. Actually, I've (svadim) already submitted a couple of patches for mwoffliner when it was hosted at the sourceforge.

BTW To play with building Wikipedia ZIM I've also made a simple "unzimmer". It could be hardly used though for doing any kind of interactive job.

from mwoffliner.

Here the goal is to integrate nodejs-libzim, which is simply the fastest solution.

from mwoffliner.