When using kiwix-serve, random unicode appear near/in-place of where other unicode should go. Example: a em-dash – should display, but instead − appears in it's place.

When extracting the zim using zimdump, the bug appears to be gone and the input file is identical to the output file for the text (the same exact unicode appears). This implies that the information is perfectly preserved and points to kiwix-serve having the bug.

If I open the zim using the chrome extension zim-reader, the error does not appear and renders as it should. Absolutely perfectly :).

You can see this bug on our private server here . We use the Absolute Zero article for our example, but the error shows up all over the place.

To recreate this, simply use zimwriterfs on a downloaded Absolute_Zero article html file.

We have to test and make work kiwix-builder/meson on MacOSX.

This should be no real obstacle for this[*], but we have to test and add the inevitable twinks as platforms are different.

[*]

• Other dependency already compile on MacOSX
• Meson support MacOSX
• kiwix-builder is python script, this should just work.

Meson should provide a way to make a distributable tarball of the sources

kiwix-builder should handle compilation of kiwix for windows of kiwix-tools and kiwix-serve

We want to garanty that Kiwix solutions compile/work fine with Fedora. This can only be done if Fedora is part at some point of the CI. Unfortunately Travis does not do that out-of-the-box.

This ticket is a follower of #12

on rpi3 with kiwix-tools_armhf_2017-09-28.tar.gz displaying wikipedia_en_medicine_2017-09.zim in kiwix-serve:

'__CSS_LINKS' is displayed at top

'ARTICLE_CONFIGVARS_LIST' is displayed below attribution

README should clearly explain the compilation procedure. This means step-by-step procedures which can be follow blindly with short explanation about what is going on:

• What are the dependencies (with minimal version)
• How to compile under Linux static/dynamic
• Where to go to compile under Windows & if user wants to download locally the deps

on 2017-09-07-raspbian-stretch-lite.img
using nightly kiwix-tools_armhf_2017-09-19.tar.gz (also 2017-9-12)

/opt/iiab/kiwix/bin/kiwix-manage /library/zims/library.xml add /library/zims/content/wikipedia_en_medicine_2017-08.zim

results in

last cluster offset larger than file size; file corrupt
Unable to build or save library file '/library/zims/library.xml'

/opt/iiab/kiwix/bin/kiwix-manage /library/zims/library.xml add /library/zims/content/wikipedia_es_medicine_2017-04.zim succeeds (so may be the zim)

I have not observed this on linux64 for either zim.

From @abudden on January 12, 2017 13:0

If kiwix is installed behind a proxy (e.g. nginx) with a non-root path, none of the links or stylesheets work as every URL is specified as absolute (e.g. /skin/jquery-ui/external/jquery/jquery.js). Therefore if you have nginx configured to serve kiwix at /kiwix, the links resolve to /skin/... instead of /kiwix/skin/... and nothing works.

Copied from original issue: kiwix/kiwix-xulrunner#436

This issue has been opened on the wrong https://github.com/kiwix/tools repository. (kelson42/tools#1)
I report it here because this is where it should have be opened.
Please see already posted comments on the original issue.

on access to even an old zim like ted_en_science_2015-02/ jquery fails to initialize with
jquery.js:9790 Uncaught SyntaxError: Invalid or unexpected token
this is the end of the file and there are two junk characters.
problem does not occur in latest FF or chrome 55 and earlier.
breaks search, and on ted science videos do not appear.

For example with http://download.kiwix.org/zim/other/icd10_fr_all_2012-01.zim
Tested with nightly build on linux

Many of the portable zims are missing the zim file and index files. They are the lastest zims from January 19/20 20017. They all seem to be 189MB in size.
Here is an example:

http://download.kiwix.org/portable/wiktionary/kiwix-0.9+wiktionary_my_all_2017-01.zip

I never noticed this before.

using 8/28 build with url http://iiab.me:3000/wikipedia_en_medicine_2017-02/
search for cardiac and select 'containing cardiac'

all search results begin with zim:// which for me does not exist

internal page links look like http://iiab.me:3000/wikipedia_en_medicine_2017-02/A/Pulmonary_embolism.html as before

I do not think this belong here, should be kept in the 'kiwix' git repo IMO

Found this bug at the SUNY Potsdam hackthon.

Environment is a Docker container running Ubuntu Xenial with the 8/13/17 Nightly build of kiwix-tools. Here is the github project for the Docker.

Assume that library.xml is at /

root@75aabacd457d:/# pwd
/

root@75aabacd457d:/# kiwix-serve --library ./library.xml
Unable to open the XML library file './library.xml'.

root@75aabacd457d:/# kiwix-serve --library /library.xml
(server is running)

From @rashiq on June 8, 2016 13:8

Reported by kelson42 on 2015-12-08 09:11 UTC
Articles where the title has a % don't load. Try eg, "100% Chance of Rain" or "1%CLUB" in
http://download.kiwix.org/zim/wikipedia/wikipedia_en_all_2015-05.zim

Copied from original issue: kiwix/kiwix-xulrunner#222

The top nav bar of pages served by kiwix-serve have buttons for library, home, random.

add a runtime switch to remove the library button

version of the 20 or 21 october.

Here is the log

Accept-Language: en,fr;q=0.7,en-US;q=0.3
Accept-Encoding: gzip, deflate
DNT: 1
Connection: keep-alive
Upgrade-Insecure-Requests: 1
Requesting :
u : /skin/jquery-ui/images/ui-bg_highlight-soft_75_cccccc_1x100.png
m : GET
v : HTTP/1.1
Host: download.kiwix.org:8080
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:56.0)
Gecko/20100101 Firefox/56.0
Accept: */*
Accept-Language: en,fr;q=0.7,en-US;q=0.3
Accept-Encoding: gzip, deflate
Referer:
http://download.kiwix.org:8080/skin/jquery-ui/jquery-ui.theme.min.css
DNT: 1
Connection: keep-alive
Requesting :
u : /networkengineering.stackexchange.com_en_all_2017-10/tag/pfsense/1.html
m : GET
v : HTTP/1.1
Host: download.kiwix.org:8080
Connection: Keep-Alive
User-Agent: Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
From: [email protected]
Accept-Encoding: gzip,deflate
Accept: */*
Failed to find /tag/pfsense/1.html
Requesting :
u : /wikipedia_en_medicine_2017-05/A/Exercise_physiology.html
m : GET
v : HTTP/1.1
Host: download.kiwix.org:8080
Connection: keep-alive
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
From: googlebot(at)googlebot.com
User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)
Accept-Encoding: gzip,deflate,br
Requesting :
u : /
m : GET
v : HTTP/1.1
Range: bytes=0-16384
User-Agent: Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)
Accept: */*
Accept-Encoding: gzip,deflate
Host: download.kiwix.org:8080
Cache-Control: max-age=259200
Connection: keep-alive
Requesting :
u : /
m : GET
v : HTTP/1.1
User-Agent: Mozilla/5.0 (compatible; The Lounge IRC Client;
+https://github.com/thelounge/lounge)
host: download.kiwix.org:8080
Connection: close
terminate called after throwing an instance of 'std::out_of_range'
  what():  basic_string::substr
Aborted

From @rashiq on June 8, 2016 13:4

Reported by kelson42 on 2014-10-31 16:27 UTC
In certain situation, kiwix-serve seems to be unable to read the library.

kelson@camber:/tmp$ ./kiwix/bin/kiwix-serve --library wikipedia_de_all_12_2013.zim.xml
Unable to open the XML library file 'wikipedia_de_all_12_2013.zim.xml'.

kelson@camber:/tmp/kiwix/bin$ ./kiwix-serve --library ../../wikipedia_de_all_12_2013.zim.xml
error 2 opening file "/content/wikipedia_de_all_12_2013.zim": No such file or directory
The XML library file '../../wikipedia_de_all_12_2013.zim.xml' is empty (or has only remote books).
Unable to instanciate the HTTP daemon. The port 80 is maybe already occupied or need more permissions to be open. Please try as root or with a port number higher or equal to 1024.

Copied from original issue: kiwix/kiwix-xulrunner#165

For each new commit to master, CI should run and:

• Build statically on Fedora and Ubuntu
• Build Dynamically on Fedora and Ubuntu

using nightly http://download.kiwix.org/nightly/2017-05-18/kiwix-tools_linux64_2017-05-18.tar.gz

kiwix-manage /library/zims/library.xml add /library/zims/content/ICD10-fr.zim
kiwix-manage /library/zims/library.xml add /library/zims/content/wikileaks_afghanistan_war_diary.zim

both cause segfault

others tested do not

No result were found for XXX should read

No results were found for XXX

Include tags in all (new) zims as follows, in addition to any others currently required such as _ftindex:

selection_criterion_all/medicine/etc. (any string that represents criterion by which articles were selected)
size_criterion_all/50K/etc. (target number of articles)

has_videos
has_pics
has_full_text
has_audio

has_ (any current or future attributes of an article)

These tags should also be in the catalog, but are independent of the file naming convention.

I saw on sourceforge that you provided some changes in 2015 to the code of kiwix-serve, helping to run it on Bookeen Cybook ebook readers (mainly in the css section).
I got version 0.9 from 2014 of kiwix-serve (without the Bookeen additions) running on a Pocketbook 626 but wonder, if the Bookeen version could be changed to recognize Pocketbook headers too:
Mozilla/5.0 (Linux like Android; de-DE) AppleWebKit/534.34 PocketBook/630 (screen 758x1024; FW U630.5.321) Mobile
or
Mozilla/5.0 (Linux like Android; de-DE) AppleWebKit/534.34 PocketBook/626 (screen 758x1024; FW W626.5.14.964) Mobile
or
Mozilla/5.0 (Linux like Android; de-DE) AppleWebKit/534.34 PocketBook/631 (screen 1072x1448; FW U631.5.321) Mobile
or
Mozilla/5.0 (Linux like Android; en-US) AppleWebKit/534.34 PocketBook/623 (screen 758x1024; FW M623.4.3.267) Mobile

Do You see a way to include Pocketbook devices too into kiwix-serve and get a newer ARM compiled binary? I'm sure a lot of users would like to use offline wikidumps via kiwix on their device.
Regards, Uwe

We can (and must) provide the install dir when compiling kiwix with kiwix-builder.
However, source code are download and compiled in the current directory.

We must add a option to specify the working directory to not pollute the cwd.

Kiwix-serve shows a Jquery UI Accordeon to show the site selector. Animate is on here. Animate does not work at all on a long list on desktop and especially on mobile devices! It literally takes 10 seconds to animate on my Huawei P8, wich is a pretty fast phone. Problem here is Icecat, or firefox wich animates horribly. Its totally unusable this way.

The solution is very simple. Turn off the animations:
http://api.jqueryui.com/accordion/#option-animate

Like so:

  <script>
  $( function() {
    $( "#accordion" ).accordion({
         animate: 0
    });
  } );
  </script>

Using project may want to customize the headerbar from kiwix-serve. ie :

• remove library button
• add custom styling (logo, ...)

On solution is to have a more advance resourceTools that can get resources from filesystem if present or return embeded ones else.
This way, a user can provide it's own template to use without to have to recompile kiwix-serve.

version: http://download.kiwix.org/nightly/2017-05-24/kiwix-tools_armhf_2017-04-30.tar.gz (download and untar)

download and unzip http://download.kiwix.org/portable/wikipedia/kiwix-0.9+wikipedia_en_for-schools_2013-01.zip

put all .zim* files into /library/zims/content and zim.idx directory into /library/zims/index

/opt/schoolserver/kiwix/bin/kiwix-manage /library/zims/library.xml add /library/zims/content/wikipedia_en_for_schools_opt_2013.zim -i /library/zims/index/wikipedia_en_for_schools_opt_2013.zim.idx (N.B. index exists)

run /opt/schoolserver/kiwix/bin/kiwix-serve --port 3000 --library /library/zims/library.xml
navigate browser to host:3000 and load Wikipedia for Schools

enter 'children' in search field

response:
No result were found for children

previously kiwix-serve showed a list of articles with the word 'children'

kiwix-serve binary should only rely on kiwix-lib and not on libmicrohttpd, ctpp2, etc.

Reported first in #73 (comment)

By running only kiwix-serve, the message seems to appear: "The program cannot be start because msvcr90.dll is missing."

Kind of puzzle by this situation: does that sounds normal?

From @tim-moody on June 26, 2016 8:54

url rewriting did not work for me if kiwix is not mapped to the web root

so mapping to / works (as I remember), but not /kiwix

Copied from original issue: kiwix/kiwix-xulrunner#254

Just trying calling /wikipedia_en_medicine_2017-09/skin/error.php
Obviously super urgent to fix.
Tested with last nightly.

To allow smooth video streaming

The top bar's layout becomes a bit awkward when it comes to narrow screen devices. The buttons and the search bar are taking up two rows.

I think we could a) collapse buttons or b) hide text field and show only when clicked on search button

To do, one time the kiwix-lib is packaged.

I'm working on the Debian packaging for kiwix-tools and ran into the following:

Running test binary command: /home/km/projects/debian/kiwix-tools/obj-x86_64-linux-gnu/meson-private/sanitycheckcpp.exe
Native cpp compiler: c++ (gcc 6.4.0)
Appending CXXFLAGS from environment: '-g -O2 -fdebug-prefix-map=/home/km/projects/debian/kiwix-tools=. -fstack-protector-strong -Wformat -Werror=format-security'
Appending LDFLAGS from environment: '-Wl,-z,relro'
Appending CPPFLAGS from environment: '-Wdate-time -D_FORTIFY_SOURCE=2'
Build machine cpu family: x86_64
Build machine cpu: x86_64
Dependency threads found: YES
Found pkg-config: /usr/bin/pkg-config (0.29)
Determining dependency 'kiwix' with pkg-config executable '/usr/bin/pkg-config'
Native dependency kiwix found: YES 1.0
Determining dependency 'libmicrohttpd' with pkg-config executable '/usr/bin/pkg-config'
Native dependency libmicrohttpd found: YES 0.9.55
Determining dependency 'zlib' with pkg-config executable '/usr/bin/pkg-config'
Native dependency zlib found: YES 1.2.8
Determining dependency 'xapian-core' with pkg-config executable '/usr/bin/pkg-config'
Native dependency xapian-core found: YES 1.4.4
Program compile_resources.py found: NO

Meson encountered an error in file static/meson.build, line 2, column 0:
Program "compile_resources.py" not found or not executable

Is this referring to the compile_resources.py file in kiwix-lib? Why is that python file needed? And where should I be putting it to make this work?

From @kelson42 on October 13, 2016 10:15

The current custom XML feed needs to be replaced by a better standard base solution. It would be better to make it in kiwix-serve instead of a custom script because:

• This is a start point to allow kiwix-serve (like kiwix-plug) solution to act like a content repository
• We have to create a serious library.kiwix.org solution demonstrating content anyway... so even easier if the catalog is here too.

Copied from original issue: kiwix/kiwix-xulrunner#342

1. version: http://download.kiwix.org/nightly/2017-04-30/kiwix-tools_armhf_2017-04-30.tar.gz (download and untar)
2. use the hopefully attached test.zim file but any zim without internal index should work
3. create library.xml with
   /opt/schoolserver/kiwix/bin/kiwix-manage /library/zims/library.xml add /library/zims/content/test.zim -i /library/zims/index/test.zim.idx (N.B. index does not exist)
4. run /opt/schoolserver/kiwix/bin/kiwix-serve --port 3000 --library /library/zims/library.xml
5. navigate browser to host:3000 and load test
6. search for XXXXXXXXXX
   root@box:/opt/schoolserver/kiwix# /opt/schoolserver/kiwix/bin/kiwix-serve --port 3000 --library /library/zims/library.xml
   terminate called after throwing an instance of 'Xapian::InvalidArgumentError'
   Aborted

For now just, kiwix-serve is compiled. We must compile all tools in the repository.

When there is a "&" in the name of the zim, search will request to kiwix-serve with a url like this : http://127.0.0.1:8080/suggest?content=unix_&_linux_stack_exchange_en_all_2017-03&term=how

the & of the name is interpret as a & for new argument of the request

Also in this case, kiwix-serve will answer to this request with the html of the page to choose which zim to load

At BSF, we need to change how the topbar is displayed. (Css but also displayed buttons).

I've got a working version, but it is based on some kind of hidden feature of the last kiwix-serve/kiwix-lib allowing to change the embedded resources at runtime with env var.
It's working but it is not perfect and error prone. (ideascube/ideascube#590 and ideascube/ansiblecube#43)

On top of it, we will end to rewrite a part of the topbar to make it more mobile friendly. (kiwix/kiwix-xulrunner#336)
It could be a good opportunity to think about how we add the topbar.

version: http://download.kiwix.org/nightly/2017-04-30/kiwix-tools_armhf_2017-04-30.tar.gz (download and untar)

download and unzip http://download.kiwix.org/portable/wikipedia/kiwix-0.9+wikipedia_en_for-schools_2013-01.zip

put all .zim* files into /library/zims/content and zim.idx directory into /library/zims/index

/opt/schoolserver/kiwix/bin/kiwix-manage /library/zims/library.xml add /library/zims/content/wikipedia_en_for_schools_opt_2013.zim -i /library/zims/index/wikipedia_en_for_schools_opt_2013.zim.idx (N.B. index exists)

run /opt/schoolserver/kiwix/bin/kiwix-serve --port 3000 --library /library/zims/library.xml
navigate browser to host:3000 and load Wikipedia for Schools

enter 'children' in search field

-rw-r--r-- 1 root root 133637738 Apr 30 14:44 kiwix-tools_armhf_2017-04-30.tar.gz
root@box:/opt/schoolserver/kiwix/bin# /opt/schoolserver/kiwix/bin/kiwix-serve --port 3000 --library /library/zims/library.xml
terminate called after throwing an instance of 'Xapian::InvalidArgumentError'
Aborted

previously kiwix-serve showed a list of articles with the word 'children' (I take it as self-evident that this is the preferred behavior)

From @rashiq on June 8, 2016 13:4

Reported by kelson42 on 2014-10-26 11:56 UTC
The text should be rewrapped correctly.

Copied from original issue: kiwix/kiwix-xulrunner#162

As a follower of #65 by @nataren

In case kiwix-serve is started with a ZIM file with ftindex and a manually (or through the library) set external ftindex, the ftindex should have the priority. This is important to provide a workardound to people cutting files with embedded ftindex.

From @kelson42 on August 31, 2016 18:46

... and avoid displaying the catalog page if only one ZIM file.

Copied from original issue: kiwix/kiwix-xulrunner#299

When I build a library that includes a chunked zim, and perform a search inside that zim (in the UI), the kiwix-serve dies.

Steps to reproduce:

mkdir ~/tmp
cd ./tmp
 wget http://download.kiwix.org/nightly/2017-08-11/kiwix-tools_linux64_2017-08-11.tar.gz
 tar xzvf kiwix-tools_linux64_2017-08-11.tar.gz
./kiwix-manage testlibrary add /library/zims/content/wikipedia_en_medicine_2017-02.zim  --current
./kiwix-manage testlibrary add /library/zims/content/wikipedia_en_medicine_2017-08.zimaa --current
./kiwix-serve --daemon --port 8002 --library ./testlibrary

# Visit the kiwix home page on 8002
# Click on the wikipedia_en_medicine_2017-08/
# Type something in the Search entry
# Confirm that the kiwix-serve process died with: terminate called after throwing an instance of 'Xapian::InvalidArgumentError'

When kiwix-serve is used in another solution (ideascube for example, by BSF or pibox), the "home" page is not the kiwix-serve library but the solution homepage.

We should allow those solution to modify the taskbar to add a link (or change the library button) pointing to the "real" homepage.

This is especially needed by pibox solution. Pibox is creating a captive portal and use the captive portal page to display the home page. In most computer, this captive portal page is opened automatically in a web browser without navigation button. It's means that once a user has enter a zim file, he cannot go back to the home page at all.

For now, source are almost compile in place in the source directory.

We should build projects in a separated directory. This way, (patched) source are left untouched and we can compile a static/dynamic, native/cross-compiled version from the same sources.