giggls / osml10n Goto Github PK

View Code? Open in Web Editor NEW

11.0 11.0 6.0 568 KB

Localization functions for Openstreetmap

Home Page: https://tile.openstreetmap.de

License: GNU General Public License v3.0

Python 15.92% Shell 1.06% PLpgSQL 1.65% Lua 77.82% Makefile 2.26% C 1.29%

latinisation openstreetmap osm2pgsql osml10n

osml10n's People

Contributors

Stargazers

Watchers

Forkers

hixi hej2010 hholzgra agileminds1997 danieldegroot2 astridx

osml10n's Issues

make deb error

Hi everyone, I clone repository and run sudo mk-build-deps -i debian/control then when I run make deb I get following errors. I use postggresql16. Why does this happens, can someone help?

missing debhelper

for debian install, there was this issue : dpkg-checkbuilddeps: erreur: Unmet build dependencies: debhelper (>= 11).
fix with "apt install debhelper "

free(): invalid size Aborted Core dumped

SUMMARY
Im trying to get an all English titles map. This is a newly built system, that has imported the whole planet quite easily with a standard command: it resulted in a fully functional map, with non english titles on asia / north africa etc.. totally normal. I tried using Localization functions for Openstreetmap (osml10n) to get english titles, and it always core dumps. In examining
in a debuuger and valgrind it looks like it could be in osm2pgsql, but im unsure. So im submitting here to see.

i also posted on osm2pgsql.. not sure which it could be.. ( i know that might be "bad form" but its hard to tell whats causing this) --> osm2pgsql-dev/osm2pgsql#2117

DETAILS
all "make test" worked perfectly the only issue is the core dump.

when i try to use the Localization functions for Openstreetmap (osml10n) it fails with "free(): invalid size Aborted (core dumped)". (see below) Ive tried this on a number of different extracts from geofabrik and did check the md5 on all.

postgres@pb5:/osm2/osm2pgsql/build$ ./osm2pgsql -G -O flex -d gis -S openstreetmap-carto-hstore-only-l10n.lua ./japan-latest.osm.pbf
2023-12-26 15:14:51  osm2pgsql version 1.10.0 (1.10.0-9-g4a8f42b5)
2023-12-26 15:14:51  Database version: 16.1 (Ubuntu 16.1-1.pgdg22.04+1)
2023-12-26 15:14:51  PostGIS version: 3.4
2023-12-26 15:14:51  Storing properties to table '"public"."osm2pgsql_properties"'.
2023-12-26 15:14:52  WARNING: The 'area' column type is deprecated. Please read
2023-12-26 15:14:52  WARNING: https://osm2pgsql.org/doc/tutorials/switching-from-add-row-to-insert/
2023-12-26 15:14:52  WARNING: The add_row() function is deprecated. Please read
2023-12-26 15:14:52  WARNING: https://osm2pgsql.org/doc/tutorials/switching-from-add-row-to-insert/
Processing: Node(11840k 538.2k/s) Way(0k 0.00k/s) Relation(0 0.0/s)free(): invalid size
Aborted (core dumped)

I recognize that its possible that the issue is with the other project, so i ran this in gdb, trying to narrow it down

Processing: Node(11420k 571.0k/s) Way(0k 0.00k/s) Relation(0 0.0/s)free(): invalid size
Thread 1 "osm2pgsql" received signal SIGABRT, Aborted.

This didnt show me alot, so i tried valgrind. The error is in a lua utility (lua-utils.cpp:219) and it could be something on the osm2pgsql side. Valgrind details below..

(ill acknowledge that some of these tests are with different PBF files, i lost some of my testing sessions, but all sessions resulted in the same regardless of the PBF file used)


`postgres@pb5:/osm2/osm2pgsql/build$ valgrind ./osm2pgsql -G -O flex -d gis -S /home/osm/openstreetmap-carto/openstreetmap-carto-hstore-only-l10n.lua ./israel-and-palestine-latest.osm.pbf
==61966== Memcheck, a memory error detector
==61966== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==61966== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==61966== Command: ./osm2pgsql -G -O flex -d gis -S /home/osm/openstreetmap-carto/openstreetmap-carto-hstore-only-l10n.lua ./israel-and-palestine-latest.osm.pbf
==61966== 
2023-12-26 15:41:31  osm2pgsql version 1.10.0 (1.10.0-9-g4a8f42b5)
2023-12-26 15:41:32  Database version: 16.1 (Ubuntu 16.1-1.pgdg22.04+1)
2023-12-26 15:41:32  PostGIS version: 3.4
2023-12-26 15:41:32  Storing properties to table '"public"."osm2pgsql_properties"'.
2023-12-26 15:41:33  WARNING: The 'area' column type is deprecated. Please read
2023-12-26 15:41:33  WARNING: https://osm2pgsql.org/doc/tutorials/switching-from-add-row-to-insert/
2023-12-26 15:41:36  WARNING: The add_row() function is deprecated. Please read
2023-12-26 15:41:36  WARNING: https://osm2pgsql.org/doc/tutorials/switching-from-add-row-to-insert/
Processing: Node(11870k 6.3k/s) Way(0k 0.00k/s) Relation(0 0.0/s)==61966== Invalid free() / delete / delete[] / realloc()
==61966==    at 0x484B27F: free (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==61966==    by 0x601E1A9: ssh_buffer_free (in /usr/lib/x86_64-linux-gnu/libssh.so.4.8.7)
==61966==    by 0x485EE5B: freelist_free (in /usr/local/lib/lua/5.3/rex_pcre.so)
==61966==    by 0x486165E: algf_gsub (in /usr/local/lib/lua/5.3/rex_pcre.so)
==61966==    by 0x4CC1AD5: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0)
==61966==    by 0x4CCBF83: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0)
==61966==    by 0x4CC1FC7: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0)
==61966==    by 0x4CBA746: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0)
==61966==    by 0x4CBCFDA: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0)
==61966==    by 0x4CC1E9D: lua_pcallk (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0)
==61966==    by 0x339BD9: luaX_pcall(lua_State*, int, int) (lua-utils.cpp:219)
==61966==    by 0x24FB09: output_flex_t::call_lua_function(prepared_lua_function_t, osmium::OSMObject const&) (output-flex.cpp:916)
==61966==  Address 0x1ffefff300 is on thread 1's stack
==61966==  in frame #3, created by algf_gsub (???:)
==61966== 
==61966== Invalid free() / delete / delete[] / realloc()
==61966==    at 0x484B27F: free (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==61966==    by 0x485EE5B: freelist_free (in /usr/local/lib/lua/5.3/rex_pcre.so)
==61966==    by 0x486165E: algf_gsub (in /usr/local/lib/lua/5.3/rex_pcre.so)
==61966==    by 0x4CC1AD5: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0)
==61966==    by 0x4CCBF83: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0)
==61966==    by 0x4CC1FC7: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0)
==61966==    by 0x4CBA746: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0)
==61966==    by 0x4CBCFDA: ??? (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0)
==61966==    by 0x4CC1E9D: lua_pcallk (in /usr/lib/x86_64-linux-gnu/liblua5.3.so.0.0.0)
==61966==    by 0x339BD9: luaX_pcall(lua_State*, int, int) (lua-utils.cpp:219)
==61966==    by 0x24FB09: output_flex_t::call_lua_function(prepared_lua_function_t, osmium::OSMObject const&) (output-flex.cpp:916)
==61966==    by 0x24FC51: output_flex_t::get_mutex_and_call_lua_function(prepared_lua_function_t, osmium::OSMObject const&) (output-flex.cpp:928)
==61966==  Address 0x1ffefff1e0 is on thread 1's stack
==61966==  in frame #2, created by algf_gsub (???:)
==61966== 
Processing: Node(11876k 6.3k/s) Way(80k 0.16k/s) Relation(0 0.0/s)^C==61966== 
==61966== Process terminating with default action of signal 2 (SIGINT)
==61966==    at 0x50B3117: __futex_abstimed_wait_common64 (futex-internal.c:57)
==61966==    by 0x50B3117: __futex_abstimed_wait_common (futex-internal.c:87)
==61966==    by 0x50B3117: __futex_abstimed_wait_cancelable64 (futex-internal.c:139)
==61966==    by 0x50B5A40: __pthread_cond_wait_common (pthread_cond_wait.c:503)
==61966==    by 0x50B5A40: pthread_cond_wait@@GLIBC_2.3.2 (pthread_cond_wait.c:627)
==61966==    by 0x197610: void std::condition_variable::wait<osmium::thread::Queue<osmium::thread::function_wrapper>::wait_and_pop(osmium::thread::function_wrapper&)::{lambda()#1}>(std::unique_lock<std::mutex>&, osmium::thread::Queue<osmium::thread::function_wrapper>::wait_and_pop(osmium::thread::function_wrapper&)::{lambda()#1}) (condition_variable:103)
==61966==    by 0x18DA92: osmium::thread::Queue<osmium::thread::function_wrapper>::wait_and_pop(osmium::thread::function_wrapper&) (queue.hpp:188)
==61966==    by 0x2496DA: thread_pool_t::worker_thread(unsigned int) (thread-pool.cpp:57)
==61966==    by 0x24A66D: void std::__invoke_impl<void, void (thread_pool_t::*)(unsigned int), thread_pool_t*, unsigned int>(std::__invoke_memfun_deref, void (thread_pool_t::*&&)(unsigned int), thread_pool_t*&&, unsigned int&&) (invoke.h:74)
==61966==    by 0x24A587: std::__invoke_result<void (thread_pool_t::*)(unsigned int), thread_pool_t*, unsigned int>::type std::__invoke<void (thread_pool_t::*)(unsigned int), thread_pool_t*, unsigned int>(void (thread_pool_t::*&&)(unsigned int), thread_pool_t*&&, unsigned int&&) (invoke.h:96)
==61966==    by 0x24A4A6: void std::thread::_Invoker<std::tuple<void (thread_pool_t::*)(unsigned int), thread_pool_t*, unsigned int> >::_M_invoke<0ul, 1ul, 2ul>(std::_Index_tuple<0ul, 1ul, 2ul>) (std_thread.h:259)
==61966==    by 0x24A43F: std::thread::_Invoker<std::tuple<void (thread_pool_t::*)(unsigned int), thread_pool_t*, unsigned int> >::operator()() (std_thread.h:266)
==61966==    by 0x24A41F: std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (thread_pool_t::*)(unsigned int), thread_pool_t*, unsigned int> > >::_M_run() (std_thread.h:211)
==61966==    by 0x4DC9252: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30)
==61966==    by 0x50B6AC2: start_thread (pthread_create.c:442)
==61966== 
==61966== HEAP SUMMARY:
==61966==     in use at exit: 305,255,741 bytes in 8,837 blocks
==61966==   total heap usage: 15,449,926 allocs, 15,441,453 frees, 9,041,668,476 bytes allocated
==61966== 
==61966== LEAK SUMMARY:
==61966==    definitely lost: 115,456 bytes in 181 blocks
==61966==    indirectly lost: 0 bytes in 0 blocks
==61966==      possibly lost: 207,856 bytes in 1,599 blocks
==61966==    still reachable: 304,932,429 bytes in 7,057 blocks
==61966==         suppressed: 0 bytes in 0 blocks
==61966== Rerun with --leak-check=full to see details of leaked memory
==61966== 
==61966== For lists of detected and suppressed errors, rerun with: -s
==61966== ERROR SUMMARY: 364 errors from 2 contexts (suppressed: 0 from 0)

`

What version of osm2pgsql are you using?

osm2pgsql version 1.10.0 (1.10.0-9-g4a8f42b5)

What operating system and PostgreSQL/PostGIS version are you using?

Ubuntu 22
Linux HOSTNAME 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Tell us something about your system

2 CPU / 10 Cores each / 40 Threads Total
bare metal
386 GB Ram.

Unable to get any output

I'm sure I have a fundamental mis-understanding of how this is supposed to work but i'm not getting any output. If you want me to post elsewhere let me know..

Current process.

Ive built a new server Ubuntu 22.04 (only chose this because its easy, i'm happy to build a Debian X, for troubleshooting)
used the following tutorial to get a working OSM server --> https://www.linuxbabe.com/linux-server/osm-openstreetmap-tile-server-ubuntu-22-04
I'm able to import planet no problem, but upon a rebuild I'm using a very small pbf.. pitcairns-islands-latest.pbf to verify the whole system is working.

./osm2pgsql --slim -d gis --hstore --multi-geometry --number-processes 32 --tag-transform-script /home/osm/openstreetmap-carto/openstreetmap-carto.lua --style /home/osm/openstreetmap-carto/openstreetmap-carto.style -C 132000 ./pitcairn-islands-latest.osm.pbf

I then attempt to import a small pbf using yours with a small japanese island skikoku pbf like:
./osm2pgsql -G -O flex -d gis -S /osml10n/openstreetmap-carto-hstore-only-l10n.lua ./shikoku-latest.osm.pbf
I stop the render daemon
systemctl stop renderd
clear the tiles directory
rm -rf /var/cache/renderd/tiles/default/*
clear the browser cache
restart the render daemon
systemctl start renderd
use render_list to force tile creation...
render_list -m default -a -z 3 -Z 19 --num-threads=40

When I check the map, the only tiles that are created are the previous ones from pitcairns island.

transcription-daemon does not auto-start after debian packages are installed.

Hi there,

On ubuntu 22.04, with the correct python dependencies as per install manual.

After I do the following:
dpkg -i lua-unaccent_*.deb dpkg -i ../osml10n_*_all.deb

I obtain

sudo dpkg -i /home/alexander/src/osml10n_1.0.1_all.deb Selecting previously unselected package osml10n. dpkg: warning: files list file for package 'osml10n-build-deps' missing; assuming package has no files currently installed (Reading database ... 347404 files and directories currently installed.) Preparing to unpack .../src/osml10n_1.0.1_all.deb ... Unpacking osml10n (1.0.1) ... Setting up osml10n (1.0.1) ... Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142.

I've been researching about that message with no luck.

However, runnning the following command works:
/usr/bin/geo-transcript-srv.py -s -g /usr/share/osml10n/boundaries

And when I open another terminal, and run make test, it all works fine and no error from the server.

lastly, when I open another terminal and run:

`
$ systemctl status osml10n.service
× osml10n.service - OSM l10n transcription server
Loaded: loaded (/lib/systemd/system/osml10n.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sat 2023-03-18 16:23:31 EDT; 8min ago
Process: 228488 ExecStart=/usr/bin/geo-transcript-srv.py -s -g /usr/share/osml10n/boundaries (code=exited, status=1/FAILURE)
Main PID: 228488 (code=exited, status=1/FAILURE)
CPU: 88ms

Mar 18 16:23:31 127.0.0.1localhost systemd[1]: osml10n.service: Scheduled restart job, restart counter is at 5.
Mar 18 16:23:31 127.0.0.1localhost systemd[1]: Stopped OSM l10n transcription server.
Mar 18 16:23:31 127.0.0.1localhost systemd[1]: osml10n.service: Start request repeated too quickly.
Mar 18 16:23:31 127.0.0.1localhost systemd[1]: osml10n.service: Failed with result 'exit-code'.
Mar 18 16:23:31 127.0.0.1localhost systemd[1]: Failed to start OSM l10n transcription server.
`

Empty string with hepburn

This line is problematic, because w['hepburn'] can be empty.

Example: ローソン野々市本町西店 results in kanji=[{'orig': 'ローソン', 'hira': 'ろーそん', 'kana': 'ローソン', 'hepburn': 'rooson', 'kunrei': 'rooson', 'passport': 'rooson'}, {'orig': ' ', 'hira': ' ', 'kana': ' ', 'hepburn': '', 'kunrei': ' ', 'passport': ' '}, {'orig': '野々市', 'hira': 'ののいち', 'kana': 'ノノイチ', 'hepburn': 'nonoichi', 'kunrei': 'nonoiti', 'passport': 'nonoichi'}, {'orig': '本町', 'hira': 'ほんちょう', 'kana': 'ホンチョウ', 'hepburn': 'honchou', 'kunrei': 'hontyou', 'passport': 'honcho'}, {'orig': '西', 'hira': 'にし', 'kana': 'ニシ', 'hepburn': 'nishi', 'kunrei': 'nisi', 'passport': 'nishi'}, {'orig': '店', 'hira': 'みせ', 'kana': 'ミセ', 'hepburn': 'mise', 'kunrei': 'mise', 'passport': 'mise'}]

So it seems the empty space between words appears an empty string in the results array.

undefined symbol: unac_string

On ubuntu:20.04
Following your instructions, added the dependencies, lua_unac makes ok (installed), however when trying to make lua_osml10, getting the error undefined symbol: unac_string.

Additional related info: libunac1-dev is already the newest version (1.8.0-8)
lua5.3/focal,now 5.3.3-1.1ubuntu2 amd64 [installed]

Can you please provide some insight on what I'm doing wrong here ?

Here is the output of "cd lua_osml10 && make deb" :

make[2]: Entering directory '/home/renderer/src/osml10n/lua_osml10' ./tests/runtests.lua /usr/bin/lua: error loading module 'unaccent' from file '/usr/lib/x86_64-linux-gnu/lua/5.3/unaccent.so': /usr/lib/x86_64-linux-gnu/lua/5.3/unaccent.so: undefined symbol: unac_string stack traceback: [C]: in ? [C]: in function 'require' /usr/share/lua/5.3/osml10n/get_localized_name_from_tags.lua:1: in main chunk [C]: in function 'require' /usr/share/lua/5.3/osml10n/init.lua:4: in local 'import' /usr/share/lua/5.3/osml10n/init.lua:14: in main chunk [C]: in function 'require' ./tests/runtests.lua:6: in main chunk [C]: in ? make[2]: *** [Makefile:15: test] Error 1 make[2]: Leaving directory '/home/renderer/src/osml10n/lua_osml10' dh_auto_test: error: make -j8 test returned exit code 2 make[1]: *** [debian/rules:18: build] Error 2 make[1]: Leaving directory '/home/renderer/src/osml10n/lua_osml10' dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2 make: *** [Makefile:18: deb] Error 2

make test doesn't work

Hello,

Running on ubuntu 22.04

During installation, when I run make test, I get the following error.

$ sudo make test [sudo] password for alexander: cd lua_osml10/tests/ && ./runtests.lua /usr/bin/lua: ./osml10n/geo_transcript.lua:10: Can not connect to server 127.0.0.1:8033. Is geo-transcript-srv.py running? stack traceback: [C]: in function 'error' ./osml10n/geo_transcript.lua:10: in main chunk [C]: in function 'require' ./osml10n/init.lua:4: in local 'import' ./osml10n/init.lua:13: in main chunk [C]: in function 'require' ./osml10n.lua:1: in main chunk [C]: in function 'require' ./runtests.lua:6: in main chunk [C]: in ? make: *** [Makefile:31: test] Error 1
I'd appreaciate if anybody can help me.

Inconsistent changes in street name rewriting

I am comparing results from different runs and am getting inconsistent results, i.e. sometimes one, sometimes a different result is returned.

- "<U+202A>пл.дь Ленина|ploschtschad Lenina<U+202C>"
+ "<U+202A>пл. Ленина|ploschtschad Lenina<U+202C>"

The reason are these lines in the RU/UK rewrites:
https://github.com/giggls/osml10n/blob/master/lua_osml10/osml10n/street_abbrev.lua#L157
https://github.com/giggls/osml10n/blob/master/lua_osml10/osml10n/street_abbrev.lua#L171

It depends on the (undefined) order of table iteration, which one is used:
https://github.com/giggls/osml10n/blob/master/lua_osml10/osml10n/street_abbrev.lua#L209
https://github.com/giggls/osml10n/blob/master/lua_osml10/osml10n/street_abbrev.lua#L217
https://github.com/giggls/osml10n/blob/master/lua_osml10/osml10n/street_abbrev.lua#L225

There are other cases in the code when a for loop is used to iterate over tags or languages which might also lead to similar problems. I have another case where these tags:

"name:en"            = "Lantau Trail Section 3"
"name:yue"           = "<U+9CF3><U+51F0><U+5F91>"
"name:zh"            = "<U+9CF3><U+51F0><U+5F91><U+7B2C>3<U+6BB5>"

do result in one of these:

- "<U+202A><U+9CF3><U+51F0><U+5F91><U+7B2C>3<U+6BB5>|Lantau Trail Section 3<U+202C>"
+ "<U+202A><U+9CF3><U+51F0><U+5F91>|Lantau Trail Section 3<U+202C>"

I am not exactly sure what's happening here, but it looks like sometimes the yue und sometimes the zh language is chosen for that street name label.

lake of osml10n.control

CREATE EXTENSION osml10n CASCADE;
ERROR: extension "osml10n" is not available
DETAIL: Could not open extension control file "/usr/share/postgresql/15/extension/osml10n.control": No such file or directory.
HINT: The extension must first be installed on the system where PostgreSQL is running.

Osm2pgsql failed due to ERROR: Error loading lua config

i have completed all above intsallations steps but whenever i run this command :"osm2pgsql -G -O flex -d osm -S openstreetmap-carto-hstore-only-l10n.lua planet.osm.pbf"
i face an ** ERROR** :
"Osm2pgsql failed due to ERROR: Error loading lua config: error loading module 'osml10n' from file '/usr/local/share/lua/5.2/osml10n.lua':
/usr/local/share/lua/5.2/osml10n.lua:1: unexpected symbol near char(127)

Ubuntu 22.04 Luarocks command in INSTALL.md

osml10n/INSTALL.md

Line 19 in 9db770d

luarocks install lrexlib-pcre

In Ubuntu 22.04 the command must be explicit the 5.3 version:
luarocks-5.3 install lrexlib-pcre
Otherwise it will result in an error:
Error: Failed finding Lua header files. You may need to install them or configure LUA_INCDIR.

Add GitHub description

Not really an issue, just to satisfy my OCD 😉

Maybe the text from the release notes?

Localization functions for Openstreetmap

idea: using unix pipe

Have you look at the usage of unix pipe to talk with the daemon rather than using a full ip network stack?

Behaviour when server not available

It is unclear what the behaviour of the Lua code should be when the server is not available either when first connecting or when the connection fails at some point. In some contexts it might make sense to keep going and not transliterating names (for instance when this is for temporary map views only, when it is more important to show something rather than an error or so), in other cases it might make more sense to stop processing with an error or wait and retry.

To solve this properly I think we need to expose this to the "user code". So instead of magically connecting to the server when the library is loaded add an explicit init or connect call or something. If that fails the user code can then decide what to do. Similarly there should be a defined error return from the libary calls, probably using the standard Lua pcall mechanism.

run at ubuntu return error

I have 2 errors:

1: Error return when I call make deb in lua_unac directory:

root@:/home/renderaccount/data/osml10n-master/lua_unac# make deb
dpkg-buildpackage -b -uc
dpkg-buildpackage: info: source package lua-unaccent
dpkg-buildpackage: info: source version 1.8-1
dpkg-buildpackage: info: source distribution unstable
dpkg-buildpackage: info: source changed by Sven Geggus <[email protected]>
dpkg-buildpackage: info: host architecture amd64
dpkg-source --before-build .
debian/rules clean
make[1]: Entering directory '/home/renderaccount/data/osml10n-master/lua_unac'
dh clean
dh_auto_clean
make -j16 clean
make[2]: Entering directory '/home/renderaccount/data/osml10n-master/lua_unac'
rm -f unaccent.so
make[2]: Leaving directory '/home/renderaccount/data/osml10n-master/lua_unac'
dh_clean
make[1]: Leaving directory '/home/renderaccount/data/osml10n-master/lua_unac'
debian/rules build
make[1]: Entering directory '/home/renderaccount/data/osml10n-master/lua_unac'
dh build
dh_update_autotools_config
dh_autoreconf
dh_auto_configure
dh_auto_build
make -j16 "INSTALL=install --strip-program=true"
make[2]: Entering directory '/home/renderaccount/data/osml10n-master/lua_unac'
gcc -Wall -shared -fPIC -o unaccent.so -I/usr/include/lua5.3 -llua5.3 -Wl,--no-as-needed -lunac lua_unac.c
lua_unac.c:14:10: fatal error: lua.h: No such file or directory
14 | #include <lua.h>
| ^~~~~~~
compilation terminated.
make[2]: *** [Makefile:7: unaccent.so] Error 1
make[2]: Leaving directory '/home/renderaccount/data/osml10n-master/lua_unac'
dh_auto_build: error: make -j16 "INSTALL=install --strip-program=true" returned exit code 2
make[1]: *** [debian/rules:18: build] Error 2
make[1]: Leaving directory '/home/renderaccount/data/osml10n-master/lua_unac'
dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2
make: *** [Makefile:13: deb] Error 2
root@:/home/renderaccount/data/osml10n-master/lua_unac# make test
make: *** No rule to make target 'test'. Stop.

2: Error when i run osm2pgsql

renderaccount@****:~$ osm2pgsql -G -O -d gis -S ~/data/osml10n-master/openstreetmap-carto-hstore-only-l10n.lua asia-latest.osm.pbf
osm2pgsql version 1.2.1 (64 bit id space)

Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=800MB, maxblocks=12800*65536, allocation method=3
DB writer thread failed due to ERROR: Connection to database failed: FATAL: database "renderaccount" does not exist

Some tests fail: 85 tests passed, 2 tests failed

I fired up a fresh Debian VM to investigate in failing tests:

…
calling osml10n.geo_transcript("42", "thai ถนนข้าวสาร 100", { 100, 14, 101, 15 }):
[ERROR] (expected thai thanon khaosan 100, got thai thanon khaosara 100)
calling osml10n.geo_transcript("42", "อนุสาวรีย์พระยารัษฎาณุประดิษฐ์", { 100, 14, 101, 15 }):
[ERROR] (expected anusawari phraya ratsa da nu pradit, got anusawari phraya rat da nu pradit)
…
85 tests passed, 2 tests failed.
make: *** [Makefile:31: test] Error 1

I tried older versions of python-pinyin-jyutping-sentence, since it outputs trackbacks (see below) from version 1.3. With older versions of python-pinyin-jyutping-sentence the same tests fail. I think the trackback pointed me in the wrong direction.

Any ideas?

The translation service fires this trackback (repeating 4 times) with python-pinyin-jyutping-sentence==1.3:

Feb 14 12:44:53 debian11 systemd[1]: Starting OSM l10n transcription server...
Feb 14 12:44:57 debian11 systemd[1]: Started OSM l10n transcription server.
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]: Loading osml10n transcription server: --- Logging error ---
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]: Traceback (most recent call last):
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]:   File "/usr/lib/python3.9/logging/__init__.py", line 1082, in emit
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]:     stream.write(msg + self.terminator)
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]: ValueError: I/O operation on closed file.
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]: Call stack:
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]:   File "/usr/bin/geo-transcript-srv.py", line 275, in <module>
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]:     asyncio.run(main())
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]:   File "/usr/lib/python3.9/asyncio/runners.py", line 44, in run
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]:     return loop.run_until_complete(main)
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]:   File "/usr/lib/python3.9/asyncio/base_events.py", line 629, in run_until_complete
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]:     self.run_forever()
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]:   File "/usr/lib/python3.9/asyncio/base_events.py", line 596, in run_forever
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]:     self._run_once()
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]:   File "/usr/lib/python3.9/asyncio/base_events.py", line 1890, in _run_once
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]:     handle._run()
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]:   File "/usr/lib/python3.9/asyncio/events.py", line 80, in _run
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]:     self._context.run(self._callback, *self._args)
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]:   File "/usr/bin/geo-transcript-srv.py", line 248, in handle_connection
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]:     reply = tc.transcript(id,cc,name)
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]:   File "/usr/bin/geo-transcript-srv.py", line 149, in transcript
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]:     return(cantonese_transcript(unistr))
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]:   File "/usr/bin/geo-transcript-srv.py", line 93, in cantonese_transcript
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]:     transcript=pinyin_jyutping_sentence.jyutping(st, spaces=True)
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]:   File "/usr/local/lib/python3.9/dist-packages/pinyin_jyutping_sentence/__init__.py", line 390, in process_sentence_jyutping
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]:     return self.process_sentence(sentence, self.conversion_data.jyutping_word_map, self.conversion_data.jyutping_char_map, self.decode_jyutping, tone_numbers, spaces, remove_tones)
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]:   File "/usr/local/lib/python3.9/dist-packages/pinyin_jyutping_sentence/__init__.py", line 379, in process_sentence
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]:     word_list = list(seg_list)
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]:   File "/usr/local/lib/python3.9/dist-packages/jieba/__init__.py", line 325, in cut
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]:     for word in cut_block(blk):
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]:   File "/usr/local/lib/python3.9/dist-packages/jieba/__init__.py", line 250, in __cut_DAG
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]:     DAG = self.get_DAG(sentence)
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]:   File "/usr/local/lib/python3.9/dist-packages/jieba/__init__.py", line 181, in get_DAG
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]:     self.check_initialized()
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]:   File "/usr/local/lib/python3.9/dist-packages/jieba/__init__.py", line 170, in check_initialized
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]:     self.initialize()
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]:   File "/usr/local/lib/python3.9/dist-packages/jieba/__init__.py", line 113, in initialize
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]:     default_logger.debug("Building prefix dict from %s ..." % (abs_path or 'the default dictionary'))
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]: Message: 'Building prefix dict from the default dictionary ...'
Feb 14 12:45:18 debian11 geo-transcript-srv.py[11765]: Arguments: ()

My test setup:

root@debian11:~# apt-get update && apt-get -y upgrade
root@debian11:~# apt-get install -y dpkg-dev debhelper git libunac1-dev \
    luarocks lua5.3 libpcre3-dev liblua5.3-dev python3-icu python3-shapely \
    python3-pip python3-sdnotify python3-requests
root@debian11:~# luarocks install lrexlib-pcre
root@debian11:~# pip install pykakasi tltk pinyin_jyutping_sentence scikit-learn
…
Successfully built pinyin-jyutping-sentence tltk jaconv jieba fst-pso miniful sklearn
Installing collected packages: numpy, scipy, miniful, simpful, pytz, python-dateutil, fst-pso, threadpoolctl, pyfume, pandas, joblib, wrapt, tqdm, tabulate, smart-open, scikit-learn, regex, python-crfsuite, FuzzyTM, click, sklearn-crfsuite, sklearn, nltk, jieba, jaconv, gensim, deprecated, tltk, pykakasi, pinyin-jyutping-sentence
  Attempting uninstall: numpy
    Found existing installation: numpy 1.19.5
    Not uninstalling numpy at /usr/lib/python3/dist-packages, outside environment /usr
    Can't uninstall 'numpy'. No files were found to uninstall.
Successfully installed FuzzyTM-2.0.5 click-8.1.3 deprecated-1.2.13 fst-pso-1.8.1 gensim-4.3.0 jaconv-0.3.3 jieba-0.42.1 joblib-1.2.0 miniful-0.0.6 nltk-3.8.1 numpy-1.24.2 pandas-1.5.3 pinyin-jyutping-sentence-1.3 pyfume-0.2.25 pykakasi-2.2.1 python-crfsuite-0.9.9 python-dateutil-2.8.2 pytz-2022.7.1 regex-2022.10.31 scikit-learn-1.2.1 scipy-1.10.0 simpful-2.9.0 sklearn-0.0 sklearn-crfsuite-0.3.6 smart-open-6.3.0 tabulate-0.9.0 threadpoolctl-3.1.0 tltk-1.6.3 tqdm-4.64.1 wrapt-1.14.1
user@debian11:~$ git clone https://github.com/giggls/osml10n.git
user@debian11:~$ cd osml10n/lua_unac
user@debian11:~/osml10n/lua_unac$ make deb
user@debian11:~/osml10n/lua_unac$ cd ..
user@debian11:~/osml10n$ make deb
root@debian11:~# dpkg -i /home/user/osml10n/lua-unaccent_1.8-1_amd64.deb \
    /home/user/osml10n_1.0_all.deb
root@debian11:~# systemctl status osml10n.service
● osml10n.service - OSM l10n transcription server
     Loaded: loaded (/lib/systemd/system/osml10n.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2023-02-14 12:15:04 CET; 23s ago
   Main PID: 11240 (geo-transcript-)
      Tasks: 4 (limit: 2337)
     Memory: 530.6M
        CPU: 4.197s
     CGroup: /system.slice/osml10n.service
             └─11240 /usr/bin/python3 /usr/bin/geo-transcript-srv.py -s -g /usr/share/osml10n/boundaries

Feb 14 12:15:00 debian11 systemd[1]: Starting OSM l10n transcription server...
Feb 14 12:15:04 debian11 systemd[1]: Started OSM l10n transcription server.

mml file

Hello,

Sorry I am not expert in OSM can you please help me understand something abut next steps?
If I understand after packages installation and importing DB I need some custom mml file to build XML stylesheet? because with standard project.mml file from openstreetmap-carto I was not able to make renderd work properly...
I am using debian11, your code is installed and listening on port 8033.
make test result: 87 test passed, 0 tests filed.
OSM file successfully imported. Everything is done as mentioned in your installation guide.

Thanks in advance!

Server cannot utilize tltk library

Just installed on Debian Bullseye 11.5.

$ /usr/bin/geo-transcript-srv.py -v -g /usr/share/osml10n/boundaries
Loading osml10n transcription server:

ERROR: unable to load required python modules, please install them as follows:
pip install pykakasi
pip install tltk
pip install pinyin_jyutping_sentence

I figured out by comments in the script this is because of the tltk lib, but it is installed:

$ pip install tltk
Requirement already satisfied: tltk in ./python3.9/dist-packages (1.6)
Requirement already satisfied: sklearn in ./python3.9/dist-packages (from tltk) (0.0.post1)
Requirement already satisfied: gensim in ./python3.9/dist-packages (from tltk) (4.2.0)
Requirement already satisfied: nltk in ./python3.9/dist-packages (from tltk) (3.7)
Requirement already satisfied: sklearn-crfsuite in ./python3.9/dist-packages (from tltk) (0.3.6)
Requirement already satisfied: scipy>=0.18.1 in ./python3.9/dist-packages (from gensim->tltk) (1.9.3)
Requirement already satisfied: numpy>=1.17.0 in /usr/lib/python3/dist-packages (from gensim->tltk) (1.19.5)
Requirement already satisfied: smart-open>=1.8.1 in ./python3.9/dist-packages (from gensim->tltk) (6.2.0)
Requirement already satisfied: click in ./python3.9/dist-packages (from nltk->tltk) (8.1.3)
Requirement already satisfied: tqdm in ./python3.9/dist-packages (from nltk->tltk) (4.64.1)
Requirement already satisfied: regex>=2021.8.3 in ./python3.9/dist-packages (from nltk->tltk) (2022.10.31)
Requirement already satisfied: joblib in ./python3.9/dist-packages (from nltk->tltk) (1.2.0)
Requirement already satisfied: tabulate in ./python3.9/dist-packages (from sklearn-crfsuite->tltk) (0.9.0)
Requirement already satisfied: six in /usr/lib/python3/dist-packages (from sklearn-crfsuite->tltk) (1.16.0)
Requirement already satisfied: python-crfsuite>=0.8.3 in ./python3.9/dist-packages (from sklearn-crfsuite->tltk) (0.9.8)

Any suggestions?

Debian 12 Bookworm Support?

Thank you for the detailed instructions for Debian 11: https://github.com/giggls/osml10n/blob/master/INSTALL.md.

I have tested this for Debian 12 (Bookworm). Install on Debian 11 and update to Debian 12 is fine.

The first part worked fine in a new install, too:

apt install libunac1-dev luarocks lua5.3 libpcre3-dev liblua5.3-dev
luarocks install lrexlib-pcre
apt install python3-icu python3-shapely python3-pip python3-sdnotify python3-requests

But then I get into trouble

PIP

pip install is no longer possible because of https://peps.python.org/pep-0668/

$ pip install pykakasi
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.
    
    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.
    
    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.
    
    See /usr/share/doc/python3.11/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

Virtual environment

First I tried it with a virtual environment. Then I ran into the problems described here:

attapol/tltk#7

--break-system-packages

With the parameter --break-system-packages I could install everything. There were warnings.

$ pip install pykakasi --break-system-packages
Defaulting to user installation because normal site-packages is not writeable
Collecting pykakasi
  Downloading pykakasi-2.2.1-py3-none-any.whl (2.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.4/2.4 MB 1.9 MB/s eta 0:00:00
Collecting jaconv
  Downloading jaconv-0.3.4.tar.gz (16 kB)
  Preparing metadata (setup.py) ... done
Collecting deprecated
  Downloading Deprecated-1.2.14-py2.py3-none-any.whl (9.6 kB)
Collecting wrapt<2,>=1.10
  Downloading wrapt-1.15.0-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (78 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.9/78.9 kB 1.6 MB/s eta 0:00:00
Building wheels for collected packages: jaconv
  Building wheel for jaconv (setup.py) ... done
  Created wheel for jaconv: filename=jaconv-0.3.4-py3-none-any.whl size=16417 sha256=b6dd3aa752842867196dfb3bf1668eed29ba06f24d8eb42ea0010e3ce1726415
  Stored in directory: /home/vagrant/.cache/pip/wheels/36/3e/0d/fe7664bb3c7fdddde7449bc3bc85faaf673a6cda43ed7767c7
Successfully built jaconv
Installing collected packages: jaconv, wrapt, deprecated, pykakasi
Successfully installed deprecated-1.2.14 jaconv-0.3.4 pykakasi-2.2.1 wrapt-1.15.0

$ pip install tltk --break-system-packages
Defaulting to user installation because normal site-packages is not writeable
Collecting tltk
  Downloading tltk-1.6.8.tar.gz (18.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18.7/18.7 MB 887.3 kB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting gensim
  Downloading gensim-4.3.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (26.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 26.7/26.7 MB 1.4 MB/s eta 0:00:00
Collecting nltk
  Downloading nltk-3.8.1-py3-none-any.whl (1.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.5/1.5 MB 2.0 MB/s eta 0:00:00
Collecting sklearn
  Downloading sklearn-0.0.post7.tar.gz (3.6 kB)
  Preparing metadata (setup.py) ... done
Collecting sklearn_crfsuite
  Downloading sklearn_crfsuite-0.3.6-py2.py3-none-any.whl (12 kB)
Requirement already satisfied: numpy>=1.18.5 in /usr/lib/python3/dist-packages (from gensim->tltk) (1.24.2)
Collecting scipy>=1.7.0
  Downloading scipy-1.11.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (36.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 36.3/36.3 MB 1.8 MB/s eta 0:00:00
Collecting smart-open>=1.8.1
  Downloading smart_open-6.3.0-py3-none-any.whl (56 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56.8/56.8 kB 2.5 MB/s eta 0:00:00
Collecting click
  Downloading click-8.1.7-py3-none-any.whl (97 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 97.9/97.9 kB 2.2 MB/s eta 0:00:00
Collecting joblib
  Downloading joblib-1.3.2-py3-none-any.whl (302 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 302.2/302.2 kB 2.8 MB/s eta 0:00:00
Collecting regex>=2021.8.3
  Downloading regex-2023.8.8-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (782 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 782.4/782.4 kB 2.0 MB/s eta 0:00:00
Collecting tqdm
  Downloading tqdm-4.66.1-py3-none-any.whl (78 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.3/78.3 kB 2.3 MB/s eta 0:00:00
Collecting python-crfsuite>=0.8.3
  Downloading python_crfsuite-0.9.9-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.0/1.0 MB 2.3 MB/s eta 0:00:00
Requirement already satisfied: six in /usr/lib/python3/dist-packages (from sklearn_crfsuite->tltk) (1.16.0)
Collecting tabulate
  Downloading tabulate-0.9.0-py3-none-any.whl (35 kB)
Building wheels for collected packages: tltk, sklearn
  Building wheel for tltk (setup.py) ... done
  Created wheel for tltk: filename=tltk-1.6.8-py3-none-any.whl size=18868898 sha256=60d06fda13d0935b1f6d9a7fd018de6f33beb4a1a895f68d4fb6ddb6507dcb0d
  Stored in directory: /home/vagrant/.cache/pip/wheels/76/d2/3d/a1701c265f3734fd29266aa8ee4d9cb6bda32d3c2afad4b980
  Building wheel for sklearn (setup.py) ... done
  Created wheel for sklearn: filename=sklearn-0.0.post7-py3-none-any.whl size=2953 sha256=26ccddbfdab1246b276f143f8ff51a016f834e70a2875e956130673b2480689b
  Stored in directory: /home/vagrant/.cache/pip/wheels/08/17/db/46486df299847133f69165c029fe388fac45781eb5d436c569
Successfully built tltk sklearn
Installing collected packages: sklearn, python-crfsuite, tqdm, tabulate, smart-open, scipy, regex, joblib, click, sklearn_crfsuite, nltk, gensim, tltk
  WARNING: The script tqdm is installed in '/home/vagrant/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The script tabulate is installed in '/home/vagrant/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The script nltk is installed in '/home/vagrant/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The script tltk is installed in '/home/vagrant/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed click-8.1.7 gensim-4.3.2 joblib-1.3.2 nltk-3.8.1 python-crfsuite-0.9.9 regex-2023.8.8 scipy-1.11.2 sklearn-0.0.post7 sklearn_crfsuite-0.3.6 smart-open-6.3.0 tabulate-0.9.0 tltk-1.6.8 tqdm-4.66.1

$ pip install pinyin_jyutping_sentence --break-system-packages
Defaulting to user installation because normal site-packages is not writeable
Collecting pinyin_jyutping_sentence
  Downloading pinyin_jyutping_sentence-1.3.tar.gz (12.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.4/12.4 MB 2.0 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting jieba
  Downloading jieba-0.42.1.tar.gz (19.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.2/19.2 MB 1.5 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: pinyin_jyutping_sentence, jieba
  Building wheel for pinyin_jyutping_sentence (setup.py) ... done
  Created wheel for pinyin_jyutping_sentence: filename=pinyin_jyutping_sentence-1.3-py3-none-any.whl size=12492668 sha256=5402d35f65468c29f53b68eec01f339062650b565372f52e690e04c2bf7b448a
  Stored in directory: /home/vagrant/.cache/pip/wheels/ac/d9/9a/46f4077d8c97134b748d1662a829b63e4845c85f7916fac8b9
  Building wheel for jieba (setup.py) ... done
  Created wheel for jieba: filename=jieba-0.42.1-py3-none-any.whl size=19314458 sha256=175706510a98b0e2a9cca318d779cffa8ee9f3f5f8bb11e4e8ae12f2d508683a
  Stored in directory: /home/vagrant/.cache/pip/wheels/ac/60/cf/538a1f183409caf1fc136b5d2c2dee329001ef6da2c5084bef
Successfully built pinyin_jyutping_sentence jieba
Installing collected packages: jieba, pinyin_jyutping_sentence
Successfully installed jieba-0.42.1 pinyin_jyutping_sentence-1.3

Debian Pakete

Then I created the Debian packages and get the message Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 145.

apt install debhelper

/osml10n$ make deb
sed -e 's;/usr/local;/usr;g' transcription-daemon/geo-transcript-srv.service >debian/osml10n.service
dpkg-buildpackage -b -uc
dpkg-buildpackage: info: source package osml10n
dpkg-buildpackage: info: source version 1.0.1
dpkg-buildpackage: info: source distribution unstable
dpkg-buildpackage: info: source changed by Sven Geggus <[email protected]>
dpkg-buildpackage: info: host architecture amd64
 dpkg-source --before-build .
 fakeroot debian/rules clean
make[1]: Entering directory '/home/vagrant/osml10n'
dh clean
   dh_auto_clean
	make -j1 clean
make[2]: Entering directory '/home/vagrant/osml10n'
echo "there is nothing to clean"
there is nothing to clean
make[2]: Leaving directory '/home/vagrant/osml10n'
   dh_clean
make[1]: Leaving directory '/home/vagrant/osml10n'
 debian/rules build
make[1]: Entering directory '/home/vagrant/osml10n'
dh build
   dh_update_autotools_config
   dh_autoreconf
   dh_auto_configure
   dh_auto_build
	make -j1 "INSTALL=install --strip-program=true"
make[2]: Entering directory '/home/vagrant/osml10n'
echo "These are lua and python scripts, there is nothing to build here"
These are lua and python scripts, there is nothing to build here
make[2]: Leaving directory '/home/vagrant/osml10n'
   debian/rules override_dh_auto_test
make[2]: Entering directory '/home/vagrant/osml10n'
/bin/true
make[2]: Leaving directory '/home/vagrant/osml10n'
   create-stamp debian/debhelper-build-stamp
make[1]: Leaving directory '/home/vagrant/osml10n'
 fakeroot debian/rules binary
make[1]: Entering directory '/home/vagrant/osml10n'
dh binary
   dh_testroot
   dh_prep
   dh_installdirs
   debian/rules override_dh_auto_install
make[2]: Entering directory '/home/vagrant/osml10n'
dh_auto_install -- prefix=/usr
	make -j1 install DESTDIR=/home/vagrant/osml10n/debian/osml10n AM_UPDATE_INFO_DIR=no "INSTALL=install --strip-program=true" prefix=/usr
make[3]: Entering directory '/home/vagrant/osml10n'
mkdir -p /home/vagrant/osml10n/debian/osml10n/usr/share/lua/5.3
cp -a lua_osml10/osml10n /home/vagrant/osml10n/debian/osml10n/usr/share/lua/5.3/
chmod -R go+rX /home/vagrant/osml10n/debian/osml10n/usr/share/lua/5.3/osml10n
sed -e 's;/usr/local;/usr;g' transcription-daemon/geo-transcript-srv.service >debian/osml10n.service
mkdir -p /home/vagrant/osml10n/debian/osml10n/usr/bin
cp -a transcription-daemon/geo-transcript-srv.py /home/vagrant/osml10n/debian/osml10n/usr/bin
chmod 755 /home/vagrant/osml10n/debian/osml10n/usr/bin/geo-transcript-srv.py
mkdir -p /home/vagrant/osml10n/debian/osml10n/usr/share/osml10n/boundaries
cp -a boundaries/* /home/vagrant/osml10n/debian/osml10n/usr/share/osml10n/boundaries
cp transcription-cli/transcribe.py /home/vagrant/osml10n/debian/osml10n/usr/bin
chmod 755 /home/vagrant/osml10n/debian/osml10n/usr/bin/transcribe.py
make[3]: Leaving directory '/home/vagrant/osml10n'
make[2]: Leaving directory '/home/vagrant/osml10n'
   dh_installdocs
   dh_installchangelogs
   dh_installinit
   dh_installsystemd
   dh_perl
   dh_link
   dh_strip_nondeterminism
   dh_compress
   dh_fixperms
   dh_missing
   dh_installdeb
   dh_gencontrol
   dh_md5sums
   dh_builddeb
dpkg-deb: building package 'osml10n' in '../osml10n_1.0.1_all.deb'.
make[1]: Leaving directory '/home/vagrant/osml10n'
 dpkg-genbuildinfo --build=binary -O../osml10n_1.0.1_amd64.buildinfo
 dpkg-genchanges --build=binary -O../osml10n_1.0.1_amd64.changes
dpkg-genchanges: info: binary-only upload (no source code included)
 dpkg-source --after-build .
dpkg-buildpackage: info: binary-only upload (no source included)
vagrant@bookworm:~/osml10n$ cd lua_unac/
vagrant@bookworm:~/osml10n/lua_unac$ make deb
dpkg-buildpackage -b -uc
dpkg-buildpackage: info: source package lua-unaccent
dpkg-buildpackage: info: source version 1.8-1
dpkg-buildpackage: info: source distribution unstable
dpkg-buildpackage: info: source changed by Sven Geggus <[email protected]>
dpkg-buildpackage: info: host architecture amd64
 dpkg-source --before-build .
 fakeroot debian/rules clean
make[1]: Entering directory '/home/vagrant/osml10n/lua_unac'
dh clean
   dh_auto_clean
	make -j1 clean
make[2]: Entering directory '/home/vagrant/osml10n/lua_unac'
rm -f unaccent.so
make[2]: Leaving directory '/home/vagrant/osml10n/lua_unac'
   dh_clean
make[1]: Leaving directory '/home/vagrant/osml10n/lua_unac'
 debian/rules build
make[1]: Entering directory '/home/vagrant/osml10n/lua_unac'
dh build
   dh_update_autotools_config
   dh_autoreconf
   dh_auto_configure
   dh_auto_build
	make -j1 "INSTALL=install --strip-program=true"
make[2]: Entering directory '/home/vagrant/osml10n/lua_unac'
gcc -Wall -shared -fPIC -o unaccent.so -I/usr/include/lua5.3 -llua5.3 -Wl,--no-as-needed -lunac lua_unac.c
make[2]: Leaving directory '/home/vagrant/osml10n/lua_unac'
   dh_auto_test
   create-stamp debian/debhelper-build-stamp
make[1]: Leaving directory '/home/vagrant/osml10n/lua_unac'
 fakeroot debian/rules binary
make[1]: Entering directory '/home/vagrant/osml10n/lua_unac'
dh binary
   dh_testroot
   dh_prep
   dh_installdirs
   dh_auto_install --destdir=debian/lua-unaccent/
	make -j1 install DESTDIR=/home/vagrant/osml10n/lua_unac/debian/lua-unaccent AM_UPDATE_INFO_DIR=no "INSTALL=install --strip-program=true"
make[2]: Entering directory '/home/vagrant/osml10n/lua_unac'
install -m 644 unaccent.so /home/vagrant/osml10n/lua_unac/debian/lua-unaccent/usr/lib/x86_64-linux-gnu/lua/5.3/
make[2]: Leaving directory '/home/vagrant/osml10n/lua_unac'
   dh_installdocs
   dh_installchangelogs
   dh_perl
   dh_link
   dh_strip_nondeterminism
   dh_compress
   dh_fixperms
   dh_missing
   dh_strip
   dh_makeshlibs
   dh_shlibdeps
   dh_installdeb
   dh_gencontrol
dpkg-gencontrol: warning: package lua-unaccent: substitution variable ${shlibs:Depends} unused, but is defined
dpkg-gencontrol: warning: package lua-unaccent: substitution variable ${shlibs:Depends} unused, but is defined
   dh_md5sums
   dh_builddeb
dpkg-deb: building package 'lua-unaccent' in '../lua-unaccent_1.8-1_amd64.deb'.
dpkg-deb: building package 'lua-unaccent-dbgsym' in '../lua-unaccent-dbgsym_1.8-1_amd64.deb'.
make[1]: Leaving directory '/home/vagrant/osml10n/lua_unac'
 dpkg-genbuildinfo --build=binary -O../lua-unaccent_1.8-1_amd64.buildinfo
 dpkg-genchanges --build=binary -O../lua-unaccent_1.8-1_amd64.changes
dpkg-genchanges: info: binary-only upload (no source code included)
 dpkg-source --after-build .
dpkg-buildpackage: info: binary-only upload (no source included)

/osml10n$ sudo dpkg -i lua-unaccent_*.deb
Selecting previously unselected package lua-unaccent.
(Reading database ... 43498 files and directories currently installed.)
Preparing to unpack lua-unaccent_1.8-1_amd64.deb ...
Unpacking lua-unaccent (1.8-1) ...
Setting up lua-unaccent (1.8-1) ...

/osml10n$ sudo dpkg -i ../osml10n_*_all.deb
Selecting previously unselected package osml10n.
(Reading database ... 43502 files and directories currently installed.)
Preparing to unpack ../osml10n_1.0.1_all.deb ...
Unpacking osml10n (1.0.1) ...
Setting up osml10n (1.0.1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/osml10n.service → /lib/systemd/system/osml10n.service.
Could not execute systemctl:  at /usr/bin/deb-systemd-invoke line 145.

Line 145 of the file /usr/bin/deb-systemd-invoke is the one in this if-statement:

if (@start_units) {
    system('systemctl', '--quiet', @instance_args, $action, @start_units) == 0 or die("Could not execute systemctl: $!");
 }

This is my whole file /usr/bin/deb-systemd-invoke.


/usr/bin$ cat deb-systemd-invoke 
#!/usr/bin/perl
# vim:ts=4:sw=4:expandtab
# © 2013 Michael Stapelberg <[email protected]>
#
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
#
#     * Redistributions of source code must retain the above copyright
#       notice, this list of conditions and the following disclaimer.
#
#     * Redistributions in binary form must reproduce the above copyright
#       notice, this list of conditions and the following disclaimer in the
#       documentation and/or other materials provided with the distribution.
#
#     * Neither the name of Michael Stapelberg nor the
#       names of contributors may be used to endorse or promote products
#       derived from this software without specific prior written permission.
# .
# THIS SOFTWARE IS PROVIDED BY Michael Stapelberg ''AS IS'' AND ANY
# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL Michael Stapelberg BE LIABLE FOR ANY
# DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
# ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

=head1 NAME

deb-systemd-invoke - wrapper around systemctl, respecting policy-rc.d

=head1 SYNOPSIS

B<deb-systemd-invoke> [B<--user>] start|stop|restart S<I<unit file> ...>

=head1 DESCRIPTION

B<deb-systemd-invoke> is a Debian-specific helper script which asks
/usr/sbin/policy-rc.d before performing a systemctl call.

B<deb-systemd-invoke> is intended to be used from maintscripts to start
systemd unit files. It is specifically NOT intended to be used interactively by
users. Instead, users should run systemd and use systemctl, or not bother about
the systemd enabled state in case they are not running systemd.

=cut

use strict;
use warnings;
use Getopt::Long; # in core since Perl 5

if (@ARGV < 2) {
    print STDERR "Syntax: $0 <action> <unit file> [<unit file> ...]\n";
    exit 1;
}

my $is_system = 1;
my @instances = ();
my $result = GetOptions(
    "user" => sub { $is_system = 0; },
    "system" => sub { $is_system = 1; }, # default
);

my $policyhelper = '/usr/sbin/policy-rc.d';
if (length $ENV{DPKG_ROOT}) {
    $policyhelper = $ENV{DPKG_ROOT} . $policyhelper;
}
my @units = @ARGV;
my $action = shift @units;
if (-x $policyhelper) {
    for my $unit (@units) {
        system(qq|$policyhelper $unit "$action"|);

        # 0 or 104 means run
        # 101 means do not run
        my $exitcode = ($? >> 8);
        if ($exitcode == 101) {
            print STDERR "$policyhelper returned 101, not running '" . join(' ', @ARGV) . "'\n";
            exit 0;
        } elsif ($exitcode != 104 && $exitcode != 0) {
            print STDERR "deb-systemd-invoke only supports $policyhelper return codes 0, 101, and 104!\n";
            print STDERR "Got return code $exitcode, ignoring.\n";
        }
    }
}

if (!$is_system) {
    # '--machine <ID>@' was added in v250 and v249.10, before that we can't talk to arbitrary user instances
    my $systemctl_version = `systemctl --version --quiet | sed -n -r "s/systemd ([0-9]+) \\(.*/\\1/p"`;
    chomp ($systemctl_version);
    if (system('dpkg', '--compare-versions', $systemctl_version, 'ge', '249') != 0) {
            print STDERR "systemctl version $systemctl_version does not support acting on user instance, skipping\n";
            exit 0;
    }

    # Each user instance of the manager has a corresponding user@<id<.service unit.
    # Get the full list of IDs, so that we can talk to each user instance to start/stop
    # user units.
    @instances = `systemctl --no-legend --quiet list-units 'user@*' | sed -n -r 's/.*user@([0-9]+).service.*/\\1/p'`;
} else {
    push @instances, 'system';
}

# If the job is disabled and is not currently running, the job is not started or restarted.
# However, if the job is disabled but has been forced into the running state, we *do* stop
# and restart it since this is expected behaviour for the admin who forced the start.
# We don't autostart static units either.
if ($action eq "start" || $action eq "restart") {
    my $global_exit_code = 0;
    my @start_units = ();

    for my $instance (@instances) {
        my @instance_args = ();

        if ($instance eq 'system') {
            push @instance_args, '--system';
        } else {
            chomp ($instance);
            push @instance_args, '--user', '--machine', "$instance@";
        }

        for my $unit (@units) {
            my $unit_installed = 0;
            my $enabled_output = `systemctl @instance_args is-enabled -- '$unit'`;
            # matching enabled and enabled-runtime as an installed non static unit
            if ($enabled_output =~ /enabled/) {
                $unit_installed = 1;
            }
            system('systemctl', @instance_args, '--quiet', 'is-active', '--', $unit);
            my $unit_active = $?>>8 == 0 ? 1 : 0;
            if (!$unit_installed && $action eq "start") {
                print STDERR "$unit is a disabled or a static unit, not starting it.\n";
            } elsif (!$unit_installed && !$unit_active && $action eq "restart") {
                print STDERR "$unit is a disabled or a static unit not running, not starting it.\n";
            }
            else {
                push @start_units, $unit;
            }
        }
        if (@start_units) {
            system('systemctl', '--quiet', @instance_args, $action, @start_units) == 0 or die("Could not execute systemctl: $!");
        }
    }
    exit(0);
} elsif ($action eq "stop" && !$is_system) {
    my $global_exit_code = 0;

    for my $instance (@instances) {
        chomp ($instance);
        system('systemctl', '--quiet', '--user', '--machine', "$instance@", $action, @units);
    }
    exit(0);
} else {
    exec('systemctl', @ARGV);
}

sudo journalctl -u osml10n looks like this:

$ sudo journalctl -u osml10n
Aug 26 07:49:50 bookworm systemd[1]: [🡕] /lib/systemd/system/osml10n.service:10: Special user nobody configured, this is not safe!
Aug 26 07:49:50 bookworm systemd[1]: Starting osml10n.service - OSM l10n transcription server...
Aug 26 07:49:52 bookworm geo-transcript-srv.py[13842]: Loading osml10n transcription server:
Aug 26 07:49:52 bookworm systemd[1]: osml10n.service: Main process exited, code=exited, status=1/FAILURE
Aug 26 07:49:52 bookworm systemd[1]: osml10n.service: Failed with result 'exit-code'.
Aug 26 07:49:52 bookworm systemd[1]: Failed to start osml10n.service - OSM l10n transcription server.
Aug 26 07:49:52 bookworm systemd[1]: osml10n.service: Consumed 1.221s CPU time.
Aug 26 07:49:52 bookworm systemd[1]: osml10n.service: Scheduled restart job, restart counter is at 1.
Aug 26 07:49:52 bookworm systemd[1]: Stopped osml10n.service - OSM l10n transcription server.
Aug 26 07:49:52 bookworm systemd[1]: osml10n.service: Consumed 1.221s CPU time.
Aug 26 07:49:52 bookworm systemd[1]: Starting osml10n.service - OSM l10n transcription server...
Aug 26 07:49:52 bookworm geo-transcript-srv.py[13845]: Loading osml10n transcription server:
Aug 26 07:49:52 bookworm systemd[1]: osml10n.service: Main process exited, code=exited, status=1/FAILURE
Aug 26 07:49:52 bookworm systemd[1]: osml10n.service: Failed with result 'exit-code'.
Aug 26 07:49:52 bookworm systemd[1]: Failed to start osml10n.service - OSM l10n transcription server.
Aug 26 07:49:52 bookworm systemd[1]: osml10n.service: Scheduled restart job, restart counter is at 2.
Aug 26 07:49:52 bookworm systemd[1]: Stopped osml10n.service - OSM l10n transcription server.
Aug 26 07:49:52 bookworm systemd[1]: Starting osml10n.service - OSM l10n transcription server...
Aug 26 07:49:53 bookworm geo-transcript-srv.py[13847]: Loading osml10n transcription server:
Aug 26 07:49:53 bookworm systemd[1]: osml10n.service: Main process exited, code=exited, status=1/FAILURE
Aug 26 07:49:53 bookworm systemd[1]: osml10n.service: Failed with result 'exit-code'.
Aug 26 07:49:53 bookworm systemd[1]: Failed to start osml10n.service - OSM l10n transcription server.
Aug 26 07:49:53 bookworm systemd[1]: osml10n.service: Scheduled restart job, restart counter is at 3.
Aug 26 07:49:53 bookworm systemd[1]: Stopped osml10n.service - OSM l10n transcription server.
Aug 26 07:49:53 bookworm systemd[1]: Starting osml10n.service - OSM l10n transcription server...
Aug 26 07:49:53 bookworm geo-transcript-srv.py[13849]: Loading osml10n transcription server:
Aug 26 07:49:53 bookworm systemd[1]: osml10n.service: Main process exited, code=exited, status=1/FAILURE
Aug 26 07:49:53 bookworm systemd[1]: osml10n.service: Failed with result 'exit-code'.
Aug 26 07:49:53 bookworm systemd[1]: Failed to start osml10n.service - OSM l10n transcription server.
Aug 26 07:49:53 bookworm systemd[1]: osml10n.service: Scheduled restart job, restart counter is at 4.
Aug 26 07:49:53 bookworm systemd[1]: Stopped osml10n.service - OSM l10n transcription server.
Aug 26 07:49:53 bookworm systemd[1]: Starting osml10n.service - OSM l10n transcription server...
Aug 26 07:49:54 bookworm geo-transcript-srv.py[13851]: Loading osml10n transcription server:
Aug 26 07:49:54 bookworm systemd[1]: osml10n.service: Main process exited, code=exited, status=1/FAILURE
Aug 26 07:49:54 bookworm systemd[1]: osml10n.service: Failed with result 'exit-code'.
Aug 26 07:49:54 bookworm systemd[1]: Failed to start osml10n.service - OSM l10n transcription server.
Aug 26 07:49:54 bookworm systemd[1]: osml10n.service: Scheduled restart job, restart counter is at 5.
Aug 26 07:49:54 bookworm systemd[1]: Stopped osml10n.service - OSM l10n transcription server.
Aug 26 07:49:54 bookworm systemd[1]: osml10n.service: Start request repeated too quickly.
Aug 26 07:49:54 bookworm systemd[1]: osml10n.service: Failed with result 'exit-code'.
Aug 26 07:49:54 bookworm systemd[1]: Failed to start osml10n.service - OSM l10n transcription server.
Aug 26 08:04:21 bookworm systemd[1]: [🡕] /lib/systemd/system/osml10n.service:10: Special user nobody configured, this is not safe!

Are you using this repo under Debian 12 Bookworm? Am I doing something wrong? Which procedure do you recommend?
Could it be that I have a similar problem as in #25

installing on ubuntu

Hi giggls
Would you please tell me how to implement osml10n at ubuntu 20.04?
things such as installation path, any tiny tips, and tricks.
I cant install it correctly.
regards

add osml10n as extention to postgreSQL

hi,
is there any way to add osml10n as an extension to PostgreSQL?
sorry if my question is irrelevant.

Error when running make test

I get the following error when running make test on ubuntu 22.04

cd lua_osml10/tests/ && ./runtests.lua
/usr/bin/lua: ./osml10n/geo_transcript.lua:10: Can not connect to server 127.0.0.1:8033. Is geo-transcript-srv.py running?
stack traceback:
[C]: in function 'error'
./osml10n/geo_transcript.lua:10: in main chunk
[C]: in function 'require'
./osml10n/init.lua:4: in local 'import'
./osml10n/init.lua:13: in main chunk
[C]: in function 'require'
./osml10n.lua:1: in main chunk
[C]: in function 'require'
./runtests.lua:6: in main chunk
[C]: in ?
make: *** [Makefile:31: test] Error 1