santhoshtr / sfst Goto Github PK

Stuttgart Finite State Transducer system

License: GNU General Public License v2.0

F* 26.67% Makefile 0.19% Perl 0.66% CMake 0.92% Python 0.85% C++ 59.08% C 0.10% LLVM 3.00% Roff 6.46% Yacc 2.06%

pybind11 readline bison-yacc fst computational-morphologies cmake

sfst's Introduction

SFST - Stuttgart Finite State Transducer

CI	status
CMake builds
Pip builds
Wheels builds

Installation

SFST can be compiled in Unix/Linux, Windows and Mac operating systems.

The SFST command line utilities has a few external requirements.

The "Flex" scanner generator which can be downloaded from: https://github.com/westes/flex. In linux systems they can be installed using package manager. For example, in Ubuntu, apt install flex installs it.
The "Bison" parser generator available from http://www.gnu.org/software/bison. In linux systems they can be installed using package manager. For example, in Ubuntu, apt install bison installs it.

After unpacking the software package, in the top directory of source code,

mkdir build
cd build
cmake ..
make

to compile the tools. Then call

make install

to install the tools in /usr/local/bin. Change the variable DESTDIR in Makefile if you want to install to a different directory. Finally call

make maninstall

to install the manpages in /usr/local/man/man1 and you are done.

The subdirectory data contains a simple example of an English morphological analyser, the source code of the German SMOR morphology (with just a few sample lexicon entries), and the general morphology XMOR which may be used as a starting point for the development of a computational morphology.

Usage

See the manual SFST-Manual.pdf in the doc subdirectory and the man pages for more information on the SFST tools. doc/SFST-Tutorial.pdf explains how computational morphologies are implemented in SFST.

If you want to implement your own tools based on the SFST code, have a look at fst-infl.C and fst-infl2.C. They show how analysers are implemented with the standard (fst-infl) and the compact (fst-infl2) transducer format.

Author

The SFST tools have been implemented by Helmut Schmid, Institute for Computational Linguistics, University of Stuttgart, Germany and they are available under the GNU public license version 2 or higher.

Please cite the following publication if you want to refer to the SFST tools:

A Programming Language for Finite State Transducers, Proceedings of the 5th International Workshop on Finite State Methods in Natural Language Processing (FSMNLP 2005), Helsinki, Finland. pdf

Bug reports

Please send bug reports and other feedback to [email protected].

sfst's People

Contributors

Stargazers

Watchers

Forkers

sebastinf gremid ambientlighter tlmnb nolda asdofindia kaungmt bnationsdev lucalila

sfst's Issues

Provide wheel for Python 11

Phrasal verbs not recognised

The sample lexicon in data/SMOR/lexicon contains several entries for phrasal verbs, e.g.

<Base_Stems>auswendig<PREF>:<><ge>lern<V><base><nativ><VVReg>

Such verbs, however, are not recognised:

> fst-mor smor.a
reading transducer...
finished.
analyze> auswendiglernen
no result for auswendiglernen
analyze> q

The problem appears to be the symbol <ge> in the lexical entry intervening between the particle and the base verb. It is taken into account by $GE in data/SMOR/deko.fst (cf. the comments there), but not in $BDKStems$ in smor.fst.

Version reported by binaries are not same as VERSION file content

SFSTVersion in alphabet.h is hardcoded and it is not picking up the version from VERSION file. We will need this to be passed at compile time

python binding: Support compact transducer format

Explore a way to provide prefix-match option for fst-generate

For morphologically complex languages like Malayalam, it would be nice to have a prefix search to be used with autocompletion like usecases.

install locally rather than in /usr/local/bin (example for DESTDIR needed)

I am trying to build sfst in the local workspace of the current user, Ubuntu 22.04 LTS.[1] For this, I would ask you for an example of setting DESTDIR in accordance with the line "Change the variable DESTDIR in Makefile if you want to install to a different directory." from the Readme.

My naive interpretation was that this means to run the following sequence of commands:

$> git clone https://github.com/santhoshtr/sfst.git;
$> cd sfst;
$> mkdir build;
$> cd build;
$> cmake ..;
$> make DESTDIR=~/tmp;
$> make install DESTDIR=~/tmp;
$> cd ../..;

Which, however, yields

Install the project...
-- Install configuration: ""
...
-- Installing: /home/christian/tmp/usr/local/include/utf8.h
CMake Error at src/cmake_install.cmake:65 (file):
  file INSTALL cannot find
  "/home/christian/Desktop/github/nmk-corpus/scripts/sfst/build/src/fst-compiler.h":
  No such file or directory.
Call Stack (most recent call first):
  cmake_install.cmake:47 (include)

make[1]: *** [Makefile:120: install] Error 1

I guess this is has a rather obvious solution, apologies for the stupid question.

[1] I need to install somewhere in the user space, because without setting DESTDIR, I get the expected permission denied error, and on the machine I want to run it on, I don't have sudo rights.

...
Install the project...
-- Install configuration: ""
-- Installing: /usr/local/lib/libsfst.so
CMake Error at src/cmake_install.cmake:52 (file):
  file INSTALL cannot copy file
  "/home/christian/Desktop/github/nmk-corpus/scripts/sfst/build/src/libsfst.so"
  to "/usr/local/lib/libsfst.so": Permission denied.
Call Stack (most recent call first):
  cmake_install.cmake:47 (include)

PS: I tried different kinds of DESTDIR paths, both absolute and relative. I also tried using the DESTDIR flag on make install only rather than on both that and the earlier call to make. No difference.

Installation fails due to 'fst-compiler.h'

Hi, I get this message when I'm installing sfst.

CMake Error at src/cmake_install.cmake:365 (file):
  file INSTALL cannot find
  "<path_to_sfst>/src/fst-compiler.h":
  No such file or directory.
Call Stack (most recent call first):
  cmake_install.cmake:47 (include)

Could I get some help?