deniederhut / niacin Goto Github PK

View Code? Open in Web Editor NEW

16.0 16.0 2.0 2.41 MB

Enrich your data

Home Page: https://niacin.readthedocs.io

License: BSD 3-Clause "New" or "Revised" License

Python 93.01% TeX 6.99%

niacin's People

Contributors

Stargazers

Watchers

Forkers

hannahbrucemacdonald fourpenny

niacin's Issues

Hypernym and Hyponym Source

Hi!
Thank you for providing this library. It is very useful and easy to use.

May I know where you got the source of hypernym and hyponym? is it originally from wordnet?

Cheers,
Guntur

[Major Issue][Wrong Doc String]

It looks like the docstring for add_whitespace function was wrongly copied from the remove_whitespace function.
🙈

[Minor Doc Improvement]

It may not be intuitive from the get-go that the user has to perform from niacin import text, else the examples in the QuickStart Guide may fail with the error

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'text' is not defined

Just adding a line from niacin import text before examples would be useful. WDYT?

[Future Plan][Other Languages]

Hey, @deniederhut just curious to know if you plan on adding other languages as well?
If so how are you planning on obtaining the modification data for the same?
Asking because it will incur changes in the way we are currently storing/loading the data used modification.
Also, do we have any reference literature or experiments done in languages other than English to show whether these transformations are as effective as they seem to be for English?

P.S: This is not w.r.t to the review, asking just out of curiosity.

[Missing Doc Strings]

In case there are left out on purpose, the following 2 functions are currently missing doc-strings:

Error Using Back-translate Functionality

TL;DR

When trying to run backtranslation on some text data, I encountered two errors, one was a urllib.error.HTTPError and the other was with installing dependencies for backtranslation.

urllib.error.HTTPError

I was trying to run back-translation on some text data, and encountered the following error:

...
Traceback (most recent call last):
  File "C:\Code\ENVIRONMENTS\ace-env\ACE\.tmp\test2.py", line 35, in <module>
    augmented_text = tx(row[0])
  File "C:\Code\ENVIRONMENTS\ace-env\lib\site-packages\niacin\text\en\sentence.py", line 180, in add_backtranslation
    t = _Translator()
  File "C:\Code\ENVIRONMENTS\ace-env\lib\site-packages\niacin\text\en\sentence.py", line 46, in __init__
    self.load_models()
  File "C:\Code\ENVIRONMENTS\ace-env\lib\site-packages\niacin\text\en\sentence.py", line 64, in load_models
    cls.translators["en2de"] = torch.hub.load(
  File "C:\Code\ENVIRONMENTS\ace-env\lib\site-packages\torch\hub.py", line 362, in load
    repo_or_dir = _get_cache_or_reload(repo_or_dir, force_reload, verbose)
  File "C:\Code\ENVIRONMENTS\ace-env\lib\site-packages\torch\hub.py", line 162, in _get_cache_or_reload
    _validate_not_a_forked_repo(repo_owner, repo_name, branch)
  File "C:\Code\ENVIRONMENTS\ace-env\lib\site-packages\torch\hub.py", line 124, in _validate_not_a_forked_repo
    with urlopen(url) as r:
  File "c:\users\illys\appdata\local\programs\python\python39\lib\urllib\request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "c:\users\illys\appdata\local\programs\python\python39\lib\urllib\request.py", line 523, in open
    response = meth(req, response)
  File "c:\users\illys\appdata\local\programs\python\python39\lib\urllib\request.py", line 632, in http_response
    response = self.parent.error(
  File "c:\users\illys\appdata\local\programs\python\python39\lib\urllib\request.py", line 561, in error
    return self._call_chain(*args)
  File "c:\users\illys\appdata\local\programs\python\python39\lib\urllib\request.py", line 494, in _call_chain
    result = func(*args)
  File "c:\users\illys\appdata\local\programs\python\python39\lib\urllib\request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: rate limit exceeded

Which led me to see if the error was due to dependencies not being installed.

Installing Dependencies for Backtranslation.

I ran the command pip install niacin[backtranslate] in Windows 10 Powershell, to see if it would fix the urllib.error.HTTPError. However, I received the following error:

...
Building wheel for fastbpe (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: 'c:\code\environments\ace-env\scripts\python.exe' -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\IllyS\\AppData\\Local\\Temp\\pip-install-ksmbdnd6\\fastbpe_27df61036fe14cf4a894595f913b7246\\setup.py'"'"'; __file__='"'"'C:\\Users\\IllyS\\AppData\\Local\\Temp\\pip-install-ksmbdnd6\\fastbpe_27df61036fe14cf4a894595f913b7246\\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\IllyS\AppData\Local\Temp\pip-wheel-4khl2d2a'
       cwd: C:\Users\IllyS\AppData\Local\Temp\pip-install-ksmbdnd6\fastbpe_27df61036fe14cf4a894595f913b7246\
  Complete output (19 lines):
  running bdist_wheel
  running build
  running build_py
  package init file 'fastBPE\__init__.py' not found (or not a regular file)
  running build_ext
  building 'fastBPE' extension
  creating build
  creating build\temp.win-amd64-3.9
  creating build\temp.win-amd64-3.9\Release
  creating build\temp.win-amd64-3.9\Release\fastBPE
  C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IfastBPE -Ic:\code\environments\ace-env\include -Ic:\users\illys\appdata\local\programs\python\python39\include -Ic:\users\illys\appdata\local\programs\python\python39\include -IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\ATLMFC\include -IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\include -IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt /EHsc /TpfastBPE/fastBPE.cpp /Fobuild\temp.win-amd64-3.9\Release\fastBPE/fastBPE.obj -std=c++11 -Ofast -pthread
  cl : Command line warning D9025 : overriding '/Os' with '/Ot'
  cl : Command line warning D9002 : ignoring unknown option '-std=c++11'
  cl : Command line warning D9002 : ignoring unknown option '-Of'
  cl : Command line warning D9002 : ignoring unknown option '-Oa'
  cl : Command line warning D9002 : ignoring unknown option '-pthread'
  fastBPE.cpp
  C:\Users\IllyS\AppData\Local\Temp\pip-install-ksmbdnd6\fastbpe_27df61036fe14cf4a894595f913b7246\fastBPE\fastBPE.hpp(15): fatal error C1083: Cannot open include file: 'sys/mman.h': No such file or directory
  error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.26.28801\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2
  ----------------------------------------
  ERROR: Failed building wheel for fastbpe
  Running setup.py clean for fastbpe
Failed to build fastbpe
Installing collected packages: fastbpe
    Running setup.py install for fastbpe ... error
    ERROR: Command errored out with exit status 1:
     command: 'c:\code\environments\ace-env\scripts\python.exe' -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\IllyS\\AppData\\Local\\Temp\\pip-install-ksmbdnd6\\fastbpe_27df61036fe14cf4a894595f913b7246\\setup.py'"'"'; __file__='"'"'C:\\Users\\IllyS\\AppData\\Local\\Temp\\pip-install-ksmbdnd6\\fastbpe_27df61036fe14cf4a894595f913b7246\\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\IllyS\AppData\Local\Temp\pip-record-7y660u4y\install-record.txt' --single-version-externally-managed --compile --install-headers 'c:\code\environments\ace-env\include\site\python3.9\fastbpe'
         cwd: C:\Users\IllyS\AppData\Local\Temp\pip-install-ksmbdnd6\fastbpe_27df61036fe14cf4a894595f913b7246\
    Complete output (19 lines):
    running install
    running build
    running build_py
    package init file 'fastBPE\__init__.py' not found (or not a regular file)
    running build_ext
    building 'fastBPE' extension
    creating build
    creating build\temp.win-amd64-3.9
    creating build\temp.win-amd64-3.9\Release
    creating build\temp.win-amd64-3.9\Release\fastBPE
    C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IfastBPE -Ic:\code\environments\ace-env\include -Ic:\users\illys\appdata\local\programs\python\python39\include -Ic:\users\illys\appdata\local\programs\python\python39\include -IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\ATLMFC\include -IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\include -IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt /EHsc /TpfastBPE/fastBPE.cpp /Fobuild\temp.win-amd64-3.9\Release\fastBPE/fastBPE.obj -std=c++11 -Ofast -pthread
    cl : Command line warning D9025 : overriding '/Os' with '/Ot'
    cl : Command line warning D9002 : ignoring unknown option '-std=c++11'
    cl : Command line warning D9002 : ignoring unknown option '-Of'
    cl : Command line warning D9002 : ignoring unknown option '-Oa'
    cl : Command line warning D9002 : ignoring unknown option '-pthread'
    fastBPE.cpp
    C:\Users\IllyS\AppData\Local\Temp\pip-install-ksmbdnd6\fastbpe_27df61036fe14cf4a894595f913b7246\fastBPE\fastBPE.hpp(15): fatal error C1083: Cannot open include file: 'sys/mman.h': No such file or directory
    error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.26.28801\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2
    ----------------------------------------
ERROR: Command errored out with exit status 1: 'c:\code\environments\ace-env\scripts\python.exe' -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\IllyS\\AppData\\Local\\Temp\\pip-install-ksmbdnd6\\fastbpe_27df61036fe14cf4a894595f913b7246\\setup.py'"'"'; __file__='"'"'C:\\Users\\IllyS\\AppData\\Local\\Temp\\pip-install-ksmbdnd6\\fastbpe_27df61036fe14cf4a894595f913b7246\\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\IllyS\AppData\Local\Temp\pip-record-7y660u4y\install-record.txt' --single-version-externally-managed --compile --install-headers 'c:\code\environments\ace-env\include\site\python3.9\fastbpe' Check the logs for full command output.

Please could you advise if there is anything I can do to remedy these errors?

What are the steps to setup and run tests?

Hey, @deniederhut It will be great if the install from source section under README.md can also have a subsection of setting up the test environment and running the test folder.
It will help validate the installation and also newcomers who would add tests for new features.

add_synonym systematically convert "was" into "wa"

As per the title, when using add_synonyms, and whatever probability is used, any occurence of "was" in the input is converted to "wa".

To reproduce:

!pip install niacin

from niacin.text import en
en.word.add_synonyms("he was fine", p=0.00001)

--> Output: 'he wa fine'

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.