Comments (8)
mikahama
commented on August 15, 2024
2
The normalization model is a torch model and the generation model is a tensorflow model, so you can use them directly too without opennmt in the middle. The models are LSTM based so they won't work on CTranslate2 or huggingface.
from murre.
mikahama
commented on August 15, 2024
Normalization should work because it uses pytorch, not tensorflow. Tensorflow has issues on M1 Macs. You could try running your terminal through Rosetta and installing an amd64 version of Python.
from murre.
SimbaNinja917
commented on August 15, 2024
Thank you for your answer.
I tried again with Rosetta and python3 dialectalize.py gives:
(murre-master) (base) macbook@macbooks-MacBook-Pro murre-master % python3 dialectalize.py
Metal device set to: Apple M1
systemMemory: 16.00 GB
maxCacheSize: 5.33 GB
Traceback (most recent call last):
File "/Users/macbook/.local/share/virtualenvs/murre-master-mnROP8xX/lib/python3.10/site-packages/tensorflow/python/framework/ops.py", line 4199, in _get_op_def
return self._op_def_cache[type]
KeyError: 'Addons>GatherTree'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/macbook/.local/share/virtualenvs/murre-master-mnROP8xX/lib/python3.10/site-packages/tensorflow/python/saved_model/load.py", line 958, in load_partial
loader = Loader(object_graph_proto, saved_model_proto, export_dir,
File "/Users/macbook/.local/share/virtualenvs/murre-master-mnROP8xX/lib/python3.10/site-packages/tensorflow/python/saved_model/load.py", line 154, in __init__
function_deserialization.load_function_def_library(
File "/Users/macbook/.local/share/virtualenvs/murre-master-mnROP8xX/lib/python3.10/site-packages/tensorflow/python/saved_model/function_deserialization.py", line 416, in load_function_def_library
func_graph = function_def_lib.function_def_to_graph(
File "/Users/macbook/.local/share/virtualenvs/murre-master-mnROP8xX/lib/python3.10/site-packages/tensorflow/python/framework/function_def_to_graph.py", line 82, in function_def_to_graph
graph_def, nested_to_flat_tensor_name = function_def_to_graph_def(
File "/Users/macbook/.local/share/virtualenvs/murre-master-mnROP8xX/lib/python3.10/site-packages/tensorflow/python/framework/function_def_to_graph.py", line 252, in function_def_to_graph_def
op_def = default_graph._get_op_def(node_def.op) # pylint: disable=protected-access
File "/Users/macbook/.local/share/virtualenvs/murre-master-mnROP8xX/lib/python3.10/site-packages/tensorflow/python/framework/ops.py", line 4203, in _get_op_def
pywrap_tf_session.TF_GraphGetOpDef(c_graph, compat.as_bytes(type),
tensorflow.python.framework.errors_impl.NotFoundError: Op type not registered 'Addons>GatherTree' in binary running on macbooks-MacBook-Pro.local. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/macbook/Downloads/coding/murre-hello-world/murre-master/dialectalize.py", line 2, in <module>
dialectalize_sentence("kodin takana on koira", "Pohjois-Savo")
File "/Users/macbook/Downloads/coding/murre-hello-world/murre-master/murre/__init__.py", line 6, in dialectalize_sentence
return dialectalize_sentences([sentence], dialect)[0]
File "/Users/macbook/Downloads/coding/murre-hello-world/murre-master/murre/generator.py", line 44, in generate
r_parts = _translate(parts)
File "/Users/macbook/Downloads/coding/murre-hello-world/murre-master/murre/generator.py", line 68, in _translate
_load_model()
File "/Users/macbook/Downloads/coding/murre-hello-world/murre-master/murre/generator.py", line 62, in _load_model
model = tf.saved_model.load(script_path("models/generate/flags_dist"))
File "/Users/macbook/.local/share/virtualenvs/murre-master-mnROP8xX/lib/python3.10/site-packages/tensorflow/python/saved_model/load.py", line 828, in load
result = load_partial(export_dir, None, tags, options)["root"]
File "/Users/macbook/.local/share/virtualenvs/murre-master-mnROP8xX/lib/python3.10/site-packages/tensorflow/python/saved_model/load.py", line 961, in load_partial
raise FileNotFoundError(
FileNotFoundError: Op type not registered 'Addons>GatherTree' in binary running on macbooks-MacBook-Pro.local. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.
You may be trying to load on a different device from the computational device. Consider setting the `experimental_io_device` option in `tf.saved_model.LoadOptions` to the io_device such as '/job:localhost'.
and python3 normalize.py gives:
(murre-master) (base) macbook@macbooks-MacBook-Pro murre-master % python3 normalize.py
Metal device set to: Apple M1
systemMemory: 16.00 GB
maxCacheSize: 5.33 GB
Traceback (most recent call last):
File "/Users/macbook/Downloads/coding/murre-hello-world/murre-master/normalize.py", line 2, in <module>
normalize_sentence("mä syön paljo karkkii")
File "/Users/macbook/Downloads/coding/murre-hello-world/murre-master/murre/normalizer.py", line 64, in normalize_sentence
res = _normalize_chunks(chunks, wnut19_model=wnut19_model, model_path=model_path, n_best=n_best, language=language)
File "/Users/macbook/Downloads/coding/murre-hello-world/murre-master/murre/normalizer.py", line 85, in _normalize_chunks
res = call_onmt(chunks, model_name, n_best=n_best)
File "/Users/macbook/.local/share/virtualenvs/murre-master-mnROP8xX/lib/python3.10/site-packages/natas/normalize.py", line 164, in call_onmt
fields, model, model_opt = _give_model(model_name)
File "/Users/macbook/.local/share/virtualenvs/murre-master-mnROP8xX/lib/python3.10/site-packages/natas/normalize.py", line 154, in _give_model
_load_model(name)
File "/Users/macbook/.local/share/virtualenvs/murre-master-mnROP8xX/lib/python3.10/site-packages/natas/normalize.py", line 149, in _load_model
m = load_test_model(opt)
File "/Users/macbook/.local/share/virtualenvs/murre-master-mnROP8xX/lib/python3.10/site-packages/onmt/model_builder.py", line 90, in load_test_model
checkpoint = torch.load(model_path,
File "/Users/macbook/.local/share/virtualenvs/murre-master-mnROP8xX/lib/python3.10/site-packages/torch/serialization.py", line 795, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "/Users/macbook/.local/share/virtualenvs/murre-master-mnROP8xX/lib/python3.10/site-packages/torch/serialization.py", line 1012, in _legacy_load
result = unpickler.load()
File "/Users/macbook/.local/share/virtualenvs/murre-master-mnROP8xX/lib/python3.10/site-packages/torch/serialization.py", line 828, in find_class
return super().find_class(mod_name, name)
ModuleNotFoundError: No module named 'onmt.inputters.text_dataset'
from murre.
mikahama
commented on August 15, 2024
For the normalization model, try installing OpenNMT-py==2.1.2. Did you run "python3 -m murre.download" by any chance? Also, forcing OpenNMT-tf==2.18.1 might help for dialect generation.
from murre.
SimbaNinja917
commented on August 15, 2024
Yes python3 -m murre.download downloads 4 normalizer and 4 generators.
pip3 install OpenNMT-py==2.1.2 gives
Collecting OpenNMT-py==2.1.2
Using cached OpenNMT_py-2.1.2-py3-none-any.whl (212 kB)
Requirement already satisfied: configargparse<2,>=1.2.3 in /Users/macbook/miniconda3/lib/python3.10/site-packages (from OpenNMT-py==2.1.2) (1.5.3)
Collecting pyyaml==5.4
Using cached PyYAML-5.4.tar.gz (174 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: pyonmttok<2,>=1.23 in /Users/macbook/miniconda3/lib/python3.10/site-packages (from OpenNMT-py==2.1.2) (1.35.0)
Requirement already satisfied: tensorboard<3,>=2.3 in /Users/macbook/miniconda3/lib/python3.10/site-packages (from OpenNMT-py==2.1.2) (2.11.0)
Requirement already satisfied: tqdm<5,>=4.51 in /Users/macbook/miniconda3/lib/python3.10/site-packages (from OpenNMT-py==2.1.2) (4.64.1)
Collecting waitress==1.4.4
Using cached waitress-1.4.4-py2.py3-none-any.whl (58 kB)
ERROR: Could not find a version that satisfies the requirement torch==1.6.0 (from opennmt-py) (from versions: 1.11.0, 1.12.0, 1.12.1, 1.13.0, 1.13.1)
ERROR: No matching distribution found for torch==1.6.0
pip3 install OpenNMT-tf==2.18.1 gives:
(base) macbook@macbooks-MacBook-Pro murre-master % pip3 install OpenNMT-tf==2.18.1
Collecting OpenNMT-tf==2.18.1
Using cached OpenNMT_tf-2.18.1-py3-none-any.whl (154 kB)
Requirement already satisfied: rouge<2,>=1.0 in /Users/macbook/miniconda3/lib/python3.10/site-packages (from OpenNMT-tf==2.18.1) (1.0.1)
ERROR: Could not find a version that satisfies the requirement ctranslate2<2,>=1.18.1; platform_system == "Linux" or platform_system == "Darwin" (from opennmt-tf) (from versions: 2.16.0, 2.17.0, 2.18.0, 2.19.0, 2.19.1, 2.20.0, 2.21.0, 2.21.1, 2.22.0, 2.23.0, 2.24.0, 3.0.0, 3.0.1, 3.0.2, 3.1.0, 3.2.0, 3.3.0)
ERROR: No matching distribution found for ctranslate2<2,>=1.18.1; platform_system == "Linux" or platform_system == "Darwin"
If this is too complicated, should I just try on another machine? or maybe on an online platform ?
from murre.
SimbaNinja917
commented on August 15, 2024
I tried to start everything from scratch deleted and reset all the python ecosystem I have on my M1.
0- cd ~/downloads/coding && mkdir murre-hello-world && cd murre-hello-world
0a- I run arch I get: i386
1- virtualenv murre
2- source murre/bin/activate
3- python3 --version returns Python 3.10.9
4- pip3 --version return pip 22.3.1 from /Users/macbook/Downloads/coding/murre-hello-world/murre/lib/python3.10/site-packages/pip (python 3.10)
5- which python3 returns /Users/macbook/Downloads/coding/murre-hello-world/murre/bin/python3
6- which pip3 return /Users/macbook/Downloads/coding/murre-hello-world/murre/bin/pip3
7- I run: pip3 install murre==1.4
(murre) macbook@macbooks-MacBook-Pro murre-hello-world % pip3 install murre==1.4
Collecting murre==1.4
Using cached murre-1.4.0-py2.py3-none-any.whl (11 kB)
ERROR: Could not find a version that satisfies the requirement tensorflow==2.4.0 (from murre) (from versions: 2.8.0rc0, 2.8.0rc1, 2.8.0, 2.8.1, 2.8.2, 2.8.3, 2.8.4, 2.9.0rc0, 2.9.0rc1, 2.9.0rc2, 2.9.0, 2.9.1, 2.9.2, 2.9.3, 2.10.0rc0, 2.10.0rc1, 2.10.0rc2, 2.10.0rc3, 2.10.0, 2.10.1, 2.11.0rc0, 2.11.0rc1, 2.11.0rc2, 2.11.0)
ERROR: No matching distribution found for tensorflow==2.4.0
8- I try again: pip3 install murre==1.4
(murre) macbook@macbooks-MacBook-Pro murre-hello-world % pip3 install murre==1.4
Collecting murre==1.4
Using cached murre-1.4.0-py2.py3-none-any.whl (11 kB)
ERROR: Could not find a version that satisfies the requirement tensorflow-addons==0.12.1 (from murre) (from versions: 0.16.1, 0.17.0, 0.17.1, 0.18.0, 0.19.0)
ERROR: No matching distribution found for tensorflow-addons==0.12.1
(murre) macbook@macbooks-MacBook-Pro murre-hello-world %
9- I run: pip3 install tensorflow-macos tensorflow-metal
Goes well.
10- I run: pip3 install mikatools natas tensorflow-addons
Goes well.
11- I run: pip3 install OpenNMT-py==2.1.2 --upgrade --force-reinstall
(murre) macbook@macbooks-MacBook-Pro murre-hello-world % pip3 install OpenNMT-py==2.1.2 --upgrade --force-reinstall
Collecting OpenNMT-py==2.1.2
Using cached OpenNMT_py-2.1.2-py3-none-any.whl (212 kB)
ERROR: Could not find a version that satisfies the requirement torch==1.6.0 (from opennmt-py) (from versions: 1.11.0, 1.12.0, 1.12.1, 1.13.0, 1.13.1)
ERROR: No matching distribution found for torch==1.6.0
12- I run: pip3 install OpenNMT-tf==2.18.1 --upgrade --force-reinstall
(murre) macbook@macbooks-MacBook-Pro murre-hello-world % pip3 install OpenNMT-tf==2.18.1 --upgrade --force-reinstall
Collecting OpenNMT-tf==2.18.1
Using cached OpenNMT_tf-2.18.1-py3-none-any.whl (154 kB)
ERROR: Could not find a version that satisfies the requirement tensorflow-addons<0.13,>=0.12 (from opennmt-tf) (from versions: 0.16.1, 0.17.0, 0.17.1, 0.18.0, 0.19.0)
ERROR: No matching distribution found for tensorflow-addons<0.13,>=0.12
13- I run : pip3 install murre
I get:
Collecting murre
Using cached murre-1.4.0-py2.py3-none-any.whl (11 kB)
Requirement already satisfied: pyonmttok>=1.23.0 in ./murre/lib/python3.10/site-packages (from murre) (1.35.0)
Using cached murre-1.3.0-py2.py3-none-any.whl (11 kB)
Collecting OpenNMT-tf==2.10.0
Using cached OpenNMT_tf-2.10.0-py2.py3-none-any.whl (140 kB)
Requirement already satisfied: natas in ./murre/lib/python3.10/site-packages (from murre) (1.2.0)
ERROR: Cannot install murre==1.3.0 and murre==1.4.0 because these package versions have conflicting dependencies.
The conflict is caused by:
murre 1.4.0 depends on tensorflow-addons==0.12.1
murre 1.3.0 depends on pyonmttok==1.10.1
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
14- I run: python3 -c "import tensorflow as tf; print(tf.__version__)"
I get:
(murre) macbook@macbooks-MacBook-Pro murre-hello-world % python3 -c "import tensorflow as tf; print(tf.__version__)"
2023-01-11 00:12:08.732973: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE4.1 SSE4.2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2.11.0
15- I try:
pip3 install https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-2.4.0-cp36-cp36m-macosx_10_14_x86_64.whl
I get:
(murre) macbook@macbooks-MacBook-Pro murre-hello-world % pip3 install https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-2.4.0-cp36-cp36m-macosx_10_14_x86_64.whl
ERROR: tensorflow-2.4.0-cp36-cp36m-macosx_10_14_x86_64.whl is not a supported wheel on this platform.
from murre.
mikahama
commented on August 15, 2024
The opennmt versions I stated previously are the ones I have tested murre with. Newer versions might work too if they are available on your platform. Your example of installing tensorflow from that wheel only works if you have python 3.6 and you are running an amd64 python. Any other python version or the native arm version will result in the error code you got.
It also looks like you are installing murre 1.3 and 1. 4 simultaneously 😵💫.
from murre.
SimbaNinja917
commented on August 15, 2024
The opennmt versions I stated previously are the ones I have tested murre with. Newer versions might work too if they are available on your platform. Your example of installing tensorflow from that wheel only works if you have python 3.6 and you are running an amd64 python. Any other python version or the native arm version will result in the error code you got.
It also looks like you are installing murre 1.3 and 1. 4 simultaneously 😵💫.
I give up. This is getting 🥴
@mikahama any chance I can deploy it using an alternative way? Maybe on HuggingFace transformers? Pytorch? CTranslate2 ?
from murre.
Related Issues (3)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from murre.