Comments (9)
horw
commented on June 17, 2024
1
Thanks, the error stack info is a bit obscure, does your change fix the error
Yes, there is nothing special in this stack trace.
Actually, the steps to reproduce it are quite easy.
For debugging, just play with this normalization layer.
You can try the changes in this PR; at least for me, it ran successfully.
from mlflow.
horw
commented on June 17, 2024
1
Yes, this works in my environment. Have you tried it?
from mlflow.
horw
commented on June 17, 2024
I suggest there is a problem with the Normalization layer. According to the documentation, the mean value and variance should be precalculated and not passed as variables:
>>> input_data = np.array([[1.], [2.], [3.]], dtype='float32')
>>> layer = keras.layers.Normalization(mean=3., variance=2.)
>>> layer(input_data)
array([[-1.4142135 ],
[-0.70710677],
[ 0. ]], dtype=float32)
So maybe solutions should be like this?
mean = np.mean(train_x, axis=0)
var = np.var(train_x, axis=0)
model = keras.Sequential(
[
keras.Input([train_x.shape[1]]),
keras.layers.Normalization(mean=mean, variance=var),
keras.layers.Dense(64, activation="relu"),
keras.layers.Dense(1),
]
)
from mlflow.
WeichenXu123
commented on June 17, 2024
could you paste the full stack trace ? :)
from mlflow.
horw
commented on June 17, 2024
could you paste the full stack trace ? :)
Yes, sure
2024-04-29 15:28:47.104174: I external/local_tsl/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2024-04-29 15:28:47.162832: I external/local_tsl/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2024-04-29 15:28:47.391369: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-04-29 15:28:48.260030: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
0%| | 0/8 [00:00<?, ?trial/s, best loss=?]
2024-04-29 15:28:51.166355: W tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: INVALID_ARGUMENT: Input to reshape is a tensor with 1 values, but the requested shape has 11
job exception: {{function_node __wrapped__Reshape_device_/job:localhost/replica:0/task:0/device:CPU:0}} Input to reshape is a tensor with 1 values, but the requested shape has 11 [Op:Reshape]
Traceback (most recent call last):
File "/home/horw/PycharmProjects/tt/model-train.py", line 94, in <module>
best = fmin(
File "/home/horw/PycharmProjects/tt/venv3/lib/python3.10/site-packages/hyperopt/fmin.py", line 540, in fmin
return trials.fmin(
File "/home/horw/PycharmProjects/tt/venv3/lib/python3.10/site-packages/hyperopt/base.py", line 671, in fmin
return fmin(
File "/home/horw/PycharmProjects/tt/venv3/lib/python3.10/site-packages/hyperopt/fmin.py", line 586, in fmin
rval.exhaust()
File "/home/horw/PycharmProjects/tt/venv3/lib/python3.10/site-packages/hyperopt/fmin.py", line 364, in exhaust
self.run(self.max_evals - n_done, block_until_done=self.asynchronous)
File "/home/horw/PycharmProjects/tt/venv3/lib/python3.10/site-packages/hyperopt/fmin.py", line 300, in run
self.serial_evaluate()
File "/home/horw/PycharmProjects/tt/venv3/lib/python3.10/site-packages/hyperopt/fmin.py", line 178, in serial_evaluate
result = self.domain.evaluate(spec, ctrl)
File "/home/horw/PycharmProjects/tt/venv3/lib/python3.10/site-packages/hyperopt/base.py", line 892, in evaluate
rval = self.fn(pyll_rval)
File "/home/horw/PycharmProjects/tt/model-train.py", line 72, in objective
result = train_model(
File "/home/horw/PycharmProjects/tt/model-train.py", line 31, in train_model
model = keras.Sequential(
File "/home/horw/PycharmProjects/tt/venv3/lib/python3.10/site-packages/keras/src/models/sequential.py", line 74, in __init__
self._maybe_rebuild()
File "/home/horw/PycharmProjects/tt/venv3/lib/python3.10/site-packages/keras/src/models/sequential.py", line 139, in _maybe_rebuild
self.build(input_shape)
File "/home/horw/PycharmProjects/tt/venv3/lib/python3.10/site-packages/keras/src/layers/layer.py", line 223, in build_wrapper
original_build_method(*args, **kwargs)
File "/home/horw/PycharmProjects/tt/venv3/lib/python3.10/site-packages/keras/src/models/sequential.py", line 183, in build
x = layer(x)
File "/home/horw/PycharmProjects/tt/venv3/lib/python3.10/site-packages/keras/src/utils/traceback_utils.py", line 122, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/home/horw/PycharmProjects/tt/venv3/lib/python3.10/site-packages/tensorflow/python/eager/execute.py", line 53, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __wrapped__Reshape_device_/job:localhost/replica:0/task:0/device:CPU:0}} Input to reshape is a tensor with 1 values, but the requested shape has 11 [Op:Reshape]
from mlflow.
WeichenXu123
commented on June 17, 2024
Thanks, the error stack info is a bit obscure, does your change fix the error ? #11848
from mlflow.
WeichenXu123
commented on June 17, 2024
thanks!
does this work in your test ?
def train_model(params, epochs, train_x, train_y, valid_x, valid_y, test_x, test_y):
# Define model architecture
mean = np.mean(train_x)
var = np.var(train_x)
model = keras.Sequential(
[
keras.Input([train_x.shape[1]]),
keras.layers.Normalization(mean=mean, variance=var),
keras.layers.Dense(64, activation="relu"),
keras.layers.Dense(1),
]
)
...
from mlflow.
horw
commented on June 17, 2024
Sorry for misleading you, but this is incorrect:
mean = np.mean(train_x)
var = np.var(train_x)
This won't work because it will only give one number as a result, whereas each feature should have its own mean and var. The difference between np.mean and np.mean(axis=0) is in the dimension.
For:
mean = np.mean(train_x)
var = np.var(train_x)
The result is:
- Mean: 18.47878942776987
- Variance: 1729.9163381735489
For:
mean = np.mean(train_x, axis=0)
var = np.var(train_x, axis=0)
The result is:
- Mean: [6.86621852e+00, 2.80377808e-01, 3.32597005e-01, 6.42164738e+00, 4.55513955e-02, 3.53556841e+01, 1.38792376e+02, 9.94074221e-01, 3.18919333e+00, 4.88396869e-01, 1.05005673e+01]
- Variance: [7.27006361e-01, 1.04782780e-02, 1.41982590e-02, 2.63994276e+01, 4.46194261e-04, 2.83726450e+02, 1.74920842e+03, 9.23159572e-06, 2.25891723e-02, 1.24958983e-02, 1.49906782e+00]
from mlflow.
horw
commented on June 17, 2024
It also works with this:
model = keras.Sequential(
[
keras.Input([train_x.shape[1]]),
keras.layers.Normalization(mean=np.mean(train_x, axis=0), variance=np.var(train_x, axis=0)),
keras.layers.Dense(64, activation="relu"),
keras.layers.Dense(1),
]
)
from mlflow.
Related Issues (20)
- [FR] ability to drop columns that get used for split, but not for training HOT 1
- Hello, I encountered an issue while attempting to run the official MLflow image. Can anyone assist me with this? I'm a beginner. Thanks HOT 6
- Add load_delta support with `deltalake` package HOT 2
- [BUG] Exception During Model Logging with Custom Ollama Class HOT 6
- comparing metrics with timestamp is not working if step is same between them [BUG] HOT 4
- [BUG] Can't delete runs with mlflow gc due to api timeout HOT 11
- [FR] Display the complete dataset profile in the data details view HOT 1
- Improve `_init_server` HOT 2
- [FR] Improve UI stability to corrupt metric files HOT 2
- [BUG] UI Crash - Unterminated string in JSON at position 5000 for mlflow.log-model.history HOT 4
- [BUG] pyfunc.load_model ignores logged model with trust_remote_code set to True HOT 1
- [BUG] HOT 2
- [BUG]Prompt Engineering request from UI to Deployments Server Connection TimeOut HOT 5
- [FR]MLflow Deployments Server Support inside corporate proxy HOT 3
- Fix typos
- Fix docstrings in `mlflow/tracing` HOT 1
- [FR] Multiple retrievers with mlflow.langchain.log_model HOT 1
- [BUG] MLFlow Deployment Server for LLMs using chatCompletion on Azure OpenAI text-davinci-003 HOT 4
- [SETUP-BUG] Multi-Cloud artifact-destination migration HOT 3
- mlflow.pyfunc.load_model is loading model of class <class 'mlflow.pyfunc.PyFuncModel'> instead of original class HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mlflow.