Comments (12)
@tjrileywisc I have a branch for Python 3 support that I had been working on locally. I just pushed it up, but it's not merged in yet because I've only "lightly tested" it on my Mac w/ Spark Standalone... However, if you're already experimenting, feel free to try it out and see if it works for you...
from tensorflowonspark.
@leewyang Thanks. I checked out the branch. I'm still stuck with the same error unfortunately but I'll keep trying.
from tensorflowonspark.
FYI, we've merged in the python3 compatible code. Unfortunately, I doubt that it will fix this issue (which seems to be Windows-specific pickling issue), so I'll update the title accordingly.
from tensorflowonspark.
I'm also trying python 3.5 + tensorflow + windows solution and I also get the pickling issue. I tried using pathos multiprocess to replace multiprocessing, it seems the pickling issue is gone but the workers crashed for unknown reason. If possible could you try using pathos and see if it can work on Linux?
from tensorflowonspark.
@goodwanghan May I ask how you used pathos to resolve the pickling issue? I'm also facing the same issue on windows right now.
from tensorflowonspark.
This is probably clear by now, but it seems that the ability to pickle under windows is affected by the lack of ability to fork() in the multiprocess package (a spawn() occurs instead). It was resolved in another context (ouspg/trytls#197) by moving the call but not sure if that's relevant here.
from tensorflowonspark.
I could not install this on CDH hence took this route of installing on windows.
I stood up a standalone spark cluster using this https://sachingpta.gitlab.io/_posts/installing-spark-cluster-in-windows.html. I could run the first step but train of the model fails. I'm stuck on this pickling error too. Let me know if you find a resolution to this.
from tensorflowonspark.
Is there any way to run this on windows and avoid the pickling error?
from tensorflowonspark.
im not sure
C:\Python35\lib\multiprocessing\reduction.py
import dill as pickle
from tensorflowonspark.
Is this still an issue?
from tensorflowonspark.
Added a note that Windows isn't currently supported due to this issue.
from tensorflowonspark.
I found a solution to this: I removed all classes definition from my main execution file (as instructed in DataLoader source here) and the pickling error no longer appears.
I'm running Windows.
from tensorflowonspark.
Related Issues (20)
- MNIST SPARK on Standalone Cluster inside Docker Container HOT 11
- Writing checkpoints to HDFS takes long HOT 2
- when using mnist_spark.py , serializer.dump_stream Timeout while feeding partition HOT 2
- pkg_resources.DistributionNotFound: The 'tensorflow' distribution was not found and is required by the application HOT 3
- MNIST example - Exception in TF background thread HOT 2
- the doubt about the data policy HOT 1
- Performance issues in the program HOT 2
- Performance issues in examples/mnist/estimator (by P3) HOT 3
- Retaining original columns after inference HOT 2
- tensorflow.python.framework.errors_impl.UnimplementedError: File system scheme 'cosn' not implemented HOT 2
- Model Saved with TF-2.5.0 HOT 3
- How to integrate a model into Spark cluster HOT 12
- Get stuck at "Added broadcast_0_piece0 in memory on" while runing Spark standalone cluster HOT 1
- ExitCode: 13 executing mnist_data_setup.py on a yarn cluster HOT 3
- can it run on tensorflow-cpu? HOT 1
- can it run use ParameterServerStrategy HOT 3
- do we support scala & java code write tensorflow model with tenorflow-core-api ? HOT 3
- Evalator hangs while training HOT 1
- yarn mode error HOT 1
- error while running mnist_tf_ds.py HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tensorflowonspark.