Comments (7)
mfeurer
commented on May 13, 2024
How large is the temporary output directory which contains all the models? Maybe auto-sklearn is still trying to load all models into main memory. This would be the case if the model directory is larger than the amount of main memory used by auto-sklearn.
We have a fix for this in the development branch and I hope that we'll merge it soon.
from auto-sklearn.
Motorrat
commented on May 13, 2024
the temp directory size is 36GB. I have also edited my post above - Once I have issued (CTRL+C) in the python command line for the script it has completed and shown better results as the last time with less execution time. So it worked. It also shows ensemble building completion at 22:32. So may be it is even not autosklearn itself?
36GB is shown as the amount of memory taken by the python process and 42GB as the virtual memory there. 42 is the memory_limit I have set for the AutoSklearnClassifier
from auto-sklearn.
mfeurer
commented on May 13, 2024
When you hit CTRL+C, does it show you a stack trace?
from auto-sklearn.
Motorrat
commented on May 13, 2024
^CError in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/home/ekobylkin/anaconda2/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
Process PoolWorker-9:
Process PoolWorker-7:
Process PoolWorker-8:
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
File "/home/ekobylkin/anaconda2/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
File "/home/ekobylkin/anaconda2/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
File "/home/ekobylkin/anaconda2/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
func(*targs, **kargs)
File "/home/ekobylkin/anaconda2/lib/python2.7/multiprocessing/util.py", line 305, in _exit_function
File "/home/ekobylkin/anaconda2/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "/home/ekobylkin/anaconda2/lib/python2.7/multiprocessing/pool.py", line 102, in worker
self.run()
File "/home/ekobylkin/anaconda2/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "/home/ekobylkin/anaconda2/lib/python2.7/multiprocessing/pool.py", line 102, in worker
task = get()
File "/home/ekobylkin/anaconda2/lib/python2.7/multiprocessing/queues.py", line 376, in get
_run_finalizers(0)
File "/home/ekobylkin/anaconda2/lib/python2.7/multiprocessing/util.py", line 274, in _run_finalizers
finalizer()
racquire()
File "/home/ekobylkin/anaconda2/lib/python2.7/multiprocessing/util.py", line 207, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/home/ekobylkin/anaconda2/lib/python2.7/multiprocessing/pool.py", line 495, in _terminate_pool
KeyboardInterrupt
cls._help_stuff_finish(inqueue, task_handler, len(pool))
File "/home/ekobylkin/anaconda2/lib/python2.7/multiprocessing/pool.py", line 480, in _help_stuff_finish
inqueue._rlock.acquire()
KeyboardInterrupt
task = get()
File "/home/ekobylkin/anaconda2/lib/python2.7/multiprocessing/queues.py", line 376, in get
racquire()
KeyboardInterrupt
Error in sys.exitfunc:
Traceback (most recent call last):
File "/home/ekobylkin/anaconda2/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/home/ekobylkin/anaconda2/lib/python2.7/multiprocessing/util.py", line 305, in _exit_function
_run_finalizers(0)
File "/home/ekobylkin/anaconda2/lib/python2.7/multiprocessing/util.py", line 274, in _run_finalizers
finalizer()
File "/home/ekobylkin/anaconda2/lib/python2.7/multiprocessing/util.py", line 207, in __call__
res = self._callback(*self._args, **self._kwargs)
File "/home/ekobylkin/anaconda2/lib/python2.7/multiprocessing/pool.py", line 495, in _terminate_pool
cls._help_stuff_finish(inqueue, task_handler, len(pool))
File "/home/ekobylkin/anaconda2/lib/python2.7/multiprocessing/pool.py", line 480, in _help_stuff_finish
inqueue._rlock.acquire()
KeyboardInterrupt
self.run()
File "/home/ekobylkin/anaconda2/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "/home/ekobylkin/anaconda2/lib/python2.7/multiprocessing/pool.py", line 102, in worker
task = get()
File "/home/ekobylkin/anaconda2/lib/python2.7/multiprocessing/queues.py", line 376, in get
racquire()
KeyboardInterrupt
Process PoolWorker-4:
Traceback (most recent call last):
File "/home/ekobylkin/anaconda2/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/home/ekobylkin/anaconda2/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "/home/ekobylkin/anaconda2/lib/python2.7/multiprocessing/pool.py", line 102, in worker
task = get()
File "/home/ekobylkin/anaconda2/lib/python2.7/multiprocessing/queues.py", line 376, in get
racquire()
KeyboardInterrupt
Process PoolWorker-5:
Traceback (most recent call last):
File "/home/ekobylkin/anaconda2/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/home/ekobylkin/anaconda2/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "/home/ekobylkin/anaconda2/lib/python2.7/multiprocessing/pool.py", line 102, in worker
task = get()
File "/home/ekobylkin/anaconda2/lib/python2.7/multiprocessing/queues.py", line 376, in get
racquire()
KeyboardInterrupt
^C^C^C
from auto-sklearn.
mfeurer
commented on May 13, 2024
This issue seems to be related to the use of SMAC2 (written in Java). @Motorrat did you also observe it with a version of auto-sklearn >= 0.1.0?
from auto-sklearn.
Motorrat
commented on May 13, 2024
I don't think I do.
from auto-sklearn.
mfeurer
commented on May 13, 2024
Thanks for letting me know. I'm closing this for now.
from auto-sklearn.
Related Issues (20)
- Revisited: Missing multiclass-multioutput support HOT 1
- [Question] What is the clear termination condition for the autoML algorithm?
- [Question] when auto-sklean will be compatible with scikit-learn 1.0 or above.
- [Question] An error occurs while installing autosklearn HOT 9
- Add support to Python 3.10 and 3.11. Before that – update documentation HOT 5
- Auto Sklearn never stop training model HOT 1
- [Question] On the Preprocessing Problem of the Best Model for Autosklearn Training Model
- auto-sklearn holds requirements.txt indexed from external file
- [Question] How to Implement and Utilize MultiObjectiveDummyEnsemble in auto-sklearn?
- [Question] How to get the pre-processed data used by `auto-sklearn` to train a model?
- Install auto-sklearn in Colab HOT 1
- [Question] Is there a way to configure n_configs of the SMAC's module ?
- calling model.show_models() give error as
- calling model.show_models() give error HOT 2
- Set Preprocess = None, but still get SimpleImputer problem; besides some warnings alongside
- ValueError: (' Dummy prediction failed with run state StatusType.CRASHED and additional output HOT 1
- [Question] TimeSeriesSplit to use auto-sklearn for time series doesn't work
- [Question] What kind of Hyperparameter optimisation technique does AutoSklearn use?
- [Question] How do I fix this issue? HOT 20
- [Question] Where does the Bayesian Optimisation is working for Hyperparameter search? HOT 16
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from auto-sklearn.