Comments (3)
Thanks for the bug report. A couple questions:
- Is this happening with the latest version of
isotree
? Currently, this is version 0.5.17-5. - Does it also happen if you use the method
export_model
instead ofpickle
? - Are you passing non-default parameters to
pickle
? (e.g. some specific protocol, compression, etc.) - What kind of hyperparameters are you using?
- Are you using a regular HDD/SDD or are you trying to write to e.g. a networked drive, or some other special storage media?
from isotree.
Below are my answers in bold
-
Is this happening with the latest version of isotree? Currently, this is version 0.5.17-5. -
using 0.5.17 -
Does it also happen if you use the method export_model instead of pickle?
- Using both the methods size is 65GB only after export -
Are you passing non-default parameters to pickle? (e.g. some specific protocol, compression, etc.)
joblib.dump(iso_model, 'model.pkl'.format(prod))
joblib.dump(iso_model, 'model_compresses.pkl'.format(prod), compress = 9)
** with export_model same size issue** -
What kind of hyperparameters are you using?
iso_model = IsolationForest( ntrees=20,nthreads=1,categ_split_type="single_categ",scoring_metric="density",ndim=1,
missing_action="impute", random_state = 1)
iso_model.fit_transform(data_sub.drop(drop,axis = 1)) -
Are you using a regular HDD/SDD or are you trying to write to e.g. a networked drive, or some other special storage media?
I am running the code on Sagemaker Studio Notebook and using S3 for storage
from isotree.
Thanks for the information. So it seems there's no bug in here: if you call fit_transform
, then there will not be any row sub-sampling applied to trees, and if you use the default value for max_depth
, then that in turn will be determined from the full number of rows. And if the number of rows is very large, then the depth of each tree will also be very large, which will lead to producing very heavy models.
Additionally, if using it as a missing value imputer (as it does when using fit_transform
), the trees that it builds for imputation are expected to be much heavier than the trees for producing anomaly scores.
As for what you could do: if the amount of rows is very large, you should call fit
and perhaps manually change the number of rows per tree (parameter sample_size
). Then the models should become smaller. Also, if you don't plan to use it as a missing value imputer, you shouldn't call fit_transform
, nor pass build_imputer
.
from isotree.
Related Issues (20)
- Question: When categorical attributes are split into subsets, are the subsets random? HOT 2
- What is equivalent to contamination parameter in sklearn? HOT 1
- Mac arm nthreads problem HOT 1
- Error when using fit() and build_imputer=True is misleading HOT 2
- Same category is always imputed when enough trees are grown HOT 3
- pip install issue: cc1plus no such file or directory HOT 5
- pip install issue not fixed but closed HOT 1
- Reproductibility problems with Extented Isolation Forest HOT 8
- How to re-train when a few values are marked as anomalies when they should not have been? HOT 3
- Training Isotree with Python on Windows then Deserializing with C++ on Raspberry Pi 3B+ (Linux) HOT 7
- Can't install Isotree anymore HOT 3
- memory issues with build_imputer=True HOT 9
- Unexpected behaviour during imputations with all NA features HOT 7
- Problems installing isotree in Sagemaker Studio HOT 4
- Using with python3.8 and numpy==1.24.4 HOT 1
- Isotree==0.5.22 is failing on databricks cluster HOT 3
- R package build from source failing on Conda Forge HOT 9
- Running isotree in databricks environment given requirements HOT 1
- Problems building isottree library on Windows using Visual Studio 2022 and CMake HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from isotree.