Comments (4)
jcreinhold
commented on May 23, 2024
Thanks for the code. I was trying to write a consistent unit test for PR #872 (which fixes this issue and #866); however, I'm unable to reproduce the error with the following code (structured as a unit test in test_shap.py):
def test_check_additivity_handling(self):
np.random.seed(0)
n = 100
X = np.hstack([np.random.randint(2, size=(n, 1)), np.random.randint(100, size=(n, 1))])
T = np.random.randint(2, size=(n,))
Y = np.random.randint(2, size=(n,))
X_train, _, T_train, _, Y_train, _ = train_test_split(X, T, Y, test_size=0.2, random_state=42)
scaler = MinMaxScaler()
X_train[:, 1:2] = scaler.fit_transform(X_train[:, 1:2])
est = CausalForestDML(
model_y=RandomForestClassifier(),
model_t=DummyClassifier(strategy="uniform"),
random_state=123,
discrete_outcome=True,
discrete_treatment=True
)
est.fit(Y_train, T_train, X=X_train)
shap_values = est.shap_values(X_train[:20])
assert shap_values is not NoneAs discussed in my second comment in #866, I don't believe there's a consistent way to reproduce this error across machines—for various reasons. However, #872 will fix this issue regardless.
from econml.
jcreinhold
commented on May 23, 2024
@yanisvdc Do you have the literal values for X_train, T_train, and y_train that create the error (after scaling)? I can try hard-coding those values in and seeing if that reproduces the error. Given they're size 80, this seems feasible.
Could you please try the following (on the set of variables that cause the error):
arrays = {
"X": X_train, # after scaling (potentially call .to_numpy() on it if it's a df)
"T": T_train.squeeze(),
"Y": y_train.squeeze(),
}
for name, arr in arrays.items():
mlw = 20 if name != "X" else 50
arr_str = np.array2string(arr, separator=", ", floatmode="maxprec", precision=16, max_line_width=mlw)
print(f"{name}:\n\n{arr_str}")and copy the results here in a markdown codeblock for each variable?
from econml.
yanisvdc
commented on May 23, 2024
Thanks for your comments.
Your test named test_check_additivity_handling() should fail by adding "astype(float)" to X. However, your test passes by removing it. The problem is when you remove "astype(float)" to X, the MinMaxScaler outputs zeros as it casts the values to int to match the dtype. Thus, your current test passes but that is only because X_train ends up being an array with binary integer values. In the more general case, with "astype(float)" added to X, MinMaxScaler outputs floats between 0 and 1, and the error displayed below will be raised (see after the code containing "astype(float)" added to X)
np.random.seed(0)
n = 100
X = np.hstack([np.random.randint(2, size=(n, 1)), np.random.randint(100, size=(n, 1))]).astype(float)
T = np.random.randint(2, size=(n,))
Y = np.random.randint(2, size=(n,))
X_train, _, T_train, _, Y_train, _ = train_test_split(X, T, Y, test_size=0.2, random_state=42)
scaler = MinMaxScaler()
X_train[:, 1:2] = scaler.fit_transform(X_train[:, 1:2])
est = CausalForestDML(
model_y=RandomForestClassifier(),
model_t=DummyClassifier(strategy="uniform"),
random_state=123,
discrete_outcome=True,
discrete_treatment=True
)
est.fit(Y_train, T_train, X=X_train)
shap_values = est.shap_values(X_train[:20])
shap.plots.beeswarm(shap_values['Y0']['T0_1'])
Error message:
ExplainerError: Additivity check failed in TreeExplainer! Please ensure the data matrix you passed to the explainer is the same shape that the model was trained on. If your data shape is correct then please report this on GitHub. This check failed because for one of the samples the sum of the SHAP values was -0.061160, while the model output was -0.059755. If this difference is acceptable you can set check_additivity=False to disable this check.
Please let me know if you experience the same behavior.
from econml.
yanisvdc
commented on May 23, 2024
@yanisvdc Do you have the literal values for
X_train,T_train, andy_trainthat create the error (after scaling)? I can try hard-coding those values in and seeing if that reproduces the error. Given they're size 80, this seems feasible.Could you please try the following (on the set of variables that cause the error):
arrays = { "X": X_train, # after scaling (potentially call .to_numpy() on it if it's a df) "T": T_train.squeeze(), "Y": y_train.squeeze(), } for name, arr in arrays.items(): mlw = 20 if name != "X" else 50 arr_str = np.array2string(arr, separator=", ", floatmode="maxprec", precision=16, max_line_width=mlw) print(f"{name}:\n\n{arr_str}")and copy the results here in a markdown codeblock for each variable?
To answer this, here are the exact values:
X:
[[0. , 0.202020202020202 ],
[1. , 0.4040404040404041],
[1. , 0.696969696969697 ],
[0. , 0.4343434343434344],
[0. , 0.5151515151515152],
[0. , 0.2323232323232323],
[0. , 0. ],
[0. , 0.6767676767676768],
[1. , 0.5252525252525253],
[0. , 0.5454545454545455],
[0. , 0.1515151515151515],
[1. , 0.98989898989899 ],
[1. , 0.9494949494949496],
[1. , 0. ],
[0. , 0.2626262626262627],
[1. , 0.686868686868687 ],
[0. , 0.4343434343434344],
[0. , 0.8686868686868687],
[1. , 0.98989898989899 ],
[0. , 0.797979797979798 ],
[1. , 0.3535353535353536],
[1. , 0.0303030303030303],
[1. , 0.0303030303030303],
[0. , 0.5252525252525253],
[1. , 0.9494949494949496],
[1. , 0.4848484848484849],
[1. , 0.303030303030303 ],
[1. , 0.9595959595959597],
[1. , 0.6565656565656566],
[1. , 1. ],
[0. , 0.6464646464646465],
[0. , 0.686868686868687 ],
[0. , 0.1313131313131313],
[1. , 0.2121212121212121],
[1. , 0.7676767676767677],
[1. , 0.5656565656565657],
[1. , 0.4747474747474748],
[1. , 0.101010101010101 ],
[0. , 0.4242424242424243],
[0. , 0.7272727272727273],
[1. , 0.8181818181818182],
[0. , 0.9696969696969697],
[1. , 1. ],
[1. , 0.5858585858585859],
[0. , 0.9494949494949496],
[0. , 0.1111111111111111],
[1. , 0.0202020202020202],
[0. , 0.0202020202020202],
[0. , 0.5858585858585859],
[1. , 0.6161616161616162],
[0. , 0.6060606060606061],
[1. , 0.101010101010101 ],
[0. , 0.2727272727272728],
[0. , 0.6262626262626263],
[1. , 0.1414141414141414],
[0. , 0.5050505050505051],
[1. , 0.1919191919191919],
[1. , 0.4848484848484849],
[1. , 0.6666666666666667],
[1. , 0.1414141414141414],
[0. , 0.3636363636363636],
[1. , 0.0303030303030303],
[1. , 0.7777777777777778],
[0. , 0.5050505050505051],
[1. , 0.8484848484848485],
[1. , 0.8282828282828284],
[1. , 0.696969696969697 ],
[1. , 0.7575757575757577],
[0. , 0.3535353535353536],
[1. , 0.1313131313131313],
[1. , 0.9595959595959597],
[0. , 0.3838383838383839],
[1. , 0.98989898989899 ],
[0. , 0.2424242424242424],
[0. , 0.494949494949495 ],
[1. , 0.4141414141414142],
[1. , 0.3232323232323233],
[0. , 0.5858585858585859],
[0. , 0.7272727272727273],
[1. , 0.6767676767676768]]
T:
[0, 1, 0, 0, 1, 0,
0, 1, 1, 0, 0, 1,
1, 1, 1, 0, 1, 0,
1, 1, 1, 1, 1, 1,
0, 1, 0, 0, 0, 1,
1, 1, 0, 1, 1, 1,
1, 0, 0, 0, 1, 1,
1, 1, 1, 1, 1, 1,
0, 1, 0, 0, 1, 1,
1, 0, 0, 1, 1, 1,
1, 1, 0, 1, 1, 0,
1, 1, 1, 1, 1, 0,
1, 0, 0, 0, 1, 1,
1, 0]
Y:
[1, 1, 1, 0, 1, 1,
0, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 0,
1, 0, 0, 0, 1, 0,
0, 0, 0, 0, 1, 1,
0, 0, 0, 0, 0, 0,
1, 0, 1, 1, 0, 0,
1, 0, 1, 1, 1, 0,
0, 1, 1, 1, 1, 1,
1, 1, 0, 0, 1, 0,
1, 1, 1, 0, 0, 0,
1, 0, 1, 0, 0, 1,
1, 1, 1, 1, 1, 1,
0, 0]
from econml.
Related Issues (20)
- Is a feature engineered from treatment T another treatment to consider for CATE?
- Will DRIV be able to support multiple treatments via multiple instruments?
- DynamicDML() issue: AttributeError: Provided crossfit folds contain training splits that don't contain all treatments DynamicDML HOT 5
- Inconsistent ATE estimation HOT 3
- Confidence Interval for categorical outcome HOT 3
- [Bug] fit_cate_incercept argument in econml.dml.DML does not add intercept correctly HOT 5
- `shap_values` for tree-based models doesn't set `check_additivity=False` as expected HOT 3
- A column-vector y was passed when a 1d array was expected (however, y is already a 1d array) HOT 1
- Individual Treatment Effects HOT 1
- How to get the Confidence Interval for ATE instead of CATE HOT 1
- Converting to Python object not allowed without gil HOT 1
- Questions regarding DRPolicyForest results HOT 2
- DRtester does not work for binary treatment AND binary outcome HOT 4
- Confounder adjusting before applying the ITE model to observational data
- Calculation of confidence intervals in NormalInferenceResults becomes very slow when passing big dataframes HOT 2
- DML discrete outcome HOT 1
- High memory footprint for big dataframes in CausalForest model HOT 3
- Questions about econml and CausalForestDML
- Reduce residual confounding in time series
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from econml.