Comments (5)
Hi! Yes I've found the c-index can be quite unstable. However, for large datasets (such as the KKBox dataset) it gives more stable results. If you're working on small datasets, you probably can't trust the c-index results that much, and if you're doing cross-validation the variability between the folds is likely larger than than the variability between the models. There's no easy way around this sadly.
One way to compare models is to use different seeds and check which models gets the best c-index for each seed. I've found that often the best model is best for many seeds, even thought the c-index varies a lot between seed. Using this evaluation, you essentially use the c-index to rank your models, but you don't really care about the value of the c-index.
from pycox.
Thanks for your patient and timely reply!
Actually, I am applying the time-varying cox models in another scenario (to be specific, for user churn analysis in the video game). And I am trying to find the best model, covariates, and parameters to fit my problem.
Unfortunately, the best models are not always the same for different seeds. Do you think it is reasonable to choose the model with the least loss on validation set?
And I also wonder why the model with significant smaller validation loss may perform worse on C-index (on validation set). In my opinion, the loss and C-index should be consistent, according to their calculation. Have you noticed the similar phenomena?
from pycox.
Which model selection criterion you use should probably be reflected by how you want to apply your model in practise. So if you only want to rank your users compared to each other (e..g, report the highest 10%), a high concordance is important. However, if you want to give probabilities for churn for the users, concordance is likely not the best scoring rule and something like the log-likelihood (validation loss) might make more sense. For example, if you divide all your survival predictions by 10, you concordance is unchanged while your loss should be much worse.
Ideally, I should probably encourage you to use multiple evaluation metrics, but I agree that it's probably fine to just consider the validation loss.
If I recall correctly, a theoretically optimal model (perfect survival estimates) should obtain an optimal concordance and loss. However, all other models might have a tradeoff between the two. For instance, DeepHit is typically good at ranking (high concordance) but often produce very bad survival estimates (which would give a bad survival likelihood).
from pycox.
Thanks a lot for your comment! It really helps a lot.
I read more works about survival analysis this week. You are right about the difference between likelihood and concordance. And in our application, I think it is better to choose models based on likelihood.
Thanks again!
from pycox.
Happy to help!
from pycox.
Related Issues (20)
- Issue in function cox_time.py HOT 1
- L1 and L2 penalty coxph HOT 1
- AssertionError: assert durations.shape[0] == surv.shape[1] == surv_idx.shape[0] == events.shape[0]
- METABRIC Covariates Subset HOT 1
- AttributeError: 'Series' object has no attribute 'is_monotonic' HOT 18
- about hazard value! HOT 2
- Reproduction of the results in JMLR19 paper HOT 1
- Calculating Estimated Population Survival Curve HOT 4
- Some question about the result of deephit_competing_risks HOT 2
- AttributeError: 'DeepHitSingle' object has no attribute 'state_dict' HOT 1
- ValueError: cannot convert float NaN to integer HOT 1
- Softmax layer and residual connections in DeepHitSingle model HOT 1
- _initialization of _internal failed
- TypeError: forward() missing 1 required positional argument: 'events'
- ValueError: cannot convert float NaN to integer HOT 1
- A model to add
- Auto-encoder pycox implementation for 3D images instead of tabular data
- performance for ordinal categorical covariates
- what kind of model in pycox works for sequential patterns
- Newton-Raphson optimization
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pycox.