Challenges

• Data format: should they operate on Pandas DataFrame?

libxgboostwrapper.so sgould be removed from source i get a ELF error (64 Bit vs 32 Bit). It is anyway included in xgboost.

Current way:

Featurizer(handlers: List[Handler])

An alternative way

listable = lambda t: Union[t, List[t]]

Featurizer(
    no_op: listable(str)=None,
    categorical: listable(str)=None, 
    one_hot: listable(str)=None, 
    binning: listable(str)=None, 
    mappers: listable(Tuple[listable[str], listable[str], Callable]), 
    modules: listable(module)=None,   # if there are many or complex mappers, better define them in a module
    ... , 
    others: listable(Handler)=None)

For example

feat = Featurizer(
    no_op=['Age', 'Capital Gain', 'Hours per week', 'Years in Education', 'Capital Loss'],
    categorical=['Education', 'Marital Status'],
    one_hot='Native Country'
)

This would make data scientist's work cleaner.

@txhai, I will explain this idea to you later.

Since the random_sweep function won't accept random state, RandomizedSearchCV runs with the default random state, which is None. This makes the result different every time one uses this function for the same data.

CategoricalHandler can deal with missing values properly. Other common handlers such as NumericHandler and BinNormalizer currently treat nan as 0.