Giter Club home page Giter Club logo

ibisml's Introduction

IbisML

Build status Docs License PyPI

ibisml is a work-in-progress library for developing Machine Learning feature engineering pipelines using ibis. These pipelines can then be used to transform and feed data to other machine learning libraries like xgboost or scikit-learn.

import ibis
import ibisml as ml

# A recipe for a feature engineering pipeline that:
# - imputes missing values in numeric columns with their mean
# - applies standard scaling to all numeric columns
# - one-hot-encodes all nominal columns
recipe = ml.Recipe(
    ml.ImputeMean(ml.numeric()),
    ml.ScaleStandard(ml.numeric()),
    ml.OneHotEncode(ml.nominal()),
)

# Use the recipe inside of a larger Scikit-Learn pipeline
from sklearn.pipeline import Pipeline
pipeline = Pipeline([("recipe", recipe), ("model", LinearSVC())])

# Fit the recipe against some local training data,
# just as you would with any other scikit-learn model
X, y = load_training_data()
pipeline.fit(X, y)

# Evaluate the model against some local testing data.
X_test, y_test = load_testing_data()
pipeline.score(X_test, y_test)

# Now apply the same preprocessing pipeline against any of ibis's
# supported backends
con = ibis.connect(...)
X_remote = con.table["mytable"]
for batch in recipe.to_pyarrow_batches(X_remote):
    ...

By using ibis for preprocessing and feature engineering, feature engineering pipelines may be compiled to SQL and executed on a wide range of performant and scalable backends. No more need to rewrite code for production deployments, pipelines may be developed locally (against e.g. duckdb) and deployed to production (against e.g. spark) with only a single line of code change.

Help Wanted!

ibisml is a work-in-progress. If you're interested in getting involved (whether through feature requests, PRs, or just sharing opinions), we'd love to hear from you.

ibisml's People

Contributors

jcrist avatar deepyaman avatar indexseek avatar jitingxu1 avatar toohsk avatar gforsyth avatar sfc-gh-twhite avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.