Topic: apache-spark Goto Github

👇 Here are 1612 public repositories matching this topic...

1duo / awesome-ai-infrastructures
apache-spark,Infrastructures™ for Machine Learning Training/Inference in Production.
User: 1duo
artificial-intelligence machine-learning deep-learning machine-learning-systems awesome-list deep-learning-framework kubernetes apache-spark apache-arrow apache-mesos

aloneguid / parquet-dotnet
apache-spark,Fully managed Apache Parquet implementation
User: aloneguid
Home Page: https://aloneguid.github.io/parquet-dotnet/
apache-parquet dotnet dotnet-core dotnet-standard apache-spark linux windows ios xamarin xbox

awesome-spark / awesome-spark
apache-spark,A curated list of awesome Apache Spark packages and resources.
Organization: awesome-spark
apache-spark pyspark awesome sparkr

awesome-spark / spark-gotchas
apache-spark,Spark Gotchas. A subjective compilation of the Apache Spark tips and tricks
Organization: awesome-spark
apache-spark book guide pyspark

big-data-europe / docker-spark
apache-spark,Apache Spark docker image
Organization: big-data-europe
spark-kubernetes kubernetes k8s-spark docker apache-spark

cartershanklin / pyspark-cheatsheet
apache-spark,PySpark Cheat Sheet - example code to help you learn PySpark and develop apps faster
User: cartershanklin
apache-spark big-data pyspark spark

cerndb / dist-keras
apache-spark,Distributed Deep Learning, with a focus on distributed training, using Keras and Apache Spark.
Organization: cerndb
Home Page: http://joerihermans.com/work/distributed-keras/
machine-learning deep-learning apache-spark data-parallelism distributed-optimizers keras optimization-algorithms tensorflow data-science hadoop

cuebook / cuelake
apache-spark,Use SQL to build ELT pipelines on a data lakehouse.
Organization: cuebook
Home Page: https://cuelake.cuebook.ai
apache-iceberg delta lakehouse datalake data-lake elt etl data-engineering data-integration data-ingestion

databricks / learningsparkv2
apache-spark,This is the github repo for Learning Spark: Lightning-Fast Data Analytics [2nd Edition]
Organization: databricks
Home Page: https://learning.oreilly.com/library/view/learning-spark-2nd/9781492050032/
apache-spark spark structured-streaming spark-sql spark-mllib mllib mlflow delta-lake

databricks / spark-sklearn
apache-spark,(Deprecated) Scikit-learn integration package for Apache Spark
Organization: databricks
apache-spark scikit-learn grid-search parameter-tuning machine-learning

datamechanics / delight
apache-spark,A Spark UI and Spark History Server alternative with CPU and Memory metrics! Delight is free, cross-platform, and open-source.
Organization: datamechanics
Home Page: https://www.datamechanics.co/delight
apache-spark cpu dashboard delight kubernetes memory monitoring netapp-public spark spark-history-server spark-ui

dmmiller612 / sparktorch
apache-spark,Train and run Pytorch models on Apache Spark.
User: dmmiller612
pytorch sparktorch deep-learning apache-spark pipelines inference distributed-computing

dotnet / spark
apache-spark,.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.
Organization: dotnet
Home Page: https://dot.net/spark
spark csharp dotnet analytics bigdata spark-streaming spark-sql machine-learning fsharp dotnet-core

ekampf / pyspark-boilerplate
apache-spark,A boilerplate for writing PySpark Jobs
User: ekampf
python apache-spark pyspark boilerplate

feathr-ai / feathr
apache-spark,Feathr – A scalable, unified data and AI engineering platform for enterprise
Organization: feathr-ai
Home Page: https://join.slack.com/t/feathrai/shared_invite/zt-1ffva5u6v-voq0Us7bbKAw873cEzHOSg
feature-engineering feature-store artificial-intelligence mlops data-engineering data-quality machine-learning apache-spark azure data-science

hortonworks-spark / spark-atlas-connector
apache-spark,A Spark Atlas connector to track data lineage in Apache Atlas
Organization: hortonworks-spark
apache-spark apache-atlas

hydrospheredata / mist
apache-spark,Serverless proxy for Spark cluster
Organization: hydrospheredata
Home Page: http://hydrosphere.io/mist/
apache-spark api big-data serverless

infoslack / awesome-kafka
apache-spark,A list about Apache Kafka
User: infoslack
kafka streaming-data data-pipeline stream-processing apache-kafka apache-spark kafka-streams infrastructure data-processing

intel-analytics / bigdl-2.x
apache-spark,BigDL: Distributed TensorFlow, Keras and PyTorch on Apache Spark/Flink & Ray
Organization: intel-analytics
Home Page: https://bigdl.readthedocs.io
apache-spark deep-neural-network distributed-deep-learning keras-tensorflow bigdl analytics-zoo python scala pytorch

japila-books / apache-spark-internals
apache-spark,The Internals of Apache Spark
Organization: japila-books
Home Page: https://books.japila.pl/apache-spark-internals
apache-spark spark book internals

japila-books / spark-sql-internals
apache-spark,The Internals of Spark SQL
Organization: japila-books
Home Page: https://books.japila.pl/spark-sql-internals
apache-spark spark-sql internals spark mkdocs-material book

japila-books / spark-structured-streaming-internals
apache-spark,The Internals of Spark Structured Streaming
Organization: japila-books
Home Page: https://books.japila.pl/spark-structured-streaming-internals
apache-spark spark structured-streaming internals book mkdocs-material

kubeflow / spark-operator
apache-spark,Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.
Organization: kubeflow
kubernetes kubernetes-operator apache-spark kubernetes-crd kubernetes-controller spark google-cloud-dataproc

lensacom / sparkit-learn
apache-spark,PySpark + Scikit-learn = Sparkit-learn
Organization: lensacom
scikit-learn apache-spark machine-learning distributed-computing python

lifeomic / sparkflow
apache-spark,Easy to use library to bring Tensorflow on Apache Spark
Organization: lifeomic
tensorflow apache-spark deep-learning dataframe spark-ml pipeline lifeomic

lucacanali / miscellaneous
apache-spark,Includes notes on using Apache Spark in general, notes on using Spark for Physics, how to run TPCDS on PySpark, how to create histograms with Spark, tools for performance testing CPUs, Jupyter notebooks examples for Spark, examples for Oracle and other DB systems.
User: lucacanali
apache-spark database jupyter-notebooks performance-analysis performance-monitoring performance-testing

lucacanali / sparkmeasure
apache-spark,This is the development repository for sparkMeasure, a tool and library designed for efficient analysis and troubleshooting of Apache Spark jobs. It focuses on easing the collection and examination of Spark metrics, making it a practical choice for both developers and data engineers.
User: lucacanali
spark apache-spark performance-troubleshooting performance-metrics python scala

lw-lin / coolplayspark
apache-spark,酷玩 Spark: Spark 源代码解析、Spark 类库等
User: lw-lin
spark spark-streaming structured-streaming sparkcore apache-spark

lw-lin / streaming-readings
apache-spark,Streaming System 相关的论文读物
User: lw-lin
stream-processing streaming flink spark-streaming storm heron dataflow drizzle millwheel s4

mahmoudparsian / data-algorithms-book
apache-spark, MapReduce, Spark, Java, and Scala for Data Algorithms Book
User: mahmoudparsian
Home Page: http://mapreduce4hackers.com
hadoop-mapreduce java distributed-computing scala mapreduce data-algorithms python machine-learning pyspark distributed-algorithms

microsoft / data-accelerator
apache-spark,Data Accelerator for Apache Spark simplifies onboarding to Streaming of Big Data. It offers a rich, easy to use experience to help with creation, editing and management of Spark jobs on Azure HDInsights or Databricks while enabling the full power of the Spark engine.
Organization: microsoft
spark spark-streaming spark-sql sparksql streaming-data streaming servicefabric nodejs docker hdinsight

microsoft / mobius
apache-spark,C# and F# language binding and extensions to Apache Spark
Organization: microsoft
spark apache-spark rdd dataframe dstream dataset streaming csharp mobius kafka-streaming

microsoft / synapseml
apache-spark,Simple and Distributed Machine Learning
Organization: microsoft
Home Page: http://aka.ms/spark
spark pyspark azure scala microsoft ml machine-learning databricks cognitive-services lightgbm

miguno / kafka-storm-starter
apache-spark,[PROJECT IS NO LONGER MAINTAINED] Code examples that show to integrate Apache Kafka 0.8+ with Apache Storm 0.9+ and Apache Spark Streaming 1.1+, while using Apache Avro as the data serialization format.
User: miguno
Home Page: http://www.michael-noll.com/blog/2014/05/27/kafka-storm-integration-example-tutorial/
apache-kafka kafka apache-storm storm spark apache-spark scala integration avro apache-avro

miguno / wirbelsturm
apache-spark,[PROJECT IS NO LONGER MAINTAINED] Wirbelsturm is a Vagrant and Puppet based tool to perform 1-click local and remote deployments, with a focus on big data tech like Kafka.
User: miguno
Home Page: http://www.michael-noll.com/blog/2014/03/17/wirbelsturm-one-click-deploy-storm-kafka-clusters-with-vagrant-puppet/
vagrant puppet kafka apache-kafka storm apache-storm spark apache-spark

mingchen0919 / learning-apache-spark
apache-spark,Notes on Apache Spark (pyspark)
User: mingchen0919
apache-spark pyspark-tutorial machine-learning

mlflow / mlflow
apache-spark,Open source platform for the machine learning lifecycle
Organization: mlflow
Home Page: https://mlflow.org
machine-learning ai ml mlflow apache-spark model-management

mrpowers / quinn
apache-spark,pyspark methods to enhance developer productivity 📣 👯 🎉
User: mrpowers
Home Page: https://mrpowers.github.io/quinn/
pyspark apache-spark

nchammas / flintrock
apache-spark,A command-line tool for launching Apache Spark clusters.
User: nchammas
apache-spark ec2 apache-spark-cluster orchestration spark-ec2

opencypher / morpheus
apache-spark,Morpheus brings the leading graph query language, Cypher, onto the leading distributed processing platform, Spark.
Organization: opencypher
apache-spark cypher scala apache2 graph big-data

openscoring / openscoring
apache-spark,REST web service for the true real-time scoring (<1 ms) of Scikit-Learn, R and Apache Spark models
Organization: openscoring
api real-time pmml r scikit-learn apache-spark xgboost lightgbm

oryxproject / oryx
apache-spark,Oryx 2: Lambda architecture on Apache Spark, Apache Kafka for real-time large scale machine learning
Organization: oryxproject
Home Page: http://oryx.io
lambda-architecture oryx apache-spark machine-learning kafka apache-kafka java cloudera

ptyadana / sql-data-analysis-and-visualization-projects
apache-spark,SQL data analysis & visualization projects using MySQL, PostgreSQL, SQLite, Tableau, Apache Spark and pySpark.
User: ptyadana
sql mysql mysql-notes exercises mysqlworkbench mysql-database data-analysis postgresql postgres sqlite

rjurney / agile_data_code_2
apache-spark,Code for Agile Data Science 2.0, O'Reilly 2017, Second Edition
User: rjurney
Home Page: http://bit.ly/agile_data_science
agile-data agile-data-science airflow amazon-ec2 amazon-web-services analytics apache-kafka apache-spark data data-science data-syndrome kafka machine-learning machine-learning-algorithms predictive-analytics python python-3 python3 spark vagrant

san089 / goodreads_etl_pipeline
apache-spark,An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.
User: san089
etl-pipeline etl-framework spark apache-spark apache-airflow airflow redshift emr-cluster livy s3

spark-notebook / spark-notebook
apache-spark,Interactive and Reactive Data Science using Scala and Spark.
Organization: spark-notebook
apache-spark notebook scala data-science spark reactive

sparklyr / sparklyr
apache-spark,R interface for Apache Spark
Organization: sparklyr
Home Page: https://spark.rstudio.com/
apache-spark distributed dplyr ide livy machine-learning r remote-clusters rstats spark sparklyr

tirthajyoti / spark-with-python
apache-spark,Fundamentals of Spark with Python (using PySpark), code examples
User: tirthajyoti
pyspark spark apache-spark dataframe mlib machine-learning big-data database map-reduce python

treeverse / lakefs
apache-spark,lakeFS - Data version control for your data lake | Git for data
Organization: treeverse
Home Page: https://docs.lakefs.io
data-engineering data-versioning go object-storage data-lake aws-s3 data-quality azure-blob-storage google-cloud-storage golang

tweag / sparkle
apache-spark,Haskell on Apache Spark.
Organization: tweag
apache-spark spark haskell analytics