Lighter is an opensource application for interacting with Apache Spark on Kubernetes or Apache Hadoop YARN. It is hevily inspired by Apache Livy and has some overlaping features.

Lighter supports:

• Interactive Python Sessions through Sparkmagic kernel
• Batch application submissions through the REST API

⚠️ If you are using interactive sessions: While we have tested batch applications quite extensively, there might be some problems with interactive sessions, consider current release of Lighter as alpha.

You can read a breaf description on how Lighter works here.

• Installation on Kubernetes
• Installation on Docker
• Configuration Properties
• Configuring Sparkmagic
• Using REST API

To build Docker image, containing application, you need to run

docker build -t lighter .

Lighter will always support latest patch version for all officially recommended Apache Spark releases i.e.: 3.0.3 and 3.1.2.

See Contribution guide

Lighter is MIT liceansed.