Giter Club home page Giter Club logo

spark-training's Introduction

Spark Training Repository

This repository contains many different examples, exercises and tutorials for Spark and Hadoop trainings performed by dimajix. You can always find the latest version on GitHub at

https://github.com/dimajix/spark-training

Contents

The repository contains different types of documents

  • Source Code for Spark/Scala
  • Jupyter Notebooks for PySpark
  • Zeppelin Notebooks for Spark/Scala
  • Hive SQL scripts
  • Pig scripts
  • ...and much more

External Dependencies

Some notebooks require some test data provided by dimajix on S3 at s3://dimajix-training/data/.

Building Executables

The source code can be built using Maven, simply by running

mvn install

from the root directory.

Running Examples

Most code is either provided as interactive Notebooks (Jupyter and/or Zeppelin) or as compilable programs. Programs which create jar files always contain start scripts, which take care of setting any environment variables and Spark configuration properties.

spark-training's People

Contributors

kupferk avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar

Forkers

thejanagala mallik-g csbigdatahub naakheel chazchandler jhseidel wuxiaolei499390725 vivek-bombatkar plthiyagu tomyc blondas hosnimed sushant20 ersawant ampolusantosh msellamitn hemant-rout gopinath678 shubhmshr gitpradeep54 anishekkamal pawandsingh er-vivekkumar trivenigk preshi mmortazavi adityakonda tangoankur vamosraghava galesun mithranvm mesunilpatil slachiewicz srinivasgutta7 srinath0812 ibelieveai baichenjing devops-best-practices sabyadg anhmike zhuohuwu0603 vlobachev deepakpanda93 nancunpei hypathia databill86 techtinkerer42 rudrabarik perrypvo ajbd2106 rezaprimasatya lokeshk1438 gshreyash hskapasi zqcsrz o-choksi

spark-training's Issues

JAVA_HOME has to point to a JRE inside a JDK

I just tried to build the training ressources with mvn package and experienced the following error:

[INFO] hadoop-skeleton .................................... FAILURE [  0.697 s]
…
[ERROR] Failed to execute goal on project hadoop-skeleton: Could not resolve dependencies for project de.dimajix.training:hadoop-skeleton:jar:1.0.0: Could not find artifact jdk.tools:jdk.tools:jar:1.8 at specified path /usr/lib/jvm/java-8-openjdk-amd64/jre/../lib/tools.jar -> [Help 1]

My local solution was to set JAVA_HOME to a directory of a JRE that is located within a JDK. Thereafter, $JAVA_HOME/../lib/tools.jar (.. seems to go from the inner JRE into the outer JDK directory) can be successfully resolved and the build works on my machine.

I encountered this error on Windows 10 (that has this JDK/JRE-structure) as well as in my Ubuntu shell (that does not have this JDK/JRE-structure and I would have to create a symlink to satisfy the implicit assumption).

I see two possible solutions here:

  • fix the resolution of tools.jar
  • document the dependency to a JRE that is located inside a JDK in the README.md.

Hope this helps you,
Stephan

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.