How to use with AWS Glue? about spark-extension HOT 9 CLOSED

Glue 3.0 uses Spark 3.1, so you should use spark-extension_2.12:2.1.0-3.1, not ...-3.3.

@EnricoMi Thanks. And you have inserted the s3 path of jar file for Python library path or Dependent JARs path ?

As Python library path. I uploaded the jar file as is.

from spark-extension.

You should be able to use this as any other Spark package. Try to find out how this is done in general, it should apply to spark-extension as well.

https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-python-libraries.html#aws-glue-programming-python-libraries-job
https://stackoverflow.com/questions/53382166/how-to-import-spark-packages-in-aws-glue

from spark-extension.

@leewalter78 do you have some suggestions, looks like you have used this with Glue.

from spark-extension.

#64 (comment)

from spark-extension.

I get an error something like this.

from spark-extension.

You are using Python? Which spark-extension version did you use? Which Glue version?

Do you have step-by-step instructions for me to reconstruct this error on Glue?

from spark-extension.

You are using Python? Which spark-extension version did you use? Which Glue version?

Do you have step-by-step instructions for me to reconstruct this error on Glue?

I was using spark-extension_2.12:2.1.0-3.3 with Glue 3.0, and python 3

from spark-extension.

Glue 3.0 uses Spark 3.1, so you should use spark-extension_2.12:2.1.0-3.1, not ...-3.3.

from spark-extension.

Glue 3.0 uses Spark 3.1, so you should use spark-extension_2.12:2.1.0-3.1, not ...-3.3.

@EnricoMi Thanks. And you have inserted the s3 path of jar file for Python library path or Dependent JARs path ?

from spark-extension.