Hello, there is a problem with starting pyspark about glow HOT 10 CLOSED

Are you sure that you're launching the pyspark shell from the binary distribution with scala 2.11? Maybe you're launching a pip or conda installed version? I'm pretty sure that your error is caused by a scala version mismatch.

from glow.

@henrydavidge I am sure that the version of scala I am using is 2.11.12, which comes with the sprak-bin. There is no problem with using spark-shell to start glow. If it is caused by the scala version, the spark-shell should not work properly. , But only pyspark will report an error, the following is the spark-shell startup interface:

from glow.

@lsygxw Yep, the fact that you only get an error when using pyspark makes me think that the pyspark shell corresponds to a different spark installation with the wrong scala version.

Try running these commands in the pyspark shell:

>>> spark._jvm.scala.util.Properties.versionString()
>>> spark._jvm.scala.util.Properties.javaVersion()

from glow.

@henrydavidge First of all, thank you for your help. The result is as follows after executing in pyspark according to your method:

Are the results as expected?

from glow.

Hm, that does look like the right scala and java version. How exactly are you installing glow?

from glow.

@henrydavidge Same as the demo on your document pip3 install glow.py
Is it possible to lack some dependency or configuration? My 'spark-env.sh' configuration is as follows:

from glow.

I get the same error with the versions below.

 ____              __
 / __/__  ___ _____/ /__
_\ \/ _ \/ _ `/ __/  '_/

/__ / .__/_,// //_\ version 2.4.3
//

Using Python version 3.6.9 (default, Apr 18 2020 01:56:04)
SparkSession available as 'spark'.

spark._jvm.scala.util.Properties.versionString()
'version 2.11.12'
spark._jvm.scala.util.Properties.javaVersion()
'1.8.0-adoptopenjdk'

from glow.

In my case the problem was that I used the dowloaded glow jar file instead of the --packages option because of a corporate proxy. That did not load the dependencies and caused problems.

this was required to get proxy and certificates working:
pyspark --conf "spark.driver.extraJavaOptions=-Djava.net.useSystemProxies=true -Djavax.net.ssl.trustStore=keystore.jks" --packages io.projectglow:glow_2.11:0.4.0

from glow.

Ah, thank you for the response @magelan! That explains why I couldn't reproduce. @lsygxw is that your problem as well?

from glow.

I'm tentatively going to close. @lsygxw feel free to reopen if your issue persists.

from glow.