indy-3rdman / docker-dotnet-spark Goto Github PK

View Code? Open in Web Editor NEW

19.0 2.0 9.0 15.35 MB

A .NET for Apache Spark docker image (3rdman/dotnet-spark)

Home Page: https://hub.docker.com/r/3rdman/dotnet-spark

License: Apache License 2.0

Shell 59.55% Dockerfile 23.09% C# 1.37% Jupyter Notebook 12.96% Python 3.04%

spark csharp dotnet-core apache-spark docker spark-master spark-worker docker-image

docker-dotnet-spark's People

Contributors

Stargazers

Watchers

Forkers

brunosaboia padamshrestha kimkihyuk andrey-puzyr wesdoyle maciusik portalmedia

docker-dotnet-spark's Issues

Azure Data Lake Store Support

Would you be interested in a PR adding support for communicating with Azure Datalake?

The changes I have made to enable it locally:

Switch the spark package to use "without-hadoop"
Download the full hadoop dist.
Upgrade Hadoop version to 3.2.2
Add a couple of environment variables for HADOOP
Add the necessary hadoop/azure jar files to $SPARK_HOME/jars

I am specifically interested in having it in the runtime image but could see its use in the interactive as well.

Pass packages parameter in interactive docker (jupyter)

@indy-3rdman
I've encountered some problem using interactive docker (jupyter)

To run my spark program, I have to pass a parameter like
spark-submit --packages org.apache.spark:spark-avro_2.11:2.4.6 ...

but i don't get it how to do it in jupyter.

what should i do to solve it?

[2020-11-26T00:28:42.3520910Z] [c2435164f8aa] [Error] [JvmBridge] org.apache.spark.sql.AnalysisException: Failed to find data source: avro. Avro is built-in but external data source module since Spark 2.4. Please deploy the application as per the deployment section of "Apache Avro Data Source Guide".;
	at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:666)
	at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSourceV2(DataSource.scala:733)
	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:248)
	at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:232)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.api.dotnet.DotnetBackendHandler.handleMethodCall(DotnetBackendHandler.scala:159)
	at org.apache.spark.api.dotnet.DotnetBackendHandler.$anonfun$handleBackendRequest$2(DotnetBackendHandler.scala:99)
	at org.apache.spark.api.dotnet.ThreadPool$$anon$1.run(ThreadPool.scala:34)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Issue with some methods from Microsoft.Spark.Sql while using Docker for debug from VS

Hello,
I'm getting an error during debug from Visual Studio. Docker container 3rdman/dotnet-spark:latest is up and running with port mapping -p 5567:5567.

Below sample code is failing at last First() method. Show() and Count() works fine, but First() is failing due to connection error to 127.0.0.1:40687 Similar issue is with another methods.

Any idea what I've missed? Has anyone faced similar problem?

Sample dotnet code:

          dataFrameTest.Sort().Show();
          dataFrameTest.Count();
          var firstRowFromDataFrameTest = dataFrameTest.First();

Log from dotnet app in VS:


[Info] [ConfigurationService] Using port 5567 for connection.
[Info] [JvmBridge] JvMBridge port is 5567

System.Net.Internals.SocketExceptionFactory.ExtendedSocketException: 'No connection could be made because the target machine actively refused it. 127.0.0.1:40687'

Log from Docker:

INFO DotnetRunner: Starting DotnetBackend with .
INFO DotnetBackend: The number of DotnetBackend threads is set to 10.
INFO DotnetRunner: Port number used by DotnetBackend is 5050
INFO SparkContext: Running Spark version 3.0.1
...
* .NET Backend running debug mode. Press enter to exit *

...
INFO DAGScheduler: Job 0 finished: count at NativeMethodAccessorImpl.java:0, took 1.889003 s
Exception in thread "serve-DataFrame" java.net.SocketTimeoutException: Accept timed out
	at java.net.PlainSocketImpl.socketAccept(Native Method)
	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
	at java.net.ServerSocket.implAccept(ServerSocket.java:560)
	at java.net.ServerSocket.accept(ServerSocket.java:528)
	at org.apache.spark.security.SocketAuthServer$$anon$1.run(SocketAuthServer.scala:58)

It seems it's failing within Microsoft.Spark.Sql.DataFrame class where socket.Connect is being called with parameters coming from GetConnectionInfo

        private IEnumerable<Row> GetRows(string funcName, params object[] args)
        {
            (int port, string secret, _) = GetConnectionInfo(funcName, args);
            using ISocketWrapper socket = SocketFactory.CreateSocket();
            socket.Connect(IPAddress.Loopback, port, secret);
            foreach (Row row in new RowCollector().Collect(socket))
            {
                yield return row;
            }
        }

indy-3rdman / docker-dotnet-spark Goto Github PK

docker-dotnet-spark's People

Contributors

Stargazers

Watchers

Forkers

docker-dotnet-spark's Issues

Azure Data Lake Store Support

Pass packages parameter in interactive docker (jupyter)

Issue with some methods from Microsoft.Spark.Sql while using Docker for debug from VS

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent