Getting Java heap space error while executing it via spark-submit/spark-shell/zeppelin notebook about spark-excel HOT 6 CLOSED

Can you find out what the heap size for the master node is and if you can increase it?

from spark-excel.

I dont know how to get that.
Can u help me .

Thanks

from spark-excel.

This is the stacktrace which i am geting.

Job 20180716-132201_1796942326 is finished, status: ERROR, exception: null, result: %text java.lang.OutOfMemoryError: Java heap space
at java.io.ByteArrayOutputStream.(ByteArrayOutputStream.java:77)
at org.apache.poi.openxml4j.util.ZipInputStreamZipEntrySource$FakeZipEntry.(ZipInputStreamZipEntrySource.java:125)
at org.apache.poi.openxml4j.util.ZipInputStreamZipEntrySource.(ZipInputStreamZipEntrySource.java:56)
at org.apache.poi.openxml4j.opc.ZipPackage.(ZipPackage.java:100)
at org.apache.poi.openxml4j.opc.OPCPackage.open(OPCPackage.java:324)
at org.apache.poi.ss.usermodel.WorkbookFactory.create(WorkbookFactory.java:184)
at org.apache.poi.ss.usermodel.WorkbookFactory.create(WorkbookFactory.java:149)
at com.crealytics.spark.excel.ExcelRelation$$anonfun$openWorkbook$2.apply(ExcelRelation.scala:60)
at com.crealytics.spark.excel.ExcelRelation$$anonfun$openWorkbook$2.apply(ExcelRelation.scala:60)
at scala.Option.getOrElse(Option.scala:121)
at com.crealytics.spark.excel.ExcelRelation.openWorkbook(ExcelRelation.scala:60)
at com.crealytics.spark.excel.ExcelRelation.excerpt$lzycompute(ExcelRelation.scala:64)
at com.crealytics.spark.excel.ExcelRelation.excerpt(ExcelRelation.scala:63)
at com.crealytics.spark.excel.ExcelRelation$$anonfun$inferSchema$1.apply(ExcelRelation.scala:257)
at com.crealytics.spark.excel.ExcelRelation$$anonfun$inferSchema$1.apply(ExcelRelation.scala:256)
at scala.Option.getOrElse(Option.scala:121)
at com.crealytics.spark.excel.ExcelRelation.inferSchema(ExcelRelation.scala:256)
at com.crealytics.spark.excel.ExcelRelation.(ExcelRelation.scala:84)
at com.crealytics.spark.excel.DefaultSource.createRelation(DefaultSource.scala:36)
at com.crealytics.spark.excel.DefaultSource.createRelation(DefaultSource.scala:14)
at com.crealytics.spark.excel.DefaultSource.createRelation(DefaultSource.scala:8)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:340)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:239)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:227)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:174)
... 7 elided

from spark-excel.

Have you tried using the maxRowsInMemory option?
https://github.com/crealytics/spark-excel#create-a-dataframe-from-an-excel-file

from spark-excel.

I think that configuration worked for me.
I have executed that on small subset of data.
What should be the ideal number for maxRowsInMemory??

Thanks

from spark-excel.

It's probably not that important which exact number you use. The main difference is that using this setting switches to a streaming parser that does not keep all data in memory.
I would go for something like 100.

from spark-excel.