Allow reading only a subset of rows about spark-excel HOT 10 CLOSED

That sounds good. I will take a look at doing it that way.

from spark-excel.

Another option would be to add a virtual rowNumber column to the exported DataFrame and implement PrunedFilteredScan in addition to PrunedScan in ExcelRelation.
Then you could do something like

val preview = existingExcelDf.where(col("rowNumber") >= 5 && col("rowNumber") <= 1005)

What I'm not sure about though is if you to have to handle all filters (meaning Spark will not do any post-filtering) when you implement PrunedFilteredScan.

from spark-excel.

Ok, actually reading the documentation link I posted would have answered my question 😜
Spark will still do post-filtering, so it's OK to return false positives.
What do you think of implementing PrunedFilteredScan then and using that to do the start and end row logic?

from spark-excel.

Is anyone working on it? My project would benefit from such a feature a lot.

from spark-excel.

Just iterating over all issues again. Having this implemented should also fix #59.
@davidw76 @jaceklaskowski are you still interested in this?

from spark-excel.

Yes, for us it's more important to be able to ingest the first n rows, but for others I guess skipping n rows is more useful. So it would be good to be able to specify a range.

from spark-excel.

What I'm wondering is if we should handle the "skipping n rows because the header row is in line n+1" separately (see #65)?
Then the proposed solution with the artificial row_number column would still work and would allow additional features like

• returning the first 100 and the last 100 rows (the end of the file might be interesting, too)
• getting a column that uniquely identifies each row for free

OTOH, if all the majority of users need is a start and an end row then it would be overkill...

from spark-excel.

I just released 0.11.0. Please have a look at the corresponding CHANGELOG and the README about the new dataAddress option which supersedes sheetName, startColumn, endColumn and skipFirstRows and should hopefully cover this use-case as well.

from spark-excel.

@nightscape - I do not see skipFirstRows as one of the ExcelOptions. Do we have the feature of skipping initial rows in the excel before we start reading the header columns? Please let me know and I am in urgent need.

from spark-excel.

Check out dataAddress

from spark-excel.