java.lang.StackOverflowError when using AvroBigQueryInputFormat with complex tables about hadoop-connectors HOT 11 CLOSED

@nevillelyh - Can you give an estimate of how deep the nested records go?

from hadoop-connectors.

4-5 layers, and some of them are Array or Map.

from hadoop-connectors.

This is also happening to Avro files exported via bigquery console. Should I file a bug to the BQ folks?

from hadoop-connectors.

Hi I'm hitting this issue again with Dataflow Java SDK now that it exports BigQuery as Avro.
Here are some example files that trigger the error:
gs://neville-steel-eu/avro-test/

It might be related to this:
http://stackoverflow.com/questions/24130615/circular-references-not-handled-in-avro

Can anyone look into it?

from hadoop-connectors.

@nevillelyh - Can you provide a full object name in your avro-test bucket that has public read access? I don't have permission to list objects within that bucket.

from hadoop-connectors.

While not a stack overflow, I've managed to produce bad avro files (parse errors that shouldn't be parse errors). I'm currently filing a bug with BQ, but your test case would also be nice in case they are separate bugs.

from hadoop-connectors.

@AngusDavis and I are collaborating internally and I can share with him what he needs -- pretty sure these are the same underlying cause at the end of the day.

from hadoop-connectors.

@AngusDavis I've made the avro file public: gs://neville-steel-eu/avro-debug/000000000000
@dhalperi yes that's my guess too. Thanks for looking into this!

from hadoop-connectors.

@nevillelyh - Thanks, got it.

from hadoop-connectors.

Hi Neville, I believe the underlying bug in the BigQuery Avro file generator has been fixed. Thanks for the report and the reproduction -- this was crucial to successful resolution!

from hadoop-connectors.

@dhalperi, @nevillelyh - Thanks both. Marking this issue closed.

from hadoop-connectors.