Serializing objects with iterable properties about serde HOT 3 CLOSED

So... this is an interesting problem space. I've been pondering it, and need to think aloud a bit...

I've wanted to figure out how to do lazy stream generation, like for large CSV files. I figured that would involve some way of serializing an array, rather than an object, and thus probably a new frontend than Serde. However, between this issue and the discussion in #5, I'm beginning to think that a single-property object is the way to go. That's not quite as flexible for the user (they have to define such an object), but perhaps that's OK if the overall process ends up being cleaner and has fewer moving parts.

Ideally, everything should be symmetric. However, serialization is way easier than deserialization, so there's already asymmetry for json-stream anyway. So perhaps that's not an issue.

Also, an array is iterable, so if you deserialize into an array on an iterable property, that's still completely legal. If the data set is large, then you'd want to deserialize into a generator or something, maybe, but that sounds really really hard. 😄

I... think probably the default behavior should be that custom \Traversable classes should be treated as a normal class. You're serializing its state, not its output. If you want to serialize its output, you can always add a new Exporter that attaches to \Traversable objects and calls iterator_to_array() on them. That's best handled via a custom Exporter, I'm fairly sure. (Though that may be worth documenting somewhere.)

Another concern: if you have a lazy iterable property, presumably you want it to generate lazily so you never have the whole thing in memory at once. However, if you're serializing to a string then you're going to have the whole thing in memory at some point anyway. So, do we do anything different?

Does anything change in the above if we're talking about an object with a bunch of properties, only some of which are iterable, rather than a single wrapper iterable object? I... don't know. I don't think so, but I don't know.

I think I need to noodle on this for a bit to figure out how to best handle it. There's probably some odd edge cases we haven't thought of.

I am currently between jobs, so if you are able to sponsor my time that would increase it's priority. 😸 Off the top of my head, I think I'd estimate this at around 5-8 hours of work to flesh through the details. (Though as usual, no plan survives first contact with the enemy.)

from serde.

Some preliminary work in #18. Still want to noodle a bit more, but there's good progress there.

from serde.

Resolved by #18

from serde.