alan-knight / minimal_serialization Goto Github PK

Comments

First, I'd like to comment on a couple of issues mentioned in Alen's email

For example, none of the examples I saw here included enough information to identify the type of the object,

Type information is known from context (e.g URI+request type+Http headers, etc), no need to include it. Maximum that has to be included automatically is "schema version", but it's not standard.

And it might be a good idea to sort the rules somehow in case JS hashtables have a different order than in the VM

Doesn't LinkedHashMap solve this problem? (AFAIK, all evergreen browsers seems to preserve order in objects -they are very unlikely to break it in the future)

Now, some thoughts from myself.

The way it's implemented in serialization package (through Rules) is not very efficient: we still have to deal with intermediate maps. Library provides large API, with many classes, special cases etc,
which, IMO, is too heavyweight for the task at hand.

For a simplified serialization in question, I'd like to propose a very different and simple design, which nonetheless makes possible to handle variety of special cases mentioned in mailing list discussion, with performance comparable with popular java serializers.

Design is based on MapViews (explained below) and makes use of two classes: TypeMetadata and FieldMetadata.

The goal of MapView is to create illusion for serializaer that it works with regular maps/lists, while in fact it will work with objects (hidden in MapView implementation)

First, let's define auxiliary classes.

TypeMetadata

TypeMetadata is generated for each type used in serialization, and includes the following:

• type name (e.g. Person; List; Map<String, Person>)

  Note that when transformer encounters generic type like List, it automatically generates TypeMetadata for Person, too)

• constructor: function that returns new object of this type, e.g. ()=> new List

  constructor in some cases may return the object of type different from "this type" - e.g. for Uint8List, it will return new List, because serializer needs method "add"; final conversion is handled by encoder/decoder (see below)

• child type - e.g. for List child type is "Person". Null if not applicable

• category: one of "struct", "list", "map", "primitive"

  The need for this field may not be obvious, but as it turns out, it makes things simpler for implementation.

The following two function are used for special types that need to be encoded/decoded (e.g. DateTime should be encoded/decoded to/from String and other basic types. Uint8List should be decoded from List). For the purposes of discussion, assume our type is DateTime, for which we have no standard support in JSON

• encoder: function that takes DateTime and returns String representation of it
• decoder: function that takes a String, parses it and returns DateTime

Basic types here are those directly supported by JSON - they are not limited to String/int, it can be Lists or Maps too. E.g., for Uint8List, de-serializer will be lead to believe our real type is List, but when placed into object, this List gets converted to Uint8List, due to decoder.

All generated type metadata gets assembled in a single place - see "type catalog" section below.

FieldMetadata

FieldMetadata is generated for each field (attribute) of struct. E.g. for Person with (firstName, lastName), we will generate two FieldMetadata objects. Included in FieldMetadata are:

• attribute name (e.g. "yearOfBirth")
• type name (if generics are used, should include them, e.g. List)
• setter functon for attribute, e.g. for "yearOfBirth", it will look like `(obj, value) =>
• getter function for attribute, e.g. (obj)=>obj.yearOfBirth

Together, these 4 parameters per attribute + 5 parameters from TypeMetadata provide enough information for a variety of serialization formats.

MapView and Serializers

MapView is a base class that provides [] and []= and other standard Map methods for serializable classes, based on generated metadata. E.g. for Person class, transformer will generate (simplified; details may vary):

class PersonJsonSerializer extends MapView {
  final static _fieldMetadata=[
     new FieldMetadata("firstName", "String", (obj,v)=>obj.firstName=v, (obj)=>obj.firstName),
     // etc.
  ];
  PersonJsonSerializer(Person p):super(p, _fieldMetadata) {};

  String stringify() { ... calls JSON serializer }
  static Person parse(String s) { ... calls JSON deserializer }

}  

The idea of MapView is obvious: if all our data were represented as Maps and Lists and primitives, we woould already know how to solve the problem: standard JSON package works very well. To address the case of structs, let's reduce the problem to the one we already know how to solve - by pretending our object is a map (but without creating real map and copying the entire content there)

To use generated serializers, user simply calls

var person = PersonJsonSerializer.parse(jsonAsString); // string to object
var str = new PersonJsonSerializer(personObject).stringify(); // object to string

TypeCatalog

TypeCatalog is a class that collects all information about generated instances of TypeMetadata (it's just a map from type name to TypeMetadata instance).

To support custom types (such as DateTime, Uint8List, Point etc), we need to include appropriate TypeMetadata into catalog. Library can come with built-in TypeMetadata for some popular classes. For other things, user has to write TypeMetadata manually and include into catalog (how exactly - TBD). This can cover many corner cases, but probably not all of them. Some of the "missing features" can be added, if they are simple enough and generic. For the rest, no solution is provided.

Restrictions on serializable classes

As we discussed, design targets primarily classes with all public fields (structs), but this restriction might be too harsh. In fact, we can relax it without making our mechanism more complicated,
by just saying that when you annotate any class with @serializable, transformer will look for public fields only, ignoring the rest - so class may have methods and private fields, too, but they are all ignored by serialization mechanism. Public no-argument constructor is a must; inheritance can be supported, too, if transformer is smart enough to figure out public fields of base class. (Not sure how useful this would be)

Another small issue: what to do when JSON string contains some attributes not declared in the class? It would be good if we could call methodNotFound, as if setter was invoked for an attribute, but we can't simulate this call (we can't create Invocation object - everything is private there). We have to either introduce special method like fieldNotFound, or invoke (from serializer) some fake method with predefined name, so methodNotFound will be automatically called.

Implementation

Implementation is rather straightforward (with the possible exception of transformer itself). I have pigeon_map that does most of the same things, but I made a mistake of introducing "special kind of structs", instead of MapViews. Anyway, it's easy to adapt, I can do it if design is approved (or just forget about it otherwise).

My current implementation makes use of json package (which was abandoned by dart in favor of convert and not maintained any more). Whether it can be made a part of "convert" library is not clear (I don't know enough about it).

As an aside, json serializer (in both packages) is very slow - at least by 2 times slower than expected, based on average ratio between serialization/parsing, according to this table. This is certainly fixable.

To simplify, can we add the example into here? One less package to manage.

Given a choice between 'serialization' and 'minimal_serialization'... not clear what the relationship is. Do they compete? Does one rely on the other?

Suggest that we move this transformer into the serialization package. One stop shop!