[feature request] validator for codec when packing the object about lumos HOT 5 CLOSED

Idea for Unpack

When an error occoured on unpack process, a tree view of that tells developer how program works can take more useful information for debugging.

Here I will give some examples.

Example 1

The type schema is vector Bytes <byte>

# there are 6 items
06 00 00 00
# six items
23 01 00 00, 56 04 00 00, 90 78 00 00, 0a 00 00 00, bc 00 00    , ef 0d 00 00
#                                                                                               ↑
#                                                                         this section is missing one byte

The error message is

vector<Byte>
    |-- _size: 6 (06 00 00 00)
    |-- [0]: 0x123  (23 01 00 00)
    |-- [1]: 0x456  (56 04 00 00)
    |-- [2]: 0x7890  (90 78 00 00)
    |-- [3]: 0xa (0a 00 00 00)
    |-- [4]: 0xef0000bc (bc 00 00 ef)
    |-- [5]: Failed, try parse 4 bytes but got 3 bytes: 0d 00 00

When the developer see this error message, it's expected result is [0x123, 0x456, 0x7890, 0xa, 0xbc, 0xdef].
So he can find the bytes in vec[4] position have missed a 00 byte.

Consider a more complex condition.

The type shema is table MixedType { f1: Bytes, f2: byte, f3: Uint32, f4: Byte3, f5: vec byte }
And the input is

# the full size is 43 (0x2b) bytes
2b 00 00 00
# five offsets (20 bytes in total)
18 00 00 00, 1c 00 00 00, 1d 00 00 00, 21 00 00 00, 24 00 00 00
# five items (19 bytes in total)
# the full size is 43 (0x2b) bytes
2b 00 00 00
# five offsets (20 bytes in total)
18 00 00 00, 1c 00 00 00, 1d 00 00 00, 21 00 00 00, 24 00 00 00
# five items (19 bytes in total)
00 00 00 00
ab
23 01 00 00
45 67 89
03 00 00 00, ab ef
#                       ↑
#               missing a byte in vector

The error message is

Table MixedType
  |- _size: 0x2b(2b 00 00 00)
  |- _offsets: 0x18, 0x1c, 0x1d, 0x21, 0x24(18 00 00 00, 1c 00 00 00, 1d 00 00 00, 21 00 00 00, 24 00 00 00) 
  |- f1(vec byte):
            |----  _length: 0x0(00 00 00 00)
  |-  f2(byte): 0xab(ab)
  |-  f3(Uint32): 0x123(23 01 00 00)
  |-  f4(Byte3): 0x456789(45 67 89)
  |-  f5(vec bytes)
             |- _length: 0x3(03 00 00 00)
             |- [0]: 0xab(ab)
             |- [1]: 0xef(ef)
             |- [2]: Failed, try parse a byte but got nothing(meet end of input)

Although sometimes the position triggers error is not the root cause, Like case in example 1, the missing bytes is not in the end, but developers can still fix their bug by the error message.

implementation of pack/unpack

in current codec structure, we need a extra context. How to pass it through various type codec method?

It have some approaches.

Pass context as parameter

we should change the pack/unpack function parameters.

  pack: (packable: Packable, context?: CodecContext) => Packed;
  unpack: (unpackable: Unpackable?: CodecContext) => Unpacked;

And we hide the CodecContext constructor(just don't export it in module). then the developer can only pass the first parameter.

• Pros: Easy to implement
• Cons: take confuse to developers. they will think: Why there is the second parameter that I can't use?

pass the context by this keywords

  pack: (this: CodecContext, packable: Packable, context?: CodecContext) => Packed;
  unpack: (this: CodecContext, unpackable: Unpackable?: CodecContext) => Unpacked;

and when we called subtype codec methods, we should do some extra work.

items.map((item, index) => {
   return this.context.apply(index, item.codec, buf.slice(0, 114514))
})

• Pros: codec API user friendly. Nothing changes on the API
• Cons: TBD

CodecContext

This class is a rough draft. Maybe we need more discussion.

export class CodecContext<Packed = any, Unpacked = any> {
  private subcontextKeyMap: WeakMap<
    CodecContext,
    string | number
  > = new WeakMap();
  private subcontexts: CodecContext[] = [];
  public type:
    | "Uint8"
    | "Uint16LE"
    | "Uint16BE"
    | "table"
    | "array"
    | "option"
    | "struct"
    | "vector"
    | "union"
    | "unknown" = "unknown";
  public failed = false;

  apply(
    path: string,
    fn: Codec<Packed, Unpacked>["pack"],
    packable: Unpacked
  ): Packed;
  apply(
    path: string,
    fn: Codec<Packed, Unpacked>["unpack"],
    unpackable: Packed
  ): Unpacked;
  apply(path: string, fn: any, input: any): any {
    const subcontext = this.createSubcontext(path);
    return (fn as Function).call(subcontext, input);
  }

  createSubcontext(path: string) {
    const subcontext = new CodecContext();
    this.subcontexts.push(subcontext);
    this.subcontextKeyMap.set(subcontext, path);
    return subcontext;
  }

  get lastSubcontext(): CodecContext | undefined {
    return this.subcontexts[this.subcontexts.length - 1];
  }

  get codecPackError(): Error | undefined {
    let context: CodecContext | undefined = this;
    let errorPaths: (string | number)[] = [];
    while (context !== undefined) {
      const lastSubcontext: CodecContext | undefined = context.lastSubcontext;
      if (lastSubcontext !== undefined) {
        errorPaths.push(context.subcontextKeyMap.get(lastSubcontext)!);
      } else if (!context.failed) {
        return;
      }
      context = lastSubcontext;
    }
    const errorPath = errorPaths.reduce(
      (acc, cur) => (acc + typeof cur === "number" ? `[${cur}]` : `.${cur}`),
      ""
    );

    return new Error(
      `${errorPath} is incorrect.
Expected: ${this.type}
`
    );
  }
}

from lumos.

Idea for Unpack

Thx for such a detailed reply for unpack enhancement design

I believe the byte layout view in console is helpful for understanding molecule.

But in all of the above scenarios, we presuppose that the content of unpack is PARTIAL incorrect, the molecule header is correct. The Most common error we make is actually exactly "unpacking an unknown binary data source via a Codec". For example

MyDataCodecForA.unpack(cellA.data) // correct
MyDataCodecForA.unpack(cellB.data) // incorrect

from lumos.

Seems we can not display the actual type in error message.
but tell user the expected type is not difficult.
Like this error message.

Expect type Uint8BE at arrayField[2] but got error: Value must be between 0 and 255, but got 1131796

from lumos.

Expect type Uint8BE at arrayField[2] but got error: Value must be between 0 and 255, but got 1131796

got it, we can display the error message and the path, it is enough for troubleshooting

from lumos.

Seems we can not display the actual type in error message. but tell user the expected type is not difficult. Like this error message.

Expect type Uint8BE at arrayField[2] but got error: Value must be between 0 and 255, but got 1131796

I create a new PR, but it is not include unpack implementation.
#403

from lumos.