xomachine / nesm Goto Github PK

Hi,
When declaring a field as a set of an enum the size of this field is wrongly calculated. The problem comes from here.

This is invalid with enumtype.maxcount values such as 8, 16 and so on. I think it should be more like this:

result = context.genBasic(int(enumtype.maxcount + 7) div 8)

Test case:

Example:
serializable:
  static:
    type
      Flag* = enum
        Bit1 = 0
        Bit8 = 7

      Obj* = object
        flags: set[Flag]

echo Obj.size

This should print 1 but currently prints 2.

I'm trying to generate docs for https://github.com/disruptek/swayipc but things go haywire, seemingly as a result of nesm. NESM works absolutely perfectly otherwise, so thanks for that! 😄

My use of nesm seems pretty trivial to me; you can find it here:
https://github.com/disruptek/swayipc/blob/master/swayipc.nim#L260

This is the (first) error I get when I attempt to nim doc swayipc.nim:

/.nimble/pkgs/nesm-0.4.5/nesm/generator.nim(130, 12) Error: Type Header is not a basic type nor a complex type under 'serializable' block!

Things run right off the rails from there...

Any ideas? Should I expect nim doc to work?

Looks like a great library, thanks!
But I can't get it to work, example:

import nesm

serializable:
  type Ball = object
    weight: float32
    diameter: int32
    isHollow: bool

let a = Ball(weight: 5.0, diameter: 3, isHollow: true)
echo a.serialize()

It throws an error like:

gaga.nim(3, 1) template/generic instantiation from here
/home/tiber/.nimble/pkgs/nesm-0.3.1/nesm/basics.nim(40, 3) Error: undeclared identifier: 'weight'

I'm experimenting with the format described here: http://yann.lecun.com/exdb/mnist/

Everything bigEndian
TRAINING SET IMAGE FILE (train-images-idx3-ubyte):

[offset] [type]          [value]          [description] 
0000     32 bit integer  0x00000803(2051) magic number 
0004     32 bit integer  60000            number of images 
0008     32 bit integer  28               number of rows 
0012     32 bit integer  28               number of columns 
0016     unsigned byte   ??               pixel 
0017     unsigned byte   ??               pixel 
........ 
xxxx     unsigned byte   ??               pixel

The following compiles on 0.17.2 but not on devel

import nesm

type MNIST_imgs* = object
  magic_number*: int32
  n_imgs*: int32
  n_rows*: int32
  n_cols*: int32
  data*: seq[uint8]

toSerializable(MNIST_imgs, endian: bigEndian)

I get:

serialization_nesm.nim(10, 15) template/generic instantiation from here
$USER/.nimble/pkgs/nesm-0.3.1/nesm/generator.nim(100, 10) Error: undeclared identifier: 'data'

here's an example that's not (AFAICT) supported.

import nesm

serializable:
  type O = object
    when defined(debug):
      qname: string

    other: int

hi.

thanks for NESM. i think it is a good macro.

However, i wonder that someday i can use in js backend

thank you

https://github.com/xomachine/NESM/blob/master/nesm/enums.nim checks whether an integer can be converted to a valid enum by comparing the stringifying of the enum with "(invalid data!)". However, ARC/ORC changes its behavior. Instantiating invalid enums now gives an empty string.

Solution:

• add logics of checking empty strings for ARC/ORC
• following the solution from https://forum.nim-lang.org/t/8188

ref nim-lang/Nim#19006
ref nim-lang/Nim#19972

When building with -d:release all assert() calls are optimized out by the compiler. You should use doAssert() instead or move the calls to stream.readData() out of the assert() call

I propose to add a pragma, that can be attached to fields, and that causes those fields to be skipped completely during de-/serialization.

When I attempt to declare a serializable seq of a dynamic variant type, a type mismatch error occurs and it fails to compile.

Simplest example code:

import NESM

serializable:
    type
      VariantType = object
        case a: uint8
        of 1:
          b: string
        else:
          c: uint8
      PeriodicContainer = object
        a: seq[VariantType]

Compilation fails with the following message. Please ignore the backtrace paths. (This is from a unit test that I added with the above code.)

./NESM/tests/complex.nim(7, 7) template/generic instantiation of `suite` from here
./NESM/tests/complex.nim(91, 8) template/generic instantiation of `test` from here
./NESM/tests/complex.nim(92, 5) template/generic instantiation of `serializable` from here
./NESM/nesm/periodic.nim(123, 9) Error: type mismatch: got <Natural, void>
but expected one of:
proc `+=`[T: SomeInteger](x: var T; y: T)
  first type mismatch at position: 2
  required type for y: T: SomeInteger
  but expression '
case obj_436208960.a[index].a
of 1:
  result += 0 + (4 + len(obj_436208960.a[index].b) * 1)
else:
  result += 1
result += 1' is of type: void

I get errors when I attempt to use object variants with NESM. Specifically:

assignment to discriminant changes object branch; compile with -d:nimOldCaseObjects for a transition period [FieldDefect]

I have worked around this for now by adding the define, but that affects all of the code in the project. Nim now has https://nim-lang.org/docs/manual.html#types-cast-uncheckedassign to address this as of version 1.6.

I'd like to propose the use of the uncheckedAssign macro.

Thanks for the very useful library. NESM looks like the ideal candidate to create and parse network packets and implement protocols.
Would you consider supporting network/host endianness?
Also serialize() could call an optional Scapy-like helper that could set default and/or random values and fill in checksums before serialization. Thanks!

This snippet:

import nesm

serializable:
    type MyType = object
        kind: byte
        size: uint32
        data: seq[byte] as {size: {}.size}

Generates the following when compiling with nim c -d:debug foo.nim:

proc deserialize(a167008: typedesc[MyType]; thestream: Stream): MyType =
  discard
  assert(1 ==
      thestream.readData(result.kind.unsafeAddr, 1),
         "Stream was not provided enough data")
  assert(4 ==
      thestream.readData(result.size.unsafeAddr, 4),
         "Stream was not provided enough data")
  result.data = newSeq[byte](result.size)
  var length167004: int32
  assert(4 ==
      thestream.readData(length167004.unsafeAddr, 4),
         "Stream was not provided enough data")
  result.data = newSeq[byte](length167004)
  if result.data.len() > 0:
    assert(result.data.len() * 1 ==
        thestream.readData(result.data[0].unsafeAddr, result.data.len() *
        1), "Stream was not provided enough data")

So the code reads the size field as it should and uses it to allocate a sequence. But it also reads another size and allocates the sequence a second time. This second length is probably what is generated when not having the custom size specified and in this case it's not wanted because the length is read from the user specified field.

I tried this with latest (devel) Nim and installed nesm with nimble.

I've found a strange behaviour that I can't understand.
An object of the SaveGame type serializes without trouble, and I can save it to and properly deserialize it from a file:

serializable:
  type
    SavedSector* = object
      name*: string
      id*: uint8

    SaveGame = object
      sectors: seq[SavedSector]

However, if I remove the name field from SavedSector,

serializable:
  type
    SavedSector* = object
      id*: uint8

    SaveGame = object
      sectors: seq[SavedSector]

the program will no longer compile, instead giving an error:

but expected one of:
proc `*`[T](x, y: set[T]): set[T]
  first type mismatch at position: 1
  required type: set[T]
  but expression 'len(obj274023.sectors)' is of type: int
proc `*`(x, y: int32): int32
  first type mismatch at position: 1
  required type: int32
  but expression 'len(obj274023.sectors)' is of type: int
proc `*`(x, y: float): float
  first type mismatch at position: 1
  required type: float
  but expression 'len(obj274023.sectors)' is of type: int
proc `*`(x, y: int16): int16
  first type mismatch at position: 1
  required type: int16
  but expression 'len(obj274023.sectors)' is of type: int
proc `*`(x, y: int): int
  first type mismatch at position: 2
  required type: int
  but expression '
result += 1
result += 0' is of type: void
proc `*`(x, y: int64): int64
  first type mismatch at position: 2
  required type: int64
  but expression '
result += 1
result += 0' is of type: void
proc `*`[T: SomeUnsignedInt](x, y: T): T
  first type mismatch at position: 1
  required type: T: SomeUnsignedInt
  but expression 'len(obj274023.sectors)' is of type: int
proc `*`(x, y: float32): float32
  first type mismatch at position: 1
  required type: float32
  but expression 'len(obj274023.sectors)' is of type: int
proc `*`(x, y: int8): int8
  first type mismatch at position: 1
  required type: int8
  but expression 'len(obj274023.sectors)' is of type: int

expression: len(obj274023.sectors) *
  result += 1
  result += 0

Basically, it seems like SavedSector has to have a string field for some reason, but I don't understand why that would be.

Here are some enhancements that I feel would greatly improve the quality of life of those implementing certain protocols. It would be greatly beneficial if we were able to set the type used to prefix a string/seq natively without making a new type custom type. An example of how this could be implemented would be like this:

serializable:
  type
    CoolStrings* = object
      set: {endian: bigEndian, strLength: int16, seqLength:  int64}
      strings*: seq[string]

This would provide the user with a level of flexibility that is honestly so nice due to the simplicity of toggling said option.

Another feature which would be nice would be the ability to set the default value of a field, at least for me I'm re-implementing a certain games protocol and some packets might have additional bytes at the end which might get parsed into their respective variables. An example of this would be this:

serializable:
  type
    SpecialPacketWithFieldsThatMightExist* = object
      name*: string
      specialfield*: int32 as {default: 40}

This would give us the flexibility to give fields a default value other than 0 as is currently implemented in nim. This would allow us to parse packets which have the possibility of containing optional/additional data.

The comments I leave for objects are not present in the generated documentation.

Example:

import NESM
serializable:
  type MyType = object
    ## Documentation on MyType
    field1: string  ## Documentation on field1

When I run nim doc on this, I expect to see both nimdoc comments, but only the field comment is present.