Suppose I have a table of offsets to some child objects, given that the offsets in the table are relative to the parent:

    class BlobIndex < BinData::Record
        uint32be :type
        uint32be :offs
    end

    class Blob < BinData::Record; end;

    class SuperBlob < BinData::Record
        uint32be :len
        array :blob_offsets, :type => :blob_index, :initial_length => :len
        # Blob objects follow, according to the offsets, the offset of the first blob will be equal to the length of the blob_offsets array in bytes
    end

I couldn't find how can I make such structure into an array of Blob records. I have also tried:

array :blob_offsets, :type => :blob_index, :initial_length => :len
array :blobs, :initial_length => :len do
   blob :adjust_offset => lambda { offset[index].offs }
end

Which didn't work properly, probably because the offset is adjusted related to the start of the IO, not the start of SuperBlob object. Is it possible to rebase the offset calculation to a given field in the SuperBlob object. I'm trying to port the following code from Python (notice it has a feature called Anchor: https://github.com/comex/cs/blob/master/macho_cs.py)

Binary Data and CRC checking is a likely combination ?

Is there a recommended approach to having a Record (with potentially nested records managed off a selection and choice) to have a structure that will read from a two byte synchronization point, the length of declarative structure, followed by a 32bit CRC32 check-sum, that is based on all the data read inclusive of the synchronization bytes through to the last byte prior to the CRC element ?

I've looked a Zlip::crc32 but am not sure exactly how to call it, and is this something that could be overloaded into a :check_value ?

This project is very Good!

Is it possible to change parameters of a record field after declaration, or in a subclass? Ex.:

class Foo < BinData::Record
  uint8 :type, :assert => ->{(1..5).member? value}
end

class FooType3 < Foo
  set_params? :type, :value => 3
end

In Foo, type can be any number from 1 to 5, but in FooType3, it's explicitly 3.

It seem that the base ruby types have many features that that are not present in the bindata primities.

If say I parse some binary data, and say define a structure that has an element of Uint8le and I copy this value put into another Ruby class (MyClass) member, the class member is of class Unit8le not Fixnum.

In an example where I then try and for example try and convert an instance of MyClass to_json I get a run-time error as that implementation is based on the standard primitive types, such as Integer, String, Float, Bignum, Fixnum, Boolean etc. Or their ability to support to_hash/to_h or be able to retrieve it's instance variables.

In some cases there are multiple option, perhaps the preference can be defined in the DSL ?

Is it fair to say based of the underlying bindata types that it should be able to automatically coerce values to these base types.

My current work around is to add to_i or to_s etc. as applicable as I copy the values out but this is far from DRY.

I'd argue that once parsed, the data should fall into the classic Ruby data type framework at a primitive level to ensure support from the rest of the run-time ?

If a record is defined such as

class MyRecord < ACEARecord

  CHOICES = {
    1 =>  RecordA,
    12 => RecordB,
    20 => RecordC,        
    # ... etc.
  }

  uint8  :message_type, :initial_value => 12
  choice :message_body, :selection => :message_type,
              :choices => CHOICES
end

If one instantiates an instance of MyRecord and queries the type of message body:

a = MyRecord.new

a.message_body.class
=> BinData::Choice

What would be preferable (in the case of the initial_value) is:

=> RecordB

Looking at the code the instantiate code seems to perform a new but there does not seem to be a method for accessing it ?

The end goal would be to have behavior based of the class of the selection such that the following code would work:

case a.message_body
  when RecordA
    # do something based of the context of RecordA
  when RecordB
    # do something based of the context of RecordB
  when RecordC
    # do something based of the context of RecordC

  # ... etc
end

Related to #40.

It is really hard to tell the difference between :length and :read_length for fixed strings. :length works with :asserted_value but when I decided to no longer assert on that field, I also had to switch to :read_length as "params 'length' and 'value' are mutually exclusive". As a n00b with you library, I don't understand how "fixed length string" is never one fixed length for all purposes.

Hello,
Are virtual fields ok to use now, or should I wait for a future version to enable it by default?

For example, on https://github.com/dmendel/bindata/wiki/Records the bullet-list links to https://github.com/dmendel/bindata/wiki/Records#specifying-default-endian but the anchor is https://github.com/dmendel/bindata/wiki/Records#wiki-specifying-default-endian.

I'm writing a file format using your gem. I have a 3 classes, each nested and inheriting from the BinData::Record class.

When I reference the module in anyway, for example, I was doing FileFormat.constants just as a test - I get a TypeError from bindata.

unknown type 'uint32' in BinData::Array

Here's a basic overview, with fake class names:

module FileFormat
class Header < BinData::Record
  class TypeHeader < BinData::Record
    class TypeABody < BinData::Record
      def version
        parent.parent.version # grab Header version
      end

      # some fields

      uint8 :thing_count
      array :things, initial_length: :thing_count do
        stringz     :string_field
        uint32  :uint32_field, onlyif: lambda { version >= 1 } # this is where the errors happening
      end

    end

    class TypeBBody < BinData::Record
      # some fields
    end

    uint8 :type
    choice :body, selection: :type, choices: [:type_a_body, :type_b_body]
  end

  uint32 :magic
  uint16 :version
  uint32 :file_size

  array  :type_headers, type: :type_header, read_until: :eof
end
end

Would this be happening because of my nested classes? It works when i comment them out and only define the Header class.

Hello. I am the author of trema/pio, a pure Ruby packet parser and generator built on BinData.

Pio: https://github.com/trema/pio

I've found the BinData's examples/ very useful when writing Pio, but I think it would be helpful for other users if BinData could provide some pointers to more practical, real world use-cases.

So, if possible, please add links to Pio in README or Wiki pages. Thanks in advance.

It would be nice to read an int and look it up in an array of symbols like this:

enum :status, :values => [:pending, :successful, :failed]

such that the read and write methods work transparently and also check the validity of the enum field.

According to https://github.com/dmendel/bindata/wiki/PrimitiveTypes#asserted_value , I can use value and assert together in cases where value is a constant and assert is a lambda but I get the error "params 'value' and 'assert' are mutually exclusive" instead.

Hi

I realize I'm probably doing something wrong...

I have an unsigned integer that is packed to be a direction (0-360 Degrees)

class MyDirection < BinData::Primitive
  Uint16le :val

  def get
    (self.val * 360.0 / 65536.0)
  end

  def set(v)
    self.val = (v * 65536.0 / 360.0).round
  end

end

However is I read data I seem to be getting negative values.

When I read the value from the stream if get -15541 but should get 49995. The data bytes are 0x4B and 0xC3.

I'm expecting the 'U' modifier in the Uint16le definition of :val to treat the data as unsigned ?

All help greatly appreciated.

I've been getting the following exception while trying to decode a big structure I have defined:

/usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:279:in instantiate_obj_at': methodmethod_missing' called on hidden T_DATA object (0x00000002660e20 flags=0xc klass=0x0) (NotImplementedError)

I've been decoding data of this very structure many many times (for a couple of years) with no issue, and I only seem to notice this exception whenever I try to decode a large number of files (1000+) in a row. Adding a delay of about 10-15 seconds between each decode seems to workaround that issue, although obviously I'd prefer to just perform them as quickly as possible :)

This is on ruby 1.9.3p0 (2011-10-30 revision 33570) [x86_64-linux].

/usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:279:in instantiate_obj_at': methodmethod_missing' called on hidden T_DATA object (0x00000002660e20 flags=0xc klass=0x0) (NotImplementedError)
from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:267:in find_obj_for_name' from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:164:inblock in snapshot'
from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:163:in each' from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:163:insnapshot'
from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/array.rb:108:in block in snapshot' from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/array.rb:108:incollect'
from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/array.rb:108:in snapshot' from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:165:inblock in snapshot'
from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:163:in each' from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:163:insnapshot'
from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/array.rb:108:in block in snapshot' from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/array.rb:108:incollect'
from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/array.rb:108:in snapshot' from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:165:inblock in snapshot'
from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:163:in each' from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:163:insnapshot'
from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/array.rb:108:in block in snapshot' from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/array.rb:108:incollect'
from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/array.rb:108:in snapshot' from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:165:inblock in snapshot'
from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:163:in each' from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:163:insnapshot'
from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:165:in block in snapshot' from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:163:ineach'
from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:163:in snapshot' from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:165:inblock in snapshot'
from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:163:in each' from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:163:insnapshot'
from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/array.rb:108:in block in snapshot' from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/array.rb:108:incollect'
from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/array.rb:108:in snapshot' from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:165:inblock in snapshot'
from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:163:in each' from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:163:insnapshot'
from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/choice.rb:137:in snapshot' from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:165:inblock in snapshot'
from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:163:in each' from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:163:insnapshot'

Hi,
Firts of all, thank you for this gem, it seems that you have thought to everything.

To customize my test output, i did this way

require 'bindata'

class TodSnapshot
  def initialize(value)
    @value = value
  end

  def inspect
    "#{@value}(#{"0x%08x" % @value})"
  end
end

class Tod < BinData::Primitive
  endian :big
  uint32 :val

  def get ; self.val; end
  def set(v) ; self.val = v ; end

  def snapshot
    TodSnapshot.new(self.val)
  end
end

class Pouet < BinData::Record
  endian :big
  tod    :sent_tod
end

pouet = Pouet.new.read("\x01\xab\x23\xcd")

require 'pp'
pp pouet.snapshot

My output is the expected one:

{:sent_tod=>27993037(0x01ab23cd)}

And my question is the following : is there any more "direct" way to do this ?

Thanks in advance,

Jérôme

I just noticed your commit disabling Ruby 2.1.0 in Travis CI. Just wanted to chime in with something I recently discovered: the allow_failures option. Consider using it instead of explicitly disabling versions of Ruby you know may fail. E.g. see my code here https://github.com/olerass/sourceless/blob/master/.travis.yml

I would like a generic PascalString type which allows me to handle strings with different length field widths in a DRY way like this:

U16BEPascalString = GenericPascalString.new :length_type => :uint16be
U16BEPascalString.read "\000\004ABCD"
U32BEPascalString = GenericPascalString.new :length_type => :uint32be
U32BEPascalString.read "\000\000\000\004ABCD"

I tried

class GenericPascalString < BinData::Primitive
  mandatory_parameter :length_type
  length_type :len
  string :data, :read_length => :len
end

and also

class GenericPascalString < BinData::Primitive
  mandatory_parameter :length_type
  array :len, :type => :length_type, :initial_length => 1
  string :data, :read_length => lambda {len[0]}
end

But with both attempts, I get "TypeError: unknown type 'length_type' in GenericPascalString". Is there a way to accomplish this?

I'm trying to create two records which refer to each other by type, and I was running into an issue. I've laid out the code with two less than desirable solutions, and was wondering if I was doing something wrong, or if this is the way to go about it.

https://gist.github.com/nicklink483/83f8c236aa991760aab6

Any help or insight would be appreciated.

• Dominic

If a declarative structure defines a length (as read for the stream) and then an array with an initial length set to that length, I want to be able to confirm once the array has been read that the length is actually conforming to the length supplied in the stream.

I'm trying to read this response of a call to my Yubikey:

"\x72\x03foo\x72\x03bar\x90\x00"

into the following BinData record:

class Entry < BinData::Record
  uint8 :name_list_tag, assert: 0x72
  uint8 :name_length
  string :name, read_length: :name_length
end

class Response < BinData::Record
  array :codes, type: :entry
end

As you can see, there's no leading byte to indicate how many codes are returned, so I can't set the initial_length of the array, and there are trailing bytes which are unrelated, so I can't use :read_until => :eof.

I'm thinking that a nextbyte variable should be exposed to the lambda passed to read_until, so that I can do something like array :codes, type: :entry, read_until: lambda { nextbyte != 0x72 }.

From a quick look I'd need to define IO#peekbytes(n) and add :nextbyte => self.peekbytes(1) to the variables Hash inside ReadUntilPlugin#do_read(io).

Thoughts?

When reading in larger files, it would be nifty to have a way to print progress to the terminal.

Is there a way to define a bit field record that is Signed, for a field that has a size that is not divisible by 8?

For example, to create a Signed 16-bit field, I can do:

 # Signed 16-bit (bit-aligned)
 class SBIT16 < BinData::Int16be
   bit_aligned
 end

However, I also need to support a Signed 6-bit field, but cannot use the same approach , since there is no Int6be (and trying to define something like it returns: RuntimeError: nbits must be divisible by 8).

Thanks.

Hello!
I think it will be useful to write a wiki about the analogue of the sizeof(field_name) function.
This can be used when you want to unload a set of values ​​in a string with no conversion to the array.
Something like:

class SomeClass < BinData::Record
  endian :little
  # get the s_count elements of sizeof(uint16_t)
  uint16 :s_count
  string :samples, :read_length => lambda { s_count * s_count.do_num_bytes }
  ...
end

Of course this example works only if the dimension s_count coincides with the dimension of each element.
Maybe think of something like uint16.size?

I wrote some test code to learn bindata and I got an error I wasn't expecting.
I had assumed, as the documentation had implied for value (and therefore implied for asserted_value), that the identity string was a fixed length constant of three letters for the purposes of reading, writing, and asserting. Writing went great but reading resulted in a length of zero. I know adding the "read_length" attirbute will fix the issue but it took a trace of the code to figure that out. Doesn't seem intuitive. I would be happy to write a patch if you agree that this is not the expected behavior. Thank you for your review. Here's the code:

require 'bindata'

class TestFormat < BinData::Record
  endian :big
  string :identity, :asserted_value => "xyz"
  uint8 :version, :initial_value => 1
end

test1 = TestFormat.new
test1.version = 65
File.open 'test.bin', 'wb' do |file|
  test1.write( file )
end

test2 = TestFormat.new
File.open 'test.bin', 'rb' do |file|
  test2.read( file )
end

puts test2

Hi,

Love the library. I've been playing with the :adjust_offset option and ran into an issue. The issue may be with my understanding or the code. :) I'll demonstrate with an example

require 'bindata'
io = ["6B7963680001000000000010000000140000000000006A640000000B"].pack "H*"

class KeychainFile < BinData::Record
  endian :big
  string :magic, length: 4, asserted_value: 'kych'
  string :padding, read_length: 8
  uint32 :schema_offset # in this example, the value is 20
  uint32 :table_count, adjust_offset: lambda { schema_offset + 4 }
end

kf = KeychainFile.read io
kf.schema_offset.abs_offset # is 12, as expected
kf.table_count.abs_offset # is 16, where i would expect 24

It appears that abs_offset does not take into account the value returned by adjust_offset. The value of kf.table_count is located at offset 24 and my understanding of abs_offset is that it returns the location within the IO that houses the data. Is this a correct understanding? If so, I think there may be a bug but I am not clever enough to fix it :)

I need to calculate a checksum over the binary data below (from version to content) and then include it in the crc field. How would I do this in a BinData way?

class Heartbeat < BinData::Record
      endian :big

      uint8 :version, :value => 0
      uint16 :seq_num, :value => 99
      uint32 :id, :value => 0
      uint8 :cmd_type, :value => 1
      uint16 :timeout, :value => 65535
      string :content, :value => 'Heartbeat'
      uint16 :crc
    end

I have some records which are like this:

class CONSTANT_Methodref_info < BinData::Record
  endian  :big
  uint8   :tag
  uint16  :class_index
  uint16  :name_and_type_index
end

class CONSTANT_NameAndType_info < BinData::Record
  endian  :big
  uint8   :tag
  uint16  :name_index
  uint16  :descriptor_index
end

As you can see, they always share a uint8 :tag which decides which record you're dealing with. I think this comes up a lot in id+length+data structures.

I tried to model this with bindata:

class Cp_info < BinData::Record
  endian  :big
  uint8   :tag
  choice  :payload, :selection => :tag
    :constant_methodref_info 10
    :constant_name_and_type_info 12
  end
end

The problem is, :tag is already outside the structure but also defined in the structure. I know that I could remove it from inside each structure, but I would like to maintain similarity with the structures as they were originally defined. This is sort of a type of struct-based inheritance, so I wondered if there was a more elegant way to do it.

My own DSL's syntax has a way:

structure :cp_info do
  uint8   :tag
  switch  :tag, :replaces_this_structure => true do |value|
    case value
      when  1 ; :CONSTANT_Utf8_info
      when  3 ; :CONSTANT_Integer_info
      when  4 ; :CONSTANT_Float_info
      when  5 ; :CONSTANT_Long_info
      when  6 ; :CONSTANT_Double_info
      when  7 ; :CONSTANT_Class_info
      when  8 ; :CONSTANT_String_info
      when  9 ; :CONSTANT_Fieldref_info
      when 10 ; :CONSTANT_Methodref_info
      when 11 ; :CONSTANT_InterfaceMethodref_info
      when 12 ; :CONSTANT_NameAndType_info
      else raise("Value has no mapping: #{value}")
    end
  end
end

:replaces_this_structure => true means that it literally replaces the entire structure but this felt like a hack when I wrote it and still feels like one today.

I'm actually trying to change over to bindata for the structure definitions as I found it to be much better than my own DSL in many ways.

Hi everyone, it seems like bindata uses synchronous interfaces when reading data. Do you, guys, have any examples of doing it in asynchronous environments?

Thanks

I'm trying the tcp/ip example and the 'choices' doesn't appear to be finding a match. I'm reading in some binary data from a wireshark capture. Here's my trace:

obj.dst-internal-.octets[0] => 132
obj.dst-internal-.octets[1] => 181
obj.dst-internal-.octets[2] => 156
obj.dst-internal-.octets[3] => 194
obj.dst-internal-.octets[4] => 46
obj.dst-internal-.octets[5] => 225
obj.dst => "84:b5:9c:c2:2e:e1"
obj.src-internal-.octets[0] => 0
obj.src-internal-.octets[1] => 224
obj.src-internal-.octets[2] => 134
obj.src-internal-.octets[3] => 19
obj.src-internal-.octets[4] => 59
obj.src-internal-.octets[5] => 221
obj.src => "00:e0:86:13:3b:dd"
obj.vlan_type => 33024
obj.dot1q_priority => 0
obj.dot1q_CFI => 0
obj.dot1q_vlan_id => 750
obj.ether_type => 2048
obj.payload-selection- => 2048
obj.payload => "E\x00\x03$\xD2\x82\x00\x00@\x0...

From the trace, the ether_type appears to match the the option called out in the choice (they're both 2048).

Note that I've had to use a different record for my ethernet frame because it's got a vlan tag. Here is my modified ethernet record:

class Etherdot1q < BinData::Record
  IPV4 = 0x8100 
  endian :big
  mac_addr :dst
  mac_addr :src
  uint16   :vlan_type
  bit3     :dot1q_priority
  bit1     :dot1q_CFI
  bit12    :dot1q_vlan_id
  uint16   :ether_type   
  choice   :payload, :selection => :ether_type do
    ip_pdu IPV4
    rest   :default
  end
end

If I comment out the default option, I get this:

choice.rb:119:in `instantiate_choice': selection '2048' does not exist in :choices for obj.payload (IndexError)

I tried playing around with the choice syntax, and have tried both styles and it does the same thing. Is this a thing, or am I doing it wrong?

Update -

Apologies, I see from the closed issues that it's a ruby 2.1.0 thing. Sorry 'bout that.

I would like to create BinData::Record subclasses that support either endian-ness. I know I can use BinData::Struct to dynamically create these types, but using these types is clunky.

Contrived example of what I want:

require 'bindata'

class Vector < BinData::Record
  mandatory_parameter :endian
  uint32 :x
  uint32 :y
  uint32 :z
end

class SphereLE < BinData::Record
  endian :little
  vector :location
  uint32 :radius
end

class SphereBE < BinData::Record
  endian :big
  vector :location
  uint32 :radius
end

puts SphereLE.new.read("\001\000\000\000\002\000\000\000\003\000\000\000\004\000\000\000")
puts SphereBE.new.read("\000\000\000\001\000\000\000\002\000\000\000\003\000\000\000\004")

It would be really cool if vector_le and vector_be were automatically supported as well.

Hi! I was wondering if there is any support for enum primitive types. For example, consider the following definitions from the TLS spec:

struct {
    uint8 major;
    uint8 minor;
} ProtocolVersion;

enum {
    change_cipher_spec(20), alert(21), handshake(22),
    application_data(23), (255)
} ContentType;

struct {
    ContentType type;
    ProtocolVersion version;
    uint16 length;
    opaque fragment[TLSPlaintext.length];
} TLSPlaintext;

Obviously it would be simple enough to model ContentType as a uint8. But since this is an enum, only a few possible values are valid. It might be nice if you could say something like this:

class ContentType < BinData::Enum
  bytesize 1
  values(change_cipher_spec: 20,
         alert: 21,
         handshake: 22,
         application_data: 23)
end

And get validation for free, maybe along with syntactic sugar to let you assign the symbolic names to enum fields.

Does BinData support something like this currently? Or has it been considered for inclusion at any point? Alternately, how would you model this specification?

Thanks!

Hi

Per request to pen another issues. You solution works as per the prior issue

In this example (data structure slightly obfuscated to protect underlying IP):

class ArrayMessage < RecordBase

  uint8 :version_num, :check_value => lambda { 1 == value }
  uint8 :selected_element_id
  uint8 :num_elements, :value => lambda { data_elements.length } 
  array :data_elements, :read_until => lambda { index == num_elements - 1 } do 
    uint8  :element_id
    uint48 :time_stamp
    uint32 :parent_id
    uint16 :data_1
    uint16 :data_2
    uint16 :data_3
    uint16 :data_4
    bit2   :data_1_valid
    bit2   :data_2_valid
    bit2   :data_3_valid
    bit2   :data_4_valid
  end

end 

This part of a larger structure:

class DataMessage < RecordBase

  CHOICES = {
     # ... other choices
    20 => DisplayTextMessage, # per other issue        
    # ... other choices
    44 => ArrayMessage,
    # ... other choices
  }

  uint8  :message_type, :initial_value => 20
  uint48 :time_stamp
  uint32 :source_id
  uint16 :message_length 
  choice :message_body, :selection => :message_type,
              :choices => DataMessage::CHOICES,
              :read_length => :message_length,
              :default => DisplayTextMessage  
end

and for completeness:

class RecordBase < BinData::Record

  endian :little

  # support JSON Serialization
  def as_json(*a)
    {
      'json_class'   => self.class.name,
      'data'         => self.snapshot
    }.as_json(*a)
  end

  # support JSON Desrialization
  def self.json_create(o)
   cls = o['json_class'].constantize
   obj = cls.new
   obj.assign(o['data'])
   obj
  end
end

The implementation open a TCPSocket and makes subsequent reads on the DataMessage. When the first ArrayMessage is read all seems to be correct; but subsequent reads seem to (per trace_reading) start the next index based of the last index of the prior read + 1 and continue, so the :read_until => lambda { index == num_elements - 1 } never resolves to true and the call gets stuck in the read.

It seem that index is not reset back to zero for each subsequent read.

Also per ArrayMessage declaration and feedback for the prior issues:

  uint8 :num_elements, :value => lambda { data_elements.length } 
  array :data_elements, :read_until => lambda { index == num_elements - 1 } do 

Is seem that to have the mutually dependent definition of num_elements and data_elements, and to essentially override the num_elements value from the stream seems to be incorrect ? However removing this makes no difference.

The following is a sample program that works fine in 2.0 but not in 2.1.

require 'bindata'

class ChoiceObject < BinData::Choice
  uint16le 0
  uint16be 1
end

class ChoiceRecord < BinData::Record
  uint8 :flag
  choice_object :payload, :selection => :flag
end

puts ChoiceRecord.read([0x00, 0x01, 0x02].pack("C*"))
puts ChoiceRecord.read([0x01, 0x01, 0x02].pack("C*"))

In Ruby 2.0.0p353, this results in:

{:flag=>0, :payload=>513}
{:flag=>1, :payload=>258}

In Ruby 2.1.0p0, I get a frustratingly misleading error about not finding the selection (when it obviously should):

/usr/local/rvm/gems/ruby-2.1.0/gems/bindata-2.0.0/lib/bindata/choice.rb:175:in `instantiate_choice': selection '0' does not exist in :choices for obj.payload (IndexError)
        from /usr/local/rvm/gems/ruby-2.1.0/gems/bindata-2.0.0/lib/bindata/choice.rb:169:in `current_choice'
        from /usr/local/rvm/gems/ruby-2.1.0/gems/bindata-2.0.0/lib/bindata/choice.rb:153:in `do_read'
        from /usr/local/rvm/gems/ruby-2.1.0/gems/bindata-2.0.0/lib/bindata/struct.rb:208:in `block in do_read'
        from /usr/local/rvm/gems/ruby-2.1.0/gems/bindata-2.0.0/lib/bindata/struct.rb:208:in `each'
        from /usr/local/rvm/gems/ruby-2.1.0/gems/bindata-2.0.0/lib/bindata/struct.rb:208:in `do_read'
        from /usr/local/rvm/gems/ruby-2.1.0/gems/bindata-2.0.0/lib/bindata/base.rb:168:in `read'
        from /usr/local/rvm/gems/ruby-2.1.0/gems/bindata-2.0.0/lib/bindata/base.rb:51:in `read'
        from choice_test.rb:14:in `<main>'

Interestingly, if I change :selection => :flag to :selection => 0 or :selection => 1, they are selected. Not a useful work-around, but maybe it will help you find the problem.

From a little bit of digging around in the code, it looks like LazyEvaluator is going to try to call __self__.flag on the ChoiceObject class, so maybe that behavior has changed in 2.1?

Right now, i can define

array :msg, type: :uint8, read_until: :eof

but i can't define

string :msg, read_until: :eof

The latter would be very useful in a protocol i'm consuming right now. Is there any way to simulate this or have this added to the API?

I've found out that bindata works in a special way. It write complex data structure to given IO by fields calling #do_write on each field.

I think in most cases when people work with files there is unnecessary low-level detail. Unfortunately in my case I'm working with SerialPort and hardware expects all binary data as a solid stream.

I suspect that this can be fixed with io.sync = false and later io.flush after BinData#write, but haven't try it yet as a hardware I worked with is inaccessible now for sometime. I'll check it and write you back here.

Nevertheless, I think it mentioning this tiny details somewhere in BinData guide could save for people several hours/days of guessing and debugging.

First off, thanks a lot for this fantastic DSL, saving precious seconds every day!

The data I'm working with is a database format, simplified example:

class DBC < BinData::Record
  uint32 :record_count
  array :records, type: Record, initial_length: :record_count
end

dbc = DBC.read(io)
p dbc.records[499]

Both the actual Record-structure as well as the amount of records differ per database file, resulting in a significantly higher memory footprint as BinData processes the entire file, not just the 500th record I'm interested in.

Is there an easy way at present to instruct BinData or Array specifically, to load records on demand instead of all at once?

Some binary formats rely on compiler-time struct packing. This really sucks.
This happens when applications just dump memory into file.

So basically you have to deal with compiler alignments. It's quite complicated with bindata ATM.

Example:

class MyAwesomeAlignedStruct 
   string :magic_word, read_length: 4
   string :version, read_length: 5
# here we suck balls,
# because compiler aligned two previous fields (9 bytes) with 7 bytes of \x0
# so double located at nice positon of 16 (for 64-bit systems, multiple of 8)
# or 12 (for 32-bit systems, multiple of 4)
   double :magic_double
end

Is there any possible way to detect/predict alignments?

For example reading int/long/float/double from position not a multiple of 4/8.

Hello,

I noticed this example code does not do what you would expect:

class Test < BinData::Record

  uint8 :type
  choice :data, :selection => :type do
    uint8 1
    string 2
  end
end

Test.new(type: 1)

It produces an IndexError: selection '1' does not exist in :choices for obj.data because the type of the type field is BindData::Uint8, but the key in the choices hash is a Fixnum and thus the lookup fails.

I'm not sure what the best option here is. Maybe int fields should be coerced to Fixnums before doing hash lookups?

class MyObject < BinData::Record
  endian :big
  uint8 :length_field
  string :string_field, :read_length => 'length_field'
end

MyObject.read("\x02\x00\x05")

The above code should raise a TypeError: cannot convert String into Integer.

This is difficult to track down without digging into bindata so I was wondering if we could get an error/warning or possibly force the the variable to a symbol when the interpreter is parsing the MyObject class.

Hi,

How can I define a structure like this?

require 'bindata'

class Foo < BinData::Record
    uint32le :baz
    bar :barfield
end

class Bar < BinData::Record
    uint32le :baz
    foo :foofield
end

With the above code It complains '<class:Foo>': unknown type 'bar' in Foo (TypeError). Thanks!

Hi Just upgraded to 1.6.0 code and find a new error that did not occur prior to upgrade:

ArgumentError: wrong number of arguments calling `eql?` (1 for 0)
    from org/jruby/RubyHash.java:1118:in `[]'
    from /Users/simon/.rvm/gems/jruby-1.7.4@acds-new/bundler/gems/bindata-7f9bfc438514/lib/bindata/sanitize.rb:132:in `[]'
    from /Users/simon/.rvm/gems/jruby-1.7.4@acds-new/bundler/gems/bindata-7f9bfc438514/lib/bindata/choice.rb:173:in `instantiate_choice'
    from /Users/simon/.rvm/gems/jruby-1.7.4@acds-new/bundler/gems/bindata-7f9bfc438514/lib/bindata/choice.rb:169:in `current_choice'
    from /Users/simon/.rvm/gems/jruby-1.7.4@acds-new/bundler/gems/bindata-7f9bfc438514/lib/bindata/choice.rb:153:in `do_read'
    from /Users/simon/.rvm/gems/jruby-1.7.4@acds-new/bundler/gems/bindata-7f9bfc438514/lib/bindata/trace.rb:84:in `do_read_with_hook'
    from /Users/simon/.rvm/gems/jruby-1.7.4@acds-new/bundler/gems/bindata-7f9bfc438514/lib/bindata/struct.rb:208:in `do_read'
    from org/jruby/RubyArray.java:1617:in `each'
    from /Users/simon/.rvm/gems/jruby-1.7.4@acds-new/bundler/gems/bindata-7f9bfc438514/lib/bindata/struct.rb:208:in `do_read'
    from /Users/simon/.rvm/gems/jruby-1.7.4@acds-new/bundler/gems/bindata-7f9bfc438514/lib/bindata/base.rb:168:in `read'

As indicative of the call stack, my defined record structure is utilizing a choice.

The error occurs with and without tracing.

As before I'm running JRuby 1.7.4 in ruby 2.0.0 compatibility mode.

I see a significant refactor in this area 2194607

This is a breaking change for me if require me to update to get this other issue #10 resolved.

Please advise...

Is there a nice way to set the 'selection' parameter in response to changes in the choice value

consider

class container <BinData::Record
  uint8 :type
  choice :payload, selection: lambda{type} do
      type1 0x01
      type2 0x02
  end
end

That works great with deserializing, but not so good when I want to create and send it off the other way.

container.new(payload: type1.new)

Won't work as type is incorrect

I've noticed the form

 container.new(type:0x01, payload: type1.new)

Seems to work, but I'm kind-of supplying redundant information here.

Also I'm willing to accept I may be using BinData completely wrong...

I encountered a problem while implementing parser for SWF file. Here is Rectangle struct:

It would be awesome if BinData supports BitField's length as parameter:

class Rectangle < BinData::Record
  ubit5 :nbit
  sbit :xmin, :length => :nbit
  sbit :xmax, :length => :nbit
  sbit :ymin, :length => :nbit
  sbit :ymax, :length => :nbit
end

I'm trying to understand how to create a buffer of uint8s that will hold any kind of data.

I'm looking to do something similar to the following C code:

uint32_t buffer_length = 10 ;
uint8_t* buffer = malloc( sizeof(uint8_t) * buffer_length ) ;
char sample_string[] = "1234567890" ;
uint32_t sample_int = 0x12345678 ;

memcpy( buffer, sample_string, buffer_length ) ;
// do stuff with buffer
bzero( buffer, buffer_length ) ;
memcpy( buffer, sample_int, sizeof(uint32_t) ) ;

Ideally, the solution would be to create a data structure similar to the following:

class MyData < BinData::Record
  uint32 length
  buffer    data
end

foo = MyData.new
foo.length = 10
foo.data = 0x12345678
# foo is now \01\02\03\04\05\06\07\08\09\00
foo.data = "ABCD"
# foo is now \41\42\43\44

I've tried doing this as follows:

class MyData < BinData::Record
  uint32 length
  string buffer
end

foo.length = 1
foo.data = "0"
foo.to_binary_s  #\01\00\00\000
# when this is sent over the network, the actual data is:
#   \01\00\00\00\30   \30 is the ASCII value for 0

I've tried using an array to do this as well, but it's really complicated to serialize different data, at which point, I might as well use Array.pack

Thank you

RubyGems.org doesn't report a license for your gem. This is because it is not specified in the gemspec of your last release.

via e.g.

spec.license = 'MIT'
# or
spec.licenses = ['MIT', 'GPL-2']

Including a license in your gemspec is an easy way for rubygems.org and other tools to check how your gem is licensed. As you can image, scanning your repository for a LICENSE file or parsing the README, and then attempting to identify the license or licenses is much more difficult and more error prone. So, even for projects that already specify a license, including a license in your gemspec is a good practice. See, for example, how rubygems.org uses the gemspec to display the rails gem license.

There is even a License Finder gem to help companies/individuals ensure all gems they use meet their licensing needs. This tool depends on license information being available in the gemspec. This is an important enough issue that even Bundler now generates gems with a default 'MIT' license.

I hope you'll consider specifying a license in your gemspec. If not, please just close the issue with a nice message. In either case, I'll follow up. Thanks for your time!

Appendix:

If you need help choosing a license (sorry, I haven't checked your readme or looked for a license file), GitHub has created a license picker tool. Code without a license specified defaults to 'All rights reserved'-- denying others all rights to use of the code.
Here's a list of the license names I've found and their frequencies

p.s. In case you're wondering how I found you and why I made this issue, it's because I'm collecting stats on gems (I was originally looking for download data) and decided to collect license metadata,too, and make issues for gemspecs not specifying a license as a public service :). See the previous link or my blog post about this project for more information.

In my environment(bindata 1.6.0 or 2.0.0, Ruby 2.0.0, Windows 7 64bit), the Choice's example in wiki doesn't work.

class MyInt16 < BinData::Record
  uint8  :e, :check_value => lambda { value == 0 or value == 1 }
  choice :int, :selection => :e,  # does not work well
               :choices => {0 => :int16be, 1 => :int16le}
end

MyInt16.read "\x00\x00\x01"

# IndexError: selection '0' does not exist in :choices for obj.int

I spent all night finded the reason is :e's type is Uint8 and it is not a valid index for Hash.

choices = { 0 => :int16be, 1 => :int16le }
assert choices[0] == :int16be
assert choices[Uint8.new(0)] == nil  # !!!

I modified :selection's value and it works well now.

class MyInt16 < BinData::Record
  uint8  :e, :check_value => lambda { value == 0 or value == 1 }
  choice :int, :selection => lambda { e.value },  # works well
               :choices => {0 => :int16be, 1 => :int16le}
end

MyInt16.read "\x00\x00\x01"

Hello,

Thanks for the great library! I'm trying to achieve something not described in the documentation, and I'm wondering if it is at all possible. I want to have a Hash-like object, which I can read/write in Ruby code, and then get it serialized using BinData.

foo = MyObject
foo["test"] = 123

The above code is expected to automagically change the len property if needed, and insert the item into the internal array (again, if there was no such item with the given string tag). I somehow managed to get MyObject represented as Hash when I puts it, by defining it as a Primitive and implementing the get function (which, however, doesn't let me to access entries like puts foo["test"], I still can only access items directly), and I am completely unclear on how could I implement setting. Could you please advice a correct way to handle such a Hash-like structure in BinData? Thanks!

class MyObject < BinData::Primitive
    endian :big
    int16 :len, :value => lambda { items.length }
    array :items, :initial_length => :len do
        string :tag
        term :item
    end

    # i added this to illustrate what kind of access i want to achieve
    def get
        hash = Hash.new
        items.each do |item|
            hash[item.tag] = item.item
        end
        hash
    end
    # problem
    def set(v)
       ???
    end
end
``

I'm trying to read data with the following class:

class IpfixSubTemplateMultiList < BinData::Record
  endian :big
  uint8  :short_element_length
  uint16 :long_element_length, :onlyif => lambda { short_element_length == 255 }
  uint8  :semantic
  array  :elements, :read_until => lambda {
    array.num_bytes == (long_element_length or short_element_length) - 1
  } do
    uint16 :element_template_id
    uint16 :element_template_length
    string :read_length => lambda { element_template_length - 4 }
  end
end

However the :read_until condition is checked after at least one element has been read, and there are valid situations where the array has no elements so it would be useful if the :read_until condition is checked once before any reading is done; something slotted in around https://github.com/dmendel/bindata/blob/master/lib/bindata/array.rb#L297-298 should do the trick however I'm unsure how best to handle the various lambda variables.

A common sight in binary formats is a "table of contents" at the start of a file, which may contain an array of offsets to records later in the file. The records may be sparsely populated in the file.

What is the best pattern to deal with this? I understand that bindata has support for seeking forwards, but not necessarily "rewinding" back to the original point. Is this beyond the scope of this project and should be handled at a higher level?