dmendel / bindata Goto Github PK
View Code? Open in Web Editor NEWBinData - Reading and Writing Binary Data in Ruby
License: BSD 2-Clause "Simplified" License
BinData - Reading and Writing Binary Data in Ruby
License: BSD 2-Clause "Simplified" License
RubyGems.org doesn't report a license for your gem. This is because it is not specified in the gemspec of your last release.
via e.g.
spec.license = 'MIT'
# or
spec.licenses = ['MIT', 'GPL-2']
Including a license in your gemspec is an easy way for rubygems.org and other tools to check how your gem is licensed. As you can image, scanning your repository for a LICENSE file or parsing the README, and then attempting to identify the license or licenses is much more difficult and more error prone. So, even for projects that already specify a license, including a license in your gemspec is a good practice. See, for example, how rubygems.org uses the gemspec to display the rails gem license.
There is even a License Finder gem to help companies/individuals ensure all gems they use meet their licensing needs. This tool depends on license information being available in the gemspec. This is an important enough issue that even Bundler now generates gems with a default 'MIT' license.
I hope you'll consider specifying a license in your gemspec. If not, please just close the issue with a nice message. In either case, I'll follow up. Thanks for your time!
Appendix:
If you need help choosing a license (sorry, I haven't checked your readme or looked for a license file), GitHub has created a license picker tool. Code without a license specified defaults to 'All rights reserved'-- denying others all rights to use of the code.
Here's a list of the license names I've found and their frequencies
p.s. In case you're wondering how I found you and why I made this issue, it's because I'm collecting stats on gems (I was originally looking for download data) and decided to collect license metadata,too, and make issues for gemspecs not specifying a license as a public service :). See the previous link or my blog post about this project for more information.
Hi,
Firts of all, thank you for this gem, it seems that you have thought to everything.
To customize my test output, i did this way
require 'bindata'
class TodSnapshot
def initialize(value)
@value = value
end
def inspect
"#{@value}(#{"0x%08x" % @value})"
end
end
class Tod < BinData::Primitive
endian :big
uint32 :val
def get ; self.val; end
def set(v) ; self.val = v ; end
def snapshot
TodSnapshot.new(self.val)
end
end
class Pouet < BinData::Record
endian :big
tod :sent_tod
end
pouet = Pouet.new.read("\x01\xab\x23\xcd")
require 'pp'
pp pouet.snapshot
My output is the expected one:
{:sent_tod=>27993037(0x01ab23cd)}
And my question is the following : is there any more "direct" way to do this ?
Thanks in advance,
Jérôme
It seem that the base ruby types have many features that that are not present in the bindata primities.
If say I parse some binary data, and say define a structure that has an element of Uint8le and I copy this value put into another Ruby class (MyClass) member, the class member is of class Unit8le not Fixnum.
In an example where I then try and for example try and convert an instance of MyClass to_json I get a run-time error as that implementation is based on the standard primitive types, such as Integer, String, Float, Bignum, Fixnum, Boolean etc. Or their ability to support to_hash/to_h or be able to retrieve it's instance variables.
In some cases there are multiple option, perhaps the preference can be defined in the DSL ?
Is it fair to say based of the underlying bindata types that it should be able to automatically coerce values to these base types.
My current work around is to add to_i or to_s etc. as applicable as I copy the values out but this is far from DRY.
I'd argue that once parsed, the data should fall into the classic Ruby data type framework at a primitive level to ensure support from the rest of the run-time ?
I'm trying to create two records which refer to each other by type, and I was running into an issue. I've laid out the code with two less than desirable solutions, and was wondering if I was doing something wrong, or if this is the way to go about it.
https://gist.github.com/nicklink483/83f8c236aa991760aab6
Any help or insight would be appreciated.
I've been getting the following exception while trying to decode a big structure I have defined:
/usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:279:in instantiate_obj_at': method
method_missing' called on hidden T_DATA object (0x00000002660e20 flags=0xc klass=0x0) (NotImplementedError)
I've been decoding data of this very structure many many times (for a couple of years) with no issue, and I only seem to notice this exception whenever I try to decode a large number of files (1000+) in a row. Adding a delay of about 10-15 seconds between each decode seems to workaround that issue, although obviously I'd prefer to just perform them as quickly as possible :)
This is on ruby 1.9.3p0 (2011-10-30 revision 33570) [x86_64-linux].
/usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:279:in instantiate_obj_at': method
method_missing' called on hidden T_DATA object (0x00000002660e20 flags=0xc klass=0x0) (NotImplementedError)
from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:267:in find_obj_for_name' from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:164:in
block in snapshot'
from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:163:in each' from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:163:in
snapshot'
from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/array.rb:108:in block in snapshot' from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/array.rb:108:in
collect'
from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/array.rb:108:in snapshot' from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:165:in
block in snapshot'
from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:163:in each' from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:163:in
snapshot'
from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/array.rb:108:in block in snapshot' from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/array.rb:108:in
collect'
from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/array.rb:108:in snapshot' from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:165:in
block in snapshot'
from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:163:in each' from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:163:in
snapshot'
from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/array.rb:108:in block in snapshot' from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/array.rb:108:in
collect'
from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/array.rb:108:in snapshot' from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:165:in
block in snapshot'
from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:163:in each' from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:163:in
snapshot'
from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:165:in block in snapshot' from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:163:in
each'
from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:163:in snapshot' from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:165:in
block in snapshot'
from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:163:in each' from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:163:in
snapshot'
from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/array.rb:108:in block in snapshot' from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/array.rb:108:in
collect'
from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/array.rb:108:in snapshot' from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:165:in
block in snapshot'
from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:163:in each' from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:163:in
snapshot'
from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/choice.rb:137:in snapshot' from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:165:in
block in snapshot'
from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:163:in each' from /usr/lib/ruby/gems/1.9.1/gems/bindata-1.8.1/lib/bindata/struct.rb:163:in
snapshot'
Hi everyone, it seems like bindata uses synchronous interfaces when reading data. Do you, guys, have any examples of doing it in asynchronous environments?
Thanks
In my environment(bindata 1.6.0 or 2.0.0, Ruby 2.0.0, Windows 7 64bit), the Choice
's example in wiki doesn't work.
class MyInt16 < BinData::Record
uint8 :e, :check_value => lambda { value == 0 or value == 1 }
choice :int, :selection => :e, # does not work well
:choices => {0 => :int16be, 1 => :int16le}
end
MyInt16.read "\x00\x00\x01"
# IndexError: selection '0' does not exist in :choices for obj.int
I spent all night finded the reason is :e
's type is Uint8
and it is not a valid index for Hash.
choices = { 0 => :int16be, 1 => :int16le }
assert choices[0] == :int16be
assert choices[Uint8.new(0)] == nil # !!!
I modified :selection
's value and it works well now.
class MyInt16 < BinData::Record
uint8 :e, :check_value => lambda { value == 0 or value == 1 }
choice :int, :selection => lambda { e.value }, # works well
:choices => {0 => :int16be, 1 => :int16le}
end
MyInt16.read "\x00\x00\x01"
I just noticed your commit disabling Ruby 2.1.0 in Travis CI. Just wanted to chime in with something I recently discovered: the allow_failures option. Consider using it instead of explicitly disabling versions of Ruby you know may fail. E.g. see my code here https://github.com/olerass/sourceless/blob/master/.travis.yml
A common sight in binary formats is a "table of contents" at the start of a file, which may contain an array of offsets to records later in the file. The records may be sparsely populated in the file.
What is the best pattern to deal with this? I understand that bindata
has support for seeking forwards, but not necessarily "rewinding" back to the original point. Is this beyond the scope of this project and should be handled at a higher level?
Is there a way to define a bit field record that is Signed, for a field that has a size that is not divisible by 8?
For example, to create a Signed 16-bit field, I can do:
# Signed 16-bit (bit-aligned)
class SBIT16 < BinData::Int16be
bit_aligned
end
However, I also need to support a Signed 6-bit field, but cannot use the same approach , since there is no Int6be (and trying to define something like it returns: RuntimeError: nbits must be divisible by 8
).
Thanks.
When reading in larger files, it would be nifty to have a way to print progress to the terminal.
Hello,
Thanks for the great library! I'm trying to achieve something not described in the documentation, and I'm wondering if it is at all possible. I want to have a Hash-like object, which I can read/write in Ruby code, and then get it serialized using BinData.
foo = MyObject
foo["test"] = 123
The above code is expected to automagically change the len
property if needed, and insert the item into the internal array (again, if there was no such item with the given string tag
). I somehow managed to get MyObject
represented as Hash when I puts
it, by defining it as a Primitive and implementing the get
function (which, however, doesn't let me to access entries like puts foo["test"]
, I still can only access items
directly), and I am completely unclear on how could I implement setting. Could you please advice a correct way to handle such a Hash-like structure in BinData? Thanks!
class MyObject < BinData::Primitive
endian :big
int16 :len, :value => lambda { items.length }
array :items, :initial_length => :len do
string :tag
term :item
end
# i added this to illustrate what kind of access i want to achieve
def get
hash = Hash.new
items.each do |item|
hash[item.tag] = item.item
end
hash
end
# problem
def set(v)
???
end
end
``
class MyObject < BinData::Record
endian :big
uint8 :length_field
string :string_field, :read_length => 'length_field'
end
MyObject.read("\x02\x00\x05")
The above code should raise a TypeError: cannot convert String into Integer.
This is difficult to track down without digging into bindata so I was wondering if we could get an error/warning or possibly force the the variable to a symbol when the interpreter is parsing the MyObject class.
I have some records which are like this:
class CONSTANT_Methodref_info < BinData::Record
endian :big
uint8 :tag
uint16 :class_index
uint16 :name_and_type_index
end
class CONSTANT_NameAndType_info < BinData::Record
endian :big
uint8 :tag
uint16 :name_index
uint16 :descriptor_index
end
As you can see, they always share a uint8 :tag
which decides which record you're dealing with. I think this comes up a lot in id+length+data structures.
I tried to model this with bindata:
class Cp_info < BinData::Record
endian :big
uint8 :tag
choice :payload, :selection => :tag
:constant_methodref_info 10
:constant_name_and_type_info 12
end
end
The problem is, :tag is already outside the structure but also defined in the structure. I know that I could remove it from inside each structure, but I would like to maintain similarity with the structures as they were originally defined. This is sort of a type of struct-based inheritance, so I wondered if there was a more elegant way to do it.
My own DSL's syntax has a way:
structure :cp_info do
uint8 :tag
switch :tag, :replaces_this_structure => true do |value|
case value
when 1 ; :CONSTANT_Utf8_info
when 3 ; :CONSTANT_Integer_info
when 4 ; :CONSTANT_Float_info
when 5 ; :CONSTANT_Long_info
when 6 ; :CONSTANT_Double_info
when 7 ; :CONSTANT_Class_info
when 8 ; :CONSTANT_String_info
when 9 ; :CONSTANT_Fieldref_info
when 10 ; :CONSTANT_Methodref_info
when 11 ; :CONSTANT_InterfaceMethodref_info
when 12 ; :CONSTANT_NameAndType_info
else raise("Value has no mapping: #{value}")
end
end
end
:replaces_this_structure => true
means that it literally replaces the entire structure but this felt like a hack when I wrote it and still feels like one today.
I'm actually trying to change over to bindata for the structure definitions as I found it to be much better than my own DSL in many ways.
I need to calculate a checksum over the binary data below (from version to content) and then include it in the crc field. How would I do this in a BinData way?
class Heartbeat < BinData::Record
endian :big
uint8 :version, :value => 0
uint16 :seq_num, :value => 99
uint32 :id, :value => 0
uint8 :cmd_type, :value => 1
uint16 :timeout, :value => 65535
string :content, :value => 'Heartbeat'
uint16 :crc
end
Hi,
How can I define a structure like this?
require 'bindata'
class Foo < BinData::Record
uint32le :baz
bar :barfield
end
class Bar < BinData::Record
uint32le :baz
foo :foofield
end
With the above code It complains '<class:Foo>': unknown type 'bar' in Foo (TypeError)
. Thanks!
If a record is defined such as
class MyRecord < ACEARecord
CHOICES = {
1 => RecordA,
12 => RecordB,
20 => RecordC,
# ... etc.
}
uint8 :message_type, :initial_value => 12
choice :message_body, :selection => :message_type,
:choices => CHOICES
end
If one instantiates an instance of MyRecord and queries the type of message body:
a = MyRecord.new
a.message_body.class
=> BinData::Choice
What would be preferable (in the case of the initial_value) is:
=> RecordB
Looking at the code the instantiate
code seems to perform a new
but there does not seem to be a method for accessing it ?
The end goal would be to have behavior based of the class of the selection such that the following code would work:
case a.message_body
when RecordA
# do something based of the context of RecordA
when RecordB
# do something based of the context of RecordB
when RecordC
# do something based of the context of RecordC
# ... etc
end
Right now, i can define
array :msg, type: :uint8, read_until: :eof
but i can't define
string :msg, read_until: :eof
The latter would be very useful in a protocol i'm consuming right now. Is there any way to simulate this or have this added to the API?
I'm trying to read this response of a call to my Yubikey:
"\x72\x03foo\x72\x03bar\x90\x00"
into the following BinData record:
class Entry < BinData::Record
uint8 :name_list_tag, assert: 0x72
uint8 :name_length
string :name, read_length: :name_length
end
class Response < BinData::Record
array :codes, type: :entry
end
As you can see, there's no leading byte to indicate how many codes are returned, so I can't set the initial_length
of the array, and there are trailing bytes which are unrelated, so I can't use :read_until => :eof
.
I'm thinking that a nextbyte
variable should be exposed to the lambda passed to read_until
, so that I can do something like array :codes, type: :entry, read_until: lambda { nextbyte != 0x72 }
.
From a quick look I'd need to define IO#peekbytes(n)
and add :nextbyte => self.peekbytes(1)
to the variables
Hash inside ReadUntilPlugin#do_read(io)
.
Thoughts?
According to https://github.com/dmendel/bindata/wiki/PrimitiveTypes#asserted_value , I can use value and assert together in cases where value is a constant and assert is a lambda but I get the error "params 'value' and 'assert' are mutually exclusive" instead.
Hello,
I noticed this example code does not do what you would expect:
class Test < BinData::Record
uint8 :type
choice :data, :selection => :type do
uint8 1
string 2
end
end
Test.new(type: 1)
It produces an IndexError: selection '1' does not exist in :choices for obj.data
because the type of the type
field is BindData::Uint8, but the key in the choices hash is a Fixnum and thus the lookup fails.
I'm not sure what the best option here is. Maybe int fields should be coerced to Fixnums before doing hash lookups?
Suppose I have a table of offsets to some child objects, given that the offsets in the table are relative to the parent:
class BlobIndex < BinData::Record
uint32be :type
uint32be :offs
end
class Blob < BinData::Record; end;
class SuperBlob < BinData::Record
uint32be :len
array :blob_offsets, :type => :blob_index, :initial_length => :len
# Blob objects follow, according to the offsets, the offset of the first blob will be equal to the length of the blob_offsets array in bytes
end
I couldn't find how can I make such structure into an array of Blob
records. I have also tried:
array :blob_offsets, :type => :blob_index, :initial_length => :len
array :blobs, :initial_length => :len do
blob :adjust_offset => lambda { offset[index].offs }
end
Which didn't work properly, probably because the offset is adjusted related to the start of the IO, not the start of SuperBlob
object. Is it possible to rebase the offset calculation to a given field in the SuperBlob
object. I'm trying to port the following code from Python (notice it has a feature called Anchor
: https://github.com/comex/cs/blob/master/macho_cs.py)
For example, on https://github.com/dmendel/bindata/wiki/Records the bullet-list links to https://github.com/dmendel/bindata/wiki/Records#specifying-default-endian but the anchor is https://github.com/dmendel/bindata/wiki/Records#wiki-specifying-default-endian.
Hello,
Are virtual fields ok to use now, or should I wait for a future version to enable it by default?
I encountered a problem while implementing parser for SWF file. Here is Rectangle struct:
Field | Type | Comment |
---|---|---|
Nbits | UB[5] | Bits used for each subsequent field |
Xmin | SB[Nbits] | x minimum position for rectangle |
Xmax | SB[Nbits] | x maximum position for rectangle |
Ymin | SB[Nbits] | y minimum position for rectangle |
Ymax | SB[Nbits] | y maximum position for rectangle |
It would be awesome if BinData supports BitField's length as parameter:
class Rectangle < BinData::Record
ubit5 :nbit
sbit :xmin, :length => :nbit
sbit :xmax, :length => :nbit
sbit :ymin, :length => :nbit
sbit :ymax, :length => :nbit
end
The following is a sample program that works fine in 2.0 but not in 2.1.
require 'bindata'
class ChoiceObject < BinData::Choice
uint16le 0
uint16be 1
end
class ChoiceRecord < BinData::Record
uint8 :flag
choice_object :payload, :selection => :flag
end
puts ChoiceRecord.read([0x00, 0x01, 0x02].pack("C*"))
puts ChoiceRecord.read([0x01, 0x01, 0x02].pack("C*"))
In Ruby 2.0.0p353, this results in:
{:flag=>0, :payload=>513}
{:flag=>1, :payload=>258}
In Ruby 2.1.0p0, I get a frustratingly misleading error about not finding the selection (when it obviously should):
/usr/local/rvm/gems/ruby-2.1.0/gems/bindata-2.0.0/lib/bindata/choice.rb:175:in `instantiate_choice': selection '0' does not exist in :choices for obj.payload (IndexError)
from /usr/local/rvm/gems/ruby-2.1.0/gems/bindata-2.0.0/lib/bindata/choice.rb:169:in `current_choice'
from /usr/local/rvm/gems/ruby-2.1.0/gems/bindata-2.0.0/lib/bindata/choice.rb:153:in `do_read'
from /usr/local/rvm/gems/ruby-2.1.0/gems/bindata-2.0.0/lib/bindata/struct.rb:208:in `block in do_read'
from /usr/local/rvm/gems/ruby-2.1.0/gems/bindata-2.0.0/lib/bindata/struct.rb:208:in `each'
from /usr/local/rvm/gems/ruby-2.1.0/gems/bindata-2.0.0/lib/bindata/struct.rb:208:in `do_read'
from /usr/local/rvm/gems/ruby-2.1.0/gems/bindata-2.0.0/lib/bindata/base.rb:168:in `read'
from /usr/local/rvm/gems/ruby-2.1.0/gems/bindata-2.0.0/lib/bindata/base.rb:51:in `read'
from choice_test.rb:14:in `<main>'
Interestingly, if I change :selection => :flag
to :selection => 0
or :selection => 1
, they are selected. Not a useful work-around, but maybe it will help you find the problem.
From a little bit of digging around in the code, it looks like LazyEvaluator
is going to try to call __self__.flag
on the ChoiceObject
class, so maybe that behavior has changed in 2.1?
First off, thanks a lot for this fantastic DSL, saving precious seconds every day!
The data I'm working with is a database format, simplified example:
class DBC < BinData::Record
uint32 :record_count
array :records, type: Record, initial_length: :record_count
end
dbc = DBC.read(io)
p dbc.records[499]
Both the actual Record-structure as well as the amount of records differ per database file, resulting in a significantly higher memory footprint as BinData processes the entire file, not just the 500th record I'm interested in.
Is there an easy way at present to instruct BinData or Array specifically, to load records on demand instead of all at once?
I'm trying the tcp/ip example and the 'choices' doesn't appear to be finding a match. I'm reading in some binary data from a wireshark capture. Here's my trace:
obj.dst-internal-.octets[0] => 132
obj.dst-internal-.octets[1] => 181
obj.dst-internal-.octets[2] => 156
obj.dst-internal-.octets[3] => 194
obj.dst-internal-.octets[4] => 46
obj.dst-internal-.octets[5] => 225
obj.dst => "84:b5:9c:c2:2e:e1"
obj.src-internal-.octets[0] => 0
obj.src-internal-.octets[1] => 224
obj.src-internal-.octets[2] => 134
obj.src-internal-.octets[3] => 19
obj.src-internal-.octets[4] => 59
obj.src-internal-.octets[5] => 221
obj.src => "00:e0:86:13:3b:dd"
obj.vlan_type => 33024
obj.dot1q_priority => 0
obj.dot1q_CFI => 0
obj.dot1q_vlan_id => 750
obj.ether_type => 2048
obj.payload-selection- => 2048
obj.payload => "E\x00\x03$\xD2\x82\x00\x00@\x0...
From the trace, the ether_type appears to match the the option called out in the choice (they're both 2048).
Note that I've had to use a different record for my ethernet frame because it's got a vlan tag. Here is my modified ethernet record:
class Etherdot1q < BinData::Record
IPV4 = 0x8100
endian :big
mac_addr :dst
mac_addr :src
uint16 :vlan_type
bit3 :dot1q_priority
bit1 :dot1q_CFI
bit12 :dot1q_vlan_id
uint16 :ether_type
choice :payload, :selection => :ether_type do
ip_pdu IPV4
rest :default
end
end
If I comment out the default option, I get this:
choice.rb:119:in `instantiate_choice': selection '2048' does not exist in :choices for obj.payload (IndexError)
I tried playing around with the choice syntax, and have tried both styles and it does the same thing. Is this a thing, or am I doing it wrong?
Update -
Apologies, I see from the closed issues that it's a ruby 2.1.0 thing. Sorry 'bout that.
Is there a nice way to set the 'selection' parameter in response to changes in the choice value
consider
class container <BinData::Record
uint8 :type
choice :payload, selection: lambda{type} do
type1 0x01
type2 0x02
end
end
That works great with deserializing, but not so good when I want to create and send it off the other way.
container.new(payload: type1.new)
Won't work as type is incorrect
I've noticed the form
container.new(type:0x01, payload: type1.new)
Seems to work, but I'm kind-of supplying redundant information here.
Also I'm willing to accept I may be using BinData completely wrong...
Hi
I realize I'm probably doing something wrong...
I have an unsigned integer that is packed to be a direction (0-360 Degrees)
class MyDirection < BinData::Primitive
Uint16le :val
def get
(self.val * 360.0 / 65536.0)
end
def set(v)
self.val = (v * 65536.0 / 360.0).round
end
end
However is I read data I seem to be getting negative values.
When I read the value from the stream if get -15541
but should get 49995
. The data bytes are 0x4B and 0xC3.
I'm expecting the 'U' modifier in the Uint16le
definition of :val
to treat the data as unsigned ?
All help greatly appreciated.
I would like to create BinData::Record subclasses that support either endian-ness. I know I can use BinData::Struct to dynamically create these types, but using these types is clunky.
Contrived example of what I want:
require 'bindata'
class Vector < BinData::Record
mandatory_parameter :endian
uint32 :x
uint32 :y
uint32 :z
end
class SphereLE < BinData::Record
endian :little
vector :location
uint32 :radius
end
class SphereBE < BinData::Record
endian :big
vector :location
uint32 :radius
end
puts SphereLE.new.read("\001\000\000\000\002\000\000\000\003\000\000\000\004\000\000\000")
puts SphereBE.new.read("\000\000\000\001\000\000\000\002\000\000\000\003\000\000\000\004")
It would be really cool if vector_le
and vector_be
were automatically supported as well.
If a declarative structure defines a length (as read for the stream) and then an array with an initial length set to that length, I want to be able to confirm once the array has been read that the length is actually conforming to the length supplied in the stream.
Hi,
Love the library. I've been playing with the :adjust_offset
option and ran into an issue. The issue may be with my understanding or the code. :) I'll demonstrate with an example
require 'bindata'
io = ["6B7963680001000000000010000000140000000000006A640000000B"].pack "H*"
class KeychainFile < BinData::Record
endian :big
string :magic, length: 4, asserted_value: 'kych'
string :padding, read_length: 8
uint32 :schema_offset # in this example, the value is 20
uint32 :table_count, adjust_offset: lambda { schema_offset + 4 }
end
kf = KeychainFile.read io
kf.schema_offset.abs_offset # is 12, as expected
kf.table_count.abs_offset # is 16, where i would expect 24
It appears that abs_offset
does not take into account the value returned by adjust_offset
. The value of kf.table_count
is located at offset 24 and my understanding of abs_offset
is that it returns the location within the IO that houses the data. Is this a correct understanding? If so, I think there may be a bug but I am not clever enough to fix it :)
Hi
Per request to pen another issues. You solution works as per the prior issue
In this example (data structure slightly obfuscated to protect underlying IP):
class ArrayMessage < RecordBase
uint8 :version_num, :check_value => lambda { 1 == value }
uint8 :selected_element_id
uint8 :num_elements, :value => lambda { data_elements.length }
array :data_elements, :read_until => lambda { index == num_elements - 1 } do
uint8 :element_id
uint48 :time_stamp
uint32 :parent_id
uint16 :data_1
uint16 :data_2
uint16 :data_3
uint16 :data_4
bit2 :data_1_valid
bit2 :data_2_valid
bit2 :data_3_valid
bit2 :data_4_valid
end
end
This part of a larger structure:
class DataMessage < RecordBase
CHOICES = {
# ... other choices
20 => DisplayTextMessage, # per other issue
# ... other choices
44 => ArrayMessage,
# ... other choices
}
uint8 :message_type, :initial_value => 20
uint48 :time_stamp
uint32 :source_id
uint16 :message_length
choice :message_body, :selection => :message_type,
:choices => DataMessage::CHOICES,
:read_length => :message_length,
:default => DisplayTextMessage
end
and for completeness:
class RecordBase < BinData::Record
endian :little
# support JSON Serialization
def as_json(*a)
{
'json_class' => self.class.name,
'data' => self.snapshot
}.as_json(*a)
end
# support JSON Desrialization
def self.json_create(o)
cls = o['json_class'].constantize
obj = cls.new
obj.assign(o['data'])
obj
end
end
The implementation open a TCPSocket
and makes subsequent reads on the DataMessage
. When the first ArrayMessage
is read all seems to be correct; but subsequent reads seem to (per trace_reading) start the next index based of the last index of the prior read + 1 and continue, so the :read_until => lambda { index == num_elements - 1 }
never resolves to true and the call gets stuck in the read.
It seem that index is not reset back to zero for each subsequent read.
Also per ArrayMessage declaration and feedback for the prior issues:
uint8 :num_elements, :value => lambda { data_elements.length }
array :data_elements, :read_until => lambda { index == num_elements - 1 } do
Is seem that to have the mutually dependent definition of num_elements and data_elements, and to essentially override the num_elements value from the stream seems to be incorrect ? However removing this makes no difference.
Hi! I was wondering if there is any support for enum primitive types. For example, consider the following definitions from the TLS spec:
struct {
uint8 major;
uint8 minor;
} ProtocolVersion;
enum {
change_cipher_spec(20), alert(21), handshake(22),
application_data(23), (255)
} ContentType;
struct {
ContentType type;
ProtocolVersion version;
uint16 length;
opaque fragment[TLSPlaintext.length];
} TLSPlaintext;
Obviously it would be simple enough to model ContentType
as a uint8
. But since this is an enum, only a few possible values are valid. It might be nice if you could say something like this:
class ContentType < BinData::Enum
bytesize 1
values(change_cipher_spec: 20,
alert: 21,
handshake: 22,
application_data: 23)
end
And get validation for free, maybe along with syntactic sugar to let you assign the symbolic names to enum fields.
Does BinData support something like this currently? Or has it been considered for inclusion at any point? Alternately, how would you model this specification?
Thanks!
It would be nice to read an int and look it up in an array of symbols like this:
enum :status, :values => [:pending, :successful, :failed]
such that the read and write methods work transparently and also check the validity of the enum field.
I would like a generic PascalString type which allows me to handle strings with different length field widths in a DRY way like this:
U16BEPascalString = GenericPascalString.new :length_type => :uint16be
U16BEPascalString.read "\000\004ABCD"
U32BEPascalString = GenericPascalString.new :length_type => :uint32be
U32BEPascalString.read "\000\000\000\004ABCD"
I tried
class GenericPascalString < BinData::Primitive
mandatory_parameter :length_type
length_type :len
string :data, :read_length => :len
end
and also
class GenericPascalString < BinData::Primitive
mandatory_parameter :length_type
array :len, :type => :length_type, :initial_length => 1
string :data, :read_length => lambda {len[0]}
end
But with both attempts, I get "TypeError: unknown type 'length_type' in GenericPascalString". Is there a way to accomplish this?
I wrote some test code to learn bindata and I got an error I wasn't expecting.
I had assumed, as the documentation had implied for value (and therefore implied for asserted_value), that the identity string was a fixed length constant of three letters for the purposes of reading, writing, and asserting. Writing went great but reading resulted in a length of zero. I know adding the "read_length" attirbute will fix the issue but it took a trace of the code to figure that out. Doesn't seem intuitive. I would be happy to write a patch if you agree that this is not the expected behavior. Thank you for your review. Here's the code:
require 'bindata'
class TestFormat < BinData::Record
endian :big
string :identity, :asserted_value => "xyz"
uint8 :version, :initial_value => 1
end
test1 = TestFormat.new
test1.version = 65
File.open 'test.bin', 'wb' do |file|
test1.write( file )
end
test2 = TestFormat.new
File.open 'test.bin', 'rb' do |file|
test2.read( file )
end
puts test2
I'm writing a file format using your gem. I have a 3 classes, each nested and inheriting from the BinData::Record class.
When I reference the module in anyway, for example, I was doing FileFormat.constants
just as a test - I get a TypeError from bindata.
unknown type 'uint32' in BinData::Array
Here's a basic overview, with fake class names:
module FileFormat
class Header < BinData::Record
class TypeHeader < BinData::Record
class TypeABody < BinData::Record
def version
parent.parent.version # grab Header version
end
# some fields
uint8 :thing_count
array :things, initial_length: :thing_count do
stringz :string_field
uint32 :uint32_field, onlyif: lambda { version >= 1 } # this is where the errors happening
end
end
class TypeBBody < BinData::Record
# some fields
end
uint8 :type
choice :body, selection: :type, choices: [:type_a_body, :type_b_body]
end
uint32 :magic
uint16 :version
uint32 :file_size
array :type_headers, type: :type_header, read_until: :eof
end
end
Would this be happening because of my nested classes? It works when i comment them out and only define the Header class.
Hello. I am the author of trema/pio, a pure Ruby packet parser and generator built on BinData.
Pio: https://github.com/trema/pio
I've found the BinData's examples/ very useful when writing Pio, but I think it would be helpful for other users if BinData could provide some pointers to more practical, real world use-cases.
So, if possible, please add links to Pio in README or Wiki pages. Thanks in advance.
I've found out that bindata
works in a special way. It write complex data structure to given IO
by fields calling #do_write
on each field.
I think in most cases when people work with files there is unnecessary low-level detail. Unfortunately in my case I'm working with SerialPort
and hardware expects all binary data as a solid stream.
I suspect that this can be fixed with io.sync = false
and later io.flush
after BinData#write
, but haven't try it yet as a hardware I worked with is inaccessible now for sometime. I'll check it and write you back here.
Nevertheless, I think it mentioning this tiny details somewhere in BinData
guide could save for people several hours/days of guessing and debugging.
Binary Data and CRC checking is a likely combination ?
Is there a recommended approach to having a Record (with potentially nested records managed off a selection and choice) to have a structure that will read from a two byte synchronization point, the length of declarative structure, followed by a 32bit CRC32 check-sum, that is based on all the data read inclusive of the synchronization bytes through to the last byte prior to the CRC element ?
I've looked a Zlip::crc32 but am not sure exactly how to call it, and is this something that could be overloaded into a :check_value ?
Some binary formats rely on compiler-time struct packing. This really sucks.
This happens when applications just dump memory into file.
So basically you have to deal with compiler alignments. It's quite complicated with bindata ATM.
Example:
class MyAwesomeAlignedStruct
string :magic_word, read_length: 4
string :version, read_length: 5
# here we suck balls,
# because compiler aligned two previous fields (9 bytes) with 7 bytes of \x0
# so double located at nice positon of 16 (for 64-bit systems, multiple of 8)
# or 12 (for 32-bit systems, multiple of 4)
double :magic_double
end
Is there any possible way to detect/predict alignments?
For example reading int/long/float/double from position not a multiple of 4/8.
I'm trying to read data with the following class:
class IpfixSubTemplateMultiList < BinData::Record
endian :big
uint8 :short_element_length
uint16 :long_element_length, :onlyif => lambda { short_element_length == 255 }
uint8 :semantic
array :elements, :read_until => lambda {
array.num_bytes == (long_element_length or short_element_length) - 1
} do
uint16 :element_template_id
uint16 :element_template_length
string :read_length => lambda { element_template_length - 4 }
end
end
However the :read_until condition is checked after at least one element has been read, and there are valid situations where the array has no elements so it would be useful if the :read_until condition is checked once before any reading is done; something slotted in around https://github.com/dmendel/bindata/blob/master/lib/bindata/array.rb#L297-298 should do the trick however I'm unsure how best to handle the various lambda variables.
Hello!
I think it will be useful to write a wiki about the analogue of the sizeof(field_name)
function.
This can be used when you want to unload a set of values in a string with no conversion to the array.
Something like:
class SomeClass < BinData::Record
endian :little
# get the s_count elements of sizeof(uint16_t)
uint16 :s_count
string :samples, :read_length => lambda { s_count * s_count.do_num_bytes }
...
end
Of course this example works only if the dimension s_count
coincides with the dimension of each element.
Maybe think of something like uint16.size
?
Is it possible to change parameters of a record field after declaration, or in a subclass? Ex.:
class Foo < BinData::Record
uint8 :type, :assert => ->{(1..5).member? value}
end
class FooType3 < Foo
set_params? :type, :value => 3
end
In Foo, type can be any number from 1 to 5, but in FooType3, it's explicitly 3.
This project is very Good!
Related to #40.
It is really hard to tell the difference between :length
and :read_length
for fixed strings. :length
works with :asserted_value
but when I decided to no longer assert on that field, I also had to switch to :read_length
as "params 'length' and 'value' are mutually exclusive". As a n00b with you library, I don't understand how "fixed length string" is never one fixed length for all purposes.
I'm trying to understand how to create a buffer of uint8s that will hold any kind of data.
I'm looking to do something similar to the following C code:
uint32_t buffer_length = 10 ;
uint8_t* buffer = malloc( sizeof(uint8_t) * buffer_length ) ;
char sample_string[] = "1234567890" ;
uint32_t sample_int = 0x12345678 ;
memcpy( buffer, sample_string, buffer_length ) ;
// do stuff with buffer
bzero( buffer, buffer_length ) ;
memcpy( buffer, sample_int, sizeof(uint32_t) ) ;
Ideally, the solution would be to create a data structure similar to the following:
class MyData < BinData::Record
uint32 length
buffer data
end
foo = MyData.new
foo.length = 10
foo.data = 0x12345678
# foo is now \01\02\03\04\05\06\07\08\09\00
foo.data = "ABCD"
# foo is now \41\42\43\44
I've tried doing this as follows:
class MyData < BinData::Record
uint32 length
string buffer
end
foo.length = 1
foo.data = "0"
foo.to_binary_s #\01\00\00\000
# when this is sent over the network, the actual data is:
# \01\00\00\00\30 \30 is the ASCII value for 0
I've tried using an array to do this as well, but it's really complicated to serialize different data, at which point, I might as well use Array.pack
Thank you
Hi Just upgraded to 1.6.0 code and find a new error that did not occur prior to upgrade:
ArgumentError: wrong number of arguments calling `eql?` (1 for 0)
from org/jruby/RubyHash.java:1118:in `[]'
from /Users/simon/.rvm/gems/jruby-1.7.4@acds-new/bundler/gems/bindata-7f9bfc438514/lib/bindata/sanitize.rb:132:in `[]'
from /Users/simon/.rvm/gems/jruby-1.7.4@acds-new/bundler/gems/bindata-7f9bfc438514/lib/bindata/choice.rb:173:in `instantiate_choice'
from /Users/simon/.rvm/gems/jruby-1.7.4@acds-new/bundler/gems/bindata-7f9bfc438514/lib/bindata/choice.rb:169:in `current_choice'
from /Users/simon/.rvm/gems/jruby-1.7.4@acds-new/bundler/gems/bindata-7f9bfc438514/lib/bindata/choice.rb:153:in `do_read'
from /Users/simon/.rvm/gems/jruby-1.7.4@acds-new/bundler/gems/bindata-7f9bfc438514/lib/bindata/trace.rb:84:in `do_read_with_hook'
from /Users/simon/.rvm/gems/jruby-1.7.4@acds-new/bundler/gems/bindata-7f9bfc438514/lib/bindata/struct.rb:208:in `do_read'
from org/jruby/RubyArray.java:1617:in `each'
from /Users/simon/.rvm/gems/jruby-1.7.4@acds-new/bundler/gems/bindata-7f9bfc438514/lib/bindata/struct.rb:208:in `do_read'
from /Users/simon/.rvm/gems/jruby-1.7.4@acds-new/bundler/gems/bindata-7f9bfc438514/lib/bindata/base.rb:168:in `read'
As indicative of the call stack, my defined record structure is utilizing a choice
.
The error occurs with and without tracing.
As before I'm running JRuby 1.7.4 in ruby 2.0.0 compatibility mode.
I see a significant refactor in this area 2194607
This is a breaking change for me if require me to update to get this other issue #10 resolved.
Please advise...
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.