Giter Club home page Giter Club logo

sbe-python's Introduction

sbe-python

Easy to use, fast, pure Python FIX SBE encoder and decoder.

Install

pip install sbe

Usage

Simple Decoding

Decode SBE to Python dictionaries in one line. Good for exploratory data analysis.

import sbe

with open('your-schema.xml', 'r') as f:
  schema = sbe.Schema.parse(f)

wtih open('your-data.sbe', 'rb') as f:
  buf = f.read()

# Get a Python dict in one-line
x = schema.decode(buf)

x.name  # The template message name

x.value
# {'userId': 11,
# 'timestamp': 1598784004840,
# 'orderSize': 0,
# 'price': 5678.0,
# ...

# If you need an initial offset, apply it Pythonically
schema.decode(buf[19:])

# decode_header to filter out messages based on header to avoid decoding
# message bodies that are not needed
schema.decode_header(buf)['templateId']

High Performance Decoding (Wrapping)

This gives you decent performance while still retaining high code readability.

import sbe

with open('your-schema.xml', 'r') as f:
  schema = sbe.Schema.parse(f)

wtih open('your-data.sbe', 'rb') as f:
  buf = f.read()

# Wrap the buffer without decoding it, fields are converted to Python variables
# on demand
x = schema.wrap(buf)

x.header['templateId']
x.body['price']
x.body['someGroup'][2]['price']

Direct Access with Pointers

get_raw_pointer gives you the required information to unpack a variable from memoryview / bytes. This gets you very close to the fastest achievable performance in Python:

import sbe

with open('your-schema.xml', 'r') as f:
  schema = sbe.Schema.parse(f)

header_pointer = schema.header_wrapper.get_raw_pointer('templateId')

# Let's say we are only interested in messages of templateId == 3
price_pointer = schema.message_wrappers[3].get_raw_pointer('price')

wtih open('your-data.sbe', 'rb') as f:
  buf = f.read()

# pass `memoryview` to `unpack` to avoid copying
buf = memoryview(buf)[initial_offset:]
template_id = header_pointer.unpack(buf)  # calls buf[offset:offset+size].cast("I")[0] directly

if template_id == 3:
  print(price_pointer.unpack(buf))

Encoding

import sbe

with open('./your-schema.xml', 'r') as f:
  schema = sbe.Schema.parse(f)

# message_id from the schema you want to encode
message_id = 3

obj = {
  'userId': 11,
  'price': 5678.0,
  # ...
}

# Encode from Python dict in one-line
schema.encode(schema.messages[3], obj)

# You can supply your header values as a dict
schema.encode(schema.messages[3], obj, headers)

sbe-python's People

Contributors

baryluk avatar gburca avatar jonyii avatar kizzx2 avatar verrocchimarco avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

sbe-python's Issues

Constants not returned in DecodedMessage values dict

Hi,
I like the library, ease of use is great. We have a common constant field defined across all templates in my project that can be used for semantic grouping however the DecodedMessage value dict doesn't not contain representation of the constant. I'd like to use this library in a project that uses some SBE constant fields however I can see in the code that constants are specifically ignored.
Can the constant value be added to the values dict instead of being ignored?
Thanks,
Mike

An example of how to encode message

Would be great to have an example of how to encode the message. There is a snippet in the documentation, but I'm not sure I understand what obj is:

import sbe

with open('./your-schema.xml', 'r') as f:
  schema = sbe.Schema.parse(f)

# message_id from the schema you want to encode
message_id = 3

# Encode from Python dict in one-line
schema.encode(schema.messages[3], obj)

# You can supply your header values as a dict
schema.encode(schema.messages[3], obj, headers)

AssertionError: Unknown tag 'ref'

Hello

I think it does not support referencing custom data type. The xml i fail to parse is here:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<sbe:messageSchema xmlns:sbe="http://fixprotocol.io/2016/sbe"
                   package="proper.market.api"
                   id="1"
                   version="1"
                   semanticVersion="1.0"
                   description="for market data transfer"
                   byteOrder="littleEndian">
    <types>
        <composite name="messageHeader" description="Message identifiers and length of message root">
            <type name="blockLength" primitiveType="uint16"/>
            <type name="templateId" primitiveType="uint16"/>
            <type name="schemaId" primitiveType="uint16"/>
            <type name="version" primitiveType="uint16"/>
        </composite>

        <enum name="innerthing" encodingType="uint8" description="trading exchange">
            <validValue name="value0">0</validValue>
            <validValue name="value1">1</validValue>
        </enum>
        <composite name="something">
            <ref name="innerthing" type="innerthing"/>
        </composite>
    </types>
    <sbe:message name="someMsg" id="1">
        <field name="something" id="1" type="something"/>
    </sbe:message>

</sbe:messageSchema>

decode function potential bug

Hi guys, just wanted to run this by here. Would the body of the second if statement ever be hit? Doesn't seem like it would. Is this a bug?

            if isinstance(f, Group):
                assert cursor.val >= header.value['blockLength']
                if cursor.val < header.value['blockLength']:
                    format_str_parts.append(str(header.value['blockLength'] - cursor.val) + 'x')

Link to line of the code

Schema parser requires message attribute blockLength which should be optional

When parsing a schema with message tags that don't specify the blockLength attribute, I get this error:

...
...\lib\site-packages\sbe\__init__.py", line 1088, in _parse_schema      
    blockLength=int(attrs['blockLength']),
KeyError: 'blockLength'

It looks like the parser depends on this attribute being there, however SBE specifies that blockLength is optional:

<xs:attribute name="blockLength" type="xs:nonNegativeInteger" use="optional">

Attribute "description" ignored for many tags

I've noticed that after parsing a schema, the description field of instances of several classes remains None. I haven't exhaustively tested this, but at least for message and field tags the description attribute does not get propagated to the corresponding objects.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.