Avro output about kodiaq HOT 6 CLOSED

Build notes for avro:

• the avro c++ docs are here: http://avro.apache.org/docs/current/api/cpp/html/
• the avro c++ driver package is available here: http://mirror.switch.ch/mirror/apache/dist/avro/avro-1.7.7/ under cpp
• avro requires libboost (best bet just install all the boost libraries) >= version 1.55. The Ubuntu 14.04 core repo version as of now is 1.54. Avro does not compile with this version so you have to compile boost from source.

Setting up boost:

1. Get the source. I got version 1.57 (newest) here: http://sourceforge.net/projects/boost/files/boost/1.57.0

2. Untar the package (tar -zxvf). You can access docs by opening index.html in the extracted dir.

3. Follow the instructions in the docs. Otherwise these commands should work:

     ./bootstrap.sh --prefix=/usr/local
     sudo ./b2 install
   

   1. Get a coffee. Or lunch. It takes a while.

Install avro

Next you can install avro. Get it at the links above. Pro tip: if you mess something up and you want to change cmake options you have to delete the cmake cache. Or just delete the avro folder and untar it again.

1. Use the following commands to install avro

           ./build.sh test
           ./build.sh install

 2. That should be it. Avro is not installed in /usr/local

Keep in mind that /usr/local/include and /usr/local/lib should be in your path and library path.

from kodiaq.

I would wait a little on this...

from kodiaq.

Why? Are we considering other formats for 1T data? If so what?

from kodiaq.

Avro seems good, but I mainly want to see what the outcome of the CM is. On the 20th of this month, we'll hear about a grant for an engineer that would be related to data formats. Maybe they come up with something smarter (and can help also the kodiaq implementation if you want)? Basho was telling me that this Avro stuff wasn't too straightforward in C++.

Let's discuss in person, but what if we just did BSON? That way there is no extra dependency for example. I sort of wanted to see how our DAQ test goes and if the file builder (e.g. avro) is a bottleneck. Afterwards, we can know how we finalize it.

from kodiaq.

We can freeze Avro and it's still much better than XED. However, we can also do better if you find the external dependency to be a burden.

from kodiaq.

Won't fix. Using BSON output for 1T.

from kodiaq.