partial update to avro bytes about avro HOT 16 CLOSED

@nrwiersma Here a sample code which reproduces the panic:

package main

import (
	"github.com/sirupsen/logrus"
	"github.com/hamba/avro"
)

func main() {
        schema, _ := avro.Parse(`{"namespace": "People","type": "record","doc": "People information","name": "People",
"fields": [{"name": "name","type": ["null",{"type": "string","avro.java.string": "String"}],"doc": "Name of the person.","default": null},
{"name": "address","type": ["null",{"type": "record","doc": "Address information","name": "Address","fields": [{"name": "name","type": ["null",{"type": "string","avro.java.string": "String"}],"doc": "An optional name of the recipient","default": null}],"doc": "","default": null}]}]}`)

	p := People{
		Name:    "TD",
		Address: &Address{Name: "DT"},
	}

	pBytes, err := avro.Marshal(schema, &p)
	if err != nil {
		logrus.New().Fatal(err)
	}

	resMap := make(map[string]interface{})

	err = avro.Unmarshal(schema, pBytes, &resMap)
	if err != nil {
		logrus.New().Fatal(err)
	}

	pBytes, err = avro.Marshal(schema, &resMap)
	if err != nil {
		logrus.New().Fatal(err)
	}

}

type People struct {
	Name    string   `avro:"name"`
	Address *Address `avro:"address"`
}

type Address struct {
	Name string `avro:"string"`
}

from avro.

@xmlking it is not important for Marshal, you must pass a pointer for Unmarshal.

from avro.

No. This is an Avro library. That is out of scope for this library.

from avro.

this is the flow I can think of. looking for better/efficient process for partial updating avro raw data.

var genericRecord interface{}
_ = avro.Unmarshal(schema, data, &genericRecord)

// extract PartialRecord( a struct) from genericRecord ???
// work on PartialRecord and update PartialRecord
// patch genericRecord with updated PartialRecord ???
// marshal patched genericRecord into byte array and write to outgoing message

from avro.

There is a way to do what you want, but it is not fast. You could decode into a map[string]interface{} and make your changes to that, then encode the map. This will have the effect you are looking for, but it will be a little slow.

Struct parsing is much faster, but the trick would be to know where in your byte slice the partial data starts and stops to encode and replace. It would also depend on your actual data schemas.

Depending on the amount of change you need, it could be feasible to use Reader and Writer to roll your own "reader" that read past the parts you dont need to change (Reader has skip functions), write the new data, and then copy to rest of the bytes. I can see how this would work, but it would take some doing and a good understanding of Avro.

from avro.

I am looking into this solution:
Are you aware of any libs or util functions that can convert a Struct to map[string]interface{} vice versa? provided my Struct has json / avro tags.

ConvertMapToStruct(aMap map[string]interface{}, &Address{}) 
ConvertStructToMap(&Address{}, aMap map[string]interface{})

type Address struct {
	street  string
        city      string
        state   string
}

    var genericFullRecord, addressMap map[string]interface{}

    _ = avro.Unmarshal(schema, data, &genericFullRecord)

     addressMap = genericFullRecord["address"]
     address = &Address{}
    ConvertMapToStruct(addressMap, address)
    address.City =  "new city"
    ConvertStructToMap(address, addressMap)

    genericFullRecord["address"] = addressMap

   // encode genericFullRecord
}

from avro.

I tried some structToMap utils

1. struct-to-map
2. ConvertStructToMap
3. mapstructure

non of the above converters produce the map[string]interface{} that is equivalent to hamba generic map.

hamba's generic map has extra namespace keys that are derived from the Schema.

I am doing

1. original bytes --> native map ( i.e., unmarshal to genericOriginalMap)
2. original bytes --> PartialStruct( i.e., address) <-- onemore time unmarshal to PartialStruct. I need to convert to a struct, as I need to pass it to an address standardization service (RPC)
3. modify fields of address struct ( i.e., modifiedAddress)
4. modifiedAddress --> native map (i.e., modifiedAddressMap) <-- I am stuck here
5. set it back to original native map i.e, genericOriginalMap["address"] = modifiedAddress
6. serialize updated genericOriginalMap back to bytes.

Am extracting PartialMyStruct that I have to convert back to native map[string]interface{} to update the original native map and serialize.

from avro.

I'm looking for a similar functionality.

I convert the data to a map(using avro.Unmarshal), modify the fields I'm interested in, and then again use avro.Marshal() on the map.

In the above process avro.Marshal() panics at https://github.com/hamba/avro/blob/master/codec_record.go#L293

from avro.

@DasTushar Do you have some example code to reproduce the panic? What is interesting is that it is trying to encode a struct in this code, not a map.

from avro.

@DasTushar I see the issue, in fact I see 2 here. Can you open a separate issue for this please.

from avro.

@nrwiersma Have created the issue

Thank you

from avro.

for avro.Marshal() for second argument, should we pass value or reference (pointer)?

from avro.

are there any helper methods that take a struct pointer and convert it into generic interface{} in hamba?

func GenericFromSpecific(cus *Customer) map[string]interface{}

type Customer struct {
    Id        int        `avro:"id"`
    Name      string     `avro:"name"`
    Addresses *[]Address `avro:"addr"`
    Married   bool       `avro:"married"`
}
type Address struct {
    HouseNo string `avro:"houseNo"`
    Street  string `avro:"street"`
    City    string `avro:"city"`
    State   string `avro:"state"`
    Country string `avro:"country"`
    ZipCode int    `avro:"zipCode"`
}

My goal is to in-place patch an other big generic interface{}

var genericMap map[string]interface{}
avro.Unmarshal(schema, Data, &genericMap);
genericMap["customer"] = GenericFromSpecific(&cus)
avro.Marshal(schema, genericMap);

from avro.

one issue I have is:
GenericFromSpecific(schema, &cus) should generate map[string]interface{} where keys for record type should have namespace prefix. That means this converter function should be aware of schema and aware namespaces of all records in the schema.

{
  "request": {
    "body": {
      "address": {
        "com.sumo.demo.v1.Address": {
          "street": {},
          "state": {},
          "city": {}
        }
      },
      "account": {
        "com.sumo.demo.v2.Account": {
          "addresses": [],
          "dob": {},
          "emails": {},
          "gender": -1450016819,
          "names": [],
          "phones": {}
        }
      }
    },
    "session": {
      "com.sumo.demo.v3.Session": {
        "uuid": "qjvsenxjvbkegvmv",
        "cust": "ibreo"
      }
    }
  },
...

from avro.

looks like I can patch big genericRecord (of type map[string]interface{}) with a struct pointer without first converting it from Struct --> map[string]interface{}

// patch `genericRec` record with specific  Struct 
keyRec := genericRec["request"].([]interface{})[i].(map[string]interface{})
keyRec["account"] = map[string]interface{}{
	"com.sumo.demo.v2.Account": &accountStruct,
}

my issue is solved.

from avro.

from avro.