Support for `Provider_name` and `Data[x]` notation to the field mapping about hayabusa HOT 4 CLOSED

Specification memo:

• If possible, make Provider_Name optional
• If possible, allow multiple values ​​to be set for Provider_Name(It is preferable to have one yaml file)
• Do not output the value before field value conversion to ExtraFieldInfo (csv/json)

from hayabusa.

• Do not output the value before field value conversion to ExtraFieldInfo (csv/json)

I have looked into the feasibility of implementation and it does not appear to be easy to achieve 🤔
The main reasons are the following two points:

• Under the Data field is an array (not simple key value)
• The exclusion process for Details and ExtraFieldInfo is a bit complicated
  • e.g. #1146
    • This is a different issue, but I think the fix will be as complicated as this one.

@YamatoSecurity
I think I'm going to create the issue of output to ExtraFieldInfo as a separate issue and hold off on implementation, what do you think?

from hayabusa.

@fukusuket I see, sure, we can hold off on the implementation for now. I'm thinking it might be better to save the Data fields as different fields instead of in an array for JSON. For example: Data-1, Data-2, etc... This way it will be easier to deal with when importing records into elastic stack, etc...

from hayabusa.

@YamatoSecurity

Do not output the value before field value conversion to ExtraFieldInfo (csv/json)

Sorry many times, I looked into it more and found how to implement this just now(though my current implementation is even more complicated...😇). I'll create PR.

I see, It would be nice if eliminating Array would make it easier to integrate with things like ElasticSearch :)

from hayabusa.