curious, not an issue about py-idstools HOT 11 CLOSED

ip_options_raw and tcp_options_raw do not yet exist in any other fields. I may add decoders for known options in the future though. The IP6 address raw fields do exist in a decoded format, so they do exist elsewhere. As packet.py is general purpose, it can be useful to have the raw address should you need to do something with the raw bytes.

It also depens on you search patterns. For instance, I log very little to Elastic Search, but I do include the full packet, so my JSON looks like:

{
    "timestamp": "2014-11-14T09:06:41.025965-0600",
    "sensor_id": 0,
    "event_type": "alert",
    "src_ip": "82.165.177.154",
    "src_port": 80,
    "dest_ip": "10.16.1.11",
    "dest_port": 51959,
    "proto": "TCP",
    "alert": {
      "action": "allowed",
      "gid": 1,
      "signature_id": 498,
      "rev": 11,
      "signature": "INDICATOR-COMPROMISE id check returned root",
      "category": "Potentially Bad Traffic",
      "severity": 2
    },
    "packets": [
      {
        "timestamp": "2014-11-14T09:06:41.025965-0600",
        "packet": "ACSMDjFUABUXDQb3CABFAAErLsxAADIGCadSpbGaChABCwBQyvcsXagVAHqfDVAYAHPWhwAASFRUUC8xLjEgMjAwIE9LDQpEYXRlOiBGcmksIDE0IE5vdiAyMDE0IDE1OjA2OjQwIEdNVA0KU2VydmVyOiBBcGFjaGUNCkxhc3QtTW9kaWZpZWQ6IE1vbiwgMTUgSmFuIDIwMDcgMjM6MTE6NTUgR01UDQpFVGFnOiAiMTgxYzg0OWEtMjctNDI3MWM1ZjFhYzRjMCINCkFjY2VwdC1SYW5nZXM6IGJ5dGVzDQpDb250ZW50LUxlbmd0aDogMzkNCkNvbnRlbnQtVHlwZTogdGV4dC9odG1sDQoNCnVpZD0wKHJvb3QpIGdpZD0wKHJvb3QpIGdyb3Vwcz0wKHJvb3QpCg=="
      }
    ]
}

Which gives me enough information to present an event display like:

But, I also have full packet capture running, so I can pivot to another tool I have that will extract the packets for review in wireshark based on details in the event.

Cool about the tool - I look forward to seeing it.

from py-idstools.

Where you have "packets":["packet"] does that represent one of these payload fields:
tcp_payload, udp_payload, icmp_payload ?

I did try EveBox via docker, which worked fine after doing this:

./launcher start -i eth0  <<<--- did not work ???
... but this did:
VOLUMES="-v $(pwd)/data:/data"
VOLUMES="${VOLUMES} -v $(pwd)/image:/image"
VOLUMES="${VOLUMES} -v $(pwd)/data/log:/var/log"
sudo docker run --rm --cidfile=cid --net=host -i -t ${PORTS} ${VOLUMES} -e HOST_UID=$(id -u) jasonish/suricata-elk "-i eth0"
rm -f cid

My client uses a version of my code that's similar to EveBox, but I wanted to write something that's
open source, generic, and provides all of the data that's available from idstools, so I am just trying to
make sure I don't miss any fields. I'm probably storing too much for what most people will want to search, but I don't think it will to difficult to edit a python module in order to customize the fields to be stored in elasticsearch. I couldn't think of a easy (and fast) to list desired fields and have the program read/use that instead (more research needed).

I didn't notice the pivot to wireshark in evebox ... that would be a nice feature, so I will have to look closer at evebox.
Thanks again for all of your help.

from py-idstools.

Where you have "packets":["packet"] does that represent one of these payload fields:
tcp_payload, udp_payload, icmp_payload ?

No, its just a base64 of the raw packet from a unified2 packet record. I'm not actually using the packet decoder in my json writer (yet). In fact, I should probably rename u2json to u2eve as its purpose is to be fully compatible with Suricata's eve log format so the same Kibana dashboards can be used for Suricata and Snort alerts.

I think my plan is to decode the options from the raw packet in the UI layer.

Thanks for the feedback on the docker image, I'll have to take a look.

from py-idstools.

While I'm not 100% satisfied, I did release:

• Uni2EsPy indexer
  • this includes an args file to index in a way that's compatible with EveBox
• Nsm4Es

I'm not really into blogging so I'm not sure how to get the word out, or even
if there would be any interest. I think it's a good step forward to replace Barnyard2 (no offense as I've used it for years) and to use Elasticsearch instead of any SQL databases.

Any issues, suggestions or feedback is appreciated.

from py-idstools.

Cool stuff. One comment while browsing the Uni2EsPy source - license. Put
your license in early to make intentions clear, ideally at the top of every
source file.

I hope to checkout the rest soon. Obviously our interests align somewhere,
would love to discuss more outside of GitHub issues!

On Mon, Nov 17, 2014 at 12:58 AM, chris [email protected] wrote:

While I'm not 100% satisfied, I did release:
Uni2EsPy https://github.com/cleesmith/uni2espy indexer
Nsm4Es https://github.com/cleesmith/nsm4es

—
Reply to this email directly or view it on GitHub
#10 (comment).

from py-idstools.

Sure, no problem, just didn't want to spam you with questions. Plus sometimes it helps others to see another's struggles and misunderstandings. What license do you recommend ? By that I mean what is appropriate when the code actually relies on others code. What's the license on Idstools ?

from py-idstools.

I'm a fan of BSD (2 clause), MIT or Apache. But note that these licenses
allow someone to modify your code and release it as closed source. IMO,
this can lead to adoption by companies, and in a way, can lead to more
contributions back. If you don't want to allow closed source releases,
then look at the GPL.

For the last 15 years I've worked for companies that have made use of open
source. I've rarely been involved with using GPL code internally and
contributing changes back.. But instead have preferred BSD code, as we
could use it with less restrictions which have led to more contributions
back. I think its something to consider.

Another guideline, if its a library, MIT or BSD.. If its end-user facing,
GPL. The idea here is you want widest adoption possible of your toolkit
type code, but you don't want someone taking an end-user type product and
selling it as their own.

I've also played license cop in the past when pulling in open source code
into code bases, so I may be a bit more opinionated than others.

On Mon, Nov 17, 2014 at 1:23 AM, chris [email protected] wrote:

Sure, no problem, just didn't want to spam you with questions. Plus
sometimes it helps others to another struggles and misunderstandings. What
license do you recommend ?

—
Reply to this email directly or view it on GitHub
#10 (comment).

from py-idstools.

I added a MIT LICENSE.md file to both projects. I guess I don't really care how the software is used, but it's nice to give credit where credit is due. So in some files I have left your license in place when I did not substantially change the file ... hope that's ok. I don't see Nsm4Es as anything exceptional, it just background stuff missing from kibana. But Uni2EsPy seems like a reasonable replacement for the ever-slow-to-change Barnyard2, plus python is everywhere ... so uni2espy seems far more useable.

from py-idstools.

I didn't notice the pivot to wireshark in evebox ... that would be a nice feature, so I will have to look closer at evebox.

Right now its tied to my dump project, https://github.com/jasonish/dumpy, which I find much simpler than the alternatives. Once that is setup and going, copy sample-config.js to config.js and provide the dumpy URL.

from py-idstools.

Thanks, I was wondering how that worked. I will use that feature, but most of my clients don't use pcap's which I find difficult to understand ... maybe they don't have the storage space.

from py-idstools.

The Elastic Search index used now defaults to "logstash-*" but is also configurable.

from py-idstools.