riolet / sam Goto Github PK

View Code? Open in Web Editor NEW

175.0 10.0 18.0 4.05 MB

System Architecture Mapper

Home Page: https://www.samapper.com

License: GNU General Public License v3.0

Python 32.60% HTML 5.39% CSS 24.68% JavaScript 37.34%

system-architecture diagnostics networking analysis security

sam's Introduction

SAM - System Architecture Mapper

SAM is a tool designed to map a network based on the data log of a router. It runs as a local python-based server and displays the a map and statistics on the browser.

Check out the website for details about the project and a demo!

Quickstart (using pip)

install SAM with pip:

pip install samapper

Collect network data with tcpdump and run the http server:

sudo tcpdump -i any -f --immediate-mode -l -n -Q inout -tt | samapper --local --whois --format=tcpdump

tcpdump will probably need to be run with sudo to allow it to capture network traffic from your devices.
Only tcpdump format works locally via pipe at the moment.

Or, run the http server without collecting data:

samapper --local --whois --format=none

Known issue:

When running samapper in local mode using sqlite (the default) the database will sometimes lock up when the collector is inserting and you are viewing the display. If this is happening, just run the collector for a while, stop it, and run the http server on its own.

Installation (using git)

Prerequisites

(optional) mysql - database software that will work better for this than sqlite.

apt-get install mysql-server libmysqlclient-dev python python-dev

pip - to install python packages

apt-get install python-pip

Installing

Clone this repository
Run pip install -r requirements.txt from within the directory to install necessary packages.
Set environment variables for credentials and settings. See sam/default.cfg.

e.g:
export SAM__DATABASE__DBN=mysql
export SAM__DATABASE__USER=root
export SAM__DATABASE__PW=mypassword

Usage

Start the server locally by running: python -m sam.launcher --target=webserver For a more robust deployment, SAM supports the WSGI interface (python sam/server_webserver.py) and can be run through a different web server.
Create a data source to use in the settings page, or use the default empty data source provided.
For static analysis, import your log files into the database by running the following scripts, where log_file is the path to your log file and destination is the name of the data source you wish to fill.

python -m sam.launcher --target=import --format=<format> --dest=<destination> <log_file>

Log formats currently supported include:
1. paloalto: The paloalto syslog format is expected.
2. nfdump: Binary files from nfcapd are expected. nfdump must be installed.
3. asa: Cisco ASA logs, Partial support. Thanks to Emre for contributing.
4. aws: AWS VPC Flow logs: Partial support. Thanks to Emre for contributing. VPC log spec
5. tcpdump: Designed to work with live local mode. See quickstart above
6. tshark: Partial support.
For live analysis,
1. On the settings page, choose a data source for your live data to be funneled into then create a Access Key
2. Edit default.cfg or set an environment variable (SAM__COLLECTOR__UPLOAD_KEY) to your new access key
3. Start the aggregator (this loads log data into the database)
  - python -m sam.launcher --target=aggregator
4. Start the collector (this listens to port 514 and translates syslog lines)
  - python -m sam.launcher --target=collector
  - You will need priviledges to bind to system port 514.
  - It should print "Testing connection... Succeeded."
5. Tell your router to output it's log files to that freshly opened socket.
Navigate your browser to localhost:8080 and explore your network!

sam's People

Contributors

Stargazers

Watchers

Forkers

joepelz thepartyplatypus blastarindia moandcompany nikhil-bhat eldhos kcdragon emresaglam maniacs-ops ivosandoval mgoerwell runt18 vermuz

sam's Issues

NameError: global name 'convert' is not defined

Traceback (most recent call last):
File "import_paloalto.py", line 117, in
main(sys.argv)
File "import_paloalto.py", line 109, in main
import_file(argv[1])
File "import_paloalto.py", line 68, in import_file
translated_line = translate(line, lineNum);
File "import_paloalto.py", line 53, in translate
return (convert(*(split_data[7].split("."))),
NameError: global name 'convert' is not defined

Host level zooming returns blank

samapper.portLUT not automatically created?

Feature: Use Elasticsearch ES5.x o ES6.x

Hi,

This project is awesome, but I think that if you migrate SQL to ES, it will be more faster in query and more easy deploy.

Even, it's possible to get pcap and export it to Elasticsearch database, and another feature will be able to get data from ES to SMAMapper !

Thanks!

Make IP addresses selectable

For copying and pasting.

Display the destination port of connections

render issue at max zoom

When fully zoomed in (at host level) there are sometimes issues in the rendering of circles and arrows.
Problems have been observed in Firefox on Linux, and Opera (chromium) on Windows.

zooming out from max level fixes issue. Could be floating point precision somewhere along the line?
attached: image of two arrows not drawing correctly.

And the intended #appearance:

filter by string

the map's port filter accepts strings and the server produces errors when casting to integer.

Delay between clicking on node and information side bar refreshing

If you click on node A and click on node B, information for node A is displayed while information for node A is fetched.

AWS importer update

The AWS importer was broken by the new importer format. It needs to updated to the new importer format and to include timestamps.

https://github.com/riolet/SAM/blob/master/import_aws.py

Subnet circles intersect

Even when there's enough space for them to be drawn without intersecting or overlapping.

Host config parameter ignored

Here
db = web.database(dbn='mysql', user=dbconfig.params['user'], pw=dbconfig.params['passwd'], db=dbconfig.params['db'], port=dbconfig.params['port'])

https://github.com/riolet/SAM/blob/dev/common.py#L28

Filter by destination port

incorrect connection lines

Incorrect connection lines are being drawn when both Inbound and Outbound traffic are enabled. And some incorrect lines even when just Inbound is enabled.
Fix:

Split database Links# tables for each direction into LinksIn# and LinksOut#. This is to because of the clustering in the links table
1. Instead, have clustered links be: 172.-1.-1.-1?
Ensure all links are preprocessed identically so any duplicate lines will overlap entirely.

Node size

A map node changes size under certain conditions when auto-refresh is enabled:
Zoom in on it.
Select it.
Wait for an auto-refresh tick.
Zoom out. The node changes size incorrectly.

Possible cause: selection holds onto an orphaned reference to the old node (that should have been replaced during refresh) This old reference doesn't have it's size updated correctly.

Solution: clear old selection and reselect new object after refresh.

MySQLdb used directly

MySQL is used directly in import script to do multiple inserts.
Should be converted to use web.db.DB.multipleInsert(...)

https://github.com/riolet/SAM/blob/dev/import_paloalto.py#L124

abstract log importer

The import_* files are all very similar and contain duplicated code.
Rather than continue to duplicate code in many places, a standard interface or superclass should be built. In many cases merely overriding the translate(...) function will be sufficient to cover a new format.

Port Links break on table sort

When the ports tab of the selection-details panel is sorted by clicking on one of the column headers, the port tooltips and popup stop working.
Something in the tablesort plugin removes the event associations from the anchor element.

ASA importer update

The Cisco ASA importer was broken by the new importer format. It needs to updated to the new importer format and to include timestamps.

https://github.com/riolet/SAM/blob/master/import_asa.py