ciscocsirt / gosint Goto Github PK
View Code? Open in Web Editor NEWThe GOSINT framework is a project used for collecting, processing, and exporting high quality indicators of compromise (IOCs).
License: BSD 3-Clause "New" or "Revised" License
The GOSINT framework is a project used for collecting, processing, and exporting high quality indicators of compromise (IOCs).
License: BSD 3-Clause "New" or "Revised" License
Hi, I'm trying to bash install GOSINT on my Ubuntu installation. Somehow the binary is not build during the bash install. Trying it manually, godep go build -o gosint gives me the error sh: 11: godep not found. What am I missing here?
Thanks in advance for any help.
Uncaught TypeError: Cannot read property 'length' of undefined
at countUp (metrics.js:50)
at metrics.js:23
at d3.min.js:1
at Object. (d3.min.js:1)
at Object.t (d3.min.js:1)
at XMLHttpRequest.u (d3.min.js:1)
error occured in metrics.js:50.
My browser is Chrome 60.0.3112.113(64)
In accordance with the configuration instructions I created a twitter app added the appropriate consumer key, access token and their secrets. A number of twitter handles were added to follow and are confirmed to issue "live" intel but nothing is being picked from the twitter feed.
While creating recipes, it appears that any Operators selected will not be applied to the final recipe. While VirusTotal is my only available operator, it is consistently failing to be added to my recipes. All of my recipes state "- no operators selected -".
Web not displaying after visiting localhost....
I was able to get past my AJAX error by starting the gosint process. Although the Welcome to GOSINT GUI comes up, the GUI does not load any indicator metrics. Also are there any recommended feeds? I have twitter and alienvault setup I just don't get anything listed in the GUI.
I am requesting this framework to support something like STIX or TAXII according to https://www.us-cert.gov/Information-Sharing-Specifications-Cybersecurity
I've followed the installation guide step by step, however ended up unable to open the web application itself. Machine is Ubuntu 16.04.02 64bit
Seems that gosint starts ok:
gosint@xenial:~/projects/src/GOSINT$ ./gosint
_____ _______ _____ _____ _____ _____
/\ \ /::\ \ /\ \ /\ \ /\ \ /\ \
/::\ \ /::::\ \ /::\ \ /::\ \ /::\____\ /::\ \
/::::\ \ /::::::\ \ /::::\ \ \:::\ \ /::::| | \:::\ \
/::::::\ \ /::::::::\ \ /::::::\ \ \:::\ \ /:::::| | \:::\ \
/:::/\:::\ \ /:::/~~\:::\ \ /:::/\:::\ \ \:::\ \ /::::::| | \:::\ \
/:::/ \:::\ \ /:::/ \:::\ \ /:::/__\:::\ \ \:::\ \ /:::/|::| | \:::\ \
/:::/ \:::\ \ /:::/ / \:::\ \ \:::\ \:::\ \ /::::\ \ /:::/ |::| | /::::\ \
/:::/ / \:::\ \ /:::/____/ \:::\____\ ___\:::\ \:::\ \ ____ /::::::\ \ /:::/ |::| | _____ /::::::\ \
/:::/ / \:::\ ___\|:::| | |:::| |/\ \:::\ \:::\ \ /\ \ /:::/\:::\ \ /:::/ |::| |/\ \ /:::/\:::\ \
/:::/____/ ___\:::| |:::|____| |:::| /::\ \:::\ \:::\____/::\ \/:::/ \:::\____/:: / |::| /::\____\/:::/ \:::\____\
\:::\ \ /\ /:::|____|\:::\ \ /:::/ /\:::\ \:::\ \::/ \:::\ /:::/ \::/ \::/ /|::| /:::/ /:::/ \::/ /
\:::\ /::\ \::/ / \:::\ \ /:::/ / \:::\ \:::\ \/____/ \:::\/:::/ / \/____/ \/____/ |::| /:::/ /:::/ / \/____/
\:::\ \:::\ \/____/ \:::\ /:::/ / \:::\ \:::\ \ \::::::/ / |::|/:::/ /:::/ /
\:::\ \:::\____\ \:::\__/:::/ / \:::\ \:::\____\ \::::/____/ |::::::/ /:::/ /
\:::\ /:::/ / \::::::::/ / \:::\ /:::/ / \:::\ \ |:::::/ /\::/ /
\:::\/:::/ / \::::::/ / \:::\/:::/ / \:::\ \ |::::/ / \/____/
\::::::/ / \::::/ / \::::::/ / \:::\ \ /:::/ /
\::::/ / \::/____/ \::::/ / \:::\____\ /:::/ /
\::/____/ ~~ \::/ / \::/ / \::/ /
\/____/ \/____/ \/____/
2017/08/12 23:22:27 Connecting to database...
2017/08/12 23:22:27 Successfully connected to backend database.
2017/08/12 23:22:27 Feeds not running, proceeding to start.
2017/08/12 23:22:27 Missing Twitter API keys... cannot start Twitter stream.
2017/08/12 23:22:27 Orchestration started.
2017/08/12 23:22:27 Missing Alienvault OTX API key... cannot start feed.
2017/08/12 23:22:28 No feeds found.
The web application is not found:
I guess I'm missing something trivial and I'd like to adjust the install guide.
[root@instance-1 GOSINT]# ./gosint --serve localhost:8020
_____ _______ _____ _____ _____ _____
/\ \ /::\ \ /\ \ /\ \ /\ \ /\ \
/::\ \ /::::\ \ /::\ \ /::\ \ /::\____\ /::\ \
/::::\ \ /::::::\ \ /::::\ \ \:::\ \ /::::| | \:::\ \
/::::::\ \ /::::::::\ \ /::::::\ \ \:::\ \ /:::::| | \:::\ \
/:::/\:::\ \ /:::/~~\:::\ \ /:::/\:::\ \ \:::\ \ /::::::| | \:::\ \
/:::/ \:::\ \ /:::/ \:::\ \ /:::/__\:::\ \ \:::\ \ /:::/|::| | \:::\ \
/:::/ :::\ \ /:::/ / :::\ \ :::\ :::\ \ /::::\ \ /:::/ |::| | /::::\ \
/:::/ / :::\ \ /:::// :::_\ :::\ :::\ \ ____ /::::::\ \ /:::/ |::| | _____ /::::::\ \
/:::/ / :::\ |:::| | |:::| |/\ :::\ :::\ \ /\ \ /:::/:::\ \ /:::/ |::| |/\ \ /:::/:::\ \
/:::// :::| |:::|| |:::| /::\ :::\ :::_/::\ /:::/ :::_/:: / |::| /::_/:::/ :::_
:::\ \ /\ /:::||:::\ \ /:::/ /:::\ :::\ ::/ :::\ /:::/ ::/ ::/ /|::| /:::/ /:::/ ::/ /
:::\ /::\ ::/ / :::\ \ /:::/ / :::\ :::\ // :::/:::/ / // // |::| /:::/ /:::/ / //
:::\ :::\ // :::\ /:::/ / :::\ :::\ \ ::::::/ / |::|/:::/ /:::/ /
:::\ :::_\ :::_/:::/ / :::\ :::_\ ::::// |::::::/ /:::/ /
:::\ /:::/ / ::::::::/ / :::\ /:::/ / :::\ \ |:::::/ /::/ /
:::/:::/ / ::::::/ / :::/:::/ / :::\ \ |::::/ / //
::::::/ / ::::/ / ::::::/ / :::\ \ /:::/ /
::::/ / ::// ::::/ / :::_\ /:::/ /
::// ~~ ::/ / ::/ / ::/ /
// // /__/
2017/09/07 06:04:57 Connecting to database...
2017/09/07 06:04:57 Successfully connected to backend database.
2017/09/07 06:04:57 Feeds not running, proceeding to start.
2017/09/07 06:04:57 Missing Twitter API keys... cannot start Twitter stream.
2017/09/07 06:04:57 Orchestration started.
2017/09/07 06:04:57 Missing Alienvault OTX API key... cannot start feed.
2017/09/07 06:04:57 No feeds found.
^Z
[1]+ Stopped ./gosint --serve localhost:8020
[root@instance-1 GOSINT]# bg 1
[1]+ ./gosint --serve localhost:8020 &
[root@instance-1 GOSINT]# netstat -ano|grep 8020
[root@instance-1 GOSINT]#
**_I set gosint to listen on 8020 but nothing happend.
When i use nginx's proxy_pass to upstream this port,error occured like below:_**
2017/09/07 06:09:52 [error] 30956#0: *82 connect() failed (111: Connection refused) while connecting to upstream, client: 156.73.169.73, server: 156.77.144.211, request: "GET /inc/css/bootstrap.min.css.map HTTP/1.1", upstream: "http://127.0.0.1:8020/inc/css/bootstrap.min.css.map", host: "156.77.144.211"
2017/09/07 06:09:52 [error] 30956#0: *82 connect() failed (111: Connection refused) while connecting to upstream, client: 156.73.169.73, server: 156.77.144.211, request: "GET /inc/css/bootstrap.min.css.map HTTP/1.1", upstream: "http://127.0.0.1:8020/inc/css/bootstrap.min.css.map", host: "156.77.144.211"
2017/09/07 06:09:52 [error] 30956#0: *92 no live upstreams while connecting to upstream, client: 156.73.169.73, server: 156.77.144.211, request: "GET /api/post/ HTTP/1.1", upstream: "http://localhost/api/post/", host: "156.77.144.211", referrer: "https://156.77.144.211/transfer-station.php"
2017/09/07 06:09:52 [error] 30956#0: *93 connect() failed (111: Connection refused) while connecting to upstream, client: 156.73.169.73, server: 156.77.144.211, request: "GET /favicon.ico HTTP/1.1", upstream: "http://127.0.0.1:8020/favicon.ico", host: "156.77.144.211", referrer: "https://156.77.144.211/transfer-station.php"
2017/09/07 06:09:52 [error] 30956#0: *93 connect() failed (111: Connection refused) while connecting to upstream, client: 156.73.169.73, server: 156.77.144.211, request: "GET /favicon.ico HTTP/1.1", upstream: "http://127.0.0.1:8020/favicon.ico", host: "156.77.144.211", referrer: "https://156.77.144.211/transfer-station.php"
Wait for your solution.
Hey!
Macine is Ubuntu 16.04. I used bash install. Фfter installation at the address 127.0.0.1:8000 404 page not found.
Output of command -lntup
tcp 0 0 127.0.1.1:53 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:8000 0.0.0.0:* LISTEN 4326/gosint
tcp 0 0 127.0.0.1:27017 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN -
tcp6 0 0 ::1:631 :::* LISTEN -
udp 0 0 0.0.0.0:58709 0.0.0.0:* -
udp 0 0 0.0.0.0:38657 0.0.0.0:* -
udp 0 0 127.0.1.1:53 0.0.0.0:* -
udp 0 0 0.0.0.0:68 0.0.0.0:* -
udp 0 0 0.0.0.0:5353 0.0.0.0:* -
udp6 0 0 :::51135 :::* -
udp6 0 10240 :::5353 :::* -
API response from http://localhost/api/pre/ and get a JSON response of {"message": "none found"}
PHP it's OK.
Help me, please.
Both are located under https://github.com/ciscocsirt/gosint#step-by-step
sudo useradd -m gosint
su gosint
sudo su gosint
git clone https://github.com/ciscocsirt/GOSINT
cd gosint
cd GOSINT
I have done up to creating a Twitter application and also got the API key of Twitter application & Alienvault API key. But I don't know, where to put my API keys.
Tons of thanks in advance!
Kapin Raj
Looks like a bug when whatever ASN lookup you do returns multiple numbers:
2018/02/22 09:43:48 Error doing reverse lookup on %v: %v 103.235.224.237 Could not parse ASN (4808 23724 56282): %!s(MISSING): strconv.Atoi: parsing "4808 23724 56282": invalid syntax
2018/02/22 09:43:52 Error doing reverse lookup on %v: %v 103.75.156.237 Could not parse ASN (134877 135216): %!s(MISSING): strconv.Atoi: parsing "134877 135216": invalid syntax
2018/02/22 09:43:59 Error doing reverse lookup on %v: %v 106.75.101.69 Could not parse ASN (4808 23724): %!s(MISSING): strconv.Atoi: parsing "4808 23724": invalid syntax
2018/02/22 09:43:59 Error doing reverse lookup on %v: %v 106.75.18.132 Could not parse ASN (4808 23724): %!s(MISSING): strconv.Atoi: parsing "4808 23724": invalid syntax
These IPs are from the CI Army blacklist, http://cinsscore.com/list/ci-badguys.txt, there's about 160 IPs out of the 15k on the list that trigger the error. GOSINT version is latest master as of today, on Ubuntu 16.04.2 LTS, installed through the bash install script.
Nginx dont install with bash 1-install.sh
I installed the master branch, and see the following in a chrome debug console:
Failed to load resource: the server responded with a status of 404 (Not Found)
It's referenced in 2 files:
$ grep -r tags.js *
post-processing.php:<script src="inc/js/general/tags.js"></script>
pre-processing.php:<script src="inc/js/general/tags.js"></script>
But it doesn't exist:
$ ls -l website/inc/js/general/
total 192
-rw-rw-r--. 1 gosint gosint 36816 Jun 4 19:58 bootstrap.min.js
-rwxrwxr-x. 1 gosint gosint 16767 Jun 4 19:58 bootstrap-tokenfield.js
-rw-rw-r--. 1 gosint gosint 1612 Jun 4 19:58 generate.js
-rwxrwxr-x. 1 gosint gosint 11 Jun 4 19:58 index.php
-rw-rw-r--. 1 gosint gosint 92594 Jun 4 19:58 jquery-1.9.1.min.js
-rwxrwxr-x. 1 gosint gosint 4707 Jun 4 19:58 jquery.modal.min.js
-rwxrwxr-x. 1 gosint gosint 24820 Jun 4 19:58 jquery.noty.packaged.min.js
I also don't see it in the repository. I don't know if it impacts functionality or not.
HOW DO I ADD MY API KEYS
Thanks
Hello all,
I'm not sure if I'm missing something obvious but I've been trying to get this working for a number of hours now and I don't know what step I appear to be missing or doing wrong that's causing AJAX errors. I've followed the manual installation guide as outlined, provided any prerequisites and yet still when I try to load the settings page:
If I complete any of the fields and try to Update, instead of 2 AJAX errors I get 1 "submitted to Endpoint" response and 1 AJAX error popup. But refreshing the page simply wipes all the fields and I'm back to where I started. I'm deploying on Ubuntu Server 16.04.
Using docker setup way.
$ sudo docker run -i -t -p 443:443 jsitech/gosint
[sudo] password for mint64:
[ ok ] Starting database: mongodb.
We will now create Credentials to access the Gosint UI
Type a username: ***********
Type a Password: ***********
_____ _______ _____ _____ _____ _____
/\ \ /::\ \ /\ \ /\ \ /\ \ /\ \
/::\ \ /::::\ \ /::\ \ /::\ \ /::\____\ /::\ \
/::::\ \ /::::::\ \ /::::\ \ \:::\ \ /::::| | \:::\ \
/::::::\ \ /::::::::\ \ /::::::\ \ \:::\ \ /:::::| | \:::\ \
/:::/\:::\ \ /:::/~~\:::\ \ /:::/\:::\ \ \:::\ \ /::::::| | \:::\ \
/:::/ \:::\ \ /:::/ \:::\ \ /:::/__\:::\ \ \:::\ \ /:::/|::| | \:::\ \
/:::/ \:::\ \ /:::/ / \:::\ \ \:::\ \:::\ \ /::::\ \ /:::/ |::| | /::::\ \
/:::/ / \:::\ \ /:::/____/ \:::\____\ ___\:::\ \:::\ \ ____ /::::::\ \ /:::/ |::| | _____ /::::::\ \
/:::/ / \:::\ ___\|:::| | |:::| |/\ \:::\ \:::\ \ /\ \ /:::/\:::\ \ /:::/ |::| |/\ \ /:::/\:::\ \
/:::/____/ ___\:::| |:::|____| |:::| /::\ \:::\ \:::\____/::\ \/:::/ \:::\____/:: / |::| /::\____\/:::/ \:::\____\
\:::\ \ /\ /:::|____|\:::\ \ /:::/ /\:::\ \:::\ \::/ \:::\ /:::/ \::/ \::/ /|::| /:::/ /:::/ \::/ /
\:::\ /::\ \::/ / \:::\ \ /:::/ / \:::\ \:::\ \/____/ \:::\/:::/ / \/____/ \/____/ |::| /:::/ /:::/ / \/____/
\:::\ \:::\ \/____/ \:::\ /:::/ / \:::\ \:::\ \ \::::::/ / |::|/:::/ /:::/ /
\:::\ \:::\____\ \:::\__/:::/ / \:::\ \:::\____\ \::::/____/ |::::::/ /:::/ /
\:::\ /:::/ / \::::::::/ / \:::\ /:::/ / \:::\ \ |:::::/ /\::/ /
\:::\/:::/ / \::::::/ / \:::\/:::/ / \:::\ \ |::::/ / \/____/
\::::::/ / \::::/ / \::::::/ / \:::\ \ /:::/ /
\::::/ / \::/____/ \::::/ / \:::\____\ /:::/ /
\::/____/ ~~ \::/ / \::/ / \::/ /
\/____/ \/____/ \/____/
2021/03/15 01:54:35 Connecting to database...
2021/03/15 01:54:35 Successfully connected to backend database.
2021/03/15 01:54:35 Feeds not running, proceeding to start.
2021/03/15 01:54:35 Missing Twitter API keys... cannot start Twitter stream.
2021/03/15 01:54:35 Orchestration started.
2021/03/15 01:54:35 Missing Alienvault OTX API key... cannot start feed.
2021/03/15 01:54:35 No feeds found.
2021/03/15 01:55:54 Settings updated.
2021/03/15 01:55:54 Missing Twitter API keys... cannot start Twitter stream.
2021/03/15 01:55:54 Alienvault feed not running, proceeding to start.
2021/03/15 01:55:56 Alienvault feed started.
2021/03/16 00:00:36 Error doing reverse lookup on %v: %v 190.247.139.101 Could not parse ASN (7303 10318): %!s(MISSING): strconv.Atoi: parsing "7303 10318": invalid syntax
2021/03/16 00:00:45 Error doing reverse lookup on %v: %v 45.153.203.124 lookup 124.203.153.45.origin.asn.cymru.com on 10.95.38.38:53: no such host
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x6dae2f]
goroutine 2808 [running]:
main.GetAlienvault()
/go/src/GOSINT/feeds.go:490 +0x6af
main.AlienvaultFeedCron.func1()
/go/src/GOSINT/feeds.go:451 +0x20
GOSINT/vendor/gopkg.in/robfig/cron%2ev2.FuncJob.Run(0x7cdaf0)
/go/src/GOSINT/vendor/gopkg.in/robfig/cron.v2/cron.go:94 +0x25
created by GOSINT/vendor/gopkg.in/robfig/cron%2ev2.(*Cron).run
/go/src/GOSINT/vendor/gopkg.in/robfig/cron.v2/cron.go:189 +0x452
My mongo db stuff works, and when I run the gosint file itself, it has the standard running thing. But, I am having problems with nginx. My nginx file looks like:
# For more information on configuration, see:
# * Official English Documentation: http://nginx.org/en/docs/
# * Official Russian Documentation: http://nginx.org/ru/docs/
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
# Load dynamic modules. See /usr/share/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
# Load modular configuration files from the /etc/nginx/conf.d directory.
# See http://nginx.org/en/docs/ngx_core_module.html#include
# for more information.
include /etc/nginx/conf.d/*.conf;
server {
listen 80;
root /home/gosint/projects/src/GOSINT/website;
index index.php index.html index.htm;
try_files $uri $uri/ @apachesite;
server_name someserver.yourcompany.com;
gzip on;
gzip_proxied any;
gzip_types
text/css
text/javascript
text/xml
text/plain
application/javascript
application/x-javascript
application/json;
#location / {
# try_files $uri $uri/ =404;
#}
error_page 404 /404.html;
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
location @apachesite {
proxy_pass http://localhost:8000;
}
location ~ \.php$ {
try_files $uri =404;
fastcgi_split_path_info ^(.+\.php)(/.+)$;
# PHP 7
fastcgi_pass unix:/var/run/php/php7.0-fpm.sock;
# PHP 5
# fastcgi_pass unix:/run/php5-fpm.sock;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include fastcgi_params;
}
}
# Settings for a TLS enabled server.
#
# server {
# listen 443 ssl http2 default_server;
# listen [::]:443 ssl http2 default_server;
# server_name _;
# root /usr/share/nginx/html;
#
# ssl_certificate "/etc/pki/nginx/server.crt";
# ssl_certificate_key "/etc/pki/nginx/private/server.key";
# ssl_session_cache shared:SSL:1m;
# ssl_session_timeout 10m;
# ssl_ciphers HIGH:!aNULL:!MD5;
# ssl_prefer_server_ciphers on;
#
# # Load configuration files for the default server block.
# include /etc/nginx/default.d/*.conf;
#
# location / {
# }
#
# error_page 404 /404.html;
# location = /40x.html {
# }
#
# error_page 500 502 503 504 /50x.html;
# location = /50x.html {
# }
# }
}
What do I need to do to get this to work properly?
The animated GIF is playing to fast and looks blurred
Can you update that, or create a video on youtube?
Hi all,
I think it's better if we have a Dockerfile for GOSINT ;) Thanks.
Best Regards,
Hello,
Help please, I can not figure it out. After installing Gosint and when I go to the settings page, I always get error messages that appear on the screen.
Having glanced in the console I saw that the page refers to directories which apparently should have appeared after installation. Still getting a number of errors.
How can I get these directories? or under what conditions?
I encountered the following error in ubuntu18 and manjaro, why?
Setting up environment...
I will now run a test Hello World program with Go to ensure things are OK...
helloworld.go:2:8: cannot find package "fmt" in any of:
/home/gosint/go/src/fmt (from $GOROOT)
/home/gosint/projects/src/fmt (from $GOPATH)
package main
imports runtime: cannot find package "runtime" in any of:
/home/gosint/go/src/runtime (from $GOROOT)
/home/gosint/projects/src/runtime (from $GOPATH)
Some error encountered...
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.