certsocietegenerale / fame Goto Github PK
View Code? Open in Web Editor NEWFAME Automates Malware Evaluation
Home Page: https://certsocietegenerale.github.io/fame/
License: GNU General Public License v3.0
FAME Automates Malware Evaluation
Home Page: https://certsocietegenerale.github.io/fame/
License: GNU General Public License v3.0
If module returns Russian symbols in detailed results, we see a mess
Write sample module, that will return string "фывапролдж"
see "фывапролдж" in web interface
fame@fame-server:~/fame$ utils/run.sh utils/troubleshoot.py
[+] Using existing virtualenv.
########## VERSION ##########
OS: Linux-4.4.0-83-generic-x86_64-with-Ubuntu-16.04-xenial
Python: 2.7.12
########## DEPENDENCIES ###########
alabaster==0.7.10
amqp==2.2.1
androguard==3.0.1
Babel==2.4.0
bamfdetect==1.6.13
beautifulsoup4==4.6.0
billiard==3.5.0.3
bs4==0.0.1
celery==4.0.2
certifi==2017.4.17
chardet==3.0.4
click==6.7
docutils==0.13.1
Flask==0.12.2
Flask-Classy==0.6.10
Flask-Login==0.3.2
Flask-Negotiation==0.1.9
flask-paginate==0.5.0
future==0.16.0
gitdb2==2.0.2
GitPython==2.1.5
googleplay-api==0.1.0
idna==2.5
ijson==2.3
imagesize==0.7.1
itsdangerous==0.24
Jinja2==2.9.6
kombu==4.0.2
LEPL==5.1.3
markdown2==2.3.4
MarkupSafe==1.0
oletools==0.51
pbkdf2==1.3
pefile==2016.3.28
pkg-resources==0.0.0
protobuf==3.3.0
pycrypto==2.6.1
pyelftools==0.24
Pygments==2.2.0
pymongo==3.4.0
python-magic==0.4.13
pytz==2017.2
rarfile==3.0
requests==2.18.1
rfc6266==0.0.4
six==1.10.0
smmap2==2.0.3
snowballstemmer==1.2.1
Sphinx==1.6.3
sphinx-rtd-theme==0.2.4
sphinxcontrib-httpdomain==1.5.0
sphinxcontrib-websupport==1.0.1
typing==3.6.1
urllib3==1.21.1
uWSGI==2.0.15
vine==1.1.4
volatility==2.6
Werkzeug==0.12.2
yara-python==3.6.3
zxcvbn==1.0
########## MongoDB ##########
Version: 3.4.6
Authorization check: True
########## Configuration ##########
types: True
virustotal: False
email: False
malware_config: False
volatility: True
Modules:
McAfee Antivirus Disabled Configured
Sophos Antivirus Disabled Configured
Symantec Antivirus Disabled Not Configured
apk Processing Disabled Configured
apk_verification Processing Disabled Not Configured
bamfdetect Processing Enabled Configured
cuckoo Processing Enabled Configured
cuckoo_modified Processing Disabled Configured
cutthecrap Processing Disabled Not Configured
eml Processing Enabled Configured
joe Processing Disabled Not Configured
marcher_config Processing Disabled Configured
mem_yara Processing Disabled Not Configured
office_macros Processing Enabled Configured
pdf Processing Enabled Configured
rat_decoders Processing Enabled Configured
url_download Processing Enabled Configured
zip Processing Enabled Configured
slack Reporting Disabled Not Configured
Yeti Threat Intelligence Disabled Not Configured
kvm Virtualization Disabled Configured
virtualbox Virtualization Disabled Configured
In testing FAME when unzipped files do not have an extension attached, just the MD5 as the filename. FAME does detect the correct file type through strings or the file command and reports that data in the UI. But, the office_macros module errors on a word document. I'm assuming the error is due to no file extension attached to the file. The file does not have macros, so I guess I would expect the results to indicate that.
Example file: https://www.virustotal.com/en/file/ef9495d4d279e595083a92d044561da1f7c48f8885e98eab2bc745e48cb93028/analysis/, but when you download it, rename to the md5hash with no extension.
Steps to Reproduce
Upload a word document file with no extension inside a zip file.
Expected behavior
Either a message is returned that the document does not contain macros, or appending the extension to the file based on other static data information at processing time.
Actual behavior
The office_macros module errors out and I do not get the macros for further analysis.
Debug
2017-03-28 10:25: debug: Trying to run office_macros
2017-03-28 10:25: error: office_macros: Could not run on /home/disdude/fame/storage/<storage_loc>/a4aac4740b67cdf90f1068353376d28d.
Traceback (most recent call last):
File "/home/disdude/fame/fame/core/module.py", line 471, in _try_each
return self.each_with_type(target, file_type)
File "/home/disdude/fame/fame/core/module.py", line 430, in each_with_type
return self.each(target)
File "/home/disdude/fame/fame/modules/community/processing/office_macros/office_macros.py", line 45, in each
analysis = sorted(analysis, key=lambda type_decoded_encoded: len(type_decoded_encoded[2]), reverse=True)
TypeError: 'NoneType' object is not iterable
2017-03-28 10:25: debug: Done with office_macros
Hello,
Do you plan to add some capacities to delete/cancel/rerun analysis ?
In some case (when misconfiguration) some tasks are staying in 'pending' status. There is a way to delete them or rerun them ?
Best regards,
After loging into fame.. i get the error that : TemplateNotFound: analysis/index.html
I'm looking for suggestions on how to best debug "Could not send to Yeti" errors for Fame observables. I am getting a "200" status code on the Yeti side, but no other indications of errors.
Hints would be appreciated.
[can you help me, i cannot running "utils/run.sh webserver.py" it says " WARNING: Do not use the development server in a production environment." and "utils/run.sh utils/install.py" it says "Could not connect to MongoDB database", thank you.]
[Describe the steps to reproduce]
[How are you expecting the application to behave?]
[How is the application behaving? (include any stacktraces, logs, screenshots, etc.)]
[Include the output of utils/run.sh utils/troubleshoot.py
]
Where to enter the initial user account
Install from git for a development environment
The administrator guide describes a user account entered during setup process.
No user email or account provided.
########## VERSION ##########
OS: Linux-4.9.16-1-lts-x86_64-with-glibc2.2.5
Python: 2.7.13
########## DEPENDENCIES ###########
alabaster==0.7.10
amqp==2.1.4
androguard==3.0
appdirs==1.4.3
Babel==2.3.4
bamfdetect==1.6.13
beautifulsoup4==4.5.3
billiard==3.5.0.2
bs4==0.0.1
celery==4.0.2
click==6.7
docutils==0.13.1
Flask==0.12
Flask-Classy==0.6.10
Flask-Login==0.3.2
Flask-Negotiation==0.1.9
flask-paginate==0.4.5
future==0.16.0
gitdb2==2.0.0
GitPython==2.1.3
googleplay-api==0.1.0
ijson==2.3
imagesize==0.7.1
itsdangerous==0.24
Jinja2==2.9.5
kombu==4.0.2
markdown2==2.3.3
MarkupSafe==1.0
oletools==0.50
packaging==16.8
pbkdf2==1.3
pefile==2016.3.28
protobuf==3.2.0
pycrypto==2.6.1
Pygments==2.2.0
pymongo==3.4.0
pyparsing==2.2.0
python-magic==0.4.13
pytz==2016.10
rarfile==3.0
requests==2.13.0
six==1.10.0
smmap2==2.0.1
snowballstemmer==1.2.1
Sphinx==1.5.3
sphinx-rtd-theme==0.2.4
sphinxcontrib-httpdomain==1.5.0
vine==1.1.3
volatility==2.6
Werkzeug==0.12.1
yara-python==3.5.0
zxcvbn==1.0
########## MongoDB ##########
Version: 3.4.2
Authorization check: True
########## Configuration ##########
Traceback (most recent call last):
File "utils/troubleshoot.py", line 83, in
main()
File "utils/troubleshoot.py", line 80, in main
configuration()
File "utils/troubleshoot.py", line 58, in configuration
for config in Config.find():
File "/home/tnormand/Repositories/github/fame/fame/common/mongo_dict.py", line 24, in find
for obj in objs:
File "/home/tnormand/Repositories/github/fame/env/lib/python2.7/site-packages/pymongo/cursor.py", line 1114, in next
if len(self.__data) or self._refresh():
File "/home/tnormand/Repositories/github/fame/env/lib/python2.7/site-packages/pymongo/cursor.py", line 1036, in _refresh
self.__collation))
File "/home/tnormand/Repositories/github/fame/env/lib/python2.7/site-packages/pymongo/cursor.py", line 928, in __send_message
helpers._check_command_response(doc['data'][0])
File "/home/tnormand/Repositories/github/fame/env/lib/python2.7/site-packages/pymongo/helpers.py", line 210, in _check_command_response
raise OperationFailure(msg % errmsg, code, response)
pymongo.errors.OperationFailure: not authorized on fame to execute command { find: "settings", filter: {} }
After installing and configuring, cannot find the Login page. This is installed in an Ubuntu 16.04 LTS vm that is being used as an offline analysis environment. The documentation doesn't state where the login page will be, so I have used the vm's IP address
Should go to the login page, but no matter what I set the domain to, I don't get a login page
I wouldn't know, I can't even get the login page
[+] Using existing virtualenv.
########## VERSION ##########
OS: Linux-4.8.0-52-generic-x86_64-with-Ubuntu-16.04-xenial
Python: 2.7.12
########## DEPENDENCIES ###########
alabaster==0.7.10
amqp==2.1.4
androguard==3.0.1
Babel==2.4.0
bamfdetect==1.6.13
beautifulsoup4==4.6.0
billiard==3.5.0.2
bs4==0.0.1
celery==4.0.2
certifi==2017.4.17
chardet==3.0.4
click==6.7
docutils==0.13.1
Flask==0.12.2
Flask-Classy==0.6.10
Flask-Login==0.3.2
Flask-Negotiation==0.1.9
flask-paginate==0.4.6
future==0.16.0
gitdb2==2.0.2
GitPython==2.1.3
googleplay-api==0.1.0
idna==2.5
ijson==2.3
imagesize==0.7.1
itsdangerous==0.24
Jinja2==2.9.6
kombu==4.0.2
LEPL==5.1.3
markdown2==2.3.4
MarkupSafe==1.0
oletools==0.50
pbkdf2==1.3
pefile==2016.3.28
protobuf==3.3.0
pycrypto==2.6.1
pyelftools==0.24
Pygments==2.2.0
pymongo==3.4.0
python-magic==0.4.13
pytz==2017.2
rarfile==3.0
requests==2.17.3
-e git+https://github.com/g2p/rfc6266@cad58963ed13f5e1068fcc9e4326123b6b2bdcf8#egg=rfc6266
six==1.10.0
smmap2==2.0.2
snowballstemmer==1.2.1
Sphinx==1.6.2
sphinx-rtd-theme==0.2.4
sphinxcontrib-httpdomain==1.5.0
sphinxcontrib-websupport==1.0.1
typing==3.6.1
urllib3==1.21.1
vine==1.1.3
volatility==2.6
Werkzeug==0.12.2
yara-python==3.6.1
zxcvbn==1.0
########## MongoDB ##########
Version: 3.4.4
Authorization check: True
########## Configuration ##########
types: True
virustotal: False
email: False
malware_config: False
volatility: True
Modules:
McAfee Antivirus Disabled Configured
Sophos Antivirus Disabled Configured
Symantec Antivirus Disabled Not Configured
apk Processing Disabled Configured
apk_verification Processing Disabled Not Configured
bamfdetect Processing Disabled Configured
cuckoo_modified Processing Disabled Configured
cutthecrap Processing Disabled Not Configured
eml Processing Disabled Configured
joe Processing Disabled Not Configured
marcher_config Processing Disabled Configured
mem_yara Processing Disabled Not Configured
office_macros Processing Disabled Configured
pdf Processing Disabled Configured
url_download Processing Disabled Configured
zip Processing Disabled Configured
slack Reporting Disabled Not Configured
Yeti Threat Intelligence Disabled Not Configured
virtualbox Virtualization Disabled Configured
[+] Generating SSH key ... ImportError: No module named rfc6266
During fresh Installation of fame I've encountered the following error:
[+] Generating SSH key ...
Traceback (most recent call last):
File "utils/install.py", line 227, in
main()
File "utils/install.py", line 221, in main
perform_local_installation(context)
File "utils/install.py", line 143, in perform_local_installation
from fame.core import fame_init
File "/home/lacerator/fame/fame/core/init.py", line 2, in
from fame.core.module_dispatcher import dispatcher
File "/home/lacerator/fame/fame/core/module_dispatcher.py", line 8, in
from fame.common.utils import get_class, iterify, unique_for_key
File "/home/lacerator/fame/fame/common/utils.py", line 10, in
from rfc6266 import parse_requests_response
ImportError: No module named rfc6266
I have sudo pip install rfc6266 and still the same issue happens.
I'm running inside a VM using VMware workstation 12
[How are you expecting the application to behave?]
Fame to install without errors.
[How is the application behaving? (include any stacktraces, logs, screenshots, etc.)]
[+] Generating SSH key ...
Traceback (most recent call last):
File "utils/install.py", line 227, in
main()
File "utils/install.py", line 221, in main
perform_local_installation(context)
File "utils/install.py", line 143, in perform_local_installation
from fame.core import fame_init
File "/home/lacerator/fame/fame/core/init.py", line 2, in
from fame.core.module_dispatcher import dispatcher
File "/home/lacerator/fame/fame/core/module_dispatcher.py", line 8, in
from fame.common.utils import get_class, iterify, unique_for_key
File "/home/lacerator/fame/fame/common/utils.py", line 10, in
from rfc6266 import parse_requests_response
ImportError: No module named rfc6266
[Include the output of utils/run.sh utils/troubleshoot.py
]
utils/run.sh utils/troubleshoot.py
[+] Using existing virtualenv.
########## VERSION ##########
OS: Linux-4.4.0-66-generic-x86_64-with-Ubuntu-16.04-xenial
Python: 2.7.12
########## DEPENDENCIES ###########
alabaster==0.7.10
amqp==2.1.4
Babel==2.4.0
billiard==3.5.0.2
celery==4.0.2
certifi==2017.4.17
chardet==3.0.4
click==6.7
docutils==0.13.1
Flask==0.12.2
Flask-Classy==0.6.10
Flask-Login==0.3.2
Flask-Negotiation==0.1.9
flask-paginate==0.5.0
gitdb2==2.0.2
GitPython==2.1.5
idna==2.5
imagesize==0.7.1
itsdangerous==0.24
Jinja2==2.9.6
kombu==4.0.2
LEPL==5.1.3
markdown2==2.3.4
MarkupSafe==1.0
Pygments==2.2.0
pymongo==3.4.0
python-magic==0.4.13
pytz==2017.2
requests==2.18.1
-e git+https://github.com/g2p/rfc6266@cad58963ed13f5e1068fcc9e4326123b6b2bdcf8#egg=rfc6266
six==1.10.0
smmap2==2.0.3
snowballstemmer==1.2.1
Sphinx==1.6.2
sphinx-rtd-theme==0.2.4
sphinxcontrib-httpdomain==1.5.0
sphinxcontrib-websupport==1.0.1
typing==3.6.1
urllib3==1.21.1
vine==1.1.3
Werkzeug==0.12.2
zxcvbn==1.0
########## MongoDB ##########
Version: 3.4.5
Authorization check: True
########## Configuration ##########
Modules:
We use Yeti and FAME with TLS certificates from an internal Certificate Authority. When using the Yeti Threat Intelligence Module, the Yeti module fails because Python requests cannot verify the certificate. Short of not verifying TLS connections, is there a workaround where I can add my CA certificate to FAME?
Use a self-signed or internal CA-signed Yeti connection within FAME.
The Yeti Module should work as expected
I am getting the following error message:
2017-05-09 19:11: error: error in threat intelligence module 'Yeti': [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
Thanks for making FAME available to the community.
Issue1: when trying to run the PROD version of the FE with nginx it outputs a 500.
nginx_access.logx.x.x.x- - [27/Mar/2017:19:44:52 +0200] "GET / HTTP/1.1" 500 32 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:50.0) Gecko/20100101 Firefox/50.0"
nginx_debug.log
2017/03/27 19:52:52 [debug] 2302#2302: *1 uwsgi param: "SERVER_NAME: "
2017/03/27 19:52:52 [debug] 2302#2302: *1 uwsgi param: "HTTP_HOST: hostname:port"
2017/03/27 19:52:52 [debug] 2302#2302: *1 uwsgi param: "HTTP_USER_AGENT: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:50.0) Gecko/20100101 Firefox/50.0"
2017/03/27 19:52:52 [debug] 2302#2302: *1 uwsgi param: "HTTP_ACCEPT: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
2017/03/27 19:52:52 [debug] 2302#2302: *1 uwsgi param: "HTTP_ACCEPT_LANGUAGE: en-US,en;q=0.5"
2017/03/27 19:52:52 [debug] 2302#2302: *1 uwsgi param: "HTTP_ACCEPT_ENCODING: gzip, deflate"
2017/03/27 19:52:52 [debug] 2302#2302: *1 uwsgi param: "HTTP_COOKIE: session=.COOKIE_STRING"
2017/03/27 19:52:52 [debug] 2302#2302: *1 uwsgi param: "HTTP_DNT: 1"
2017/03/27 19:52:52 [debug] 2302#2302: *1 uwsgi param: "HTTP_CONNECTION: keep-alive"
2017/03/27 19:52:52 [debug] 2302#2302: *1 uwsgi param: "HTTP_UPGRADE_INSECURE_REQUESTS: 1"
2017/03/27 19:52:52 [debug] 2302#2302: *1 http cleanup add: 0000561043AC8698
2017/03/27 19:52:52 [debug] 2302#2302: *1 get rr peer, try: 1
2017/03/27 19:52:52 [debug] 2302#2302: *1 stream socket 9
2017/03/27 19:52:52 [debug] 2302#2302: *1 epoll add connection: fd:9 ev:80002005
2017/03/27 19:52:52 [debug] 2302#2302: *1 connect to unix:///tmp/fame.sock, fd:9 #2
2017/03/27 19:52:52 [debug] 2302#2302: *1 connected
2017/03/27 19:52:52 [debug] 2302#2302: *1 http upstream connect: 0
2017/03/27 19:52:52 [debug] 2302#2302: *1 posix_memalign: 0000561043AA8E40:128 @16
2017/03/27 19:52:52 [debug] 2302#2302: *1 http upstream send request
2017/03/27 19:52:52 [debug] 2302#2302: *1 http upstream send request body
2017/03/27 19:52:52 [debug] 2302#2302: *1 chain writer buf fl:0 s:978
2017/03/27 19:52:52 [debug] 2302#2302: *1 chain writer in: 0000561043AC86D0
2017/03/27 19:52:52 [debug] 2302#2302: *1 writev: 978 of 978
2017/03/27 19:52:52 [debug] 2302#2302: *1 chain writer out: 0000000000000000
2017/03/27 19:52:52 [debug] 2302#2302: *1 event timer add: 9: 60000:1490637232929
2017/03/27 19:52:52 [debug] 2302#2302: *1 http finalize request: -4, "/?" a:1, c:2
2017/03/27 19:52:52 [debug] 2302#2302: *1 http request count:2 blk:0
2017/03/27 19:52:52 [debug] 2302#2302: *1 post event 0000561043AEF320
2017/03/27 19:52:52 [debug] 2302#2302: *1 post event 0000561043AEF380
2017/03/27 19:52:52 [debug] 2302#2302: *1 delete posted event 0000561043AEF320
2017/03/27 19:52:52 [debug] 2302#2302: *1 http run request: "/?"
2017/03/27 19:52:52 [debug] 2302#2302: *1 http upstream check client, write event:1, "/"
2017/03/27 19:52:52 [debug] 2302#2302: *1 http upstream recv(): -1 (11: Resource temporarily unavailable)
2017/03/27 19:52:52 [debug] 2302#2302: *1 delete posted event 0000561043AEF380
2017/03/27 19:52:52 [debug] 2302#2302: *1 http upstream request: "/?"
2017/03/27 19:52:52 [debug] 2302#2302: *1 http upstream dummy handler
2017/03/27 19:52:52 [debug] 2302#2302: *1 post event 0000561043AEF380
2017/03/27 19:52:52 [debug] 2302#2302: *1 delete posted event 0000561043AEF380
2017/03/27 19:52:52 [debug] 2302#2302: *1 http upstream request: "/?"
2017/03/27 19:52:52 [debug] 2302#2302: *1 http upstream dummy handler
2017/03/27 19:52:52 [debug] 2302#2302: *1 post event 0000561043ADD370
2017/03/27 19:52:52 [debug] 2302#2302: *1 post event 0000561043AEF380
2017/03/27 19:52:52 [debug] 2302#2302: *1 delete posted event 0000561043ADD370
2017/03/27 19:52:52 [debug] 2302#2302: *1 http upstream request: "/?"
2017/03/27 19:52:52 [debug] 2302#2302: *1 http upstream process header
2017/03/27 19:52:52 [debug] 2302#2302: *1 malloc: 0000561043AAD3F0:4096
2017/03/27 19:52:52 [debug] 2302#2302: *1 recv: fd:9 104 of 4096
2017/03/27 19:52:52 [debug] 2302#2302: *1 http uwsgi status 500 "500 Internal Server Error"
2017/03/27 19:52:52 [debug] 2302#2302: *1 http uwsgi header: "Connection: close"
2017/03/27 19:52:52 [debug] 2302#2302: *1 http uwsgi header: "Content-Type: text/plain"
2017/03/27 19:52:52 [debug] 2302#2302: *1 http uwsgi header done
2017/03/27 19:52:52 [debug] 2302#2302: *1 xslt filter header
2017/03/27 19:52:52 [debug] 2302#2302: *1 HTTP/1.1 500 Internal Server Error
Server: nginx/1.10.0 (Ubuntu)
Date: Mon, 27 Mar 2017 17:52:52 GMT
Content-Type: text/plain
Transfer-Encoding: chunked
Connection: keep-alive
Issue2: when trying to run a remote worker (this time with DEV version because I can't run PROD) I get this on the worker: 500 Server Error: INTERNAL SERVER ERROR for url: http://172.16.20.10:4200/modules/download
I also tried to point the connection to the extranet address I gave to this host but didn't work.
And this on the server:
172.16.20.50 - - [27/Mar/2017 20:01:08] "GET /modules/download HTTP/1.1" 500 -
Traceback (most recent call last):
File "/home/gg/git_code/fame/env/lib/python2.7/site-packages/flask/app.py", line 1994, in __call__
return self.wsgi_app(environ, start_response)
File "/home/gg/git_code/fame/env/lib/python2.7/site-packages/flask/app.py", line 1985, in wsgi_app
response = self.handle_exception(e)
File "/home/gg/git_code/fame/env/lib/python2.7/site-packages/flask/app.py", line 1540, in handle_exception
reraise(exc_type, exc_value, tb)
File "/home/gg/git_code/fame/env/lib/python2.7/site-packages/flask/app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "/home/gg/git_code/fame/env/lib/python2.7/site-packages/flask/app.py", line 1614, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/gg/git_code/fame/env/lib/python2.7/site-packages/flask/app.py", line 1517, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/gg/git_code/fame/env/lib/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/home/gg/git_code/fame/env/lib/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/gg/git_code/fame/env/lib/python2.7/site-packages/flask_classy.py", line 200, in proxy
response = view(**request.view_args)
File "/home/gg/git_code/fame/web/views/helpers.py", line 109, in inner
return func(*args, **kwargs)
File "/home/gg/git_code/fame/web/views/modules.py", line 448, in download
f, path = mkstemp(dir=fame_config.temp_path)
File "/usr/lib/python2.7/tempfile.py", line 314, in mkstemp
return _mkstemp_inner(dir, prefix, suffix, flags)
File "/usr/lib/python2.7/tempfile.py", line 244, in _mkstemp_inner
fd = _os.open(file, flags, 0600)
OSError: [Errno 2] No such file or directory: '/home/gg/git_code/fame/temp/tmpn3cyU8'
I have installed FAME as described on the Documentation web page.
I am running a dedicated instance FE for the WEB server, a dedicated instance DB and trying to run a dedicated remote worker instance.
The only step not done from the Documentation is the DB auth (is this critical? I hope this is not the issue.)
[+] Using existing virtualenv.
########## VERSION ##########
OS: Linux-4.4.0-66-generic-x86_64-with-Ubuntu-16.04-xenial
Python: 2.7.12
########## DEPENDENCIES ###########
alabaster==0.7.10
amqp==2.1.4
appdirs==1.4.3
Babel==2.4.0
billiard==3.5.0.2
celery==4.0.2
click==6.7
docutils==0.13.1
Flask==0.12
Flask-Classy==0.6.10
Flask-Login==0.3.2
Flask-Negotiation==0.1.9
flask-paginate==0.4.5
gitdb2==2.0.0
GitPython==2.1.3
imagesize==0.7.1
itsdangerous==0.24
Jinja2==2.9.5
kombu==4.0.2
markdown2==2.3.3
MarkupSafe==1.0
packaging==16.8
Pygments==2.2.0
pymongo==3.4.0
pyparsing==2.2.0
python-magic==0.4.13
pytz==2016.10
requests==2.13.0
six==1.10.0
smmap2==2.0.1
snowballstemmer==1.2.1
Sphinx==1.5.3
sphinx-rtd-theme==0.2.4
sphinxcontrib-httpdomain==1.5.0
vine==1.1.3
Werkzeug==0.12.1
zxcvbn==1.0
########## MongoDB ##########
Version: 3.4.2
Authorization check: True
########## Configuration ##########
types: True
virustotal: False
email: False
malware_config: False
volatility: True
Modules:
McAfee Antivirus Disabled Configured
Sophos Antivirus Disabled Configured
Symantec Antivirus Disabled Not Configured
apk Processing Enabled Configured
apk_verification Processing Disabled Not Configured
bamfdetect Processing Enabled Configured
cuckoo_modified Processing Disabled Configured
eml Processing Enabled Configured
joe Processing Disabled Not Configured
marcher_config Processing Disabled Configured
mem_yara Processing Enabled Configured
office_macros Processing Enabled Configured
pdf Processing Enabled Configured
url_download Processing Enabled Configured
zip Processing Enabled Configured
slack Reporting Disabled Not Configured
Yeti Threat Intelligence Disabled Not Configured
Could you please help me?
Thanks!
regards.
Guido.
Remote worker's don't appear to be subscribed to the 'updates' queue. When the modules repository is updated from the GUI, it puts the message on the 'messages' Mongo collection, but the remote workers never consume it because of the following code in worker.py:
# A local worker should also take care of updates
if not fame_config.remote:
queues.append('updates')
I had to remove the 'if' statement in order to get remote workers to use the 'updates' queue to sync the new repo.
[when I managed to access the fame, I got into trouble with the detail analysis menu, where when I tried to do the analysis I only get some menu just like: file details, execution path, logs and dimenu path there is still some constraint, when I read in fame documentation there some menu options such as: execution path, observables, extracttions, detailed results, and logs.
roughly for that problem how is the solution? thank you very much.]
[I hope fame that I install can run according to the documentation and can run well]
[I only get some menu just like: file details, execution path, logs ]
[How is the application behaving? (include any stacktraces, logs, screenshots, etc.)]
[Include the output of utils/run.sh utils/troubleshoot.py
]
FAME cannot be installed when using latest pip versions.
Follow the installation manual.
Working installation.
~/fame$ utils/run.sh utils/install.py
[+] Using existing virtualenv.
[+] Installing requirements ...
~/fame$
This is because pip.main
no longer exists, it is now pip._internal.main
. This technique is NOT used only in the install.
We did a quick review of fame and we really like the approach, the overall design and the modularity. As we are working on the object model in MISP (to be released soon), we were wondering if the fame format json output as seen in this example.
Thank you very much
Setting this up in a pretty contained environment with a need to go through the corporate proxy for outbound requests. I'm not seeing a configuration option for this and have been unsuccessful trying to get this to work. Would it be possible to add this option?
[when running the script : "utils/run.sh utils/install.py" i have message 127.0.0.1:27017: [Errno 111] Connection refused
]
[Describe the steps to reproduce]
[can do installation fame]
[How is the application behaving? (include any stacktraces, logs, screenshots, etc.)]
[root@bismillah-VirtualBox:/home/bismillah/fame# utils/run.sh utils/troubleshoot.py
[+] Using existing virtualenv.
########## VERSION ##########
OS: Linux-4.10.0-28-generic-x86_64-with-Ubuntu-16.04-xenial
Python: 2.7.12
########## DEPENDENCIES ###########
alabaster==0.7.11
amqp==2.3.2
Babel==2.6.0
billiard==3.5.0.4
celery==4.1.1
certifi==2018.4.16
chardet==3.0.4
click==6.7
docutils==0.14
Flask==1.0.2
Flask-Classy==0.6.10
Flask-Login==0.3.2
Flask-Negotiation==0.1.9
flask-paginate==0.5.1
gitdb2==2.0.4
GitPython==2.1.11
idna==2.6
imagesize==1.0.0
itsdangerous==0.24
Jinja2==2.10
kombu==4.2.1
LEPL==5.1.3
markdown2==2.3.5
MarkupSafe==1.0
packaging==17.1
Pygments==2.2.0
pymongo==3.6.1
pyparsing==2.2.0
python-magic==0.4.15
pytz==2018.5
requests==2.18.4
rfc6266==0.0.4
six==1.11.0
smmap2==2.0.4
snowballstemmer==1.2.1
Sphinx==1.7.6
sphinx-rtd-theme==0.3.1
sphinxcontrib-httpdomain==1.6.1
sphinxcontrib-websupport==1.1.0
typing==3.6.4
urllib3==1.22
vine==1.1.4
Werkzeug==0.14.1
zxcvbn==1.0
########## MongoDB ##########
Could not connect to MongoDB: localhost:27017: [Errno 111] Connection refused
]
[how to connect modules like apk verification, cuckoo, cuckoo modified, joe, and office_macros with fame, thanks.]
[.]
[Fame can use available modules like apk verification, cuckoo, cuckoo modified, joe, and office_macros and other]
[fame can not use the available modules like apk verification, cuckoo, cuckoo modified, joe, and office_macros and other.]
[root@bismillah-VirtualBox:~/fame# utils/run.sh utils/troubleshoot.py
[+] Using existing virtualenv.
########## VERSION ##########
OS: Linux-4.10.0-28-generic-x86_64-with-Ubuntu-16.04-xenial
Python: 2.7.12
########## DEPENDENCIES ###########
alabaster==0.7.10
amqp==2.3.1
Babel==2.6.0
billiard==3.5.0.3
celery==4.1.1
certifi==2018.4.16
chardet==3.0.4
click==6.7
docutils==0.14
Flask==1.0.2
Flask-Classy==0.6.10
Flask-Login==0.3.2
Flask-Negotiation==0.1.9
flask-paginate==0.5.1
gitdb2==2.0.3
GitPython==2.1.10
idna==2.6
imagesize==1.0.0
itsdangerous==0.24
Jinja2==2.10
kombu==4.2.0
LEPL==5.1.3
markdown2==2.3.5
MarkupSafe==1.0
packaging==17.1
Pygments==2.2.0
pymongo==3.6.1
pyparsing==2.2.0
python-magic==0.4.15
pytz==2018.4
requests==2.18.4
rfc6266==0.0.4
six==1.11.0
smmap2==2.0.3
snowballstemmer==1.2.1
Sphinx==1.7.5
sphinx-rtd-theme==0.3.1
sphinxcontrib-httpdomain==1.6.1
sphinxcontrib-websupport==1.0.1
typing==3.6.4
urllib3==1.22
vine==1.1.4
Werkzeug==0.14.1
zxcvbn==1.0
You are using pip version 10.0.1, however version 18.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
########## MongoDB ##########
Version: 3.6.4
Authorization check: True
########## Configuration ##########
types: True
virustotal: True
email: False
malware_config: False
volatility: True
Modules:
McAfee Antivirus Disabled Configured
Sophos Antivirus Disabled Configured
Symantec Antivirus Disabled Not Configured
apk Processing Enabled Configured
apk_verification Processing Disabled Not Configured
bamfdetect Processing Enabled Configured
cuckoo Processing Enabled Configured
cuckoo_modified Processing Enabled Configured
cutthecrap Processing Disabled Not Configured
eml Processing Enabled Configured
joe Processing Disabled Not Configured
marcher_config Processing Disabled Configured
mem_yara Processing Disabled Not Configured
office_macros Processing Enabled Configured
pdf Processing Enabled Configured
rat_decoders Processing Enabled Configured
url_download Processing Enabled Configured
zip Processing Enabled Configured
slack Reporting Disabled Not Configured
Yeti Threat Intelligence Disabled Not Configured
kvm Virtualization Disabled Configured
virtualbox Virtualization Disabled Configured
root@bismillah-VirtualBox:~/fame#
]
Capability to delete users from the fame portal.
I have verified that Cuckoo was able to download and analyze the file so I'm looking for suggestions as to how to proceed. The parameters for the Cuckoo module are:
WAIT_TIMEOUT 5400
WAIT_STEP 30
ANALYSIS_TIME 300
2018-10-16 08:52: debug: Trying to run cuckoo
2018-10-16 09:05: error: cuckoo: Could not run on http://microsoftupdate.dynamicdns.org.uk/host/290.exe.
Traceback (most recent call last):
File "/home/cirt/fame/fame/core/module.py", line 492, in _try_each
return self.each_with_type(target, file_type)
File "/home/cirt/fame/fame/modules/community/processing/cuckoo/cuckoo.py", line 97, in each_with_type
self.process_report()
File "/home/cirt/fame/fame/modules/community/processing/cuckoo/cuckoo.py", line 159, in process_report
self.extract_info(response)
File "/home/cirt/fame/fame/modules/community/processing/cuckoo/cuckoo.py", line 166, in extract_info
for prefix, event, value in parser:
File "/home/cirt/fame/env/local/lib/python2.7/site-packages/ijson/common.py", line 65, in parse
for event, value in basic_events:
File "/home/cirt/fame/env/local/lib/python2.7/site-packages/ijson/backends/python.py", line 185, in basic_parse
for value in parse_value(lexer):
File "/home/cirt/fame/env/local/lib/python2.7/site-packages/ijson/backends/python.py", line 119, in parse_value
for event in parse_object(lexer):
File "/home/cirt/fame/env/local/lib/python2.7/site-packages/ijson/backends/python.py", line 163, in parse_object
for event in parse_value(lexer, None, pos):
File "/home/cirt/fame/env/local/lib/python2.7/site-packages/ijson/backends/python.py", line 119, in parse_value
for event in parse_object(lexer):
File "/home/cirt/fame/env/local/lib/python2.7/site-packages/ijson/backends/python.py", line 163, in parse_object
for event in parse_value(lexer, None, pos):
File "/home/cirt/fame/env/local/lib/python2.7/site-packages/ijson/backends/python.py", line 116, in parse_value
for event in parse_array(lexer):
File "/home/cirt/fame/env/local/lib/python2.7/site-packages/ijson/backends/python.py", line 138, in parse_array
for event in parse_value(lexer, symbol, pos):
File "/home/cirt/fame/env/local/lib/python2.7/site-packages/ijson/backends/python.py", line 119, in parse_value
for event in parse_object(lexer):
File "/home/cirt/fame/env/local/lib/python2.7/site-packages/ijson/backends/python.py", line 163, in parse_object
for event in parse_value(lexer, None, pos):
File "/home/cirt/fame/env/local/lib/python2.7/site-packages/ijson/backends/python.py", line 116, in parse_value
for event in parse_array(lexer):
File "/home/cirt/fame/env/local/lib/python2.7/site-packages/ijson/backends/python.py", line 138, in parse_array
for event in parse_value(lexer, symbol, pos):
File "/home/cirt/fame/env/local/lib/python2.7/site-packages/ijson/backends/python.py", line 119, in parse_value
for event in parse_object(lexer):
File "/home/cirt/fame/env/local/lib/python2.7/site-packages/ijson/backends/python.py", line 163, in parse_object
for event in parse_value(lexer, None, pos):
File "/home/cirt/fame/env/local/lib/python2.7/site-packages/ijson/backends/python.py", line 119, in parse_value
for event in parse_object(lexer):
File "/home/cirt/fame/env/local/lib/python2.7/site-packages/ijson/backends/python.py", line 170, in parse_object
pos, symbol = next(lexer)
File "/home/cirt/fame/env/local/lib/python2.7/site-packages/ijson/backends/python.py", line 65, in Lexer
data = f.read(buf_size)
File "/home/cirt/fame/env/lib/python2.7/codecs.py", line 488, in read
newdata = self.stream.read(size)
File "/usr/lib/python2.7/socket.py", line 384, in read
data = self._sock.recv(left)
error: [Errno 104] Connection reset by peer
[when I managed to access the fame, I got into trouble with the detail analysis menu, where when I tried to do the analysis I only get some menu just like: file details, execution path, logs and dimenu path there is still some constraint, when I read in fame documentation there some menu options such as: execution path, observables, extracttions, detailed results, and logs.
roughly for that problem how is the solution?
Your help is very much needed, thank you.]
[Describe the steps to reproduce]
[How are you expecting the application to behave?]
[How is the application behaving? (include any stacktraces, logs, screenshots, etc.)]
[Include the output of utils/run.sh utils/troubleshoot.py
]
Hello I like to have fame on my Raspberry Pi 3B but since mongodb is only version v2.4.14 I can't rely on your documentation. Can you explain me how to correctly make the admin user and configuring the databases?
Greetings,
Would it be possible to provide some default values for this configuration ?
I am repeatedly receiving errors for the community modules when following the production installation instructions.
apt-get -qq update
apt-get install git python-pip python-dev p7zip-full
pip install virtualenv
pip install --upgrade pip
cp -v /usr/local/bin/pip /usr/bin/pip
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 9DA31620334BD75D9DCB49F368818C72E52529D4
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/4.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.0.list
apt-get -qq update
apt-get install -y mongodb-org
systemctl enable mongod
systemctl start mongod
mongo
> use admin
> db.createUser({ user: "admin", pwd: "example", roles: [ { role: "userAdminAnyDatabase", db: "admin" } ] })
> use fame
> db.createUser({ user: "fame", pwd: "example", roles: [ { role: "dbOwner", db: "fame" } ] })
security:
authorization: enabled
systemctl restart mongod
cd fame
pip install uwsgi
/etc/systemd/system/fame_web.service
[Unit]
Description=FAME web server
[Service]
Type=simple
ExecStart=/bin/bash -c "cd /fame && uwsgi -H /fame/env --uid root --gid root --socket /tmp/fame.sock --chmod-socket=660 --chown-socket root:www-data -w webserver --callable app"
[Install]
WantedBy=multi-user.target
[Unit]
Description=FAME workers
[Service]
Type=simple
User=root
ExecStart=/bin/bash -c 'cd /fame && utils/run.sh worker.py'
[Install]
WantedBy=multi-user.target
utils/run.sh utils/install.py
systemctl enable fame_web
systemctl enable fame_worker
systemctl start fame_web
systemctl start fame_worker
apt-get install nginx
rm /etc/nginx/sites-enabled/default
/etc/nginx/sites-available/fame
upstream fame {
server unix:///tmp/fame.sock;
}
server {
listen 80 default_server;
# Allows big file upload
client_max_body_size 0;
location / {
include uwsgi_params;
uwsgi_pass fame;
}
location /static/ {
alias /fame/web/static/;
}
}
All modules should not return any errors and should be in a usable state.
apk_verification, cuckoo, cuckoo_modified, joe, and office_macros all return EnvironmentErrors.
/fame/fame/modules/community/processing/apk_verification/requirements.txt: error on 'circle':
Requirement already satisfied: androguard in ./env/lib/python2.7/site-packages (from -r /fame/fame/modules/community/processing/apk_verification/requirements.txt (line 1)) (3.2.1)
Collecting googleplay-api (from -r /fame/fame/modules/community/processing/apk_verification/requirements.txt (line 2))
Using cached https://files.pythonhosted.org/packages/d2/74/e089b2d8b9caf88c7738f631a22fee675db6f17d05df9a9ceec99117f601/googleplay_api-0.1.0.tar.gz
Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/tmp/pip-req-tracker-jACyBx/4bd56651a47a3f4a642aaa5b8ab809c8219f0c6e035325978b12dcc6'
/fame/fame/modules/community/processing/cuckoo/requirements.txt: error on 'circle':
Collecting ijson (from -r /fame/fame/modules/community/processing/cuckoo/requirements.txt (line 1))
Using cached https://files.pythonhosted.org/packages/7f/e9/8508c5f4987ba238a2b169e582c1f70a47272b22a2f1fb06b9318201bb9e/ijson-2.3-py2.py3-none-any.whl
Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/tmp/pip-req-tracker-jACyBx/af051b681629ce9f0f028b1d3e10f6238379d2a9b8f081e8705190c0'
/fame/fame/modules/community/processing/cuckoo_modified/requirements.txt: error on 'circle':
Requirement already satisfied: requests in ./env/lib/python2.7/site-packages (from -r /fame/fame/modules/community/processing/cuckoo_modified/requirements.txt (line 1)) (2.18.4)
Collecting ijson (from -r /fame/fame/modules/community/processing/cuckoo_modified/requirements.txt (line 2))
Using cached https://files.pythonhosted.org/packages/7f/e9/8508c5f4987ba238a2b169e582c1f70a47272b22a2f1fb06b9318201bb9e/ijson-2.3-py2.py3-none-any.whl
Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/tmp/pip-req-tracker-jACyBx/af051b681629ce9f0f028b1d3e10f6238379d2a9b8f081e8705190c0'
/fame/fame/modules/community/processing/joe/requirements.txt: error on 'circle':
Requirement already satisfied: requests in ./env/lib/python2.7/site-packages (from -r /fame/fame/modules/community/processing/joe/requirements.txt (line 1)) (2.18.4)
Collecting ijson (from -r /fame/fame/modules/community/processing/joe/requirements.txt (line 2))
Using cached https://files.pythonhosted.org/packages/7f/e9/8508c5f4987ba238a2b169e582c1f70a47272b22a2f1fb06b9318201bb9e/ijson-2.3-py2.py3-none-any.whl
Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/tmp/pip-req-tracker-jACyBx/af051b681629ce9f0f028b1d3e10f6238379d2a9b8f081e8705190c0'
/fame/fame/modules/community/processing/office_macros/requirements.txt: error on 'circle':
Collecting oletools (from -r /fame/fame/modules/community/processing/office_macros/requirements.txt (line 1))
Using cached https://files.pythonhosted.org/packages/79/f5/9b1a89145ac9bce77c235fee549fc7af617d778bb29af4c8dd1561813a10/oletools-0.53.1.zip
Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/tmp/pip-req-tracker-jACyBx/1242e770b9a28f026a448edb32b189c35bb53e95ba2bae9227a930a0'
bamfdetect returns an AttributeError.
python /fame/fame/modules/community/processing/bamfdetect/install.py: error on 'circle':
Traceback (most recent call last):
File "/fame/fame/modules/community/processing/bamfdetect/install.py", line 8, in <module>
main()
File "/fame/fame/modules/community/processing/bamfdetect/install.py", line 5, in main
pip.main(['install', '--no-deps', 'git+https://github.com/bwall/bamfdetect#egg=BAMF_Detect'])
AttributeError: 'module' object has no attribute 'main'
The other modules do not immediately return an error and I have not tested all of them. The pdf module returns a KeyError, though, when ran against a tested PDF file.
2018-07-24 15:44: debug: Trying to queue module 'pdf'
2018-07-24 15:44: debug: Trying to run pdf
2018-07-24 15:44: error: pdf: Could not run on /fame/storage/4daf4edbb04383b93094e89636f303bb11ab687636a4d40813e930c213c3b513/eicar.pdf.
Traceback (most recent call last):
File "/fame/fame/core/module.py", line 492, in _try_each
return self.each_with_type(target, file_type)
File "/fame/fame/core/module.py", line 450, in each_with_type
return self.each(target)
File "/fame/fame/modules/community/processing/pdf/pdf.py", line 41, in each
if element['vuln_cve_list']:
KeyError: 'vuln_cve_list'
2018-07-24 15:44: debug: Done with pdf
root@circle:/fame# utils/run.sh utils/troubleshoot.py
[+] Using existing virtualenv.
########## VERSION ##########
OS: Linux-4.13.0-36-generic-x86_64-with-Ubuntu-16.04-xenial
Python: 2.7.12
########## DEPENDENCIES ###########
alabaster==0.7.11
amqp==2.3.2
androguard==3.2.1
asn1crypto==0.24.0
Babel==2.6.0
backports.functools-lru-cache==1.5
backports.shutil-get-terminal-size==1.0.0
billiard==3.5.0.4
celery==4.1.1
certifi==2018.4.16
chardet==3.0.4
click==6.7
colorama==0.3.9
cycler==0.10.0
decorator==4.3.0
docutils==0.14
enum34==1.1.6
Flask==1.0.2
Flask-Classy==0.6.10
Flask-Login==0.3.2
Flask-Negotiation==0.1.9
flask-paginate==0.5.1
future==0.16.0
gitdb2==2.0.4
GitPython==2.1.11
idna==2.6
imagesize==1.0.0
ipython==5.7.0
ipython-genutils==0.2.0
itsdangerous==0.24
Jinja2==2.10
kiwisolver==1.0.1
kombu==4.2.1
LEPL==5.1.3
lxml==4.2.3
markdown2==2.3.5
MarkupSafe==1.0
matplotlib==2.2.2
networkx==2.1
numpy==1.15.0
packaging==17.1
pathlib2==2.3.2
pexpect==4.6.0
pickleshare==0.7.4
prompt-toolkit==1.0.15
ptyprocess==0.6.0
pyelftools==0.24
Pygments==2.2.0
pymongo==3.6.1
pyparsing==2.2.0
python-dateutil==2.7.3
python-magic==0.4.15
pytz==2018.5
requests==2.18.4
rfc6266==0.0.4
scandir==1.7
simplegeneric==0.8.1
six==1.11.0
smmap2==2.0.4
snowballstemmer==1.2.1
Sphinx==1.7.6
sphinx-rtd-theme==0.3.1
sphinxcontrib-httpdomain==1.6.1
sphinxcontrib-websupport==1.1.0
subprocess32==3.5.2
traitlets==4.3.2
typing==3.6.4
urllib3==1.22
vine==1.1.4
volatility==2.6
wcwidth==0.1.7
Werkzeug==0.14.1
yara-python==3.7.0
zxcvbn==1.0
########## MongoDB ##########
Version: 4.0.0
Authorization check: True
########## Configuration ##########
types: True
virustotal: False
email: False
malware_config: False
volatility: True
Modules:
McAfee Antivirus Disabled Configured
Sophos Antivirus Disabled Configured
Symantec Antivirus Disabled Not Configured
apk Processing Disabled Configured
apk_verification Processing Disabled Not Configured
bamfdetect Processing Disabled Configured
cuckoo Processing Disabled Configured
cuckoo_modified Processing Disabled Configured
cutthecrap Processing Disabled Not Configured
eml Processing Disabled Configured
joe Processing Disabled Not Configured
marcher_config Processing Disabled Configured
mem_yara Processing Disabled Not Configured
office_macros Processing Disabled Configured
pdf Processing Disabled Configured
rat_decoders Processing Disabled Configured
url_download Processing Disabled Configured
zip Processing Disabled Configured
slack Reporting Disabled Not Configured
Yeti Threat Intelligence Disabled Not Configured
kvm Virtualization Disabled Configured
virtualbox Virtualization Disabled Configured
Feature request
Serve a zip with a simple password would prevent accidentally running malware, when downloading a sample.
I guess this project is being used by a lot of people and it would be nice to have one line installer script or ansible playbook to automate entire installation and configuration process.
Its kind of a feature request, but if it is in scope then I can write it and send a PR
Hi,
Do you know if Fame works on vmware esxi???
Thx
Installing fame and getting error mentioned after SSH key generates
makflwana@ubuntu:~/fame$ sudo utils/run.sh utils/install.py
[+] Using existing virtualenv.
[+] Installing requirements ...
[?] MongoDB host [localhost]:
[?] MongoDB port [27017]:
[?] MongoDB database [fame]:
Choose your installation type:
[?] Installation type [1]: 1
[?] FAME's URL for users (e.g. https://fame.yourdomain/): https://fame.localhost.com
[+] Creating configuration file ...
[+] SSH key already exists.
Traceback (most recent call last):
File "utils/install.py", line 227, in
main()
File "utils/install.py", line 221, in main
perform_local_installation(context)
File "utils/install.py", line 144, in perform_local_installation
fame_init()
File "/home/makflwana/fame/fame/core/init.py", line 6, in fame_init
store.connect()
File "/home/makflwana/fame/fame/core/store.py", line 40, in connect
self.files.create_index([("$**", TEXT)], background=True)
File "/home/makflwana/fame/env/local/lib/python2.7/site-packages/pymongo/collection.py", line 1529, in create_index
self.__create_index(keys, kwargs)
File "/home/makflwana/fame/env/local/lib/python2.7/site-packages/pymongo/collection.py", line 1437, in __create_index
sock_info, index, True, False, False, wcn)
File "/home/makflwana/fame/env/local/lib/python2.7/site-packages/pymongo/collection.py", line 562, in _insert
check_keys, manipulate, write_concern, op_id, bypass_doc_val)
File "/home/makflwana/fame/env/local/lib/python2.7/site-packages/pymongo/collection.py", line 551, in _insert_one
self.__write_response_codec_options)
File "/home/makflwana/fame/env/local/lib/python2.7/site-packages/pymongo/collection.py", line 488, in _legacy_write
rqst_id, msg, max_size, acknowledged)
File "/home/makflwana/fame/env/local/lib/python2.7/site-packages/pymongo/pool.py", line 473, in legacy_write
return helpers._check_gle_response(response)
File "/home/makflwana/fame/env/local/lib/python2.7/site-packages/pymongo/helpers.py", line 250, in _check_gle_response
raise OperationFailure(details["err"], code, result)
pymongo.errors.OperationFailure: text search not enable
Also, I am testing the FAME installation on VM.
This is more of a feature request, but would it be possible to add hyperlinks to the PCAP files that are created during the analysis?
During initial install process ($ utils/run.sh utils/install.py), administrative user can be created with a malformed email address. But upon use in HTTP UI, validations are made and prevent the use of malformed address.
Client side validation can be bypassed by removing "input type=email", and authentication can proceed as expected.
<input type="email" name="email" placeholder="Enter email" class="form-control">
Expected initial account creation to throw warning.
No warning is thrown.
...
[+] Creating first user (as administrator) ...
[?] Full Name: firstuser
[?] Email Address: firstuser
[?] Groups (comma-separated) [cert]:
[?] Password:
[?] Confirm:
[+] User created.
...
nothing to add.
[excusme, i have trouble from update module repositori, and i have message : ValueError: Reserved characters such as ':' must be escaped according RFC 2396. An IPv6 address literal must be enclosed in '[' and ']' according to RFC 2732.
can you help me, thankyou.]
[Describe the steps to reproduce]
[How are you expecting the application to behave?]
[How is the application behaving? (include any stacktraces, logs, screenshots, etc.)]
[Include the output of utils/run.sh utils/troubleshoot.py
]
Hey,
When I tried to run utils/troubleshoot.py, I got the following output.
fame@fame:~/fame$ utils/run.sh utils/troubleshoot.py
[+] Using existing virtualenv.
########## VERSION ##########
OS: Linux-4.4.0-92-generic-x86_64-with-Ubuntu-16.04-xenial
Python: 2.7.12
########## DEPENDENCIES ###########
alabaster==0.7.10
amqp==2.2.1
androguard==3.0.1
Babel==2.4.0
bamfdetect==1.6.13
beautifulsoup4==4.6.0
billiard==3.5.0.3
bs4==0.0.1
celery==4.1.0
certifi==2017.7.27.1
chardet==3.0.4
click==6.7
docutils==0.14
Flask==0.12.2
Flask-Classy==0.6.10
Flask-Login==0.3.2
Flask-Negotiation==0.1.9
flask-paginate==0.5.0
future==0.16.0
gitdb2==2.0.2
GitPython==2.1.5
googleplay-api==0.1.0
idna==2.6
ijson==2.3
imagesize==0.7.1
itsdangerous==0.24
Jinja2==2.9.6
kombu==4.1.0
LEPL==5.1.3
markdown2==2.3.4
MarkupSafe==1.0
oletools==0.51
pbkdf2==1.3
pefile==2017.8.1
protobuf==3.4.0
pycrypto==2.6.1
pyelftools==0.24
Pygments==2.2.0
pymongo==3.5.0
python-magic==0.4.13
pytz==2017.2
rarfile==3.0
requests==2.18.4
rfc6266==0.0.4
six==1.10.0
smmap2==2.0.3
snowballstemmer==1.2.1
Sphinx==1.6.3
sphinx-rtd-theme==0.2.4
sphinxcontrib-httpdomain==1.5.0
sphinxcontrib-websupport==1.0.1
typing==3.6.2
urllib3==1.22
vine==1.1.4
volatility==2.6
Werkzeug==0.12.2
yara-python==3.6.3
zxcvbn==1.0
########## MongoDB ##########
Version: 3.4.7
Authorization check: True
########## Configuration ##########
Traceback (most recent call last):
File "utils/troubleshoot.py", line 83, in
main()
File "utils/troubleshoot.py", line 80, in main
configuration()
File "utils/troubleshoot.py", line 58, in configuration
for config in Config.find():
File "/home/fame/fame/fame/common/mongo_dict.py", line 24, in find
for obj in objs:
File "/home/fame/fame/env/local/lib/python2.7/site-packages/pymongo/cursor.py", line 1134, in next
if len(self.__data) or self._refresh():
File "/home/fame/fame/env/local/lib/python2.7/site-packages/pymongo/cursor.py", line 1057, in _refresh
self.__collation))
File "/home/fame/fame/env/local/lib/python2.7/site-packages/pymongo/cursor.py", line 949, in __send_message
helpers._check_command_response(doc['data'][0])
File "/home/fame/fame/env/local/lib/python2.7/site-packages/pymongo/helpers.py", line 210, in _check_command_response
raise OperationFailure(msg % errmsg, code, response)
pymongo.errors.OperationFailure: not authorized on fame to execute command { find: "settings", filter: {} }
[when i try to configure module i told to enter identity, "Triggered By" to fill in what, and if possible ask for an example ]
[Describe the steps to reproduce]
[can running module]
[How is the application behaving? (include any stacktraces, logs, screenshots, etc.)]
[[+] Using existing virtualenv.
########## VERSION ##########
OS: Linux-4.15.0-29-generic-x86_64-with-Ubuntu-16.04-xenial
Python: 2.7.12
########## DEPENDENCIES ###########
alabaster==0.7.10
amqp==2.3.1
androguard==3.2.1
asn1crypto==0.24.0
Babel==2.6.0
backports.functools-lru-cache==1.5
backports.shutil-get-terminal-size==1.0.0
beautifulsoup4==4.6.0
billiard==3.5.0.3
bs4==0.0.1
celery==4.1.1
certifi==2018.4.16
chardet==3.0.4
click==6.7
colorama==0.3.9
cycler==0.10.0
decorator==4.3.0
docutils==0.14
enum34==1.1.6
Flask==1.0.2
Flask-Classy==0.6.10
Flask-Login==0.3.2
Flask-Negotiation==0.1.9
flask-paginate==0.5.1
future==0.16.0
gitdb2==2.0.3
GitPython==2.1.10
googleplay-api==0.1.0
idna==2.6
ijson==2.3
imagesize==1.0.0
ipython==5.7.0
ipython-genutils==0.2.0
itsdangerous==0.24
Jinja2==2.10
kiwisolver==1.0.1
kombu==4.2.0
LEPL==5.1.3
lxml==4.2.3
markdown2==2.3.5
MarkupSafe==1.0
matplotlib==2.2.2
networkx==2.1
numpy==1.15.0
oletools==0.53.1
packaging==17.1
pathlib2==2.3.2
pbkdf2==1.3
pefile==2017.11.5
pexpect==4.6.0
pickleshare==0.7.4
prompt-toolkit==1.0.15
protobuf==3.6.0
ptyprocess==0.6.0
pycrypto==2.6.1
Pygments==2.2.0
pymongo==3.6.1
pyparsing==2.2.0
python-dateutil==2.7.3
python-magic==0.4.15
pytz==2018.4
rarfile==3.0
requests==2.18.4
rfc6266==0.0.4
scandir==1.7
simplegeneric==0.8.1
six==1.11.0
smmap2==2.0.3
snowballstemmer==1.2.1
Sphinx==1.7.5
sphinx-rtd-theme==0.3.1
sphinxcontrib-httpdomain==1.6.1
sphinxcontrib-websupport==1.0.1
subprocess32==3.5.2
traitlets==4.3.2
typing==3.6.4
urllib3==1.22
vine==1.1.4
virtualenv==16.0.0
volatility==2.6
wcwidth==0.1.7
Werkzeug==0.14.1
yara-python==3.7.0
zxcvbn==1.0
########## MongoDB ##########
Version: 3.6.4
Authorization check: True
########## Configuration ##########
types: True
virustotal: True
email: False
malware_config: False
volatility: True
Modules:
McAfee Antivirus Disabled Configured
Sophos Antivirus Disabled Configured
Symantec Antivirus Disabled Not Configured
apk Processing Enabled Configured
apk_verification Processing Enabled Configured
bamfdetect Processing Disabled Configured
cuckoo Processing Enabled Configured
cuckoo_modified Processing Enabled Configured
cutthecrap Processing Disabled Not Configured
eml Processing Enabled Configured
joe Processing Disabled Not Configured
marcher_config Processing Disabled Configured
mem_yara Processing Disabled Not Configured
office_macros Processing Enabled Configured
pdf Processing Enabled Configured
rat_decoders Processing Enabled Configured
url_download Processing Enabled Configured
zip Processing Enabled Configured
slack Reporting Disabled Not Configured
Yeti Threat Intelligence Disabled Not Configured
kvm Virtualization Disabled Configured
virtualbox Virtualization Disabled Configured ]
[failed to connect the module with fame, confused in placing the creation class for the module]
[I tried to create a python class and sample documentation like on the web, but I am still confused in placing a class, so that it becomes inaccessible.]
[I hope the module and fame can connect and I can use the features that are fame as well as possible.
]
[fame not running]
[root@bismillah-VirtualBox:~/fame# utils/run.sh utils/troubleshoot.py
[+] Using existing virtualenv.
########## VERSION ##########
OS: Linux-4.10.0-28-generic-x86_64-with-Ubuntu-16.04-xenial
Python: 2.7.12
########## DEPENDENCIES ###########
alabaster==0.7.10
amqp==2.3.1
androguard==3.2.1
asn1crypto==0.24.0
Babel==2.6.0
backports.functools-lru-cache==1.5
backports.shutil-get-terminal-size==1.0.0
beautifulsoup4==4.6.0
billiard==3.5.0.3
bs4==0.0.1
celery==4.1.1
certifi==2018.4.16
chardet==3.0.4
click==6.7
colorama==0.3.9
cycler==0.10.0
decorator==4.3.0
docutils==0.14
enum34==1.1.6
Flask==1.0.2
Flask-Classy==0.6.10
Flask-Login==0.3.2
Flask-Negotiation==0.1.9
flask-paginate==0.5.1
future==0.16.0
gitdb2==2.0.3
GitPython==2.1.10
googleplay-api==0.1.0
idna==2.6
ijson==2.3
imagesize==1.0.0
ipython==5.7.0
ipython-genutils==0.2.0
itsdangerous==0.24
Jinja2==2.10
kiwisolver==1.0.1
kombu==4.2.0
LEPL==5.1.3
lxml==4.2.3
markdown2==2.3.5
MarkupSafe==1.0
matplotlib==2.2.2
networkx==2.1
numpy==1.15.0
oletools==0.53.1
packaging==17.1
pathlib2==2.3.2
pbkdf2==1.3
pefile==2017.11.5
pexpect==4.6.0
pickleshare==0.7.4
prompt-toolkit==1.0.15
protobuf==3.6.0
ptyprocess==0.6.0
pycrypto==2.6.1
pyelftools==0.24
Pygments==2.2.0
pymongo==3.6.1
pyparsing==2.2.0
python-dateutil==2.7.3
python-magic==0.4.15
pytz==2018.4
rarfile==3.0
requests==2.18.4
rfc6266==0.0.4
scandir==1.7
simplegeneric==0.8.1
six==1.11.0
smmap2==2.0.3
snowballstemmer==1.2.1
Sphinx==1.7.5
sphinx-rtd-theme==0.3.1
sphinxcontrib-httpdomain==1.6.1
sphinxcontrib-websupport==1.0.1
subprocess32==3.5.2
traitlets==4.3.2
typing==3.6.4
urllib3==1.22
vine==1.1.4
volatility==2.6
wcwidth==0.1.7
Werkzeug==0.14.1
yara-python==3.7.0
zxcvbn==1.0
########## MongoDB ##########
Version: 3.6.4
Authorization check: True
########## Configuration ##########
types: True
virustotal: True
email: False
malware_config: False
volatility: True
Modules:
McAfee Antivirus Disabled Configured
Sophos Antivirus Disabled Configured
Symantec Antivirus Disabled Not Configured
apk Processing Enabled Configured
apk_verification Processing Disabled Not Configured
bamfdetect Processing Disabled Configured
cuckoo Processing Enabled Configured
cuckoo_modified Processing Enabled Configured
cutthecrap Processing Disabled Not Configured
eml Processing Enabled Configured
joe Processing Disabled Not Configured
marcher_config Processing Disabled Configured
mem_yara Processing Disabled Not Configured
office_macros Processing Enabled Configured
pdf Processing Enabled Configured
rat_decoders Processing Enabled Configured
url_download Processing Enabled Configured
zip Processing Enabled Configured
slack Reporting Disabled Not Configured
Yeti Threat Intelligence Disabled Not Configured
kvm Virtualization Disabled Configured
virtualbox Virtualization Disabled Configured]
PyPi package zxcvbn
taken over by zxcvbn-python
(see discussion here).
Pinning zxcvbn==1.0
in requirements.txt
solves the issue.
As of 11 Apr 2018,
$ git clone https://github.com/certsocietegenerale/fame
$ cd fame
$ utils/run.sh utils/install.py
$ utils/run.sh webserver.py
FAME webserver should start.
(env) fame@fame ~/fame $ utils/run.sh webserver.py
[+] Using existing virtualenv.
Traceback (most recent call last):
File "webserver.py", line 20, in <module>
from web.views.users import UsersView
File "/home/fame/fame/web/views/users.py", line 14, in <module>
auth_module = import_module('web.auth.{}.views'.format(fame_config.auth))
File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
File "/home/fame/fame/web/auth/user_password/views.py", line 3, in <module>
from zxcvbn import password_strength
ImportError: cannot import name password_strength
(env) fame@fame ~/fame $ utils/run.sh utils/troubleshoot.py
[+] Using existing virtualenv.
########## VERSION ##########
OS: Linux-4.10.0-38-generic-x86_64-with-LinuxMint-18.3-sylvia
Python: 2.7.12
########## DEPENDENCIES ###########
alabaster==0.7.10
amqp==2.2.2
Babel==2.5.3
billiard==3.5.0.3
celery==4.1.0
certifi==2018.1.18
chardet==3.0.4
click==6.7
docutils==0.14
Flask==0.12.2
Flask-Classy==0.6.10
Flask-Login==0.3.2
Flask-Negotiation==0.1.9
flask-paginate==0.5.1
gitdb2==2.0.3
GitPython==2.1.9
idna==2.6
imagesize==1.0.0
itsdangerous==0.24
Jinja2==2.10
kombu==4.1.0
LEPL==5.1.3
markdown2==2.3.5
MarkupSafe==1.0
packaging==17.1
Pygments==2.2.0
pymongo==3.6.1
pyparsing==2.2.0
python-magic==0.4.15
pytz==2018.4
requests==2.18.4
rfc6266==0.0.4
six==1.11.0
smmap2==2.0.3
snowballstemmer==1.2.1
Sphinx==1.7.2
sphinx-rtd-theme==0.3.0
sphinxcontrib-httpdomain==1.6.1
sphinxcontrib-websupport==1.0.1
typing==3.6.4
urllib3==1.22
vine==1.1.4
Werkzeug==0.14.1
zxcvbn==4.4.25
########## MongoDB ##########
Version: 3.6.4
Authorization check: True
########## Configuration ##########
types: True
virustotal: False
email: False
malware_config: False
volatility: True
Modules:
McAfee Antivirus Disabled Configured
Sophos Antivirus Disabled Configured
Symantec Antivirus Disabled Not Configured
apk Processing Disabled Configured
apk_verification Processing Disabled Not Configured
bamfdetect Processing Disabled Configured
cuckoo Processing Disabled Configured
cuckoo_modified Processing Disabled Configured
cutthecrap Processing Disabled Not Configured
eml Processing Disabled Configured
joe Processing Disabled Not Configured
marcher_config Processing Disabled Configured
mem_yara Processing Disabled Not Configured
office_macros Processing Disabled Configured
pdf Processing Disabled Configured
rat_decoders Processing Disabled Configured
url_download Processing Disabled Configured
zip Processing Disabled Configured
slack Reporting Disabled Not Configured
Yeti Threat Intelligence Disabled Not Configured
kvm Virtualization Disabled Configured
virtualbox Virtualization Disabled Configured
web
web/views
web/views/mixins.py
web/views/helpers.pyc
web/views/constants.pyc
web/views/analyses.py
web/views/analyses.pyc
web/views/users.pyc
web/views/modules.pyc
web/views/search.pyc
web/views/negotiation.py
web/views/constants.py
web/views/init.pyc
web/views/negotiation.pyc
web/views/files.pyc
web/views/search.py
web/views/init.py
web/views/configs.py
web/views/configs.pyc
web/views/modules.py
web/views/helpers.py
web/views/mixins.pyc
web/views/files.py
web/views/users.py
web/init.pyc
web/init.py
web/static
web/static/js
web/static/js/bootstrap-notify.min.js
web/static/js/bootstrap.min.js
web/static/js/fame.js
web/static/js/tagsinput.js
web/static/js/checkbox.js
web/static/js/bootstrap.js
web/static/js/typeahead.bundle.min.js
web/static/js/template.js
web/static/js/npm.js
web/static/js/highlight.min.js
web/static/js/fileinput.min.js
web/static/js/modules-typeahead.js
web/static/js/jquery-1.11.3.min.js
web/static/js/fame-file.js
web/static/js/handlebars.min.js
web/static/fonts
web/static/fonts/Pe-icon-7-stroke.woff
web/static/fonts/Pe-icon-7-stroke.eot
web/static/fonts/fontawesome-webfont.svg
web/static/fonts/fontawesome-webfont.woff2
web/static/fonts/glyphicons-halflings-regular.ttf
web/static/fonts/Pe-icon-7-stroke.svg
web/static/fonts/glyphicons-halflings-regular.woff2
web/static/fonts/Pe-icon-7-stroke.ttf
web/static/fonts/glyphicons-halflings-regular.eot
web/static/fonts/fontawesome-webfont.ttf
web/static/fonts/fontawesome-webfont.woff
web/static/fonts/fontawesome-webfont.eot
web/static/fonts/glyphicons-halflings-regular.woff
web/static/fonts/glyphicons-halflings-regular.svg
web/static/fonts/FontAwesome.otf
web/static/favicon.png
web/static/img
web/static/img/computer_code.jpg
web/static/img/sidebar.jpg
web/static/img/mask.png
web/static/img/avatars
web/static/img/avatars/58da87c8fb99aa13c2f5c47d.png
web/static/img/avatars/default.png
web/static/css
web/static/css/bootstrap.css
web/static/css/fileinput.min.css
web/static/css/bootstrap-theme.css
web/static/css/bootstrap-theme.min.css
web/static/css/template.css
web/static/css/bootstrap.css.map
web/static/css/pe-icon-7-stroke.css
web/static/css/font-awesome.min.css
web/static/css/animate.css
web/static/css/bootstrap-theme.css.map
web/static/css/bootstrap.min.css
web/static/css/fame.css
web/static/css/highlight.railscasts.min.css
web/auth
web/auth/user_password
web/auth/user_password/user_management.py
web/auth/user_password/views.pyc
web/auth/user_password/views.py
web/auth/user_password/init.pyc
web/auth/user_password/init.py
web/auth/user_password/templates
web/auth/user_password/templates/password_reset_form.html
web/auth/user_password/templates/mail_user_creation.html
web/auth/user_password/templates/mail_reset_password.html
web/auth/user_password/templates/auth_profile.html
web/auth/user_password/templates/login.html
web/auth/user_password/templates/base_unauthenticated.html
web/auth/user_password/templates/password_reset.html
web/auth/user_password/user_management.pyc
web/auth/saml
web/auth/saml/user_management.py
web/auth/saml/views.py
web/auth/saml/init.py
web/auth/saml/config
web/auth/saml/config/init.py
web/auth/saml/config/.gitignore
web/auth/saml/config/custom_mappings.py.sample
web/auth/saml/config/settings.json.sample
web/auth/init.pyc
web/auth/init.py
web/templates
web/templates/configs
web/templates/configs/single_block.html
web/templates/configs/target_attributes.html
web/templates/configs/botnet_list.html
web/templates/configs/index.html
web/templates/configs/show.html
web/templates/base.html
web/templates/users
web/templates/users/_form.html
web/templates/users/new.html
web/templates/users/index.html
web/templates/users/profile.html
web/templates/search.html
web/templates/analyses
web/templates/analyses/details.html
web/templates/analyses/new.html
web/templates/analyses/index.html
web/templates/analyses/show.html
web/templates/analyses/_options.html
web/templates/analyses/list.html
web/templates/files
web/templates/files/details.html
web/templates/files/index.html
web/templates/files/show.html
web/templates/files/list.html
web/templates/modules
web/templates/modules/module_configuration.html
web/templates/modules/templates.html
web/templates/modules/configuration.html
web/templates/modules/index.html
web/templates/modules/repository_new.html
web/templates/modules/_configuration.html
After enabling the url_download module, files were only downloadable via the local fame worker with the server. When a remote worker consumed a task to download from the url, the following error occurred:
2018-10-20 21:26: debug: Trying to queue module 'url_download'
2018-10-20 21:26: debug: Trying to run url_download
2018-10-20 21:26: debug: Adding extracted file '/fame/temp/5bd59cca9a6747959a194db4260430e8/com.parental.control.v4.apk'
2018-10-20 21:26: error: url_download: Could not run on http://xxx.xxx.xxx.xxx:8000/com.parental.control.v4.apk.
Traceback (most recent call last):
File "/fame/fame/core/module.py", line 492, in _try_each
return self.each_with_type(target, file_type)
File "/fame/fame/core/module.py", line 450, in each_with_type
return self.each(target)
File "/fame/fame/modules/community/processing/url_download.py", line 35, in each
self.add_extracted_file(filepath)
File "/fame/fame/core/module.py", line 345, in add_extracted_file
self._analysis.add_extracted_file(location)
File "/fame/fame/core/analysis.py", line 90, in add_extracted_file
response = send_file_to_remote(filepath, '/files/')
File "/fame/fame/common/utils.py", line 74, in send_file_to_remote
response.raise_for_status()
File "/fame/env/local/lib/python2.7/site-packages/requests/models.py", line 935, in raise_for_status
raise HTTPError(http_error_msg, response=self)
HTTPError: 500 Server Error: INTERNAL SERVER ERROR for url: http://fame:4200/files/
2018-10-20 21:26: debug: Done with url_download
The FAME server shows a traceback that indicates the files.py post() view function crashed because it was trying to render a template. The fame server is trying to render a template that doesn't exist. This is fixed by adding an "Accept: application/json" header into the requests.post call in send_file_to_remote() in utils.py to indicate that the server should respond to the worker with a JSON encoded payload.
Subsequently, if this change is added in, the core/analysis.py file add_extracted_file() function will need to load the response payload as bson so that it can resolve the File object correctly.
common/utils.py
def send_file_to_remote(file, url):
if isinstance(file, basestring):
file = open(file, 'rb')
url = urljoin(fame_config.remote, url)
response = requests.post(url, files={'file': file}, headers={'X-API-KEY': fame_config.api_key,
'Accept': 'application/json'})
response.raise_for_status()
file.close()
return response
core/analysis.py
from bson.json_util import loads
...
def add_extracted_file(self, filepath):
self.log('debug', "Adding extracted file '{}'".format(filepath))
fd = open(filepath, 'rb')
filename = os.path.basename(filepath)
f = File(filename=filename, stream=fd, create=False)
if not f.existing:
if fame_config.remote:
response = send_file_to_remote(filepath, '/files/')
f = File(loads(response.text)['file'])
else:
f = File(filename=os.path.basename(filepath), stream=fd)
f.analyze(self['groups'], self['analyst'], None, self['options'])
fd.close()
self.append_to('extracted_files', f['_id'])
f.add_parent_analysis(self)
Hope this helps! I have tested this and it works in my forked version (feel free to pull in that code if these changes seem reasonable)
I get the following error message when trying to execute an analysis by selecting Bamfdetect:
2017-04-01 08:42: error: Could not find execution path to target bamfdetect
Submit sample file select bamfdetect as analyser and click submit
Give a result not an error in logs section
Log section of the analysis shows:
2017-04-01 08:48: error: Could not find execution path to target bamfdetect
fame@ubuntu:~/fame$ utils/run.sh utils/troubleshoot.py
[+] Using existing virtualenv.
########## VERSION ##########
OS: Linux-4.4.0-62-generic-x86_64-with-Ubuntu-16.04-xenial
Python: 2.7.12
########## DEPENDENCIES ###########
alabaster==0.7.10
amqp==2.1.4
appdirs==1.4.3
Babel==2.4.0
billiard==3.5.0.2
celery==4.0.2
click==6.7
docutils==0.13.1
Flask==0.12.1
Flask-Classy==0.6.10
Flask-Login==0.3.2
Flask-Negotiation==0.1.9
flask-paginate==0.4.5
gitdb2==2.0.0
GitPython==2.1.3
imagesize==0.7.1
itsdangerous==0.24
Jinja2==2.9.5
kombu==4.0.2
markdown2==2.3.3
MarkupSafe==1.0
packaging==16.8
Pygments==2.2.0
pymongo==3.4.0
pyparsing==2.2.0
python-magic==0.4.13
pytz==2017.2
requests==2.13.0
six==1.10.0
smmap2==2.0.1
snowballstemmer==1.2.1
Sphinx==1.5.3
sphinx-rtd-theme==0.2.4
sphinxcontrib-httpdomain==1.5.0
vine==1.1.3
Werkzeug==0.12.1
zxcvbn==1.0
########## MongoDB ##########
Version: 3.4.3
Authorization check: True
########## Configuration ##########
types: True
virustotal: False
email: False
malware_config: False
volatility: True
Modules:
McAfee Antivirus Disabled Configured
Sophos Antivirus Disabled Configured
Symantec Antivirus Disabled Not Configured
apk Processing Disabled Configured
apk_verification Processing Disabled Not Configured
bamfdetect Processing Enabled Configured
cuckoo_modified Processing Disabled Configured
eml Processing Disabled Configured
joe Processing Disabled Not Configured
marcher_config Processing Disabled Configured
mem_yara Processing Disabled Not Configured
office_macros Processing Disabled Configured
pdf Processing Disabled Configured
url_download Processing Disabled Configured
zip Processing Disabled Configured
slack Reporting Disabled Not Configured
Yeti Threat Intelligence Disabled Not Configured
Got this error in configuration tab for AntiVirus
python /home/USER/fame/fame/modules/community/antivirus/mail/install.py: error on 'fame':
Missing dependency: 7z
Looks like a requirement is need. (Running on fresh install of Ubuntu 16.04)
Please add this package for install:
p7zip-full
Best regards,
[problem in the menu Analysis Details, should there are some menus, for example execution path, observables, extracttions, detail results, and logs and while my menu is only there file details, execution path, logs.]
[Describe the steps to reproduce]
[How are you expecting the application to behave?]
[How is the application behaving? (include any stacktraces, logs, screenshots, etc.)]
[Include the output of utils/run.sh utils/troubleshoot.py
]
Please how do i delete the samples already submitted and lastly a sample i have submited says #PENDING and it been long
I submitted a couple of samples on fame for analysis and then manually stopped the jobs at the cuckoo-modified server. Before all this, I modified the cuckoo-modified configuration and disabled the time-out options. Now both analysis on fame are running perpetually.
Is there a way to cancel these jobs on the fame portal? If not, how to cancel them from the shell?
Thanks in advance for the help and for releasing this tool to the public.
[Please provide a description of the issue encountered]
When submiting a Hash as analysis through API if reulstant hash file is not found FAME returns bit html code instead of json reply.
[Describe the steps to reproduce]
headers = {'Accept': 'application/json',
'X-API-KEY': 'my_api_key'}
params2 = {
'options[allow_internet_access]':'on',
'options[analysis_time]':"300", 'groups':'*','options[tag]':'Honeypot','hash':'8971fc79d73e2541cf5a27e8ad5e971c'
}
r2 = requests.post(submit_url,data=params2,headers=headers,verify=False)
[How are you expecting the application to behave?]
I was expecting json reply similar to
u'{"analysis": {"support_files": {}, "logs": ["2017-07-25 11:23: debug: Trying to queue module \'bamfdetect\'", "2017-07-25 11:23: debug: Trying to queue module \'eml\'", "2017-07-25 11:23: debug: Trying to queue module \'office_macros\'", "2017-07-25 11:23: debug: Trying to queue module \'pdf\'", "2017-07-25 11:23: debug: Trying to queue module \'zip\'", "2017-07-25 11:23: debug: Trying to queue module \'fireeye_ax\'", "2017-07-25 11:23: debug: Trying to queue module \'virustotal_report\'", "2017-07-25 11:23: debug: Trying to queue module \'payload_security\'"], "extractions": [], "results": {}, "module": null, "date": {"$date": 1500981796720}, "file": {"$oid": "59772a24e6c7db09969802f5"}, "iocs": [], "executed_modules": [], "probable_names": [], "extracted_files": [], "status": "pending", "tags": [], "groups": ["*"], "pending_modules": ["fireeye_ax", "virustotal_report", "payload_security"], "analyst": {"$oid": "59663083e6c7db099698027a"}, "waiting_modules": [], "canceled_modules": [], "threat_intelligence": {}, "generated_files": {}, "_id": {"$oid": "59772a24e6c7db09969802f6"}, "options": {}}}'
[How is the application behaving? (include any stacktraces, logs, screenshots, etc.)]
API call returned following html
u'<!doctype html>\n\n\n\t\n\t\n\t\n\n\t<title>FAME</title>\n\n\t<meta content='width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=0' name='viewport' />\n \n\n\n \n \n\n \n \n\n \n \n\n \n \n\n \n \n <link href='https://fonts.googleapis.com/css?family=Roboto:400,700,300\' rel='stylesheet' type='text/css'>\n \n\n \n \n\n \n <script src="/static/js/jquery-1.11.3.min.js" type="text/javascript"></script>\n\t<script src="/static/js/bootstrap.min.js" type="text/javascript"></script>\n\n \n\t<script src="/static/js/template.js"></script>\n <script src="/static/js/checkbox.js"></script>\n\n \n <script src="/static/js/bootstrap-notify.min.js"></script>\n\n \n <script src="/static/js/fame.js"></script>\n\n \n \n <script src="/static/js/fileinput.min.js"></script>\n\n \n <script src="/static/js/highlight.min.js"></script>\n\n \n <script src="/static/js/tagsinput.js"></script>\n\n \n <script src="/static/js/typeahead.bundle.min.js"></script>\n\n \n <script src="/static/js/handlebars.min.js"></script>\n\n\n\n
{{description}}
\n[Include the output of utils/run.sh utils/troubleshoot.py
]
I'm writing a TI module to enrich analysis using an internal data source. I implemented both "ioc_submission" and "ioc_lookup" methods. How can I test this now? Nothing in the interface shows up similar to the "Send to Yeti".
It may be convenient to add a way to test TI modules like processing modules "single_module.py".
Hello,
Is it possible to add a context when we send IOCs to Yeti ?
Best regards,
cant import new self created processing module to FAME
when trying to run "utils/run.sh utils/single_module.py" i get:
[+] Using existing virtualenv.
[+] Enabling test mode.
/!\ Could not find module 'XXX'
i saved the code in "/fame/fame/modules/community/processing" under new folder with the module name
also tried creating new module folder under "/fame/fame/modules/processing"
both times i tried "reload" on "Module Repositories" in configuration page
as i understand a new module should be available in configuration under "processing" in configuration page with the new module name
no new module shown under "processing" in configuration page
[Include the output of utils/run.sh utils/troubleshoot.py
]
[+] Using existing virtualenv.
########## VERSION ##########
OS: Linux-4.4.0-21-generic-x86_64-with-Ubuntu-16.04-xenial
Python: 2.7.12
########## DEPENDENCIES ###########
alabaster==0.7.11
amqp==2.3.2
androguard==3.2.1
asn1crypto==0.24.0
Babel==2.6.0
backports.functools-lru-cache==1.5
backports.shutil-get-terminal-size==1.0.0
bamfdetect==1.6.13
beautifulsoup4==4.6.3
billiard==3.5.0.4
bs4==0.0.1
celery==4.1.1
certifi==2018.8.24
chardet==3.0.4
click==6.7
colorama==0.3.9
cycler==0.10.0
decorator==4.3.0
docutils==0.14
enum34==1.1.6
Flask==1.0.2
Flask-Classy==0.6.10
Flask-Login==0.3.2
Flask-Negotiation==0.1.9
flask-paginate==0.5.1
future==0.16.0
gitdb2==2.0.4
GitPython==2.1.11
googleplay-api==0.1.0
idna==2.6
ijson==2.3
imagesize==1.0.0
ipython==5.8.0
ipython-genutils==0.2.0
itsdangerous==0.24
Jinja2==2.10
kiwisolver==1.0.1
kombu==4.2.1
LEPL==5.1.3
lxml==4.2.4
markdown2==2.3.5
MarkupSafe==1.0
matplotlib==2.2.3
networkx==2.1
numpy==1.15.1
oletools==0.53.1
packaging==17.1
pathlib2==2.3.2
pbkdf2==1.3
pefile==2018.8.8
pexpect==4.6.0
pickleshare==0.7.4
prompt-toolkit==1.0.15
protobuf==3.6.1
ptyprocess==0.6.0
pycrypto==2.6.1
pyelftools==0.25
Pygments==2.2.0
pymongo==3.6.1
pyparsing==2.2.0
python-dateutil==2.7.3
python-magic==0.4.15
pytz==2018.5
rarfile==3.0
requests==2.18.4
rfc6266==0.0.4
scandir==1.9.0
simplegeneric==0.8.1
six==1.11.0
smmap2==2.0.4
snowballstemmer==1.2.1
Sphinx==1.7.8
sphinx-rtd-theme==0.3.1
sphinxcontrib-httpdomain==1.6.1
sphinxcontrib-websupport==1.1.0
subprocess32==3.5.2
traitlets==4.3.2
typing==3.6.6
urllib3==1.22
vine==1.1.4
volatility==2.6
wcwidth==0.1.7
Werkzeug==0.14.1
yara-python==3.8.1
zxcvbn==1.0
########## MongoDB ##########
Version: 4.0.2
Authorization check: True
########## Configuration ##########
types: True
virustotal: True
email: False
malware_config: False
volatility: True
Modules:
McAfee Antivirus Disabled Configured
Sophos Antivirus Disabled Configured
Symantec Antivirus Disabled Not Configured
apk Processing Enabled Configured
apk_verification Processing Disabled Not Configured
bamfdetect Processing Enabled Configured
cuckoo Processing Disabled Configured
cuckoo_modified Processing Disabled Configured
cutthecrap Processing Disabled Not Configured
eml Processing Enabled Configured
joe Processing Disabled Not Configured
marcher_config Processing Disabled Configured
mem_yara Processing Enabled Configured
office_macros Processing Enabled Configured
pdf Processing Enabled Configured
rat_decoders Processing Enabled Configured
url_download Processing Disabled Configured
zip Processing Enabled Configured
slack Reporting Disabled Not Configured
Yeti Threat Intelligence Disabled Not Configured
kvm Virtualization Disabled Configured
virtualbox Virtualization Disabled Configured
When submitting a new sample, the webserver returned an Internal error.
Install from scratch and submission of the sample Shamoon from theZoo repo
[How are you expecting the application to behave?]
20170821-fame-issue_on-submit.txt
[i tried fame installation by following tutorial, but i do not know when i want to try to access fame did not work, can anyone help me?
thank you]
[Describe the steps to reproduce]
[How are you expecting the application to behave?]
[How is the application behaving? (include any stacktraces, logs, screenshots, etc.)]
[Include the output of utils/run.sh utils/troubleshoot.py
]
Hey All,
Anyone get SAML working successfully with Fame?
Thx,
Jim
Hello,
I didn't see this available in any of the models, so I added it to the file model.
diff --git a/fame/core/file.py b/fame/core/file.py
index 288a478..ab8816b 100755
--- a/fame/core/file.py
+++ b/fame/core/file.py
@@ -124,6 +124,7 @@ class File(MongoDict):
self['detailed_type'] = magic.from_file(self['filepath'])
self['mime'] = magic.from_file(self['filepath'], mime=True)
self['analysis'] = []
+ self['size'] = os.path.getsize(self['filepath'])
# Init antivirus status
self['antivirus'] = {}
diff --git a/web/templates/files/details.html b/web/templates/files/details.html
index e6d36dc..ce05576 100755
--- a/web/templates/files/details.html
+++ b/web/templates/files/details.html
@@ -26,6 +26,10 @@
</div>
{% if not file.type == 'url' %}
<div class="row">
+ <div class="col-sm-2 text-right"><strong>File Size</strong></div>
+ <div class="col-sm-10">{{file.size}} bytes</div>
+ </div>
+ <div class="row">
<div class="col-sm-2 text-right"><strong>MD5</strong></div>
<div class="col-sm-10">{{file.md5}}</div>
</div>
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.