Giter Club home page Giter Club logo

vfeed's Introduction

**CHECK THE NEW LIBRARY **https://github.com/vfeedio/pyvfeed

vFeed The Correlated Vulnerability and Threat Intelligence Database Wrapper

vFeed

Build Status Code Health Compatibility Compatibility Compatibility

vFeed Python Wrapper / Database is a CVE, CWE, and OVAL Compatible naming scheme concept that provides extra structured detailed third-party references and technical characteristics for a CVE entry through an extensible XML/JSON schema. It also improves the reliability of CVEs by providing a flexible and comprehensive vocabulary for describing the relationship with other standards and security references.

vFeed API generates a JSON-based format outputs to describe in detail vulnerabilities. It can be leveraged as input by security researchers, practitioners, and tools as part of their vulnerability description. The standard syntax is easy to interpret by humans and systems.

The mandatory associated vFeed DB (The Correlated Vulnerability and Threat Intelligence Database) is a detective and preventive security information repository used for gathering vulnerability and mitigation data from scattered internet sources into an unified database. The vFeed DB must be obtained directly from vFeed IO.

  • Open security standards:

  • Vulnerability Assessment & Exploitation IDs (Metasploit, SAINT Corporation, Tenable's Nessus Plugin IDs, Nmap, Exploit-DB)

  • Vendors Security Alerts:

    • Microsoft MS
    • Mandriva
    • Redhat
    • Cisco
    • Sun
    • Gentoo
    • Ubuntu
    • And more ...

Key features

  • Registered as CVE, CWE, and OVAL Compatible by the Mitre Corporation
  • Support Open Standards such as CVE, CPE, CWE, CAPEC, WASC, CVSS and more
  • Downloadable as SQLite database
  • Support correlation with 3rd-party security references IAVA, OVAL and more
  • Support correlation with security assessment and patch vendors (Nessus, Exploit-DB, Redhat, Microsoft..)
  • Easy and ready-to-use python Wrapper

More features at vFeed, Inc.

Target Audience

  • Penetration testers who want to analyze CVEs and gather extra information to help shape avenues to exploit vulnerabilities.
  • Security auditors who want to report accurate information about findings. vFeed could be the best way to describe a CVE with attributes based on standards and 3rd party references as vendors or companies involved into standarization efforts.
  • Security tools vendors / security open source developers who need to implement libraries to enumerate useful information about CVEs without wasting time to correlate and to create a proprietary database. vFeed is by far the best solution. Methods can be invoked from programs or scripts with a simple call.
  • Any security hacker who is conducting research and needs a very fast and accurate way to enumerate available exploits or techniques to check a vulnerability.

How to ?

Run vfeedcli.py -h for help. Refer to the Documentation official documentation page.

Latest release

1.0

0.7.2.1

  • [Fix] Fixed Migrate() module (SQLite to MongoDB). Thanks to Thiago Palmeira from Infolink for reporting the bug.

0.7.2

  • Added support to CAPEC v2.10. Check the full changelog.
  • Added support to CWE v2.11. Check the full changelog.
  • Added support to the new Microsoft security update
  • [Improve] Improved the get_ms method to returns both all and new Microsoft bulletins and KBs.
  • [Improve] Fixed issue #65. Cleaned the database from Reject entries.
  • [Doc] Documentation updated to reflect the new changes. All changes are immediate for consultancy / integrator license customers. The CE database will be available by the end of the month

0.7.1

  • [New] Reactivated the ability to automate the download process for Consultancy / Integrator plans using private Dropbox repository.
  • [Improve] Improved the mongo.py to check whether SQLite exists. Thanks to Alex Faraino (https://github.com/AlexFaraino/vFeed)
  • [Fix] Modified vfeedcli from API to wrapper.
  • [Doc] Documentation updated to reflect the new changes.

vfeed's People

Contributors

andresriancho avatar savon-noir avatar toolswatch avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

vfeed's Issues

Raised exception in exportXML with a missed CVEs

tornado:vFeed Tornado$ python vFeedAPI_calls_2.py exportXML CVE-2013-1950
[warning] Entry CVE-2013-1950 missed from vFeed Database
[warning] Entry CVE-2013-1950 missed from vFeed Database
Traceback (most recent call last):
File "vFeedAPI_calls_2.py", line 313, in
main()
File "vFeedAPI_calls_2.py", line 305, in main
vfeed.exportXML()
File "/Users/Tornado/Documents/Working/Coding/vFeedPython/beta001/vFeed/lib/vFeedApi.py", line 483, in exportXML
{'published' : self.cveInfo['publishedDate'],

Search when vfeed.db is missing

WARNING - vfeed.db is missing.
[+] Querying information for daktronics ...
Traceback (most recent call last):
File "./vfeedcli.py", line 38, in
Search(args.search)
File "/Users/dev/Documents/vFeed/test/vFeed/lib/core/search.py", line 18, in init
self.detect_entry()
File "/Users/dev/Documents/vFeed/test/vFeed/lib/core/search.py", line 38, in detect_entry
self.search_summary()
File "/Users/dev/Documents/vFeed/test/vFeed/lib/core/search.py", line 162, in search_summary
('%' + self.entry + '%',))
sqlite3.OperationalError: no such table: nvd_db

It will be fixed

Can I separate vFeed.db and client into two hosts?

Like the title mentioned, I want to host a local version of vFeed.db. In the meanwhile, many of instances will install just a vfeed client. These clients will periodically query the local vFeed.db on another host. My local version of vFeed.db will be updated periodically through a cron job. Could you give me some suggestions regarding the setup steps for vfeed? It seems like current vfeed.db and vfeed client can not be separated into different hosts.

Thanks,

Su Zhang

Tagged release?

Hello,
It would be great to make a tagged release in git for the next releases.
In kali we have tools that monitors web pages listing release, and it works well with github pages showing git tags. If you make a tagged release we will be automatically informed and we will update the package quickly.
Thanks

UnicodeEncodeError with get_cve()

from vfeed import vFeed

vfeed = vFeed("CVE-2003-0497")
cve_info = vfeed.get_cve()

./vfeedcli.py update

python test.py
Traceback (most recent call last):
File "test.py", line 4, in
cve_info = vfeed.get_cve()
File "/home/mikko.lehtisalo/vFeed/vfeed/api.py", line 66, in get_cve
self.cveInfo['summary'] = str(self.data[3])
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 4: ordinal not in range(128)

Add a Stats Table

KPIs as
Total CVE
Total MSF sploits
Total Sploits (MSF + exploitdb + SaintExploits)
Total unique CPE (+ most vulnerable CPEs etc etc
Total CWE addressed (most CWE ...)
Total top vulnerable applications ...
Total Nessus scripts
Total Patchs ....
etc etc

Not so pythonic

There are a couple of things about the API which I dislike:

  • vFeed shouldn't take a CVE as init parameter. The best way to architecture this is:
    • Have a vFeedConnection class which takes an optional vfeed.db object as init param
    • vFeedConnection has a search method, which takes a CVE id and returns a Vulnerability object
    • The vulnerability object has all the information CVE, CWE, CPE, MS, etc. already set as attributes
    • The user gets the information from the vulnerability.
vfeed_conn = vFeedConnection()
vulnerability = vfeed_conn.search('CVE-...')
print vulnerability.get_MS()

Please note that I also propose a change from "checkMS" to "get_MS" (and the same for all the other "check*" calls.

Finally, the "get_MS" / "get_KB" shouldn't return dictionaries, it is very ugly to access the result of the query using result['foo'], I would prefer result.foo (see https://pypi.python.org/pypi/easydict). Also achievable with namedtuples.

You can go ahead and create different issues for fixing these, or simply ignore all.

Exit from parent script when CVE is missed

When I try to search a CVE that is not on vFeed DB, using api on python script, the process is killed.
Debuggin the code I saw that on line 46 in /lib/common/database.py you do a sys.exit command, that kill also the parent python process.

code:

from vfeed.lib.core.methods import CveInfo
CveInfo( "CVE-2015-6038" ).get_cve( )
print "This print not work! Process killed before" 

By commenting the line 46 the script works.

Bug when exporting info of CVE-2013-2465

./vfeedcli.py export CVE-2013-2465
Traceback (most recent call last):
File "./vfeedcli.py", line 376, in
main()
File "./vfeedcli.py", line 364, in main
vfeed.export()
File "/Users/toolswatch/Documents/coding/test/vFeed/vfeed/exportxml.py", line 397, in export
'file': self.redhat_oval_url + self.REDHAT_id[i]['oval'].split('oval:com.redhat.rhsa:def:')[1] + '.xml',
IndexError: list index out of range

Changing methods names

According to a discussion and proposal made by Andres Riancho. It"s better having method names as follow

get_cve instead of checkCVE
get_msf instead of checkCVSS
get_capec instead of checkCAPEC
get_cpe and so on

I agree. This is somehow more appropriate way to deal with methods.

What do you think ?

It should be implemented with v0.4.0

saves the database to current directory versus somewhere better like ~/.vfeed/

The database is saved to and used from the current directory versus somewhere better like ~/.vfeed/. This hampers several things, but most notably putting a symlink somewhere in my path to be able to call/use vfeedcli.py from any directory (cd ~/bin; ln -s path/to/vFeed/vfeedcli.py vfeed) like a system-wide command.

Btw, I have a patch for this already and will sent a PR shortly.

Expand CheckEDB to handle edb database

CheckEDB is limited today to the cross references supplied by CVE/NVD.

The best approach is to scan for edb exploit database files and extract CVE + any other information (as well as OSVDB, exploit name, exploit file etc etc).

ERROR UPDATING DATABASE

Hi, i am using vfeed on Kali Linux and i am trying to update using: python ./vfeedcli.py --update but it reports to me:

python vfeedcli.py --update
[+] Checking connectivity to http://www.toolswatch.org/vfeed/
[+] New install. Downloading the Correlated Vulnerability Database.
[+] Receiving 5709824 out of 45563168 Bytes of vfeed.db.tgz ( 13 %)Traceback (most recent call last):
File "vfeedcli.py", line 40, in
Update().update()
File "/home/criestlav/Downloads/vFeed-master/lib/core/update.py", line 34, in update
self.download(self.remote_db)
File "/home/criestlav/Downloads/vFeed-master/lib/core/update.py", line 63, in download
self.buffer = self.u.read(self.block_sz)
File "/usr/lib/python2.7/socket.py", line 380, in read
data = self._sock.recv(left)
File "/usr/lib/python2.7/httplib.py", line 561, in read
s = self.fp.read(amt)
File "/usr/lib/python2.7/socket.py", line 380, in read
data = self._sock.recv(left)
socket.error: [Errno 104] Connection reset by peer

What does it mean?

Thank you a lot!!

vFeed database building tools

At the moment the excellent vFeed queries information from an sqlite database that is updated periodically but there are no tools for constructing the database from scratch.

This means that if the database provider takes a holiday/goes rogue/loses interest the information provided by vFeed becomes stale/misleading/stranded. Additionally it would allow users to have a quicker database update period if they have the resources to do so...

Sanitize queries

Andres Riancho suggested to sanitize the queries against SQL injection (not for local use but a next online web based vFeed db)

I added a small patch to get CVE details as following

query = (myCVE,)
cur.execute('SELECT * FROM nvd_db WHERE cveid=?', query)
data = cur.fetchone()

The code is working. Need to recheck if SQLi is sanitized.

Otherwise, any smart way to construct secure queries ?

ps : not mandatory for now.

checkCVE function crash with 'CVE-2010-2343' parameter

In version 0.3.5, a vFeed instance with CVE-2010-2343 parameter raise a python error:

import vFeedApi
VF=vFeedApi.vFeed('CVE-2010-2343')
VF.checkCVE()
Traceback (most recent call last):
File "", line 1, in
File "vFeedApi.py", line 107, in checkCVE
self.cveInfo['cveDescription'] = str(self.data[3])
IndexError: tuple index out of range

For now, this is the only CVE when I have found this problem.
There is a description available for this CVE on http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2010-2343, so the error seems not coming from a lack of information.

Add full support to Metasploit

Add new method checkMSF() to verify the availability of Metasploit Exploits (File, script name, description ...).
The mapper should scan the whole metasploit directory and parse exploit to extract the relevant information.

It will expand the simplified method (extracting Metasploit reference from NVD xml)

Core vFeed code

Hi Nabil,

I wish to know if the core vFeed source code will be available soon.

Thanks

Ambrose

distribution request for Blackarch.org

I am a developer for blackarch.org, we are building an unoffiical repo for archlinux for pentesters. I had considered adding this tool as soon as I saw it on toolswatch.org. I have a pkgbuild already built for it, I will let you know if I have any issues. Thanks alot for developing this tool.

Difficult to do "vulnerable prior version" searches

When using vFeed's search to check for NTP vulnerabilities a difficult case occurs:

# ./vfeedcli.py search cpe:/a:ntp:ntp:4.2.6

[+] Querying information for cpe:/a:ntp:ntp:4.2.6 ...
     [-] Total Unique CVEs        [0] 
     [-] Total Found CPEs         [0] 
[+] Nothing found in the database

# ./vfeedcli.py search CVE-2014-9295

[+] Querying information for CVE-2014-9295 ...
    [-] Multiple stack-based buffer overflows in ntpd in NTP before 4.2.8 allow remote attackers to execute arbitrary code via a crafted packet, related to (1) the crypto_recv function when the Autokey Authentication feature is used, (2) the ctl_putdata function, and (3) the configure function.

[INFO] Try vfeedcli.py export CVE-2014-9295 for more information !!

# ./vfeedcli.py search cpe:/a:ntp:ntp:4.2.7

[+] Querying information for cpe:/a:ntp:ntp:4.2.7 ...
     [-] Total Unique CVEs        [5] 
     [-] Total Found CPEs         [1] 
[+] Gathering information ... 
    [-] cpe:/a:ntp:ntp:4.2.7
        [-] CVE-2013-5211 | CVSS Base :5.0
            [->] The monlist feature in ntp_request.c in ntpd ....
            [!] 1 Metasploit Exploit(s) Found
            [!] 1 Exploit DB sploits Found
        [-] CVE-2014-9293 | CVSS Base :5.0
            [->] The config_auth function in ntpd in NTP ....
        [-] CVE-2014-9294 | CVSS Base :5.0
            [->] util/ntp-keygen.c in ntp-keygen in NTP before 4.2.7p230 ....
        [-] CVE-2014-9295 | CVSS Base :7.5
            [->] Multiple stack-based buffer overflows in ntpd in ....
        [-] CVE-2014-9296 | CVSS Base :5.0
            [->] The receive function in ntp_proto.c in ntpd ....

(http://www.eecis.udel.edu/~ntp/ntp_spool/ntp4/NEWS also confirms there are issues that affect 4.2.6).

It would be useful if there was a search feature that allowed you to find out if vulnerabilities affecting later versions also affect the version you are looking at...

Database should not exit program if CVE is not found

Hello,

I am using vFeed as a Python library and it's working great, but I think it should raise an exception if it does not find a CVE, instead of sys.exit() the program.

In my case, my workflow follows this process:

  1. Get CVE from scanner
  2. Try to find said CVE in vFeed
  3. Do something if found, do something else if not
  4. Continue happily with other CVEs

The try-catch workflow fails with current code because because vFeed exits the whole program when not finding sometrhing...

Have a good day!

Minimal bug in exploit-db link

Due to the new change i have made to exploit-db, I merged the IDs and the Link.

It will be fixed. I'll modify the map_cve_exploitdb table in vfeed.db and add a new column (id, script, cve).

But the great news is that get_edb reports now the link to the exploit file platforms/hp-ux/local/28984.pl (inside the archive file downloaded from http://exploit-db.com/)

download vFeed.db.tgz gives 404 error

I attempted executing the vFeed_update.py as well as several different manual download attempts. All efforts generate a 404 file not found error. Please advise.

vfeed.db not updated for 12 days

I cannot find any post related to the update frequency of vfeed.db. Right now, on June 4th, the last vfeed.db is dated May 23rd. Is this project still active or should I be patient and be aware of the release timing for vfeed.db?

So the issue here is that I have a feeling that this project is not active because there's no post (or wiki text) specifying the update frequency of the db (and it's already close to 2 weeks old).

Release dates and searching via date

I looked through the database tables and some schema contain release dates like the Redhat table. However, tables like the Microsoft table don't. It would be awesome if all the tables could have a release date field for when the vulnerability was released to the public and also a way to search via release date from the API and CLI tool. I'm not sure how you are compiling your database but if you need any help doing this I'd be happy to help.

--update stops at 12%

$ python vfeedcli.py -u
[+] Checking connectivity to http://www.toolswatch.org/vfeed/
[+] New install. Downloading the Correlated Vulnerability Database.
[+] Receiving 5709824 out of 47720679 Bytes of vfeed.db.tgz ( 12 %)

Seems to stop at 12% , retried a few times and in 3 attempts has not got past 12%

Using Mac OSX 10.11.3

Also tried on CentOS 6.7:

[root@localhost vFeed]# python vfeedcli.py --update
[+] Checking connectivity to http://www.toolswatch.org/vfeed/
[+] New install. Downloading the Correlated Vulnerability Database.
[+] Receiving 5701632 out of 47720679 Bytes of vfeed.db.tgz

vfeed.db updates

Quick question. How often is the master vfeed database updated? I tried to pull up cve's from today and three days ago after updating but kept getting warning messages about the cve's not being found in the database.

Output:
r@14TLS:~/vFeed$ python vfeedcli.py update
[info] checking for the latest vfeed.db
[progress 100 %] receiving 49 out of 49 Bytes of update.dat
[info] You have the latest vfeed.db vulnerability database
[info] Cleaning compressed database and update file

r@14TLS:~/vFeed$ python vfeedcli.py get_cve CVE-2014-3437
[warning] Entry CVE-2014-3437 is missed from vFeed Database
[hint] Update your local vfeed.db

Search for Category or Products

a cool enchantment could be the possibility to search CVE -> Suricata Rules by the category or the products object of the vulnerability.

like for example: get_ssid/cve "Apache 2*"

could be a good improvment?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.