osquery / osquery Goto Github PK

SQL powered operating system instrumentation, monitoring, and analytics.

License: Other

C 22.86% C++ 67.57% Thrift 0.03% Objective-C++ 1.43% Shell 0.23% Python 1.70% PowerShell 0.37% Roff 0.03% CMake 5.68% Dockerfile 0.02% Assembly 0.08% Makefile 0.01% Objective-C 0.01%

hacktoberfest intrusion-detection monitoring security sql

osquery's Introduction

osquery

osquery is a SQL powered operating system instrumentation, monitoring, and analytics framework.
Available for Linux, macOS, and Windows.

Information and resources

Homepage: osquery.io
Downloads: osquery.io/downloads
Documentation: ReadTheDocs
Stack Overflow: Stack Overflow questions
Table Schema: osquery.io/schema
Query Packs: osquery.io/packs
Slack: Browse the archives or Join the conversation
Build Status:
CII Best Practices:

What is osquery?

osquery exposes an operating system as a high-performance relational database. This allows you to write SQL-based queries to explore operating system data. With osquery, SQL tables represent abstract concepts such as running processes, loaded kernel modules, open network connections, browser plugins, hardware events or file hashes.

SQL tables are implemented via a simple plugin and extensions API. A variety of tables already exist and more are being written: https://osquery.io/schema. To best understand the expressiveness that is afforded to you by osquery, consider the following SQL queries:

List the users:

SELECT * FROM users;

Check the processes that have a deleted executable:

SELECT * FROM processes WHERE on_disk = 0;

Get the process name, port, and PID, for processes listening on all interfaces:

SELECT DISTINCT processes.name, listening_ports.port, processes.pid
  FROM listening_ports JOIN processes USING (pid)
  WHERE listening_ports.address = '0.0.0.0';

Find every macOS LaunchDaemon that launches an executable and keeps it running:

SELECT name, program || program_arguments AS executable
  FROM launchd
  WHERE (run_at_load = 1 AND keep_alive = 1)
  AND (program != '' OR program_arguments != '');

Check for ARP anomalies from the host's perspective:

SELECT address, mac, COUNT(mac) AS mac_count
  FROM arp_cache GROUP BY mac
  HAVING count(mac) > 1;

Alternatively, you could also use a SQL sub-query to accomplish the same result:

SELECT address, mac, mac_count
  FROM
    (SELECT address, mac, COUNT(mac) AS mac_count FROM arp_cache GROUP BY mac)
  WHERE mac_count > 1;

These queries can be:

performed on an ad-hoc basis to explore operating system state using the osqueryi shell
executed via a scheduler to monitor operating system state across a set of hosts
launched from custom applications using osquery Thrift APIs

Download & Install

To download the latest stable builds and for repository information and installation instructions visit https://osquery.io/downloads.

We use a simple numbered versioning scheme X.Y.Z, where X is a major version, Y is a minor, and Z is a patch. We plan minor releases roughly every two months. These releases are tracked on our Milestones page. A patch release is used when there are unforeseen bugs with our minor release and we need to quickly patch. A rare 'revision' release might be used if we need to change build configurations.

Major, minor, and patch releases are tagged on GitHub and can be viewed on the Releases page. We open a new Release Checklist issue when we prepare a minor release. If you are interested in the status of a release, please find the corresponding checklist issue, and note that the issue will be marked closed when we are finished the checklist. We consider a release 'in testing' during the period of hosting new downloads on our website and adding them to our hosted repositories. We will mark the release as 'stable' on GitHub when enough testing has occurred, this usually takes two weeks.

Build from source

Building osquery from source is encouraged! Check out our build guide. Also check out our contributing guide and join the community on Slack.

Osquery fleet managers

There are many osquery fleet managers out there. The osquery project does not endorse, recommend, or test these. They are provided as a starting point

Project	License
Fleet	Open Core
Kolide	Commercial
OSCTRL	Open Source
Zentral	Open Source

License

By contributing to osquery you agree that your contributions will be licensed as defined on the LICENSE file.

Vulnerabilities

We keep track of security announcements in our tagged version release notes on GitHub. We aggregate these into SECURITY.md too.

Learn more

The osquery documentation is available online. Documentation for older releases can be found by version number, as well.

If you're interested in learning more about osquery read the launch blog post for background on the project, visit the users guide.

Development and usage discussion is happening in the osquery Slack, grab an invite here!

osquery's People

Contributors

Stargazers

Watchers

Forkers

symbiansymoh fspadini nishanthvasudevan networkelements dhanunjaya basicsbeauty simplyahmazing yannick reedloden neuroradiology ruo91 shankarj astanway bruce2008github chubbymaggie zwass chrisrammy jdolitsky vmauge ecin sethkontny congnghia0609 wonderisland zixuanhe sumansai14 uikit0 xjzhou zclfly hitomitanaka raceli imcom barroco59 mmallad robustfengbin msurendra reaperlee yoyzhou dldev0 bill2004158 wmx3ng hu174 trentluo yanlinaung brightcui yetanotherhacker umeshptiwari sechacking rjeczalik friginal tweets magicmonkey konstantinpae emaxerrno eastebry creepig01 tzudot rmoorman crandallbrady angie0angie zogwei vicenteforever sdu2011 ezc jamalahmedmaaz akshaydixi firewash sukelluskello ltd-beget mjdineshm ahmsayat licong techhat nmadhok kartikeyap butangero daheon deniszh mezianehadjadj entroplexity umarags nvdnkpr wheelcomplex qingsing voilet covering made-by-love xiangcao claudiouzelac spectrumjade decli easyfmxu kumargk yfix wlan0 bigrats zofuthan hsin hjwsm1989 johnnylai theopolis

osquery's Issues

Virtual table for the "last" command

Pretty self explanatory, I want to wrap last into a vtable.

NSPropertyListSerialization propertyListWithData doesn't release heap allocated memory

NSPropertyListSerialization propertyListWithData, which is called in plist.mm to do property list parsing doesn't release heap allocated memory that it acquires during normal operations.

Installed Applications virtual table

List all applications (perhaps all dirs ending in .app in /Applications?) with relevant information such as version, vendor, etc.

Networking settings virtual table

Probably a one row table with columns such as:

is promiscuous mode turned on?
if ip forwarding enabled?
what is the IP and MAC of the default gateway?
what are the DNS nameservers?

eapoclient virtual table

On OS X, expose eapoclient configurations so that we can ensure that /Library/Preferences/SystemConfiguration/com.apple.network.eapolclient.configuration.plist isn't maliciously configured

running processes which no longer exist on disk vtable

Lets play find the shell. After an attacker logs in, runs it, rm's it and it stays running.

Combine this with the lsof table and the system updates table, and you have running processes which no longer exist on disk which have established web sockets to non rfc1918 hosts which were not part of a system update or original build.

Full-disk Encryption virtual table

It would be nice to have a virtual table which exposes the configurations of the current state of OS X's full-disk encryption.

Open file handles (ala lsof) vtable

Scenario: pwned remotely and sshd replaced/ preloaded/tampered with in whatever way.

A way to determine which copies of ssh accross a fleet of machines that have an open backdoor socket to allthepwns-whatever-attacker-host.com.

select * from open_file_handles where socket_inet_endpoint LIKE 'someip:%'

Virtual table for all active shells

Including information exposed via ps -ef

Create infrastructure for creating packages with a private set of scheduled queries

Crontab virtual table

Platform agnostic virtual table which exposes the data in cron.

Especially useful on OS X since nobody is supposed to use cron on OS X anyway.

User login (su, ssh, interactive, etc) virtual table

Record time, (optional source), requested user, success/failure for user logins. SSH is a great place to start.

/etc/hosts virtual table

Vtable which exposes the entries in /etc/hosts

Process parent-child relationship virtual table

Hard request, but when a process forks or spawns a child, record the PID, CID and for context the process name/argv data (not integral information).

QuarantineEvents Virtual table

http://www.forensicswiki.org/wiki/Mac_OS_X#Quarantine_event_database

Easy post-phish payload location.

Browser History virtual table

This would be useful for performing forensics: a forensically sound way of acquiring a host's browser history. Can probably use libtsk for this.

Sleuth Kit timeline virtual table

A table which uses libtsk to generate a sleuth kit timeline

alf_explicit_auths table not returning results

Returns empty strings:

osquery> select * from alf_explicit_auths;
processes
----------

cacerts table has int casting error

osquery> SELECT common_name FROM cacerts WHERE not_valid_before = -1;
W0821 17:34:13.723083 2072097552 cacerts.cpp:173] Error casting 4248201426to int
common_name
---------------------------------------------
UTN-USERFirst-Client Authentication and Email
osquery>

When a bad_lexical_cast is thrown in the generated vtable, the int is replaced with a -1.

Investigate the performance of the disk serializer

Serializing into a HistroicalQurtyResult type is slower than it should be. Make sure the scheme is correct. If necessary, break it into multiple keys in the column family and use thrift's serialization to store it (http://techblog.ridewithvia.com/post/37118173720/take-care-when-serializing-and-deserializing-thrift).

Apple Firewall virtual table

This is a virtual table which aggregates the data in /Library/Preferences/com.apple.alf.plist. This is going to be really similar to #3 and can probably re-use a lot of functionality from there.

Columns should include relevant plist keys in com.apple.alf.plist such as:

AllowSignedEnabled
FirewallUnload
GlobalState
LoggingEnabled
PreviousOnState
StealthEnabled
Version
Exceptions
ExplicitAuths
Applications

Create a mechanism to leave out privacy violating tables from the daemon

Certain tables like browser history, browser cookies, etc. can be seen as having a huge potential to violate user privacy if an org were to schedule queries to persistently query those tables. If certain vtables are only to be used in the event of forensic investigation / compromise, we shouldn't build those tables into the osqueryd daemon, which would ensure that they couldn't be used in scheduled queries.

Systems updates vtable

To see what updates are installed on a host

LaunchAgent/Daemon virtual table

Create a virtual table specifically for OS X which analyzes LaunchAgents and LaunchDaemons on a host.

Each row in the table should represent a LaunchAgent/Daemon where each column represents a LA/D key:

RunAtLoad
WatchPaths
KeepAlive
StartInterval
StartOnMount
OnDemand
QueueDirectories
StandardInPath
StandardOutPath
StandardErrorPath
Debug
LaunchOnlyOnce
Sockets
OSAXHandlers
LSEnvironment
CFBundleVersion

In addition to the above, there should be a column for the following:

The Program key
A hash of the executable which the program key points to
The ProgramArguments key
A hash of the executable which the ProgramArguments key points to

For ultimate ballerness, this should be implemented in Objective-C++.

Consider changing name of cacerts table

Could it be ca_certs? system_ca_certs? Something that signifies where these specific certificate authorities are trusted?

NVRAM variables virtual table

NVRAM can be organized into named variables nvram -p, changes include editing default boot (OSX), to EFI security variables (SecureBoot). There are also other system and OS specific ways to store/query NVRAM or SPI data such as the efivars sysfs /sys/firmware/efi/efivars.

Memory leak in the scheduler

valgrind --tool=memcheck --leak-check=yes ./build/osqueryd --config_path=/Users/marpaia/Desktop/osquery.conf

==71163== 10,808 bytes in 1 blocks are definitely lost in loss record 567 of 567
==71163==    at 0x6DFB: malloc (in /usr/local/Cellar/valgrind/3.9.0/lib/valgrind/vgpreload_memcheck-amd64-darwin.so)
==71163==    by 0x6866E5: gmtsub (in /usr/lib/system/libsystem_c.dylib)
==71163==    by 0x10001FC1C: boost::date_time::c_time::gmtime(long const*, tm*) (in ./build/osqueryd)
==71163==    by 0x10001FAE5: boost::date_time::microsec_clock<boost::posix_time::ptime>::create_time(tm* (*)(long const*, tm*)) (in ./build/osqueryd)
==71163==    by 0x10001FA23: boost::date_time::microsec_clock<boost::posix_time::ptime>::universal_time() (in ./build/osqueryd)
==71163==    by 0x10001F9FC: boost::asio::time_traits<boost::posix_time::ptime>::now() (in ./build/osqueryd)
==71163==    by 0x10001F90C: boost::asio::detail::deadline_timer_service<boost::asio::time_traits<boost::posix_time::ptime> >::expires_from_now(boost::asio::detail::deadline_timer_service<boost::asio::time_traits<boost::posix_time::ptime> >::implementation_type&, boost::posix_time::time_duration const&, boost::system::error_code&) (in ./build/osqueryd)
==71163==    by 0x10001D43E: boost::asio::deadline_timer_service<boost::posix_time::ptime, boost::asio::time_traits<boost::posix_time::ptime> >::expires_from_now(boost::asio::detail::deadline_timer_service<boost::asio::time_traits<boost::posix_time::ptime> >::implementation_type&, boost::posix_time::time_duration const&, boost::system::error_code&) (in ./build/osqueryd)
==71163==    by 0x10001D32A: boost::asio::basic_deadline_timer<boost::posix_time::ptime, boost::asio::time_traits<boost::posix_time::ptime>, boost::asio::deadline_timer_service<boost::posix_time::ptime, boost::asio::time_traits<boost::posix_time::ptime> > >::basic_deadline_timer(boost::asio::io_service&, boost::posix_time::time_duration const&) (in ./build/osqueryd)
==71163==    by 0x10001D014: boost::asio::basic_deadline_timer<boost::posix_time::ptime, boost::asio::time_traits<boost::posix_time::ptime>, boost::asio::deadline_timer_service<boost::posix_time::ptime, boost::asio::time_traits<boost::posix_time::ptime> > >::basic_deadline_timer(boost::asio::io_service&, boost::posix_time::time_duration const&) (in ./build/osqueryd)
==71163==    by 0x10001C397: osquery::scheduler::initialize() (in ./build/osqueryd)
==71163==    by 0x1000092A5: boost::detail::thread_data<void (*)()>::run() (in ./build/osqueryd)

Configure TravisCI

I want to be able to run all of the projects tests on all supported platforms with one command. Presumably something like make test, which would do something like:

# run tests on OS X
mkdir -p build/osx
cd build/osx && cmake ../.. && make -j5
find build/osx -name "*_tests" -type f -exec '{}' \; 

# build the virtual machines via vagrant, provision them,
# build osquery and run the tests on there as well
vagrant provision

To start, I'd like to support the following operating systems:

OS X
Ubuntu 14.04 (via the Vagrant chef/ubuntu-14.04 box)
CentOS 6.5 (via the Vagrant chef/centos-6.5 box)

Note that robust Chef cookbooks will also be helpful for users of/contributors to the open source product.

System-trusted root certificates

In OSX changes to the "System Roots"/Certificates keychain, on Linux /etc/ssl/certs/* or where ca-bundle.crt and friends live.

Improve the sqlite shell's display callbacks

Modify the sqlite shell's shell_exec function to accept an optional additional pointer. Right now, shell_exec is called with xCallback = shell_callback. Modify shell_callback to create a vector of rows and modify shell_exec to handle another callback which should print the content of the vector more nicely than how shell_exec currently does.

Currently, output looks like this:

osquery> select path, pid from processes order by pid limit 5;
path           pid
-------------  ----------
/sbin/launchd  1
/usr/libexec/  13
/usr/libexec/  14
/usr/libexec/  15
/usr/sbin/not  16

Notice how 13 characters are printed for path because the length of /sbin/launchd is 13. I would rather this look more like:

osquery> select path, pid from processes order by pid limit 5;
+---------------------+ +-------+
|        path         | |  pid  |
+---------------------+ +-------+
| /sbin/launchd       | | 1     |
| /usr/libexec/foobar | | 13    |
| /usr/libexec/afile  | | 14    |
| /usr/libexec/baz    | | 15    |
| /usr/sbin/charlie   | | 16    |
+---------------------+ +-------+

Create LaunchDaemon

Memory leak in nvram table

Figure out a more reliable way to build libtsk

I just built libtsk and the java bindings caused a compilation error because of a jar not existing somewhere on the internet. Luckily, I didn't need the Java bindings, so I edited the Makefile to not compile the bindings by removing $(JAVA_BINDINGS) from line 282 (SUBDIRS = tsk tools tests samples man $(UNIT_TESTS) $(JAVA_BINDINGS)).

Poor experience if this is going to be a core part of osquery.

Filesystems virtual table

This should expose information about the filesystem that are loaded on the host.

Are there ramdisks loaded?
NFS?
etc

Memory leak in RocksDB DBHandle class

valgrind --tool=memcheck --leak-check=yes ./build/osqueryd --config_path=/Users/marpaia/Desktop/osquery.conf

==71163== 16 bytes in 1 blocks are definitely lost in loss record 102 of 567
==71163==    at 0x6DFB: malloc (in /usr/local/Cellar/valgrind/3.9.0/lib/valgrind/vgpreload_memcheck-amd64-darwin.so)
==71163==    by 0x45028D: operator new(unsigned long) (in /usr/lib/libc++.1.dylib)
==71163==    by 0x13B701: std::__1::__hash_table<std::__1::__hash_value_type<unsigned int, void (*)(void*)>, std::__1::__unordered_map_hasher<unsigned int, std::__1::__hash_value_type<unsigned int, void (*)(void*)>, std::__1::hash<unsigned int>, true>, std::__1::__unordered_map_equal<unsigned int, std::__1::__hash_value_type<unsigned int, void (*)(void*)>, std::__1::equal_to<unsigned int>, true>, std::__1::allocator<std::__1::__hash_value_type<unsigned int, void (*)(void*)> > >::__rehash(unsigned long) (in /usr/local/lib/librocksdb.dylib)
==71163==    by 0x13B537: std::__1::__hash_table<std::__1::__hash_value_type<unsigned int, void (*)(void*)>, std::__1::__unordered_map_hasher<unsigned int, std::__1::__hash_value_type<unsigned int, void (*)(void*)>, std::__1::hash<unsigned int>, true>, std::__1::__unordered_map_equal<unsigned int, std::__1::__hash_value_type<unsigned int, void (*)(void*)>, std::__1::equal_to<unsigned int>, true>, std::__1::allocator<std::__1::__hash_value_type<unsigned int, void (*)(void*)> > >::__node_insert_unique(std::__1::__hash_node<std::__1::__hash_value_type<unsigned int, void (*)(void*)>, void*>*) (in /usr/local/lib/librocksdb.dylib)
==71163==    by 0x13B27E: std::__1::unordered_map<unsigned int, void (*)(void*), std::__1::hash<unsigned int>, std::__1::equal_to<unsigned int>, std::__1::allocator<std::__1::pair<unsigned int const, void (*)(void*)> > >::operator[](unsigned int const&) (in /usr/local/lib/librocksdb.dylib)
==71163==    by 0x13A98F: rocksdb::ThreadLocalPtr::StaticMeta::SetHandler(unsigned int, void (*)(void*)) (in /usr/local/lib/librocksdb.dylib)
==71163==    by 0x5B2C5: rocksdb::ColumnFamilyData::ColumnFamilyData(unsigned int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, rocksdb::Version*, rocksdb::Cache*, rocksdb::ColumnFamilyOptions const&, rocksdb::DBOptions const*, rocksdb::EnvOptions const&, rocksdb::ColumnFamilySet*) (in /usr/local/lib/librocksdb.dylib)
==71163==    by 0x5C57D: rocksdb::ColumnFamilySet::ColumnFamilySet(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, rocksdb::DBOptions const*, rocksdb::EnvOptions const&, rocksdb::Cache*) (in /usr/local/lib/librocksdb.dylib)
==71163==    by 0xBFE2F: rocksdb::VersionSet::VersionSet(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, rocksdb::DBOptions const*, rocksdb::EnvOptions const&, rocksdb::Cache*) (in /usr/local/lib/librocksdb.dylib)
==71163==    by 0x6A646: rocksdb::DBImpl::DBImpl(rocksdb::DBOptions const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) (in /usr/local/lib/librocksdb.dylib)
==71163==    by 0x8135F: rocksdb::DB::Open(rocksdb::DBOptions const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<rocksdb::ColumnFamilyDescriptor, std::__1::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::__1::vector<rocksdb::ColumnFamilyHandle*, std::__1::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**) (in /usr/local/lib/librocksdb.dylib)
==71163==    by 0x10008CF8E: osquery::db::DBHandle::DBHandle(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool) (in ./build/osqueryd)
==71163==
==71163== 32 bytes in 1 blocks are definitely lost in loss record 176 of 567
==71163==    at 0x6DFB: malloc (in /usr/local/Cellar/valgrind/3.9.0/lib/valgrind/vgpreload_memcheck-amd64-darwin.so)
==71163==    by 0x45028D: operator new(unsigned long) (in /usr/lib/libc++.1.dylib)
==71163==    by 0x13B25E: std::__1::unordered_map<unsigned int, void (*)(void*), std::__1::hash<unsigned int>, std::__1::equal_to<unsigned int>, std::__1::allocator<std::__1::pair<unsigned int const, void (*)(void*)> > >::operator[](unsigned int const&) (in /usr/local/lib/librocksdb.dylib)
==71163==    by 0x13A98F: rocksdb::ThreadLocalPtr::StaticMeta::SetHandler(unsigned int, void (*)(void*)) (in /usr/local/lib/librocksdb.dylib)
==71163==    by 0x5B2C5: rocksdb::ColumnFamilyData::ColumnFamilyData(unsigned int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, rocksdb::Version*, rocksdb::Cache*, rocksdb::ColumnFamilyOptions const&, rocksdb::DBOptions const*, rocksdb::EnvOptions const&, rocksdb::ColumnFamilySet*) (in /usr/local/lib/librocksdb.dylib)
==71163==    by 0x5C57D: rocksdb::ColumnFamilySet::ColumnFamilySet(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, rocksdb::DBOptions const*, rocksdb::EnvOptions const&, rocksdb::Cache*) (in /usr/local/lib/librocksdb.dylib)
==71163==    by 0xBFE2F: rocksdb::VersionSet::VersionSet(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, rocksdb::DBOptions const*, rocksdb::EnvOptions const&, rocksdb::Cache*) (in /usr/local/lib/librocksdb.dylib)
==71163==    by 0x6A646: rocksdb::DBImpl::DBImpl(rocksdb::DBOptions const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) (in /usr/local/lib/librocksdb.dylib)
==71163==    by 0x8135F: rocksdb::DB::Open(rocksdb::DBOptions const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<rocksdb::ColumnFamilyDescriptor, std::__1::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::__1::vector<rocksdb::ColumnFamilyHandle*, std::__1::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**) (in /usr/local/lib/librocksdb.dylib)
==71163==    by 0x10008CF8E: osquery::db::DBHandle::DBHandle(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool) (in ./build/osqueryd)
==71163==    by 0x10008C593: osquery::db::DBHandle::DBHandle(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool) (in ./build/osqueryd)
==71163==    by 0x10008D4AD: osquery::db::DBHandle::getInstance(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool) (in ./build/osqueryd)

valgrind --tool=memcheck --leak-check=yes ./build/osquery/database/db_handle_tests

==70748== 2,048 bytes in 1 blocks are definitely lost in loss record 279 of 289
==70748==    at 0x74F9: calloc (in /usr/local/Cellar/valgrind/3.9.0/lib/valgrind/vgpreload_memcheck-amd64-darwin.so)
==70748==    by 0x59100A: strerror (in /usr/lib/system/libsystem_c.dylib)
==70748==    by 0xD9662: rocksdb::(anonymous namespace)::PosixEnv::GetChildren(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >*) (in /usr/local/lib/librocksdb.dylib)
==70748==    by 0x3CAD6: rocksdb::DBImpl::FindObsoleteFiles(rocksdb::DBImpl::DeletionState&, bool, bool) (in /usr/local/lib/librocksdb.dylib)
==70748==    by 0x3F1F4: rocksdb::DBImpl::DeleteObsoleteFiles() (in /usr/local/lib/librocksdb.dylib)
==70748==    by 0x52C99: rocksdb::DB::Open(rocksdb::DBOptions const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<rocksdb::ColumnFamilyDescriptor, std::__1::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::__1::vector<rocksdb::ColumnFamilyHandle*, std::__1::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**) (in /usr/local/lib/librocksdb.dylib)
==70748==    by 0x100067EFE: osquery::db::DBHandle::DBHandle(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool) (in ./build/osquery/database/db_handle_tests)
==70748==    by 0x100067503: osquery::db::DBHandle::DBHandle(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool) (in ./build/osquery/database/db_handle_tests)
==70748==    by 0x10006841D: osquery::db::DBHandle::getInstance(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool) (in ./build/osquery/database/db_handle_tests)
==70748==    by 0x100068610: osquery::db::DBHandle::getInstanceAtPath(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) (in ./build/osquery/database/db_handle_tests)
==70748==    by 0x100003AF9: osquery::db::DBHandleTests_test_create_new_database_on_disk_Test::TestBody() (in ./build/osquery/database/db_handle_tests)
==70748==    by 0x10005D4A2: void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (in ./build/osquery/database/db_handle_tests)

Add backslashes to the spaces in the path in the program column of the launchd table

High performance executables virtual table

It would be nice to have a way to say something like SELECT * FROM all_executables_on_the_host;

On OS X this could perhaps done by using the mdfind API. Try running the following in a terminal: mdfind kMDItemContentType==public.unix-executable

installed browser plugins virtual table

An OS X specific virtual table that will allow one to query upon which browser plugins are installed on a host.

Virtual table for the "last" command

Make the command history into a virtual table

Startup Items virtual table

Let's create a virtual table which displays all of the items that will be executed when a host boots.

For OS X, there's a neat tool which already does this: https://github.com/synack/knockknock. We should port it's functionality to C++ and get this in there.

Listening network ports/processes virtual table

Table should protocol, port, pid and optionally pid name/path.

Loaded kernel extensions virtual table

This virtual table should expose the kernel extensions that are currently loaded on a host. This should be cross-platform, but let's focus on OS X for now.

This virtual table should have the following columns:

address
size
wired
name
signing entity

Network routing table (IPv4/IPv6) virtual table

The table will include details (network, gateway, interface, metric) for the v4 and v6 routing tables.

Much more than just the default interface, this can be used to debug/ID VPN issues, also noted here: #10

"Strings" virtual table

Expose the output of the "strings" command via a virtual table.

Add more introspection into NSData items in property lists

Follow up from #58.

Property Lists are allowed to contain data of the type NSData on OS X. Unfortunately, when a plist contains NSData, it cannot be serialized into JSON, because it's not valid JSON (or whatever). filterPlist performs some runtime introspection into the parsed plist data structure and if it identifies any data as being NSData, it overwrites it with an NSString which has the content "NSData".

Let's fix that so that some introspection can be performed to retrieve the NSData's contents.

Memory leak occurs when linking against the Objective-C runtime.

Given code that (literally) just links against osquery, here are some interesting valgrind results:

With neither RocksDB or Obj-C:

==62958== LEAK SUMMARY:
==62958==    definitely lost: 0 bytes in 0 blocks
==62958==    indirectly lost: 0 bytes in 0 blocks
==62958==      possibly lost: 0 bytes in 0 blocks
==62958==    still reachable: 4,096 bytes in 1 blocks
==62958==         suppressed: 25,768 bytes in 377 blocks

With RocksDB:

==69688== LEAK SUMMARY:
==69688==    definitely lost: 0 bytes in 0 blocks
==69688==    indirectly lost: 6,240 bytes in 6 blocks
==69688==      possibly lost: 0 bytes in 0 blocks
==69688==    still reachable: 6,992 bytes in 188 blocks
==69688==         suppressed: 26,768 bytes in 371 blocks

With RocksDB and Obj-C:

==64333== LEAK SUMMARY:
==64333==    definitely lost: 1,656 bytes in 35 blocks
==64333==    indirectly lost: 7,376 bytes in 10 blocks
==64333==      possibly lost: 2,576 bytes in 53 blocks
==64333==    still reachable: 111,364 bytes in 750 blocks
==64333==         suppressed: 193,892 bytes in 464 blocks

==71163== 56 bytes in 1 blocks are definitely lost in loss record 324 of 567
==71163==    at 0x6DFB: malloc (in /usr/local/Cellar/valgrind/3.9.0/lib/valgrind/vgpreload_memcheck-amd64-darwin.so)
==71163==    by 0x45028D: operator new(unsigned long) (in /usr/lib/libc++.1.dylib)
==71163==    by 0x100145AEC: int osquery::tables::sqlite3_attach_vtable<osquery::tables::sqlite3_etc_hosts>(sqlite3*, char const*, sqlite3_module const*) (in ./build/osqueryd)
==71163==    by 0x100145A7C: osquery::tables::etcHostsTablePlugin::attachVtable(sqlite3*) (in ./build/osqueryd)
==71163==    by 0x100141BF0: osquery::tables::attachVirtualTables(sqlite3*) (in ./build/osqueryd)
==71163==    by 0x1000372EB: osquery::core::createDB() (in ./build/osqueryd)
==71163==
==71163== 128 bytes in 1 blocks are definitely lost in loss record 355 of 567
==71163==    at 0x6DFB: malloc (in /usr/local/Cellar/valgrind/3.9.0/lib/valgrind/vgpreload_memcheck-amd64-darwin.so)
==71163==    by 0x45028D: operator new(unsigned long) (in /usr/lib/libc++.1.dylib)
==71163==    by 0x10014D68C: int osquery::tables::sqlite3_attach_vtable<osquery::tables::sqlite3_interface_addresses>(sqlite3*, char const*, sqlite3_module const*) (in ./build/osqueryd)
==71163==    by 0x10014D61C: osquery::tables::interfaceAddressesTablePlugin::attachVtable(sqlite3*) (in ./build/osqueryd)
==71163==    by 0x100141BF0: osquery::tables::attachVirtualTables(sqlite3*) (in ./build/osqueryd)
==71163==    by 0x1000372EB: osquery::core::createDB() (in ./build/osqueryd)
==71163==
==71163== 176 bytes in 1 blocks are definitely lost in loss record 362 of 567
==71163==    at 0x6DFB: malloc (in /usr/local/Cellar/valgrind/3.9.0/lib/valgrind/vgpreload_memcheck-amd64-darwin.so)
==71163==    by 0x45028D: operator new(unsigned long) (in /usr/lib/libc++.1.dylib)
==71163==    by 0x1001606AC: int osquery::tables::sqlite3_attach_vtable<osquery::tables::sqlite3_routes>(sqlite3*, char const*, sqlite3_module const*) (in ./build/osqueryd)
==71163==    by 0x10016063C: osquery::tables::routesTablePlugin::attachVtable(sqlite3*) (in ./build/osqueryd)
==71163==    by 0x100141BF0: osquery::tables::attachVirtualTables(sqlite3*) (in ./build/osqueryd)
==71163==    by 0x1000372EB: osquery::core::createDB() (in ./build/osqueryd)
==71163==
==71163== 296 bytes in 1 blocks are definitely lost in loss record 370 of 567
==71163==    at 0x6DFB: malloc (in /usr/local/Cellar/valgrind/3.9.0/lib/valgrind/vgpreload_memcheck-amd64-darwin.so)
==71163==    by 0x45028D: operator new(unsigned long) (in /usr/lib/libc++.1.dylib)
==71163==    by 0x10015450C: int osquery::tables::sqlite3_attach_vtable<osquery::tables::sqlite3_interface_details>(sqlite3*, char const*, sqlite3_module const*) (in ./build/osqueryd)
==71163==    by 0x10015449C: osquery::tables::interfaceDetailsTablePlugin::attachVtable(sqlite3*) (in ./build/osqueryd)
==71163==    by 0x100141BF0: osquery::tables::attachVirtualTables(sqlite3*) (in ./build/osqueryd)
==71163==    by 0x1000372EB: osquery::core::createDB() (in ./build/osqueryd)
==71163==
==71163== 1,232 (80 direct, 1,152 indirect) bytes in 1 blocks are definitely lost in loss record 478 of 567
==71163==    at 0x6DFB: malloc (in /usr/local/Cellar/valgrind/3.9.0/lib/valgrind/vgpreload_memcheck-amd64-darwin.so)
==71163==    by 0x45028D: operator new(unsigned long) (in /usr/lib/libc++.1.dylib)
==71163==    by 0x1001645AC: int osquery::tables::sqlite3_attach_vtable<osquery::tables::sqlite3_time>(sqlite3*, char const*, sqlite3_module const*) (in ./build/osqueryd)
==71163==    by 0x10016453C: osquery::tables::timeTablePlugin::attachVtable(sqlite3*) (in ./build/osqueryd)
==71163==    by 0x100141BF0: osquery::tables::attachVirtualTables(sqlite3*) (in ./build/osqueryd)
==71163==    by 0x1000372EB: osquery::core::createDB() (in ./build/osqueryd)
==71163==