ericsson / ecaudit Goto Github PK

View Code? Open in Web Editor NEW

43.0 14.0 38.0 1.91 MB

Ericsson Audit plug-in for Apache Cassandra

License: Apache License 2.0

Java 97.25% Shell 2.75%

cassandra java audit hacktoberfest

ecaudit's Introduction

ecAudit

With ecAudit you get auditing and query logger functionality for Apache Cassandra 3.0 and 3.11.

Features include:

Detailed audit records for CQL operations and login attempts
Customizable audit record format
Obfuscation of sensitive password information
Powerful filtering with centralized or local whitelists
Logger backends with SLF4J/Logback or Chronicle

Example of audit records created by ecAudit:

15:42:41.644 - client:'127.0.0.1'|user:'cassandra'|status:'ATTEMPT'|operation:'INSERT INTO ecks.ectbl (partk, clustk, value) VALUES (?, ?, ?)[1, '1', 'valid']'
15:42:41.646 - client:'127.0.0.1'|user:'cassandra'|status:'ATTEMPT'|operation:'SELECT * FROM ecks.ectbl WHERE partk = ?[1]'
15:42:41.650 - client:'127.0.0.1'|user:'cassandra'|status:'ATTEMPT'|operation:'DELETE FROM ecks.ectbl WHERE partk = ?[1]'
15:42:41.651 - client:'127.0.0.1'|user:'cassandra'|status:'ATTEMPT'|operation:'INSERT INTO ecks.ectbl (partk, clustk, value) VALUES (?, ?, ?)[2, '2', 'valid']'
15:42:41.653 - client:'127.0.0.1'|user:'cassandra'|status:'ATTEMPT'|operation:'SELECT * FROM ecks.ectbl WHERE partk = ?[2]'
15:42:41.655 - client:'127.0.0.1'|user:'cassandra'|status:'ATTEMPT'|operation:'DELETE FROM ecks.ectbl WHERE partk = ?[2]'

Checkout the detailed description for a more comprehensive list of features, limitations and operational impact.

Getting Started

ecAudit integrates with Apache Cassandra using its existing plug-in points.

Download

Official releases of ecAudit can be downloaded from Maven Central. Get the ecAudit flavor for your Cassandra version.

For a detailed description of compatible Cassandra versions, refer to the Cassandra Compatibility Matrix.

Maintenance

The following flavors of ecAudit are in maintenance mode and will get critical fixes, but no new features.

Setup

Install and configure ecAudit using the setup guide for your Cassandra version.

Upgrade

Upgrade ecAudit using then upgrade guide for your Cassandra version

Issues & Contributions

Report an issue if you're having trouble to use ecAudit or have an idea for an improvement.

Want to contribute to ecAudit? Check out our contribution guide.

Credits

The following developers have contributed to the ecAudit project:

Per Otterström
Tobias Eriksson
Laxmikant Upadhyay
Anuj Wadhera
Marcus Olsson
Ted Petersson
Pushpendra Rajpoot
Tommy Stendahl

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

ecaudit's People

Contributors

Stargazers

Watchers

ecaudit's Issues

Exception encountered during startup

ERROR [main] 2019-02-16 00:25:25,789 CassandraDaemon.java:708 - Exception encountered during startup
java.lang.NoSuchMethodError: org.apache.cassandra.cql3.statements.ParsedStatement.prepare()Lorg/apache/cassandra/cql3/statements/ParsedStatement$Prepared;
at com.ericsson.bss.cassandra.ecaudit.auth.WhitelistDataAccess.prepare(WhitelistDataAccess.java:273) ~[ecaudit_c3.0.11-2.1.0-SNAPSHOT.jar:na]
at com.ericsson.bss.cassandra.ecaudit.auth.WhitelistDataAccess.setup(WhitelistDataAccess.java:91) ~[ecaudit_c3.0.11-2.1.0-SNAPSHOT.jar:na]
at com.ericsson.bss.cassandra.ecaudit.filter.role.RoleAuditFilter.setup(RoleAuditFilter.java:59) ~[ecaudit_c3.0.11-2.1.0-SNAPSHOT.jar:na]
at com.ericsson.bss.cassandra.ecaudit.facade.DefaultAuditor.setup(DefaultAuditor.java:45) ~[ecaudit_c3.0.11-2.1.0-SNAPSHOT.jar:na]
at com.ericsson.bss.cassandra.ecaudit.AuditAdapter.setup(AuditAdapter.java:82) ~[ecaudit_c3.0.11-2.1.0-SNAPSHOT.jar:na]
at com.ericsson.bss.cassandra.ecaudit.auth.AuditPasswordAuthenticator.setup(AuditPasswordAuthenticator.java:87) ~[ecaudit_c3.0.11-2.1.0-SNAPSHOT.jar:na]
at org.apache.cassandra.service.StorageService.doAuthSetup(StorageService.java:1091) ~[apache-cassandra-3.11.2.jar:3.11.2]
at org.apache.cassandra.service.StorageService.finishJoiningRing(StorageService.java:1081) ~[apache-cassandra-3.11.2.jar:3.11.2]
at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1003) ~[apache-cassandra-3.11.2.jar:3.11.2]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:682) ~[apache-cassandra-3.11.2.jar:3.11.2]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:613) ~[apache-cassandra-3.11.2.jar:3.11.2]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:379) [apache-cassandra-3.11.2.jar:3.11.2]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:602) [apache-cassandra-3.11.2.jar:3.11.2]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691) [apache-cassandra-3.11.2.jar:3.11.2]

Add support for post-logging

In ecAudit we are using pre-logging by logging an ATTEMPT before passing the operation down to the QueryProcessor. Further, in the event of failure we will also log a matching FAILURE message.

In C* 4.0 we took a different approach. One message will always be logged after the operation has completed and at the same time we're indicating if operation was successful or not.

Upside of pre-logging is that we will have a audit event even if a request triggers a fatal stop of the Cassandra node. Downside is the possibly double messages (performance and sometimes unexpected behavior).

We should add support for post-logging to align with Cassandra 4.0, and make that behavior default.

Investigate possibility to automate releases

Would be nice to perform releases via Travis CI.

Things to research:

How to set parameters for build, such as release version and new development version.
Handle branches (flavours)
What about PGP signatures

Integration Tests in 2.2 branch: Insufficient disk space

The integration tests in release/c2.2 branch pass, but every time the embedded daemon "shutdown" it will print this ERROR similar to this:

08:08:24.532 [MemtableFlushWriter:1] ERROR o.a.c.service.CassandraDaemon - Exception in thread Thread[MemtableFlushWriter:1,5,main]
java.lang.RuntimeException: Insufficient disk space to write 50 bytes
	at org.apache.cassandra.db.Directories.getWriteableLocation(Directories.java:349) ~[cassandra-all-2.2.14.jar:2.2.14]
	at org.apache.cassandra.db.Memtable.flush(Memtable.java:325) ~[cassandra-all-2.2.14.jar:2.2.14]
	at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1191) ~[cassandra-all-2.2.14.jar:2.2.14]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_201]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_201]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_201]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_201]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_201]

Can't say this is influencing reliability of tests, but let's consider this a low-priority left-over from #111

Add support for Chronicle logging backend - align with C* 4.0

The audit logging feature in C* 4.0 have support for two different backends, SLF4J and Chronicle-Queues.

The Chronicle backend is interesting since it is more performant than the SLF4J/Logback backend. With this ticket we should explore ways to add support for Chronicle in ecAudit. We should consider compatibility with C* 4.0 when it comes to configuration options, record format in the Chronicle Queue as well as the tools used to view/get records from the Chronicle Queue.

Inconsistent behavior of batches

When a batch of updates is created manually (e.g. in cqlsh), the full batch will be logged as one single audit record. However, if a prepared batch is executed, then individual records will be created for the different operations in the batch (and correlated with a batch id).

This behavior is not consistent and will confuse users. We should investigate if we can get separated batch entries always.

FWIW, the same behavior is present in Cassandra 4.0.

Avoid logging large values

It would be nice if we could configure a size limit on values before logging them to the audit log. For instance it may not be useful to log 1M blobs into the audit log.

It should be possible to configure a size limit, and if the size of a value exceeds this limit, it should be replaced by a special character sequence. For performance reasons it would be desirable if the size could be checked before the value is extracted and escaped.

Primarily this should be implemented for prepared statements, but simple statements would be a nice bonus as well.

Developer script for installing ecAudit in current ccm cluster

We should create a simple setup script to "install" ecAudit in the current ccm cluster. This would be a great way to speed up development, or just take ecAudit for a test drive.

The script should be easily accessible in the ecAudit repo.
It should put the ecAudit jar on the class-path of the current ccm cluster nodes (preferrably without changing things in the ~/.ccm/repository directory.
It should modify cassandra.yaml and cassandra-env.sh files such that ecAudit plug-ins are enabled.

Optionally skip values when logging prepared statements

ecAudit will include values associated with prepared statements when logging audit records. This is not the case when logging audit records in Cassandra 4.0. However, Cassandra does include values when creating records for the FQL (Full Query Logger).

We should give end users a configuration option to include values or not.

Add support for whitelist filtering based on operation type

Currently ecAudit support filtering based on user/resource.

We should add support for whitelist filters based on type of operation. This is mostly useful for filtering on data, like having the ability to log writes, but not queries. Even though this may be less useful for other resources (like roles and schema changes), the solution must provide a reasonable solution for all cases.

In Cassandra 4.0 there is a similar filtering mechanism based on "categories" which in turn are derived from "types" (i.e type of operation).

Fix default path for audit.yaml

Currently ecAudit is looking for the audit.yaml file in /etc/cassandra/conf/. It would be nice if ecAudit was looking for audit.yaml in the directory specified by the $CASSANDRA_CONF environment variable. The /etc/cassandra/conf/ should still be the fallback if $CASSANDRA_CONF is not defined

Finally we should remain backwards compatible with the current option which allows you to configure the path using the Java property named com.ericsson.bss.cassandra.eaudit.config. This option takes precedence of $CASSANDRA_CONF. BTW, this should be documented.

Grant whitelist on a resource requires permission to alter role

In order to whitelist another role for a specific resource, the user performing the whitelist operaiton need AUTHORIZE permission on the resource as well. But currently this also require the performer to have ALTER permission on the assignee role as well.

admin@cqlsh> ALTER ROLE reader_role WITH OPTIONS = { 'grant_audit_whitelist_for_all' : 'data' };
Unauthorized: Error from server: code=2100 [Unauthorized] message="User admin does not have sufficient privileges to perform the requested operation"
admin@cqlsh> exit

cassandra@cqlsh> GRANT ALTER ON ROLE reader_role TO cadmin;
cassandra@cqlsh> exit

admin@cqlsh> ALTER ROLE reader_role WITH OPTIONS = { 'grant_audit_whitelist_for_all' : 'data' };

Would be neat if ecAudit would allow changes on the OPTIONS part of a role but not the other attributes like password and such.

Include client port number in audit record

In C* 4.0 the client port number is included in the audit record. The port field is not present in all kinds of records which is indicated with a value of '0' (zero).

We should add support for this in ecAudit as well.

Error when enabling ecAudit in live system

In the scenario where ecAudit is enabled in a live system by restarting one Cassandra node at a time, requests will fail temporarily. The error is manifesting itself as several rejected requests with AuthenticationException at the client side. This goes on for ~10 seconds just after the first Cassandra node is restarted. After this, traffic will work as expected. Subsequent Cassandra node restarts will not cause any issues. The issue was observed in a multi DC cluster.

The following stack trace is collected from failing node:

2019-01-25T11:45:38.760+0100 [SharedPool-Worker-14] ERROR o.a.c.t.Message$UnexpectedChannelExceptionHandler:625 apply Unexpected exception during request; channel = [id: 0x2ee41da4, L:/10.61.186.80:12742 - R:/10.61.186.78:33773]
com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 3 responses.
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) ~[guava-18.0.jar:na]
        at com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[guava-18.0.jar:na]
        at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) ~[guava-18.0.jar:na]
        at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) ~[guava-18.0.jar:na]
        at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) ~[apache-cassandra-3.0.17-E001.jar:3.0.17-E001]
        at org.apache.cassandra.auth.RolesCache.getRoles(RolesCache.java:44) ~[apache-cassandra-3.0.17-E001.jar:3.0.17-E001]
        at org.apache.cassandra.auth.Roles.getRoles(Roles.java:38) ~[apache-cassandra-3.0.17-E001.jar:3.0.17-E001]
        at com.ericsson.bss.cassandra.ecaudit.filter.role.RoleAuditFilter.isFiltered(RoleAuditFilter.java:46) ~[ecaudit_c3.0-1.0.0.jar:na]
        at com.ericsson.bss.cassandra.ecaudit.filter.yamlandrole.YamlAndRoleAuditFilter.isFiltered(YamlAndRoleAuditFilter.java:48) ~[ecaudit_c3.0-1.0.0.jar:na]
        at com.ericsson.bss.cassandra.ecaudit.facade.DefaultAuditor.audit(DefaultAuditor.java:46) ~[ecaudit_c3.0-1.0.0.jar:na]
        at com.ericsson.bss.cassandra.ecaudit.AuditAdapter.auditAuth(AuditAdapter.java:178) ~[ecaudit_c3.0-1.0.0.jar:na]
        at com.ericsson.bss.cassandra.ecaudit.auth.AuditPasswordAuthenticator$AuditPlainTextSaslAuthenticator.getAuthenticatedUser(AuditPasswordAuthenticator.java:140) ~[ecaudit_c3.0-1.0.0.jar:na]
        at org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:78) ~[apache-cassandra-3.0.17-E001.jar:3.0.17-E001]
        at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:513) [apache-cassandra-3.0.17-E001.jar:3.0.17-E001]
        at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:407) [apache-cassandra-3.0.17-E001.jar:3.0.17-E001]
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.44.Final.jar:4.0.44.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.0.44.Final.jar:4.0.44.Final]
        at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) [netty-all-4.0.44.Final.jar:4.0.44.Final]
        at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348) [netty-all-4.0.44.Final.jar:4.0.44.Final]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_192]
        at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) [apache-cassandra-3.0.17-E001.jar:3.0.17-E001]
        at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-3.0.17-E001.jar:3.0.17-E001]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_192]
Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 3 responses.
        at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:517) ~[apache-cassandra-3.0.17-E001.jar:3.0.17-E001]
        at org.apache.cassandra.auth.CassandraRoleManager.getRoles(CassandraRoleManager.java:282) ~[apache-cassandra-3.0.17-E001.jar:3.0.17-E001]
        at org.apache.cassandra.auth.RolesCache.lambda$new$0(RolesCache.java:36) ~[apache-cassandra-3.0.17-E001.jar:3.0.17-E001]
        at org.apache.cassandra.auth.AuthCache$1.load(AuthCache.java:183) ~[apache-cassandra-3.0.17-E001.jar:3.0.17-E001]
        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) ~[guava-18.0.jar:na]
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) ~[guava-18.0.jar:na]
        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) ~[guava-18.0.jar:na]
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) ~[guava-18.0.jar:na]
        ... 22 common frames omitted
Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 3 responses.
        at org.apache.cassandra.service.ReadCallback.awaitResults(ReadCallback.java:132) ~[apache-cassandra-3.0.17-E001.jar:3.0.17-E001]
        at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:137) ~[apache-cassandra-3.0.17-E001.jar:3.0.17-E001]
        at org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:145) ~[apache-cassandra-3.0.17-E001.jar:3.0.17-E001]
        at org.apache.cassandra.service.StorageProxy$SinglePartitionReadLifecycle.awaitResultsAndRetryOnDigestMismatch(StorageProxy.java:1716) ~[apache-cassandra-3.0.17-E001.jar:3.0.17-E001]
        at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1665) ~[apache-cassandra-3.0.17-E001.jar:3.0.17-E001]
        at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1603) ~[apache-cassandra-3.0.17-E001.jar:3.0.17-E001]
        at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1522) ~[apache-cassandra-3.0.17-E001.jar:3.0.17-E001]
        at org.apache.cassandra.db.SinglePartitionReadCommand$Group.execute(SinglePartitionReadCommand.java:1133) ~[apache-cassandra-3.0.17-E001.jar:3.0.17-E001]
        at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:303) ~[apache-cassandra-3.0.17-E001.jar:3.0.17-E001]
        at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:264) ~[apache-cassandra-3.0.17-E001.jar:3.0.17-E001]
        at org.apache.cassandra.auth.CassandraRoleManager.getRoleFromTable(CassandraRoleManager.java:525) ~[apache-cassandra-3.0.17-E001.jar:3.0.17-E001]
        at org.apache.cassandra.auth.CassandraRoleManager.getRole(CassandraRoleManager.java:507) ~[apache-cassandra-3.0.17-E001.jar:3.0.17-E001]
        ... 29 common frames omitted

Add description, topics

By adding a description and topics, it will be easier for others to find the project in the Github search engine.

Isolate flavour differences

In order to support different versions of Cassandra we're using the concept of flavors in ecAudit, which basically mean we maintain different branches of ecAudit - one for each supported version of Cassandra.

The differences between flavors is mostly limited to some specific parts of the code, but it is scattered around the code base which makes it error prone and time consuming as we're merging features/fixes between flavors/branches.

We should try to make this more explicit in our design by breaking out favor-differences into distinct classes or methods with intention-reviling names.

Possibility to log permission denied

In order to detect unauthorized use it would be beneficial to be able to specify that all failed (unauthorized) operations should be logged, regardless if the role is white-listed or not.

If a role has been white-listed for all data access operations (i.e. for simplicity) but only has access to a subset of the keyspaces/tables then it would make sense to log whenever the role tries to access another keyspace.

Deal with ignored unit tests in master branch

There are a hand full of unit tests in the master branch that need attention.

Refactor ecAudit initialization procedure

Given that ecAudit is utilizing 4 different plug-in points in Cassandra we need to refactor the initialization strategy to make sure that things are wired properly at startup.

In particular we should make sure that only one instance of logic components are created. Current procedure has flaws as two different AuditAdapterFactory instances (created from AuditPasswordAuthenticator and AuditQueryHandler) could create and AuditAdapter each. Not likely but possible.

Also, there are a bunch of static interfaces that we could loose in order to simplify testability. For instance the cache is using statics which in turn resolve the RoleManager via the DatabaseDescriptor. This is messy in it self, and also unnecessary since all calls could go directly to the WhitelistDataAccess instance.

Failed batch statements never get logged

Since batch statements are parsed and prepared in BatchMessage.execute(), the operation will be aborted before it gets to AuditQueryHandler.processBatch() in case the operation is invalid.

An example being when a batch statement includes an update on a non-existing table.

Chronicle log files use a lot of disk space

We should investigate the possibility to make Chronicle disk footprint smaller.
Comparing the log size of the audit files created when running the run_ccm_performance_test.sh (runs cassandra-stress with 3 million requests), we can see that the SLF4J logfiles take ~ 1.6G disc space, while Chronicle take ~2.1G.

We should investigate ways to lower this... A quick test showed that shortening the WireTag keys (the Strings describing the fields when writing data) made the usage much lower ~ 1.8G.
Maybe we don't even need those tags (not all of them)?

Flaky test in c2.2 ITVerifyChronicleBackend.testFailedAuthenticationRequest

Observed in 2.2 flavour.

[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] ITVerifyChronicleBackend.testFailedAuthenticationRequest:177
Expected size:<2> but was:<1> in:
<[com.ericsson.bss.cassandra.ecaudit.common.record.StoredAuditRecord@530216c5]>
[INFO]
[ERROR] Tests run: 113, Failures: 1, Errors: 0, Skipped: 2

https://travis-ci.org/eperott/ecaudit/jobs/605134126#L9860

Log only keys

Sometimes values contain sensitive or less useful information. In order to improve some specific security scenarios and performance it would be nice if we could limit what type of value types end up in the audit log. For instance it could be useful to only log values from the primary key.

Configurable format of printed records with eclog tool

The eclog tool which is used to extract and print Chronicle logs will print one entry per line using a hard coded format. We should investigate ways to make the print format configurable, much the same way as we have for the Slf4jLogger backend.

Ideally the format definition for eclog should be the same as the one used by the Slf4jLogger backend. Still I think we should make this configuration independent of the audit.yaml file, but this is open for discussion.

Fault handling around ResourceFactory methods

The ResourceFactory utility class throws IllegalArgumentException if an invalid resource String is passed in to any of its toResource() methods. This potential fault is not properly handled in WhitelistDataAccess and AuditWhitelistCache.

Error scenarios should result in proper Cassandra exception and reject client request. Behavior should be verified with integration tests.

Role based whitelists for non-existing ks/table doesn't work

While it is possible to whitelist a non-existing table, there will still be entries in the audit log if a client attempt to access that non-existing table.

For instance, this shows in the integration tests if we upgrade java-driver version to 3.6.0.

org.apache.cassandra.exceptions.InvalidRequestException: unconfigured table peers_v2
	at org.apache.cassandra.thrift.ThriftValidation.validateColumnFamilyWithCompactMode(ThriftValidation.java:119) ~[cassandra-all-3.0.16.jar:3.0.16]
	at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:897) ~[cassandra-all-3.0.16.jar:3.0.16]
	at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:892) ~[cassandra-all-3.0.16.jar:3.0.16]
	at org.apache.cassandra.cql3.QueryProcessor.getStatement(QueryProcessor.java:520) ~[cassandra-all-3.0.16.jar:3.0.16]
	at com.ericsson.bss.cassandra.ecaudit.AuditEntryBuilderFactory.createEntryBuilder(AuditEntryBuilderFactory.java:76) ~[classes/:na]
	at com.ericsson.bss.cassandra.ecaudit.AuditAdapter.auditRegular(AuditAdapter.java:83) [classes/:na]
	at com.ericsson.bss.cassandra.ecaudit.handler.AuditQueryHandler.process(AuditQueryHandler.java:98) [classes/:na]
	at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:115) [cassandra-all-3.0.16.jar:3.0.16]
	at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:513) [cassandra-all-3.0.16.jar:3.0.16]
	at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:407) [cassandra-all-3.0.16.jar:3.0.16]
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.44.Final.jar:4.0.44.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.0.44.Final.jar:4.0.44.Final]
	at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35) [netty-all-4.0.44.Final.jar:4.0.44.Final]
	at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348) [netty-all-4.0.44.Final.jar:4.0.44.Final]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_181]
	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164) [cassandra-all-3.0.16.jar:3.0.16]
	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [cassandra-all-3.0.16.jar:3.0.16]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]

Add support for host address in log message

In Cassandra 4.0 the IP address of the coordinator host i part of the audit log message. We should add this to ecAudit in order to close the feature gap.

Use queryStartNanoTime in ecaudit_c3.11

In #27 we introduced support for timestamps generated by ecAudit itself rather than relying on the logger backend to generate one at the time of writing. This improved accuracy a bit.

In ecaudit_c3.0 and ecaudit_c3.0.11 we're generating a timestamp ourself in the plugin; in the AuditQueryHandler for CQL queries. However, in ecaudit_c3.11 we get a timestamp (named queryStartNanoTime) from Cassandra into the method signatures. We should consider to use it in our audit records rather than generating one on our own.

Configurable format of logger backend

In order to implement #27 and #28 without breaking backwards compatibility we should investigate ways to make the logging format of the Slf4JAuditLogger configurable.

The default format should stay compatible with existing format.

Remove unit tests from Travis integration test suite

Travis runs unit tests (UT) and integration tests (IT) in parallel.
So there is no need to run the UTs also in the IT suite.

Change logFormat to log_format in audit.yaml

All documentation examples suggests to use logFormatin the audit.yaml configuration file. We should change this to log_format in order to use the same style as the Cassandra yaml configuration file.

Unable to re-create legacy whitelist table

When upgrading to ecAudit 2.0.0 or later a new format for the internal whitelist table is introduced. Once the upgrade is completed it is recommended to drop the legacy whitelist table using a custom cql command.

If for some reason the operator trigger this command to early (before all nodes are upgraded to ecAudit 2.0.0 or later), then it should be possible to re-create the legacy whitelist table simply by restarting one of the nodes which is still running on the old version. However, this doesn't work. Instead nodes fail to start:

2019-11-14T08:21:11.625+0100 [main] ERROR o.a.c.service.CassandraDaemon:726 exitOrFail Exception encountered during startup
java.lang.AssertionError: org.apache.cassandra.exceptions.InvalidRequestException: unconfigured table role_audit_whitelists
        at com.ericsson.bss.cassandra.ecaudit.auth.WhitelistDataAccess.prepare(WhitelistDataAccess.java:158) ~[ecaudit_c3.0-1.0.0.jar:na]
        at com.ericsson.bss.cassandra.ecaudit.auth.WhitelistDataAccess.setup(WhitelistDataAccess.java:59) ~[ecaudit_c3.0-1.0.0.jar:na]
        at com.ericsson.bss.cassandra.ecaudit.auth.AuditWhitelistManager.setup(AuditWhitelistManager.java:75) ~[ecaudit_c3.0-1.0.0.jar:na]
        at com.ericsson.bss.cassandra.ecaudit.auth.AuditRoleManager.setup(AuditRoleManager.java:78) ~[ecaudit_c3.0-1.0.0.jar:na]
        at org.apache.cassandra.service.StorageService.doAuthSetup(StorageService.java:1006) ~[apache-cassandra-3.0.17-E004.jar:3.0.17-E004]
        at org.apache.cassandra.service.StorageService.finishJoiningRing(StorageService.java:997) ~[apache-cassandra-3.0.17-E004.jar:3.0.17-E004]
        at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:934) ~[apache-cassandra-3.0.17-E004.jar:3.0.17-E004]
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:659) ~[apache-cassandra-3.0.17-E004.jar:3.0.17-E004]
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:572) ~[apache-cassandra-3.0.17-E004.jar:3.0.17-E004]
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:362) [apache-cassandra-3.0.17-E004.jar:3.0.17-E004]
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:585) [apache-cassandra-3.0.17-E004.jar:3.0.17-E004]
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:713) [apache-cassandra-3.0.17-E004.jar:3.0.17-E004]
Caused by: org.apache.cassandra.exceptions.InvalidRequestException: unconfigured table role_audit_whitelists
        at org.apache.cassandra.thrift.ThriftValidation.validateColumnFamilyWithCompactMode(ThriftValidation.java:119) ~[apache-cassandra-3.0.17-E004.jar:3.0.17-E004]
        at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:897) ~[apache-cassandra-3.0.17-E004.jar:3.0.17-E004]
        at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:892) ~[apache-cassandra-3.0.17-E004.jar:3.0.17-E004]
        at com.ericsson.bss.cassandra.ecaudit.auth.WhitelistDataAccess.prepare(WhitelistDataAccess.java:154) ~[ecaudit_c3.0-1.0.0.jar:na]
        ... 11 common frames omitted

Neither is it possible to create this table manually (since this is not allowed in system keyspaces), leaving the operator in a rather tricky situation.

Only login "Authentication attempt " operations are logged

Hi ,

I started to test audit plugin for Cassandra 3.11.2.

Followed the steps as per https://github.com/Ericsson/ecaudit/blob/release/c3.0/doc/setup.md .

i was testing from cql.
But only login "Authentication attempt " operations are logged .
Select ,insert,delete and other operations are not logged in the audit log file .

Is there any extra steps needed be done

Testing:

[cassandra@xxxxxxxxxxxx:/opt/app/cassandra/conf]$ cqlsh hostname -u coach -p 'All4One2day!'
Connected to xxx-cluster at xxxxxxxxxx:9042.
[cqlsh 5.0.1 | Cassandra 3.11.2 | CQL spec 3.4.4 | Native protocol v4]
Use HELP for help.
coach@cqlsh> INSERT INTO cycling.cyclist_name (id, lastname, firstname) VALUES (5b6962dd-3f90-4c93-8f61-eabfa4a803d4, 'VOSAA','Marianneee');
coach@cqlsh>
coach@cqlsh> select * from cycling.cyclist_name;

(2 rows)
coach@cqlsh>

Checking the audit.log

Add keyspace field in audit log

We currently don't have ks field. To make it compatible with 4.0, we should have it.

Make configuration updateable

Make the audit.yaml configuration updatable, so that is possible to dynamically update configuration parameters (e.g. logger_backend, log_timing_strategy, log_format) without having to restart Cassandra.

This would simplify and speed up integration test so that a new integration-test-* module does not have to be created every time a new parameter is added. Having "too many" integration test modules will prolong the test execution time since C* has to be started / torn down for every module.

Query logs without authentication backend

The full-query-logger in C* 4.0 support logging of CQL queries without any specific dependency to authentication/authorization.

Let's explore if ecAudit could support a similar setup, where records could be created without any specific information/relation about users or permissions.

Backport ecAudit to 2.2.x

Potential users have requested a backport of ecAudit that would be compatible with Cassandra 2.2.8. We should investigate if this is possible, with our without compromises in supported features.

Ideally we should have the very same features on all flavors of ecAudit, if nothing else for sake of maintainability.

Also, since 2.2.8 is a rather old release of the 2.2.x series, we should aim to start with a backport to 2.2.14 (latest at the time of writing). If necessary we'd make a backport further to 2.2.8 if necessary.

Configurable fields in Chronicle backend

We should make it possible to configure what fields to include in each record when using the Chronicle backend. This would make it possible for users to save both CPU and disk resources.

With this option we probably should include a bit-field after the record header to indicate what information is included in the record. The eclog tool must then be able to consume and print fields accordingly.

UUID lock contention

While testing with a lot of concurrent batch operations it seems that generating random UUIDs is causing a lot of contention. Under the hood random UUIDs uses SecureRandom which uses a synchronized code path (as well as occasionally using /dev/random or /dev/urandom).
On a node handling ~800 batches/s around 5% of the batch requests got blocked by this.

It would be good if we could postpone the UUID generation in some way or creating it in a way that avoids synchronization.

Bound values to simple statements are not logged

Bound values to prepared statements are nicely logged by ecAudit.
For example:
PreparedStatement preparedStatement = session.prepare("INSERT INTO dataks.tbl (key, value) VALUES (?, ?)");
testSession.execute(preparedStatement.bind("myKey", 42));
Will log:
"INSERT INTO dataks.tbl (key, value) VALUES (?, ?)['myKey', 42]"

But it is also possible to provide bound values to simple statements, see:
https://docs.datastax.com/en/developer/java-driver/3.7/manual/statements/simple/#using-values

Example:
session.execute("SELECT * FROM dataks.tbl WHERE key = ?", 42)
Will "only" log the statement, not the bound value:
"SELECT * FROM dataks.tbl WHERE key = ?"

In AuditQueryHandler (the plugin-pint for ecAudit) the process(...) method is called for simple statements. The problem here is that the QueryOptions object does not have a ColumnSpecification which is needed to convert the bound values into strings.

Investigate if this can be accomplished in a "simple way", to be able to reuse the code for stringifying bound values for prepared statements.

Coding standard

Investigate ways to include settings for coding standards in repo.

Should be aligned with Cassandra coding style.

Preferably support for IntelliJ and Eclipse.

Query logs without authorization backend

This ticket is created as a subtask to #77. In this ticket we're dealing with the authorization part, and #77 will deal with the authentication part.

Unstable Test in ITVerifyThreadedAudit

ITVerifyThreadedAudit.testValidPreparedStatementsAreLogged sometimes fails on Travis CI.
Test/code must be updated to get a predictable test result.

Scan code base for TODO comments and create tickets for them

Remove TODO comments where resonable

Audit metrics

For debugging purposes it would be good to get some insights into how much time is spent in ecAudit. It would good if we could expose timer metrics through JMX to make this information readily available. I propose that we add two timer metrics to AuditAdapter called Attempt and Failed (based on the enum Status) and then add a timer to each audit operation.

Optionally we could separate it a bit more and:

Add specific timers for authentication.
Add specific timers for when we actually audit something.

Add category field in audit log

We currently don't have category field. To make it compatible with 4.0, we should have it.

Add support for system timestamp in log message - align with C* 4.0

In Cassandra 4.0 audit log implementation the system timestamp is embedded into the audit log message.

We should add this to ecAudit to close feature gap.

In C* 4.0, documentation claim "unix timestamp", but implementation is using System.currentTimeMillis().

Align WhitelistDataAccess statements

Today there are two four types of statements in WDA:

Delete
Select
Update (add)
Update (remove)

Delete and select are using prepared statements while updates are using raw queries. IMO we should either align so that all non-traffical requests uses raw statements or so that all requests uses prepared statements.

compatible with Cassandra 3.11.3

@eperott Does ecaudit will be compatible with Cassandra 3.11.3 ?

Avoid logging BLOB values

BLOBs are probably not that interesting to log. Also they take up a lot of space when printing in ASCII-format. Logging just a few large blobs (can be of MB size) will fill up the logs.

Example log output from a prepared statement (with small blobs):
1570025092487|127.0.0.1:45176|127.0.0.1|cassandra|ATTEMPT|UPDATE "standard1" SET "C0" = ?,"C1" = ?,"C2" = ? WHERE KEY=12345 [0x28029cc8622a9724bf1cf40452459cffcf8d59c9ab5ebb00c91ff72de420d1a346b9, 0x6b986a3cb238eafd3953c7109ca2b92c7ece025076ffa573623b7994a915fb9ea209, 0x11c6d2fb29c9f643dae00c72e9f79a9f0858fd93bf1bbd79cd24d4124971a99ba9e8]

It should be possible to configure whether or not to print the blob values when logging prepared statements. The actual blob could be replaced by a "marker" (e.g. <BLOB>) to indicate that a blob was present. For example:
1570025092487|127.0.0.1:45176|127.0.0.1|cassandra|ATTEMPT|UPDATE "standard1" SET "C0" = ?,"C1" = ?,"C2" = ? WHERE KEY=12345 [<BLOB>, <BLOB>, <BLOB>]

This issue is related/similar to #124. Maybe both of them are relevant?

Note that CollectionTypes (LIST, SET, MAP) can also contain blobs. These values should also be considered (not logged) if "normal" blobs are configured not to be logged.