Hello,

I'm working on packaging Falco in Debian (see #842306).

In CMakeLists.txt, there is the inclusion of directories from Sysdig :

CMakeLists.txt:#add_subdirectory("${SYSDIG_DIR}/driver" "${PROJECT_BINARY_DIR}/driver")
CMakeLists.txt:#add_subdirectory("${SYSDIG_DIR}/userspace/libscap" ${PROJECT_BINARY_DIR}/userspace/libscap")
CMakeLists.txt:#add_subdirectory("${SYSDIG_DIR}/userspace/libsinsp" ${PROJECT_BINARY_DIR}/userspace/libsinsp")

It would make our life way easier if these files were actually part of Falco itself. I currently don't see any other way around.
Do you think it would be possible ?

Thanks,
Julien

Add example rules that show off Falco's k8s integration.

just like rules!
related to the more exploratory #65

An extra null pointer check is not needed in the function "falco_init".

I added the rule at the bottom of this message to the rules.yaml file. When I do a curl, I would expect the rule to be hit multiple times. However, I don't see any alert.

• rule: system_binaries_network_activity
  desc: any network activity performed by system binaries that are not expected to send or receive any network traffic
  condition: fd.sockfamily = ip
  output: "Known system binary sent/received network traffic"
  priority: WARNING

Add support for glob matching on pathnames in filters. For example you could write 'evt.arg[0] glob /bin/*' or similar.

This may have to be handled at the sysdig level, but we could also possibly translate it into a regex match.

It may be a useful exercise to document some comparisons between falco and something like auditd. For example, auditd also is built around the concept of identifying syscall conditions and logging the results (or otherwise delivering them somewhere). It can also be configured to halt the system if triggered events cannot be delivered to their destination, which is certainly a highly-valued feature when you want to make sure an adversary isn't trying to cover their tracks (e.g. either stopping auditing or redirecting events to a blackhole, etc.). Clearly falco seems a bit more expressive and user-friendly than auditd from a configuration and usability point-of-view, but what types of mandatory controls can be configured or introduced to ensure that what you see in your audits is actually what happened?

If I find e.g. php doing suspicious things, it doesn't really help me find the source of the attack (except for the uid). What I'd love would be to see the open file list at the time a rule fires to see what the vulnerable script was.

Maybe instead of putting this exact feature into falco (which would probably lead to a slippery slope of storing more and more data, e.g. syscall history before rule fired) a flight data recorder would be good enough. I.e. keep recording all the traced data a'la sysdig -W and remove the files that were recorded during no rule triggers (or maybe keep a couple before a rule fired etc.) -- then the user could simply sysdig -r the relevant file.

The sad thing is, I'd probably love to see the incoming request, which means not excluding read/write calls, at least for network I/O over a bunch of ips/ports.

Come to think of it, as long as falco is detect-only, maybe sysdig flight data recorder + falco reading from scaps (e.g. a minute delay) would be sufficient (it wouldn't stop the attack anyway).

With the probable fix for #57 in #64, rules and macros are starting to look a lot more similar. Maybe we should unify them and say that any rule without an output is a macro, or allow referring to rules in other rules.

Some parameters (like "${SYSDIG_DIR}/driver" and "${PROJECT_SOURCE_DIR}/scripts/debian/falco") are passed to CMake commands in your build scripts without enclosing them by quotation marks. I see that these places will result in build difficulties if the contents of the used variables will contain special characters like semicolons.

I would recommend to apply advices from a wiki article.

Add new fields name and description to rules that allow for uniquely identifying any rule and providing inline documentation for any rule.

Currently, falco can read multiple rules files and load them all into its running instance. However, the rules are always appended the current ruleset. This makes it impossible for example to have a second "overrides" rules file that changes the behavior of rules in the first file.

Also, you can not change a macro or list in a second rules file and have it change the behavior of an already loaded rule that referred to that macro/list. The best fix for this might be to implement #146.

Hi,

if in falco config i select the json output as true, the real json output doesnt include the priority or the description. Lets give an example:

• rule: ssh_incoming_connections
  desc: ips connected to ssh server
  condition: ssh_port and inbound
  output: "sshd incoming connection (connection=%fd.name)"
  priority: WARNING

this is rendered as

{"fd.name":"2.2.2.2:60310->1.1.1.1:22"}
if i disable the json output the output gives much much more info

May 22 16:13:38 test02 falco: sshd incoming connection (connection=2.2.2.2:60502->1.1.1.1:22)

is intended behaviour? is there any way to affect json output?

Thanks in advance i really do love falco :)

Within sysdig, chisels are really useful as they allow for lua-driven aggregations across multiple events. It'd be nice to have the same type of functionality where you could define a chisel to perform aggregation and expose the chisel's outputs to make them available in falco rules.

One example might be duplicating the functionality of topprocs_* chisels, and use that for a rule similar to "alert when a process is sending excess network traffic" or similar.

Add output filtering based on rule severity. That way, if you want to ignore Informational rules you can.

Add a tag field to rules and allow multiple tags per rule. We'll use this capability to run a subset of the rules based on tags.

Add example rules showing off falco's mesos integration.

Handle tmux: client/tmux: server as acceptable shell spawning programs.

This is somewhat blocked on draios/sysdig#718, although if necessary it could be handled by a substring match on proc.pname.

(Content copied from #99)

ikoniaris commented on Aug 19 • edited
Hi @mstemm, does this really work and how can I debug it?

For example, testing the program_output as so:

program_output:
enabled: true
program: logger -t falco-test
doesn't seem to do anything.

...

ikoniaris commented 2 days ago
I tried with Trusty, same thing. mailx works, logger doesn't. Not sure where the problem lies. Do you think I might be missing some lua-related libraries @mstemm? Is falco self-contained?

Falco should be self-contained wrt lua libraries and runtime. Just to be sure, can you attach your falco.yaml file so we can compare configurations?

Other things to check would be that you can run logger by hand to send messages and that logger is in your path, etc.

I think another thing you could try to do is to run sysdig to monitor the actions that falco performs, writing its events to a trace file. Hopefully the trace file will help diagnose the problem. sudo sysdig -w /tmp/falco_logger.scap "proc.name=falco or proc.name=logger" would be a good command line to run.

I don't see a copy of the install script here, but figure it's an ok place to report this. Please let me know if there is a better place to send this.

Within install-script which I have just retrieved from https://s3.amazonaws.com/download.draios.com/stable/install-falco I have noticed the following concerning lines:

curl -s -o /etc/yum.repos.d/draios.repo http://download.draios.com/stable/rpm/draios.repo

curl -s -o /etc/apt/sources.list.d/draios.list http://download.draios.com/stable/deb/draios.list

These are both plaintext requests, which I believe are a poor practice and best avoided when you're installing an application intending to increase your security posture.

I can verify that the repo files can be retrieved fine via HTTPS from the s3 bucket at for example https://s3.amazonaws.com/download.draios.com/stable/rpm/draios.repo and given that the bucket is referenced directly elsewhere in the install script it should be fine to change this line also.

Cheers.

Suggestion from @KenanSulayman--have the ability to load subsequent sets of rules on the fly conditioned on rules triggering. For example, if an initial rule detected whether a container was running privileged and had a given image name, a second set of rules could be loaded that were specific to the use case of a privileged container with that image.

This could also be used to handle the administration problem, where certain actions like writing to binary directories, etc. are generally not allowed, but would be expected if a program like dpkg, rpm, etc were running. It would be nice if falco could notice that dpkg/rpm was exec()ed and switch to a different set of rules that allowed software installation tasks, and disabled those rules once the dpkg/rpm exited.

Sysdig's build system supports a global flag which tells CMake that all of the dependencies will be provided and that it shouldn't download them if needed. This is helpful when compiling Sysdig on systems like NixOS which don't allow network access during the build process.

See flag use here:
https://github.com/draios/sysdig/blob/dev/CMakeLists.txt#L98

On release 0.3.0:

Build of lyaml failed when testing if luajit-prefix/src/luajit is a Lua interpreter.

Adding a dependency upon juajit for lyaml in the CMakeLists.txt allos the compilation to go further

Categorize the existing set of rules into profiles like 'production server', 'desktop', 'devops management', etc, using the tags we'll define in #58.

Idea brought up during some feedback meetings...

In addition to explicit activity, it might be useful to consider excessive resource usage (like numbers of open fds, etc.) as suspicious.

This starts to blur the line between general resource usage monitoring and security-specific monitoring, but it may have some value to look for, say, dramatic increases in the number of connections or processes.

Based on experiences with writing rulesets to restrict activities of pipe installer programs. It'd be nice if you could generate a list of files/items/etc on the fly from a script or similar, and use that list in rules.

For example, you could find all the locations listed in apt source files below /etc/apt/sources.list.d, and write a rule that warns when any outbound connection is made to a host not in that list.

A while ago we had rulesets that defined the acceptable ports, etc used for a bunch of common servces like apache, cassandra, etc. We disabled those as having them all enabled slowed down falco too much, but now that we initially filter based on event type we may have the headroom again. Also, fixing #149 would also make it easy to enable/disable these rules only when the process is running.

When the falco engine matches an event to a rule, it returns the rule's name, priority, and output string.

Later, in the outputs module, the output string is prefixed with '*' to ensure that if an event doesn't have a matching value for a given %xxx field, the rest of the values are filled in. See https://github.com/draios/sysdig/wiki/Sysdig-User-Guide#output-formatting for details.

The engine should really just add the '*' prefix itself so the outputs module doesn't have to.

Yaml supports the notion of aliases, which allows for defining blocks of yaml with a name and referring to those blocks by that name. An example is:

aliases:
    # example of an alias that defines just a single piece of data
    - &environ [DAMAGE_ENTITY_EXPLOSION, DAMAGE_FIRE,  DAMAGE_CONTACT, DAMAGE_DROWNING, DAMAGE_FALL, DAMAGE_SUFFOCATION]

rule:
   - tool: *environ

This would be a good way to support lists natively instead of having a user-defined convention. We may want to do this for macros as well, although macros do have some validation on conditions which would go away if the expansion appeared before it made it to falco.

@ahl thanks for the suggestion!

Suggestion from rez410 on /r/netsec--add the ability to read trace files over network connections. The original comment was asking for reading all kinds of logs and not specifically sysdig trace files, but the network read idea is worth considering.

sysdig and falco installed through the project's Debian repository on existing systems running Jessie and Kubernetes 1.4

falco process consumes 2.8 GB of resident memory after running on a cluster node for more than 3 hours. Syslog has many lines which look like the following

Dec  5 20:34:33 ip-10-53-83-111 falco: 20:34:33.211744777: Warning File opened for read/write by non-privileged container (user=root command=iptables -w -N KUBE-SERVICES -t filter k8s_kube-proxy.19117fe_kube-proxy-ip-10-53-83-111.us-west-2.compute.internal_kube-system_fe7124acf1c281948c5fa97e777ed495_3048a81d (id=a4190e1cec0a) file=/lib/libip4tc.so.0)
Dec  5 20:34:33 ip-10-53-83-111 falco: 20:34:33.211772926: Warning File opened for read/write by non-privileged container (user=root command=iptables -w -N KUBE-SERVICES -t filter k8s_kube-proxy.19117fe_kube-proxy-ip-10-53-83-111.us-west-2.compute.internal_kube-system_fe7124acf1c281948c5fa97e777ed495_3048a81d (id=a4190e1cec0a) file=/lib/libip6tc.so.0)
Dec  5 20:34:33 ip-10-53-83-111 falco: 20:34:33.211795245: Warning File opened for read/write by non-privileged container (user=root command=iptables -w -N KUBE-SERVICES -t filter k8s_kube-proxy.19117fe_kube-proxy-ip-10-53-83-111.us-west-2.compute.internal_kube-system_fe7124acf1c281948c5fa97e777ed495_3048a81d (id=a4190e1cec0a) file=/lib/libxtables.so.10)
Dec  5 20:34:33 ip-10-53-83-111 falco: 20:34:33.211820300: Warning File opened for read/write by non-privileged container (user=root command=iptables -w -N KUBE-SERVICES -t filter k8s_kube-proxy.19117fe_kube-proxy-ip-10-53-83-111.us-west-2.compute.internal_kube-system_fe7124acf1c281948c5fa97e777ed495_3048a81d (id=a4190e1cec0a) file=/lib/x86_64-linux-gnu/libm.so.6)
Dec  5 20:34:33 ip-10-53-83-111 falco: 20:34:33.211844532: Warning File opened for read/write by non-privileged container (user=root command=iptables -w -N KUBE-SERVICES -t filter k8s_kube-proxy.19117fe_kube-proxy-ip-10-53-83-111.us-west-2.compute.internal_kube-system_fe7124acf1c281948c5fa97e777ed495_3048a81d (id=a4190e1cec0a) file=/lib/x86_64-linux-gnu/libc.so.6)
Dec  5 20:34:33 ip-10-53-83-111 falco: 20:34:33.211877232: Warning File opened for read/write by non-privileged container (user=root command=iptables -w -N KUBE-SERVICES -t filter k8s_kube-proxy.19117fe_kube-proxy-ip-10-53-83-111.us-west-2.compute.internal_kube-system_fe7124acf1c281948c5fa97e777ed495_3048a81d (id=a4190e1cec0a) file=/lib/x86_64-linux-gnu/libdl.so.2)

There are hundreds of thousands of warnings like this in /var/log/syslog. I'm unsure why these warnings consume resident memory within the falco process. When I add the following change to /etc/falco_rules.yaml all warnings disappear and there is no longer resident memory growth.

- macro: trusted_containers
  condition: (container.image startswith sysdig/agent or container.image startswith sysdig/falco or container.image startswith sysdig/sysdig or container.image startswith gcr.io/google_containers/hyperkube or container.image startswith gcr.io/google_containers/kube-proxy)

- rule: File Open by Privileged Container
  desc: Any open by a privileged container. Exceptions are made for known trusted images.
  condition: (open_read or open_write) and container and container.privileged=true and not trusted_containers
  output: File opened for read/write by non-privileged container (user=%user.name command=%proc.cmdline %container.info file=%fd.name)
  priority: WARNING

Not making a PR for this since it's a configuration default. I believe this decision lies on the side of the authors. It is worth noting this detail in the documentation since the default configuration makes falco non-functional on a Kubetnetes cluster of arbitrary size.

Support reading rule files from a directory in addition to from a series of files. The argument to -r can be tested and if it's a directory every file below it will be read as a rules file.

Currently, all falco rules provide general information for each notification that contains host-level information like the process name, arguments, file being read/written/etc, network connection, etc.

It would be nice if you could also provide container level information like the container name, kubernetes level information like the pod, mesos level information like the app, etc. in the notification output, without having to add it to each rule's output format string.

Sysdig does this via -pk, -pc, and -pm arguments that change the default output format. We could do a similar thing here.

Hi there,
As the wiki page https://github.com/draios/falco/wiki/Falco-Rules says, for performance consideration, the sysdig logs of a list of system calls are discarded before falco processing. However, I would like to collect the complete system call list of a running process.
I wonder how I can stop falco discard system calls before processing.

Thanks,
Zhiyuan

A solution to address man-in-the-middle attacks against curl | sh installers is to come up with a variant of sh that can be tracked by falco. All children processes of that sh variant will have restrictions on the things they can do.

We should package that variant in the distribution. The current best solution is a c program that does the fork + exec, but hopefully we can do something better.

Now that we have the ability to build a container from a local package, we should be able to test that the built container runs successfully.

While working through some updates to the falco rules, I found that with the current set of rules, network file descriptors won't have their state (hostname and port) fully resolved. For example, the installer_bash_starts_network_server rule can't show the address/port on which the process is trying to listen.

The reason for this is that file descriptor resolution generally occurs during a bind. However, as no current rule looks for bind events, the file descriptor meta-information isn't associated with the fd.

We'll have to figure out a way to get the bind events to libsinsp so the meta-information can be saved.

I noticed that the output would be occationally truncated:

Thu May  5 22:34:41 2016: Parsed rules from file ../../falco/rules/falco_rules.yaml
22:34:44.860506950: Warning Unexpected shell (mstemm bash emacs > rt_sigaction
22:34:44.860507719: Warning Unexpected shell (mstemm bash emacs < rt_sigaction
22:34:44.860513008: Warning Unexpected shell (mstemm bash emacs > stat
22:34:44.860520440: Warning Unexpected shell (mstemm bash emacs < stat res=-2(ENOENT) path=/var/mail/mstemm

The reason for this is that sinsp_evt_formatter::tostring stops and returns false if not all fields in an output string are present. You can mark all fields as optional if the format string starts with a *.

We should update the rules to reflect this or flag incomplete output as such.

When starting, falco does several tests to find an appropriate configuration file. It tries these files in this order:

• <source dir>/falco.yaml
• /etc/falco.yaml

The first file found is used.

However, the logic for testing the files is backwards. When it tests to see if a file was opened, it accepts the file if it is not available.

The result of this is that it always chooses the source directory file if it is not available.

This bug was introduced in fc9690b, so 0.3.0 is not affected.

Hello,

It would be nice if falco could produce a full record of applications call.
something that can produce similar view that these products do
https://www.carbonblack.com/2016/05/10/patterns-of-attack-youre-looking-for-answers-not-just-indicators/
http://www.cisco.com/c/en/us/products/collateral/security/fireamp-endpoints/datasheet-c78-733181.html

Thanks

Add the ability to run only those rules matching one or more tags. This will allow us to come up with profiles for 'production server', 'desktop', etc.

Implement as much of the CIS benchmark as possible using Falco rules. Some components, specifically those that map to auditd rules, should be easy. Some will not be possible, such as checking ulimit values. But I bet we can implement a pretty significant subset.

Reported by a sysdig cloud customer:

Capturing a SYSDIG capture causes quite a nice number of alerts.
Name: Change thread namespace
Description: Namespace change (setns) by unexpected program (user=root command=python2.7 /opt/draios/bin/sdchecks container=53761a2ccbfe)
Scope: host.mac=06:7c:60:21:91:a2 and container.id=53761a2ccbfe
Tags source: falco_engine

This doesn’t make a lot of sense since the rule has a specific call out to prevent sdchecks from firing.

For example
sysdig -r ssh-key-management.scap -p "%proc.name %24proc.pname %proc.cmdline" evt.type = setns and not proc.pname in (setup-backend, dragent, sdchecks)

Returns nothing.

Investgate and update the rule set as needed.

I am using the container install from : https://github.com/draios/falco/wiki/Running%20Falco
The issue I am seeing is the kernel I am using it named differently than the standard. I have manually installed the sysdig-probe.ko. Yet when I run the falco container it unloads it and then fails at loading a new one. How can override this function?

moby:~# lsmod
Module                  Size  Used by
sysdig_probe          412164  0
moby:~# docker run -i -t --name falco --privileged -v /var/run/docker.sock:/host/var/run/docker.sock -v /dev:/host/dev -v /proc:/host/proc:ro -v /boot:/host/boo
t:ro -v /lib/modules:/host/lib/modules:ro -v /usr:/host/usr:ro sysdig/falco
* Setting up /usr/src links from host
ls: cannot access '/host/usr/src': No such file or directory
* Unloading sysdig-probe, if present
* Running dkms autoinstall
Error! echo
Your kernel headers for kernel 4.4.17-moby cannot be found at
/lib/modules/4.4.17-moby/build or /lib/modules/4.4.17-moby/source.
* Trying to load a system sysdig-probe, if present
* Trying to load a dkms sysdig-probe, if present
* Trying to find precompiled sysdig-probe for 4.4.17-moby
Found kernel config at /proc/config.gz
* Trying to download precompiled module from https://s3.amazonaws.com/download.draios.com/stable/sysdig-probe-binaries/sysdig-probe-0.11.0-x86_64-4.4.17-moby-7662b930caa21f46ec44b4ed9eaab027.ko
Download failed, consider compiling your own sysdig-probe and loading it or getting in touch with the sysdig community
Wed Aug 24 13:43:52 2016: Falco initialized with configuration file /etc/falco.yaml
Wed Aug 24 13:43:52 2016: Parsed rules from file /etc/falco_rules.yaml
Wed Aug 24 13:43:52 2016: Unable to load the driver. Exiting.
Wed Aug 24 13:43:52 2016: Runtime error: error opening device /host/dev/sysdig0. Make sure you have root credentials and that the sysdig-probe module is loaded.. Exiting.
moby:~# uname -a
Linux moby 4.4.17-moby #1 SMP Thu Aug 11 03:47:40 UTC 2016 x86_64 Linux
moby:~# uname -r
4.4.17-moby
moby:~# 

Add a preprocessor step that supports an include mechanism so you can include one set of rules from one file into another.

Be awesome if Falco could be configured to send email alerts to one ro more configured email addresses.

Might be a nicer way of achieving what is currently done via commenting (https://github.com/draios/falco/pull/71/files)

Whether can trace completely a session of the user who made sudo? Now if the user made "sudo su -" or "sudo - s" it is logged as root

Idea: it would be nice to have a Puppet module (or other type of configuration management software "recipe") to deploy and manage falco.

https://github.com/juju4/ansible-falco/

If I can have feedback to on my rules work in progress, would be great too
https://github.com/juju4/ansible-falco/blob/master/templates/falco_rules_w_exceptions.yaml.j2

Thanks