cbsd expose clears host's existing pf rules about cbsd HOT 3 OPEN

Firewall management in CBSD is quite limited and creates very simple rules. This control is suitable if you do not use your own rules because, as you can see, it is quite difficult to mix/combine them together. As a rule, if you use your own rules, you should turn off the firewall in the cbsd ( cbsd natcfg + cbsd natoff ) for better control.

However, there are several ways to combine:

1. Checkout https://github.com/cbsd/cbsd/blob/develop/etc/defaults/pf.conf.tpl template. It regulates the sequence/order of includes and you can use it to include your rules. For example:

/storage/cbsd/etc/pf.conf:

## include CBSD NAT rules
include "/storage/cbsd/etc/pfnat.conf"

## include system pf RULES
include "/etc/pf.conf"

## include CBSD RDR rules
include "/storage/cbsd/etc/pfrdr.conf"

In this case, you should also change default config path for /etc/rc.d/pf service:

sysrc pf_rules="/storage/cbsd/etc/pf.conf"

/storage/cbsd/etc/pf.conf is static while "/storage/cbsd/etc/{pfnat,pfrdr}.conf is dynamic and managed by CBSD.

2. Use CBSD hooks for integration with your firewall rules management. By placing your own scripts in different directories in ~cbsd/jails-system/ENV/master_*.d, you can perform the actions you need in different states of the virtual environment. In these scripts you can use internal CBSD facts (you only need $jname and $ip4_addr)

For example for jail named dnsmasq create executable file ~cbsd/jails-system/dnsmasq/master_poststart.d/0_set_rules.sh:

#!/bin/sh

cat > /tmp/0_set_rules.txt <<EOF
My jail name: ${jname}
My Full IP address list: ${ip4_addr}

Other useful facrts:

ipv4_first_public="${ipv4_first_public}"
ipv4_first_private="${ipv4_first_private}"
ipv4_first="${ipv4_first}"
ipv6_first_public="${ipv6_first_public}"
ipv6_first_private="${ipv6_first_private}"
ipv6_first="${ipv6_first}"
nic_address="${nic_address}"

nic_hwaddr0="${nic_hwaddr0}"
nic_address0="${nic_address0}"
nic_parent0="${nic_parent0}"
interface="${interface}"
EOF

chmod +x ~cbsd/jails-system/dnsmasq/master_poststart.d/0_set_rules.sh

after the container (or VM) starts, look at the /tmp/0_set_rules.txt. Instead of saving facts to a log file, you can create and delete ( via master_poststop.d hooks directory ) corresponding rules at runtime.

if you want hook scripts to be created by default along with the new environment, put these files in the appropriate skeleton directory ( ~cbsd/share/jail-system-default ) or create your own profile.

less /storage/cbsd/etc/pfnat.conf-> contains all the private network ranges' nat pf rules, not just those IP ranges the jails using

you can adjust this, see: https://github.com/cbsd/cbsd/blob/develop/etc/defaults/cbsd-pf.conf

For example, override this settings via ~cbsd/etc/cbsd-pf.conf:

# misc settings. Default RFC1918 network for NAT net/rules
cbsd_nat_networks="10.0.0.0/8 172.16.0.0/12"

# don't create (when '1') NAT rules for SAME network as NATIP
cbsd_nat_skip_natip_network=0

^^ exclude 192.168.x.x from NAT networks.

to apply:

cbsd natoff
cbsd naton

from cbsd.

Thank you for the details, and possible workarounds really helpful!

I have not checked the MyBee or ClonOS yet: how is this whole host pf.conf area handled in their case? If I understand well MyBee more cli and provides APIs, ClonOS has the "built-in" web-ui like Proxmox.

1. For host pf rules - I'll test out the second version you wrote and write back the working way, eg. what could be a host pf.conf template which always works - only the allowed service/application ports need to be varied. In other words - providing a skeleton for host rules. It gives two directions:

• if someone starts a new host with CBSD -> CBSD provide the host rules from the first step
• if someone already has rules with a configured/running host and starts CBSD later -> can help how to strip down their original /etc/pf.conf with the CBSD host pf template/sample.

It could be a base of a tui or cli based import portion during cbsd expose apply... command.
In general, I found the best base for populating a pf.conf ruleset at DigitalOcena pf FB12.1, It controls outbound as well (not a bad thing either, if I run multiple VMs/Jail even better, like myBee ), but missing nat/rdr samples for CBSD purposes.

2. So for handling the RFC1918, editing/crop-to-target A/B/C network range is the simplest way in /path_to/cbsd/etc/defaults/cbsd-pf.conf OK

For including there are multiple ways, for first sight I would opt for including the host's /etc/pf.conf into

/etc/pf.conf into the /path/to/cbsd/etc/pf.conf:

## include system pf RULES
include "/etc/pf.conf"

3. Last thing, as of now the expose command's rdr rules does not includes the ability to add rdr rules allowed for specific IPs only
   /path_to/cbsd/etc/pfrdr.conf
   rdr pass proto tcp from any to x.x.x.x port 111 -> y.y.y.y port 1110 # // Setup by CBSD expose: tcp-1110-jailname

I've learned by trial, that skipping to specify the source ip in rdr rules makes the packet filtering block useless in the pf.conf (host based or not) - the minute I opened a desired application port, got some bots connecting already.
A desired version of that sample would be:
rdr pass proto tcp from 8.8.8.8 to x.x.x.x port 111 -> y.y.y.y port 1110 # // Setup by CBSD expose: tcp-1110-jailname
Here let only Google's DNS ip (8.8.8.8) to connec to the port 111.

Would you consider adding this "from-ip" capability into the expose command, as optional parameter?

from cbsd.

@olevole

After some quick tests, host /etc/pf.conf content can't be aded as just one file into the CBSD working directory structure into a special pf configuration file.
One can make his/her system wide /etc/pf.conf relevant part duplicated into CBSD -> It need to parsed, cut in two (half or depends on the actual host's pf.conf )

• cut after scrub or queue insert his/her first part before pfnat file inclusion line
• and the rest after pfrdr inclusion line.

I think less people will or should go in this direction, this is basically duplicates a forsen version of original pf.conf into an another place belongs to a non-base application, while CBSD's nat ad rdr will be dynamic via cbsd naton/natof & cbsd expose

My proposition is following the reverse logic, as the simplest & best solution: right after CBSD install, as a post-process hook just include CBSD's pfnat & pfrdr files automatically - before the filtering block of any /etc/pf.conf -> Include statement existing since FreeBSD 9.0 Release: man pf.conf I did that now manually, which solves the original issue for me.

Other, more fine-graned possibility post-process hooks placements:
Include the CBSD's /path_to/cbsd/etc/pfnat.conf after cbsd naton command (and remove if natoff has ran).
Then include the /path_to/cbsd/etc/pfrdr.conf into the host's /etc/pf.conf after expose apply (and remove if with expose clear|flush)

Details on pf.conf parsing and possible post-process hook scripting:

The /etc/pf.conf parsing:

• ext_if="int_name" for example, etc. (Macros), -> this set lives anywhere up until the first use of macroname[s]
• ... (Tables), -> this set lives anywhere within the pf.conf, based on my tests
• set skip on lo and its friends (Options), -> this set lives anywhere within the pf.conf, based on my tests
• after scrub , (Normalization) or,
• after queue (Queueing) (search for "queue ssh_bulk bandwidth 50% priority 0 cbq(borrow)" in man pf.conf )
• ... include /path_to/cbsd/etc/pfnat.conf (Translation),
• ... include /path_to/cbsd/etc/pfrdr.conf (Translation),
• before anchors, which are coming before,
• the blockfiltering rules (Filtering),
• and pass filtering rules (Filtering)

A script needs to do:
On install:

• backup original pf.conf as cp /etc/p.conf /etc/pf.conf.cbsd.save
• find in /etc/pf.conf if 'queue' exist, if yes csplit it into a temp-file1 till the last /^queue./ line and temp-file2. OR if not do the same with /^scrub./. [Other perhaps better direction(?) candidates are new files before first /^block./ or /^pass./]
  In my case this works csplit -f pf.conf. /etc/pf.conf /^scrub.*/+1 BUT needs additional testing. I have no queue settings/lines in actual pf.conf. For that version of the oneliner, it needs to get the last line with queue
• create a 3rd temp-file: which has the 2 include lines with pfnat & pfrdr line, echo -e '\n# Adding CBSD NAT and RDR rules\ninclude "/path_to/cbsd/etc/pfnat.conf"\ninclude "/path_to/cbsd/etc/pfrdr.conf"\n' > pf.conf.incl
• cat pf.conf.00 pf.conf.incl pf.conf.02 > pf.conf.new

On uninstall:

• reverse it via the backup file (yes, but the /etc/pf.conf file could be evolved since that)
• reverse it via comment out the two include lines with a sed oneliner sed -i.cbsd.restore -e 's/^[^#]*\/cbsd\/etc\/pfnat/#&/' -e 's/^[^#]*\/cbsd\/etc\/pfrdr/#&/' /etc/pf.conf (best). Of course its worth to do a check first, if they are there at all grep -e "/cbsd/etc/pfnat.conf\|/cbsd/etc/pfrdr.conf" /etc/pf.conf Note: if this is a chosen direction, install step should look for commented out pfnat & pfrdr include lines if yes/remove and then readd them

from cbsd.