If we want to look at using Cyclonus for conformance testing I think we should move it upstream into this repo.

I took a stab at this a while back #29 but need someone to help push it forward.

Following up on the meeting on Apr, 25th here are some use cases, that might be interesting for consideration when thinking about NP API, egress controllers etc

Suppose we have a K8s cluster that hosts a bunch of applications. Suppose those applications provide user interface to scientific instruments. Scientific instruments are essentially a bunch of hardware, for simplicity let's assume they are just network entry points, listening on TCP fo some predefined commands. Applications in K8s and that hardware lives in different networks.

What be super useful is to ability in a declarative way to restrict applications to access only specific hardware. Say I have applications A1 and A1 and my hardware is X, Y, Z. I want A1 to be able to access X, Y and A2 -- Y, Z.

Communication is bidirectional i.e. X may push updates to applications, while Y, Z can be only polled.

I recall we discussed this scenario within the scope of egress controller, but I may be mistaken.

Hope this whole story makes any sense at all.

• Add docs outlining conformance tests - what are core v/s extended features
• Add docs for the test grid
• Add docs for contribution instructions to adding a new test
• Add docs for how implementations can build and run these tests downstream
• Add docs for conformance profile
• Add docs for conformance reporting and badges

Currently all conformance tests are only testing on a namespace level (users should use NetworkPolicy if they want to fall to a pod-level which is why the first round of tests are not giving importance to this). We need to also add tests that use podSelector since that is a supported part of the API.

Add extended conformance test for

We would like to provide the community with a set of examples and curated recipes for using the new AdminNetworkPolicy feature. Recipes are to be provided here in this repo.

Currently all conformance tests are testing on a namespace level. Once #104 is fixed, we need coverage for example, when the labels of a pod get updated and policy stops matching OR when labels of a pod are changed and the new ones start matching an existing policy etc. We should add coverage for these.

/area conformance

This issue will track the addition of release tooling and documentation. Much of it can be carried over from the gateway-api project, except for now we won't need to handle release channels since we're just getting started.

This will allow us to finish #90

This was discussed in yesterday's policy-api meeting and we are going to do something similar to what gateway folks have: https://gateway-api.sigs.k8s.io/geps/overview/

Currently all conformance tests are testing on a namespace level. However we don't have coverage for example, when the labels of a namespace get updated and policy stops matching OR when labels of a namespace are changed and the new ones start matching an existing policy etc. We should add coverage for these.

/area conformance

We just tried to install the crd directory on antrea live and messed up :)

Error from server (Invalid): error when creating "STDIN": CustomResourceDefinition.apiextensions.k8s.io "adminnetworkpolicies.policy.networking.k8s.io" is invalid: [spec.validation.openAPIV3Schema.properties[spec].properties[egress].items.properties[ports].items.properties[portNumber].properties[protocol].allOf[0].default: Forbidden: must be undefined to be structural, spec.validation.openAPIV3Schema.properties[spec].properties[egress].items.properties[ports].items.properties[portNumber].properties[protocol].allOf[1].default: Forbidden: must be undefined to be structural, spec.validation.openAPIV3Schema.properties[spec].properties[egress].items.properties[ports].items.properties[portRange].properties[protocol].allOf[0].default: Forbidden: must be undefined to be structural, spec.validation.openAPIV3Schema.properties[spec].properties[egress].items.properties[ports].items.properties[portRange].properties[protocol].allOf[1].default: Forbidden: must be undefined to be structural, spec.validation.openAPIV3Schema.properties[spec].properties[ingress].items.properties[ports].items.properties[portNumber].properties[protocol].allOf[0].default: Forbidden: must be undefined to be structural, spec.validation.openAPIV3Schema.properties[spec].properties[ingress].items.properties[ports].items.properties[portNumber].properties[protocol].allOf[1].default: Forbidden: must be undefined to be structural, spec.validation.openAPIV3Schema.properties[spec].properties[ingress].items.properties[ports].items.properties[portRange].properties[protocol].allOf[0].default: Forbidden: must be undefined to be structural, spec.validation.openAPIV3Schema.properties[spec].properties[ingress].items.properties[ports].items.properties[portRange].properties[protocol].allOf[1].default: Forbidden: must be undefined to be structural, metadata.annotations[api-approved.kubernetes.io]: Required value: protected groups must have approval annotation "api-approved.kubernetes.io", see https://github.com/kubernetes/enhancements/pull/1111]

How can we install these? We tried

-> % kustomize build . | kubectl apply -f -

as well as just kubectl create....

We're maintaining a ton of scripts here, it'd be nice to have a shell script specific linter like https://www.shellcheck.net/ or something to make sure we don't miss silly things :)

Ensure we test

Now that the API has physically merged #30 We should look at adding conformance testing, cyclonus already does this for netpol so it seems to make sense to extend the tool for the AdminNetworkPolicy API as well.

There are a few kube-builder features that would be useful for the Admin Network Policy API:

• Short Names of anp and banp
• Printer columns like priority, creation timestamp

In the KEP we specify https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/2091-admin-network-policy#priority:

The relative precedence of the rules within a single ANP object (all of which share a priority) will be determined by the order in which the rule is written. Thus, a rule that appears at the top of the ingress/egress rules would take the highest precedence.

But I don't think I know this from reading the actual API types :)

Pretty simple update to the API comments should fix this

Update the KEP https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/2091-admin-network-policy

To be up to date with the API which actually merged.

Including,

• BaselineAdminNetworkPolicy
• Selector changes
  etc

Follow up to #47 (review)

We should update our README to have pointers back to the K8s contributor guidelines and maybe even add our own?

We might consider changing the image on the website to use "Evaluated first" and "Evaluated Last" instead of a "priority" arrow, which I felt a little confusing. (sorry about the crappy Paint job 😄 )

Current image: site-src/images/ANP-api-model.png

New proposal

In the current API, port numbers/ranges are given priority over protocol:

ports:
- portNumber:
    port: 443
    protocol: TCP

This makes it tricky to express rules for port-less IP protocols such as ICMP, OSPF, etc. This is probably fine for IPv4, but I think this is a problem for IPv6 due it's heavy reliance on ICMP. I expect a use-case such as "allow (TCP, 443) and ICMP traffic" to become increasingly common as IPv6 adoption improves. Maybe ICMP should be special-cased? But that seems a bit awkward.

I'm wondering whether this use-case warrants making protocol a top-level field in AdminNetworkPolicyPort. E.g:

ports:
- protocol: TCP
  portNumber:
    port: 443
- protocol: ICMP

Or perhaps making protocols the one-of variants. E.g:

protocols:
- tcp:
    portNumber:
      port: 443
- udp:
    portRange:
      start: 0
      end: 1024
- icmp:
    type: 3 # Destination unreachable

This is a much larger change, but it enables custom selectors for each protocol. Looks like the CNI-specific NetworkPolicy APIs have gone in this direction as well:

• https://docs.tigera.io/calico/3.25/network-policy/policy-rules/icmp-ping
• https://docs.cilium.io/en/latest/security/policy/language/#limit-icmp-icmpv6-types

Apologies if this has already been discussed elsewhere, I wasn't able to find any previous threads on this topic.

We don't actually need Same / NotSame for namespaces, you can just specify SameLabels: ["kubernetes.io/metadata.name"] / NotSameLabels: ["kubernetes.io/metadata.name"] ...

The KEP mentions that there were no valid use cases for having more than one BANP on the cluster. Of course if we have valid use cases this could change in the future, but for now, we don't have that plan.

Keeping that in mind it would be nice if we could restrict that somehow using kubebuilder validation.

An equivalent of:

// BaselineAdminNetworkPolicyList contains a list of BaselineAdminNetworkPolicy
type BaselineAdminNetworkPolicyList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:"metadata,omitempty"`
	// +kubebuilder:validation:MaxItems:=1
	Items           []BaselineAdminNetworkPolicy `json:"items"`
}

Currently looking at how this can be achieved so that implementations don't have to worry about getting more than one BANP which needs an additional check on the implementation layer.

Currently the object definitions for the AdminNetworkPolicyAPI live in /apis/v1alpha1 However in the future we'll most likely add more APIs which fall under the
sig-network-policy-api umbrella such as networkpolicy V2.... And when those land they will also be in /apis/v1alpha1

So this issue just needs to answer:

Do we need separate directories for each API? i.e (/apis/admin-network-policy-api/v1alpha1/.. and /apis/network-policy-v2-api/v1alpha1 etc)

OR

Should we really be really be only defining a single API per repo (would result in alot of duplicated code)

Follow the instructions outlined by core K8s to do so -> https://www.kubernetes.dev/resources/rename/

@ahmetb has a great collection of NetworkPolicy recipes at https://github.com/ahmetb/kubernetes-network-policy-recipes
It would be nice to import these samples in to official network policy repository.

@Dyanngg : It seems we only generated typed and scheme portions. We also need fake client-set for tests. When I generated them locally using a manual command I had these fake objects created automatically, so I am surprised this didn't happen in your PR, I'll double check the command we are using in the repo to generate the clientgen/listers/informers.

As part of this repo we've created the following website: https://network-policy-api.sigs.k8s.io/

Currently it takes you straight to the AdminNetworkPolicy API information however the network-policy-api repo was designed to be broader than that and instead house extra information and tools for all past and present policy API's

Therefore we should refactor the website to follow suite. Specifically we should

• Make a new landing page describing the subgroup and what we do -> #91
• Add a section for APIs
• [ ]Add a section for Tooling
  • [ ]Add a bit about Cyclonus and why we have it
• [ ]Add a section for In progress Initiatives and design documents

along with anything else we may see fit :)

Link for subscribe to an RSS feed in https://github.com/kubernetes-sigs/network-policy-api/blob/master/SECURITY.md#security-announcements section broken

See: #47 (comment) for details. Once fixed please update hack/verify-gofmt.sh and remove the grep -v "\/cmd\/cyclonus\/"

Multiple link broken in network-policy-api/blob/master/site-src/index.md

• [API overview]
• [Examples]
• get involved

We've(the network-policy-api-subgroup) have been working on this API since it's initial merge in #30 and agree that's it's time too cut an initial release for v1alpha1 of this API.

This issue will track that work!

• document how it needs to be done
• should we be vendoring in the config part of the repo as well? -> I'd say yes rather than letting each plugin generate their own yamls, we should be using the kustomization.yaml uniformly
• If kustomization.yaml isn't used and plugins directly generate yamls it will break them at install time since the api-approved annotation will be missing.

These are some issues that we ran into when doing OVNK implementation for making the controller install CRD, so we should certainly clarify these.

I am waiting for the clientget changes to land first to try the vendoring again since there is some reordering of repos as a result of that.

Per sig network - the idea of mesh like features for choke points / egress policies, seems to have gotten a second wind.

maybe we should reopen #9

@rikatz :)

Firstly, do we plan to have v1alpha2 and then move to beta or do we want to go straight to beta?

Checklist of items before graduation to beta (if we decide to do some of this in v1alpha2 itself, then we can move them over to that accordingly): https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/2091-admin-network-policy#graduation-criteria

• Gather feedback from developers and surveys

• ONGOING

• At least 2 implementors must provide a functional and scalable implementation for the complete set of alpha features.

• ovn-org/ovn-kubernetes#3489 - ONGOING

• Conformance tests

TODO; can happen only when we have a successful implementation.

• Evaluate the need for multiple Subjects per ANP.

This was brought up in the sig-network-policy-api meeting on March 14th; see the meeting notes/video recording

• Evaluate "future work" items based on feedback from community and challenges faced by implementors.
• Ensure extensibility of adding new fields. i.e. adding new fields do not "fail-open" traffic for older clients.
• Revisit the topic of whether this API should cover north-south traffic.

IMHO we should do this in alpha itself
ONGOING, see #86

• Scale? -> can we make this beta to GA criteria?
• What else?

While trying to pull in the vendoring for ANP implementation, we realised that we are missing the nonNamespaced.

#80 will close this.

See 63394cc#r1200872040 for details.

• See if we can improve the parser to be more accurate on what an error means
• See if we need to retry the connection errors similar to how its done in netpol v1 tests

/area conformance

@danwinship Has been extremely active within this community since the very beginning when we first implemented the API following KEP completion. We also recently went from 3 down to 2 maintainers and are feeling the strain of this transition. Due to this it makes perfect sense to have him become a maintainer here. I'm opening this issue to allow any contributors to voice any concerns or give their +1's.

I messed up with the new CRDs (ANP + BANP) and kubebuilder is generating namespaced scoped objects rather than cluster scoped ones.

Add GH actions to ensure the generated types have been created and gofmt has been run on any changes to the api itself (see the make all makefile target)

#35 depends on this one.
Downstream issue of #62

Currently Cyclonus engine does not support ANP and BANP.

We need to have a ton of documentation for the new AdminNetworkPolicy API

I've started putting together a website in #33 but there is still much more to do, we need specifically

• More Yaml examples of AdminNetworkPolicies and BaselineAdminNetworkPolicies
• Update the KEP to reflect the design changes that occurred during API review
• Spruce up this repo's readme explaining what it contains and it's relevance

In an effort to have a secure network many projects will have a set of default k8s network policies deployed. It would be useful to be able to get an idea of how the current traffic would be impacted by a network policy before deploying it. It would be nice to know what traffic will be blocked by a given policy so that we could more confidentially create exceptions, and ensure there will be as little downtime as possible on deployment.

Since we have alot of vendors (calico, antrea, cillium) providing different types of implementations of network policy admin and cluster APIs, i.e.

This is an open ended issue, talked with @stromqueen1990 about it this morning...

Calico

Calico has a "Global network policy"

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: allow-tcp-port-6379

Antrea

Antrea has a "ClusterNetworkPolicy"

apiVersion: security.antrea.tanzu.vmware.com/v1alpha1
kind: Tier
apiVersion: security.antrea.tanzu.vmware.com/v1alpha1
kind: ClusterNetworkPolicy
apiVersion: security.antrea.tanzu.vmware.com/v1alpha1
kind: NetworkPolicy

Cillium

Cillium has a "ClusterWideNetworkPolicy", which is similar ... but not identical

CiliumClusterwideNetworkPolicy

Goal

There are probably two ways to do this ...

One way to do this would be a long blog post w/ yaml snippets showing how to test these out...

• Examples of how to do L7 network policies
  • Calico
  • blah blah
• Examples of CNPs
  • Antrea
  • Cillium

... And we could publish it in the CNCF blogs..

I think the goal would be to have a table:

| ClusterScopedNetworkPolicy | | | | |

See #98 (comment) for details
/area conformance

The behaviour for both NetworkPolicy and ANP is not defined in the docs for host networked pods, which causes confusion on why those pods are [not] affected by the netpol rules. Need to clarify that in the docs.

Follow up on #34 -- what is the future of this network policy tool?

I suggest considering:

• rename to something appropriate for kubernetes-sigs (positive side effect: disambiguates from my personal project)
• define goals
  • engine
    • features
      • ANP
      • BANP
    • applications
      • understand use cases -- CI, end user, #46
  • conformance suite
    • ANP
    • BANP
  • builds, distros, devex
    • binaries
    • images
    • kubectl plugin
    • yaml for k8s deploys

Note: this is just off the top of my head, I'm sure there are more things to consider.

As per a discussion between SIG-Security and SIG-Network: We want to ensure that CNIs are using named ports so that NetworkPolicy can be used to allow access with DNS.

Our proposed first step is to write an e2e test that checks if a NetworkPolicy allowing egress to the named port 'dns' allows access to DNS regardless of what port the pods are listening on. This would then be followed up with discussion in the SIG-Network meeting and communicated through the mailing list.

Downstream issue from #62

Now that Cyclonus is merged into upstream, we want to publish Cyclonus images to GHCR (via this repo).

Images should go to:

ghcr.io/kubernetes-sigs/network-policy-api/cyclonus:v0.0.1

• 0.0.1 being an example, use Semver and auto-increment on release.

It may be worth looking at how other sigs are handling:
https://github.com/orgs/kubernetes-sigs/packages

Todos:

• Discuss release management/versioning (auto-increment, manually tag, etc.)

• Currently the ingress-rules test focus on testing ingress traffic towards subject, even with these rules in place, we need to add a new test or two to those files to test egress from subject works fine since there are 0 egress rules in those files.
• Currently the egress-rules test focus on testing egress traffic towards subject, even with these rules in place, we need to add a new test or two to those files to test ingress from subject works fine since there are 0 ingress rules in those files.