Network Service Mesh's exclude prefix mechanism is designed to protect the IPAM/Route Assignments an NSE makes from colliding with the prefixes already in use by the workloads environment (like the K8s cluster prefixes) and with the prefixes used by other NSEs to which the workload is connected.

Normally this means that if a workload connects to NSE-1 which assigns the workload an IP address and some routes, those should be added to the exclude prefix list for Requests to future NSEs.

One side effect of this is that it preclude the same IP address from being assigned to the interfaces from two different NSEs.

This is done because NSEs are presumed to be mutually ignorant of one another. There is no safe way to allow IP space overlap between two mutually ignorant NSEs.

In #119 a really good use case was raised for mutually aware NSEs in the form of multihoming a loadbalancer that assigns the same VIP address coming from multiple NSEs. It's important to support this use case.

In order for NSM to safely allow mutually aware NSEs to have overlapping IP space, it is necessary for NSEs to have a way to credibly signal that they are part of a group of mutually aware NSEs.

My initial idea for accomplishing this would be to add a field to ConnectionContext to hold a key unique to each 'awareness group' to indicate membership in that group.

There is some additional thought necessary around how those 'awareness group' ids associate to exclude prefixes so that the correct entities in the chain (like forwarders) can know its OK to allow overlapping prefixes, even if different connections are handled by different forwarwders.

@szvincze Thoughts?

Description

Current implementation of authorization for registry may be insecure when registry restarts.

Current behavior

When NSE goes to registry to register itself, registry stores it's spiffeID in a map. Registry uses this map to check that different spiffeIDs don't register NSEs with the same names. When registry restarts, the map is removed, and bad NSE has a possibility to register itself under the name of a good NSE.

Solution

• We can use gRPC metadata to transfer path, which is private and constracted during Registeration of Endpoint. Authorize chain element can get private path from gRPC medatada and check policies.
• We can store spiffeID in NSE entry in etcd. When a bad NSE tries to register itself as a good NSE, we can get entry from etcd and check that spiffeID of a good one is equal to spiffeID of a bad one. If spiffeIDs are different, registry won't allow bad NSE to register. It would work even if registry restarted, because spiffeID of good NSE is stored in etcd. (Options for how we can store spiffe id are listed below)

Options

We can store additional information in NSE CRD that can be helpful for future authorization cases. Also we can use NSE labels to store spiffeID.

Option 1: Store only NSE spiffeID in NSE CRD

Fixes registry security issue.

message NetworkServiceEndpoint {
    string name = 1;
    repeated string network_service_names = 2;
    map<string, NetworkServiceLabels> network_service_labels = 3;
    string url = 4;
    string id = 5;
    google.protobuf.Timestamp expiration_time = 6;
    google.protobuf.Timestamp initial_registration_time = 7;
}

Option 2: Store a list of spiffeIDs in NSE CRD from the path that was constructed during initial NSE registration

Fixes registry security issue. It may be useful for authorization cases that we haven't considered yet

message NetworkServiceEndpoint {
    string name = 1;
    repeated string network_service_names = 2;
    map<string, NetworkServiceLabels> network_service_labels = 3;
    string url = 4;
    repeated string path_ids = 5;
    google.protobuf.Timestamp expiration_time = 6;
    google.protobuf.Timestamp initial_registration_time = 7;
}

Option 3: Store spiffeID of NSE in NSE labels.

Fixes registry sequrity issue. Doesn't require any changes in API

nse := &registryapi.NetworkServiceEndpoint{
		Name: "nse",
		NetworkServiceLabels: map[string]*registryapi.NetworkServiceLabels{
			"networkservice": {
				Labels: map[string]string{
					"id": "spiffe://test.com/nse",
				},
			},
		},
		NetworkServiceNames: []string{"networkservice"},
	}

Motivation

Mechanisms have not general convention about return parameters. Suggested to use grpc convention about return parameters for mechanisms.
At this moment, should be refactored the next mechanisms:

mechanisms/srv6

Related to #28 (comment)

The current mechanism name is SRIOV_USERSPACE. This is a network service centric view of the world. The client connects in via a standard interface type which we didn't historically have in NSM.

@przemeklal In order to retain convention and accuracy, can you suggest a new name and open a PR refactoring the name? I think you're in the best position to suggest a good name here.

https://docs.google.com/document/d/1JCeBn4QoEl0Nh7Krq9JryFcs-qSX66DGDoewilSv4go/edit?usp=sharing

We need to recheck all helper functions in api and probabbly cover them by unut tests.

Some of helper functions are wrong.

1. See at https://github.com/networkservicemesh/api/blob/main/pkg/api/networkservice/ipcontext_helpers.go#L192 //Note: this makes ipNet incorrect

2. This is not using

   api/pkg/api/networkservice/connection_helpers.go

   Line 37 in ad449d9

   func (x *Connection) GetSourceNetworkServiceManagerName() string {

   and its wrong

3. This is not using

   api/pkg/api/networkservice/connection_helpers.go

   Line 48 in ad449d9

   func (x *Connection) GetDestinationNetworkServiceManagerName() string {

   and its wrong

Solution

1. Cleanup helpers
2. (?) Add unit tests

Add a parameter to to kernel.Mechanism:

ROUTE_LOCALNET = "route_localnet"

With getters and setters in the helpers.go:

func (m* Mechanism) SetRouteLocalNet(bool) *Mechanism
func (m* Mechanism) GetRouteLocalNet() bool // defaults to false

Motivation

Need to migrate patch networkservicemesh/networkservicemesh#2127 to this repository

Description

Currently, we support DNSConfigs related to the concrete DNS servers.
https://github.com/networkservicemesh/api/blob/master/pkg/api/networkservice/connectioncontext.proto#L66-L71.

Probably it can be used to add an additional config into https://github.com/networkservicemesh/api/blob/master/pkg/api/networkservice/connectioncontext.proto#L73
That will represent static DNS entry.

message StaticEntry {
    string domain = 1;
    string ip = 2;
}

message DNSContext {
    repeated DNSConfig configs = 1;
  + repeated StaticEntry static_entries = 2;
}

It can be used to configure DNS static entries on the client-side. Probably it is useful for external applications that using DNS domains inside.

Question

Is it make sense?

Currently proto and port fields of PolicyRoute struct are mandatory in ConnectionContext.

For this reason, if the user sets source port as any which means port range of 0-65535, then the whole range should be requested, which would end up quite a lot of policy routes.

Therefore these fields are expected to be optional.

If it implies change of the API, backward compatibility shall be considered.

Motivation

Current registry query implementation allows checking only on-field contains. We need to expand our query parameters to support operations such as AND, OR, NOT, MORE, LESS, EQUAL, CONTAINS

@edwarnicke Do we plan to use this thing somewhere?

Description

MatchesMonitorScopeSelector function takes into account only PathSegment.Name - https://github.com/networkservicemesh/api/blob/main/pkg/api/networkservice/connection_helpers.go#L150

This is wrong, because there are other fields in the PathSegment - at least we use connection ID in some places:
https://github.com/networkservicemesh/sdk/blob/main/pkg/networkservice/common/heal/eventloop.go#L60-L65
https://github.com/networkservicemesh/cmd-nsc/blob/main/main.go#L253-L256

Description

status field can be super useful for watching systems (Find(Watch=true)).
This can show that NSE/NS running or terminating and so on.

The field can be also used for heal scenarios.

Implemetation

Add into networkservices and networkserviceendpoints status enum with next states:

type Status uint32
const Unknown Status = 0
const Running Status = 1
const Terminating Status = 2

Overview

Link to the Google Docs document

https://docs.google.com/document/d/108QWsficn9adJrCppuHZcDi-AqlV7WFOaKKwYvIs-5k/edit?usp=sharing
( see at point 4.3)

Service proto

service IPAMService {
  rpc Allocate(IPContext) returns (IPContext);
  rpc Free(IPContext) returns (google.protobuf.Empty);
}

References

The alternative way for networkservicemesh/sdk#1192

I'd highly appreciate if you could provide some documentation/explanation regarding the Connection States and Connection Event Types declared in https://github.com/networkservicemesh/api/blob/main/pkg/api/networkservice/connection.proto.

I'd be especially interested from the point of Monitor Connection API (to keep track of operation state, "healthiness" of certain connections.)

#42 (comment)

Suggestion

Add registration_time for the network service endpoints.

message NetworkServiceEndpoint {
    string name = 1;
    repeated string network_service_names = 2;
    map<string, NetworkServiceLabels> network_service_labels = 3;
    string url = 4;
    google.protobuf.Timestamp expiration_time = 5;
    google.protobuf.Timestamp initial_registration_time = 6;
}

How it will be used

expiration_time updates by registry by each refresh register.

registration_time will be set once per registration by the registry side.

registration_time can be used for statistics goals and also for identity issues and sorting issues.

Motivation

1. Solves networkservicemesh/sdk#1079
2. Solves networkservicemesh/cmd-nse-vl3-vpp#3

Traffic with the VIP address as source should exit the POD via the nsmX interface instead of the default interface of the POD.

Currently the VIP address put on the loopback interface and routed by ip rules:
ip rule add from 20.0.0.1/32 table 1
ip route add table 1 nexthop via 167.0.0.1

We need to add to /api wireguard remote mechanism to be able to use wireguard with a new SDK style.

This new mechanism should be used by vlan forwarder. It should be very basic subset of kernel mechanism. The main purpose for this is to assure that the vlan forwarder will be selected by nsmgr in case of multiple forwarders.

Description

Add new field groupId to ConnectionContext to enable possibility for mutually aware nse(which belongs to the same group)

message ConnectionContext {
    IPContext ip_context = 1; 
    DNSContext dns_context = 2; 
    EthernetContext ethernet_context = 3;
    map<string, string> extra_context = 4; 
    uint32 MTU = 5;
    string group_id =6 /*new field,*/
}

Issue

part of #121

Definition of done

• Add new field to proto file (it has to be at the bottom)
• Regenerate all code

Motivation

Mechanisms have not general convention about return parameters. Suggested to use grpc convention about return parameters for mechanisms.
At this moment, should be refactored the next mechanisms:

mechanisms/vxlan

Related to #28 (comment)

go generate ./... produce *.pb.go into wrong folders, instead of proper place it put them into subfolders of every root containing .proto files.

Introduce new constant for parameters for kernel.Mechanism:

IPTABLES4_NAT_TEMPLATE = "iptables__nat_template"

And provide helpers:

# Getters and Setters
func (m *Mechanism) SetIPTables4NatTemplate(tmpl string) *Mechanism {}
func (m *Mechanism) GetIPTables4NatTemplate(tmpl string) string {}

# Helper to evaluate the iptables template using a *networkserviceConnection
func (m *Mechanism) EvaluateIPTables4NatTemplate(conn *networkservice.Connection) {}

The ip tables nat template is expected to be a golang text/template to inject a struct with fields:

• NsmInterfaceName - the name of the interface from conn.GetMechanism().GetInterfaceName()
• NsmSrcIPs - the ipv4 addresses from conn.GetContext().GetIPContext(). GetSrcIpAddrs()
• NsmDstIPs - the ipv4 addresses from conn.GetContext().GetIPContext().GetDstIPAddrs()

Example template:

-N NSM_PREROUTE
-A NSM_PREROUTE -j ISTIO_REDIRECT
-I PREROUTING 1 -p tcp -i {{ .NsmInterfaceName }} -j NSM_PREROUTE
-N NSM_OUTPUT
-A NSM_OUTPUT -j DNAT --to-destination {{ .NsmSrcIps[0] }}
-A OUTPUT -p tcp -s 127.0.0.6 -j NSM_OUTPUT
-N NSM_POSTROUTING
-A NSM_POSTROUTING -j SNAT --to-source ${NsmDstIPs[0]}
-A POSTROUTING -p tcp -o ${NSM_INTERFACE} -j NSM_POSTROUTING

Intro

Currently, we have two remote mechanisms: VXLAN (Ethernet) and Wireguard (IP).

The selection for those mechanisms is working fine because they have a diff by Payload.

What NSM should do if we will have a few IP mechanisms or a few Ethernet mechanisms?

Problems

1. How to select a remote mech for a pair nsc/nse?
2. How to select a remote mech for a couple of pairs nsc/nse?
3. How to select a remote mech for a composition of nse?
4. How to select a remote mech for a couple of compositions of nse?

Solutions

1. Extend match metadata
2. Your options

Description

We can add the SupportedMechanisms field into NetworkServiceEndpoint to directly indicate what kinds of mechanisms the Endpoint could handle.

SupportedMechanisms is a list of Mechanisms kind, such as 'MEMIF', 'KERNEL', and so on.

Motivation

This can reduce the count of NSMgr tries on selecting cross-nse or nse. It can make Request for the Connection much faster on load. (NSMgr could just filter all NSEs from the registry before do Request to the next point)

Closes networkservicemesh/sdk#802