Giter Club home page Giter Club logo

jalapeno's Introduction

Jalapeno

A cloud-native infrastructure platform to enable development of network services

Project Jalapeno combines existing open source tools with some new stuff we've developed into an infrastructure platform intended to enable development of Cloud Native Network Services (CNNS). Think of it as applying microservices architecture to SDN: give developers the ability to quickly and easily build microservice control planes (CNNS) on top of a common data collection and warehousing infrastructure (Jalapeno).

This repository provides documentation and shell scripts to install Jalapeno's component images.

To install Jalapeno and get started, visit the Getting Started guide.

view - Documentation

High level architecture

jalapeno_architecture

High level architecture

jalapeno_architecture

Platform Overview: SDN is a Database Problem

At the heart of Jalapeno is the concept that all SDN use cases are really virtual topologies whose type and characteristics are driven by dataplane encapsulations and other meta data. Thus, SDN is database problem. With this framework in mind, Jalapeno has the theoretical ability to address any kind of virtual topology use case, for example:

  • Internal Traffic Engineering (TE) - engineered tunnels traversing a network under common management (BGP-LS use cases - see note below**)
  • Egress Peer Engineering (EPE) - engineered tunnels sending traffic out a specific egress router/interface to an external network
  • SD-WAN - various combinations of TE and EPE
  • VPN overlays - engineered tunnels creating point-to-point or multipoint overlay virtual networks
  • Network Slicing - see VPN overlays
  • VPN overlays with TE, EPE, SDWAN
  • Service Chaining - engineered tunnels, potentially a series of them, linked together via or seamlessly traversing midpoint service nodes
  • Service Chaining - engineered tunnels, potentially a series of them, linked together via or seamlessly traversing midpoint service nodes

Some project principles and goals

  • Give applications the ability to directly choose their service/SLA (path through the network)
  • Enable development of an ecosystem of Network Service tools and capabilities
  • The Host may be the control/encapsulation point (linux, fd.io, eBPF, other)
  • Microservice architecture
  • Combine network and application performance data
  • Emphasize the use of APIs over Protocols for greater agility

Jalapeno's key components

Jalapeno is comprised of a series of microservices which can be summarized as:

  • Collectors - capture network topology and performance data and feed the data to Kafka. Eventually we wish to incorporate application and server/host performance data as well. The collection stack also includes Influx TSDB and Grafana for data visualization

  • Data Processors, Graph Database, and Time-Series Database - Jalapeno has two classes of processors:

    • Base data processors: parse topology and performance data coming off Kafka and populate the Influx TSDB and base data collections in the Arango graph database. The Topology and Telegraf pods are base processors.
    • Virtual Topology or Edge processors: mine the graph and TSDB data collections and then populate virtual topology Edge collections in the graph DB. Linkstate-edge is an one such processor: https://github.com/cisco-open/jalapeno/tree/main/linkstate-edge
  • Data Processors, Graph Database, and Time-Series Database - Jalapeno has two classes of processors:

    • Base data processors: parse topology and performance data coming off Kafka and populate the Influx TSDB and base data collections in the Arango graph database. The Topology and Telegraf pods are base processors.
    • Virtual Topology or Edge processors: mine the graph and TSDB data collections and then populate virtual topology Edge collections in the graph DB. Linkstate-edge is an one such processor: https://github.com/cisco-open/jalapeno/tree/main/linkstate-edge
  • API-GW - expose Jalapeno's virtual topology data for application consumption (API-GW is under construction)

  • SR-Apps - mini-applications that mine the graph and time-series databases for the label stack or SRv6 SRH data needed to execute topology or traffic engineering use cases. Each SR-App should have its own API to field client requests for Segment Routing network services.

Jalapeno's kubernetes architecture make it inherently extensible, and we imagine the number of collectors, graphDB virtual topology use cases, and SR-Apps to expand significantly as our community grows.

In this example app an end user or application would like to send their backup/background traffic to its destination via the least utilized path, and thus preserve more capacity on the routing protocol's chosen best path. Jalapeno responds to the request with a segment routing label stack that, when appended to outbound packets, will steer traffic over the least utilized path. The app then re-queries Jalapeno every 10 seconds and updates the SR label stack should the least utilized path change. In this example app an end user or application would like to send their backup/background traffic to its destination via the least utilized path, and thus preserve more capacity on the routing protocol's chosen best path. Jalapeno responds to the request with a segment routing label stack that, when appended to outbound packets, will steer traffic over the least utilized path. The app then re-queries Jalapeno every 10 seconds and updates the SR label stack should the least utilized path change.

** Note on BGP-LS

The key to developing and supporting virtual topology use cases is the programmatic acquisition of topology data. Traditional SDN-TE platforms focus on Internal-TE and therefore leverage BGP-LS. With Jalapeno we wish to eventually support all the above categories of use case, and therefore we use BGP Monitoring Protocol (BMP) and leverage the GoBMP collector: https://github.com/sbezverk/gobmp. BMP provides a superset of topology data, including: The key to developing and supporting virtual topology use cases is the programmatic acquisition of topology data. Traditional SDN-TE platforms focus on Internal-TE and therefore leverage BGP-LS. With Jalapeno we wish to eventually support all the above categories of use case, and therefore we use BGP Monitoring Protocol (BMP) and leverage the GoBMP collector: https://github.com/sbezverk/gobmp. BMP provides a superset of topology data, including:

  • BGP-LS topology data
  • iBGP and eBGP IPv4, IPv6, and labeled unicast topology data
  • BGP VPNv4, VPNv6, and EVPN topology data

jalapeno's People

Contributors

0x2142 avatar aishsree avatar aishusreeni avatar brmcdoug avatar byzek avatar dependabot[bot] avatar illidom avatar jdanielmyers avatar jklaiber avatar justaugustus avatar lelia avatar mbongard avatar mcaulfield avatar pgduda avatar remingtonc avatar sbezverk avatar skkumaravel avatar stephenrlouie avatar ubaumann avatar virlos avatar ziausyed avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

jalapeno's Issues

Wrong entries in ArangoDB

Sometimes documents in ArangoDB contain wrong entries. In the following example, the _from and _to values are incorrect and show the same information (the remaining values of the document are correct) :

wrong-entry-ls_node_edge

The reason for these wrong entries is not clear. Maybe there is a problem with an update of the documents.

Kubernetes pods are stuck in "Pending" state

Hi together,

After installing kubeadm and the deployment of the Jalapeno application with the script deploy_jalapeno.sh, all pods were stuck in the Pending state. (Could be observed with the command kubectl get all --all-namespaces)

After taking a closer look at the pods with the command kubectl describe pod/<pod-name> -n <namespace-name> I could find an error message like that:

$ kubectl describe pod/grafana-deployment-7d4dd466f5-brvp9 -n jalapeno

Name:           grafana-deployment-7d4dd466f5-brvp9
Namespace:      jalapeno
Priority:       0
Node:           <none>
Labels:         app=grafana
                pod-template-hash=7d4dd466f5
Annotations:    <none>
Status:         Pending
IP:
IPs:            <none>
Controlled By:  ReplicaSet/grafana-deployment-7d4dd466f5
Containers:
  grafana:
    Image:      grafana/grafana:6.6.1
    Port:       3000/TCP
    Host Port:  0/TCP
    Command:
      grafana-server
    Args:
      --homepath=/usr/share/grafana
      --config=/etc/grafana/grafana.ini
    Environment:  <none>
    Mounts:
      /etc/grafana from grafana-config (rw)
      /var/lib/grafana from grafana-lib (rw)
      /var/log/grafana from grafana-log (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-kwjlp (ro)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  grafana-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      grafana-config
    Optional:  false
  grafana-log:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  grafana-lib:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  default-token-kwjlp:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-kwjlp
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  89s (x9 over 7m37s)  default-scheduler  0/1 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate. 

The problem is connected to Kubernetes Taints and could be solved with the help of calebhailey/homelab#3

To solve the problem, the command kubectl taint nodes --all node-role.kubernetes.io/master- has to be executed, and the Jalapeno application has to be redeployed. After that, it should work.

Security Policy violation Binary Artifacts

This issue was automatically created by Allstar.

Security Policy Violation
Project is out of compliance with Binary Artifacts policy: binaries present in source code

Rule Description
Binary Artifacts are an increased security risk in your repository. Binary artifacts cannot be reviewed, allowing the introduction of possibly obsolete or maliciously subverted executables. For more information see the Security Scorecards Documentation for Binary Artifacts.

Remediation Steps
To remediate, remove the generated executable artifacts from the repository.

Artifacts Found

  • bin/linkstate-edge
  • bin/topology

Additional Information
This policy is drawn from Security Scorecards, which is a tool that scores a project's adherence to security best practices. You may wish to run a Scorecards scan directly on this repository for more details.


This issue will auto resolve when the policy is in compliance.

Issue created by Allstar. See https://github.com/ossf/allstar/ for more information. For questions specific to the repository, please contact the owner or maintainer.

BMP misconfiguration leads to unicast prefix entries having incorrect key

best practice is for RRs to BMP monitor their clients rather than the other way around. if an RR client runs BMP on their RR session the collector/topology processor will write an incorrect entry into the graphDB:

"key": "10.0.12.0_24_10.0.0.10",
"peer_ip": "10.0.0.10",
"nexthop": "10.0.0.12",
"bmp_ip": "10.0.0.8" <-- this RR client was configured to collect/pass BMP data from the RR to Jalapeno

}

Key should be: 10.0.12.0_24_10.0.0.12

Topology processor should probably check BMP message peer_ip == nexthop, and if != then use nexthop in key construction

Security Policy violation Branch Protection

Allstar has detected that this repository’s Branch Protection security policy is out of compliance. Status:
No protection found for branch master

This issue will auto resolve when the policy is in compliance.

Issue created by Allstar. See https://github.com/ossf/allstar/ for more information. For questions specific to the repository, please contact the owner or maintainer.

Security Policy violation Outside Collaborators

Allstar has detected that this repository’s Outside Collaborators security policy is out of compliance. Status:
Found 1 outside collaborators with admin access.
This policy requires all users with this access to be members of the organisation. That way you can easily audit who has access to your repo, and if an account is compromised it can quickly be denied access to organization resources. To fix this you should either remove the user from repository-based access, or add them to the organization.

OR

If you don't see the Settings tab you probably don't have administrative access. Reach out to the administrators of the organisation to fix this issue.

This issue will auto resolve when the policy is in compliance.

Issue created by Allstar. See https://github.com/ossf/allstar/ for more information. For questions specific to the repository, please contact the owner or maintainer.

Add IGP Router ID to ls_node_edge documents

Problem Statement

Currently, the ls_node_edge collection lacks igp_router_id and remote_igp_router_id, identifiers that are available in other datasets like ls_prefix and ls_srv6_sid. Adding these IGP Router IDs into the ls_node_edge collection would simplify retrieving related information such as Prefix or SRv6 SID data.

Proposed Solution

Add the igp_router_id and remote_idp_router_id to the ls_node_edge.

Additional Context

I'm using the ls_node_edge collection to build a network graph in my app, where nodes are routers and edges are the connections between them. The calculation is triggered with a source and destination IPv6 address. Thus, I need the ls_prefix collection to identify the source and destination router that announces the source/destination networks where the IPv6 addresses are included. The calculation result should be transformed into a SID list. Thus, I also need the ls_srv6_sid.
Adding the igp_router_id and remote_igp_router_id directly in the ls_node_edge documents makes retrieving the related documents much more straightforward.

Do you agree that this would be handy? If so, I would create a PR.

False outside port in Kafka broker-cfg

Hi together,

I think there is a false port referenced in the broker config of kafka...
In the moment the outside port is referenced to the outside port (tcp/32400) of service telegraf-ingress-np.
But it should be referenced to the outside port (tcp/30092) of service kafka
https://github.com/jalapeno/jalapeno/blob/4828a47bbda0baed563942cc6b71cb96b4f407f7/infra/kafka/2-broker-cfg.yaml#L30

I noticed the problem when I tried to access Kafka Topics from external. When doing so, strange DNS responses like the one below came back:

2021/07/16 07:22:34 failed to read messages:dial tcp: lookup broker.jalapeno on 10.43.0.10:53: no such host

As soon as the OUTSIDE_PORT variable is set to tcp/30092 then it works.

Best regards,
Julian

Wrong Processor Path after Renaming

Hi together,

After the renaming of the ls-edge-processor, the deployment is failing.
Following an example of the error during the deployment process:

Deploying LS Edge Processor
error: the path "/home/test/jalapeno/processors/ls-edge-processor/ls-edge-processor.yaml" does not exist

The solution is straight forward:
In the file processors/deploy_processors.sh following change has to be made:

Change:

echo "Deploying LS Edge Processor"
${KUBE} create -f ${PWD}/${BASEDIR}/ls-edge-processor/ls-edge-processor.yaml

to:

echo "Deploying LS Edge Processor"
${KUBE} create -f ${PWD}/${BASEDIR}/lsnode-edge/lsnode-edge.yaml

After the change, the deployment is working successfully.

Thanks.

Best
Severin

lsv6_processor crashes when no SRv6 SIDs are active in the network

Creating LSv6_Topology collection in Arango
LSv6_Topology collection exists: entering collection.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/pyArango/query.py", line 120, in getitem
return self.result[i]
IndexError: list index out of range
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "create_lsv6_topology.py", line 108, in
main()
File "create_lsv6_topology.py", line 54, in main
local_srv6_info = get_srv6_info(database, local_igpid)[0]
File "/usr/local/lib/python3.6/site-packages/pyArango/query.py", line 122, in getitem
self.nextBatch()
File "/usr/local/lib/python3.6/site-packages/pyArango/query.py", line 88, in nextBatch
raise StopIteration("That was the last batch")
StopIteration: That was the last batch

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.