The structured log field for the attribute "request.address" is actually logged as ":". This makes sorting/grouping by IP address difficult. If the source port is important (which IMO is not), it should be logged in a separate field.

Hello folks!

I'm now messing around with the webpa stack and I'm having some trouble registering a hook listener to tr1d1um.
I'm currently using the latest tags from all webpa stack on my k8s cluster and they seem to communicate with each other just fine.

So I've port forwarded to tr1d1um and I'm attempting to post an hook with this body:

{
"config": {
"url": "https://my-own-ngrok-service-address/api/v1/events/hook_listener",
"content_type": "json"
},
"matcher": {
"device_id": [
"."
]
},
"failure_url": "https://my-own-ngrok-service-address/api/v1/events/failure_listener",
"events": [
"."
]
}

I've also checked this api doc and messed around with the body (adding secret, changed failure_url to address, added until (I've saw in webpa code that duration is deprecated so I didn't added it) and even added a secret under the config object, however I get Internal Server Error every time.

The error seems to be pretty self explanatory

{
"message": "failed to add webhook to registry: statusCode 400: argus rejected the request as invalid"
}

To see if argus is working I've also port forwarded to it and attempted all the methods and they work ok and they show up in YB DB.

Argus console logs shows no errors whatsoever and tr1d1um pretty much logs the same as the message above except the msg is "Argus responded with a non-successful status code for a PushItem request".

Sorry if this is an Argus issue (seems to be(?)) however, as I said above all the endpoints documented here seems to be working great.
Any tip on where might be my problem?

Thanks in advance

/iot was a temporary endpoint added to support some internal API<->device calls

Today, we no longer need it and so we can get rid of the 'hack' we had to introduce!

Please remove all logic related to this endpoint from code, tests and config files when applicable.

The solution was already accepted in Talaria (xmidt-org/talaria@3db2d63) and we just need to port it over here

Context:

This is related to #198 but it instead focuses on the server-side of the picture.

Current approach

For server-side instrumentation, we currently have a middleware function that does two things:

1. Start the main span for the trace going through this application.
2. If this application is the first node in the trace path of applications, the span, and trace ID are both returned as part of the HTTP response headers.

Gorilla Mux Instrumentation Candidate 🦍

Instead of performing (1) ourselves, we can leverage this to instrument gorilla mux which we already use across most (if not all) of our services.

Here's some sample code on what the integration might look like: https://github.com/xmidt-org/tr1d1um/compare/example/serverInstrumentation

The two main benefits over the current approach: (1) automatic creation of named spans based on the URL path that was matched (2) span attributes to distinguish different endpoints using the same route (i.e. HTTP request method GET /user/{id} vs PUT /user/{id}).

Side note on instrumenting the HTTP Handlers directly (not recommended)

This is also an option and it consists of instrumenting existing HTTP handlers and (non-gorilla) routes. Unlike the gorilla/mux setup which allows setting the instrumentation once through the root router, this approach requires wrapping each individual endpoint to get a similar behavior.

Currently, the source is set here on each outgoing WRP Message from tr1d1um:

Since we now simply want the Source to be `dns:{{fqdn-of-server}}:

• We no longer need to use the device component service:

  tr1d1um/translation/transport_utils.go

  Line 104 in 8995ed5

  Source: service,

• We can simply update the original code (

  tr1d1um/translation/service.go

  Line 54 in 8995ed5

  wrpMsg.Source = fmt.Sprintf("%s/%s", w.WRPSource, wrpMsg.Source)

  )
  to be
  wrpMsg.Source = w.WRPSource
  assuming the WRPSource tr1d1um config value is updated correctly.

This aligns with our original specs: https://xmidt.io/docs/wrp/basics/#service-locators

Separate main.go into at least two files:

• main.go
• primaryHandler.go

Functions specifically for http handlers and not starting the service can be put in the primaryHandler.go file.

The recent update allows you to register a url with an invalid protocol when httpsOnly is disabled. e.g.:

{"config": {"url": "spongebob://96.0.0.1:80/responder","content_type": "wrp","secret":"super_secret"},"matcher":{"device_id":[".*"]},"events": [ "XMiDT-test" ],"duration":300000}

Pre-0.6.n this was rejected with a 400. We should only allow http/https.

https://github.com/xmidt-org/tr1d1um/blob/master/translation/errors.go#L16
^ An example of an error message that's not helpful at all with respect to which part of the request was invalid.

Submitting an invalid mac address still causes a 500 error to be returned. Some simple examples are using and X instead of a 0-F or an 11 digit mac address.
e.g. submitting a request with mac:11223344556X returns a 500 as does submitting a request with mac: 11223344556
Also, adding a space, omitting the mac prefix result in 500's.

If a request's jwt doesn't pass the capabilities check, still allow the request. However, log a warning that the capabilities were wrong. Make sure to include the client id and the endpoint the request is for.

This should be configurable and exist in harmony with the capabilityCheck configuration.

Hi Team,

we have upgraded all the xmidt components in our dev environment according to xmidt-org/xmidt#43, But we are getting new traceId every time when the request is transferred from one service to another service.

Root cause of the problem,

we are not forwarding the context which is having the trace information and at the end, it tries to start the span which will result in a new traceId to downstream services.

Here we are skipping the context forwarding to the new request,

A simpler solution to fix this is we need to forward the context like this

r = r.withContext(original request context ) 

or

sc:= trace.SpanFromContext(original request context)
ctx = trace.ContextWithSpan(r.contect(),sc)
r = r.withContext(ctx)

but we need to take care of this change in every microservice.

Note: It is working with Manual instrumentation which is https://github.com/xmidt-org/tr1d1um/releases/tag/v0.5.5

cc: @joe94 @schmidtw

Version: tr1d1um-v0.5.0-1
Description:
A webhook can be "posted" using the following curl command:

curl -X "DELETE"
https://[yourtr1d1um]:8090/api/v2/hook
-H 'authorization: [snip]'
-d '{
"config": {
"url": "http://somefake.url.com:8080/responder",
"content_type": "json",
"secret":"dexterslabratory"
},
"events": [ "XMiDT-test" ],
"duration":300000000000
}

This works for other http verbs as well.
Expected:
Return code 405 or 400.

In general, we want to transition our applications over to the uber fx framework. Since tr1d1um is relatively small and straightforward, we want to start here. This is also a good opportunity to move away from the webpa-common library, which is slowly being deprecated. Below are some of the things we will need to transition:

• Use arrange to help with configuration and other uber fx wiring.
• Move to zap logging. Sallust will help with this.
• Use basculehttp Provide functions for easier setup of bascule middleware.
• Use touchstone for prometheus compatible metrics.
• Set up http servers and clients with httpaux.
• No longer import webpa-common directly into any tr1d1um package. After these other changes, the only package remaining should be device. We will need to either move the functionality to a new module (this is my preference) or put the functionality we need into tr1d1um directly.

Version: tr1d1um-v0.5.0-1
Description:
The stat endpoint /api/v2/device//stat returns 400 + error message for PATCH & HEAD http requests.
Expected:
It should return 405. This is to match the behavior for unsupported http methods.

The Dockerfile was updated during the Github Actions roll out to make it consistent across the org. It now needs to be updated to the needs of this application and the exposed ports should be double checked so they don't conflict with other instances.

It seems tr1d1um & scytale tries to bind to same metrics port (8082) although it's configured to run on different ports.
It causes tr1d1um repeatedly trying to bind hence causing higher CPU usage.

RPM used:

scytale-0.1.4-1.el7.x86_64
tr1d1um-0.1.2-1.el7.x86_64

Configuration:

/etc/tr1d1um/tr1d1um.yaml:
metrics:
address: ":9082"
/etc/scytale/scytale.yaml:
metrics:
address: ":7082"

Console Error log:

#cat /var/log/tr1d1um/tr1d1um.log
{"bindAddress":":8082","caller":"webpa.go:73","error":"listen tcp :8082: bind: address already in use","level":"debug","msg":"restartable func making a loop","serverName":"tr1d1um.metrics","ts":"2019-07-04T10:03:29.578738795Z"}
{"bindAddress":":8082","caller":"webpa.go:73","error":"listen tcp :8082: bind: address already in use","level":"debug","msg":"restartable func making a loop","serverName":"tr1d1um.metrics","ts":"2019-07-04T10:03:29.579060981Z"}

netstat -plan|grep scytale

tcp6 0 0 :::8082 :::* LISTEN 4787/scytale

When starting the tr1d1um service after talaria is in service, there is a an error message in the tr1d1um.log that indicates the following:

`ts=2017-11-29T16:58:30.035982038Z caller=tr1d1um.go:89 level=info configurationFile=/etc/tr1d1um/tr1d1um.json

ts=2017-11-29T16:58:30.036595832Z caller=sns.go:144 level=debug selfURL=https://:8080/api/v2/aws/sns/1511974710 protocol=https

ts=2017-11-29T16:58:30.036637725Z caller=sns_handler.go:95 level=debug msg="handler not nil" urlPath=/api/v2/aws/sns/1511974710

ts=2017-11-29T16:58:30.036682361Z caller=webpa.go:235 level=info msg="starting server" name=tr1d1um.health address=:8081

ts=2017-11-29T16:58:30.036694028Z caller=health.go:118 level=debug msg="Health Monitor Started"
ts=2017-11-29T16:58:30.036712033Z caller=webpa.go:249 level=info msg="starting server" name=tr1d1um address=:8080

ts=2017-11-29T16:58:30.036807806Z caller=webpa.go:51 level=error error="listen tcp :8080: bind: address already in use"

ts=2017-11-29T16:58:30.036847589Z caller=sns_handler.go:116 level=debug subscribeParams="{\n Endpoint: "https://:8080/api/v2/aws/sns/1511974710",\n Protocol: "https",\n TopicArn: "arn:aws:sns:us-east-1:999999999999:fake"\n}"

ts=2017-11-29T16:58:30.037322711Z caller=webpa.go:51 level=error error="listen tcp :8081: bind: address already in use"`

tr1d1um is configured to use port 8095 and talaria uses port 8080. Need to understand why tr1d1um is still listening on port 8080 after restarting services. Please see attachments for config and logs.

This came up during a code review: https://github.com/xmidt-org/tr1d1um/pull/205/files#r609854502
However this is resolved, it'll likely have to be propagated to our other services.

Description:
Tr1d1um returns the following error when the duration of a webhook is set to a string e.g. "duration":"5m" in a webhook registration:

{
"message": "json: cannot unmarshal object into Go value of type []webhook.W"
}
Expected:
A more directed error indicating which field and value.

Update
xmidt-org/webpa-common to v1.11.5
xmidt-org/ancla to v0.1.2

CVE-2021-22878 - Medium Severity Vulnerability

Vulnerable Library - github.com/hashicorp/hcl-d7e7bca9a90f497e93c52038633506c398faec71

HCL is the HashiCorp configuration language.

Dependency Hierarchy:

• github.com/xmidt-org/bascule/basculehttp-b408708d21bcbdb521594ea846ecb1e1dd4b1771 (Root Library)
  • github.com/xmidt-org/bascule-b408708d21bcbdb521594ea846ecb1e1dd4b1771
    • github.com/xmidt-org/arrange-514e34871983067d5e05d901ec3c0efc26d2c3cd
      • github.com/spf13/viper-4613c4a95f453a29c43f8834840b4b12669491b8
        • ❌ github.com/hashicorp/hcl-d7e7bca9a90f497e93c52038633506c398faec71 (Vulnerable Library)

Found in HEAD commit: a859a3f0c97915f193b6d6aa37de1e52d5ce7ad8

Found in base branch: main

Vulnerability Details

Nextcloud Server prior to 20.0.6 is vulnerable to reflected cross-site scripting (XSS) due to lack of sanitization in OC.Notification.show.

Publish Date: 2021-03-03

URL: CVE-2021-22878

CVSS 3 Score Details (5.3)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Network
  • Attack Complexity: High
  • Privileges Required: High
  • User Interaction: Required
  • Scope: Unchanged
• Impact Metrics:
  • Confidentiality Impact: High
  • Integrity Impact: Low
  • Availability Impact: Low

For more information on CVSS3 Scores, click here.

Suggested Fix

Type: Upgrade version

Origin: https://nextcloud.com/security/advisory/?id=NC-SA-2021-005

Release Date: 2021-03-03

Fix Resolution: v20.0.6

Step up your Open Source Security Game with WhiteSource here

Here is an example of the changes required xmidt-org/svalinn#111

We're switching from SNS to Argus as the storage service for webhook registrations.

Also have to change references to the branch in .travis.yml, README, and CONTRIBUTING. Double check any other markdown files as well - sometimes links have the branch name in them.

We recently ran into an error scenario in which it would have been helpful having access to the contents (especially metadata information) of a WRP message involved in a transaction.

Creating this ticket as a way to discuss the need and potential solutions.

add the following badges:

• maintainability
• code climate
• release

should match this README:
https://github.com/xmidt-org/svalinn

https://github.com/golang/go/wiki/CodeReviewComments#package-names

We have scytale, talaria, and tr1d1um running on a single machine with the three services currently listening:

tcp 0 0 :::8096 :::* LISTEN 4180/tr1d1um
tcp 0 0 :::8080 :::* LISTEN 4157/talaria
tcp 0 0 :::6000 :::* LISTEN 4136/scytale
tcp 0 0 :::8081 :::* LISTEN 4157/talaria
tcp 0 0 :::6001 :::* LISTEN 4136/scytale
tcp 0 0 :::6002 :::* LISTEN 4136/scytale
tcp 0 0 :::8095 :::* LISTEN 4180/tr1d1um

If a curl command is sent to retrieve TR-181 parameters using this command

curl -i http://10.11.160.49:8095/api/v2/device/mac:f8a097ea7206/config?names=Device.DeviceInfo.SerialNumber

The following is displayed:

HTTP/1.1 404 Not Found
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Tue, 05 Dec 2017 17:36:41 GMT
Content-Length: 19
404 page not found

When making requests against Tr1d1um that results in 404 errors, the response is stated to be of type text/plain even when the payload is actually in JSON.

$ curl --include --header "Authorization: Bearer $SAT" https://api.webpa.comcast.net/api/v2/device/mac:12bf60291df8/config?names=Device.MoCA.Interface.1.Enable,Device.MoCA.Interface.1.Status
HTTP/1.1 404 Not Found
X-Scytale-Build: 0.1.1-120
X-Scytale-Flavor: coke
X-Scytale-Region: ch2h
X-Scytale-Server: scytale-prod-00101-120-124-89h.xmidt.comcast.net
X-Scytale-Start-Time: 06 Sep 18 05:11 UTC
X-Tr1d1um-Build: 0.1.1-297
X-Tr1d1um-Flavor: coke
X-Tr1d1um-Region: ch2h
X-Tr1d1um-Server: tr1d1um-prod-00101-297-38-lov.webpa.comcast.net
X-Tr1d1um-Start-Time: 22 Aug 18 01:20 UTC
X-Webpa-Transaction-Id: JvB4hIZJcFG3bS5Y947k4A
X-Xmidt-Span: "https://petasos-prod-coke.xmidt.comcast.net:8080/api/v2/device/send","2018-09-10T20:45:05Z","15.552742ms"
X-Xmidt-Span: "https://petasos-prod-rum.xmidt.comcast.net:8080/api/v2/device/send","2018-09-10T20:45:05Z","68.692828ms"
X-Xmidt-Span: "https://petasos-prod-ice.xmidt.comcast.net:8080/api/v2/device/send","2018-09-10T20:45:05Z","124.900738ms"
Date: Mon, 10 Sep 2018 20:45:05 GMT
Content-Length: 87
Content-Type: text/plain; charset=utf-8

{"code": 404, "message": "Could not process device request: The device does not exist"}

CVE-2020-26160 - High Severity Vulnerability

Vulnerable Library - github.com/dgrijalva/jwt-go-dc14462fd58732591c7fa58cc8496d6824316a82

Golang implementation of JSON Web Tokens (JWT)

Dependency Hierarchy:

• github.com/xmidt-org/ancla-48f750397692b0700d6d19953124ce98ae9c55f2 (Root Library)
  • github.com/xmidt-org/bascule-72ba2cf6de33b19b64df83875ba494067dc30de4
    • ❌ github.com/dgrijalva/jwt-go-dc14462fd58732591c7fa58cc8496d6824316a82 (Vulnerable Library)

Found in HEAD commit: 995d1ed7a8357a12373101da06a7fdf00c958fbd

Found in base branch: main

Vulnerability Details

jwt-go before 4.0.0-preview1 allows attackers to bypass intended access restrictions in situations with []string{} for m["aud"] (which is allowed by the specification). Because the type assertion fails, "" is the value of aud. This is a security problem if the JWT token is presented to a service that lacks its own audience check.

Publish Date: 2020-09-30

URL: CVE-2020-26160

CVSS 3 Score Details (7.5)

Base Score Metrics:

• Exploitability Metrics:
  • Attack Vector: Network
  • Attack Complexity: Low
  • Privileges Required: None
  • User Interaction: None
  • Scope: Unchanged
• Impact Metrics:
  • Confidentiality Impact: High
  • Integrity Impact: None
  • Availability Impact: None

For more information on CVSS3 Scores, click here.

Step up your Open Source Security Game with WhiteSource here

Right now Tr1d1um returns a Content-Type: application/Json header even when it returns an empty response (e.g. 404 + Content-Length: 0 header). Some automated systems are attempting to parse the body as json when the header is present.

Past solutions included passing the header as Content Type: Text/Plain, or omitting the Content Type header altogether (which could introduce other issues).

Propagate spec file change from here to other services:
#178

Once Scytale starts owning XMiDT's webhook registry, suggested steps to deprecate the webhook endpoints include:

Phase 1

Have a flag for Tr1d1um to either (1) serve the request or (2) simply redirecting it to Scytale.

Handling request: In this mode, Tr1d1um will create a new request (must pay close attention to capability check configurations between Scytale and Tr1d1um) to Scytale and then return the response to the caller. Callers should notice no difference from the current behavior in this mode.
Redirection: Perform an HTTP redirect to the scytale endpoint. This would require Tr1d1um users to setup their services and credentials to follow redirects and ultimately update the endpoint they use for webhooks

Phase 2

Drop support for the flag and only support redirection.

Phase 3

Drop support of endpoints.

Current state:

Tr1d1um delegates the work of populating the partnerIDs field in the wrp.Message to scytale

Desired state:

Tr1d1um should do that work when creating the wrp (https://github.com/xmidt-org/tr1d1um/blob/master/translation/transport_utils.go#L99). The source data of the partnerIDs field is the incoming JWT in the Authorization header.

One thing to keep in mind: What happens if tr1d1um is not configured to accept JWTs?
Fall back to header

We need the work for https://github.com/xmidt-org/webpa-common/issues/559 to be done first.

Current state:

Once endpoint capability checks are enforced in Comcast's Scytale, Tr1d1um will no longer be able to forward its webpa JWT token credentials.

Desired state:

Tr1d1um should now create, cache and present a new xmidt JWT token to Scytale which includes:

• All endpoint capabilities Tr1d1um needs to perform as it is today
• PartnerIDs claims in the JWT: [*].
• Make enabling this behavior configurable

Deployment dependencies:

#139 (WRP Message with populated fields)
xmidt-org/scytale#83 (Validation for wildcard partnerID *)

Context:

What we're currently proposing in #197 will do the job and maybe that is all we need for a first release. It might still be worth spending a bit of time exploring the instrumentation packages OTEL already provides. The big possible advantages are that we wouldn't have to change so much code near our business logic and we wouldn't have to perform trace propagation ourselves.

HTTP Client instrumentation

Specifically, instead of manually propagating trace information like this:

candlelight.InjectTraceInformation(ctx,r.Header)

We could instead use OTEL's instrumented Transport
https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/net/http#NewTransport

The instrumentation would roughly look like this: https://github.com/xmidt-org/tr1d1um/compare/example/clientInstrumentation

This is the more generic example they provide: https://github.com/open-telemetry/opentelemetry-go-contrib/blob/main/instrumentation/net/http/otelhttp/example/client/client.go

We're seeing the wrong registered_from_address being returned by tr1d1um when asking /hooks.

#235 (comment)

As part of streamlining the error reporting for api requests:

1. payload returned for a non-success responses should be empty
2. a header (x-xmidt-message-error ) should be return with the appropriate error message

See xmidt-org/talaria#126 for context

Talaria's main.go file already has the solution model.

Version: tr1d1um-v0.5.0-1
Description:
Tr1d1um doesn't appear to validate any webhook fields, aside from requiring that they be present, if mandatory. Moving forward we want to verify these fields.
For example, the url for failover isn't verified. The server will allow you to register this failover url:
"failure_url" : "wss://https://http://saulgoodman:bettercallsaul"

Resulting in this (from /api/v2/hooks)
{
"config": {
"url": "[redacted]",
"content_type": "wrp",
"secret": "breakingbad"
},
"failure_url": "wss://https://http://saulgoodman:bettercallsaul",
"events": [
"XMiDT-test"
],
"matcher": {
"device_id": [
".*"
]
},
"duration": 300000000000,
"until": "2020-03-24T00:22:25.517636689Z",
"registered_from_address": "[redacted]"
}
Expected:
failure_url would be validated.

The capabilityConfig values should be explained in the tr1d1um.yaml file, but all values should be commented out - if anyone uses the default yaml to test or play around, they won't need to check for capabilities in the jwt.

A model format is: https://github.com/xmidt-org/svalinn/edit/master/CHANGELOG.md

Example patch: https://github.com/xmidt-org/argus/blob/c2562df18c76f64ff59262c55a45bb5804566a00/auth/logger.go#L49-L58
Related to xmidt-org/xmidt#52

Currently, this configuration is required for the tr1d1um application to start:

Ideally, we would like tracing to be optional configuration. If no configuration is provided, the application should be able to successfully start and run without tracing functionality.

The two options I know of are:

1. Implement a Discard/No-op TracerProvider (and subsequent Tracer). This would most likely be best to be implemented in candelight. Then, if tr1d1um isn't given a tracing.provider in configuration, it will fall back to creating a no-op tracer provider.
2. If tr1d1um doesn't receive a tracing.provider in configuration, the application should not attempt to create a TracerProvider or add candlelight middleware.

Note: my proposals specifically ensure that the application checks that tracing.provider is an empty string. If tr1d1um receives a tracing provider in configuration, failing to set up what has been given should still result in an error on startup.

One of our community members reported in xmidt-org/xmidt#51 that it would be nice having a way to mask sensitive information such as device IDs (information that could be traced back to an individual customer) from log entries.

Some of the log field names that include sensitive data across our microservices are:
requestURL
requestURI
requestHeaders.Referer
instance

Suggested Approach
Compile a global regex to match device IDs from within a string and replace all characters with a * except the prefix. Ex: mac:1122334455 would become mac:**********

Configuration:

log:
  ...
  # defaults to false meaning sensitive data is masked by default
  disableSensitiveDataMasking: false