microsoft / cpp_client_telemetry Goto Github PK

A single worker thread is managed by the PAL singleton. The following components use the worker thread to dispatch async tasks (via PAL::scheduleOnWorkerThread()):

• HttpClientManager
• OfflineStorageManager
• TransmissionPolicyManager
• Statistics

Maintaining only a single worker thread reduces the number of thread resources used by the SDK.

However, I would like to support a scenario where the SDK creates no threads and doesn't directly manage any thread resources. For instance, server applications often restrict components from creating their own threads and would rather manage those thread resources themselves.

I propose to accomplish this by allowing an application to implement its own IWorkerThread and pass it to the SDK during LogManager initialization. However, this will still need to work with multi-host/guest scenarios. My initial thinking is:

• Host log managers that specify their own IWorkerThread should use that worker thread for all async task dispatching.
• All other log managers should use the default, single-instance IWorkerThread implementation.

How to accomplish this? Here's my rough idea:

1. Move IWorkerThread interface to public header
2. Initialize LogManager with optional IWorkerThread parameter
3. PAL only creates its own worker thread if it not provided one by LogManager
4. IWorkerThread becomes owned by ITelemetrySystem which has the same lifetime as LogManager
5. Move PAL::scheduleOnWorkerThread() functionality to ITelemetrySystem
6. ITelemetrySystem will only use PAL's worker thread if it doesn't have its own.

Thoughts?

To reproduce, disconnect network on the client system, initialize logger, and send events. Reconnect network, possibly sending more events.
Expected result: logger initialization succeeds and events are stored in SQLite and sent when network is reconnected.
Result: the events are never sent.

Currently 1DS SDK is not built with the same set of compiler dependencies and warning levels as MS Edge. It would help if we can clean up OneDS warnings & Integrate a CI target with a very similar build setup to that of MS Edge (Windows clang-cl, Mac/Linux clang with strict warnings), with continuous run of BVT (unit and functional). That way all potential breaking issues can be caught early-on prior integrating a new SDK source code drop in MS Edge.

SDK customer must be able to tell if SDK is indeed operating in UTC channel mode.

One possible fix here is to expose a boolean flag via LogManager::GetLogConfiguration() e.g. isUTCModeActive

When OneDS is used to send sensitive application audit data, there may be a need to encrypt the offline storage to prevent attackers--or even legitimate users--from reading or modifying the data. Today, any user can open the sqlite database and see the bond-serialized data in the clear.

• Should the default offline storage perform encryption?
• How would encryption keys be managed?
  • Managed/provided by host application?
  • Managed by OneSDK via OS-specific key manager? (e.g. ncrypt for win32)
• How to encrypt? (openssl? Or OS-specific encryption APIs?)

Alternatively, should an application be responsible for injecting its own IOfflineStorage storage implementation that happens to perform encryption? Is this issue common enough that OneDS should provide encryption out of the box?

Edge ran into a build issue, because we've been relying on the AriaDecoderV3 header for build targets external to our OneDS build, but ce1e345 updated the AriaDecoderV3 header to rely on the HAVE_MAT_JSONHPP define and several different internal include paths.

I put up #44 as a fix, but I think the ideal fix would be to make a public decoder header in lib/include/public to make it more explicit that external targets rely on that header, and internal build-state and include paths shouldn't be added to that header.

Curious on opinions on this, because it's not part of the typical API, but it's also not an internal-only header.

I run Visual Leak Detector with Omaha Installer. There is a new leak showing up:

---------- Block 7092 at 0x00F20A18: 24 bytes ----------
  Leak Hash: 0x90EF44E6, Count: 1, Total 24 bytes
  Call Stack (TID 21988):
  Data:
    6E 6F 20 73    75 63 68 20    74 61 62 6C    65 3A 20 70     **no.such. table:.p**
    61 63 6B 61    67 65 73 00                                   **ackages. ........**

Visual Leak Detector detected 1 memory leak (60 bytes).
Largest number used: 1007780 bytes.

Looking at the OneDS log, this allocated string appears here:

2019-06-20 16:58:50.166 T#21988 <D> [EventsSDK.SQLiteDB] DELETE FROM packages
2019-06-20 16:58:50.166 T#21988 <D> [EventsSDK.SQLiteDB] rc=1: no such table: packages
2019-06-20 16:58:50.166 T#21988 <D> [EventsSDK.SQLiteDB] Failed to execute query: DELETE FROM packages [rc=1]

When looking in the code, you can see that errmsg ptr is not deallocated:

Ask from MS Edge team.

Multiple loggers: We want to be able to concurrently initiatize multiple loggers pointing to different transportation options (Direct upload versus UTC).

There are several fields that cannot be currently populated by SDK. Example:

• ext.app.env
• ext.container
• etc.

Feature request:

• add new CS4.0 fields definition and regenerate the C++ bond-related code
• introduce a generic schema mapping mechanism that would allow to expose any new CS protocol field via SetProperty and SetContext API to the app caller

Request comes from MS Edge team.

getOsBuildLabEx() is a method that is used to get the OS version. It uses RegGetValueA() function to access the HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\BuildLabEx registry, which contains the os version string.

On my machine, the value of the registry is 17763.1.amd64fre.rs5_release.180914-1434, but when looking at the events sent from Omaha (32-bit installer application) the version differs in the architecture part: 17763.1.x86.rs5_release.180914-1434. It is because the actual registry that is being accessed is HKLM\SOFTWARE\WOW6432Node\Microsoft\Windows NT\CurrentVersion\BuildLabEx, which is default behavior for RegGetValueA().

A OneDS crash got promoted last week for an invalid pointer execution (m_backoff) in TransmissionPolicyManager::handleNothingToUpload. Relevant crash information is in the bug: https://microsoft.visualstudio.com/Edge/_workitems/edit/22132099

Looking at the builds it got promoted from, and the changes that went in around that time, I couldn't find any relationship between us taking an updated version of OneDS, or changing how we called into OneDS, and when the bug started appearing.

I'm not very familiar with this pat of OneDS, but I see there's an unused mutex in TransmissionPolicyManager, so I wonder if checkBackoffConfigUpdate updates m_backoff on one thread, but another thread tries to execute the old m_backoff pointer? The relatively low, but consistent, number of hits makes sense to me for that kind of race condition.

Update FlushAndTeardown, or add a new function, that tries more aggressively to flush telemetry, since there’s no offline storage in "compact" mini SDK SKU flavor. This item is potentially related to #6 which is also asking for guidance on how to get certain events be "immediately" uploaded.

Another possible avenue here is to explore an enhancement that'd utilize OS implementation of Win 10 winsqlite3.dll .. That way we could have enabled the offline storage for transient installer / updater processes and maintained the same small size without sacrificing offline storage feature.

This ask comes from Office Desktop telemetry.

The new MSP flags / bitmask field allows to determine what ring / group the user belongs to; also allows to enable routing of a subset of data to Enterprise Data Platform.

Requirements:

• allow the customer to set MSP flags via API call at individual event-level (SetProperty or SetContext)
• send MSP flag on wire in Bond 3.0+ schema to Collector++

The DLL and PDB filenames contain "Windows-Managed", and if they're included in a C# project as content (as only the winmd file is included as a reference), Visual Studio interprets the dash in the filename as a language name (just like with en-US), and generates a warning during packaging since the app doesn't actually support such as "language"

1DS SDK is already available for Linux. But we need to improve / standardize certain fields:

• Device ID : currently there is no standard way of obtaining Hardware ID on Linux. SDK uses some very scientific method to anchor to certain hardware identifiers, hostname, interface mac address, etc. to generate a GUID of these. The process is not documented.

• OS Name and Version : unfortunately there is no consistent approach to labeling various Linux distros and flavors of mainline distros, as well as versioning schema differs between distros.

These are just two common examples. There could be other fields that need further exploratory work on identifying and resolving inconsistency issues. Resolution of this issue is required for delivering MS Edge on Linux.

I have just cloned the repo and have VS 2017.

When I first attempted to open the solution I was prompted to install VS tools, which I did.

Next time I was able to open the solution, but building the SampleCpp project resulted in a syntax error (attached).

When I commented that line out and rebuilt, I ended up linker errors (attached).

Since the win32-dll project actually built successfully, I decided to ignore those errors and attempt to generate a nuget package for the sdk.

Running build-nugets.cmd from /tools resulted in many errors for native and net40, with a success message for uap, though I suspect there are no files in the package. The errors seems to be due to a non-existent /dist folder and absence of any /lib or /include folder inside /tools/Nuget/native. (Attached)

I know I must be doing something fundamentally wrong, just not sure what it is.

LinkErrors.txt
PackageErrors.txt
SyntaxError.txt

Currently, the client app has no way of knowing if telemetry is working. This is useful at least in dev scenarios as there's a delay for events showing up in the portal (vs deployment scenarios where there's not much the app can do in case of telemetry errors)

This will allow additional optional initialization for the http client. For example, in case of http client based on libcurl, the curl handle will be passed to the callback and additional setup of options like SSL validation flags/callbacks can be done.

Current SDK codebase is already compiling / building for ARM64 Desktop. However, it has never been tested on Win10-ARM64. Need to instrument a lab setup to run ongoing BVT (unit and functional, plus possibly Win 10 UWP ARM64 app run) on ARM64 Desktop device.

Problem
Right now, time points and durations are represented by a pletora of types (to name a few):
int, unsigned, uint64_t, int64_t

This is undesirable, as not only is it inconsistent, but C++11 introduces the chrono header, which provides both:
std::chrono::time_point - A representation of a point in time.
std::chrono::duration - A representation of the span between two points in time.

Proposed Solution
Standardize on std::chrono types for representing durations and points in time. Use these types everywhere possible (it may not be doable in the C-API). Serialize these values to the appropriate CS types during serialization, not before.

To start, the PAL methods which return the current time should be updated to return a std::chrono type. This way, the epoch can be controlled from a central point and differing OS epochs (.NET, Win32, UNIX, etc.) can be abstracted away from the caller.

Non Goals
Will not address time representations in the public API surface of the SDK, as that's a bigger conversation.

Describe how to compile C++ SDK for iOS.

SetPolicyBitFlags(policyBitFlags) API should accept MICROSOFT_EVENTTAG_MARK_PII flag to tag event as containing private and sensitive data. Such events should only be forwarded to GPDR-compliant storage and never forwarded to non-compliant destinations.

This work item covers support for this flag over direct upload channel.

Hi,
I am using Aria SDK version 3.2.999.1 on Windows 10. I am setting ANID using Semantic Context API called SetUserANID(""). I saw the documentation where it says ext.intweb.anid is the common schema now.
I cannot see the ANID property at all in fiddler capture.

Currently it is impossible to specify event priority that is required for batching and device state-based filtering logic via C API
Thanks

There were recent features added to MS ASG since July 20th. The goal is:

• integrate all these recent changes from msasg:onesdk to github:master
• make MS ASG repo read-only
• implement daily GitHub action to push from the 'master' to 'onesdk', maintaining the legacy MS ASG location as a read-only snapshot

Adaptive Cards SDK (opensource) would like to share their SDK source code alongside with 1DS C++ SDK. When they go opensource, the guys would like to avoid exposing their own PROD iKey in their source code.

The ask is that we respect an inert / inactive / known-invalid iKey. If SDK is obtaining a logger or initialized with that known invalid deadbeef- iKey, then all events sent to that iKey must be effectively discarded, akin to redirecting the output to /dev/null

MS Defender ATP runs on Mac and has the knowledge of the OS proxy settings. The ask is to implement a custom config section in ILogConfiguration that allows to pass the proxy type, host and port to SDK. Subsequently the SDK has to apply those settings at IHttpClient layer. Initial implementation is required for Mac, with Windows implementation not immediately required at the moment (because on Windows we'd typically auto-discover the proxy).

The common framework for specifying proxy settings is gonna be available cross-plat, with functional implementation on Mac, and with a stub implementation on Windows. TBD if we require the functional implementation on Windows later on.

SetPolicyBitFlags(policyBitFlags) API should accept MICROSOFT_EVENTTAG_MARK_PII flag to tag event as containing private and sensitive data. Such events should only be forwarded to GPDR-compliant storage and never forwarded to non-compliant destinations.

This work item covers support for this flag over UTC channel on Windows 10.
Separate work is needed to cover this for direct upload on Win7/8, Mac and Linux.

I have build C++ SDK locally by following instructions here:
https://microsoft.sharepoint.com/teams/1ds.sdk.cpp/SitePages/Getting-started-with-1DS-C++-SDK-on-Windows.aspx.

Then I am tried to import the ClientTelemetry.dll and ClientTelemetry.lib to our project. But I got link errors:
d:\adssnr_2\qlocal\cmd\9\private\ads.pa.jennya.product\src\jennya\jennyalib.lib(AriaLogger.obj) : error LNK2001: unresolved external symbol "public: void __cdecl Microsoft::Applications::Events::EventProperties::SetProperty(class std::basic_string<char,struct std::char_traits,class std::allocator > const &,__int64,enum Microsoft::Applications::Events::PiiKind,enum Microsoft::Applications::Events::DataCategory)" (?SetProperty@EventProperties@Events@Applications@Microsoft@@QEAAXAEBV?$basic_string@DU?$char_traits@D@std@@v?$allocator@D@2@@std@@_JW4PiiKind@234@W4DataCategory@234@@z) [d:\adssnr_2\qlocal\cmd\9\private\ads.pa.jennya.product\src\jennya\jennya.vcxproj]

Here is my PR:
https://msasg.visualstudio.com/DefaultCollection/Bing_Ads/_git/AdsSnR/pullrequest/1121632, Please help take a look. AriaLogger.cpp is the module calling Aria Api.

Got it. Thanks James!

Regards,
Reiley

From: James Todd [email protected]
Sent: Friday, August 16, 2019 9:27 AM
To: Tielman de Villiers [email protected]; Reiley Yang [email protected]; Rama Durairajan [email protected]
Subject: RE: Anaheim catchup

Here is information about how a client can get an MSA token that can be consumed by the OneDS Collector service.

The OneDS Collector service uses server-side APIs to decrypt and use information in the MSA token sent by the client. This may include information such as a user PUID value, a Windows 10 global device ID, and more.

On the client side, clients use APIs to ask for an MSA token. When asking, the client should ask for a token for the site called “events.data.microsoft.com”, which in MSA terms is SiteID number 296123.

Android support for MSA - https://identitydocs.azurewebsites.net/static/msa/android.html
iOS support for MSA - https://identitydocs.azurewebsites.net/static/msa/ios.html

Once the client has an MSA token, it can be included in the payload when the client sends a batch of events to the OneDS Collector. The technical details for @reiley are these:

• Specify the X-Tickets http header as described on https://osgwiki.com/wiki/Vortex/ClientProtocol#Http_custom_header_support to include a user MSA token.
• In the JSON representing an event that should be associated with a user, add a field in the “protocol” extension that specifies that the event has tickets that apply to it, and that one of those tickets is an application token. This is described on https://osgwiki.com/wiki/Vortex/ClientProtocol#Associating_an_event_to_an_auth_ticket.

James

Cloned from Task 20721275.

Please attempt to mitigate the load time costs during SDK initialization.

Need to start with field-by-field comparison of direct in-proc upload vs. Win 10 UTC channel.

• Direct in-proc upload uses CS3.0 Bond protocol.
• UTC upload uses CS2.1/Vortex to CS3.0/Collector JSON protocol.

The task is to identify the fields that are missing and do feasibility study what fields can be populated internally at SDK level and what not. At this time SDK is populating all mandatory fields and some optional Part A fields populated by UTC are inaccessible via SDK in direct mode.

Problem
As we add more configurability and modularity to the SDK, this inevitably leads to more arguments being passed to the methods responsible for constructing an ILogManager. This has already grown hard to sustain and maintain, and will only get worse until it is wrangled.

Proposed Solution
Methods responsible for constructing an ILogManager should take a LogManagerConfiguration structure, as of right now this is defined as:

struct LogManagerConfiguration
{
    ILogConfiguration LogConfiguration{};
    IHttpClient* HttpClient{};
    // Others currently in code review.
};

This would be passed to LogManagerProvider's methods (as well as the associated factory functions that generate an ILogManager object) to initialize an ILogManager. This then means that the set of overloads for Initialize() is kept slim and maintainable.

LogManager Initialize method ignores the transmit profile setting specified in ILogConfiguration . The workaround is to manually apply the profile after initialization, but it might be potentially causing a race condition: events might get in-flight prior to apply the active profile.

Implement pure cross-plat .netcore -compliant C# PInvoke projection layer on top of C API.

Our existing customers (MS Office, Edge) utilize OnlineId API to retrieve a global device ticket presentable to Collector++ service. That API allows to uniquely identify the global MSA Device Id, that a device gets assigned once during Windows installation.

Unfortunately there is no reference code example in SDK how to do that. Other customers are interested in enabling that flow and frequently asking for a customer sample.

Edge telemetry needs a way to stamp the events with Lab information. In order to achieve this we need an API that would allow to set below fields:

• ext_app_env
• ext_utc_flags

Please refer to Common Schema 3.0 wiki for information about the fields.

Currently it's possible only via the C++ API and missing from the C API.

Bond serializer/deserializer auto-generated code needs to be updated with new CS4.0 schema definition.

Cloned from Task 20657846 and may be related to this msvc bug:

https://developercommunity.visualstudio.com/content/problem/169569/this-threadsleep-for-hangs-when-system-clock-moved.html?childToView=484269#comment-484269

Contact to more information: James Todd.

Below is the email conversation around this problem.

From: James Todd [email protected]
Sent: Wednesday, August 7, 2019 11:02 AM
To: Aswath Mohan [email protected]; Rama Durairajan [email protected]; Kari Richardson [email protected]; LeRoy Tuttle [email protected]; Andrea Chin (FX) [email protected]; Euan Grant [email protected]; Xianfang Wang [email protected]; M. Vivekananthan [email protected]; Ashish Poddar [email protected]; Jennifer Rowe [email protected]
Cc: Felanice Yancy [email protected]
Subject: RE: Design for new data flow path - SAN personalization

Catching up on this thread from being out the last few days. @Aswath, you are correct that the transport itself is over https to the Collector. That’s true for all connections to the Collector from any OS. However, the point being made here is that there is an additional level of defense in depth that can be used on the Windows OS, which is that the client code can check to see if the SSL cert being provided by the Collector is chained to the Microsoft CA root, proving that the Collector being uploaded to is really own by Microsoft and not some man-in-the-middle attacker. The Windows UTC’s code here does this extra check as an added layer of security.

By default Windows machines trust the Microsoft cert authority, but other OSes do not. As a result non-Windows OSes do not usually upload to the endpoints that provide a Microsoft-specific SSL cert, and that is why there are different endpoints for Windows and non-Windows OSes.

It is up the client team to decide if they want to leverage this same kind of logic on the client side to double-check that the Collector’s SSL cert is signed by the Microsoft CA.

James

From: Aswath Mohan [email protected]
Sent: Monday, August 5, 2019 4:56 PM
To: Rama Durairajan [email protected]; Kari Richardson [email protected]; LeRoy Tuttle [email protected]; Andrea Chin (FX) [email protected]; James Todd [email protected]; Euan Grant [email protected]; Xianfang Wang [email protected]; M. Vivekananthan [email protected]; Ashish Poddar [email protected]; Jennifer Rowe [email protected]
Cc: Felanice Yancy [email protected]
Subject: RE: Design for new data flow path - SAN personalization

Hi Rama,
I don’t understand the specifics of the concern. In all OS implementations the data leaving the client will be encrypted as it will be sent over https. I’m guessing all major OS in the market handle security of https certs in a secure way.
Regards,
Aswath

From: Rama Durairajan [email protected]
Sent: Friday, August 2, 2019 9:27 AM
To: Aswath Mohan [email protected]; Kari Richardson [email protected]; LeRoy Tuttle [email protected]; Andrea Chin (FX) [email protected]; James Todd [email protected]; Euan Grant [email protected]; Xianfang Wang [email protected]; M. Vivekananthan [email protected]; Ashish Poddar [email protected]; Jennifer Rowe [email protected]
Cc: Felanice Yancy [email protected]
Subject: RE: Design for new data flow path - SAN personalization

Sorry for the late reply, as Kari mentioned, there is still a security issue on non-windows. Are we okay with that?

From: Aswath Mohan [email protected]
Sent: Friday, August 2, 2019 8:58 AM
To: Kari Richardson [email protected]; LeRoy Tuttle [email protected]; Andrea Chin (FX) [email protected]; James Todd [email protected]; Rama Durairajan [email protected]; Euan Grant [email protected]; Xianfang Wang [email protected]; M. Vivekananthan [email protected]; Ashish Poddar [email protected]; Jennifer Rowe [email protected]
Cc: Felanice Yancy [email protected]
Subject: Re: Design for new data flow path - SAN personalization

@Andrea Chin (FX) and @rama Durairajan

Can you work on an ETA for the client code to send data to collector ++?

I will start a weekly sync with client/collector/pdos/amc from next week to track this.

Aswath

Get Outlook for iOS

From: Aswath Mohan [email protected]
Sent: Wednesday, July 31, 2019 1:59 PM
To: Kari Richardson; LeRoy Tuttle; Andrea Chin (FX); James Todd; Rama Durairajan; Euan Grant; Xianfang Wang; M. Vivekananthan; Ashish Poddar; Jennifer Rowe
Cc: Felanice Yancy
Subject: RE: Design for new data flow path - SAN personalization

Hi All,
Here is a summary of the items discussed in this thread. Please highlight any other open items on the data flow path.

1. We plan to send 2 browse history events, one for SAN and another for HJ/CUV

2. SAN events annotated with dedicated tenant ID70109aa3567b40e3bb8ac9e67a07b58a

3. 1DS client  Collector++  PDOS and Bing G
   a. Windows end point to send tov20.events.data.microsoft.com with Microsoft rooted SSL certs
   b. Non Windows end point – functional.events.data.microsoft.com. @james Todd and@Rama Durairajan are these the right end point for this?

4. SAN events routed to Bing G –PR complete
   a. Any action needed to prevent SAN events from landing in cosmos 15 or other locations?

Regards,
Aswath

From: Kari Richardson [email protected]
Sent: Wednesday, July 31, 2019 12:05 PM
To: LeRoy Tuttle [email protected]; Andrea Chin (FX) [email protected]; James Todd [email protected]; Rama Durairajan [email protected]; Euan Grant [email protected]; Aswath Mohan [email protected]; Xianfang Wang [email protected]; M. Vivekananthan [email protected]; Ashish Poddar [email protected]; Jennifer Rowe [email protected]
Cc: Felanice Yancy [email protected]
Subject: RE: Design for new data flow path - SAN personalization

Adding @jennifer Rowe

From: LeRoy Tuttle [email protected]
Sent: Tuesday, July 30, 2019 12:18 PM
To: Andrea Chin (FX) [email protected]; James Todd [email protected]; Rama Durairajan [email protected]; Kari Richardson [email protected]; Euan Grant [email protected]; Aswath Mohan [email protected]; Xianfang Wang [email protected]; M. Vivekananthan [email protected]; Ashish Poddar [email protected]
Cc: Felanice Yancy [email protected]
Subject: RE: Design for new data flow path - SAN personalization

FYI –

A dedicated tenant ID (70109aa3567b40e3bb8ac9e67a07b58a) has been created for SAN-related traffic.

From: Andrea Chin (FX) [email protected]
Sent: Tuesday, July 30, 2019 11:55 AM
To: James Todd [email protected]; Rama Durairajan [email protected]; Kari Richardson [email protected]; Euan Grant [email protected]; Aswath Mohan [email protected]; Xianfang Wang [email protected]; M. Vivekananthan [email protected]; Ashish Poddar [email protected]; LeRoy Tuttle [email protected]
Cc: Felanice Yancy [email protected]
Subject: RE: Design for new data flow path - SAN personalization

+@leroy Tuttle

From: James Todd [email protected]
Sent: Monday, July 29, 2019 6:09 PM
To: Rama Durairajan [email protected]; Kari Richardson [email protected]; Euan Grant [email protected]; Aswath Mohan [email protected]; Xianfang Wang [email protected]; Andrea Chin (FX) [email protected]; M. Vivekananthan [email protected]; Ashish Poddar [email protected]
Cc: Felanice Yancy [email protected]
Subject: RE: Design for new data flow path - SAN personalization

I believe it goes all the way down to Windows 7 since UTC runs on Windows 7.

James

From: Rama Durairajan [email protected]
Sent: Monday, July 29, 2019 5:03 PM
To: James Todd [email protected]; Kari Richardson [email protected]; Euan Grant [email protected]; Aswath Mohan [email protected]; Xianfang Wang [email protected]; Andrea Chin (FX) [email protected]; M. Vivekananthan [email protected]; Ashish Poddar [email protected]
Cc: Felanice Yancy [email protected]
Subject: Re: Design for new data flow path - SAN personalization

Security is a concern across all platforms & we need to address it ubiquitously - any thoughts on how we can make this work across all OSes?

Also, when we say Windows OS, are we talking about a specific version or does this include all down-level versions as well?

Sent from Outlook

From: James Todd [email protected]
Sent: Monday, July 29, 2019 11:27
To: Kari Richardson [email protected]; Euan Grant [email protected]; Aswath Mohan [email protected]; Rama Durairajan [email protected]; Xianfang Wang [email protected]; Andrea Chin (FX) [email protected]; M. Vivekananthan [email protected]; Ashish Poddar [email protected]
Cc: Felanice Yancy [email protected]
Subject: RE: Design for new data flow path - SAN personalization

Agreed on the security concern since Windows OSes should use the endpoints that provide the Microsoft-rooted SSL cert for extra security, and should confirm that the SSL cert’s root is the Microsoft CA. This is what UTC does in the Windows OS. To reiterate what I said several emails below, the endpoint for functional data on Windows is “v20.events.data.microsoft.com”. So@Rama, while it complicates the client story I prefer that you do use different functional endpoints on Windows and non-Windows, and add the check for the Microsoft-rooted SSL cert on Windows.

James

We are using the v3 SDK for sending our both telemetry and Audit data.

I've read the documentation with regards to smart transmission but I'm not sure I understand the numbers below the event priority.

As our app shuts down, we currently upload everything (uploadNow()), that may cause a delay on shutdown as all pending events are uploaded.
We don't mind caching the Telemetry events for next app start but we can't wait with the Audit events.
Is it possible to send all Audit events immediately and batch Telemetry?

A bonus would be uploading all Audit events on shutdown while Telemetry will stay cached.

Thanks

Our current today's implementation of C++/CX and C++/CLI projection layer for C# does not cover strong types. All key-value pairs passed down to SDK are assumed to be string-string. Since C++ layer has the strong types support, it should be possible to expose better API to C# apps, to allow setting individual event props as strongly-typed (float, integer, Boolean, guid, etc.).

MS Edge team has asked for a new feature that would allow to emit telemetry using the same API surface (LogEvent or LogTrace) and send it to local ETW sink. There are several implementation options:

• non-routable iKey. That way telemetry can be shown in DDV tool and TRTT.

• custom routing to Event Tracing. That way the logs can be seen in ETW listener or a tool like "Microsoft Message Analyzer". It is possible to bind the LogTrace in that mode to existing SDK tracing mechanism, that allows sending ETW events as "raw strings" to custom ETW provider GUID.

Provide a set of build cmake scripts and java to compile C++ telemetry SDK for Android using Android NDK.
This will use the Android-preferred Java HttpUrlConnection library, a Java thread pool, and Java shim layer to implement an android implementation of IHttpClient.

The network detection logic is still disabled in Edge. This is somewhat related to #10 , in that it's in the network detector, but appears to be a separate issue. From my memory (I think Watson got rid of these older cabs), it particularly affected surface books, and was mainly an issue on laptops (makes sense if network is changing a lot).

Users weren't experiencing a hang, but were getting a crash. The exceptions captured indicated a thread trying to re-enter the _GetCurrentNetworkCost. An engineering manager ran into the issue and commented "Just a guess... is this because NetworkCost NetworkDetector::_GetCurrentNetworkCost() calls into netprofm!CPubINetworkListManager::GetCost with a mutex held, and then is reentered and attempts to acquire same mutex, causing an exception?", which seems plausible

There is a highly intermittent test failure happening only on GitHub Actions-hosted VM.
And only on x64-Debug. There is no local repro.

It's either:

• timing; or
• antivirus / antimalware killing the process while trying to access 0.0.0.0 or 127.0.0.1

The test is attempting to resolve non-existing DNS, and it starts with WinInet HTTP post call to 0.0.0.0 and 127.0.0.1 , so the likely cause is the process getting killed / blocked by some external monitor. It could also be due to popup that interactively asks the user to approve the process to access the network.

Those are the theories at this point. Log output observed when the test fails:

[ RUN      ] APITest.LogManager_BadNetwork_Test
##[error]Process completed with exit code -1073741819.

I observed it for the first time after HTTP and Thread refactoring last week - made in PR #68 (custom worker thread). But it may or may not be related. Main challenge is that the bug might be blocking codereviews. We don't have a local repro - it might be specific to GitHub runner VM image.

MS Edge team relies on UTC transport for Released final product on Windows 10. There is currently no debug callback that allows to inspect each individual event contents prior to event being sent to UTC. The ask is to add a new debug callback type that allows to inspect individual event contents in a human-readable format, e.g. JSON view.

Note that this debug callback may show only preliminary event contents, i.e. event fields prior to event being further enriched with certain Part A/B props by system-level Universal Telemetry Client service.

Currently there is a hardcoded list of valid prefixes for User ID field.
ms: (internal Microsoft user) is not one of the supported ID types.