Looking at #8 we need to have 4.5 tests for Core as well

Problem description

Given automated way of collecting Request and Dependency events in App Insights customers have little to no control over the amount of information their application would produce. By default we collect all requests and dependency calls and there is no way to reduce event rates if application in question is fairly large.

With static sampling feature shipped in [2.0 beta] Sdk in sprint 90 filtering feature shipped in early 91 we do provide customers a way to control volumes but questions remain.

If customer attempts to use sampling the very first problem they will run into is which value they would need to set as sampling percentage. This can be determined by looking at total volume of events for a fairly large period of time, but presents a different problem. Say, application is bursty and produces a lot of volume in a short period of time followed by a long “quiet” period. With static sampling burst period would be sampled down and would produce statistically correct data but “quiet” period may have so little telemetry events generated that static sampling will capture a few events per day/hour and make data incorrect (skewed).

Solution

Proposed solution is to develop “adaptive sampling” mechanism that would very the sampling percentage based on the observed rate of telemetry events generated by the application. As the rate increases (burst situation), sampling percentage will decrease capturing less events. When rate of telemetry events drops (“quiet period”), sampling percentage increases back capturing more events to preserve statistical data correctness.

Thus, sampling percentage may “float” based on the rate of telemetry events produced by the application. Since we’re addressing this on Sdk side, the rate of events is actually rate of events on that box/device, no across entire application.
This solution will work for .Net server-side Sdk. We can potentially port it to other server-side sdk ports later).

Adaptive sampling module can be set up in code or in configuration file. The default configuration file will include adaptive sampling module be default.

Design details

The solution uses existing concept of “telemetry processor” found in 2.0 beta Sdk of AI.

We will employ existing sampling module to do actual sampling and create another telemetry processor to do the math to figure out what sampling percentage is to be applied in a given situation.

To do that, we’ll calculate exponential moving average (see: https://en.wikipedia.org/wiki/Moving_average) of the telemetry items sent to AI data collector (rate of events after sampling). This number will be available all the time the application runs. If application has just started, the calculation of event rate will be reset. As the application continues to run, moving average will more precisely reflect rate of telemetry events.

The process will keep its state in memory without any serialization initially. So, restart of the application will reset state to the initial one.

Having average rate of events produced and effective sampling rate (current sampling rate set on the sampling telemetry processor) we can determine the ‘ideal’ sampling rate given target event rate set as configuration value. If ‘ideal’ sampling percentage is different from the currently effective one, we’ll change corresponding parameter of the sampling telemetry processor to new value.

Sampling percentage will not be changed constantly. Timeouts will be applied (different ones) before sampling percentage will be [further] decreased or increased. Changing sampling rate very frequently may result in bad behavior where request/rdd/events may not be together sampled in or out if sampling percentage changes in between, therefore certain timeout is needed.

Process parameters

The table below contains entire set of parameters used in the process. These will be settable in code or in configuration file. All the parameters may be set from code or via configuration file. Default configuration file will have only MaxTelemetryItemsPerSecond parameter set explicitly to default value.

In addition to all parameters, a callback can be set in code invoked every time sampling percentage algorithm is run (in case customer wants to track/trace sampling percentage change events).

• InitialSamplingPercentage (default: 100%) - Sampling percentage to apply when the application code starts and no state of the estimation process is available.
• MaxTelemetryItemsPerSecond (default: 5) - Target maximum number of telemetry items generated by a single box/device per second. This parameter is the main driving factor of sampling percentage changes. Generally speaking, if this parameter is set to 5 and we observe 10 telemetry events generated per second, we’ll set sampling percentage to 50%.
• MinSamplingPercentage (default: 0.1%) - As sampling percentage varies, what is the minimum value we’re allowed to set.
• MaxSamplingPercentage (default: 100%) - As sampling percentage varies, what is the maximum value we’re allowed to set.
• EvaluationIntervalSeconds (default: 15sec) - How frequently do we run sampling percentage evaluation algorithm (along with moving average algorithm).
• MovingAverageRatio (default: 0.25) - When calculating moving average of telemetry events submitted per second, how much “emphasis” to put on the most recent values vs. historical values.
  With default value we put 25% ‘emphasis” on the most recent value and 75% on historical values.
• SamplingPercentageDecreaseTimeoutSeconds (default: 2min) - When sampling percentage value changes, how soon after are we allowed to lower sampling percentage again to capture less data.
• SamplingPercentageIncreaseTimeoutSeconds (default: 15min) - When sampling percentage value changes, how soon after are we allowed to increase sampling percentage again to capture more data.

Sdk Api design

Similar to static sampling we will provide simple building block to enable customers to quickly setup adaptive sampling in code. TelemetryChannelBuilder class allowing to build a list of telemetry processors will receive new extension UseAdaptiveSampling() with the following overloads:

1. No parameters. Enables adaptive sampling with all default values for algorithm parameters. This is to be used by new customers primarily to “kick the tires”;
2. maxTelemetryItemsPerSecond parameter. Enables adaptive sampling with all default parameters of the algorithm but custom target telemetry item rate. We expect this one to be used by more advanced customers who have either fewer boxes/servers in the application and willing to capture more telemetry per box or the other way around.
3. settings, callback parameters. An overload for full customization of the algorithm allowing to set all parameters (here “settings” is a set of settable properties corresponding to all parameters outlined above for the estimation algorithm). Callback parameter allows customer to set up code that is invoked when sampling percentage evaluation algorithm runs. The following parameters will be provided to callback:
   • After-sampling rate of telemetry observed by the algorithm;
   • Current sampling percentage algorithm assumes is applied by the sampling telemetry processor;
   • New sampling percentage to set for sampling telemetry processor in order to make the rate of events “ideal”;
   • Whether or not sampling percentage will be changed after this evaluation (even though current & new sampling percentages may be different, new one may not be applied immediately due to ‘timeout’ or ‘penalty box’ situations, in other words, in cases sampling percentage was changed recently);
   • Algorithm current settings.

A separate telemetry processor AdaptiveSamplingTelemetryProcessor will also be provided. This one is used by the extensions and in itself is a combination of two telemetry processors – existing sampling processor and new sampling percentage estimator processor.

AdaptiveSamplingTelemetryProcessor also contains code to react to sampling percentage change recommendation performed by [internal] estimator processor by setting it as sampling percentage property of the sampling processor.

Sampling percentage evaluation algorithm

SamplingPercentageEstimatorTelemetryProcessor [internal] new class implements code for sampling percentage evaluation algorithm.
It sets up the timer to evaluate sampling percentage and follows this set of steps every time timer fires:

• Close next interval of the ‘moving average’ counter and get average observed after-sampling telemetry event rate;
• Calculate suggested sampling percentage so that rate would be “just under” ‘ideal’ target rate provided; adjust this suggested rate if it is below min or above max;
• Reset the timer if evaluation frequency parameter was changed;
• See if sampling percentage needs to be changed;
• Call evaluation callback if provided suppressing all exceptions (we’re on a timer thread in the process here and if that throws, the process would die);
• If sampling percentage can be changed (suggested is different from current and we’re not in any kind of ‘penalty box’), assume sampling percentage was changed by sampling telemetry processor (this is enforced by the container public telemetry processor), record current and date of change and also reset moving average counter since previous values were taken with different sampling percentage.

_From the customer issue report:_

We are trying to onboard to Application Insights for our asp.net application(.NET 4.0) which is running on Azure Cloud Services. We are running into issues while deploying the cspkg and\or while getting application start with application insights. It is happening because of version conflicts for System.Threading.Tasks which got added in our application because of AppInsights nuget package. This reference is needed for Microsoft.Bcl.Async and Microsoft.Bcl which are in turn used by AppInsights.

Stepwise Problem description with workflow, status, options tried and current status:

• Azure Web Role running ASP.NET application targetted for .NET 4.0 [Environment: OSFamily 2, .Net framework 4.0 with KB2468871 ]
  |
• Added with Application Insights + Application Insights TraceListener
  |
• Installed Nuget Package Application Insights for Web Applications (.NET 4.0)
  |
• which has dependency on
  |
• Microsoft.Bcl.Async.1.0.168 -> Uses Microsoft.Threading.Tasks.dll 1.0.168.0 -> which references-> System.Threading.Tasks, Version=1.5.11.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
  |
• which has dependency on
  |
• Microsoft.Bcl.1.1.8 -> Which adds -> System.Threading.Tasks.dll -> versions available in the package2.6.8.0 & 1.6.8.0
  |
• By default, 2.6.8.0 gets added reference to the project and gets copied to bin folder
  |
• There is assembly redirect present in web.config of web application (Web Role) for System.Threading.Tasks.dll to redirect anything from 0.0.0.0 to 2.6.8.0. But WaIISHost.exe tries to start webrole and hence it does not consider assembly redirect of web.config of our application.
  |
• WaIISHost.exe
  |
• RoleEntryPoint OnStart() -> if we try to set InstrumentationKey here(i.e. TelemetryConfiguration.Active.InstrumentationKey = RoleEnvironment.GetConfigurationSettingValue("Telemetry.AI.InstrumentationKey") runs into same exception) WebRole does not start because of Type: System.IO.FileLoadException for System.Threading.Tasks, Version=1.5.11.0. Check exception 1 attached. Same fusion log as below step
  |
• Moved that instrumenation key intialization code Global.asax.cs
  |
• Application_start() -> if we try to set here, we run into .NET runtime error for loading System.Threading.Tasks.dll 1.5.11.0 which is required by Microsoft.Threading.Tasks.dll 1.0.168.0. Check Exception 2 attached.
  |
• Whenever Microsoft.Threading.Tasks.dll is tried to load, then it tries to load System.Threading.Tasks.dll 1.5.11.0 and hence runs into System.IO.FileLoadException.
  |
• Fusion log suggests Major Version mismatch since System.Threading.Tasks.dll 2.6.8.0 is available in approot\bin but Microsoft.Threading.Tasks.dll needs -> System.Threading.Tasks.dll 1.5.11.0
  |
  Removing reference to 2.6.8.0 from solution and added 1.6.8.0 which was available in Microsoft.Bcl package but that results in Minor version mismatch and hence same FileLoadException.
  |
  Looked into Issue 9 of http://blogs.msdn.com/b/bclteam/p/asynctargetingpackkb.aspx which looked similar but it does not seem to help in this case(since I can't find System.Threading.Tasks.dll 1.5.11.0 anywhere in pacakges or in GAC)
  |
  Tried to search for 1.5.11.0 version of System.Threading.Tasks.dll in packages, nuget gallery, on internet but I could not find one.
  |
  What's the suggested approach? When Azure WebRole running WEb application targetting .NET 4.0 adds Application Insights, It looks like "Application Insights for Web Application" for .NET 4.0 together with its nuget dependencies like (Microsoft.Bcl.Async and Microsoft.Bcl) causes conflicts of versions for System.Threading.Tasks.dll 1.5.11.0
  |
  An option which I can think of is to write a startup task, which will copy these two different versioned dlls and Install them in GAC. When Startup tasks run, it will install both versions to GAC and Assembly Binder should find System.Threading.Tasks.dll 1.5.11.0 in GAC when application starts.
  |
  For above option, I need 1.5.11.0 version of System.Threading.Tasks.dll which can't be found.

Please suggest known solutions if there are any or suggestions to get over this issue.

Global.props
GlobalStaticVersion.props

Is the client side script tracking code compatible with CSP's?

I'm struggling to configure it with the NWebSec nuget package.

Refused to connect to 'http://dc.services.visualstudio.com/v2/track' because it violates the following Content Security Policy directive: "connect-src 'self'".

Specifically this one: TelemetryConfigurationExtensions

TelemetryClient checks iKey in its own Context and configuration. If it is empty in both places item is dropped. But telemetry item may have instrumentation key defined on the item level which should be checked as well.

Error reported:

error :   The name 'Value' in attribute 'DebuggerDisplay(\"Value={Value}; Name={Name}; Count={Count}; Success={Success}; Async={Async}; Timestamp={Timestamp}\")' on type 'Microsoft.ApplicationInsights.DataContracts.DependencyTelemetry'

See DependencyTelemetry.cs#L17

When upgrading using NuGet packages, the package should add bindingRedirect to app.config/web.config to prevent assembly load failures.

  <dependentAssembly>
    <assemblyIdentity name="Microsoft.ApplicationInsights" publicKeyToken="31bf3856ad364e35" culture="neutral" />
    <bindingRedirect oldVersion="0.0.0.0-1.2.0.5639" newVersion="1.2.0.5639" />
  </dependentAssembly>

hi all,
i notice that the instrumentationKey is empty in my asp.net 5 project (1.0.0-rc1-update1)

Steps to reproduce:

• create file with name config.json
• populate the file with appInisight informations:
  {
  "ApplicationInsights": {
  "InstrumentationKey": ""
  }
  }
• read and build configuration:
  public Startup(IHostingEnvironment env, IApplicationEnvironment appEnv)
  {
  Configuration = new ConfigurationBuilder()
  .SetBasePath(appEnv.ApplicationBasePath)
  .AddJsonFile("config.json")
  .Build();
  }
• insert the following line in method ConfigureServices in startup.cs:
  services.AddApplicationInsightsTelemetry(Configuration);
• insert the following lines in method Configure:
  app.UseApplicationInsightsRequestTelemetry();
  app.UseApplicationInsightsExceptionTelemetry();

now when i try to retrieve the TelemetryClient from myController(via Dependency Injection) i notice that the InstrumentationKey is Empty.

regards,

luca

We @ Enterprise Commerce in Universal Store trying to use App Insight Monitoring and Web Tests in place of current implementation of Key Note. One requirement for us is to be able to Accept Client Certificates and perform some Web Tests.

From what we tried/learned, we can't have App Insight take our Client Certificate and use it when calling Target Application.

Is this something we have currently OR something in pipeline?

Regards,
Shital Mehta
Enterprise Commerce

It seems that System.Threading.Tasks.dll assembly, contained in the Microsoft.Bcl.1.1.8 NuGet package that App Insights depends on, is version 2.6.8.0.

However, Micosoft.Threading.Tasks.dll, contained in the Microsoft.Bcl.Async.1.0.168 NuGet package that App Insights depends on, expects System.Threading.Tasks.dll to be version 1.5.11.0.

See the following assembly binder log.

*** Assembly Binder Log Entry (1/4/2016 @ 8:03:14 PM) ***

The operation failed.
Bind result: hr = 0x80131040. No description available.

Assembly manager loaded from: c:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\clr.dll
Running under executable C:\xtra\temp\vb\AppInsightsTest\debug\VirtualBench.exe
--- A detailed error log follows.

=== Pre-bind state information ===
LOG: DisplayName = System.Threading.Tasks, Version=1.5.11.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
(Fully-specified)
LOG: Appbase = file:///C:/xtra/temp/vb/AppInsightsTest/debug/
LOG: Initial PrivatePath = NULL
LOG: Dynamic Base = NULL
LOG: Cache Base = NULL
LOG: AppName = VirtualBench.exe

Calling assembly : Microsoft.Threading.Tasks, Version=1.0.12.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a.

LOG: This bind starts in default load context.
LOG: Using application configuration file: C:\xtra\temp\vb\AppInsightsTest\debug\VirtualBench.exe.Config
LOG: Using host configuration file:
LOG: Using machine configuration file from c:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\config\machine.config.
LOG: Safe mode is set (publisher policy disallowed).
LOG: Post-policy reference: System.Threading.Tasks, Version=1.5.11.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
LOG: GAC Lookup was unsuccessful.
LOG: Attempting download of new URL file:///C:/xtra/temp/vb/AppInsightsTest/debug/System.Threading.Tasks.DLL.
LOG: Assembly download was successful. Attempting setup of file: C:\xtra\temp\vb\AppInsightsTest\debug\System.Threading.Tasks.dll
LOG: Entering run-from-source setup phase.
LOG: Assembly Name is: System.Threading.Tasks, Version=2.6.8.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
WRN: Comparing the assembly name resulted in the mismatch: Major Version
ERR: The assembly reference did not match the assembly definition found.
ERR: Failed to complete setup of assembly (hr = 0x80131040). Probing terminated.

Data platform should be capable to accept property values up to 8 * 1024 symbols. However SDK sanitize values with the limit of 1024 symbols. See Property.cs#L67

When one of the types wasn't found - all other types will not be read by Application Insights SDK. See
https://github.com/Microsoft/ApplicationInsights-Home/issues/23 for details.

Configuration reading should be more resilient and extra tracing should be added.

Currently, telemetry is written to debug output before telemetry processors are executed, so it doesn't reflect items that were modified or filtered out by the processors. Debug output should be written after telemetry processors are executed and not print items that were filtered out. However, if there is no instrumentation key, debug output should still be written (as telemetry processors are not executed in that case).

Setting the user id is not resulting in the data being in the portal.
Microsoft.ApplicationInsights.Extensibility.TelemetryConfiguration.Active
.InstrumentationKey = "";
var tc = new TelemetryClient();
var request = new RequestTelemetry();
request.Context.User.Id = "1234";
request.Name = "Test Request";
tc.TrackRequest(request);

This issue was brought up here:
http://stackoverflow.com/questions/33527249/application-insights-setting-user-in-requesttelemetry-context-not-showing-up-in/33629580

StartOperaiton API doesn't support scenario when parent ID and root ID were received from different layer and we want to start operation with the existing parent and root ID.

Proposed change - allow StartOperaition to receive RequestTelemetry type as an argument

System.ArgumentNullException: Value cannot be null.
Parameter name: task
at AwaitExtensions.GetAwaiter(Task task)
at Microsoft.ApplicationInsights.Extensibility.Implementation.TaskTimer.<>c__DisplayClass1.<b__0>d__3.MoveNext()

Reported for versions: 1.0.0.4220 ; 1.1.0.1899 ; 1.2.0.5639

We are seeing a crash in our application with the following stack (we have seen ~3 occurrences of this):

System.AggregateException: A Task's exception(s) were not observed either by Waiting on the Task or accessing its Exception property. As a result, the unobserved exception was rethrown by the finalizer thread. 
---> System.Net.WebException: The request was aborted: The request was canceled.
   at Microsoft.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess(Task task)
   at Microsoft.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at Microsoft.ApplicationInsights.Channel.Transmission.<SendRequestAsync>d__24.MoveNext()
   --- End of inner exception stack trace ---
---> (Inner Exception #0) System.Net.WebException: The request was aborted: The request was canceled.
   at Microsoft.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at Microsoft.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccess(Task task)
   at Microsoft.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at Microsoft.ApplicationInsights.Channel.Transmission.<SendRequestAsync>d__24.MoveNext()<---

Because the exception occurs in a task and is unhandled, we get a TaskUnobserved exception, resulting in our app crashing. It would seem that some level of try/catch is needed to suppress the exception (probably in the Transmission class), since we have no way to reliably suppress the failure at our app's level.

Consider moving UseSampling into the same namespace as telemetryProcessorsChainBuilder to simplify discovery of this method

Hello All,

We, Enterprise Commerce @ Microsoft, build/deploy/maintain/support Transaction Processing systems. These systems are moving to IAAS/PAAS. One of the by design outcome of moving to IAAS is that many systems that doesn't require exposure to internet are kept under EBL IP ADDR, which are not exposing the systems to Internet. These servers will have provide capability to reach out to internet/azure platform.

Our systems are already equipped with Telemetry and Application Insight logging to Azure Platform. Current design allows us to have multiple set of IPs, ones with Internet exposure and ones without Internet exposure. So our App Insight logging works. As we move to IAAS, we lose those IPs with internet capability.

How do we implement Application Insight in these systems?

We've explored options like RELAY Servers with HTTP Proxies that takes the App Insight logging and port to Azure, however these are not feasible OR we don't even know how to make those to work. Feel free to contact me directly on internal skype to discuss the issue[s].

Regards,
Shital Mehta

Given that App Insights requires kb 2468871 when using .NET 4.0, we'd like to distribute an internet installer for kb 2468871 with our installer. This issue isn't with App Insights per say, but rather with our ability to deploy apps containing App Insights successfully. I've been looking online for answers to these questions:

1. Does an internet installer for kb 2468871 exist? I see installers at https://support.microsoft.com/en-us/kb/2468871 but we'd need an internet installer due to size limitations.
2. Is it okay to distribute said installers?

I apologize if this is not the correct forum for getting assistance with these issues.

As we announced here we've recently added a new concept: TelemetryProcessors. It can be used for filtering telemetry collected by ApplicationInsights SDK.

We collected a feedback and there are several modification that we would like to do.
Let me describe all the scenarios that we have and explain what we have now, what we would like to change and why.

All TelemetryProcessors in Configuration

Currently Web SDK by default adds 1 telemetry processor (AdaptiveSampling). You can add more custom telemetry processors to the ApplicationInsights configuration file. They will be automatically loaded as soon as you create TelemetryConfiguration.

Scenario is working as expected and there are no changes here.

All TelemetryProcessors in Code

You can remove AdaptiveSampling from configuration file and configure it in code. You can also add custom telemetry processors as described here.

Scenario is working as expected and there are no changes here.

Some TelemetryProcessors In Code and Some In Configuration File

Add Sampling and Custom processors after default processors from configuration file

In the next Web SDK version we would like to add several Telemetry Processors in the configuration file (e.g. filter requests out by UserAgent). At the same time if you want to provide AdaptiveSampling callback method you need to configure it in code. Right now when you configure sampling in code configuration file is ignored. That means that if you want to add AdaptiveSampling or some custom telemetry processor you also need to move all other telemetry processors that we already have in the configuration file to code.

Let's look at the example:

var builder = TelemetryConfiguration.Active.GetTelemetryProcessorChainBuilder();
builder.Use((next) => new AnotherProcessor(next));
builder.UseAdaptiveSampling();
builder.Build();

Right now in the code above

• TelemetryConfiguration.Active has list of processors defined in the configuration file
• GetTelemetryProcessorChainBuilder creates a new empty list of processors
• Use (and UseAdaptiveSampling) adds new processors to this empty list
• Build overrides processors defined in TelemetryConfiguration.Active with the list created by GetTelemetryProcessorChainBuilder

Proposal: GetTelemetryProcessorChainBuilder creates a new list from the processors defined in the configuration file. Use (as before) adds to this list.

Note: We will have default filters from the config executed first, than custom processors defined in code and sampling (if it is defined in code).

Add Custom processors before default processors from configuration file

Currently the scenario is not supported at all because as described above when you define telemetry processor in code configuration file is ignored. With implementation update described above custom telemetry processors are added after the ones that are defined in the configuration file.

If you want to add custom telemetry processors before default ones you would have 2 options

• Define all telemetry processors in code
• Define all telemetry processors in configuration file

Mixed mode will not be supported.

Update TelemetryProcessor properties in code

This scenario is added to address this issue.

The proposal is to expose readonly TelemetryProcessors collection.

public sealed class TelemetryProcessorChain: IDisposable
{
    public IReadOnlyCollection<ITelemetryProcessor> TelemetryProcessors { get; }

    public void Process(ITelemetry item);
} 

Time field of the envelope should be in UTC to be accepted by Vortex. It must be in the ISO 8601 UTC format, with a trailing ‘Z’ character. See https://en.wikipedia.org/wiki/ISO_8601#UTC and for an example see

Event date time. UTC time when the event was generated on the client. This should be in ISO 8601 format 2013-09-21T21:55:00.9839095Z

A TelemetryProcessor may be a fairly complex class using disposable objects (such as timers) and may need to be disposable. Add functionality (to TelemetryConfiguration.Dispose()) to check if any of the processors in the set configured are disposable and dispose of those.

DC wouldn't accept measurements when it is set to NaN.

...On second though maybe we need to make DC accept it as it's unclear how to tell that the metric is bad... It's better to send it than to throw exception or send zero instead.

Now item is sanitized after TelemetryInitializers but before TelemetryProcessors. But TelemetryProcessors may enrich the data so that it will exceed size limit or add some strings that need to be encoded.

Steps to reproduce:

1. New up one RequestTelemetry or EventTelemetry object "Ping"
2. New up two TelemetryClient objects with the different IKeys: IKey1 and IKey2
3. Use the same "ping" telemetry object in both clients to send "ping" telemetry periodically
4. First Client sends "Ping" telemetry item with IKey1
5. Second Client sends "Ping" telemetry with IKey1 as well, IKey2 is ignored because context was already initialized for "ping" telemetry item by the first client.
6. As a result "Ping" telemetry is only send for IKey1 and in a doubled amount...

Expected Behavior:
The correct IKey is used (all Lazy Initializers, including Context are run again), Client1 sends this telemetry item to IKey1, client2 to IKey2.

_or_

We prevent this coding pattern (either throw "Cannot use the same TelemetryItem twice" or ...)

This bug is reported by customer who tried to use the product in this way and could not understand why does it behave like that. This might be considered a "Doc bug" if we want to.

The code is simple:

            var configuration1 = TelemetryConfiguration.CreateDefault();
            configuration1.InstrumentationKey = "111";

            var configuration2 = TelemetryConfiguration.CreateDefault();
            configuration2.InstrumentationKey = "222";

            var client1 = new TelemetryClient(configuration1);
            var client2 = new TelemetryClient(configuration2);

            var tcEvent = new RequestTelemetry("Test", DateTime.Now, TimeSpan.FromSeconds(1), string.Empty, true);
            client1.TrackRequest(tcEvent);
            client2.TrackRequest(tcEvent);

or even without the TelemetryConfiguration, just with TelemetryClient:

            var client1 = new TelemetryClient(configuration1);
            client1.InstrumentationKey="111"
            var client2 = new TelemetryClient(configuration2);
            client2.InstrumentationKey="222"
            var tcEvent = new RequestTelemetry("Test", DateTime.Now, TimeSpan.FromSeconds(1), string.Empty, true);
            client1.TrackRequest(tcEvent);
            client2.TrackRequest(tcEvent);

I'm trying to integrate ApplicationInsights into my Win10 UWP app however on start up I'm seeing a slew of unmanaged exceptions.

These exceptions seems to impact app load time as well.

Exception thrown at 0x76F91048 in Yammer.App.exe: Microsoft C++ exception: _com_error at memory location 0x00CFF4AC.
Exception thrown at 0x76F91048 in Yammer.App.exe: Microsoft C++ exception: EETypeLoadException at memory location 0x0A15D864.
...

That last exception repeats like 100+ times.

Need to change Sanitize method for EventTelemetry so event with the name longer than 512 symbols wouldn't be rejected by endpoint

See aspnet/Universe#300, https://github.com/aspnet/dnx/pull/3037/files and https://github.com/davidfowl/aspnetvnextwebapiapp/blob/master/Generations.md

Internal telemetry collected by SDK must be InvariantCulture.
Examples:

TelemetryInitializer_Microsoft.ApplicationInsights.Web.AccountIdTelemetryInitializer_filed_to_initialize_telemetry_item_System.IndexOutOfRangeException:_Индекс_находился_вне_границ_массива

[msg=Log_Error];[msg=System.AggregateException:One_or_more_errors_occurred.--->_System.Exception:要求されたオブジェクトが存在しません。(HRESULT_からの例外:0x80010114)_____at_Windows.Storage.StorageFile.GetBasicPr

[msg=DependencyOperationTelemetryInitializerFailed];[msg=System.NullReferenceException:_La_référence_d'objet_n'est_pas_définie_à_une_instance_d'un_objet._____à_PSellerProfil_Web.UserIDTelemtry

In a nutshell:

I need a way of telling if the telemetry data I sent was successfully received.
Ideally having the API provide me this information, or alternatively a fair way of knowing if the telemetry server is up.

Details:

I have been using the C# API and am now additionally embedding the JS (angular) API.
The application’s backend is written in C# and the frontend in JS (management web console).

In some cases, the backend cannot reach the web whereas the frontend can.
By default, I want to send telemetry from the backend.
When not possible, I want to send from the frontend.
Also, I’d rather not send duplicates (from either the frontend and the backend, or from several frontends).

Ideally I’d have a way of telling if the attempts to send telemetry data were successful.
But it seems both the APIs are async and there’s no way to tell if the data was sent successfully after sending it.
Could you recommend a way to tell whether the telemetry data was in fact sent?

Another option is to check Internet connectivity, though this is less than ideal, or to check if the telemetry server is reachable.
Since the telemetry server’s url provides a restful API, I can’t ping it or send an HTTP GET request to it.
How would you suggest to check the telemetry server’s availability if the API doesn’t support this need?
Do you use any such mechanism?

cc: @abaranch, @briancr-ms, @BogdanBe, @vitalyf007

Stable version of Application Insights SDK doesn't support hierarchical correlation today. Correlation is implemented using two fields: RequestTelemetry.ID and ITelemetry.Context.Operaiton.Id. When these fields match - we show them as related in UI.

Cross-tier and hierarchical correlation requires more fields. Typical scenario for this type of correlation that we want to support is for every page view event (collected by JavaScript SDK) we show as "children" all ajax calls run on the page. For every ajax call we show corresponding server request collected by .NET server SDK. Every server request may have children operations like "business logic execution" or "rendering" which in turn may have calls to external dependencies as children activities. So we can troubleshoot page loading sloweness and show the root cause when sloweness was caused by database issue.

We propose the following terminology:

• Transaction Name - name of the first/topmost/root operation that initiated entire chain of execution.
• Transaction ID - correspondingly unique ID of operation described above.
• *Parent Operation ID" - unique id of the operation that was active at the moment when telemetry was reported
• Operation Name - name of the telemetry item if it is request telemetry or dependency telemetry
• Operation ID - unique id of the telemetry item if it is request telemetry or dependency telemetry

_Note:_ We propose to use term Transaction as oppose to Root Operation as we believe that transaction is easier to understand and explain.

Label for corresponding telemetry items on request details blade will be updated to this:
"All traces/exceptions/dependencies for this request transaction".

We can use the following http header names to propagate correlation information:

• x-ms-request-id for parent operation ID
• x-ms-transaction-id for transaction ID
• x-ms-transaction-name for transaction name

Going forward In order to support proper end-to-end cross-tier events correlation we want to start using concept called "correlation vector". The idea behind correlation vector is very similar to the new EventSource correlation and causation feature in FW 4.6. Here is Vance Morrison blog post explaining this feature.

Correlation vector is a string consist of base64-encoded GUID and a dot-delimited vector of integer values: HyCFaiQoBkyEp0L3.1.2

It support two method: Increment that will increment the last integer in a vector and Extend that will add .0 to the end of the vector.

Whenever new layer receives correlation vector it extends it and use it as the current operation identity. Every time new outgoing operation is started - correlation vector of current operation is incremented on one side and extended on other side of operation.

The main advantages of correlation vector comparing to traditional ParentOperaitonId, TransactionId approach are:

• Ability to build execution tree even if some events are missing
• Ordering of operation that doesn't rely on event time

So the new data field CorrelationVector will be included into Operation context.

Hovewer, there are still services using ParentOperationId, TransactionId concept for end-to-end tracking of operations. There are services receiving headers like x-ms-request-id representing operation ID that should be used as a parent operation of the service's new operation. Thus the proposal is to support fields ParentOperationId and TransactionId for these services.

The idea is that SDK developer will use ParentOperationId field for the requests called with this header. However if no operation id header was configured - CorrelationVector-based correlation will be preferred.

Error described here should be more informative:

Hello,
after updating Application-Insights.Web from v1.1 to 1.2 we get the following error several times at runtime in our Azure cloud Service (others too http://stackoverflow.com/questions/31993394/application-insights):

[NullReferenceException]: Object reference not set to an instance of an object.
at Microsoft.ApplicationInsights.TelemetryClient.Track(ITelemetry telemetry)
at Microsoft.ApplicationInsights.TelemetryClient.TrackException(ExceptionTelemetry telemetry)
at Microsoft.ApplicationInsights.TelemetryClient.TrackException(Exception exception, IDictionary 2 properties, IDictionary 2 metrics)
at System.Web.Http.ExceptionHandling.ExceptionLogger.LogAsync(ExceptionLoggerContext context, CancellationToken cancellationToken)
at System.Web.Http.ExceptionHandling.CompositeExceptionLogger.LogAsync(ExceptionLoggerContext context, CancellationToken cancellationToken)
at System.Web.Http.HttpServer.d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
at System.Web.Http.WebHost.HttpControllerHandler.d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.TaskAsyncHelper.EndTask(IAsyncResult ar)
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

How can we avoid this?

Remove dependency on https://github.com/Microsoft/ApplicationInsights-dotnet/tree/master/Test/External/DataPlatform

New algorithm for creating operationid produces 6 char strings (much shorter than it used to be with guids). This seems to screw up sampling algorithm (DJB hash) which seems to not work properly on short strings (bad distribution, 98+% of hashes have score <= 20, score should be equal to distribution)

This makes sampling work incorrectly.

Proposed solution is to fall back to original guids instead of 6 char ids or investigate a use of a different hash algorithm which works on short strings (NOTE - both .net and JSSDK should use the same algorithm as both can have sampling ON and should sample in/out the same users)

See Open-Source-Signing for details on Open Source Signing.

Here is .NET build tool repo task to implement it.

Today csc do not support OSS signing. Work item tracking it here Roslyn #1171.

AI (Internal): TelemetryInitializer Microsoft.ApplicationInsights.Web.AccountIdTelemetryInitializer filed to initialize telemetry item System.IndexOutOfRangeException: Index was outside the bounds of the array.
at Microsoft.ApplicationInsights.Web.AccountIdTelemetryInitializer.GetAuthUserContextFromUserCookie(HttpCookie authUserCookie, RequestTelemetry requestTelemetry)
at Microsoft.ApplicationInsights.Web.AccountIdTelemetryInitializer.OnInitializeTelemetry(HttpContext platformContext, RequestTelemetry requestTelemetry, ITelemetry telemetry)
at Microsoft.ApplicationInsights.Web.Implementation.WebTelemetryInitializerBase.Initialize(ITelemetry telemetry)

See https://github.com/Microsoft/ApplicationInsights-dotnet/blob/master/Test/Persistence.Test/Persistence.Net40.Tests/PersistenceChannel.Net40.Tests.csproj#L10. Needs to be fixed to 4.0

Sampling score today is calculated based on User ID followed by Operation ID if User ID is not available. See https://github.com/Microsoft/ApplicationInsights-dotnet/blob/9f4c715fa703753c2e6107afde79051447018a02/src/TelemetryChannels/ServerTelemetryChannel/Shared/Implementation/SamplingScoreGenerator.cs

The issue is that in case when the number of sessions is much bigger than number of users and moreover - set of users is never changed - statistics may be skewed as the same user will be sampled out all the time.

Thus the idea is to use Session ID as a first ID to calculate sampling score.

If non-existing TelemetryProcessors are configured in AI.config, the loading fails - however its no visible to end user that it failed. No events are sent if this happens.

We have too many test framework projects that needs to be merged into one. Ideally we also need to eliminate the dependency on telemetry channel from test framework if possible