Giter Club home page Giter Club logo

applicationinsights-localforwarder's Introduction

Local Forwarder

Background

Local Forwarder is an agent that collects Application Insights or OpenCensus telemetry from a variety of SDKs and routes it to the Application Insights backend. It's capable of running under Windows and Linux. You may also be able to run it under macOS, but that is not officially supported at this time.

Running Local Forwarder

Local Forwarder is an open source project on GitHub. There is a variety of ways to run Local Forwarder across multiple platforms.

Windows

Windows Service

The most natural way of running Local Forwarder under Windows is by installing it as a Windows Service. The release comes with a Windows Service executable (WindowsServiceHost/Microsoft.LocalForwarder.WindowsServiceHost.exe) which can be easily registered with the OS by running a script similar to the following:

Register a service and configure it to start at system boot.

sc create "Local Forwarder" binpath= "WindowsServiceHost\Microsoft.LocalForwarder.WindowsServiceHost.exe" start= auto

Configure the service to restart automatically if it fails for any reason.

sc failure "Local Forwarder" reset= 432000 actions= restart/1000/restart/1000/restart/1000

Start the service immediately.

sc start "Local Forwarder"

Once the service is registered, use Windows tools to manage it.

Console application

For certain use cases it might be beneficial to run Local Forwarder as a console application. The release comes with the following executable versions of the console host:

  • a framework-dependent .NET Core binary /ConsoleHost/publish/Microsoft.LocalForwarder.ConsoleHost.dll. Running this binary requires a .NET Core runtime to be installed; refer to this download page for details.
E:\uncdrop\ConsoleHost\publish>dotnet Microsoft.LocalForwarder.ConsoleHost.dll
  • a self-contained .NET Core set of binaries for x86 and x64 platforms. These don't require .NET Core runtime to run. /ConsoleHost/win-x86/publish/Microsoft.LocalForwarder.ConsoleHost.exe, /ConsoleHost/win-x64/publish/Microsoft.LocalForwarder.ConsoleHost.exe.
E:\uncdrop\ConsoleHost\win-x86\publish>Microsoft.LocalForwarder.ConsoleHost.exe
E:\uncdrop\ConsoleHost\win-x64\publish>Microsoft.LocalForwarder.ConsoleHost.exe

Linux

Same as for Windows, the release comes with the following executable versions of the console host:

  • a framework-dependent .NET Core binary /ConsoleHost/publish/Microsoft.LocalForwarder.ConsoleHost.dll. Running this binary requires a .NET Core runtime to be installed; refer to this download page for details.
dotnet Microsoft.LocalForwarder.ConsoleHost.dll
  • a self-contained .NET Core set of binaries for linux-64. This one doesn't require .NET Core runtime to run. /ConsoleHost/linux-x64/publish/Microsoft.LocalForwarder.ConsoleHost.
user@machine:~/ConsoleHost/linux-x64/publish$ sudo chmod +x Microsoft.LocalForwarder.ConsoleHost
user@machine:~/ConsoleHost/linux-x64/publish$ ./Microsoft.LocalForwarder.ConsoleHost

Many Linux users will want to run Local Forwarder as a daemon. Linux systems come with a variety of solutions for service management, like Upstart, sysv, or systemd. Whatever your particular version is, you can use it to run Local Forwarder in a way which is most appropriate for your scenario.

As an example, let's create a daemon service using systemd. We'll use the framework-dependent version, but the same can be done for a self-contained one as well.

  • create the following service file named localforwarder.service and place it into /lib/systemd/system. This sample assumes your user name is SAMPLE_USER and you've copied Local Forwarder framework-dependent binaries (from /ConsoleHost/publish) to /home/SAMPLE_USER/LOCALFORWARDER_DIR.
# localforwarder.service
# Place this file into /lib/systemd/system/
# Use 'systemctl enable localforwarder' to start the service automatically on each boot
# Use 'systemctl start localforwarder' to start the service immediately

[Unit]
Description=Local Forwarder service
After=network.target
StartLimitIntervalSec=0

[Service]
Type=simple
Restart=always
RestartSec=1
User=SAMPLE_USER
WorkingDirectory=/home/SAMPLE_USER/LOCALFORWARDER_DIR
ExecStart=/usr/bin/env dotnet /home/SAMPLE_USER/LOCALFORWARDER_DIR/Microsoft.LocalForwarder.ConsoleHost.dll noninteractive

[Install]
WantedBy=multi-user.target
  • Run the following command to instruct systemd to start Local Forwarder on every boot
systemctl enable localforwarder
  • Run the following command to instruct systemd to start Local Forwarder immediately
systemctl start localforwarder
  • Monitor the service by inspecting *.log files in the /home/SAMPLE_USER/LOCALFORWARDER_DIR directory.

Mac

You may be able to run Local Forwarder under macOS, but that is not officially supported at this time.

Self-hosting

Local Forwarder is also distributed as a .NET Standard NuGet package, allowing you to host it inside your own .NET application.

//!!! TODO include details for downloading NuGet

using Library;
...
Host host = new Host();

// see section below on configuring Local Forwarder
string configuration = ...;
    
host.Run(config, TimeSpan.FromSeconds(5));
...
host.Stop();

Docker

Until we publish the official image, feel free to build one yourself. You may find an example in https://github.com/Microsoft/ApplicationInsights-LocalForwarder/blob/master/examples/opencensus.

Configuring Local Forwarder

  • When running one of Local Forwarder's own hosts (Console Host or Windows Service Host), you will find LocalForwarder.config placed next to the binary.
  • When self-hosting the Local Forwarder NuGet, the configuration of the same format must be provided in code (see section on self-hosting). For the configuration syntax, please see LocalForwarder.config in the GitHub repository. Note that configuration may change from release to release, so pay attention to which version you're using.

Monitoring Local Forwarder

Traces are written out to the file system next to the executable that runs Local Forwarder (look for *.log files). You can place a file with a name of NLog.config next to the executable to provide your own configuration in place of the default one. See documentation for the description of the format. If no configuration file is provided (which is the default), Local Forwarder will use the default configuration which can be found here.

Distributed tracing with OpenCensus

OpenCensus is a vendor-agnostic single distribution of libraries to provide metrics collection and tracing for your services. OpenCensus for Python and Go allows to export data to LocalForwarder (and support for Java is on the way). You may find more information on how to configure it on https://docs.microsoft.com/en-us/azure/application-insights/opencensus-python and https://docs.microsoft.com/en-us/azure/application-insights/opencensus-go.

Please check out the full example demonstrating OpenCensus instrumentation for golang and python web server applications in this repo: https://github.com/Microsoft/ApplicationInsights-LocalForwarder/blob/master/examples/opencensus

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct.

For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

applicationinsights-localforwarder's People

Contributors

csteegz avatar dnduffy avatar microsoftopensource avatar msftgits avatar tokaplan avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

applicationinsights-localforwarder's Issues

Azure App Service (Linux)

If I have an application that is running on app service, do I have any options for sending opencensus telemetry to appinsights? Is a forwarder available in that environment or can I send opencensus data directly to appinsights?

Sampling configuration

Is there a way to configure sampling as shown on [https://docs.microsoft.com/en-us/azure/application-insights/app-insights-sampling](this page) ?
My main requirement is to make sure all Exception telemetry is sent to AI.
Being able to tweak some adaptive sampling parameters would be nice.

Moreover, the FAQ on the same linked page suggests to use adaptive sampling to define a good value on fixed-rate sampling. This is what I would like to do since I have multiple services and I'd like to have the same fixed-rate sampling on each one (so that end-to-end transaction in azure always show the full operation across all services).
But local-forwarder configuration don't seem to allow fixed-rate sampling. Am I wrong ?

My services use Java SDK. Can I use fixed-rate sampling on the Java side, and disable (adaptive) sampling on the local-forwarder side to achieve fixed-rate sampling ?

Thanks in advance for any feedback :)

OpenCensusTelemetryConverter.cs -> Null pointer can occure at line 337 on annotation.Description.value

Hi,
we try to use the localforwarder for our traces, with the following flow:

  • Opentracing (Jaeger) -> OpenCensus Receiver (Jeager) -> Opencensus Exporter -> Azure Localforwarder -> App Insights

All works fine, I can see my traces in App Insights... but the dpendency call status is false?

In addition the localforwarder logg this error:


2019/07/22 08:40:34.534|ERROR|Could not track an incoming OpenCensus telemetry item. System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.LocalForwarder.Library.OpenCensusTelemetryConverter.TrackTraceFromTimeEvent(TelemetryClient telemetryClient, TimeEvent evnt, Span span, Node peerInfo, String ikey)
at Microsoft.LocalForwarder.Library.OpenCensusTelemetryConverter.TrackSpan(TelemetryClient telemetryClient, Span span, Node peerInfo, String ikey)
at Microsoft.LocalForwarder.Library.Library.OnOcBatchReceived(ExportTraceServiceRequest batch, ServerCallContext callContext)
2019/07/22 08:40:34.534|ERROR|Could not track an incoming OpenCensus telemetry item. System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.LocalForwarder.Library.OpenCensusTelemetryConverter.TrackTraceFromTimeEvent(TelemetryClient telemetryClient, TimeEvent evnt, Span span, Node peerInfo, String ikey)
at Microsoft.LocalForwarder.Library.OpenCensusTelemetryConverter.TrackSpan(TelemetryClient telemetryClient, Span span, Node peerInfo, String ikey)
at Microsoft.LocalForwarder.Library.Library.OnOcBatchReceived(ExportTraceServiceRequest batch, ServerCallContext callContext)
2019/07/22 08:40:34.534|ERROR|Could not track an incoming OpenCensus telemetry item. System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.LocalForwarder.Library.OpenCensusTelemetryConverter.TrackTraceFromTimeEvent(TelemetryClient telemetryClient, TimeEvent evnt, Span span, Node peerInfo, String ikey)


Can you help me, to understand whats going on?

Thank you
YR

Local Forwarder does not work correctly with Span requests

It never writes a response, therefore attempt to stream Spans without closing connection after each span does not work.

LocalForwarder should write to response stream after each span received similarly to how it does it for config requests.

examples/opencensus/python-app not work by google.protobuf.internal.well_known_types.ParseError

$ cd examples/opencensus
$ docker-compose up 
...
python-app_1       | Performing system checks...
python-app_1       |
python-app_1       | Unhandled exception in thread started by <function check_errors.<locals>.wrapper at 0x7f4d2bf08268>
python-app_1       | Traceback (most recent call last):
python-app_1       |   File "/usr/local/lib/python3.6/site-packages/django/utils/autoreload.py", line 228, in wrapper
python-app_1       |     fn(*args, **kwargs)
python-app_1       |   File "/usr/local/lib/python3.6/site-packages/django/core/management/commands/runserver.py", line 125, in inner_run
python-app_1       |     self.check(display_num_errors=True)
python-app_1       |   File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 359, in check
python-app_1       |     include_deployment_checks=include_deployment_checks,
python-app_1       |   File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 346, in _run_checks
python-app_1       |     return checks.run_checks(**kwargs)
python-app_1       |   File "/usr/local/lib/python3.6/site-packages/django/core/checks/registry.py", line 81, in run_checks
python-app_1       |     new_errors = check(app_configs=app_configs)
python-app_1       |   File "/usr/local/lib/python3.6/site-packages/django/core/checks/urls.py", line 16, in check_url_config
python-app_1       |     return check_resolver(resolver)
python-app_1       |   File "/usr/local/lib/python3.6/site-packages/django/core/checks/urls.py", line 26, in check_resolver
python-app_1       |     return check_method()
python-app_1       |   File "/usr/local/lib/python3.6/site-packages/django/urls/resolvers.py", line 254, in check
python-app_1       |     for pattern in self.url_patterns:
python-app_1       |   File "/usr/local/lib/python3.6/site-packages/django/utils/functional.py", line 35, in __get__
python-app_1       |     res = instance.__dict__[self.name] = self.func(instance)
python-app_1       |   File "/usr/local/lib/python3.6/site-packages/django/urls/resolvers.py", line 405, in url_patterns
python-app_1       |     patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
python-app_1       |   File "/usr/local/lib/python3.6/site-packages/django/utils/functional.py", line 35, in __get__
python-app_1       |     res = instance.__dict__[self.name] = self.func(instance)
python-app_1       |   File "/usr/local/lib/python3.6/site-packages/django/urls/resolvers.py", line 398, in urlconf_module
python-app_1       |     return import_module(self.urlconf_name)
python-app_1       |   File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
python-app_1       |     return _bootstrap._gcd_import(name[level:], package, level)
python-app_1       |   File "<frozen importlib._bootstrap>", line 994, in _gcd_import
python-app_1       |   File "<frozen importlib._bootstrap>", line 971, in _find_and_load
python-app_1       |   File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
python-app_1       |   File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
python-app_1       |   File "<frozen importlib._bootstrap_external>", line 678, in exec_module
python-app_1       |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
python-app_1       |   File "/code/app/urls.py", line 19, in <module>
python-app_1       |     import app.views
python-app_1       |   File "/code/app/views.py", line 5, in <module>
python-app_1       |     from opencensus.trace.exporters.ocagent import trace_exporter
python-app_1       |   File "/usr/local/lib/python3.6/site-packages/opencensus/trace/exporters/ocagent/trace_exporter.py", line 30, in <module>
python-app_1       |     from opencensus.trace.exporters.ocagent import utils
python-app_1       |   File "/usr/local/lib/python3.6/site-packages/opencensus/trace/exporters/ocagent/utils.py", line 17, in <module>
python-app_1       |     from google.protobuf.internal.well_known_types import ParseError
pip freeze|grep proto
googleapis-common-protos==1.6.0
protobuf==3.9.1

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.