microsoft / ambaribasedhadoopexporter Goto Github PK

Exporter based on Hadoop clusters that use Ambari as their administrative tool, leveraging Ambari API to export cluster's metrics.

License: MIT License

Dockerfile 0.47% C# 99.09% Mustache 0.45%

ambaribasedhadoopexporter's Introduction

Introduction

Apache Ambari is an open-source management tool that allows you to administer your Hadoop cluster easly. Ambari uses AMS (Ambari Metrics System) to monitor an collect logs from related Hadoop installtions in your cluster, exposing them to the user in each component dashboard. In addition, you may want to export those metrics to a cenralized tool you are using. Grafana has a built in support for that, which is great.
However, you may want to export AMS's collected metrics to a different source like Prometheus. This repo attepts to try and solve this issue, implementing a Prometheus exporter for Hadoop based on Ambari API.

Note - there are open source Prometheus exporters for some Hadoop components like YARN and HDFS, but, Ambari gives you a single point of access to multiple collected metrics for services, components and configuration. Thus our effort building an exporter based on its API.

Tested environment

We're running on Azure environment, using Azure HDInsight cluster 3.6 running HDP 2.6.
The exporter was deployed on our Kubernetes cluster running init-container that handles all secrets acquisition from an Azure Key Vault and injection to our containers.

Download from DockerHub

You can download an image from this DockerHub.

Build with Docker

Clone the project and go to its root folder.
Run docker build . -t ambari-exporter, this will create a docker image on your local machine, run docker images ambari-exporter to validate that the image exists.

Deploying to Kubernetes

Every service requires settings file to run, an example of full settings file can be found here. But, since we're using Kubernetes as our orchestrator, we can leverage helm to inject those values during deployment as environment variables.

Prerequisites

Install init-container in your cluster for Azure Key Vault integration. This is used to extract secrets in a secure way.

Deployment example

View an example deployment helm chart here.

Dependencies

All of the exporters are based on Ambari's API.

Implementation dependencies:

Prometheus-net - used to expose the API endpoint for Prometheus.
Serilog - for formatting log messages.
Newtonsoft - for Json parsing.
FluentAssertions, Moq 4, xUnit - for testing.
StyleCop - for code formatting.

Support

For any issue, request, question please feel to create an issue, we'll be trying giving you the best support as possible.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

ambaribasedhadoopexporter's People

Contributors

Stargazers

Watchers

Forkers

eveningcafe myselfpt bhaskers-blu-org2 taffywrinkle claudiusgonzo revathikotla hapylestat matt90luo saritng x-house test-mass-forker-org-1 muthhus hemanthabbina ssattari

ambaribasedhadoopexporter's Issues

SSL certs

Hi, my Ambari Server API uses SSL.

Normally I access it using TLS v1.2 and using an auth token but when I try to use this image I get an SSL negotiation error indicating that remote certificate does not pass validation.

This suggests that this image is trying to connect / auth using certificates.

Is there any way to force Token based auth and communication or a way to pass the correct certificates to the container?

Thanks in advance

Enhancement: Refactor stopwatch integration

Currently every exporter component has the same code structure of:

Creating a scoped section
Start timer
Run export action
End time and report

Suggestions: refactor this structure and prevent code duplication.

Secrets configuration

I'm trying to run container with next configuration:

{
  "LogLevel": "Debug",
  "Secrets": {
    "Path": "/app/test",
    "Mapping": {
      "ambariConnectionUsername": "AmbariConnection--Username", 
      "ambariConnectionPassword": "AmbariConnection--Password"
    }
  },
  "BaseExporterConfiguration": {
    "ClusterName": "cli-cluster",
    "AmbariServerUri": "http://10.0.0.5:8080/api/v1/clusters",
    "DefaultLabels": {
      "TestLabelName": "TestLabelValue"
    }
  },
  "PrometheusExporterConfiguration": {
    "Port": 8086
  },
  "LivenessConfiguration": {
    "SamplingPeriodInMinutes" : 1
  }
}

I'm using next command:

docker run -it --rm --network=host -v $PWD/SecretFolder:/app/test -v $PWD/appsettings.json:/app/appsettings.json ambari-exporter:latest

SecretFolder contains 2 files:

ambariConnectionPassword 
ambariConnectionUsername

I can successfully access ambari with command:

curl -u admin:password http://10.0.0.5:8080/api/v1/clusters/

But app is crashing with an error:

Fetching secrets from folder /app/test.
SecretConfigurationExtension::AddSecretProvider - resolved 2 secrets.
SecretConfigurationExtension::AddSecretProvider - mapped the following: 'ambariConnectionPassword' -> 'AmbariConnection:Password'.
SecretConfigurationExtension::AddSecretProvider - mapped the following: 'ambariConnectionUsername' -> 'AmbariConnection:Username'.
{"LogTimestamp":"2020-01-15T16:03:08.8934489Z","Level":"Debug","LevelId":1,"Message":"Hosting starting","CustomFields":{"EventId":{"Id":1},"SourceContext":"Microsoft.Extensions.Hosting.Internal.Host"}}
{"LogTimestamp":"2020-01-15T16:03:09.1784481Z","Level":"Information","LevelId":2,"Message":"Starting Prometheus MetricServer at 01/15/2020 16:03:09.","CustomFields":{"SourceContext":"App.Services.Hosted.PrometheusExporterHostedService"}}
{"LogTimestamp":"2020-01-15T16:03:09.2078564Z","Level":"Information","LevelId":2,"Message":"Starting the service - initializing and starting the timer.","CustomFields":{"SourceContext":"App.Services.Hosted.LivenessHostedService"}}
Application started. Press Ctrl+C to shut down.
Hosting environment: Production
Content root path: /app/
{"LogTimestamp":"2020-01-15T16:03:09.2100355Z","Level":"Debug","LevelId":1,"Message":"Hosting started","CustomFields":{"EventId":{"Id":2},"SourceContext":"Microsoft.Extensions.Hosting.Internal.Host"}}
{"LogTimestamp":"2020-01-15T16:03:09.2133152Z","Level":"Information","LevelId":2,"Message":"Running HealthCheck.","CustomFields":{"SourceContext":"App.Services.Hosted.LivenessHostedService"}}
{"LogTimestamp":"2020-01-15T16:03:09.3101379Z","Level":"Error","LevelId":4,"Message":"Failed to get response from Ambari: http://10.0.0.5:8080/api/v1/clusters.","Exception":{"Type":"System.Net.Http.HttpRequestException","Message":"Status code: Forbidden isnt Ok.","StackTrace":"   at Infrastructure.Providers.Concrete.AmbariApiContentProvider.GetResponseContentAsync(String uriEndpoint) in /app/src/Infrastructure/Providers/Concrete/AmbariApiContentProvider.cs:line 58"},"CustomFields":{"SourceContext":"Infrastructure.Providers.Concrete.AmbariApiContentProvider"}}
{"LogTimestamp":"2020-01-15T16:03:09.3340873Z","Level":"Error","LevelId":4,"Message":"HealthCheck failed.","Exception":{"Type":"System.Net.Http.HttpRequestException","Message":"Status code: Forbidden isnt Ok.","StackTrace":"   at Infrastructure.Providers.Concrete.AmbariApiContentProvider.GetResponseContentAsync(String uriEndpoint) in /app/src/Infrastructure/Providers/Concrete/AmbariApiContentProvider.cs:line 58\n   at App.Services.Hosted.LivenessHostedService.HealthCheck(Object state) in /app/src/App/Services/Hosted/LivenessHostedService.cs:line 73"},"CustomFields":{"SourceContext":"App.Services.Hosted.LivenessHostedService"}}

Unhandled Exception: System.Net.Http.HttpRequestException: Status code: Forbidden isnt Ok.
   at App.Services.Hosted.LivenessHostedService.HealthCheck(Object state) in /app/src/App/Services/Hosted/LivenessHostedService.cs:line 82
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

It looks like I misconfigured login and password

Exporters need to be refactored to avoid main logic duplication

all the exporters derived from IExporter implement the same basic pattern by code duplication.
This should be fixed - move the basic pattern to common code (base class) and implement the exporter specific logic only in each exporter (implementing protected virtual functions)

Configuration file

Hi,

I'm trying to use this in a non-Azure, non-Kubernetes environment but I can't find the provided configuration file example (404).

Could you please re-upload it?

Thanks in advance

Update helm charts

Servicemonitor doesn't belong in the same chart as the exporter.
Generalize the charts.

Enhancement: Adding fine-grained metrics for all exporters

Current: There are high-level service metrics that help traces errors in the exporter.
Enhancement: Add fine-grained metrics for all other component's exporters to help trace single components API availability.

App folder location

By default app is copied to root folder and caused an error:

Unhandled Exception: System.UnauthorizedAccessException: Access to the path '/proc/1/map_files' is denied. ---> System.IO.IOException: Operation not permitted

To fix this set WORKDIR for runtime container in Dockerfile:

FROM microsoft/dotnet:2.2-sdk AS build-env
WORKDIR /app

# -------------- Step 1. Copy only required files for nuget restore for better caching
# 1.1 Copy solution & nuget config file (if exists)
COPY *.sln NuGet*.Config ./
# 1.2 Copy csproj and create folders for projects
COPY ./src/*/*.csproj ./
# 1.3 Create folders for projects
RUN for file in $(ls *.csproj); do mkdir -p src/${file%.*} && mv $file src/${file%.*}/; done
# 1.4 Copy tests
COPY ./test/*/*.csproj ./
# 1.5 Create projects for tests
RUN for file in $(ls *.csproj); do mkdir -p test/${file%.*} && mv $file test/${file%.*}/; done
# 1.6 Restore
RUN dotnet restore

# -------------- Step 2. Copy all other files
COPY . .

# -------------- Step 3. Build all
RUN dotnet build -c Release --no-restore

# TODO Support flag test=false
# TODO Export test xml results  files
# -------------- Step 4. Run tests
RUN dotnet vstest test/*/bin/Release/*/*Tests.dll

#-------------- Step 5. Build runtime image
RUN dotnet publish src/App -c Release -o /app/out --no-restore

FROM microsoft/dotnet:2.2-runtime
WORKDIR /app
COPY --from=build-env /app/out .
ENTRYPOINT ["dotnet", "App.dll"]