Giter Club home page Giter Club logo

somethingdigital_influxdb's Introduction

SomethingDigital_InfluxDb

A set of tools that are useful when connecting Magento to InfluxDb

Features

  • "InfluxDB" section under the "Advanced" tab for configuring connection
  • SomethingDigital_InfluxDb_Model_Api class for communicating with InfluxDB
  • SomethingDigital_InfluxDb_Model_MeasurementInterface interface for creating measurements
  • Measurements in app/code/community/SomethingDigital/InfluxDb/Model/Measurement
  • SomethingDigital_Shell_Influxdb script for executing measurement groups
  • Send a customer response header (Sd-Influxdb-Route) with routing information

Sending Measurements

The Magento cron has a bad habit of getting stuck. In fact, it's one of the most important things that you should be monitoring! As such we don't rely on it delivery of measurements to InfluxDb. Instead, we add additional crontab entries to execute specific measurement groups via shell/sd_influxdb.php. For example...

crontab

* * * * * php /var/www/html/shell/sd_influxdb.php --group one_minute

local.xml

<config>
    <sd_influxdb>
        <groups>
            <one_minute>
                <inventory>sd_influxdb/measurement_inventory</inventory>
            </one_minute>
        </groups>
    </sd_influxdb>
</config>

The send method will be called on each model specified under the measurement group. In send you can get the data you need and send it to InfluxDB using SomethingDigital_InfluxDb_Model_Api. See the measurements in app/code/community/SomethingDigital/InfluxDb/Model/Measurement for reference.

You can add additional groups and add additional measurements to groups as needed.

Sending Routing info in custom response header

Routing information for a given request can be sent in a response header. A common use case for this would be to see response codes counts on a route-by-route basis. A new Apache CustomLog format can be used to save the response headers to the log files.

Turning on the custom header

local.xml

<config>
    <sd_influxdb>
        <route_response_headers>
            <enabled>1</enabled>
        </route_response_headers>
    </sd_influxdb>
</config>

In order to add a response headers to full page cache hits SomethingDigital_InfluxDb_Model_RequestProcessor also needs to be registered as a request processor and must go last.

enterprise.xml

<config>
    <global>
        <cache>
            <request_processors>
                <ee>Enterprise_PageCache_Model_Processor</ee>
                <zz_sd_influxdb>SomethingDigital_InfluxDb_Model_RequestProcessor</zz_sd_influxdb>
            </request_processors>
        </cache>
    </global>
</config>

Compatibility with custom request processors

If, for some reason, your site is not using Enterprise_PageCache_Model_Processor as the processor for FPC, you'll need to declare that processor as the metadata_source.

local.xml

<config>
    <sd_influxdb>
        <route_response_headers>
            <enabled>1</enabled>
            <metadata_source>Elastera_EnterprisePageCacheSSL_Model_Processor</metadata_source>
        </route_response_headers>
    </sd_influxdb>
</config>

somethingdigital_influxdb's People

Contributors

avocadoslab avatar mpchadwick avatar philwinkle avatar toddbc avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

somethingdigital_influxdb's Issues

Add support for writing to specific retention policies

We may want to vary the retention policy by measurement rather than just always writing to the DEFAULT RP. This could potentially be handled in the config. E.g.

<sd_influxdb>
    <groups>
        <one_minute>
            <inventory>
                <model>sd_influxdb/measurement_inventory</model>
                <retention_policy>ninety_days</retention_policy>
            </inventory>
        </one_minute>
    </groups>
</sd_influxdb>

Add the ability to set a global measurement prefix

I'm still working through the best schema, but it may be useful to have the ability to add a prefix to all measurements that get sent to InfluxDb. For example, we send measurements with names like "cron" and "changelog". In InfluxDb measurements may come from other source, though, such as telegraf. To reduce the risk of collision it may be useful to have the ability to prefix all measurements (e.g. magento_cron, magento_chagenlog, etc...)

Improve defense if InfluxDb goes down for FpcHitRate

What happens if InfluxDb goes down

  • In progress requests pile up and are never deleted
  • If this isn't addressed, redis store could fill up which could create performance issues for the application as every request will try to write to the redis store.

Solutions

  • Set an expiry when creating InProgress keys. We lose the data if InfluxDb goes down for an extended period of time, but we decrease possibility of jeopardizing application performance
  • Also, check for InProgress keys during send. Use timestamp added to in progress flag when sending to InfluxDb

Add a measurement for index changelog version(s)

Would be interesting to track the highest version Id in each changelog table as well as the current version in the metadata table. We could learn about whether lag is an issue, understand cache invalidations due to partial reindexes and, I'm sure, more...

Add some browser instrumentation

JS to post to a Magento route

  • route
  • URL
  • JS http request count
  • JS http request bytes downloaded
  • CSS http request count
  • CSS http request bytes downloaded
  • IMG http requests count
  • IMG http requests bytes donwloaded
  • Page timings

Magento route stores it in Redis where it is queued to be synced to InfluxDb by shell/sd_influxdb.php

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.