Giter Club home page Giter Club logo

google-analytics-super-proxy's Introduction

Google Analytics superProxy

The Google Analytics superProxy allows you to publicly share your Google Analytics reporting data. Use it to power your own custom dashboards and widgets, transform responses to various formats, manage your quota efficiently, test, and more. It handles authentication, caching, and response-formatting for you.

It works with the following Google Analytics APIs: Core Reporting API, Real Time Reporting API, and Multi-Channel Funnels Reporting API

Quick Links

Feature Highlights

  • Public access to your Google Analytics data
  • Use the proxy to power your own custom dashboards
  • Convert to CSV, Data Table, TSV
  • Relative dates are supported (e.g. last 7 days)
  • Automatically refreshes report data
  • Caching - fast responses and efficient quota usage

The Google Analytics superProxy is a web application that runs in the Google App Engine python environment.

Setting up a local development environment

  1. If necessary, download and install Python 2.7
  2. Download and install the App Engine SDK for Python
  3. Create an APIs Console Project.
  • Go to the Services pane to activate the Analytics API service.
    • Go to the API Access pane and create an OAuth 2.0 Client. For Client ID settings select Web Application. For the hostname click more options and then add the following to the Authorized Redirect URIs field: http://localhost:8080/admin/auth. Note: you may use a different port, but you need to use that port consistently throughout your project. Click on Create client ID.
  1. Edit config.py in the Google Analytics superProxy src directory. Update OAUTH_CLIENT_ID, OAUTH_CLIENT_SECRET, and OAUTH_REDIRECT_URI with the corresponding values from the OAuth 2.0 Client you created in the previous step. The redirect URI does not need to include /admin/auth, this will be added for you.
  2. (Optional): The default the timezone for resolving relative dates is the Pacific timezone. The default setting for "anonymizing" Core Reporting API responses is set to False. Both of these options can be configured in src/controllers/util/co.py.
  3. Add the Google Analytics superProxy app to the Google App Engine Launcher (File->Add Existing Application) or start the app using dev_appserver.py. For additional details see The Development Environment. Make sure to serve the application using the same port as set in the list of Authorized Redirect URIs for your APIs Console Project, and in config.py.
  4. View the app by visiting http://localhost:8080/admin (replace 8080 with the correct port number).

Hosting the application on App Engine

  1. Setup the local development environment as described above.
  2. Register an application ID for your application using the App Engine Administration Console. This will give you a free hostname on appspot.com.
  3. Edit the app.yaml file in the src directory of the Google Analytics superProxy and set the first line to the application ID you registered in the previous step. E.g. application: your-application-id.
  4. Edit or create a new APIs Console Project and add the full URL of your application + /admin/auth as an Authorized Redirect URI for the OAuth 2.0 Client. If using the free appspot.com domain, the redirect URI will look something like https://your-applciation-id.appspot.com/admin/auth.
  5. Edit config.py in the Google Analytics superProxy src directory and update OAUTH_CLIENT_ID, OAUTH_CLIENT_SECRET, and OAUTH_REDIRECT_URI if necessary.
  6. Using the Google App Engine Launcher or appcfg.py deploy the application. For details see Uploading Your Application.
  7. View the app by visiting the /admin page of your application. E.g. https://your-application-id.appspot.com/admin.

Creating your first public query

  1. See instructions above to get up and running, either with a local dev environment or on App Engine.
  2. View the application by visiting the /admin page. E.g. https://your-application-id.appspot.com/admin.
  3. Follow the instructions to authenticate and authorize the application to access your Google Analytics account.
  4. Create a new query (a Core Reporting API query URI) and specify:
  5. Click Save & Schedule Query to save the query and start scheduling the query for automatic refresh. A new Public Request Endpoint (URL) will be created for this report.

Requests to the public endpoint URL will return the API response for the specific report created. Authorization will not be required to access the report data and it will automatically refresh.

Features

  • OAuth 2.0 for authentication and it's all handled for you server side
  • Multiple users - each with their own set of queries
  • Automatic scheduling to refresh data at a configurable time interval
  • Caching of responses (saves on quota and it's fast)
  • API Query stats (last request time and number public requests)
  • Transform responses to CSV, Data Table, or TSV.
  • Relative dates are supported for reporting queries (e.g. last 7 days).
  • Timezone for relative dates can be configured (North American timezones and UTC).
  • Auto-scheduling. Scheduling for an "abandoned" API query (i.e. hasn't been publicly requested for a long time) will be automatically paused, resuming only if is subsequently requested.
  • Responses can "anonymized". If enabled, then Google Analytics profile IDs and other account information is removed from the public response.
  • Error logging. Errors for scheduled API queries are logged. After an API Query (default) has 10 error responses, scheduling for the query is paused.
    • The Google Analytics superProxy will not publicly return error responses. Instead the last successful response will be returned.
  • Public endpoints can be enabled/disabled if you want to stop sharing.
  • Queries can be be refreshed on an adhoc basis instead of waiting for the next scheduled refresh.
  • JSONP (add a callback parameter to the Public Endpoint request URL).

Changelog

2013-07-19

  • Initial launch...super sweet!

2013-08-06

  • Multiple Google Charts on a single page works again! (Fixes Issue #3)
  • Added a sample Google Charts HTML/JS demo.

google-analytics-super-proxy's People

Contributors

drrataplan avatar pfrisella avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

google-analytics-super-proxy's Issues

SyntaxError: EOL while scanning string literal

'OAUTH_REDIRECT_URI' gives a syntax/EOF error on the ',' (line 42). Removing the string separation and just adding the redirect solves this issue for the moment.

'OAUTH_REDIRECT_URI': '%s%s' % (
        'https://REPLACE_THIS_WITH_YOUR_APPLICATION_NAME.appsot.com OR http://localhost:8080',
        '/admin/auth')
}

Obviously works just fine.

'OAUTH_REDIRECT_URI':  'https://REPLACE_THIS_WITH_YOUR_APPLICATION_NAME.appsot.com

Missing Something

Followed the instructions to the letter. But after setting up AppEngine Hosting, then trying to hit:

https://your-application-id.appspot.com/admin

get a 404 error. Hitting just:

https://your-application-id.appspot.com

does show 'Hello World' in the browser.

AppEngine Dashboard shows an error on the /admin URI at 100%. Seems like I've missed something or instructions are missing something.

Reviewed for a 4th time and realized that when using the AppEngine Launcher I used the 'Create New Application' instead of 'Add Existing Application' option. Then re-deployed to AppEngine and tried the admin URL again:

https://your-application-id.appspot.com/admin

Then got the correct 'authorization message' as expected, rather than the 404 error. So not an issue. But suggest the readme file under the AppEngine Launcher instructions also note the use of this option. I only picked this up when reading the instructions for the Local setup. CLOSED.

Dynamically loading queries into server

I am submitting this request after discussing this issue with Pete.

I have multiple users of My website, each has his own dashboard on my site.
I need each to view statistics from my GA account (events) which are attributed to him (each event has a label code which stands for the user code on my site).

My problem obviously is that queries need to be set manually in advance on superProxy right now (to the best of my knowledge), meaning that new users cannot be added automatically.

I need an API for me to upload a new query to superProxy from my server, to be available immediately upon upload.

Thank you so much

Bug with query data refresh scheduler

My query pulls data from 7 days ago up through yesterday. On the scheduler, I have the refresh rate set at 86400 seconds. My understanding is that the data will refresh once a day but if a user executes the query and there has not been a refresh, the system will automatically refresh the data before delivering the results to the user. By doing so, the elapsed time from the last refresh gets reset.

The user is executing the URL once a day to pull data. The data is refreshed sometimes and sometimes it is not. We are not getting consistent results each day during the URL execution to get data up through the previous day.

For example, today is 12/16. I should data up through 12/15. Instead, I am seeing data only up through 12/14.

But on 12/15, when I executed the URL, I see the data was refreshed up through 12/14.

I agree with other postings that an enhancement need to be made to allow users to better control the data fresh schedule. But in the meantime, can this bug be addressed?

Thank you.

Dynamic variables

Hi Pete,

We tested superproxy to make a public dashboard and this is really a promising tool !
Actually we would really like to use it for one of our clients.
But we need to insert a dynamic value into the request :

https://www.googleapis.com/analytics/v3/data/ga?ids=ga%3A25810321&dimensions=ga%3AyearMonth&metrics=ga%3Avisitors&segment=dynamic%3A%3Aga%3AcustomVarValue1%3D~DYNAMICVALUE&start-date=2013-01-01&end-date={today}&max-results=5

We need the final user to be able to choose a dynamic value to update this request.
Would it be possible in the future to predetermine some custom dynamic variables to make dynamic requests ?

Thanks a lot,
Benjamin

Query Scheduler Bug

I have set refresh interval to be 15seconds and hitting public endpoint in ever 20 seconds, but query scheduler is not dispatching the query. The only way i got into mannually click on refresh now button.

Excel - Data from Web

It would be great if this could support outputing to excel, maybe via pasting the URL into the data connections area,

I used to use a similar system with another Web Analytics provider, the tables were just structured with html.

s~appname : uncaught application failure

Hi, I configured and deployed the source code correctly and it works but sometimes I get this error. And during this time I can't change anything in the queries and the queries doesn't show any response data at the public endpoint. Can you help me to solve this issue?

Google Analytics Super Proxy Not Displaying Japanese Character Data

Hi,

I'm using Google Analytics Super-Proxy to make data from Google Analytics publicly available at our website. Currently I have a custom query that I'm creating using Google Query Explorer. This contains the top search keywords for a particular User Type, which contains a Japanese word as well. The Japanese term is getting displayed in the Query Explorer website. But when I use the same custom query and run it on GA super proxy site, I get no data in the JSON Response. Any ideas why ?

Regards,
Alok

sP stripping out "+" from regex filter?

Hi, I am using this working regex filter in the Query Explorer:

ga:pagePath=~-AB(a|b|cd)?[0-9]+(-CD[0-9]+|.htm)

The copy & paste URL that QE creates looks like this:

https://www.googleapis.com/analytics/v3/data/ga?ids=ga%3[[ID_HERE]]
&metrics=ga%3Apageviews
&filters=ga%3ApagePath%3D~-AB(a%7Cb%7Ccd)%3F%5B0-9%5D%2B(-CD%5B0-9%5D%2B%7C%5C.xyz)
&start-date=2014-04-15
&end-date=2014-04-29

In the above URL it looks like the "+"s are still in there as "%2B"s

The superProxy response for the filter looks like this though; it's missing the "+"s (spaces are there instead):

{...u'filters': u'ga:pagePath=~-AB(a|b|cd)?[0-9](-CD[0-9] |.xyz)',...}

Whether this is the issue or not, sP is incorrectly returning 0 pageviews for this query

Is there something I need to do to the URL to make sure they don't get stripped?

Is this a bug?

Thanks :)

{today} - allow setting timezone in the app

I am in GMT+11, which means {today} is usually yesterday.

It would be good if we could set the timezone in the admin screen, or retrieve the timezone from the GA web property.

How to pass start_date and end_date to superProxy?

I setup the superProxy and everything works fine. However, I want to get the reporting data according to dates. I know that the Google Analytics superProxy does not provide that feature. How can I change superProxy's source code to accomplish that? It is written in Python and I don't have any serious experience that programming language. Could you help me with that please?

Visitors' Locations

Hi -

Not sure if this is possible, but I'm trying to show a map of visitor locations. So I have a page on my site that will have a map which pulls google analytics visitor / location data.

Here's what I have so far -

http://honeymanghost.com/?page=classy_map.php

The locations are incorrect though - as most of my site visitors are in America. I think I built the query wrong - I chose visitors as the metric, and lat / lng as dimensions.

Any help would be greatly appreciated !

How to configure super proxy to request authorization for both the core & real-time apis?

Thanks so much for your great work. I followed your video tutorial and things went swimmingly. :) Pie-chart goodness working. Awesome!

Along with data from the Core Reporting API, I want to pull data from the Real Time Reporting API. I attempted to set up a real time query through the super proxy admin and received an error:

GET
https://www.googleapis.com/analytics/v3/data/realtime?ids=ga%3A{MY_PROFILE_ID}&metrics=ga%3AactiveVisitors&max-results=1

RETURNED
{u'error': {u'code': 403, u'message': u'Your project does not have access to this feature.', u'errors': [{u'domain': u'global', u'message': u'Your project does not have access to this feature.', u'reason': u'insufficientPermissions'}]}}

I've read through the page on authorization for real time api requests at https://developers.google.com/analytics/devguides/reporting/realtime/v3/authorization
and it says "You will get a 403 status code if the authorized user does not have access to the view (profile). Make sure you are authorized with the correct user and that they indeed have the view (profile) you have selected."
I am authorizing with the correct user and I'm able to access the real time data through the main Google Analytics UI and through the test page at https://developers.google.com/analytics/devguides/reporting/realtime/v3/reference/data/realtime/get using the same user.

In your analytics_auth_helper.py, you set the OAUTH scope to https://www.googleapis.com/auth/analytics.readonly
As far as I can tell, that works for the real time API too.

The only other thing that I've spotted is that the real time authorization documentation mentions "Activate the Real Time Reporting API in the Services pane of the Google APIs Console. (If it isn't listed in the Console, then skip this step.)" It wasn't there.

I'm stumped. Any ideas?

Thanks!

Changing default labels of analytics querys

Hi, is there any way to change the default labels and formats of the querys to display in charts? ex.: I´m creating a chart to show visits and new visits by date, and i want to change "ga:visits" to just "Visits" and "ga:newVisits" to just "New Visitors".
Also i would like to insert a hype to the date format so it looks "2014-03-15".
Is there any way to achive this?
Example image:
chart

Service Down?

This has been working for several months flawlessly, but this evening I got this error:
Error: Server Error
The server encountered an error and could not complete your request.

Please try again in 30 seconds.

Most recently I'm not getting that error, just a blank page with a title of :

uncaught application failure

Any thoughts?

Unable to get JSON query with jQuery

I am getting an error that says:
No 'Access-Control-Allow-Origin' header is present on the requested resource

I have been looking for the file I need to edit to make it so I can pull the Query data using jQuery, but I can't find it.

I am using this to populate a dashboard that is client friendly, and without this feature, I can't use the SuperProxy. Please let me know what file I need to update, or push an update allowing us to grab the file using jQuery!

Super Proxy support for GA4?

Is someone planning to add support for Google Analytics 4?
We are using Super Proxy to implement a "most read" widget for our website, but I am afraid that sooner or later it will stop working if it is no longer actively maintained.
Is there some alternartive?
Thanks everyone, ciao,
Cesare

Ability to email if scheduled tasks fail

Hi,
Is there a way for superproxy to email out if a test is paused due to increased errors from the Analytics API? I would rather not have to set up something like Pingdom to watch all my tests :/

Thanks,

Analytics Super proxy scheduler not working

The scheduler works the first time I create the queries but if I ever stop the appengine hosting the superproxy or restart the host computer, the schedule never runs at its scheduled interval again. The only way I can get it to run appropriately is to delete the query add it back and update all of my applciation to the new link since it generates a new link each time

Query Explorer Not Functional

Hi there. The query explorer that you link to seems to be having issues with authenticating and allowing me to build queries through it. I know it isn't part of this codebase but i'm having trouble figuring out who has ownership of it so I can alert them of the problem.

URL in question: http://ga-dev-tools.appspot.com/explorer/

Thank you for everything!

Google Analytics Super Proxy Crash While Creating Query

Hi,

I created around 10-15 queries today using google analytics super proxy, while creating the 16th query, the query creation page went blank. I'm not able to create any more queries, neither am I able to view the previously created queries.

Any idea as to why this might be happening ? Any way to rectify this error ?

Regards,
Alok Nath Saha

Feature Request

Please provide a simple 'delete user' mechanism. Easy to add users - less easy to delete currently.

Thanks

Doug

/admin/auth 404 not found

I want to setup my own proxy but I can't get past square 1.

When I run locally I get 404 not found for http://localhost:8080/admin/auth. Same for /admin. I deployed via the app engine with the same result.

The app is running, http://localhost:8080 results in "Hello world!". I can see this is the default in main.py... but I don't know how to troubleshoot request routing in a Python app.

Cheers,
Morgan.

Typo in config.py: appsot

In the config.py, the example value for for the OAUTH_REDIRECT_URI has a typo in it, the value is appsot instead of appspot.

Could someone fix it, or should I create a fork + pull request?

Hebrew chars output and DataTable format

Hello there,
no respond when hebrew chars are involved. see images below
thanks,

**JSON format respond:
image

**DataTable (both formats) respond is white screen, this is what source code looks like:
image

Problem with query. Dates are returned as strings.

I built a simple query using the Explorer to bring back the number of visits for each day during a 60 day period which is working and have the URL copied to the superProxy in the apps engine. When I create a web page using the URL and display the results using Google Charts the labels along the x-axis are in the yyyymmdd string format. When I examine the data returned using the Manage Query page the data is listed as STRING no matter what format I click on (DataTable JSON Response, JSON String or just JSON) I have tried placing option=dataTable in the query string but that just seems to break the query. Is there some setting I am missing or is this a bug?

Super proxy API request doesnt recognize russian character Data

Hello, thanks for the superb application!!

Im using GA reports with data which have russian words.
Detect problem while add filter with russian words. Get API Request with encoding russian words, for eg.

use filter like: rt:eventLabel=~телефоны 
in request its look like:
filters=rt%3AeventLabel%3D~%D1%82%D0%B5%D0%BB%D0%B5%D1%84%D0%BE%D0%BD
and as result superproxy return error:
{'error': 'No JSON object could be decoded'}
but if use in english: rt:eventLabel=~phone
in request: filters=rt%3AeventLabel%3D~phone
all works well.. dont no how to go arround this

Please, make some updates to fix this... its very pitty that cant use this tool in work :(

Assigning a time to the schedule (enhancement)

Hi,

Requesting that an option be added to assign when the request should start/happen in addition to the time interval between requests field

This could just be a drop down list of hours of the day for example, even on the hour would be fine

Issue arises because:

  1. Having something run during the work day / normal waking hours often (perhaps usually) isn't ideal

E.g., one would need to wake up at 4:00am in order to schedule something that takes place daily at 4:00am (which of course is 7:00am EST if one lives on the west coast)

  1. Sometimes one may want to check things out immediately after a change, but Refreshing would mean resetting the schedule (I think); having a set, scheduled Refresh time that persists even after you manually click Refresh would be advantageous for this case

Thank You :)

Also mentioned on Google SP forum here: https://groups.google.com/forum/#!topic/google-analytics-super-proxy/zKx_VeXKBfA

Internal Server Error

Been using SuperProxy for several years.

As of Friday, August 13th, 2021, continually receive an Internal 500 Error message error either 1) accessing endpoints or 2) accessing the site directly via a browser.

The issue started between 10:33am Eastern (my last successful access and report download) and 3:23PM Eastern when I first noticed the Internal 500 Error. As of today - Monday, August 16, 2021 at 12:30PM, still receiving 500 errors.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.