Giter Club home page Giter Club logo

hyou's People

Contributors

adamchainz avatar ddoskind avatar endast avatar nya3jp avatar pacohh avatar sfontana avatar

Watchers

avatar avatar avatar avatar avatar avatar

hyou's Issues

Retry 100s user quota errors

Hello gents,

I noticed that some of our builds started to fail when they access the Sheets API with an "Read requests per user per 100 seconds" quota error.
To me it looks a bit like the error message for that limit changed, because we do have retrying built into hyou.
It might be just a matter of changing the logic here https://github.com/wbrp/hyou/blob/master/hyou/api.py#L74, but additional diagnosis might be required.

Thanks & Cheers,
Dorian

An example of the new error message:

<HttpError 429 when requesting https://sheets.googleapis.com/v4/spreadsheets/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/values/%27DE%27%21A1%3AZ1032?alt=json returned "Quota exceeded for quota group 'ReadGroup' and limit 'Read requests per user per 100 seconds' of service 'sheets.googleapis.com' for consumer 'project_number:xxxxxxxxxxxx'.">

Send numbers to the API instead of strings

Expected behaviour

Number values should be sent through the API as integers or floats, not as strings.

Current behaviour

We transform all values to strings before sending them to the API. This can cause some problems, e.g. if a cell is formatted as a percentage and you upload a float with hyou it will remove the percentage formatting from the cell.

Retry ConnectionResetError exceptions

From time to time I get a ConnectionResetError while getting the data.

We should retry these (we don't see any downside with that...yet).
Make sure to check whether ConnectionResetError is the same kind of exception in both py3 and py2.

Here's an example stacktrace:

Traceback (most recent call last):
  File "/Users/stefano/.virtualenvs/lara/lib/python3.6/site-packages/IPython/core/formatters.py", line 224, in catch_format_error
    r = method(self, *args, **kwargs)
  File "/Users/stefano/.virtualenvs/lara/lib/python3.6/site-packages/IPython/core/formatters.py", line 702, in __call__
    printer.pretty(obj)
  File "/Users/stefano/.virtualenvs/lara/lib/python3.6/site-packages/IPython/lib/pretty.py", line 402, in pretty
    return _repr_pprint(obj, self, cycle)
  File "/Users/stefano/.virtualenvs/lara/lib/python3.6/site-packages/IPython/lib/pretty.py", line 697, in _repr_pprint
    output = repr(obj)
  File "/Users/stefano/.virtualenvs/lara/lib/python3.6/site-packages/hyou/view.py", line 115, in __repr__
    return str('View(%r)') % (self._view_rows,)
  File "/Users/stefano/.virtualenvs/lara/lib/python3.6/site-packages/hyou/view.py", line 217, in __repr__
    return repr(list(self))
  File "/Users/stefano/.virtualenvs/lara/lib/python3.6/site-packages/hyou/view.py", line 212, in __iter__
    self._view._ensure_cells_fetched()
  File "/Users/stefano/.virtualenvs/lara/lib/python3.6/site-packages/hyou/api.py", line 45, in wrapper
    return _do_exp_backoff(partial_func, MAX_WAIT_TIME)
  File "/Users/stefano/.virtualenvs/lara/lib/python3.6/site-packages/hyou/api.py", line 99, in _do_exp_backoff
    return func()
  File "/Users/stefano/.virtualenvs/lara/lib/python3.6/site-packages/hyou/view.py", line 58, in _ensure_cells_fetched
    **self._fetch_params).execute()
  File "/Users/stefano/.virtualenvs/lara/lib/python3.6/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/Users/stefano/.virtualenvs/lara/lib/python3.6/site-packages/googleapiclient/http.py", line 837, in execute
    method=str(self.method), body=self.body, headers=self.headers)
  File "/Users/stefano/.virtualenvs/lara/lib/python3.6/site-packages/googleapiclient/http.py", line 162, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
  File "/Users/stefano/.virtualenvs/lara/lib/python3.6/site-packages/google_auth_httplib2.py", line 198, in request
    uri, method, body=body, headers=request_headers, **kwargs)
  File "/Users/stefano/.virtualenvs/lara/lib/python3.6/site-packages/httplib2/__init__.py", line 1514, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/Users/stefano/.virtualenvs/lara/lib/python3.6/site-packages/httplib2/__init__.py", line 1264, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/Users/stefano/.virtualenvs/lara/lib/python3.6/site-packages/httplib2/__init__.py", line 1217, in _conn_request
    response = conn.getresponse()
  File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
  File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
  File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
  File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 1009, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 871, in read
    return self._sslobj.read(len, buffer)
  File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 631, in read
    v = self._sslobj.read(len, buffer)
ConnectionResetError: [Errno 54] Connection reset by peer

Some HTTP 429 errors are not retried

From time to time Google adapts the message sent along with the HTTP 429 code.
This means that we need to quickly adjust hyou as soon as they do it.

Give that there is not daily usage limit

This version of the Google Sheets API has a limit of 500 requests per 100 seconds per project, and 100 requests per 100 seconds per user. Limits for reads and writes are tracked separately. There is no daily usage limit.

https://developers.google.com/sheets/api/limits

So since there is no daily quota it means that we can always retry 429 errors without checking the message.

Relax requirements

Since hyou is a library it shouldn't have pinned requirements otherwise conflicts arise when another library pins the same package but with a different version.

Upgrade to google-api-python-client v2

According to Google, the 2.0 release of google-api-python-client is a significant upgrade compared to v1, however, there don't seem to be too many changes required see here.

The main reason to update this is to avoid requirement conflicts in projects that use google-api-python-client v2.

Expected behaviour

hyou uses google-api-python-client v2

Current behaviour

hyou uses google-api-python-client v1

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.