Giter Club home page Giter Club logo

data-api's People

Contributors

dependabot[bot] avatar gassc avatar

Stargazers

avatar avatar

Watchers

avatar avatar

Forkers

holocene-systems

data-api's Issues

failing requests from app due to Redis memory issue

Need to investigate and out of memory issue with Redis that is causing rainfall retrieval requests to fail. This only affects the aync ReST API endpoint used by the Rainfall web app.

Sample logs:

2023-05-19T12:48:45.122985+00:00 app[worker.1]: Traceback (most recent call last):
2023-05-19T12:48:45.122999+00:00 app[worker.1]: File "/app/manage.py", line 21, in <module>
2023-05-19T12:48:45.123083+00:00 app[worker.1]: main()
2023-05-19T12:48:45.123085+00:00 app[worker.1]: File "/app/manage.py", line 17, in main
2023-05-19T12:48:45.123128+00:00 app[worker.1]: execute_from_command_line(sys.argv)
2023-05-19T12:48:45.123137+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.10/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line
2023-05-19T12:48:45.123217+00:00 app[worker.1]: utility.execute()
2023-05-19T12:48:45.123218+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.10/site-packages/django/core/management/__init__.py", line 440, in execute
2023-05-19T12:48:45.123285+00:00 app[worker.1]: self.fetch_command(subcommand).run_from_argv(self.argv)
2023-05-19T12:48:45.123286+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.10/site-packages/django/core/management/base.py", line 402, in run_from_argv
2023-05-19T12:48:45.123353+00:00 app[worker.1]: self.execute(*args, **cmd_options)
2023-05-19T12:48:45.123354+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.10/site-packages/django/core/management/base.py", line 448, in execute
2023-05-19T12:48:45.123417+00:00 app[worker.1]: output = self.handle(*args, **options)
2023-05-19T12:48:45.123418+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.10/site-packages/django_rq/management/commands/rqworker.py", line 128, in handle
2023-05-19T12:48:45.123457+00:00 app[worker.1]: w.work(
2023-05-19T12:48:45.123458+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.10/site-packages/rq/worker.py", line 571, in work
2023-05-19T12:48:45.123541+00:00 app[worker.1]: self.set_state(WorkerStatus.STARTED)
2023-05-19T12:48:45.123542+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.10/site-packages/rq/worker.py", line 374, in set_state
2023-05-19T12:48:45.123618+00:00 app[worker.1]: connection.hset(self.key, 'state', state)
2023-05-19T12:48:45.123627+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.10/site-packages/redis/commands/core.py", line 4859, in hset
2023-05-19T12:48:45.124379+00:00 app[worker.1]: return self.execute_command("HSET", name, *items)
2023-05-19T12:48:45.124379+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.10/site-packages/redis/client.py", line 1258, in execute_command
2023-05-19T12:48:45.124653+00:00 app[worker.1]: return conn.retry.call_with_retry(
2023-05-19T12:48:45.124661+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.10/site-packages/redis/retry.py", line 46, in call_with_retry
2023-05-19T12:48:45.124727+00:00 app[worker.1]: return do()
2023-05-19T12:48:45.124728+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.10/site-packages/redis/client.py", line 1259, in <lambda>
2023-05-19T12:48:45.124935+00:00 app[worker.1]: lambda: self._send_command_parse_response(
2023-05-19T12:48:45.124945+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.10/site-packages/redis/client.py", line 1235, in _send_command_parse_response
2023-05-19T12:48:45.125157+00:00 app[worker.1]: return self.parse_response(conn, command_name, **options)
2023-05-19T12:48:45.125157+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.10/site-packages/redis/client.py", line 1275, in parse_response
2023-05-19T12:48:45.125363+00:00 app[worker.1]: response = connection.read_response()
2023-05-19T12:48:45.125372+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.10/site-packages/redis/connection.py", line 827, in read_response
2023-05-19T12:48:45.125499+00:00 app[worker.1]: raise response
2023-05-19T12:48:45.125523+00:00 app[worker.1]: redis.exceptions.ResponseError: OOM command not allowed when used memory > 'maxmemory'.
2023-05-19T12:48:45.320864+00:00 heroku[worker.1]: source=worker.1 dyno=heroku.175537191.280be8e0-5e3b-4772-967e-eb52de41404b sample#memory_total=101.65MB sample#memory_rss=97.64MB sample#memory_cache=4.01MB sample#memory_swap=0.00MB sample#memory_pgpgin=27207pages sample#memory_pgpgout=1696pages sample#memory_quota=1024.00MB
2023-05-19T12:48:45.457728+00:00 heroku[worker.1]: Process exited with status 1
2023-05-19T12:48:45.486736+00:00 heroku[worker.1]: State changed from up to crashed

3RWW SSO Middleware

Need middleware and a REST API endpoint to handle authenticating with 3RWW's existing legacy SSO system.

Initially this will be used by the Sewer Atlas v3.

  • create a service that is referrer limited that returns a good token
  • tie into SSO

Memory leak after large queries

Failed queries like this appear to be failing because they are exceeding the memory available in the Heroku Dyno web process (not the workers). After failing, they seem to be leaving behind a whatever they had in memory, in memory.

May be able to address some cases with complete switch from PETL to Pandas. Also switching away from the DRF serializer should help. Off-loading some of the transforms to the database is worth exploring here too.

failing requests from the app due Redis SSL issue

All requests to the async endpoint from the web app are failing due to a Redis connection issue.

This may be a result of the Heroku-imposed upgrade from Redis 5.x to 7 on June 16 today.

Logs:

2023-06-27T00:04:22.000000+00:00 app[redis-sinuous-87740]: Error accepting a client connection: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
2023-06-27T00:04:22.080126+00:00 app[web.1]: 2023-06-26 20:04:22 [10] [ERROR] pathname=/app/.heroku/python/lib/python3.10/site-packages/django/utils/log.py lineno=241 funcname=log_response Internal Server Error: /admin/django_rq/queue/
2023-06-27T00:04:22.080148+00:00 app[web.1]: Traceback (most recent call last):
2023-06-27T00:04:22.080148+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/redis/connection.py", line 598, in connect
2023-06-27T00:04:22.080148+00:00 app[web.1]: sock = self.retry.call_with_retry(
2023-06-27T00:04:22.080149+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/redis/retry.py", line 46, in call_with_retry
2023-06-27T00:04:22.080149+00:00 app[web.1]: return do()
2023-06-27T00:04:22.080149+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/redis/connection.py", line 599, in <lambda>
2023-06-27T00:04:22.080150+00:00 app[web.1]: lambda: self._connect(), lambda error: self.disconnect(error)
2023-06-27T00:04:22.080150+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/site-packages/redis/connection.py", line 998, in _connect
2023-06-27T00:04:22.080150+00:00 app[web.1]: sslsock = context.wrap_socket(sock, server_hostname=self.host)
2023-06-27T00:04:22.080150+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/ssl.py", line 513, in wrap_socket
2023-06-27T00:04:22.080151+00:00 app[web.1]: return self.sslsocket_class._create(
2023-06-27T00:04:22.080151+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/ssl.py", line 1071, in _create
2023-06-27T00:04:22.080151+00:00 app[web.1]: self.do_handshake()
2023-06-27T00:04:22.080152+00:00 app[web.1]: File "/app/.heroku/python/lib/python3.10/ssl.py", line 1342, in do_handshake
2023-06-27T00:04:22.080152+00:00 app[web.1]: self._sslobj.do_handshake()
2023-06-27T00:04:22.080154+00:00 app[web.1]: ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:997)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.