aschn / drf-tracking Goto Github PK
View Code? Open in Web Editor NEWUtils to track requests to Django Rest Framework API views
Home Page: http://drf-tracking.readthedocs.org/
License: ISC License
Utils to track requests to Django Rest Framework API views
Home Page: http://drf-tracking.readthedocs.org/
License: ISC License
We need to add the now required on_delete attribute to the user ForeignKey in the BaseAPIRequestLog.
I tried to implement the tracking for the endpoint / views that needed EMAIL
field in the request data, and it failing
File "/.virtualenvs/lib/python2.7/site-packages/rest_framework_tracking/mixins.py", line 72, in initial
self.request.log.data = self._clean_data(self.request.data)
File "/.virtualenvs/lib/python2.7/site-packages/rest_framework_tracking/mixins.py", line 142, in _clean_data
value = ast.literal_eval(value)
File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 49, in literal_eval
node_or_string = parse(node_or_string, mode='eval')
File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 37, in parse
return compile(source, filename, mode, PyCF_ONLY_AST)
File "<unknown>", line 1
[email protected]
^
SyntaxError: invalid syntax
i've already using the dependencies from repo but it still failing
I noticed that the APIRequestLogAdmin
class allows users to add and edit records. If that's not needed, I would suggest making all of the fields read-only and adding this:
def has_add_permission(self, request):
return False
If it is needed then could it be a setting so that developers can more easily disable the adding and editing of records? Probably two settings, actually (one for adding and one for editing).
As for deleting, I can see why that hasn't been disabled as people may want to delete records from the admin. This might be another thing to be made into a setting, though, since some developers may want to delete records on a schedule instead and disallow deleting from the admin.
I am getting an error with logging on write operations.
The DB is PostgreSQL 9.5 and using latest Django Rest Framework.
TransactionManagementError: An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block.
Has anyone else run into this issue. I didn't notice anything in the issues.
I'm currently working on a possibility to store data either in the database or in a elastic search cluster.
If you have idea or input regarding this feature, please share your thoughts here.
The field is a float field. All other float fields work.
I can see how this would cause errors on
'capitalized_cost' (my field)
'keyless_entry'
'user_signature_jpg_url'
While I do not mind changing the name to cap_cost I'd still much rather not do it.
One possible suggestion for this fix is to add a '_' prefix / suffix.
because most of the times it will be:
'stripe_api_key'
'stripe_token'
'paypal_signature'
So change the regex line to (_api)|(api_)|(_token)|(token_)|(_key)|(key_)|(_secret)|(secret_)|(_password)|(password_)|(_signature)|(signature_)
And this has to be in the documentation.
def _clean_data(data):
"""
Clean a dictionary of data of potentially sensitive info before
sending to the database.
Function based on the "_clean_credentials" function of django
(django/django/contrib/auth/__init__.py)
"""
SENSITIVE_DATA = re.compile('api|token|key|secret|password|signature', re.I)
CLEANSED_SUBSTITUTE = '********************'
for key in data:
if SENSITIVE_DATA.search(key):
data[key] = CLEANSED_SUBSTITUTE
return data
Hi, this module is exactly what I'm looking for, in order to track usage of our APIs. After installing and adding to DRF ViewSets, some scenarios raise Exceptions instead of gracefully handling them.
File "/home/ubuntu/virtualenvs/venv-system/local/lib/python2.7/site-packages/rest_framework/authentication.py", line 185, in authenticate_credentials
raise exceptions.AuthenticationFailed(_('Invalid token.'))
AuthenticationFailed: Invalid token.
Here I was trying to test if my API returns the correct status code if Token is invalid.
File "/home/ubuntu/virtualenvs/venv-system/local/lib/python2.7/site-packages/rest_framework/authentication.py", line 188, in authenticate_credentials
raise exceptions.AuthenticationFailed(_('User inactive or deleted.'))
AuthenticationFailed: User inactive or deleted.
Here I was trying to test if my API returns the correct status code if User is inactive/deleted.
I'll try and raise a PR to handle these exceptions.
Release 1.3.1
but it don't have latest code as you have in your master branch.e.g.,
_clean_data
as separate function but now you have moved it to class function and modified it too. Release 1.3.1 still have old code for this function (I have checked source code of Release too)TypeError: expected string or bytes-like object
in line No 137 if SENSITIVE_DATA.search(key):
list
of dict()
[ {
"actor_type": "tets",
"meeting_with": "Self",
"first_name": "qwert",
"last_name": "asdfg",
"logged_at": "1448113944"
}
]
We have to make this compatible with django 2.
There are various things that are deprecated, such as, is_authenticated is not a function anymore.
I am using drf-tracking it is working great but i am getting error on class in which i am using password and tokens to perform operations.
Is there any way to avoid hide token and password values ?
Subject : When multiple query params are passed with same key, only the last query params is captured and rest all are ignored/dropped.
Steps to reproduce :
Expected behaviour : Query params field should record {"name": ["A", "B", "C"]}
Actual behaviour : Query params field records only last value {"name": "C"}
UnicodeDecodeError
'utf-8' codec can't decode byte 0xff in position 400: invalid start byte
rest_framework_tracking/base_mixins.py in _clean_data at line 163
Seems there is the same problem like the #27 but now the missing attribute is 'accepted_renderer'.
I'm using:
drf-tracking (1.1.0)
django-rest-swagger (0.3.5)
Django (1.10.5)
Thanks
Hello all,
I just want to share an issue I had.
There was an exception regarding negative response time. After reading the code a bit I see that you have a justified assumption regarding the response_ms
being positive.
The implementation of the duration is straightforward so no surprises there. My solution was to patch the duration calculation to return 0 if the delta is negative.
Something like this:
response_timedelta = now() - self.request.log.requested_at
response_ms = int(response_timedelta.total_seconds() * 1000)
response_ms = max(response_ms, 0)
I wasn't able to reproduce the issue though.
I was wondering if anyone else have seen something similar.
Thanks
In venv/lib/python3.5/site-packages/rest_framework_tracking/mixins.py", line 134, in _clean_data
if SENSITIVE_DATA.search(key):
TypeError: expected string or bytes-like object
class MyBulkCreate(LoggingMixin, generics.CreateAPIView):
def create(self, request, *args, **kwargs):
serializer = serializers.FooBarSerializer(data=data, context=context, many=True)
my post data is a json as following:
[{"foo": "200" ,"bar": "A", "fufu":"1"},{"foo": "200" ,"bar": "B", "fufu":"1"}]
When i pass the data through Mixin and it crashed already. I have to skip using the tracking function for all bulk_create api.
Will there any solutions for now?
If there is a problem with parsing the posted JSON to DRF API endpoint, drf-tracking raises this exception and causes Internal Server Error. Reason is that inside LoggingMixin.initial if there is a ParseError thrown on data_dict = request.data.dict() self.request.log is never set, but finalize_response gets called anyway.
Problem is, you never get to see the actual problem (i.e. {"detail":"JSON parse error - Expecting property name enclosed in double quotes: line 7 column 1 (char 111)"}
), as the 500 Internal Error masks the problem.
Responses with status 404 should be logged. Currently, requests to a missing url are not logged.
I saw (partially my fault) that the doc on Read the doc the latest changes has not always been put there. I think that having it twice doesn't help to maintain it.
I would suggest we remove one to keep everything up-to-date in only one place.
WDYT
Great work! I have a similar mixin and have found it useful to also track the actual response. Would that be something useful to add to this project?
Hi
If possible to log custom headers information? I want to track secret keys which are part of the Post header.
Thanks
Hello, I have stumbled upon the same thing on my test suite. I'm testing accessing the given resource without authentication.
class DriverReadFailureTests(APITestCase):
def setUp(self):
self.client = APIClient()
self.user = User.objects.create_superuser('admin', '[email protected]', 'admin123')
self.client.force_login(user=self.user)
self.token = Token.objects.create(user=self.user)
self.client.post('/api/drivers/', data=driver_data, format='json')
self.client.post('/api/drivers/', data=driver_data_second, format='json')
self.client.logout()
def test_read_no_auth_driver(self):
"""Try to retrieve json representation of previously added driver.
"""
response = self.client.get('/api/drivers/')
self.assertEqual(response.status_code, 401)
Traceback:
Error
Traceback (most recent call last):
File "/home/bkovacev/sally-projects/ops4/drivers/tests.py", line 316, in test_read_no_auth_driver
response = self.client.get('/api/drivers/')
File "/home/bkovacev/envs/ops4-env/local/lib/python2.7/site-packages/rest_framework/test.py", line 282, in get
response = super(APIClient, self).get(path, data=data, **extra)
File "/home/bkovacev/envs/ops4-env/local/lib/python2.7/site-packages/rest_framework/test.py", line 208, in get
return self.generic('GET', path, **r)
File "/home/bkovacev/envs/ops4-env/local/lib/python2.7/site-packages/django/test/client.py", line 409, in generic
return self.request(**r)
File "/home/bkovacev/envs/ops4-env/local/lib/python2.7/site-packages/rest_framework/test.py", line 279, in request
return super(APIClient, self).request(**kwargs)
File "/home/bkovacev/envs/ops4-env/local/lib/python2.7/site-packages/rest_framework/test.py", line 231, in request
request = super(APIRequestFactory, self).request(**kwargs)
File "/home/bkovacev/envs/ops4-env/local/lib/python2.7/site-packages/django/test/client.py", line 494, in request
six.reraise(*exc_info)
File "/home/bkovacev/envs/ops4-env/local/lib/python2.7/site-packages/django/core/handlers/exception.py", line 42, in inner
response = get_response(request)
File "/home/bkovacev/envs/ops4-env/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 187, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/home/bkovacev/envs/ops4-env/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/bkovacev/envs/ops4-env/local/lib/python2.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
return view_func(*args, **kwargs)
File "/home/bkovacev/envs/ops4-env/local/lib/python2.7/site-packages/django/views/generic/base.py", line 68, in view
return self.dispatch(request, *args, **kwargs)
File "/home/bkovacev/envs/ops4-env/local/lib/python2.7/site-packages/rest_framework/views.py", line 483, in dispatch
response = self.handle_exception(exc)
File "/home/bkovacev/envs/ops4-env/local/lib/python2.7/site-packages/rest_framework_tracking/mixins.py", line 80, in handle_exception
self.request.log.errors = traceback.format_exc()
File "/home/bkovacev/envs/ops4-env/local/lib/python2.7/site-packages/rest_framework/request.py", line 384, in __getattribute__
six.reraise(info[0], info[1], info[2].tb_next)
File "<string>", line 2, in reraise
AttributeError: 'Request' object has no attribute 'log'
How do I approach this?
LoggingMixin overrides request data for fields such as "password", "key" etc. not only while saving the request log, but before the request is processed. My view got a long string of asterixes instead of the user provided value for fields "Key" and "Password". Took me a good 2 hours to find the issue.
Or to call logging when some conditions were fulfilled.
Mixin logs all requests to the view :(
More of a question, rather than an issue. Why is the majority of the log construction performed in finalize_response() as opposed to initial() ?
Im looking to log the request when an unexpected exception is thrown, having the log constructed in initial() will pack up all the request information.
Django: 2.0.4, python: 3.5.3
class BookingViewSet(viewset.ModelViewSet, LoggingMixin):
def get_queryset(self):
return Booking.objects.all()
def get_serializer_class(self):
return BookingSerializer
After making multiple calls to this api viewset, no records added in API REQUEST LOGS.
is there any decorator to use it with functin based view?
Having foreign keys to User model doesn't allow us to store logs in a separate database.
Any http 500 error causes log to have status code not assigned and response ms to 0. It would be nice to see the trace instead.
I'm using Django-OAuth-Toolkit to allow client based authorization.
I have set different client for different platforms. Like for Android, a different client is used to generate the access token and for the web, a different client is used to generate the access token.
I want to log which the client is making the request.
is it possible to write log with async tasking i dont want my view thread to get blocked with drf tracking operation or to write in db. I am using celery for async task.
I'm using Django 1.11.7, and using drf-tracing
at latest master branch (b753c5e).
When I run python manage.py makemigrations
, I got the following conflict error:
CommandError: Conflicting migrations detected; multiple leaf nodes in the migration graph: (0006_auto_20180315_1442, 0006_view_and_view_method_nullable in rest_framework_tracking).
To fix them run 'python manage.py makemigrations --merge'
I can solve it by running python manage.py makemigrations --merge
but thats a bit inconvenient when using travis, since I need to run makemigrations
every time the build process running. Is there any plan to fix this?
Hi,
Thanks for the useful plugin!
Since some of our API responses can be large, the log table is becoming slow to load due to the size of the response column. They are cropped to [204K], (I think due to Django/postgres?), but this is still a lot and means we have to update the column data manually to get sensible performance.
Is there an easy way to turn off or limit the size of the response stored?
Cheers
Sim
From TravisCI support:
Please take a look at https://travis-ci.org/aschn/drf-tracking/requests, which shows that your recent build requests exceeds the limit of 200 jobs within a build, and were therefore rejected. Please try reducing the matrix size.
Reducing the build matrix to under 200 builds should resume builds on TravisCI again.
how to use it function base view
Exception occurs while presence in request, for example, dates:
def test_log_params(self):
self.client.get('/logging', {'p1': 'a', 'another': '2', 'date_field': '1970-01-01'})
log = APIRequestLog.objects.first()
self.assertEqual(ast.literal_eval(log.query_params), {
u'p1': u'a', u'another': u'2', u'date_field': u'1970-01-01'})
Also, why user can not change SENSITIVE_FIELDS by self? Currently, field named, for example, 'is_secretary' leads to broken request handling.
Add a status_code
field to the model, mixin, and admin list filter.
I want request.data too? how to override initial method? i wish to compare data before and after update. so i gone through Log Mixin , in that, you didnt update request.data into tables. so how to override my "def initial()" mixin? to insert request.data too ...
This filter
list_filter = ('user', 'path', 'method', 'status_code')
is a bit annoying if you have +1000 users.
and can you add
raw_id_fields = ('user', )
to APIRequestLogAdmin class ?
Here is what I learned from MySQL manual (http://dev.mysql.com/doc/refman/5.7/en/create-index.html):
"Prefixes must be specified for BLOB and TEXT column indexes."
Introducing on_delete=models.PROTECT
for the ApiRequestLog
object in response to #91 makes it impossible to delete users without deleting every single related API request manually beforehand.
Please change the behaviour to on_delete=models.CASCADE
, which makes sure that users can be deleted and all related API request logs will be also removed in the process.
Alternatively SET_NULL
would also be an option, if you want to keep the request logs intact also after user deletion.
Hi,
there are different python version requirements listed here in readme and at http://drf-tracking.readthedocs.io/en/latest/#requirements
README:
Web:
Hey @avelis,
I think it's would be really great to have a new version released on PYPI as the actual one has an important issue.
Do you think it's possible to do it ?
Thanks
Is there currently a way to enable/disable (toggling) logging of certain fields? For example, if I didn't want to capture the logging of response
field. Would that be possible currently? TIA.
Hi,
drf-tracking 1.2.0 crash if you use a DRF APIView that returns a FileResponse.
Could you fix this behaviour?
Thanks,
Simone
Environment: development
Request Method: GET
Request URL: http://prm.mydevelopmentenv.com/api/export/files
Django Version: 1.8.18
Python Version: 3.3.5
Installed Applications:
('django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'debug_toolbar',
'django_q',
'rest_framework',
'rest_framework.authtoken',
'rest_framework_tracking',
'maintenancemode',
'prm')
Installed Middleware:
('django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'prm.middleware.SetRemoteAddrMiddleware',
'debug_toolbar.middleware.DebugToolbarMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
'maintenancemode.middleware.MaintenanceModeMiddleware')
Traceback:
File "/home/prm/virtualenv/lib/python3.3/site-packages/django/core/handlers/base.py" in get_response
132. response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/home/prm/virtualenv/lib/python3.3/site-packages/django/views/decorators/csrf.py" in wrapped_view
58. return view_func(*args, **kwargs)
File "/home/prm/virtualenv/lib/python3.3/site-packages/django/views/generic/base.py" in view
71. return self.dispatch(request, *args, **kwargs)
File "/home/prm/virtualenv/lib/python3.3/site-packages/rest_framework/views.py" in dispatch
491. self.response = self.finalize_response(request, response, *args, **kwargs)
File "/home/prm/virtualenv/lib/python3.3/site-packages/rest_framework_tracking/mixins.py" in finalize_response
100. self.request.log.response = response.rendered_content
Exception Type: AttributeError at /api/export/files
Exception Value: 'FileResponse' object has no attribute 'rendered_content'
Thanks for your contribution first.. its working fine currently i am getting data after request like put, patch was updated one. but I want to have both updated and requested before change. is it possible ?
Hi - I am trying to get API tracking data for several APIs. In order to analyze this data later, I am wondering if there is a way to switch the data writes to a file instead of DB.
Thanks
Santi
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.