vishalanandl177 / drf-api-logger Goto Github PK
View Code? Open in Web Editor NEWAn API Logger for your Django Rest Framework project.
Home Page: https://github.com/vishalanandl177/DRF-API-Logger
License: Apache License 2.0
An API Logger for your Django Rest Framework project.
Home Page: https://github.com/vishalanandl177/DRF-API-Logger
License: Apache License 2.0
The issue still persists therefore I reopen the issue. Check the new comment on the related issue. #23
Thanks in advance...
when
DRF_API_LOGGER_METHODS = ['POST', 'DELETE', 'PUT']
GET method will be request twice.
Can have an configuration example of the parameter DRF_API_LOGGER_SKIP_URL_NAME?
I want to know how to skip some path in my project.
Thank you in advance
DOMGA
There should be an option to clear logs in Django admin. Maybe through Django admin actions
good day,
today one of my requests returned 500 status_code due to drf-api-logger internal error
is the filed type can be changed somewhere from config file ?
best regards, Askhat
Everything is perfectly working in development environment. All my APIs that I am using POSTMAN desktop application is logging in drf_api_logger model. In nutshell all is working perfect as what I am looking. But once I transfer my code to production environment drf_api_logger is not logging any API calls from POSTMAN from both version desktop and web base POSTMAN.
I tried by looking options in drf_api_logger documentation, nothing works. One thing I noticed the difference in development and production is of "http" vs "https".
Can you please advice how I can make drf_api_logger to work for my production environment.
It would be very convenient to log based on the user ID. But as far as I can see there is no such thing by default, maybe there are thoughts on how to do this?
Hi, thank you for this amazing package.
Following your set-up it looks like I am able to log just APIis that have 200 errors;
Any idea?
(A part from adding it in middleware and installeds app)
DRF_API_LOGGER_DATABASE = True # Default to False
DRF_LOGGER_QUEUE_MAX_SIZE = 100
DRF_LOGGER_INTERVAL = 10
DRF_API_LOGGER_EXCLUDE_KEYS = ['password', 'token', 'access', 'refresh']
Everything was working fine, after install this lib my django test CI failed. Test database is not destroying and raising
I am 100% sure, session is used by drf-api-logger.
django.db.utils.OperationalError: database "test_betrics_db" is being accessed by other users DETAIL: There is 1 other session using the database.
Hi,
I follow README to add app, middleware and some settings in settings.py
.
The admin panel have new page: http://localhost/admin/drf_api_logger/apilogsmodel/
But no log show up.
I called API work as expect.
method = GET
url = http://localhost:3000/
response = Hello!!
Django log
Performing system checks...
Watching for file changes with StatReloader
System check identified no issues (0 silenced).
October 16, 2022 - 11:49:57
Django version 3.2.16, using settings 'DrfLogTest.settings'
Starting development server at http://0.0.0.0:3000/
Quit the server with CTRL-BREAK.
[16/Oct/2022 11:50:09] "GET /admin/drf_api_logger/apilogsmodel/ HTTP/1.1" 200 8690
[16/Oct/2022 11:50:09] "GET /admin/jsi18n/ HTTP/1.1" 200 3195
[16/Oct/2022 11:50:13] "GET / HTTP/1.1" 200 140
[16/Oct/2022 11:50:14] "GET / HTTP/1.1" 200 140
[16/Oct/2022 11:50:15] "GET / HTTP/1.1" 200 140
[16/Oct/2022 11:50:15] "GET / HTTP/1.1" 200 140
[16/Oct/2022 11:50:15] "GET / HTTP/1.1" 200 140
[16/Oct/2022 11:50:15] "GET / HTTP/1.1" 200 140
[16/Oct/2022 11:50:15] "GET / HTTP/1.1" 200 140
I also copy the signal example code and it doesn't print anything either.
Is there anything I forget to setup??
Run pip freeze
asgiref==3.5.2
beautifulsoup4==4.11.1
bleach==5.0.1
bs4==0.0.1
certifi==2022.9.24
charset-normalizer==2.1.1
Django==4.1.2
django-tables2==2.4.1
djangorestframework==3.14.0
drf-api-logger==1.1.11
idna==3.4
keyboard==0.13.5
psycopg2==2.9.4
python-vlc==3.0.16120
pytz==2022.4
requests==2.28.1
six==1.16.0
soupsieve==2.3.2.post1
sqlparse==0.4.3
typing-extensions==4.4.0
urllib3==1.26.12
webencodings==0.5.1
settings.py
as follow.
"""
Django settings for DrfLogTest project.
Generated by 'django-admin startproject' using Django 3.0.5.
For more information on this file, see
https://docs.djangoproject.com/en/3.0/topics/settings/
For the full list of settings and their values, see
https://docs.djangoproject.com/en/3.0/ref/settings/
"""
import os
# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
from config import database_config
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/3.0/howto/deployment/checklist/
# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = '$t=lyl(()ytv0%vx9^^o_-zivv%upc&&n0o#l4%g#=o!u07!m0'
# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = True
ALLOWED_HOSTS = ['*']
# Application definition
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'django_tables2',
'rest_framework',
'crawler',
'drf_api_logger',
]
MIDDLEWARE = [
'django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
'drf_api_logger.middleware.api_logger_middleware.APILoggerMiddleware',
]
ROOT_URLCONF = 'DrfLogTest.urls'
TEMPLATES = [
{
'BACKEND': 'django.template.backends.django.DjangoTemplates',
'DIRS': [os.path.join(BASE_DIR, 'templates')]
,
'APP_DIRS': True,
'OPTIONS': {
'context_processors': [
'django.template.context_processors.debug',
'django.template.context_processors.request',
'django.contrib.auth.context_processors.auth',
'django.contrib.messages.context_processors.messages',
],
},
},
]
WSGI_APPLICATION = 'DrfLogTest.wsgi.application'
# Database
# https://docs.djangoproject.com/en/3.0/ref/settings/#databases
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': os.path.join(BASE_DIR, 'db.sqlite3')
# 'ENGINE': 'django.db.backends.postgresql_psycopg2',
# 'NAME': database_config.name,
# 'USER': database_config.user,
# 'PASSWORD': database_config.password,
# 'HOST': database_config.host,
# 'PORT': database_config.port,
}
}
# Password validation
# https://docs.djangoproject.com/en/3.0/ref/settings/#auth-password-validators
AUTH_PASSWORD_VALIDATORS = [
{
'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
},
]
# Internationalization
# https://docs.djangoproject.com/en/3.0/topics/i18n/
LANGUAGE_CODE = 'en-us'
TIME_ZONE = 'UTC'
USE_I18N = True
USE_L10N = True
USE_TZ = True
# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/3.0/howto/static-files/
STATIC_URL = '/static/'
SITE_ID = 1
# drf logger
DRF_API_LOGGER_DATABASE = True
DRF_API_LOGGER_SIGNAL = True
DRF_API_LOGGER_METHODS = ['GET', 'POST', 'DELETE', 'PUT']
DRF_API_LOGGER_STATUS_CODES = ['200', '400', '404', '500']
DRF_LOGGER_QUEUE_MAX_SIZE = 1
DRF_LOGGER_INTERVAL = 1
Hey,
It's not an issue, but an feature request.
Is there a possibility to skip the response in case of get Method and response code 200?
after one day I've got about 200MB of data overhead due to this.
I disabled now the logging for get requests.
Thank you in advance!
@vishalanandl177
I thing you should update api
field in APILogsModel
into TextField
. The reason is, when we use ModelMultipleChoiceFilter
in filterset class, it creates extra long urls like, /api/v1/events/inspections/widgets/?&client=6&client=5&client=2&widget=top_service_locations
Right now the max_length=1024
of api
field and if URL goes long than 1024, you package will throw error.
Thanks!
I made some tests and it seems that:
It logs all the API information for content type "application/json".
Does not seem to hold true.
Version drf-api-logger==1.0.5
seems to be logging all types of requests.
The response
field is skipped, but the entry is created.
Hi!
Having gone through the documentation it is my understanding that logs are written into the default
database described in the settings.py
file.
Is it possible to add a feature to write the logs into some other database while using multiple databases? If not, can you suggest any alternative work-arounds for this?
Hi, @vishalanandl177
Thanks for your great library to log DRF apis.
Could you do me a favor?
I want to log only APIs with only long responsive time. How can I set config or customize middleware?
Hope your kind answer.
Thank you!
hello,
is it possible to store log in aws cloudwatch instead of the database? I am trying to achieve it using watchtower
I have configured the app and tested for 500 status but api does not logged 500 status, though it logged 200, 400. I am following default configuration.
Have you considered adding logging of DELETE requests? These are not logged now since they do not have a content-type, but deleting resources seems to be something you would want to log. I've patched my copy and it's only a few lines of code changes to allow it to log DELETE requests also.
Hi,
I installed this package some days ago and Im really happy with it!
However, I have noticed that the logger also logs the password unencrypted to the database if REST auth is used because it's part of the request.
Is there an option to disable or encrypt the password in the requests?
Usage of multiple schemas with DRF_API_LOGGER_DATABASE does not work, because DRF-API-Logger is not aware of different schemas. The option is to use a logger signal and disable automatic logs. Unfortunately when DRF_API_LOGGER_DATABASE is set to false all database migrations are not applied.
Solution:
Make database migrations and all model function independent from DRF_API_LOGGER_DATABASE.
Thanks for this package. I'm excited to try it.
When I got it, it needed a migration, but that migration isn't saved in the package.
$ ./manage.py migrate
Operations to perform:
Apply all migrations: admin, auth, ..., sessions
Running migrations:
No migrations to apply.
Your models have changes that are not yet reflected in a migration, and so won't be applied.
Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.
$ ./manage.py makemigrations
Migrations for 'drf_api_logger':
.venv/lib/python3.9/site-packages/drf_api_logger/migrations/0001_initial.py
- Create model APILogsModel
I believe you should make the migration and check it in.
I could probably produce a pull request if you wish.
When I run tests with pytest, I get:
src/apps/accounts/tests/test_endpoint_post_login.py::test_happy_path
/usr/local/lib/python3.10/site-packages/drf_api_logger/start_logger_when_server_starts.py:14: DeprecationWarning: getName() is deprecated, get the name attribute instead
if t.getName() == LOG_THREAD_NAME:
src/apps/accounts/tests/test_endpoint_post_login.py::test_happy_path
/usr/local/lib/python3.10/site-packages/drf_api_logger/start_logger_when_server_starts.py:20: DeprecationWarning: setName() is deprecated, set the name attribute instead
t.setName(LOG_THREAD_NAME)
so that I can prevent my DB from being filled with normal APIs requests with message 200... it goes fast...
thanks for the amazing work
How to log API failures that arent handled by try-catch blocks?
Below code snippet might helpful.
SENSITIVE_KEYS = ['password']
def mask_sensitive_data(data):
"""
Hides sensitive keys specified in sensitive_keys settings.
Loops recursively over nested dictionaries.
"""
if type(data) != dict:
return data
for key, value in data.items():
if key in SENSITIVE_KEYS:
data[key] = "***FILTERED***"
if type(value) == dict:
data[key] = mask_sensitive_data(data[key])
# // new code
if type(value) == list:
data[key] = [mask_sensitive_data(item) for item in data[key]]
return data
input_data = {"username":"any name", "password": "pass",
"extra": {"email":"mail@sample", "password": "pass"},
"list_items":[{"username":"blah", "password": "blah"}]}
masked_data = mask_sensitive_data(input_data)
print(masked_data)
Can you prompt, if need logger only for some method (POST or PUT) or multiple methods (POST and PUT) - how I can do it?
Can you prompt, if need logger only for some api - how I can do it? exist DRF_API_LOGGER_SKIP_NAMESPACE and DRF_API_LOGGER_SKIP_URL_NAME, but all enumerate - its bad and can something forgot.
Hi guys.
There is a suggestion to leave the user the ability to change the default exceptions
DRF-API-Logger/drf_api_logger/utils.py
Lines 4 to 7 in 8602734
SENSITIVE_KEYS = settings.DRF_API_LOGGER_EXCLUDE_KEYS
There are few apis which consists of long query params like
{{protocol}}://{{base_url}}/api/v1/configurations/area-templates/?search=searchstring&service=6&sub_location=9&sub_location=8&sub_location=7&fields=name,id&service_location=2&service_location=3§ion=1§ion=2§ion=3§ion=4§ion=5§ion=6§ion=7§ion=8§ion=9§ion=10§ion=11§ion=12§ion=13§ion=14§ion=15§ion=16§ion=17§ion=18§ion=19§ion=20§ion=21§ion=22&no_page
but according to the model field, this can only save 512 chars and raise
*** django.db.utils.DataError: value too long for type character varying(512)
class APILogsModel(BaseModel):
api = models.CharField(max_length=512, help_text='API URL')
Please update your setup.py in https://files.pythonhosted.org/packages/e0/26/6c903af8943b1f91584401d122e8d62ca4d2b778245f2afaee37a12484db/drf_api_logger-1.0.3.tar.gz#sha256=3b0dab7b0ccd9035425a18d0599d7f0a56c28a17108a02aa667e8a61fb50efdc
Error
```
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-kuuj6y0c/drf-api-logger_49ecd2f8baa1419691e19dcc415c23a3/setup.py'"'"'; file='"'"'/tmp/pip-install-kuuj6y0c/drf-api-logger_49ecd2f8baa1419691e19dcc415c23a3/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-pu29f_p6
cwd: /tmp/pip-install-kuuj6y0c/drf-api-logger_49ecd2f8baa1419691e19dcc415c23a3/
Complete output (7 lines):
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-install-kuuj6y0c/drf-api-logger_49ecd2f8baa1419691e19dcc415c23a3/setup.py", line 16, in
long_description=get_long_desc(),
File "/tmp/pip-install-kuuj6y0c/drf-api-logger_49ecd2f8baa1419691e19dcc415c23a3/setup.py", line 5, in get_long_desc
with open("README.md", "r") as fh:
FileNotFoundError: [Errno 2] No such file or directory: 'README.md'
----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/e0/26/6c903af8943b1f91584401d122e8d62ca4d2b778245f2afaee37a12484db/drf_api_logger-1.0.3.tar.gz#sha256=3b0dab7b0ccd9035425a18d0599d7f0a56c28a17108a02aa667e8a61fb50efdc (from https://pypi.org/simple/drf-api-logger/) (requires-python:>=3.5). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement drf-api-logger==1.0.3
ERROR: No matching distribution found for drf-api-logger==1.0.3
you have spelling mistake in reading README.md file in setup.py
Internal Server Error: /manage-console/drf_api_logger/apilogsmodel/
Traceback (most recent call last):
File "/Python/env/lib/python3.8/site-packages/asgiref/sync.py", line 458, in thread_handler
raise exc_info[1]
File "/Python/env/lib/python3.8/site-packages/django/core/handlers/exception.py", line 38, in inner
response = await get_response(request)
File "/Python/env/lib/python3.8/site-packages/django/core/handlers/base.py", line 233, in _get_response_async
response = await wrapped_callback(request, *callback_args, **callback_kwargs)
File "/Python/env/lib/python3.8/site-packages/asgiref/sync.py", line 423, in call
ret = await asyncio.wait_for(future, timeout=None)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/tasks.py", line 455, in wait_for
return await fut
File "/Python/env/lib/python3.8/site-packages/asgiref/current_thread_executor.py", line 22, in run
result = self.fn(*self.args, **self.kwargs)
File "/Python/env/lib/python3.8/site-packages/asgiref/sync.py", line 462, in thread_handler
return func(*args, **kwargs)
File "/Python/env/lib/python3.8/site-packages/django/contrib/admin/options.py", line 616, in wrapper
return self.admin_site.admin_view(view)(*args, **kwargs)
File "/Python/env/lib/python3.8/site-packages/django/utils/decorators.py", line 130, in _wrapped_view
response = view_func(request, *args, **kwargs)
File "/Python/env/lib/python3.8/site-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func
response = view_func(request, *args, **kwargs)
File "/Python/env/lib/python3.8/site-packages/django/contrib/admin/sites.py", line 232, in inner
return view(request, *args, **kwargs)
File "/Python/env/lib/python3.8/site-packages/drf_api_logger/admin.py", line 90, in changelist_view
filtered_query_set = response.context_data["cl"].queryset
KeyError: 'cl'
This is happening when i try to delete all the logs from django admin
Hi,
When I select one or more actions and I try to use the "Delete selected API logs" in the API Logs view in the django admin I get the following error:
http://0.0.0.0:16000/unadmin/drf_api_logger/apilogsmodel/
2.2.12
KeyError
'cl'
/usr/local/lib/python3.8/site-packages/drf_api_logger/admin.py in changelist_view, line 86
/usr/local/bin/python
3.8.5
If you need more info just let me know.
Thanks in advance!
Use case:
In Production, I want to deprecate an API endpoint. For this I need to know whether there are still clients that use this endpoint. I do not want any overhead for other endpoints.
Hi,
Great thanks, for this package!
I got bug from django "KeyError: 'content-type' in APILoggerMiddleware"
probably, you should use method ".get()" to get key 'content-type' from response
line 76:
if 'content-type' in response and
response.get('content-type') == 'application/json' or
response.get('content-type') == 'application/vnd.api+json':
I'm using Postman and the body field remains empty. The payload isn't being logged!
I've set it up as per the readme!
Any suggestions would be really appreciated!
@vishalanandl177
The JWT
token is stored in the headers
as follows.
headers:
{
"AUTHORIZATION": "Bearer [JWT]",
}
When a user sends a GET /log
request, I want to filter the APILogsModel
based on user_id
. (i.e. users can only see their own request history)
Pseudo code:
class LogView(generics.ListAPIView):
serializer_class = LogSerializer
def get_queryset(self):
token_str = self.request.META.get("HTTP_AUTHORIZATION", " ").split(" ")[1]
access_token = AccessToken(token_str)
##################################################
Filter by access_token
##################################################
return queryset
There is no field for user information in APILogsModel
. Therefore, it must be extracted from the JWT
stored in headers
. However, it seems inefficient to extract it from all headers
every time a request comes in.
Is there any other efficient way? (Is it possible to override a model or middleware?)
Please
Is it possible this package to add the username or user ID (if available) to the log messages?
Thank you in advance for your response
drf-api-logger throws an exception where there is no Host header in the request.
api_logger_middleware.py", line 93, in __call__
host = request.META['HTTP_HOST']
KeyError: 'HTTP_HOST'
I noticed this when running unit tests with the api logger enabled.
I have added the settings for different db for logs
DRF_API_LOGGER_DEFAULT_DATABASE = 'logs'
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': os.environ.get('DATABASE_NAME'),
'USER': os.environ.get('DATABASE_USER'),
'PASSWORD': os.environ.get('DATABASE_PASSWORD'),
'HOST': os.environ.get('DATABASE_HOST', 'localhost'),
'PORT': os.environ.get('DATABASE_PORT', 5432),
},
'logs': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': os.environ.get('LOGS_DATABASE_NAME'),
'USER': os.environ.get('LOGS_DATABASE_USER'),
'PASSWORD': os.environ.get('LOGS_DATABASE_PASSWORD'),
'HOST': os.environ.get('LOGS_DATABASE_HOST', 'localhost'),
'PORT': os.environ.get('LOGS_DATABASE_PORT', 5432),
}
}
But it shows error in django admin
ProgrammingError at /manage-console/drf_api_logger/apilogsmodel/
relation "drf_api_logs" does not exist
LINE 1: SELECT COUNT(*) AS "__count" FROM "drf_api_logs" WHERE ("drf...
in this file:
https://github.com/vishalanandl177/DRF-API-Logger/blob/master/drf_api_logger/templates/charts_change_list.html
you use two online file
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/Chart.js/2.8.0/Chart.min.css" />
<script src="https://cdnjs.cloudflare.com/ajax/libs/Chart.js/2.8.0/Chart.bundle.min.js"></script>
The page cannot be displayed correctly in the offline environment, how to fix it.
thanks.
{"date": "{{ item.added_on__date|date:'Y-m-d' }}", "y": {{ item.total }}},
When I add the lib to the installed apps and to the middleware the wsgi server instance returns only 404. The dev server still works fine though. Any ideas on why this occures ?
When deploying on a non-root path i.e. http://myserver.com/myapp/ the drf-api-logger middleware reports a 404 on all routes. This can easily be reproduced on a dev server by specifying FORCE_SCRIPT_NAME="/myapp/" in settings.py
The problem is request.path contains the myapp prefix and the path does not resolve. Switching to request.path_info will fix this.
Nothing apache-specific about it but I think this is what the person who reported #23 was experiencing.
TypeError: Object of type UUID is not JSON serializable
Specifically here: d['headers'] = json.dumps(d['headers'], indent=4)
The field is of type uuid.UUID .
I have found using the DjangoJSONEncoder allows for the encoding of these odd django fields
d['headers'] = json.dumps(d['headers'], indent=4, cls=DjangoJSONEncoder)
Django sites framework implementation is not considered. When using sites, app shows both application's api calls.
Is there any way to add user details to the logs for eg: username or user id.
One functionality i'll find useful is the ability to filter requests by a certain user for debugging purpose.
Perhaps an api is slow or failing for a certain user. Not sure if this is desirable for the entire project. I'll like to open a PR for it if it is. @vishalanandl177
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.