Overview

The JWT token is stored in the headers as follows.

headers:

{
"AUTHORIZATION": "Bearer [JWT]",
}

When a user sends a GET /log request, I want to filter the APILogsModel based on user_id. (i.e. users can only see their own request history)

Pseudo code:

class LogView(generics.ListAPIView):
    serializer_class = LogSerializer

    def get_queryset(self):
        token_str = self.request.META.get("HTTP_AUTHORIZATION", " ").split(" ")[1]
        access_token = AccessToken(token_str)

        ##################################################
        Filter by access_token
        ##################################################

        return queryset

Problem

There is no field for user information in APILogsModel. Therefore, it must be extracted from the JWT stored in headers. However, it seems inefficient to extract it from all headers every time a request comes in.

Conclusion

Is there any other efficient way? (Is it possible to override a model or middleware?)

Hi,

When I select one or more actions and I try to use the "Delete selected API logs" in the API Logs view in the django admin I get the following error:

http://0.0.0.0:16000/unadmin/drf_api_logger/apilogsmodel/
2.2.12
KeyError
'cl'
/usr/local/lib/python3.8/site-packages/drf_api_logger/admin.py in changelist_view, line 86
/usr/local/bin/python
3.8.5

If you need more info just let me know.

Thanks in advance!

Usage of multiple schemas with DRF_API_LOGGER_DATABASE does not work, because DRF-API-Logger is not aware of different schemas. The option is to use a logger signal and disable automatic logs. Unfortunately when DRF_API_LOGGER_DATABASE is set to false all database migrations are not applied.

Solution:
Make database migrations and all model function independent from DRF_API_LOGGER_DATABASE.

@vishalanandl177
I thing you should update api field in APILogsModel into TextField. The reason is, when we use ModelMultipleChoiceFilter in filterset class, it creates extra long urls like, /api/v1/events/inspections/widgets/?&client=6&client=5&client=2&widget=top_service_locations
Right now the max_length=1024 of api field and if URL goes long than 1024, you package will throw error.
Thanks!

How can I fix problem with Unicode Escape?

good day,
today one of my requests returned 500 status_code due to drf-api-logger internal error
is the filed type can be changed somewhere from config file ?
best regards, Askhat

There are few apis which consists of long query params like
{{protocol}}://{{base_url}}/api/v1/configurations/area-templates/?search=searchstring&service=6&sub_location=9&sub_location=8&sub_location=7&fields=name,id&service_location=2&service_location=3&section=1&section=2&section=3&section=4&section=5&section=6&section=7&section=8&section=9&section=10&section=11&section=12&section=13&section=14&section=15&section=16&section=17&section=18&section=19&section=20&section=21&section=22&no_page

but according to the model field, this can only save 512 chars and raise
*** django.db.utils.DataError: value too long for type character varying(512)

class APILogsModel(BaseModel):
    api = models.CharField(max_length=512, help_text='API URL')

Have you considered adding logging of DELETE requests? These are not logged now since they do not have a content-type, but deleting resources seems to be something you would want to log. I've patched my copy and it's only a few lines of code changes to allow it to log DELETE requests also.

Can you prompt, if need logger only for some api - how I can do it? exist DRF_API_LOGGER_SKIP_NAMESPACE and DRF_API_LOGGER_SKIP_URL_NAME, but all enumerate - its bad and can something forgot.

There should be an option to clear logs in Django admin. Maybe through Django admin actions

TypeError: Object of type UUID is not JSON serializable

Specifically here: d['headers'] = json.dumps(d['headers'], indent=4)

The field is of type uuid.UUID .

I have found using the DjangoJSONEncoder allows for the encoding of these odd django fields
d['headers'] = json.dumps(d['headers'], indent=4, cls=DjangoJSONEncoder)

One functionality i'll find useful is the ability to filter requests by a certain user for debugging purpose.
Perhaps an api is slow or failing for a certain user. Not sure if this is desirable for the entire project. I'll like to open a PR for it if it is. @vishalanandl177

Everything is perfectly working in development environment. All my APIs that I am using POSTMAN desktop application is logging in drf_api_logger model. In nutshell all is working perfect as what I am looking. But once I transfer my code to production environment drf_api_logger is not logging any API calls from POSTMAN from both version desktop and web base POSTMAN.

I tried by looking options in drf_api_logger documentation, nothing works. One thing I noticed the difference in development and production is of "http" vs "https".

Can you please advice how I can make drf_api_logger to work for my production environment.

I'm using Postman and the body field remains empty. The payload isn't being logged!
I've set it up as per the readme!
Any suggestions would be really appreciated!
@vishalanandl177

when

DRF_API_LOGGER_METHODS = ['POST', 'DELETE', 'PUT']

GET method will be request twice.

Below code snippet might helpful.

SENSITIVE_KEYS = ['password']

def mask_sensitive_data(data):
    """
    Hides sensitive keys specified in sensitive_keys settings.
    Loops recursively over nested dictionaries.
    """

    if type(data) != dict:
      return data

    for key, value in data.items():
      if key in SENSITIVE_KEYS:
        data[key] = "***FILTERED***"

      if type(value) == dict:
        data[key] = mask_sensitive_data(data[key])
      
      # // new code
      if type(value) == list:
          data[key] = [mask_sensitive_data(item) for item in data[key]]

    return data


input_data = {"username":"any name", "password": "pass",
     "extra": {"email":"mail@sample", "password": "pass"},
     "list_items":[{"username":"blah", "password": "blah"}]}

masked_data = mask_sensitive_data(input_data)
print(masked_data)

Hi,
Great thanks, for this package!
I got bug from django "KeyError: 'content-type' in APILoggerMiddleware"

probably, you should use method ".get()" to get key 'content-type' from response

line 76:
if 'content-type' in response and
response.get('content-type') == 'application/json' or
response.get('content-type') == 'application/vnd.api+json':

How to log API failures that arent handled by try-catch blocks?

Django sites framework implementation is not considered. When using sites, app shows both application's api calls.

I have configured the app and tested for 500 status but api does not logged 500 status, though it logged 200, 400. I am following default configuration.

Hi guys.
There is a suggestion to leave the user the ability to change the default exceptions

SENSITIVE_KEYS = settings.DRF_API_LOGGER_EXCLUDE_KEYS

drf-api-logger throws an exception where there is no Host header in the request.

api_logger_middleware.py", line 93, in __call__
    host = request.META['HTTP_HOST']
KeyError: 'HTTP_HOST'

I noticed this when running unit tests with the api logger enabled.

Thanks for this package. I'm excited to try it.

When I got it, it needed a migration, but that migration isn't saved in the package.

$ ./manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, ..., sessions
Running migrations:
  No migrations to apply.
  Your models have changes that are not yet reflected in a migration, and so won't be applied.
  Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.

$ ./manage.py makemigrations
Migrations for 'drf_api_logger':
  .venv/lib/python3.9/site-packages/drf_api_logger/migrations/0001_initial.py
    - Create model APILogsModel

I believe you should make the migration and check it in.

I could probably produce a pull request if you wish.

I have added the settings for different db for logs
DRF_API_LOGGER_DEFAULT_DATABASE = 'logs'

DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': os.environ.get('DATABASE_NAME'),
'USER': os.environ.get('DATABASE_USER'),
'PASSWORD': os.environ.get('DATABASE_PASSWORD'),
'HOST': os.environ.get('DATABASE_HOST', 'localhost'),
'PORT': os.environ.get('DATABASE_PORT', 5432),
},
'logs': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': os.environ.get('LOGS_DATABASE_NAME'),
'USER': os.environ.get('LOGS_DATABASE_USER'),
'PASSWORD': os.environ.get('LOGS_DATABASE_PASSWORD'),
'HOST': os.environ.get('LOGS_DATABASE_HOST', 'localhost'),
'PORT': os.environ.get('LOGS_DATABASE_PORT', 5432),
}
}

But it shows error in django admin
ProgrammingError at /manage-console/drf_api_logger/apilogsmodel/

relation "drf_api_logs" does not exist
LINE 1: SELECT COUNT(*) AS "__count" FROM "drf_api_logs" WHERE ("drf...

It would be very convenient to log based on the user ID. But as far as I can see there is no such thing by default, maybe there are thoughts on how to do this?

Can have an configuration example of the parameter DRF_API_LOGGER_SKIP_URL_NAME?
I want to know how to skip some path in my project.
Thank you in advance

DOMGA

Use case:
In Production, I want to deprecate an API endpoint. For this I need to know whether there are still clients that use this endpoint. I do not want any overhead for other endpoints.

Hey,

It's not an issue, but an feature request.
Is there a possibility to skip the response in case of get Method and response code 200?
after one day I've got about 200MB of data overhead due to this.

I disabled now the logging for get requests.

Thank you in advance!

Hi,

I follow README to add app, middleware and some settings in settings.py.

The admin panel have new page: http://localhost/admin/drf_api_logger/apilogsmodel/

But no log show up.

I called API work as expect.

method = GET
url = http://localhost:3000/

response = Hello!!

Django log

Performing system checks...

Watching for file changes with StatReloader
System check identified no issues (0 silenced).
October 16, 2022 - 11:49:57
Django version 3.2.16, using settings 'DrfLogTest.settings'
Starting development server at http://0.0.0.0:3000/
Quit the server with CTRL-BREAK.
[16/Oct/2022 11:50:09] "GET /admin/drf_api_logger/apilogsmodel/ HTTP/1.1" 200 8690
[16/Oct/2022 11:50:09] "GET /admin/jsi18n/ HTTP/1.1" 200 3195
[16/Oct/2022 11:50:13] "GET / HTTP/1.1" 200 140
[16/Oct/2022 11:50:14] "GET / HTTP/1.1" 200 140
[16/Oct/2022 11:50:15] "GET / HTTP/1.1" 200 140
[16/Oct/2022 11:50:15] "GET / HTTP/1.1" 200 140
[16/Oct/2022 11:50:15] "GET / HTTP/1.1" 200 140
[16/Oct/2022 11:50:15] "GET / HTTP/1.1" 200 140
[16/Oct/2022 11:50:15] "GET / HTTP/1.1" 200 140

I also copy the signal example code and it doesn't print anything either.

Is there anything I forget to setup??

Run pip freeze

asgiref==3.5.2
beautifulsoup4==4.11.1
bleach==5.0.1
bs4==0.0.1
certifi==2022.9.24
charset-normalizer==2.1.1
Django==4.1.2
django-tables2==2.4.1
djangorestframework==3.14.0
drf-api-logger==1.1.11
idna==3.4
keyboard==0.13.5
psycopg2==2.9.4
python-vlc==3.0.16120
pytz==2022.4
requests==2.28.1
six==1.16.0
soupsieve==2.3.2.post1
sqlparse==0.4.3
typing-extensions==4.4.0
urllib3==1.26.12
webencodings==0.5.1

settings.py as follow.

"""
Django settings for DrfLogTest project.

Generated by 'django-admin startproject' using Django 3.0.5.

For more information on this file, see
https://docs.djangoproject.com/en/3.0/topics/settings/

For the full list of settings and their values, see
https://docs.djangoproject.com/en/3.0/ref/settings/
"""

import os

# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
from config import database_config

BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))

# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/3.0/howto/deployment/checklist/

# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = '$t=lyl(()ytv0%vx9^^o_-zivv%upc&&n0o#l4%g#=o!u07!m0'

# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = True

ALLOWED_HOSTS = ['*']

# Application definition

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',

    'django_tables2',

    'rest_framework',
    'crawler',

    'drf_api_logger',
]

MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',

    'drf_api_logger.middleware.api_logger_middleware.APILoggerMiddleware',
]

ROOT_URLCONF = 'DrfLogTest.urls'

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [os.path.join(BASE_DIR, 'templates')]
        ,
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

WSGI_APPLICATION = 'DrfLogTest.wsgi.application'

# Database
# https://docs.djangoproject.com/en/3.0/ref/settings/#databases

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.path.join(BASE_DIR, 'db.sqlite3')
        # 'ENGINE': 'django.db.backends.postgresql_psycopg2',
        # 'NAME': database_config.name,
        # 'USER': database_config.user,
        # 'PASSWORD': database_config.password,
        # 'HOST': database_config.host,
        # 'PORT': database_config.port,
    }
}

# Password validation
# https://docs.djangoproject.com/en/3.0/ref/settings/#auth-password-validators

AUTH_PASSWORD_VALIDATORS = [
    {
        'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
    },
]

# Internationalization
# https://docs.djangoproject.com/en/3.0/topics/i18n/

LANGUAGE_CODE = 'en-us'

TIME_ZONE = 'UTC'

USE_I18N = True

USE_L10N = True

USE_TZ = True

# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/3.0/howto/static-files/

STATIC_URL = '/static/'

SITE_ID = 1

# drf logger
DRF_API_LOGGER_DATABASE = True
DRF_API_LOGGER_SIGNAL = True
DRF_API_LOGGER_METHODS = ['GET', 'POST', 'DELETE', 'PUT']
DRF_API_LOGGER_STATUS_CODES = ['200', '400', '404', '500']
DRF_LOGGER_QUEUE_MAX_SIZE = 1
DRF_LOGGER_INTERVAL = 1

Internal Server Error: /manage-console/drf_api_logger/apilogsmodel/
Traceback (most recent call last):
File "/Python/env/lib/python3.8/site-packages/asgiref/sync.py", line 458, in thread_handler
raise exc_info[1]
File "/Python/env/lib/python3.8/site-packages/django/core/handlers/exception.py", line 38, in inner
response = await get_response(request)
File "/Python/env/lib/python3.8/site-packages/django/core/handlers/base.py", line 233, in _get_response_async
response = await wrapped_callback(request, *callback_args, **callback_kwargs)
File "/Python/env/lib/python3.8/site-packages/asgiref/sync.py", line 423, in call
ret = await asyncio.wait_for(future, timeout=None)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/tasks.py", line 455, in wait_for
return await fut
File "/Python/env/lib/python3.8/site-packages/asgiref/current_thread_executor.py", line 22, in run
result = self.fn(*self.args, **self.kwargs)
File "/Python/env/lib/python3.8/site-packages/asgiref/sync.py", line 462, in thread_handler
return func(*args, **kwargs)
File "/Python/env/lib/python3.8/site-packages/django/contrib/admin/options.py", line 616, in wrapper
return self.admin_site.admin_view(view)(*args, **kwargs)
File "/Python/env/lib/python3.8/site-packages/django/utils/decorators.py", line 130, in _wrapped_view
response = view_func(request, *args, **kwargs)
File "/Python/env/lib/python3.8/site-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func
response = view_func(request, *args, **kwargs)
File "/Python/env/lib/python3.8/site-packages/django/contrib/admin/sites.py", line 232, in inner
return view(request, *args, **kwargs)
File "/Python/env/lib/python3.8/site-packages/drf_api_logger/admin.py", line 90, in changelist_view
filtered_query_set = response.context_data["cl"].queryset
KeyError: 'cl'

This is happening when i try to delete all the logs from django admin

Can you prompt, if need logger only for some method (POST or PUT) or multiple methods (POST and PUT) - how I can do it?

Hi!

Having gone through the documentation it is my understanding that logs are written into the default database described in the settings.py file.

Is it possible to add a feature to write the logs into some other database while using multiple databases? If not, can you suggest any alternative work-arounds for this?

The issue still persists therefore I reopen the issue. Check the new comment on the related issue. #23

Thanks in advance...

Hi, @vishalanandl177
Thanks for your great library to log DRF apis.

Could you do me a favor?
I want to log only APIs with only long responsive time. How can I set config or customize middleware?

Hope your kind answer.

Thank you!

When I run tests with pytest, I get:

src/apps/accounts/tests/test_endpoint_post_login.py::test_happy_path
  /usr/local/lib/python3.10/site-packages/drf_api_logger/start_logger_when_server_starts.py:14: DeprecationWarning: getName() is deprecated, get the name attribute instead
    if t.getName() == LOG_THREAD_NAME:

src/apps/accounts/tests/test_endpoint_post_login.py::test_happy_path
  /usr/local/lib/python3.10/site-packages/drf_api_logger/start_logger_when_server_starts.py:20: DeprecationWarning: setName() is deprecated, set the name attribute instead
    t.setName(LOG_THREAD_NAME)

Is there any way to add user details to the logs for eg: username or user id.

Please
Is it possible this package to add the username or user ID (if available) to the log messages?
Thank you in advance for your response

Hi,
I installed this package some days ago and Im really happy with it!
However, I have noticed that the logger also logs the password unencrypted to the database if REST auth is used because it's part of the request.
Is there an option to disable or encrypt the password in the requests?

hello,

is it possible to store log in aws cloudwatch instead of the database? I am trying to achieve it using watchtower

When I add the lib to the installed apps and to the middleware the wsgi server instance returns only 404. The dev server still works fine though. Any ideas on why this occures ?

so that I can prevent my DB from being filled with normal APIs requests with message 200... it goes fast...

thanks for the amazing work

Please update your setup.py in https://files.pythonhosted.org/packages/e0/26/6c903af8943b1f91584401d122e8d62ca4d2b778245f2afaee37a12484db/drf_api_logger-1.0.3.tar.gz#sha256=3b0dab7b0ccd9035425a18d0599d7f0a56c28a17108a02aa667e8a61fb50efdc

Error
```
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-kuuj6y0c/drf-api-logger_49ecd2f8baa1419691e19dcc415c23a3/setup.py'"'"'; file='"'"'/tmp/pip-install-kuuj6y0c/drf-api-logger_49ecd2f8baa1419691e19dcc415c23a3/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-pu29f_p6
cwd: /tmp/pip-install-kuuj6y0c/drf-api-logger_49ecd2f8baa1419691e19dcc415c23a3/
Complete output (7 lines):
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-install-kuuj6y0c/drf-api-logger_49ecd2f8baa1419691e19dcc415c23a3/setup.py", line 16, in
long_description=get_long_desc(),
File "/tmp/pip-install-kuuj6y0c/drf-api-logger_49ecd2f8baa1419691e19dcc415c23a3/setup.py", line 5, in get_long_desc
with open("README.md", "r") as fh:
FileNotFoundError: [Errno 2] No such file or directory: 'README.md'
----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/e0/26/6c903af8943b1f91584401d122e8d62ca4d2b778245f2afaee37a12484db/drf_api_logger-1.0.3.tar.gz#sha256=3b0dab7b0ccd9035425a18d0599d7f0a56c28a17108a02aa667e8a61fb50efdc (from https://pypi.org/simple/drf-api-logger/) (requires-python:>=3.5). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement drf-api-logger==1.0.3
ERROR: No matching distribution found for drf-api-logger==1.0.3

you have spelling mistake in reading README.md file in setup.py

Got empty body in every request in Django Admin logs

in this file:
https://github.com/vishalanandl177/DRF-API-Logger/blob/master/drf_api_logger/templates/charts_change_list.html

you use two online file

<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/Chart.js/2.8.0/Chart.min.css" />
<script src="https://cdnjs.cloudflare.com/ajax/libs/Chart.js/2.8.0/Chart.bundle.min.js"></script>

The page cannot be displayed correctly in the offline environment, how to fix it.

thanks.

Hi, thank you for this amazing package.
Following your set-up it looks like I am able to log just APIis that have 200 errors;

Any idea?

(A part from adding it in middleware and installeds app)

Store API Logger to Database

DRF_API_LOGGER_DATABASE = True # Default to False
DRF_LOGGER_QUEUE_MAX_SIZE = 100
DRF_LOGGER_INTERVAL = 10

hide sensitive information from being exposed in the logs

DRF_API_LOGGER_EXCLUDE_KEYS = ['password', 'token', 'access', 'refresh']

@vishalanandl177

When deploying on a non-root path i.e. http://myserver.com/myapp/ the drf-api-logger middleware reports a 404 on all routes. This can easily be reproduced on a dev server by specifying FORCE_SCRIPT_NAME="/myapp/" in settings.py

The problem is request.path contains the myapp prefix and the path does not resolve. Switching to request.path_info will fix this.

Nothing apache-specific about it but I think this is what the person who reported #23 was experiencing.

I made some tests and it seems that:

It logs all the API information for content type "application/json".

Does not seem to hold true.

Version drf-api-logger==1.0.5 seems to be logging all types of requests.

The response field is skipped, but the entry is created.

Everything was working fine, after install this lib my django test CI failed. Test database is not destroying and raising
I am 100% sure, session is used by drf-api-logger.

django.db.utils.OperationalError: database "test_betrics_db" is being accessed by other users DETAIL: There is 1 other session using the database.

I am not sure, but I believe that if there is an error parsing a POST body, possibly it does not capture the body.

Here's what I see in the Django admin UI:

Unfortunately, I'm not quite sure how to easily set up a reproducible test case in your code repo. Suggestions welcome.