Giter Club home page Giter Club logo

django-pghistory's Introduction

django-pghistory

django-pghistory tracks changes to your Django models using Postgres triggers, providing:

  • Reliable history tracking everywhere with no changes to your application code.
  • Structured history models that mirror the fields of your models.
  • Grouping of history with additional context attached, such as the logged-in user.

django-pghistory has a number of ways in which you can configure history tracking for your application's needs and for performance and scale. An admin integration and middleware is included out of the box too.

Quick Start

Decorate your model with pghistory.track. For example:

import pghistory

@pghistory.track()
class TrackedModel(models.Model):
    int_field = models.IntegerField()
    text_field = models.TextField()

Above we've tracked TrackedModel. Copies of the model will be stored in a dynamically-created event model on every insert and update.

Run python manage.py makemigrations followed by migrate and voila, every change to TrackedModel is now stored. This includes bulk methods and even changes that happen in raw SQL. For example:

from myapp.models import TrackedModel

m = TrackedModel.objects.create(int_field=1, text_field="hello")
m.int_field = 2
m.save()

print(m.events.values("pgh_obj", "int_field"))

> [{'pgh_obj': 1, 'int_field': 1}, {'pgh_obj': 1, 'int_field': 2}]

Above we printed the history of int_field. We also printed pgh_obj, which references the tracked object. We'll cover how these fields and additional metadata fields are tracked later.

django-pghistory can track a subset of fields and conditionally store events based on specific field transitions. Users can also store free-form context from the application in event metadata, all with no additional database queries. See the next steps below on how to dive deeper and configure it for your use case.

Compatibility

django-pghistory is compatible with Python 3.8 - 3.12, Django 3.2 - 5.0, Psycopg 2 - 3, and Postgres 12 - 16.

Documentation

View the django-pghistory docs here to learn more about:

  • The basics and terminology.
  • Tracking historical events on models.
  • Attaching dynamic application context to events.
  • Configuring event models.
  • Aggregating events across event models.
  • The Django admin integration.
  • Reverting models to previous versions.
  • A guide on performance and scale.

There's also additional help, FAQ, and troubleshooting guides.

Installation

Install django-pghistory with:

pip3 install django-pghistory

After this, add pghistory and pgtrigger to the INSTALLED_APPS setting of your Django project.

Contributing Guide

For information on setting up django-pghistory for development and contributing changes, view CONTRIBUTING.md.

Creators

Other Contributors

  • @max-muoto
  • @shivananda-sahu
  • @asucrews
  • @Azurency
  • @dracos
  • @adamchainz
  • @eeriksp
  • @pfouque

django-pghistory's People

Contributors

adamchainz avatar dracos avatar eeriksp avatar johanvdw avatar jzmiller1 avatar madtools avatar max-muoto avatar quevon24 avatar shivananda-sahu avatar tomage avatar wesleykendall avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

django-pghistory's Issues

Is there a way to override the name of the db table used for capturing history?

It looks like the event table that gets created defaults to the db table name of ${app_label_model_name}_event. Is there a way to override the name of the table that gets generated?

From a reading of the code , it seems like create_event_model allows for the meta attribute to be passed in - but this option isn't currently exposed in the top level functions (track ?)

A model clashes with field name "event", what to do?

First of all, thanks again for the team to provide such great package. I personally think django-pghistory
is the best history tracking package out there.

My question: there's another 3rd package "django-scheduler" in my app, and it happens to have "event" in its model:

class EventRelation(models.Model):
    event = models.ForeignKey(Event, on_delete=models.CASCADE, verbose_name=_("event"))
    content_type = models.ForeignKey(ContentType, on_delete=models.CASCADE)
    object_id = models.IntegerField(db_index=True)
    content_object = fields.GenericForeignKey("content_type", "object_id")

Here is the tracking:

class OccasionsConfig(AppConfig):
    def ready(self):
        schedule_eventrelation_model = django_apps.get_model("schedule.EventRelation", require_ready=False)
        pghistory.track(
            pghistory.Snapshot('eventrelation.snapshot'),
            app_label='occasions',
        )(schedule_eventrelation_model)

So migration complains:

occasions.EventRelationEvent.pgh_obj: (fields.E302) Reverse accessor for 'occasions.EventRelationEvent.pgh_obj' clashes with field name 'schedule.EventRelation.event'.
	HINT: Rename field 'schedule.EventRelation.event', or add/change a related_name argument to the definition for field 'occasions.EventRelationEvent.pgh_obj'.

Both packages are important and using .event extensively, what should I do? Since "event" is a fairly common word, is it possible for django-pghistory to have more unique method name such as ".pghistory_event()"?

Thanks again for your suggestion

Calls to SET LOCAL quickly blind pg_stat_statements

pghistory uses SET LOCAL to store in the PostgreSQL session a context_id.
While this is perfectly fine per se, it has a terrible side effect on pg_stat_statements and any tool that use this to analyze performance: SET queries are not normalized.
pghistory will thus fill pg_stat_statements with thousands of calls.
Since forever (at least PostgreSQL 8.0, so let's say the beginning of time), there is an alternative function in PostgreSQL that has the exact same effect, but without this possibly nasty side effect, set_config().
I thus suggest switching pghistory to this solution.

Upgrading to version 2

django-pghistory tracks history with triggers that are installed by django-pgtrigger. django-pgtrigger is now integrated with the migration system, so triggers will appear in migrations from now on.

Instructions for upgrading

After upgrading to version 2 of django-pghistory, the majority of people can simply run python manage.py makemigrations to make the migrations for the triggers. If, however, you are tracking third-party models, you will need to register trackers on proxy models. Otherwise trigger migrations will be created outside of your project.

Important - Please be sure you have django-pgtrigger>=4.5 installed, otherwise django-pghistory might be susceptible to some migration-related bugs

For example, this is how you can track changes to Django's user model:

  # Track the user model, excluding the password field
  @pghistory.track(
      pghistory.Snapshot('user.snapshot'),
      exclude=['password'],
  )
  class UserProxy(User):
      class Meta:
          proxy = True

The same syntax is also used for default many-to-many "through" models. For example, this is how one tracks changes to group add/remove events on the user model:

  from django.contrib.auth.models import User

  import pghistory


  # Track add and remove events to user groups
  @pghistory.track(
      pghistory.AfterInsert('group.add'),
      pghistory.BeforeDelete('group.remove'),
      obj_fk=None,
  )
  class UserGroups(User.groups.through):
      class Meta:
          proxy = True

Maintaining legacy behavior

If you want to disable django-pgtrigger integration with migrations entirely, set settings.PGTRIGGER_MIGRATIONS to False. Setting this along with settings.PGTRIGGER_INSTALL_ON_MIGRATE to True will preserve the legacy behavior of how triggers were installed. It is not recommend to do this

Other changes

Along with this, there is no longer a dependency on django-pgconnection. You no longer have to wrap settings.DATABASES with django-pgconnection after upgrading

Issues?

If you have any issues with the upgrade, please comment here and I will try to assist

Migrations cannot be rolled back

Since the triggers are inserted without the use of the migrations system, once the table gets removed, the context tries to reinsert the triggers, which now reference tables which have been removed (at least, that's my analysis):

  Unapplying core.0057_reimbursementclaim_reimbursementclaimhistory_reimbursementremovals... OK
Traceback (most recent call last):
  File ".../site-packages/django/db/backends/utils.py", line 82, in _execute
    return self.cursor.execute(sql)
  File ".../site-packages/pgconnection/core.py", line 85, in execute
    return super().execute(sql, args)
psycopg2.errors.UndefinedTable: relation "core_reimbursementclaim" does not exist
CONTEXT:  SQL statement "CREATE TRIGGER pgtrigger_ReimbursementClaim_updates_insert_3607b
                    AFTER INSERT ON core_reimbursementclaim
                    
                    FOR EACH ROW 
                    EXECUTE PROCEDURE pgtrigger_ReimbursementClaim_updates_insert_3607b()"
PL/pgSQL function inline_code_block line 2 at SQL statement


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "manage.py", line 30, in <module>
    main()
  File "manage.py", line 26, in main
    execute_from_command_line(sys.argv)
  File ".../site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File ".../site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File ".../site-packages/django/core/management/base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File ".../site-packages/django/core/management/base.py", line 371, in execute
    output = self.handle(*args, **options)
  File ".../site-packages/django/core/management/base.py", line 85, in wrapped
    res = handle_func(*args, **kwargs)
  File ".../site-packages/django/core/management/commands/migrate.py", line 268, in handle
    self.verbosity, self.interactive, connection.alias, apps=post_migrate_apps, plan=plan,
  File ".../site-packages/django/core/management/sql.py", line 54, in emit_post_migrate_signal
    **kwargs
  File ".../site-packages/django/dispatch/dispatcher.py", line 179, in send
    for receiver in self._live_receivers(sender)
  File ".../site-packages/django/dispatch/dispatcher.py", line 179, in <listcomp>
    for receiver in self._live_receivers(sender)
  File ".../site-packages/pgtrigger/apps.py", line 9, in install
    pgtrigger.install(database=using)
  File ".../site-packages/pgtrigger/core.py", line 903, in install
    trigger.install(model)
  File ".../site-packages/pgtrigger/core.py", line 657, in install
    cursor.execute(rendered_trigger)
  File ".../site-packages/django/db/backends/utils.py", line 98, in execute
    return super().execute(sql, params)
  File ".../site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File ".../site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File ".../site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File ".../site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File ".../site-packages/django/db/backends/utils.py", line 82, in _execute
    return self.cursor.execute(sql)
  File ".../site-packages/pgconnection/core.py", line 85, in execute
    return super().execute(sql, args)
django.db.utils.ProgrammingError: relation "core_reimbursementclaim" does not exist
CONTEXT:  SQL statement "CREATE TRIGGER pgtrigger_ReimbursementClaim_updates_insert_3607b
                    AFTER INSERT ON core_reimbursementclaim
                    
                    FOR EACH ROW 
                    EXECUTE PROCEDURE pgtrigger_ReimbursementClaim_updates_insert_3607b()"
PL/pgSQL function inline_code_block line 2 at SQL statement

A fix seems to be possible if my analysis is correct, since the post_migrate signal has a "plan" argument:

The migration plan that was used for the migration run. While the plan is not public API, this allows for the rare cases when it is necessary to know the plan. A plan is a list of two-tuples with the first item being the instance of a migration class and the second item showing if the migration was rolled back (True) or applied (False).

If the 2nd member of the tuple is True, we should not reinsert the triggers, since the triggers are cleaned up successfully using cascade:

% djmanage sqlmigrate --backwards core 0057
BEGIN;
--
-- Create model ReimbursementClaimHistory
--
DROP TABLE "core_reimbursementclaimhistory" CASCADE;
--
-- Create model ReimbursementRemovals
--
DROP TABLE "core_reimbursementremovals" CASCADE;
--
-- Create model ReimbursementClaim
--
DROP TABLE "core_reimbursementclaim" CASCADE;
COMMIT;

Reverse setup doesn't install triggers

Docs of get_event_model:

Instead of using pghistory.track, which dynamically generates an event model, one can instead construct a event model themselves, which will also set up event tracking for the original model.

The second part isn't working. The triggers are not installed after migrating (\dft shows none).

Fix docs to mention that pgtrigger must be in INSTALLED_APPS

Currently the installation instructions just say that pghistory needs to be added to the INSTALLED_APPS: https://django-pghistory.readthedocs.io/en/latest/installation.html

But doing only that will not the triggers in the migration, leaving new users scratching their heads why this is happening.

Simple fix is to change the sentence about adding pghistory to INSTALLED_APPS to:

 After this, add `pghistory` and `pgtrigger` to the `INSTALLED_APPS` setting of your Django project.

#39 #46

Allow nested contexts

I'd been wondering if we could nest contexts, so that if you have a complex process happening, say, when a request is processed, it could have the option to "spawn" those children contexts.

Example

Perhaps a small example helps clarify:

def some_api_endpoint(request):
    run_something_very_complex(request.user, request.POST.get('reason'))


def run_something_very_complex(actor, reason):
    pghistory.context(actor_email=actor.email)

    something_slightly_less_complicated(reason)

    if reason == 'Just because':
        another_less_complicated_thing("Unknown")
    else:
        another_less_complicated_thing(f"Reason: {reason}")


def something_slightly_less_complicated(reason):
    pghistory.context(reason=reason)
    ... # do something complicated here ...


def another_less_complicated_thing(reason):
    pghistory.context(reason=reason)
    ... # do something complicated here ...

Of course, in a trivial example, it's probably easy to just refactor it and make the problem go away - but I have a very similar thing happening in our codebase, and it's not always so clear-cut how to deal with it.

In the above, I'm imagining that either of the slightly-less complicated functions might easily be called from other places, so they might each have their own particular reasons for why they add reason to the pghistory.context. And it might be that the reasons are of a slightly different ilk.

Reasoning

I know that one of the main design choices for the context in pghistory was to make it so that you'd only have the one context for the entire request. This is good for example for simplicity's sake, but also most definitely avoids adding too many queries to the process.

For example, if one had a for-loop over 1000 users and called run_something_very_complex() on each and every one, that'd be a lot of contexts saved. Then again... if someone is calling a very complex function 1000's of times, they better find a better way to do that - pehaps batch somehow.

Which brings me to one point we had discussed before. I came across something like this in my codebase:

def do_stuff(users):
    jobs = [create_job_for_user(user, commit=False) for user in users]
   Job.objects.bulk_create(jobs)

def create_job_for_user(user, commit=True):
    pghistory.context(username=user.username)
    job = Job(user=user, ...)
    if commit:
        job.save()

This of course caused issues (as described in #11), because the last user to be looped through was the one that got it's username set in the pghistory.context().

(Of course, this again is a bit trivial - there's no need to actually put the username into the context - you can track it using pghistory.track() on the model - but that's besides the point).

I think that no matter what, it seems unlikely that we can support bulk_operations where each item might add it's own personal kwarg to the (singular) context. This kinda means that if one need to do that, one just can't use Django's bulk operations 🤷‍♂️ .

But - perhaps a very complex function might want to call other complex function, which on their own might also be doing some pghistory tracking that might "clash" either with that originating function, or some of the other complex functions being called in the same process. Perhaps it'd be nice if we could be slightly more granular with the level at which we do create contexts.

So, for a very complex function, pehaps it'll spawn one main context. And perhaps just a few "sub" contexts.

Crude implementation

To test the idea, I did implement a partial solution - so just for reference, here it is (NOTE: there might be a few problems here - I haven't thought too carefully perhaps about threading safety, nor how the Context object gets created - but this was enough to get unit-tests passing, and I figured a proper way would be to implement in the library proper):

@contextlib.contextmanager                                                      
def new_context(*, inherit_from_parent=False, **metadata):                                                    
                                                                                
    _tracker = pghistory.tracking._tracker                                      
    if hasattr(_tracker, 'value'):                                              
        old_context = _tracker.value                                            
        _tracker.value = pghistory.tracking.Context(                            
            id=uuid.uuid4(),                                                    
            metadata={'previous_context_id': str(old_context.id)},              
        )                                                                       
    else:                                                                       
        old_context = None                                                      
                                                                                
    with pghistory.context(**metadata):                                         
        yield                                                                   
                                                                                
    if old_context:                                                             
        _tracker.value = old_context

NOTE: We can indeed track lineage of these contexts.. I.e. we can put a "parent_id" field on the contexts.
We could also make it so that child contexts inherit the metadata from their parents.

Anyway.. thoughts @wesleykendall ?

Question regarding using Retrieving and Joining AggregateEvent Metadata from the doc

Hi team, thanks again for creating such a great package, it's been a pleasure to develop with it. Here is an observation regarding " Retrieving and Joining AggregateEvent Metadata" in the doc.

https://django-pghistory.readthedocs.io/en/latest/extras.html#retrieving-and-joining-aggregateevent-metadata
When I copying the example pointing auth.User, makemigration complains:

SystemCheckError: System check identified some issues:

ERRORS:
users.CustomAggregateEvent.user: (fields.E301) Field defines a relation with the model 'auth.User', which has been swapped out.
	HINT: Update the relation to point at 'settings.AUTH_USER_MODEL'.

Thus is my user model modifying the doc example pointing settings.AUTH_USER_MODEL, which is my User model.

import pghistory
from pghistory.models import BaseAggregateEvent
from django.contrib.auth.models import AbstractUser

class User(AbstractUser):
    name = CharField("Name of User", blank=True, max_length=255)
    #...

class CustomAggregateEvent(BaseAggregateEvent):
    user = models.ForeignKey(User, on_delete=models.DO_NOTHING, null=True)
    url = models.TextField(null=True)

    class Meta:
        managed = False

makemigrations, migrate, createsuperuser and seeding data seems fine, however, it complains ValueError: Must use .target() to target an object for event aggregation in the django shell:

Python 3.9.13 (main, May 18 2022, 02:18:18)
Type 'copyright', 'credits' or 'license' for more information
IPython 8.3.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: UserAggregateEvent.objects.annotate(email=F('user__email'))
Out[1]: ---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
File /usr/local/lib/python3.9/site-packages/IPython/core/formatters.py:707, in PlainTextFormatter.__call__(self, obj)
    700 stream = StringIO()
    701 printer = pretty.RepresentationPrinter(stream, self.verbose,
    702     self.max_width, self.newline,
    703     max_seq_length=self.max_seq_length,
    704     singleton_pprinters=self.singleton_printers,
    705     type_pprinters=self.type_printers,
    706     deferred_pprinters=self.deferred_printers)
--> 707 printer.pretty(obj)
    708 printer.flush()
    709 return stream.getvalue()

File /usr/local/lib/python3.9/site-packages/IPython/lib/pretty.py:410, in RepresentationPrinter.pretty(self, obj)
    407                         return meth(obj, self, cycle)
    408                 if cls is not object \
    409                         and callable(cls.__dict__.get('__repr__')):
--> 410                     return _repr_pprint(obj, self, cycle)
    412     return _default_pprint(obj, self, cycle)
    413 finally:

File /usr/local/lib/python3.9/site-packages/IPython/lib/pretty.py:778, in _repr_pprint(obj, p, cycle)
    776 """A pprint that just redirects to the normal repr function."""
    777 # Find newlines and replace them with p.break_()
--> 778 output = repr(obj)
    779 lines = output.splitlines()
    780 with p.group():

File /usr/local/lib/python3.9/site-packages/django/db/models/query.py:256, in QuerySet.__repr__(self)
    255 def __repr__(self):
--> 256     data = list(self[:REPR_OUTPUT_SIZE + 1])
    257     if len(data) > REPR_OUTPUT_SIZE:
    258         data[-1] = "...(remaining elements truncated)..."

File /usr/local/lib/python3.9/site-packages/django/db/models/query.py:262, in QuerySet.__len__(self)
    261 def __len__(self):
--> 262     self._fetch_all()
    263     return len(self._result_cache)

File /usr/local/lib/python3.9/site-packages/django/db/models/query.py:1324, in QuerySet._fetch_all(self)
   1322 def _fetch_all(self):
   1323     if self._result_cache is None:
-> 1324         self._result_cache = list(self._iterable_class(self))
   1325     if self._prefetch_related_lookups and not self._prefetch_done:
   1326         self._prefetch_related_objects()

File /usr/local/lib/python3.9/site-packages/django/db/models/query.py:51, in ModelIterable.__iter__(self)
     48 compiler = queryset.query.get_compiler(using=db)
     49 # Execute the query. This will also fill compiler.select, klass_info,
     50 # and annotations.
---> 51 results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
     52 select, klass_info, annotation_col_map = (compiler.select, compiler.klass_info,
     53                                           compiler.annotation_col_map)
     54 model_cls = klass_info['model']

File /usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py:1162, in SQLCompiler.execute_sql(self, result_type, chunked_fetch, chunk_size)
   1160 result_type = result_type or NO_RESULTS
   1161 try:
-> 1162     sql, params = self.as_sql()
   1163     if not sql:
   1164         raise EmptyResultSet

File /usr/local/lib/python3.9/site-packages/pghistory/models.py:509, in AggregateEventQueryCompiler.as_sql(self, *args, **kwargs)
    505 base_sql, base_params = super().as_sql(*args, **kwargs)
    507 # Create the CTE that will be queried and insert it into the
    508 # main query
--> 509 cte = self.get_aggregate_event_cte()
    511 return cte + base_sql, base_params

File /usr/local/lib/python3.9/site-packages/pghistory/models.py:480, in AggregateEventQueryCompiler.get_aggregate_event_cte(self)
    478 obj = self.query.target
    479 if not obj:
--> 480     raise ValueError('Must use .target() to target an object for event aggregation')
    482 event_models = self.query.across
    483 cls = self._class_for_target(obj)

ValueError: Must use .target() to target an object for event aggregation

Thanks for any suggestions.

Information for Contributors

Hey everyone, thanks so much for your interest and help with django-pghistory.

I have a Slack group now with a few other people that have helped contribute to these libraries before, and we will be doing a much better job from now on addressing issues so that people don't have to resort to making forks.

Currently I have it on my calendar every week to be sure I have time set aside to go through issues in this repo and others. If you have issues that are preventing you or your organization from using the libraries, please feel free to escalate them to [email protected].

[Question] how do you get django to recognize the dynamically created Event Models during makemigrations?

I don't understand how you do this.

I know the following:

  1. Signal Handling in AppConfig: The class_prepared signal is connected to the pgh_setup function in the __init__ method of the PGHistoryConfig class. This ensures that as soon as a model class is prepared, the function pgh_setup is invoked.

    class_prepared.connect(pgh_setup)
  2. pgh_setup Function: This function checks if the sender model (which is now prepared) has a pghistory_setup method and if so, calls it. The pghistory_setup method is where all the dynamic model creation and setup logic will likely be located.

    def pgh_setup(sender, **kwargs):
        if hasattr(sender, "pghistory_setup"):
            sender.pghistory_setup()
  3. Event Model and pghistory_setup: This method, as a part of the Event model, does the actual setup, including adding triggers, creating the event model, etc. The model is then added to the app registry.

    @classmethod
    def pghistory_setup(cls):
        # Your setup logic here

Then i am lost. I still don't see how Django's makemigrations pick up the dynamically generated class. Because makemigrations is very strict abt looking for models in models.py.

In your docs at https://django-pghistory.readthedocs.io/en/3.0.0/event_tracking/#using-pghistorytrack you stated

Running python manage.py makemigrations will produce migrations for both the triggers and the event model.

Although event models aren't in models.py, they're still imported, migrated, and accessed like a normal model.

me: but how????????

UPDATE

I see that there's a create_event_model method

but you're accessing the tracked_model.__meta.fields and using methods like ._meta.get_field(field_name) how do you ensure that the model is properly loaded and you don't get errors like django.core.exceptions.FieldDoesNotExist

Also how do you prevent the eventmodel from facing clashes of the reverse accessor issue?

Configurable Event Model Suffix

The application I am adding django-pghistory already has a naming convention that utilizes "Event" to represent application level ends. For example we already have both Purchase model and PurchaseEvent model that both need to have change logging applied.

Not only would be slightly less than desirable to have PurchaseEventEvent (We have everal models that use the name Event already). But it also seems that migrations will fail due to a conflict when PurchaseEvent table is created automatically.. unless I change my existing model names which would be an insurmountably difficult thing for us to do atm..

I would propose making the naming convention suffix for event models to be configurable somewhere while setting up the app to allow users to change this to something like Log or Record or really anything just to avoid naming conflicts of this kind.

Multiple trackers with DRF

Making a note here in case someone encounters something similar.

With an out of the box setup for pghistory with the admin integration enabled I am running multiple trackers on a model.

@pghistory.track(
    pghistory.Snapshot(label='ap.snapshot'),
    pghistory.BeforeDelete(label='ap.beforedelete')
)

I have a DRF endpoint using filters that uses my ap model. The endpoint results in the pg history error:

ap has more than one tracker. Use the pgh_event_models dictionary to retrieve the event model by label.

Stemming from rest_framework/filters.py in get_default_valid_fields at line 230:


        model_property_names = [
            # 'pk' is a property added in Django's Model class, however it is valid for ordering.
            attr for attr in dir(model_class) if isinstance(getattr(model_class, attr), property) and attr != 'pk'
        ]

Problem when single-quote exists in context payload

We recently found an issue where if we added a string with a single-quote ' character in string, it would brick the SQL command, presumably because of some lack of escaping for single quotes.

As we already discussed this in a chat @wesleykendall , I believe you are going to see about verifying in a unit-test and look into if this is a trivial fix, or something more substantial. Feel free to reach out for more context, or if you want me/others involved in producing a PR with a fix.

How to track deletion events with snapshot?

Hi there,
It's been great using this great library, especially on the version 2.4, thanks for all your efforts!

Question: how do I track DELETE event along with snapshot?

My Event model definition:

class ModelsEvent(pghistory.get_event_model(
    Model,
    pghistory.Snapshot('model.snapshot'),
)):

This only generates triggers for create and update, but not delete:

pgtrigger.migrations.AddTrigger(
            model_name='model',
            trigger=pgtrigger.compiler.Trigger(name='model_snapshot_insert', .....
),
pgtrigger.migrations.AddTrigger(
            model_name='model',
            trigger=pgtrigger.compiler.Trigger(name='model_snapshot_update'....

Interaction with Django Admin Log to give a richer Model Change History

Seems to me that when this library is installed it could supersede the default Django Admin log functionality (django_admin_log table).

My aim in using this library is for Django history to give a more detailed explanation of what happened, i.e.

DATE/TIME USER ACTION ORIGINAL VALUE CURRENT VALUE
June 18, 2020, 9:04 p.m. stefan Added Country Name - South Africa
June 18, 2020, 9:04 p.m. stefan Added Country Alpha 2 code - ZT
June 18, 2020, 9:04 p.m. stefan Added Country Alpha 3 code - ZAX
June 19, 2020, 9:04 a.m. peter Changed Country Alpha 3 code ZAX ZAF
June 19, 2020, 10:00 a.m. - Changed Country Alpha 2 code ZT ZA

Where in the case the admin directly made a change to the database, bypassing Django then obviously there would be no USER available (hence - above).

I really like that I have access to all of the above information in Country Event table but a bit confused about the final step. Any guidance available? I think this would be a very useful demonstration of the power of this library (if the above is possible or worthwhile using this library instead of another).

Error while changing AppConfig - EventAdmin

I tried to set up a custom App Config for a app called core and it maps an error related to the EventsAdmin

File "C:\Users\Mika\PycharmProjects\palme\palmy\core\celery.py", line 9, in <module> django.setup() File "C:\Users\Mika\anaconda\envs\palme\lib\site-packages\django\__init__.py", line 24, in setup apps.populate(settings.INSTALLED_APPS) File "C:\Users\Mika\anaconda\envs\palme\lib\site-packages\django\apps\registry.py", line 124, in populate app_config.ready() File "C:\Users\Mika\PycharmProjects\palme\palmy\core\apps.py", line 21, in ready from .signals import profile # todo import not working File "C:\Users\Mika\PycharmProjects\palme\palmy\core\signals.py", line 7, in <module> from stocks.tasks import send_email_with_data File "C:\Users\Mika\PycharmProjects\palme\palmy\stocks\tasks.py", line 30, in <module> django.setup() File "C:\Users\Mika\anaconda\envs\palme\lib\site-packages\django\__init__.py", line 24, in setup apps.populate(settings.INSTALLED_APPS) File "C:\Users\Mika\anaconda\envs\palme\lib\site-packages\django\apps\registry.py", line 124, in populate app_config.ready() File "C:\Users\Mika\anaconda\envs\palme\lib\site-packages\pghistory\admin\apps.py", line 12, in ready admin.site.register(config.admin_queryset().model, config.admin_class()) File "C:\Users\Mika\anaconda\envs\palme\lib\site-packages\django\contrib\admin\sites.py", line 132, in register raise AlreadyRegistered(msg) django.contrib.admin.sites.AlreadyRegistered: The model Events is already registered with 'pghistory.EventsAdmin'. '

Doc request: FAQ on backfilling models with existing data

Suggestion: Add an FAQ on how to backfill audit events for models with existing data.

I saw this section in the docs on manual tracking, it helped me to add audit events for the current data I have: https://django-pghistory.readthedocs.io/en/2.7.0/event_tracking.html#manual-tracking

The FAQ could probably just link to that. Might also be worth noting that in the data migration you must refer to the actual model, not the fake model used in migration state, otherwise pghistory won't recognise it (there are no trackers on the fake models).

PS: Thanks for all the hard work on this useful lib 🏆

Trigger function not excluded when using non-postgres database (proposed solution)

(Apologies, I had mistakenly entered this issue in pgtrigger. Copied here since it is related to pghistory)

(Similar issue from pgtrigger: "pgtrigger doesn't play nicely (i.e., ignore) other non postgres dbs" )

In pghistory/migrations/0004_auto_20220906_1625.py, a stored procedure defined in /pghistory/models.py is referenced. There is no check to make sure that the connected database is Postgres. For my local environment, I added a check in the migration file to skip if not connected to postgres. File change below for your reference in case it is useful as a general change:

# Generated by Django 3.2.15 on 2022-09-06 21:25

from django.db import migrations

from pghistory.models import Context


def install_pgh_attach_context_func(apps, schema_editor):
    # skip creating stored function for non-postgres database
    if schema_editor.connection.vendor.startswith("postgres"):
        Context.install_pgh_attach_context_func(using=schema_editor.connection.alias)


class Migration(migrations.Migration):

    dependencies = [
        ("pghistory", "0003_auto_20201023_1636"),
    ]

    operations = [migrations.RunPython(install_pgh_attach_context_func, reverse_code=migrations.RunPython.noop)]

Cannot track Auth permission model: psycopg2.errors.UndefinedFunction

Thanks for providing such a great tool! When I tried to track Django 3.2 built-in auth.Permission model:

from django.apps import apps as django_apps

class UsersConfig(AppConfig):
    def ready(self):
        auth_permission_model = django_apps.get_model("auth.Permission", require_ready=False)
        pghistory.track(
            pghistory.Snapshot('permission.snapshot'),
            model_name='PermissionsHistory',
            related_name='history',
            app_label='users',
        )(auth_permission_model)

End up with the error when migrate:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.9/site-packages/pgconnection/core.py", line 85, in execute
    return super().execute(sql, args)
psycopg2.errors.UndefinedFunction: function _pgh_attach_context() does not exist
LINE 2: ...NEW."id", NOW(), 'permission.snapshot', NEW."id", _pgh_attac...
                                                             ^
HINT:  No function matches the given name and argument types. You might need to add explicit type casts.
QUERY:  INSERT INTO "users_permissionshistory"
                ("name", "content_type_id", "codename", "id", "pgh_created_at", "pgh_label", "pgh_obj_id", "pgh_context_id") VALUES (NEW."name", NEW."content_type_id", NEW."codename", NEW."id", NOW(), 'permission.snapshot', NEW."id", _pgh_attach_context())
CONTEXT:  PL/pgSQL function pgtrigger_permission_snapshot_insert_839b9() line 14 at SQL statement

I am using django-pghistory v1.4.0, Thanks for any suggestions!

multi table inheritance

This isn't so much of a request, but to just log my findings of getting this to work with multi table inheritance.

First, you will need to pass related_name='+' to track() in order to get pgh_obj from having conflicting related names.

Second, we need to limit the fields tracked. One way would be to change pghistory to pass include_parents=False to _meta.get_fields(), but it should also be able to just do this at the track() level as well with a decorator to wrap the pghistory decorator.

Another possibility... but I think it would not be wanted .. is to automatically walk _meta.get_parent_list and decorate the parents. I think it is probably best to make that a manual step in the hierarchy.

And best of all -- avoid multitable inheritance :)

(OH! You can't call get_fields() yet: AppRegistryNotReady ... so still trying to figure out what to so here.)

Admin for MiddlewareEvents doesn't display non-mw events

We need to track changes to a model that might happen both in normal views (so via middleware) and in shell or other operations.
As far as I can tell from the code and the docs, MiddlewareEvents should do just that, but what actually happens is this:
image

That is: the one event that I triggered via the admin is properly displayed; the other is filtered out and only counted.

Is there any workaround for this?

Tracking linked models being assigned/removed

This may not be possible, or I may have missed something obvious :) Say I have two models:

@pghistory.track(pghistory.Snapshot('author.snapshot'))
class Author(models.Model):
    name = models.CharField(max_length=100)

@pghistory.track(pghistory.Snapshot('book.snapshot'))
class Book(models.Model):
    title = models.CharField(max_length=100)
    author = models.ForeignKey(Author)

I have Book A by author AA and Book B by author BB. But it turns out Book B was actually written by AA. So I update the data:

>>> book = Book.objects.get(title='B')
>>> book.author = Author.objects.get(name='AA')
>>> book.save()

If I now look at the history of Book B, what I see is what you would expect:

>>> pghistory.models.AggregateEvent.objects.target(Book.objects.get(title='B')).order_by('pgh_created_at').values()
<AggregateEventQuerySet [
  {'pgh_diff': None,                  'pgh_id': 2, 'pgh_label': 'book.snapshot', 'pgh_data': {'id': 2, 'title': 'B', 'author_id': 2}, ... },
  {'pgh_diff': {'author_id': [2, 1]}, 'pgh_id': 3, 'pgh_label': 'book.snapshot', 'pgh_data': {'id': 2, 'title': 'B', 'author_id': 1}, ... }
]>

The history of Author AA, however is as follows:

>>> for i in pghistory.models.AggregateEvent.objects.target(Author.objects.get(name='AA')).order_by('pgh_created_at').values(): print(i)
{'pgh_id': 1, 'pgh_label': 'author.snapshot', 'pgh_data': {'id': 1, 'name': 'AA'}, 'pgh_diff': None, ... }
{'pgh_id': 1, 'pgh_label': 'book.snapshot', 'pgh_data': {'id': 1, 'title': 'A', 'author_id': 1}, 'pgh_diff': None, ... }
{'pgh_id': 3, 'pgh_label': 'book.snapshot', 'pgh_data': {'id': 2, 'title': 'B', 'author_id': 1}, 'pgh_diff': None, ... }

This does make perfect sense from AA's point of view, the Author B entry is new to it and the docs are clear that pgh_diff only shows the difference "between this event and the previous event on the same object with the same label.", but what ideally I would like is for that last row to include a diff like the one in the Book B history, saying the author ID had changed from 2 to 1. That way I can get a history of Author AA, being able to tell the difference between books being added as new entries, and added as transfers from other authors.

Is there a way to do this that I have missed/ haven't thought of?

save some fields when it's changed

well I need to save 2-3 fields when it changes
Now I have
@pghistory.track( pghistory.BeforeUpdate( ("plate_number_changed"), condition=pgtrigger.Q(old__plate_number__df=pgtrigger.F("new__plate_number")) ), fields=["plate_number",'unit_number' "updated_by"], model_name="TrailerTracker", )
So, I need to add condition if change plate number, save it, if Change unit_number - save it. And save everytime updated_by without condition.
Is it accessible? or Should I need to create new tracks for every field...

dumpdata fails unless pghistory.context is directly referenced

Hi pghistory friends!

I've noticed that when using pghistory, dumpdata fails unless the context model is specifically targetted:

python manage.py dumpdata pghistory  --traceback
  File "/projpath/manage.py", line 27, in <module>
    main()
  File "/projpath/manage.py", line 23, in main
    execute_from_command_line(sys.argv)
  File "/localpath/lib/python3.9/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line
    utility.execute()
  File "/localpath/lib/python3.9/site-packages/django/core/management/__init__.py", line 440, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/localpath/lib/python3.9/site-packages/django/core/management/base.py", line 414, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/localpath/lib/python3.9/site-packages/django/core/management/base.py", line 460, in execute
    output = self.handle(*args, **options)
  File "/localpath/lib/python3.9/site-packages/django/core/management/commands/dumpdata.py", line 265, in handle
    serializers.serialize(
  File "/localpath/lib/python3.9/site-packages/django/core/serializers/__init__.py", line 134, in serialize
    s.serialize(queryset, **options)
  File "/localpath/lib/python3.9/site-packages/django/core/serializers/base.py", line 125, in serialize
    for count, obj in enumerate(queryset, start=1):
  File "/localpath/lib/python3.9/site-packages/django/core/management/commands/dumpdata.py", line 222, in get_objects
    yield from queryset.iterator()
  File "/localpath/lib/python3.9/site-packages/django/db/models/query.py", line 401, in _iterator
    yield from self._iterable_class(
  File "/localpath/lib/python3.9/site-packages/django/db/models/query.py", line 57, in __iter__
    results = compiler.execute_sql(
  File "/localpath/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1348, in execute_sql
    sql, params = self.as_sql()
  File "/localpath/lib/python3.9/site-packages/pghistory/models.py", line 509, in as_sql
    cte = self.get_aggregate_event_cte()
  File "/localpath/lib/python3.9/site-packages/pghistory/models.py", line 480, in get_aggregate_event_cte
    raise ValueError('Must use .target() to target an object for event aggregation')
ValueError: Must use .target() to target an object for event aggregation                     

If you expressly mention the model, it works as expected:

python manage.py dumpdata pghistory.context 
[]

python=~3.9
django=4.0.5
pghistory=1.5.0

Let me know if there are other particulars needed to reproduce.

On behalf of my team that is using pghistory: thank you very much for this wonderful code. You're all legends!

Reduce likelihood of unexpected behavior by avoiding overriding of context keys

The problem

I recently discovered a tricky little bug in our system.

We've set things up so that every request, Celery task or manage.py invocation starts off a pghistory.context, which that particular request/task/manage.py-session will then use to add context to pghistory-tracked objects.

Unfortunately, what I found in a few places was that we were running particular code multiple times in a single request (way), in a way where we'd be overriding a key that was previously set.

Examples

A tivial (and kinda nonsensical) example:

def increase_payout_api_endpoint(request):
    for job in Job.objects.all():
        update_payout(job)

def update_payout(job):
    pghistory.context(old_payout=job.payout)
    job.payout += 10
    job.save()

This is nonsensical in the sense that pghistory is most likely going to track the old value on the payout in a JobEvent table. It's unnecessary really to use pghistory.context() to store the old_payout. But that's not the point.

The point is more that update_payout() might be using pghistory.context() in some more useful way, and might have been written at a time when it was really only called once per request/task. But then someone might have started to call it in a for-loop (for example).

This is unfortunate, since what'll happen is that the last time pghistory.context(old_payout=...) is called, that's the value that's going to end on the context for that whole request, so the same value will be attacked to all jobs.

Another example where this can happen:

def some_api_endpoint(request):
    
    play_musical_note(request.POST.get('musical_note'))
    send_a_small_note(request.POST.get('text_note'))

def play_musical_note(note):
    pghistory.context(note=f"Playing note {note}")
    play(note)  # This might change Model instances tracked by pghistory

def send_a_small_note(note):
    pghistory.context(note=f"Sending small note: {note}")
    send_note(note)  # This might change Model instances tracked by pghistory

Possibly solution

To track down the problematic cases, I monkey-patched pghistory.context with something like the following, and then ran our unit-tests:

orig_context = pghistory.context                                                
                                                                                
class strict_context(orig_context):                                             
    def __init__(self, **metadata):                                             
                                                                                
        _tracker = pghistory.tracking._tracker                                  
        if hasattr(_tracker, 'value'):                                          
            keys_already_set = set(metadata.keys()) & set(                      
                _tracker.value.metadata.keys()                                  
            )                                                                   
            if keys_already_set:                                                
                raise RuntimeError(                                             
                    f'The following keys have already been set in the '         
                    f'pghistory.context: {keys_already_set}'                    
                )                                                               
        super().__init__(**metadata)                                            
                                                                                
pghistory.context = strict_context

While pretty crude, it did break a bunch of unit-tests in my codebase. Most of them were somewhat trivial to fix, tho some were harder (eager execution of Celery tasks meant that many of those broke - I did fix that by hacking in sth that re-news the context for those.. not quite an elegant fix, but enough to get unit-tests to pass). I also needed to adjust the HistoryMiddleware class a bit to not make it re-insert the user again (somewhat trivial).

If we were to introduce something that disallows re-insertion of same key into the context, we could also gently introduce it, either by having a settings.py config to control it's default behavior, and/or by adding in a special flag on the context. I.e. the signature of the __init__() could reads sth as:

def __init__(self, allow_overrides=False, **metadata):
    ...

or

def __init__(self, allow_overrides=None, **metadata):
    allow_overrides = getattr(settings, 'PGHISTORY_ALLOW_OVERRIDES_DEFAULT', False)
    ...

Any thoughts @wesleykendall ?

Configurable context injection mechanisms

When using pghistory.context, variables are inject into the SQL, meaning an additional SQL statement to set variables is prepending before every SQL statement.

The rationale of this original approach was to avoid any extra surprise queries at all costs, especially for users with poor DB latencies. On the other hand, SQL is now prepended with a statement to set a variable, which fills the SQL log and also does add an overhead to every statement (albeit largely negligible)

I'm planning to spin out the SQL injection and variable setting logic into its own library since this approach is used by other libraries. I will make the variable injection configurable in this library, meaning users of pghistory will be able to configure how context (and other variables) are set

Events consolidated into single table?

I'd like to replace my current auditing system, and have been studying your code tonight. In order to change to this package I really need a way to query a single table with ordered events by time.

I've tried figuring out a way to change the Events base class to not be abstract... no success, and it would involve table inheritance. Maybe an external table with content types that point to the actual tables?

Do you have any suggestions as to how this could be done without modifying pghistory?

update primary key data type on tracking tables

currently our primary key data type for tracking tables is set to auto integers. We need to update this to be a big int as we are on pace to exhaust the integers in the near future. I have not seen anywhere in the documentation on how to update this value for existing tables. Is this possible or do we need to create a custom migration to manually set these values?

trigger condition based on pghistory context

I am looking for a way to only execute a pg history event if the pg history context metadata satisfies specific requirements. I have tried adding a condition with variations of the following only to have the migrations fail for obvious reasons. Is there a way to do this exclusion or is this something that is not supported?

condition=pgtrigger.Q(
            pghistory_context__metadata__has_key="url"
        ),

Add attribute to ignore auto_now fields?

We've noticed that sometimes the only fields that have changed are auto_now fields and they're creating lots of event records we don't really care about.

To deal with this, we made a custom tracker that has an extra parameter, ignore_auto_now. When set to true, it ignores changes to models that only affect auto_now fields.

In usage, it looks like this:

@pghistory.track(CustomSnapshot(ignore_auto_now_fields=True))
class Note(models.Model):
    ...

We'd be pleased to do a small PR with this new feature, if the maintainers were interested. Any interest?

BeforeDelete trigger raises IntegrityError on NEW.id (should be OLD.id)

Hi there,

From reading the trigger created for logging a delete event, it looks like it should be referencing OLD.id but is using NEW.id which is set to null on a delete.

My model:

class Branch(models.Model):
occ_version = models.IntegerField(default=0)
name = models.CharField(unique=True, max_length=30, blank=False)
address = models.TextField(max_length=500, blank=True)
manager = models.ForeignKey(User, null=True, on_delete=models.PROTECT)

class Branch_delete_audit(pghistory.get_event_model(Branch, pghistory.BeforeDelete('branch.delete.audit'), related_name='branch_delete', )):
pass

The error:

IntegrityError at /admin/acct/branch/2/delete/
null value in column "pgh_obj_id" violates not-null constraint
DETAIL: Failing row contains (1, 2021-01-21 23:12:21.727031+00, branch.delete.audit, 2, Test office, some address, 2, 1, 0ffd0952-4676-4b16-b3eb-89a76d611e2a, null).
CONTEXT: SQL statement "INSERT INTO "audit_branch_delete"
("occ_version", "name", "address", "manager_id", "id", "pgh_created_at", "pgh_label", "pgh_obj_id", "pgh_context_id") VALUES (OLD."occ_version", OLD."name", OLD."address", OLD."manager_id", OLD."id", NOW(), 'branch.delete.audit', NEW."id", _pgh_attach_context())"
PL/pgSQL function pgtrigger_branch_delete_audit_c2919() line 14 at SQL statement

I believe the delete trigger that was generated by pghistory should read OLD.id when referring to the record that is to be deleted.

Thanks for your help, love your work!

3rd party app objects' history is not viewable in Django Amin

Thanks again for the "Showing Event History in the Django Admin" documentation for easy history browsing in Django Admin.

However, for both django-pghistory 1.5 or 2.4 on Django 3.2, when clicking object history of 3rd party app, the custom history_view is not called and Django admin site shows original object_history.html:

This object doesn’t have a change history. It probably wasn’t added via this admin site.

For example, here is the code to add snapshot of Group model in django-pghistory 2

from django.apps import apps as django_apps

@pghistory.track(
    pghistory.Snapshot('group.snapshot'),
)
class GroupProxy(django_apps.get_model("auth.Group", require_ready=False)):
    class Meta:
        proxy = True

In the database level, the correct history data are stored in the event table, however, it's not accessible in the Django Admin.

Proxying a model that contains an event field

I'm doing some initial testing with django-pghistory.

I have tried to create a proxy class to an existing third-party object, but that fails, because that class already has an event foreign key (and most models will have it, it's an event management system).

auditlog.RoomProxyEvent.pgh_obj: (fields.E302) Reverse accessor for 'auditlog.RoomProxyEvent.pgh_obj' clashes with field name 'auditlog.RoomProxy.event'.
	HINT: Rename field 'auditlog.RoomProxy.event', or add/change a related_name argument to the definition for field 'auditlog.RoomProxyEvent.pgh_obj'.

I wonder if there is an easy solution for this. I can live with the fact that the event key is not logged.

I tried

 @pghistory.track(
         pghistory.Snapshot(), exclude=["event"])
 class RoomProxy(Room):
     class Meta:
         proxy=True

but that does not work.

Toggleable context injection

This is one of those late night questions so forgive me if I'm missing something obvious. I'm getting an error of this type:

TransactionManagementError at /admin/djstripe/apikey/add/
An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block.

The pghistory context injection is included in the traceback and I was curious if there was an easy way to disable the context injection or pghistory in general?

/usr/local/lib/python3.10/site-packages/pghistory/runtime.py, line 51, in _inject_history_context is going off before self.db.validate_no_broken_transaction() in /django/db/backends/utils.py

I attempted following the suggestion in https://django-pghistory.readthedocs.io/en/2.5.1/performance.html about setting settings.PGHISTORY_CONTEXT_FIELD to None but bumped into some issues on that path as well:

File "/usr/local/lib/python3.10/site-packages/pghistory/core.py", line 656, in _model_wrapper
  create_event_model(
File "/usr/local/lib/python3.10/site-packages/pghistory/core.py", line 516, in create_event_model
  context_field = _get_context_field(context_field=context_field, context_fk=context_fk)
File "/usr/local/lib/python3.10/site-packages/pghistory/core.py", line 399, in _get_context_field
  context_field = config.context_field()
File "/usr/local/lib/python3.10/site-packages/pghistory/config.py", line 70, in context_field
  assert isinstance(context_field, (ContextForeignKey, ContextJSONField))

Prevent database transaction when BeforeUpdate/Delete raises Exception

Would there be a way to disable final database transaction if e.g. the BeforeUpdate/Delete raises an exception?

This would be a nice way to use the 'BeforeXXX' events as kind of a request of a user to ask for changes that need to be approved by another user.
If these Events had a 'needApprove' property, nice request/grant workflows could be created with this package. An approval could then apply the BeforeXXX event to the database.

Many thanks for your opinion and reply.

RemovedInDjango41Warning: 'pghistory' defines default_app_config = 'pghistory.apps.PGHistoryConfig'.

I am getting this warning while running pytest.

RemovedInDjango41Warning: 'pghistory' defines default_app_config = 'pghistory.apps.PGHistoryConfig'. Django now detects this configuration automatically. You can remove default_app_config.
    app_config = AppConfig.create(entry)

Also, Pytest that are using database_sync_to_async throwing integrity error.

Versions:
django-pgconnection==1.0.2
django-pghistory==1.4.0
django-pgtrigger==2.4.1
django==3.2.7

pghistory adds NEW data not OLD, requiring a total database copy for robust backups

Until I looked under the hood, my mental model for pghistory was that it saved the OLD value whenever a row got an UPDATE or DELETE. That way, you could always revert.

It turns out that it saves a copy of the NEW value instead, whenever the row gets an UPDATE or INSERT.

This seems to work OK, but it seems to leave a big pathway for major problems:

  1. Create a model, add data to it. (INSERT)

  2. Add pghistory to track the model.

  3. Change the data. (UPDATE new data in the model; INSERT copy of new data into the event table)

  4. Realize the change was bad; try to revert.

  5. 💥 You can't revert. The original is gone. The event table holds the NEW data, not the OLD. 💥

Maybe I'm missing something, but if you don't have pghistory enabled from the beginning of a project, you can't revert to versions of data that existed when you started using pghistory. Is that right?

I think there are two solutions to this:

  1. Save the OLD value to the event table, not the new, whenever something is UPDATED or DELETED.

  2. Run some sort of script to save copies of the current data whenever you add pghistory to a model. This isn't great if you have a very big database like we do, since it means you have to copy the whole thing on the off chance a row is changed.

Either way, I looked through the docs a bit and I didn't see any warnings about this. It feels like it should be a big, loud warning in a couple places (FAQ, Homepage, etc.), or maybe an option when setting up triggers.

Is there a reason for the current design? Sorry if I missed something in the docs and thank you again for the great project!

django.db.utils.NotSupportedError: cannot alter type of a column used in a trigger definition

Hi,

When creating a new django migration on a model registered with pghistory, I get this error: django.db.utils.NotSupportedError: cannot alter type of a column used in a trigger definition

Here's the full error stack:

poetry run python ./src/backend/manage.py migrate 
Operations to perform:
  Apply all migrations: admin, auth, contact, contenttypes, easy_thumbnails, energy, heritage, history, invoicing, my-user_geo, knox, markets, notification, parameters, pghistory, sessions, sites, weather
Running migrations:
  Applying heritage.0051_auto_20220701_1510...Traceback (most recent call last):
  File "/home/my-user/workspace/my-project/.venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/home/my-user/workspace/my-project/.venv/lib/python3.10/site-packages/pgconnection/core.py", line 85, in execute
    return super().execute(sql, args)
psycopg2.errors.FeatureNotSupported: cannot alter type of a column used in a trigger definition
DETAIL:  trigger pgtrigger_building_snapshot_update_162c2 on table heritage_building depends on column "building_owner"


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/my-user/workspace/my-project/./src/backend/manage.py", line 58, in <module>
    execute_from_command_line(sys.argv)
  File "/home/my-user/workspace/my-project/.venv/lib/python3.10/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/home/my-user/workspace/my-project/.venv/lib/python3.10/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/my-user/workspace/my-project/.venv/lib/python3.10/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/my-user/workspace/my-project/.venv/lib/python3.10/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/home/my-user/workspace/my-project/.venv/lib/python3.10/site-packages/django/core/management/base.py", line 89, in wrapped
    res = handle_func(*args, **kwargs)
  File "/home/my-user/workspace/my-project/.venv/lib/python3.10/site-packages/django/core/management/commands/migrate.py", line 244, in handle
    post_migrate_state = executor.migrate(
  File "/home/my-user/workspace/my-project/.venv/lib/python3.10/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/home/my-user/workspace/my-project/.venv/lib/python3.10/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/home/my-user/workspace/my-project/.venv/lib/python3.10/site-packages/django/db/migrations/executor.py", line 227, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/home/my-user/workspace/my-project/.venv/lib/python3.10/site-packages/django/db/migrations/migration.py", line 126, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/home/my-user/workspace/my-project/.venv/lib/python3.10/site-packages/django/db/migrations/operations/fields.py", line 244, in database_forwards
    schema_editor.alter_field(from_model, from_field, to_field)
  File "/home/my-user/workspace/my-project/.venv/lib/python3.10/site-packages/django/db/backends/base/schema.py", line 608, in alter_field
    self._alter_field(model, old_field, new_field, old_type, new_type,
  File "/home/my-user/workspace/my-project/.venv/lib/python3.10/site-packages/django/db/backends/postgresql/schema.py", line 196, in _alter_field
    super()._alter_field(
  File "/home/my-user/workspace/my-project/.venv/lib/python3.10/site-packages/django/db/backends/base/schema.py", line 765, in _alter_field
    self.execute(
  File "/home/my-user/workspace/my-project/.venv/lib/python3.10/site-packages/django/db/backends/base/schema.py", line 145, in execute
    cursor.execute(sql, params)
  File "/home/my-user/workspace/my-project/.venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 98, in execute
    return super().execute(sql, params)
  File "/home/my-user/workspace/my-project/.venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/home/my-user/workspace/my-project/.venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/home/my-user/workspace/my-project/.venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 79, in _execute
    with self.db.wrap_database_errors:
  File "/home/my-user/workspace/my-project/.venv/lib/python3.10/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/my-user/workspace/my-project/.venv/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/home/my-user/workspace/my-project/.venv/lib/python3.10/site-packages/pgconnection/core.py", line 85, in execute
    return super().execute(sql, args)
django.db.utils.NotSupportedError: cannot alter type of a column used in a trigger definition
DETAIL:  trigger pgtrigger_building_snapshot_update_162c2 on table heritage_building depends on column "building_owner"
make: *** [Makefile:68: command] Error 1

I think it can be reproduced using the following steps :

  • Create a model and its migrations

Exemple

class Building(models.Model):
    building_owner = models.CharField(
        _("Owner"),
        max_length=300,
        blank=True,
    )
  • Register it with pghistory and create the migration:
@pghistory.track(pghistory.Snapshot("building.snapshot"))
class Building(models.Model):
    building_owner = models.CharField(
        _("Owner"),
        max_length=300,
        blank=True,
    )
  • Change the size value of a field, create a new migration:
@pghistory.track(pghistory.Snapshot("building.snapshot"))
class Building(models.Model):
    building_owner = models.CharField(
        _("Owner"),
        max_length=350,
        blank=True,
    )
  • When applying the migration I get the error:
django.db.utils.NotSupportedError: cannot alter type of a column used in a trigger definition
DETAIL:  trigger pgtrigger_building_snapshot_update_162c2 on table heritage_building depends on column "building_owner"

If I manually remove the trigger on the db and run the migration, it works and recreate the trigger.

Did I missed something ?
Do you have an idea on how we can fix this issue ?

Error: triggers not created

I have installed the package as described here and added the annotation to my model like this:

@pghistory.track(
    pghistory.Snapshot('foo.snapshot')
)
class Foo(models.Model):
    ...

After running makemigrations and migrate a new table for storing history is created in the database.
However, no triggers are set up and changes made in Foo will not be recorded in the history table.

I verified that triggers are not set up by the following query:

FROM information_schema.triggers  
GROUP BY table_name , trigger_name 
ORDER BY table_name ,trigger_name;

I also tried explicitly creating the history table:

class FooHistory(
    pghistory.get_event_model(
        Foo,
        pghistory.Snapshot("foo.snapshot"),
    )
):
    class Meta:
        db_table = "foo_history"

That yielded the exact same result: the table is created as expected, but no triggers are set up.

How to fix that?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.