Giter Club home page Giter Club logo

Comments (6)

simonw avatar simonw commented on June 5, 2024

I applied this change:

diff --git a/datasette/utils/internal_db.py b/datasette/utils/internal_db.py
index dbfcceb4..6a8c1eb8 100644
--- a/datasette/utils/internal_db.py
+++ b/datasette/utils/internal_db.py
@@ -66,9 +66,11 @@ async def init_internal_db(db):
 
 
 async def populate_schema_tables(internal_db, db):
+    print("populate_schema_tables")
     database_name = db.name
 
     def delete_everything(conn):
+        print("  delete_everything")
         conn.execute(
             "DELETE FROM catalog_tables WHERE database_name = ?", [database_name]
         )

And confirmed that the populate_schema_tables() function is called once at startup and then only after a schema change has been made.

from datasette.

simonw avatar simonw commented on June 5, 2024

Which lead me to this piece of code:

datasette/datasette/app.py

Lines 486 to 505 in 86335dc

for database_name, db in self.databases.items():
schema_version = (await db.execute("PRAGMA schema_version")).first()[0]
# Compare schema versions to see if we should skip it
if schema_version == current_schema_versions.get(database_name):
continue
placeholders = "(?, ?, ?, ?)"
values = [database_name, str(db.path), db.is_memory, schema_version]
if db.path is None:
placeholders = "(?, null, ?, ?)"
values = [database_name, db.is_memory, schema_version]
await internal_db.execute_write(
"""
INSERT OR REPLACE INTO catalog_databases (database_name, path, is_memory, schema_version)
VALUES {}
""".format(
placeholders
),
values,
)
await populate_schema_tables(internal_db, db)

I thought for a moment that the INSERT OR REPLACE INTO there meant there would be constant changes to the internal.db database, but actually that looks like it only runs if the schema version has changed.

from datasette.

simonw avatar simonw commented on June 5, 2024

To help investigate I upgraded this plugin:

It doesn't seem to be showing the _internal database being upgraded much though - it's at version 36 and sticking there.

from datasette.

simonw avatar simonw commented on June 5, 2024

Added this debug code:

diff --git a/datasette/app.py b/datasette/app.py
index 8591af6a..ea61ba75 100644
--- a/datasette/app.py
+++ b/datasette/app.py
@@ -486,8 +486,12 @@ class Datasette:
         for database_name, db in self.databases.items():
             schema_version = (await db.execute("PRAGMA schema_version")).first()[0]
             # Compare schema versions to see if we should skip it
+            print("schema_version = {}, current_schema_version = {}".format(
+                schema_version, current_schema_versions.get(database_name)
+            ))
             if schema_version == current_schema_versions.get(database_name):
                 continue
+            print("  Did not skip, gonna INSERT OR REPLACE INTO")
             placeholders = "(?, ?, ?, ?)"
             values = [database_name, str(db.path), db.is_memory, schema_version]
             if db.path is None:

And it looks like that INSERT OR REPLACE INTO is only run when the schema changes, so it shouldn't be running very often at all.

from datasette.

simonw avatar simonw commented on June 5, 2024

I'm not at all sure that internal.db is being updated often - I think this bug I'm seeing may relate to a specific configuration of the way I'm running it in Datasette Cloud.

I'll try one last thing: I'm going to run datasette --internal internal.db locally with some plugins and have a watch command that checks the MD5 hash of the database to see if it changed.

from datasette.

simonw avatar simonw commented on June 5, 2024

I ran watch md5sum internal.db and fired up Datasette, then messed around with inserting rows and altering the schema.

The MD5 of that file only updated after I altered the table schemas, which is as it should be.

So this isn't a Datasette bug after all.

from datasette.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.