Create a SQLite database from URL in append about sqlitebiter HOT 2 CLOSED

@aborruso
I appreciate if you find it useful.

sqlitebiter using "-a" creates one db and 237 tables. Is it possible to append every source table to only one destination table (the structure is the same)?

I had added an experimental feature (--convert-config option) at sqlitebiter 0.26.0 to enables the above.
You could append every source table to only one destination table by the below command and the config file:

$ sqlitebiter -a --convert-config convert_config.json url https://si.siciliafse1420.it/elencobeneficiari?page=N

convert_config.json is as follows:

{
    "url" : [
        {
            "target_url": "https://si.siciliafse1420.it/elencobeneficiari?page=",
            "rules": {
                "P.IVA/CF" : {
                    "type hint": "text"
                }
            }
        }
    ]
}

I tested https://si.siciliafse1420.it/elencobeneficiari?page=1 to ...?page=10 can be converted to one destination table.

without the configuration file, sqlitebiter detects column data types from actual data and create(or append) a table.
If a detected table schema does not match with the existing table, sqlitebiter will create another table with a different table name even if -a option specified.

For example:
https://si.siciliafse1420.it/elencobeneficiari?page=1
https://si.siciliafse1420.it/elencobeneficiari?page=2
these pages detected different schema as follows.

page=1

page=2

In this case, page=1 P.IVA/CF column type is TEXT (because include value other than numbers such as "-"), while page=2 P.IVA/CF column type is INTEGER. So, considered as different schema.

from sqlitebiter.

Hi @thombashi first of all thank you very much.

It works!!!

I'm opening a new related feature request

from sqlitebiter.