brentozarultd / soddi Goto Github PK

In v1.4 and v1.5. More info later.

In the app, if you click Help/About, everything's outdated. (Sky passed away, his web site is no longer available, Jeremiah took his mirror of the project down, and it's probably not .NET 3.5 SP1.) Let's just point people to the Github repo for now to read the readme there:

https://github.com/BrentOzarULTD/soddi

Novice users may find it difficult to craft their own ADO.NET connection string. For convenience, let's add Microsoft's DataConnectionDialog.

Please provide a zip only variant of the binaries. :-)

Start SODDI, check that box, import data, close the app, and open it back up again. The "Set Ids as Identity Fields" checkbox doesn't stay checked.

(Using v1.4.)

In SQL Server:

• RevisionGUID - is currently a char(36), should be a uniqueidentifier
• Comment - is currently an ntext, should be nvarchar(max)
• Text - is currently an ntext, should be nvarchar(max)

I am using latest version of SODDI (1.5) with latest StackOverflow dumps (122018). Import activity fails everytime for Posts.XML dump file after an approx execution time of 1 hour. Other other dumps, it successfully imports. I tried switching various values of BatchSize but it did not help.

I have verified, remote query timeout is set to 0 on SQL Server side.

In current exports:
https://archive.org/details/stackexchange

Fields:
https://meta.stackexchange.com/questions/2677/database-schema-documentation-for-the-public-data-dump-and-sede

When trying to import the latest Stackoverflow data files using SODDI v1.5, to MySQL 5.7, the Users table import fails with the error:
Unknown column AccountId in field list

Recent dump 122018 is of large size. So to make XML data import quick, I added multiple Data Files in New filegroup 'DATA' & marked it as Default. Still, the tool SODDI created all the tables in PRIMARY filegroup.

Could it be modified to create tables in DEFAULT filegroup rather then on PRIMARY.

In current exports:
https://archive.org/details/stackexchange

Fields:
https://meta.stackexchange.com/questions/2677/database-schema-documentation-for-the-public-data-dump-and-sede

In current exports:
https://archive.org/details/stackexchange

Fields:
https://meta.stackexchange.com/questions/2677/database-schema-documentation-for-the-public-data-dump-and-sede

If your table's row count is smaller than the batch size setting, it'll show 0 for "Count" even when rows are successfully imported.

Repro steps:

1. Set the "Batch Size" count higher than the smallest table's rows (the default of 5000 is fine for dba.stackexchange.com's Tags table)
2. Import a relatively small site, like dba.stackexchange.com
3. When it finishes, the "Count" column shows 0 for Tags. I'm guessing the Count is only updated each time a batch finishes. (Which is fine, we don't need an exact row count.)

Just adding this in Issues so we know we have a known issue.

Right now, every table has an Id field. It's not defined as an identity because in theory, users shouldn't be adding new rows to the Stack Overflow database - that just wasn't a goal for the export.

Unfortunately, with identity fields, you can't load them in parallel. To explicitly insert data into identity fields, you have to turn on identity_insert - but that can only be turned on for one table at a time:

https://docs.microsoft.com/en-us/sql/t-sql/statements/set-identity-insert-transact-sql

So we shouldn't do this by default across the board (because then everybody's loads will go slower).

Instead:

• Add a checkbox on the main UI to "Set Ids as Identity Fields"
• During table creations, define the Id field as an IDENTITY(1,1)
• Load tables one at a time, using SET IDENTITY_INSERT ON/OFF

Tried setting the Batch Size to 500,000 (no comma), but still went through 5,000 rows at a time.

(Using v1.4)

When opening the solution file in VS2017 for the first time there is a warning that the setup project SODDI_Setup is not supported. The way forward is unclear. A note needs to be added to provide an easier path for novice programmers that would like to complete How to Think Like the SQL Server Engine.