data-solution-automation-engine / team Goto Github PK

The creation of segments (business concepts) and PSA / STG groupings is not working due to underlying repository changes. Code has been updated to at least generate a working model, but left at that for the time being.

It was possible to update earlier version of metadata. This is now prevented by using a check to compare the current version and the highest version.

• Remove settings from the settings form
• Remove settings from being created and updated in the config file
• Rename the config file to a TEAM name (still has VEDW references)

Rudimentary handling is included, but needs more rounding and testing.

NOTE: FAML not ready at the moment. Conversations with Factil continue based on TEAM sample set.

When configuring the metadata repository database connection, it will always return an error. When the method TestConnections() is getting called the connection string is not set with the newly set values but still contains the default values. Is there an UPDATE button missing or something?

Using Virtual mode, the metadata activation checks the snapshot of the physical model. But, in Physical mode the tool checks the database / catalog for this. If no tables are present in the database for these lookups it creates an unhandled exception.

Issue with File.WriteAllText. The datatable correctly captures each change, but the IO command doesn't always trigger a change if changes are made rapidly (within second). Only happens with JSON for this reason.

Need redevelopment to move all JSON file writes to a single method so IO speeds can be throttled / wait states can be added.

Current source-to-target mappings work only for STG / PSA to INT mappings.

• Order information in datagrid
• Order information when saved in Json

This makes it easier to merge versions in source control.

The Filter Criterion is lost if we re-order the TableGrid by any other Column

The Show Error Log button is obsolete, and replaced by the event log.

The following should be done:

• Remove the Show Error Log button from the feedback screens
• Ensure that meaningful messages are also visible in the event log

TEAM Activation Metadata ends up getting confused when you have the same source, same business key but populating two different HUBs.

• Create event handling to directly update mapping metadata.

• Source tables will fail the prefix/suffix check
• STG or PSA as a target fail the 'target' check as this looks at the Integration Layer

Solution is to split out the check by table type / load vector instead of just 'source' and 'target'. This can be done by extending the method to accept the table type. All source and target columns can be handled this way.

Add BDV and DIM components to sample metadata generation, in different schemas. Extending the metadata to cover more use-cases.

I found out why in some cases a value isn't saved via the TEAM application. It's basically because upon saving a new key is generated which isn't fed back to memory (in the datagrid). So, when making another change to the same record while staying in the screen will cause the change to be lost.

I did end-to-end tests yesterday and this is all I could find. The fix is to re-bind the data to the grid after saving I think.

Ensure that all listed Attributes in Attribute Mapping exists in Source and Target.

Add validation rule to ensure schema settings in the connections are checked in the table mappings. If the schema is set, only tables / data objects in that schema should be checked.

Currently all samples require a SQL Server back-end. The samples are directly created in the database. Creating samples for the JSON would be helpful as well, as currently this only creates a dummy row.

Simplify repository by eliminating MD schema (only use INTERFACE schema / views) as repository.

Related to folder: Files
Related to file: Sample_EDW_1_Source_DML.SQL

Order of the insert statements doesn't match the foreign key relation.
Inserts for Customer_personal table should be on top.

Hello Roelant,

Happy New Year!
I was re-generating some packages today, and I think something changed in the TEAM Activation.
The column OMD_HASH_FULL_RECORD is being mapped as an ATTRIBUTE, which I think it shouldn’t.

Issue: wrong table source name in generated info statement
vedw version : 1.3.0

generated info statement for PROFILER PLAN:

-- PSA Insert Into statement for HSTG_PROFILER_PLAN
-- Generated at 31-1-2018 9:01:37

USE [EDW_100_Staging_Area]
GO

INSERT INTO [EDW_150_History_Area].[dbo].[HSTG_PROFILER_PLAN]
(
[ETL_INSERT_RUN_ID],
[LOAD_DATETIME],
[EVENT_DATETIME],
[RECORD_SOURCE],
[SOURCE_ROW_ID],
[CDC_OPERATION],
[HASH_FULL_RECORD],
[PLAN_CODE],
[PLAN_DESC],
[RENEWAL_PLAN_CODE]
)
SELECT
-1 AS ETL_INSERT_RUN_ID,
[LOAD_DATETIME],
[EVENT_DATETIME],
[RECORD_SOURCE],
[SOURCE_ROW_ID],
[CDC_OPERATION],
[HASH_FULL_RECORD],
[PLAN_CODE],
[PLAN_DESC],
[RENEWAL_PLAN_CODE]
FROM HSTG_PROFILER_PLAN

Must be from STG_PROFILER_PLAN

OMD_CURRENT_RECORD_INDICATOR is being listed as a SOURCE COLUMN to a SAT, even when the SOURCE doesn't contain this column

• Create new form, move settings there
• Ensure textboxes (values) are still accessible from other forms (they are used dynamically. OR, find a new way to centrally manage settings (repository, file etc.).

Add delete context menu in the gridviews (table and attribute mappings). Currently there is only an export-to-JSON function. Adding delete helps managing the grid, although the del key works also.

There are still a few leftover MessageBox calls in the code to catch errors, which all should be removed as it's not really good user experience.

https://github.com/RoelantVos/TEAM/blob/07009eea5e04fc7c9ebead19d296b15edeadca7e/TEAM/Form_Main.cs#L116

This should be connPsa.

Previous versions of TEAM had different logic for handling database lookups verus using the datagrid for the physical model (internal metadata). Any differences are now evaluated at the start, so the same query can be used - leaving only a single query for each step as opposed to two previously.

INFORMATION_SCHEMA is to be replaced by sys.columns etc.

In the validation of the Link Key order the business keys need to be selected, and for this all non-business key attributes need to be removed from the selection.

Currently this is done by using the ordinal position, which is not reliable (or conceptually correct) and only supports the setup where the business keys are last in the order.

Example:

                // Select only the business keys in a link table. 
                // Excluding all non-business key attributes
                workingTable = physicalModelDataTable
                    .Select("TABLE_NAME LIKE '" + FormBase.TeamConfigurationSettings.LinkTablePrefixValue +
                            "_%' AND TABLE_NAME = '" + validationObject.Item2 + "' AND **ORDINAL_POSITION > 4**",
                        "ORDINAL_POSITION ASC").CopyToDataTable();

This is only allowed in integration layer mappings.

Using the last version downloaded yesterday, there was a null reference exception with the save of the automation meta data.

An unhandled exception of type 'System.NullReferenceException' occurred in TEAM.exe
Additional information: Object reference not set to an instance of an object.

On code;
DataTable dataTableKeyChanges = ((DataTable)_bindingSourceTableMetadata.DataSource).GetChanges();

Currently VDW reads from the INTERFACE views to aquire metadata. Ideally, TEAM would export the files already in the format that VDW would need it.

Issue in saving the output path in settings:
Version 1.3.0

In settings one can change the output path.
after change, save, exit and reopening the program the default path is back again:

C:\Program Files (x86)\Virtual Enterprise Data Warehouse\Output\

The change output path should be saved.

Some work to do to remove the ORDINAL POSITION for the metadata activation, as well as making this work against the reverse-engineered metadata as well. Current fix only for default sample setup. This applies to the activation of the metadata into the MD_HUB_LINK_XREF table.

This option is misleading, because the user might enter the same information in both and doesn't know that the needs to create a LinkedServer in his SQL Server.
Maybe a quick check if the Servers are the same it will simply ignore, or maybe a TipTool?