Metadata Interchange in DMExpress 6.5
As I leave “Vegas for kids” (aka Orlando), I realize that I have seen the light. That light is metadata or, more specifically, metadata interchange. Metadata interchange is critical for our customers and for Syncsort. As the well-known television commercial says, “this changes everything.”
In my previous post on DMExpress 6.5, I talked about this concept of “snap”. I want to add some more detail about this here.
The first place to start a DI Acceleration project, or any DI/ETL project for that matter, is to understand the source systems being read and the target system(s) to write. The metadata Interchange will allow users to import the source and target schemas from DI/ETL tools (Informatica PowerCenter, DataStage, etc.), databases, files, COBOL Copybooks (really ?!?), etc. This will allow users to match the metadata in a DI Acceleration scenario. In DMExpress, this corresponds to individual sources and targets (file, pipe, database table, table in memory, etc.) and their associated metadata (delimited field layout, fixed position field layouts, XML schemas, extracted database columns, COBOL copybooks, etc.). Pretty cool so far!
Enter transformations (fanfare please).
To “jump start” a DI Acceleration project, metadata Interchange will provide the import of transformations performed in an ETL tool into DMExpress. This will assist in the development of the DI/ETL into DMExpress. Metadata Interchange will import the transformations by constructing DMExpress Source/Sort/Copy/Aggregate/Join/etc. task(s) and then connect these tasks in a DMExpress job to perform the equivalent processing. Will we get it 100%? No. As the saying goes, only death and taxes are a 100% certainty. However, when you watch this working in our labs…it’s amazing!
Probably the most important piece of metadata interchange is to maintain the data lineage. Whether users are using DMExpress as a performance augmentation to other DI/ETL tools, or using DMExpress for all ETL, users do not and cannot lose the data lineage for reporting and compliance purposes. Metadata interchange will provide this data lineage to understand and maintain the data origination, how it was transformed, and allows export of it via the bridge.
Syncsort is using the same “Switzerland” of metadata vendors that almost every other data vendor (DI, BI, modeling, etc.) uses in the industry. As one of the top DI analysts said to us recently, this is “a really big deal” for Syncsort and we believe it makes DI acceleration that much more compelling and credible for the market.