November 2011

Post by Steven Totman

Data Integration Business Unit Executive

I was on the underground in London last week on my way back from visiting a financial services customer when I heard a couple of well dressed gents carrying brollies (is it only us Brits that leave the house assuming it will rain no matter how nice it is outside?) musing over the old adage that the only thing you can count on is taxes, death and trouble (as captured in this Marvin Gaye song).

Their conversation got me thinking that instead of trouble, there is actually another thing you can rely on today – that data is only going to get bigger.  I would argue that the amount of useful information to be gleaned from this data is not growing at the same exponential rate. However, regardless of whether you consider your data ‘Big Data’ or not, you actually have to do a lot more “work” to your data as it grows to get business relevant and valuable information from it.

A good example of this is close at heart to those of us impacted by the Eurozone (I’m intentionally avoiding the long debate as to if the UK is actually a member given we’ve kept our own currency but are paying to support the euro). The financial crisis worldwide caused the rapid acceleration of new regulations and controls on markets and companies. In Europe, we already had Solvency II and Basel I, Basel II and now Basel III. These regulations are getting incredibly complex.

Calculations on “extreme” data volumes are required to remain compliant and keep senior executives from going to jail. In this case, picking the right ETL tool can be like receiving a “get out of jail free” card in Monopoly.

So why are the calculations required so complex? For starters, here in Europe we love them as evidenced by European Commission regulation (EC) 2257/94 which states – bananas must be “free from malformation of abnormal curvature.”  In the case of “extra class” bananas, there is no wiggle room but “class 1” bananas can have “slight defects of shape” while “class 2” bananas can have full-on “defects of shape.” Yes, that’s right. We have regulations about the shape and curvature of bananas and don’t even get me started on cucumbers (Commission Regulation (EEC) No 1677/88), where “class I” and “extra class” cucumbers are allowed a bend of 10mm per 10cm of length. Class II cucumbers can bend twice as much. So you can imagine how detailed our calculations must be for something like risk!

About 2 years ago, I was heavily involved with a very smart team working on industry models. To keep up with them, I decided I had to read and understand the Basel II regulations. All I will say is that whenever someone mentions they are working on a Basel project, it brings back horrible memories. I remember it being 4 a.m. on the first day of my “reading project” when I realised my brain hurt and that the scroll bar on the document didn’t look like it had moved. Tying this back to data integration, the point is that it’s definitely not just the volume of data that causes the problems for customers. More often than not, it’s the complexity of calculations or transforms they are dealing with.

Often when I’m speaking with people about data integration acceleration (a good example was the bank I visited earlier this week), they will respond that “our data isn’t really that big.” When pressed on how long it takes them to process their data and whether this satisfies the business, people usually pause and you can see the wheels turning in their head. This is regularly followed by an admission that they are in fact exceeding their service level agreements. The next question is to ask them how much data growth they are seeing and are they prepared for it. After an even longer pause, something like “we plan for 20 percent growth” (a commonly accepted average). However, I’ve heard numerous companies admit that actual data growth could range from 10 percent all the way up to 600 percent! But no one ever says their data isn’t growing. Inevitably, the conversation ends up focusing on how much time they spend tuning their existing environment, how much hardware they are buying, and how they have no better option than to push transformations into the database.

It is always a bit amusing and always very satisfying when the same people who were saying they don’t have ‘Big Data’ are suddenly advocating for why data integration acceleration is needed and makes a lot of sense. Instead of reminding them of what they said in the first place, I simply smile and mention the amount of money they will be able to save from it, as well.

Perhaps I should revisit the title of my post. Three things that are guaranteed are death, taxes and data breaking your data integration infrastructure. If you are already using DMExpress, you can forget about the last one since we have you covered. Everyone else, you are invited to have one less thing to worry about. The whole death and taxes things…we are sorry but can’t help there!

{ 3 comments }