When we announced our DMExpress Hadoop offering, we shared a set of results from benchmark testing that had been completed. Testing has continued since, and I wanted to dedicate this post to sharing some of those results.

We did a series of tests that distill down to:

• TeraSort benchmark (if you’re not familiar with this benchmark, it is worthwhile to search it online)
• Aggregation based on TPC-H generated data (aggregated on order id for line item data)

We varied two things in the tests:

• Compression in the shuffle step: no compression, GZIP
• Data volume: We ramped up to 4TB on the TeraSort and 600GB on the Aggregation

The tests were done on a 10-node cluster running CDH3u2 (Apache 0.20.2).

The results were very interesting, but not surprising.  For TeraSort:

• No compression:
  • While DMExpress was faster for smaller data volumes (under 1TB), the elapsed times were still small – 15.12 minutes for native sort vs. 11.93 minutes with DMExpress for 500GB
  • When you pump up the data volumes, DMExpress really outperformed the native sort – 240.48 minutes for native sort vs. 144.18 minutes with DMExpress for 4TB.  That’s a 40% improvement and nearly 2x faster.  That was consistent for 1TB and 2TB, as well
  • GZIP compression, the results were consistently 2x or more faster:
    • 20.82 minutes for native vs. 8.98 minutes with DMExpress for 500GB
    • 223.82 minutes vs. 84.72 minutes with DMExpress for 4TB, more than 2x faster!

For the Aggregation, we wrote the same aggregation logic in Java, Pig and DMExpress (a key benefit with DMExpress is using a GUI rather than coding, but this post is focused on performance). The compression results were consistent across the board with the non-compression results, so I will just give you the results using GZIP:

• 150GB of data
  • Java: 2.4 minutes
  • Pig: 2.92 minutes
  • DMExpress: 1.18 minutes
  • 600GB
    • Java: 7.89 minutes
    • Pig: 11.15 minutes
    • DMExpress:  4.07 minutes

DMExpress is nearly 2x faster vs. Java, and consistently more than 2x faster than Pig.

What’s that mean for you? It means that you can do more with less nodes, which has implications for the CapEx and OpEx associated with it. Simply stated, you can process more data with the cluster you already have available. If you happen to be running on a public cloud, faster processing times also mean less usage time.   

If you have any questions or want to learn more, please feel free to leave a comment.

DMExpress Hadoop: New Benchmark Results is a post from the Syncsort blog. Copyright © 2012 Syncsort, All Rights Reserved.

Earlier this year, I kicked off the “proof is in the pudding” blog series as a way to share results that DMExpress is achieving during proof of concepts (POCs) in real customer environments. The idea is to wow the loyal readers of the Syncsort blog with information about DMExpress’ speed, efficiency and ease of use.  

It has been too long since I contributed to the series, but I promise to start posting more frequently. We’ve got a lot of exciting work going on behind the scenes and plenty of information to share.

For this post, I want to focus on a recent POC involving a customer that was running up against their nightly batch window. If there was any failure at all during the evening, the customer would not be able to refresh the data warehouse leaving business users with data that is 24+ hours old. This was simply not acceptable to the business and we knew that DMExpress was just the right solution for the job.

For this POC, the environment consisted of a four-core UNIX box with ETL coded in PL/SQL (while that’s really ELT, please forgive the semantics for right now).  Another challenge this customer had was that this particular ETL flow involved nearly 900 lines of PL/SQL which was incredibly complex and nearly impossible to maintain. In fact, they really only had one person capable of maintaining it. What happens if he goes away? Hopefully this doesn’t sound too familiar to you!

The stated goal of the POC was to reduce elapsed processing time by 33%. Additionally, we were looking to demonstrate that DMExpress could significantly reduce the complexity of building and maintaining the ETL.

The particular job involved 5 data sources, identifying changed records, performing multiple joins, enhancing the information via lookup, and loading the database.  The POC ran on approximately 350,000 records, a relatively small amount of data. However, as you are about to find out, the results were quite impressive!

The original process was taking 90 minutes, so the 33% reduction that the POC targeted meant that we had to reduce it to 60 minutes. How did DMExpress do? How about only 6 minutes! That’s a 15x improvement in throughput and 93% reduction in elapsed time for those of you keeping score at home.

How about the 900 lines of PL/SQL? We took that and converted it into just 2 DMExpress jobs, now built and able to be maintained in a simple, easy-to-use graphical user interface. Needless to say, the customer was impressed.

Stay tuned for more results in the days and weeks ahead. In the meantime, don’t be shy about posting comments and questions.

We are also still open to taking on any challengers willing to put their solutions up head-to-head versus DMExpress in a benchmark. Of course, with results like the ones I’ve shared above, I guess it’s not a big surprise that we haven’t had any takers on that just yet…

DMExpress “Pudding” Series: Reducing Elapsed Processing Time by 93 Percent is a post from the Syncsort blog. Copyright © 2012 Syncsort, All Rights Reserved.

Earlier this week, I spent three great days attending Gartner’s Business Intelligence (BI) Summit in Los Angeles. All the usual suspects presented on a variety of topics ranging from Big Data to cloud computing to mobile. However, this year felt a bit different to me. There seemed to be a realization that BI (and related) technologies alone do not represent the road to perfection and information nirvana.

Instead, what I observed was organizations being careful about how they leverage the hot trends of the day. They seem to recognize that they must carefully watch the technology evolution, understand the associated risks and opportunities, and only then determine how to incorporate them into plans for supporting the business.

Here are some personal takeaways from the conference, in no particular order:

• Big Data means big noise.  As companies start to analyze Big Data, the amount of noise grows exponentially. In fact, some studies estimate the noise level to be greater than 70%. Therefore, the challenge becomes how to efficiently and effectively process all the data while filtering out the noise. As Gartner analysts mentioned, organizations need to be very careful not to add bad data in their quest to leverage Big Data. I agree.
• Information is about connecting the dots.  Once we’ve filtered out the noise, we have to connect the dots. Raw data by itself has marginal value. Connecting the dots enables us to convert data into information, adding tremendous amount of value along the way. For instance, having comprehensive data about suspected terrorists has nearly no value if we can’t intelligently connect the dots to unmask their network and predict the next move.  Data Integration plays a key role as the first line of defense not only to integrate the myriad of sources of information, but also to do so in a timely fashion.
• The decision environment has evolved. Instead of only the strategic aspect, the decision environment now also includes the management and operational aspects. This results in new requirements in terms of velocity, variety and volumes of data. For instance, operational workers need near real-time data at the lowest level of detail while strategists may look at weekly, monthly, even yearly trends of aggregated data. No wonder Gartner predicts that by 2014, most organizations will not scale to meet the requirements of Big Data! Now think about what happens if you have underperforming data integration tools pushing transformations down to the database. Performance clearly has a huge impact across the entire organization.  
• Balancing resources is a daunting, but critical task. Fortunately, this is not the case for Syncsort. At the event, Mark Beyer did a great job highlighting this challenge. In the era of Big Data, it’s more important than ever to balance resource utilization – that is CPU, memory, storage and I/O. Workloads are competing for all these resources and the variables are not static. Organizations have different workloads on different months, weeks, and days of the year. IT organizations are being forced to either “oversize” their systems or leave constant tuning/optimization cycles while living in constant fear of failure at “rush hour.” This is why having a highly scalable, self-tuning engine like DMExpress is so powerful.
• Volume is (not) a “20 mules problem.” This is a funny yet interesting analogy. Basically, it means that you could just throw more cores (mules) to parallelize a given data processing job and get it done. Of course, it’s not that simple. I would argue that with more mules also come more issues such as what to feed the mules, how to house them, clean them, keep them healthy, etc. Therefore, you might as well keep the number of mules (or cores) to a minimum! Again, this is another area where DMExpress is highly differentiated in the marketplace.

I’ve been working on shortening my blog posts. It seems that “attention span thing” always gets in the way of everything I want to share! However, if you’ve made it this far, surely you are wondering what Skynet has to do with any of this.

Well, for the first time at a BI conference (and I’ve been to many of them), I observed a subtle, but legitimate concern among attendees about how information – and more specifically algorithms and automated decision making – are shaping our lives and culture. To paraphrase Kevin Slavin, algorithms are shaping the way we live, what we read, what we write, what we consume.

Don’t believe me? Think about the last time you bought a product on Amazon, selected a movie from Netflix, or found a business through Google search.

What do you think? If you attended Gartner BI Summit, what were your takeaways? How is Big Data impacting you? Let’s keep the discussion going.

From Big Data to Skynet – Takeaways from the Gartner BI Summit is a post from the Syncsort blog. Copyright © 2012 Syncsort, All Rights Reserved.

As a disclaimer, I should point out that I have been working with very large data for all of my working life and am extremely passionate about it. In fact, I was on the team that ran the first 1 terabyte, non-extrapolated ETL benchmark 10 years ago.

However, if I’m being completely honest, I must confess that all of this talk about Big Data (including from yours truly on the Syncsort blog) has me increasingly thinking that enough is enough. Suddenly, every company is now a Big Data company. It wouldn’t shock me to find a furniture company at the next tech industry tradeshow selling special reinforced Big Data storage cabinets!

Like kissing in the school yard, Big Data is the topic that everyone is talking about but very few are doing well (if at all). CEOs and CIOs everywhere are being bombarded with messaging that makes it sound like their businesses are about to grind to a halt if they don’t redirect significant portions of their budgets to this “new” area of focus.

As an aside, my first thought for the title of this post was “Has Big Data Jumped the Shark?” before I recalled a post by a similar name from Curt Monash.  In addition to being a very good post, I loved Merv Adrian’s quote towards the end about it being Crocodile Dundee’s job to determine what is and isn’t Big Data. That said, I have grown to quite like the title that I landed on for this post.

I must admit that I do love what Big Data has done for my social street cred. It used to be that data geeks like me, with our vampire tans from being in the data center all day (somehow made worse for a Brit like me), used to be mocked. Can you believe that! Now we are data scientists that are in high demand and can earn fortunes. I have even wondered if my experience in this space will one day lead to me being called a “Big Data professor.” But I digress and it is time to get back to business…

Recently, I met with two very smart (and very talented) executives looking for guidance on how to stop their company’s “impending destruction” at the hands of Big Data.  I naturally tried to share some pearls of wisdom, but what really struck me was that it took a simple name, “Big Data,” to make all this stuff sexy. Data didn’t just become Big Data overnight. One could argue that it has always been that way! Even before I was born, Syncsort was helping customers address the challenges of handling very large data volumes to save money.

So, I’m curious. Is it just me that’s thinking the term Big Data is starting to get so overhyped that it could eventually become meaningless? Is “Big Data” poised to be simply called “data” again? Leave me a comment with your thoughts.

Regardless, I love the fact that data and all the plumbing around it are finally sexy. If this keeps up, it will only be a matter of time before we will all be able to go to a spray tan shop and get the “data scientist special.”

Has Big Data Nuked the Fridge? is a post from the Syncsort blog. Copyright © 2012 Syncsort, All Rights Reserved.

Last week,  I attended two days of sessions at GigaOM’s Structure:Data Conference  in New York City where over 700 attendees came together to discuss the business and industry-transformative nature of Big Data, and the latest technologies and approaches to best manage it all.

What struck me this year is that the conversation has evolved from Big Data being an infrastructure-only issue to now the realization that the Big Data stack requires contribution from everyone from the bottom layer of the infrastructure up through the top application layer.

The following key themes emerged from the onsite discussions and will be the focus as the community continues to develop the Big Data stack:

1) It’s all about high performance computing and speeding up analytics as data volumes grow exponentially. The pain points for unstructured versus structured data are different. While unstructured data requires better visualization of the data, structured data requires more cleansing making filtering and grouping much more critical. One of the speakers referenced a quote from Clay Shirky that, “Information overload is not the problem. It’s filter failure.”

2) The line between personal and business behavior is blurring as analytics moves out of the IT realm and into the hands of business users, and as a result there is an expectation that delivery of data can be more easily consumed, such as through visualization capabilities and collaboration.

3) Real-time decision making through predictive analytics and machine learning is becoming essential with sensor data, digital exhaust and need to get ‘insight’ to consumer behavior.

As such, there’s a realization that the Big Data market is fragmented, and there is plenty of opportunity to contribute to building the Big Data stack. Software packages and tools need to be built on top of Hadoop for example to increase enterprise adoption. Currently most of the available enterprise software is proprietary. Offering applications layered on top of Hadoop will spur the Big Data market leading to more open source contributions and additional opportunities for startups.

Syncsort has a lot to offer in the areas of performance, data integration, and processing – all critical components to the Big Data stack. We can deliver and run ETL over Hadoop without requiring a brand new development team and skill set. One of the speakers suggested that businesses should consider adopting Hadoop only if they are willing to dedicate a separate team. Syncsort’s offering eliminates this requirement for the enterprise. We can also efficiently move the data in and out of Hadoop which as John Webster points out in his CNET post continues to be an issue.

To reach the holy grail of Big Data management – the focus needs to be on building a top to bottom Big Data stack which will require different segments of the market to come together.

Takeaways from GigaOM’s Structure:Data 2012 Conference is a post from the Syncsort blog. Copyright © 2012 Syncsort, All Rights Reserved.

I recently came across a blog post from Susan Hall over at IT Business Edge on the “Seven Keys to Becoming a Data Integration Expert.” Naturally, the headline caught my attention and I soon learned that it was based on a recent post from David Linthicum on “Obtaining Mad Data Integration Skills.”

As I read through both of these posts, I started thinking. Instead of the order that the seven keys had been originally listed, what if I tried to rank them by how much time and money these things cost organizations during an average month. Here is what I came up with:

1. Performance
2. Data governance (this could arguably be number 1, but most organizations I have seen aren’t really doing wholesale data governance)
3. Security
4. Rules and routine
5. Database concepts
6. Interfaces to data
7. Data mediation and transformation

David’s performance criteria states that “…the ability to define how a data integration solution will perform over time.  This is very important.”  I couldn’t agree more!  Building performance and scalability into a DI approach is not only important today, but also for the Big Data requirements of the future. David goes on to say that many DI approaches “become useless after several years.”

We see this every day with our customers and partners.  When they’ve hit the wall with their current approach, they often try one or more of the following:

• Add hardware (CPU, memory) – this is expensive and adds to the software cost, and usually does not scale linearly
• Fine tune the approach/tool – this requires very senior IT staff and/or highly-skilled (read: expensive) consultants from the vendor
• Rip out the logic and push it into the database – now you have an ELT approach pushing the cost and complexity into hundreds of lines of SQL and PL/SQL

Syncsort helps customers solve their performance and scalability issues without needing to resort to stop-gap measures that accelerate costs.

Thanks to Susan and David for their posts and the inspiration they provided me to write this one.  I look forward to following the discussion on their blogs and reading what they write about next. In the meantime, feel free to leave a comment or challenge me if you want to debate the way I’ve ranked the list above.

So You Want to Be a Data Integration Expert… is a post from the Syncsort blog. Copyright © 2012 Syncsort, All Rights Reserved.

Recently, I had the opportunity to attend and present at a conference in Madrid commemorating the 25^th anniversary of NessPro, a key Syncsort partner in the EMEA region. More than 100 IT professionals attended the event at the core of Madrid’s financial district, La Torre de Cristal (The Tower of Glass) in the neighborhood of La Castellana. The theme of the event was increasing business profitability while reducing costs.

It’s hard to imagine a more appropriate topic for an event being held in a country struggling with growing debt and high unemployment. Indeed, the financial crisis has underscored the need to cut costs and improve productivity. However, there was a great sense of optimism and excitement on the 50^th floor of La Torre de Cristal. As another presenter mentioned, the word crisis in Chinese is the combination of two characters: danger and opportunity.

There is definitely an opportunity today for organizations to optimize business processes. Information holds the keys to reduce inventory costs, adapt in real-time to customer demands, streamline operations, uncover new business opportunities, improve productivity, and more. That’s why organizations continue to invest significantly in IT solutions that enable them to uncover a vast array of opportunities.

This is especially true in the competitive banking industry. For instance, RSI (Rural Servicios Informaticos), a joint NessPro and Syncsort customer, presented at the event on how the company is using Syncsort technology to achieve up to 9x savings by gradually migrating most of their applications off the mainframe to open systems. Today, RSI supports Grupo Caja Rural, one of the top 5 banks in Spain with more than 7 million customers, running 75 percent of their batch loads on the open systems. Syncsort’s technology is delivering up to 60 percent faster processing times and is enabling significant savings – helping RSI remain competitive and successful during challenging times.

In the end, opportunity is not just about money. While reducing costs is an opportunity to increase profitability and competitiveness, I was very pleased to see that for my Spanish colleagues, there’s also a strong sense of corporate responsibility. There was a lot of discussion about reducing energy consumption and minimizing waste to preserve the environment, and ultimately bring sustainable economic growth and leadership back to Spain. There are green initiatives in many areas such as reducing paper use, printer toner, PC power consumption, and more.

Special thanks to all my colleagues at NessPro Spain for a very exciting event and equally productive week with the characteristic Spanish hospitality. And of course, many thanks to Alberto, who not only worked closely with me to ensure a successful presentation, but also volunteered to take me through a culinary “tapas” tour at the popular Mercado de San Miguel.  For me, it was the perfect close to a great day of talking about technology and the opportunities ahead.

I wish nothing but the best for Spain and its people.  I have no doubt they will emerge stronger from the challenges they have been facing. ¡Hala, España!

Spotting Opportunity in a Time of Crisis is a post from the Syncsort blog. Copyright © 2012 Syncsort, All Rights Reserved.

Earlier this week as millions of people in the United States went to the polls to cast their votes on Super Tuesday, I was busy getting ready to present with Forrester’s Sebastian Selhorst and Noel Yuhanna on, “Big Data Integration: Achieving Positive ROI in the Era of Exponential Data Growth.”

It seems like the buzz about Big Data is everywhere lately. By now, you’ve likely read all about how Big Data is about the 3 V’s of volume, velocity and variety. However, nothing brings a topic like Big Data to life for me like a real-world example. Think for a minute about Super Tuesday. Not so long ago, people would go out and vote during the day. Later that night they might turn on the television at home for a hint at the early returns. With a bit of luck, the results would be published (and hopefully accurately!) in time to read about it over breakfast in the morning newspaper.

Taking a closer look at this scenario, the volume of information was relatively low in the form of a few pages in the newspaper. Variety was also low with information primarily coming from the newspaper, radio and television (and maybe that annoying neighbor who takes pride in being the first to know and share everything!). Velocity was also low in that you had to wait a good amount of time to get any meaningful data and results.

Fast forward to present day. While millions of people still go out to vote, they access the latest information and trends in real time on Facebook, Twitter, and maybe an online news source like www.nytimes.com or www.cnn.com. In many cases, they are accessing this data from their iPhone or Blackberry, or maybe even their iPad.

In addition to consuming information, they are also creating it. They tweet, post on Facebook, and share their views and preferences via comments on blogs and pushing the ‘like’ button. They are able to track results almost in real time, and chances are they already know which candidates have won (or are very likely to win) which states before turning on the television at night to watch their favorite reality program or sporting event (I won’t even get into how the breakdown of statistics on ESPN contributes to the Big Data phenomenon!).

Clearly, volume is so vast that you could not fit all the articles, comments, photos, podcasts, etc. into a year’s worth of newspapers from the good old days! Variety is high with data coming from a myriad of sources including social media, mobile devices, blogs, newspapers, and more. Velocity is also high, with data coming in every second. That is exactly how Big Data is shaping our lives!

For politicians, the ability to leverage all of this data and the different channels of information can play a decisive factor in gaining a competitive advantage in an election. Of course, not all of us are fortunate (or unfortunate depending on your view!) to be politicians. Whether politics or business, it seems that an organization’s survival today is highly dependent on the ability (or lack thereof) to efficiently and cost-effectively leverage the enormous amount of information being created on a daily basis.

While it is hard to deny the impact that Big Data is having on all aspects of our daily lives, many organizations don’t think of themselves as ‘Big Data’ companies. As a result, I often get the question, “Is Big Data relevant to me?” when speaking to customers and prospects about their data integration challenges. This is an easy one and I almost always respond, “Yes, it is!” The reality is that most organizations are spending more time and resources to maintain their data integration SLAs. They are in need of tools that are easier to use and increase staff productivity. They need these same tools to be able to help them process more data in less time and with fewer resources. Ultimately, they need a more cost-effective solution that meets their expectations!

If these ideas resonate with you, I’d strong encourage you to check out the replay of our ‘Big Data’ Tuesday webinar with Forrester. Additionally, you can download a full copy of Forrester’s report on, “The Total Economic Impact of Syncsort DMExpress,” and play with our online calculator to get a better understanding of the positive impact that DMExpress can have on your business.  If you have any questions or want someone from Syncsort to walk you through the online calculator, feel free to leave a comment and I’ll be happy to help connect you.

Turning Super Tuesday into ‘Big Data’ Tuesday is a post from the Syncsort blog. Copyright © 2012 Syncsort, All Rights Reserved.

Over the last week or so, I have read several articles about the challenges organizations face when deploying Hadoop. This is something that certainly caught my eye since the majority of articles and blog posts I’ve read about Hadoop have been more focused on the opportunities and promises rather than the challenges. As an aside, all the talk about elephants has brought back memories of Undercover Elephant, the good old 70s cartoon that Hanna-Barbera produced about an inspector elephant that helped to solve mysteries. But, I digress…

One of the articles from The Wall Street Journal, “Hadoop Has Promise but Also Problems,” was especially interesting since it featured commentary from executives at two major global organizations regarding Hadoop’s challenges around scalability and ease of use.

Let’s talk about scalability first. Hadoop gets its scalability by deploying a significant number of (arguably) cheap commodity servers. This way, the Hadoop framework can distribute the work among the different servers for increased performance at scale. Of course, adding commodity hardware running open source software looks like a much more cost-effective proposition than adding nodes to a high-end proprietary database appliance. However, the hardware required to cope with growing data volumes and performance SLAs can grow significantly. Therefore, it is not uncommon to find Hadoop deployments with a massive number of nodes. This obviously elevates not only capital costs but also operational costs due to hardware maintenance, cooling, power and data center costs. As one of the executives in the article mentions, this also complicates things “…partly because it requires that engineers deploy software across lots of different servers.”

This leads in perfectly to a discussion around ease of use. This aspect hits at the core of one of the major challenges that nearly every organization working with Hadoop is facing. Hadoop is not easy to develop. Among other things, coding MapReduce jobs and tuning sort operations requires very specific skills that are not only expensive but very difficult to find. For many organizations, this is the most significant barrier for Hadoop adoption.

Scalability and ease of use are both real challenges that organizations are facing today. That’s why Syncsort’s focus on Hadoop continues to center around removing the barriers for wider adoption by accelerating performance and reducing the complexity of Hadoop deployments. Our tests have shown up to 2x faster performance at scale (using same amount of hardware). This means organizations can potentially defer additional hardware purchases while coping with more data and increasing performance requirements. Additionally, developers can use DMExpress to easily leverage the Hadoop framework without the need to learn/code MapReduce jobs.

When executives make public statements in The Wall Street Journal that Hadoop “isn’t easy to work with… partly because it requires that engineers deploy software across lots of different servers,” it is important to listen. That is exactly what Syncsort is doing and why we are getting the word out that DMExpress can help organizations significantly reduce the number of servers required to meet performance goals. It is also why when these same executives say that Hadoop “comes with additional costs of hiring in-house expertise and consultants” that we nod our heads in agreement. DMExpress Hadoop Edition can help reduce those costs by providing an easy-to-use graphical interface that eliminates the need for additional MapReduce experts for constant coding and tuning.

We are listening and would love to hear from you. What has your experience been with Hadoop and are scalability and ease of use also challenges you are facing? We just might be able to help!

Hadoop: Uncovering the Elephant is a post from the Syncsort blog. Copyright © 2012 Syncsort, All Rights Reserved.

“What happens in Vegas, stays in Vegas”… at least according to the popular tourism campaign. Pardon me for breaking this rule, but I believe an exception is in order when it is the TDWI World Conference that brings you to the Entertainment Capital of the World.

As seems to be the case at every TDWI event, I’m finding a vibrant, exciting blend of IT professionals eager to share and learn about the latest industry trends. However, what has really caught my attention this time around is the increasing level of awareness and interest in data integration.

There is no doubt in my mind that “Big Data” and mobile devices have been instrumental in catapulting business intelligence (BI) to the forefront. Information that was once reserved for a few business analysts and techies has become readily available and open to the entire organization via simple, easy-to-use mobile interfaces. This “democratization” of BI is demanding more information than ever before, and creating a significant need for faster, simpler, and more agile data integration solutions.

This is exactly what I’m hearing over and over again this week at the show. It seems like people are increasingly realizing that when all is said and done, data integration is the core that makes data available to the organization. Mobility has allowed BI tools to deliver on user demands for agile, self-service BI. Therefore, organizations have a greater need than ever before for fast, easy ETL development to minimize “time to BI.” This requires a level of performance and scalability capable of delivering all of the data that users are demanding. Interestingly enough, this was one of the common themes I heard during my visit to MicroStrategy World last month.

I had the opportunity earlier this week to speak with Shawn Rogers, an industry analyst and research vice president at Enterprise Management Associates. Syncsort has worked closely with Shawn for some time, and he made a great point about the role of data integration as an “enabler.” I could not agree more with Shawn, and it seems as though more and more IT professionals are coming around to this way of thinking too. In fact, we have seen more and more organizations turning to Syncsort to enable and accelerate BI initiatives.

Are you seeing this too? Is your data integration tool enabling your BI strategy?

Enabling Your BI Strategy is a post from the Syncsort blog. Copyright © 2012 Syncsort, All Rights Reserved.

It has been a great week at Syncsort’s 2012 Global Sales Kickoff in Newport Beach, CA. As you might expect, there has been a lot of discussion about Big Data, Hadoop, ETL, data analytics, business intelligence, and more.

In addition to some great networking with colleagues from around the world, we have been fortunate to have several of our partners join us. During the event, we also announced the winners of our Partner of the Year Awards program.

However, the presentation that had everyone excited and talking was from Enterprise Strategy Group’s Julie Lockner. We’ve interacted regularly with Julie since she joined the ESG team a little more than a year ago, and find her to be really in touch with the end user community and the challenges they face. I had the chance to chat with Julie for a bit after her presentation and wanted to share a portion of the conversation. Thanks again to Julie for participating in our event!

KK: When you speak with end users, what do they tell you about the challenges they face with data integration?

JL: What is interesting is that everyone I speak with regarding data integration state that they have challenges. Aside from the common challenges associated data quality and data consistency, most talk how the development team has challenges responding to new requirements in a timely fashion.  More data, shorter windows, and new applications to be integrated all contribute to the challenges.

KK: What do you see organizations doing to overcome data integration challenges?

JL: Unfortunately, according to recent ESG research, many are still investing in more hardware and software to address the performance challenges. What is interesting, however, is more than 1/5th of the research respondents are using technology classified as data integration accelerators to complement their ETL strategies.

KK: Are you hearing more about data integration accelerators?

JL: As I mentioned in your previous question, this is absolutely becoming a viable option.  Data integration accelerators can play an important role in data transformations – especially when you consider that 80% of the tasks involve simple to complex data flows with sorts being a large part of what consumes hardware and compute resources.  Organizations I speak with that are using solutions such as DMExpress love the fact that it is fairly straightforward to implement and with little effort. They see significant performance improvements in their ETL jobs.  

KK: What are you seeing in terms of Hadoop and the profiles of early adopters?

JL: Hadoop has a lot of promise. There is a lot of excitement because it is new and with that comes hype.  What I am seeing, and what is being confirmed by ESG research, is that many are still trying to figure out how to leverage Hadoop in their environment.  This year seems to be the year of experimentation with Hadoop.  Those that are using Hadoop had an obvious need for something like Hadoop to deal with a seriously big data challenge – and Hadoop made economical sense.  The time and effort associated with coming up to speed with Hadoop and deploying a dedicated compute cluster needs to be vetted out in a business case before taking the plunge. In the meantime, I like the idea that there are a lot of “skunks work” projects out there. This will give the technical team a head start when the business case arrives at their door.

KK: Syncsort is based in New Jersey (home of the New York Giants), but has a number of employees (like myself) that either live in New England or have roots there. Do you want to go on the record with your Super Bowl prediction?

JL: I would love to see the New England Patriots win the Super Bowl this year.  With these two great teams going head to head, it will be a great game.

Big Data Q&A with ESG’s Julie Lockner is a post from the Syncsort blog. Copyright © 2012 Syncsort, All Rights Reserved.

Today, we announced some very exciting news recognizing our partners for their commitment to Syncsort’s solutions. The program is a testament to the important role partners play in Syncsort’s go-to-market strategy. Across both our data integration and data protection businesses, we continue to expand our partner ecosystem and forge deep relationships as we navigate the increasingly diverse and competitive industry landscape.

The past year has brought a lot of attention to the industries we serve. The data integration business saw discussions around the opportunities Big Data represents for companies and how Hadoop and the evolution of ETL tools, or ETL 2.0 as we call it, make capitalizing on these opportunities a reality. In the data protection world, snapshot technology came to the forefront and organizations of all sizes faced increased pressure from the C-suite to modernize backup and implement more robust disaster recovery plans. This has certainly been a positive for solutions like NSB, our integrated data protection offering with our partners at NetApp.

Across the board, our partners demonstrated a tremendous commitment to serving customers and using Syncsort technology to create cost-effective solutions that address some of the most complex IT challenges that exist today. Our partners also showed great enthusiasm for the new Syncsort certification and training programs that we rolled out throughout the year. While we appreciate the many contributions made by all of our partners, the award winners really stood out across multiple dimensions. They include:

Data Integration

• Global Systems Integrator Partner of the Year: Cognizant
• ISV Partner of the Year: Clerity Solutions
• Reseller Partner of the Year: Amdocs

 Data Protection

• Channel Partner of the Year: SwishData
• Channel Partner Rookie of the Year: Zumasys
• NSB Technical Innovator Award: Voyant Strategies

We’ve had a tremendous response from all of the award winners and couldn’t be more enthusiastic about working closely with each of them in the year ahead.  We appreciate their hard work and it’s been fantastic for partners to make statements like:

“Our strategic relationship with Syncsort enables us both to offer clients unique solutions that address the most complex data integration challenges at a fraction of the time and cost. We appreciate the recognition as Syncsort’s GSI Partner of the Year, and look forward to building on our work and success together in the marketplace in the year ahead.” – Hal Lavender, Chief Architect and AVP, Cognizant

“Our partnership with Syncsort provides significant benefits to organizations, such as Singapore Exchange, looking for effective ways to reduce IT operating expenses and enhance flexibility through mainframe migration to better prepare for future growth. By collaborating with Syncsort and utilizing their powerful data integration and sorting technology, we can bring greater value to enterprises with large datasets and batch requirements.” – Cameron Jenkins, COO, Clerity

“Syncsort data protection solutions and their partnership have been a key piece of SwishData’s success since our founding in 2006.  As virtualization continues to disrupt various data center strategies, Syncsort products not only provide our customers and partners with an integrated, simple way to protect data, but also enable them to return to production operation time by recovering applications in minutes instead of the hours commonly required with other products. Fundamentally, SwishData adds value to our customers and Syncsort solutions enhance our ability to do just that.” - Bob Kerr, Vice President, SwishData Corporation

Congratulations again to this year’s winners!

Congratulations to the Syncsort Partner of the Year Award Winners is a post from the Syncsort blog. Copyright © 2012 Syncsort, All Rights Reserved.

Each January for the last 8 years, I have had the opportunity to escape the cold winter weather of the Northeast to visit warmer places like Las Vegas and Miami (where I currently am!) to catch up on the latest BI trends at MicroStrategy World. (Full disclosure, I’m an ex-MicroStrategist).

With more than 2,000 attendees, I can honestly say this year has been one of the most exciting, energizing events I’ve attended in a long time. IT professionals from all around the world are thrilled about the seemingly endless possibilities that market disrupting forces like Big Data, cloud, social media and mobile technologies are producing. When all is said and done, these things will have made a profound impact on the way we do business as well as the way we live, communicate and interact with our world. In fact, they already have!

During my many conversations with BI professionals here at MicroStrategy World and elsewhere, they often cite a fundamental challenge that is preventing them from fully realizing the benefits of their BI applications. That challenge is how to build a strong data integration infrastructure that enables IT to capitalize on the opportunities of Big Data. This is exactly where BI meets DI.

It is clear to me that a solid data integration infrastructure not only accelerates BI initiatives, but also helps maximize the benefits by making more data, that is accurate and relevant, available in much less time. However, this is a story that is much more powerful when told by one of our customers. For those of you in Miami at MicroStrategy World, don’t miss the chance on Thursday, Jan. 26 at 11:30 a.m. to hear directly from a leader in the healthcare industry. Part of the Big Data track, the presentation will be held at the Intercontinental Miami and will focus on using Syncsort’s DMExpress to reduce the cost and complexity of ETL for better, faster BI.

For the lucky ones in Miami, see you there! For everyone else, please feel free to leave a comment. I’m interested in your thoughts and experiences.

Live at MicroStrategy World: Where Business Intelligence Meets Data Integration is a post from the Syncsort blog. Copyright © 2012 Syncsort, All Rights Reserved.

The majority of technology sales, particularly in software, require some sort of proof of concept (POC) intended to prove out the product(s) based on a customer’s requirements. Syncsort is no stranger to POCs and we have a record of producing some really impressive results. Recently, we had the opportunity to present some of these results to a respected industry analyst. He suggested we share some of our POC results on a regular basis on the Syncsort blog. What a great idea!

In writing the first in what will be a series of posts throughout the year on POC results, I was inspired by my colleague and fellow Syncsort blogger Dave Nahmias. One of the phrases that those of us who work with Dave have no doubt heard him speak at one time or another is, “the proof is in the pudding.” We have seen time and time again situations where prospects are pleasantly surprised (and even amazed!) when they get their hands on DMExpress and an up close and personal look at just how fast, efficient and simple it really is to use.

Let’s start with a relatively straight forward POC performed on a Windows machine with 8 cores. Clearly this is not a large, powerful box. This will be important to keep in mind as I share the results. This particular job joined two data sources, performed two aggregations, and then loaded the data into SQL Server and Oracle as well as wrote to a compressed file.

Here are some of the specifics:

• One of the data sources consisted of more than 100 million records (15GB of compressed data). The second data source was small (1,200 records). The reading of both files and the join took 2 minutes 25 seconds (about the same as the amount of CPU time). Only 35MB of memory was used!
• The first aggregation took just under 19 seconds, and used only 3 of the cores and 50 seconds of CPU time.  This included the write to the compressed file and the load into Oracle.
• The second aggregation took 40 seconds, using only 2 cores and 45 seconds of CPU!  This included the load into SQL Server.

Total job time: 3 minutes, 5 seconds!

So, what were we trying to beat?  How about almost 4 hours of processing that was running in the database!  Not only did we beat the times by orders of magnitude, the customer can now use a graphical interface to build and maintain the ETL.  Perhaps more importantly, the customer can also offload expensive database cycles and staging tables.

Since we do this for a living and see results like this from DMExpress all the time, it is easy to lose sight of the impressive results consistently coming from POCs. However, what we believe makes these results even more impactful is that they were achieved without consuming the entire box.

Stay tuned for more results in the days, weeks and months ahead. We’d also love to hear from anyone interested in learning more or who has seen similar results with their tools (please feel free to post a comment). We are also willing to take on those interested in challenging us to a benchmark!

The Proof is in the Pudding is a post from the Syncsort blog. Copyright © 2012 Syncsort, All Rights Reserved.

According to Albert Einstein, the definition of insanity is repeating the same actions and expecting a different result. While I won’t go quite as far as to call it insanity, it has always bothered me that people keep tuning ETL tools that can’t handle larger data volumes. Over Christmas I had an experience which helped me understand at least some of the logic behind it.

It was Christmas Day and I was staying at the home of my fiancé’s parents. I had taken an inflatable bed so that we could stay the night after indulging in way too much turkey. Having managed to shoehorn the bed into a room that was entirely too small for it, I settled down to sleep. Shortly thereafter at about 3 a.m., I woke to find that I was being swallowed by the mattress. It had developed a slow puncture. For those of you that haven’t experienced it, moving around on a deflating air mattress is not easy or fun!

Knowing that if I got up and off the mattress it was going to deposit my fiancé onto the floor, I had little choice but to inflate the mattress from where I was (waking up everyone else in the house in the process). From that point forward, I spent nearly every hour repeating the same process of inflating the mattress until it was time to get up for the day. Needless to say, I was grumpy and the rest of the house was irritable that entire day. There was also a large air mattress deposited directly into the rubbish bin!

This whole situation got me thinking. Even though I knew it wouldn’t help for more than an hour, why did I continue to inflate the mattress throughout the night?

For starters, I didn’t think that I had any other options (although the 4 hours I spent sleeping on the sofa the next day while Boxing Day chaos continued around me proved that wrong). I also thought (at least for the first inflation at 3 a.m.) that inflating the mattress would permanently solve the problem. It was after the second time (okay, probably the third) that I got wise.

Bringing my crazy story back to ETL, the vast amount of people out there “tuning” ETL tools are likely working on this same logic. The first time they do it, there is probably an assumption made about only needing to do it once. The second time, they maybe think that they just didn’t get it quite right last time and this time will work exactly right. The third time, the harsh reality of their situation starts slowly seeping in as they realise they could be doing this for the rest of eternity and not get the result they are seeking.

However, here is the thing. Ultimately, I knew I only had to keep inflating the bed that one night. The next day the leaky air mattress would be in the bin and I’d be at home sleeping in my own bed. People who “tune” ETL tools don’t have that luxury. They know data volumes are increasing (between 10% and 500% a year depending on which customer I talk to) and fundamentally their ETL tools aren’t going to help. Sure, they can try and buy more hardware (a bigger air mattress), but that’s just a temporary (and very expensive) measure because that leak is definitely going to reappear.

In fact, given all the discussion about Hadoop and Big Data, I am now picturing an elephant standing on a deflating mattress! For those of you that made it to this point in my post, thank you for sticking with me. Now it is your turn. I’d love to hear about your thoughts and experiences tuning ETL tools to handle larger data volumes. Comments are welcome!

The Relativity of Einstein, Elephants, Air Mattresses and ETL is a post from the Syncsort blog. Copyright © 2012 Syncsort, All Rights Reserved.