Data infrastructure optimization, availability & security software
Data integration & quality software
The Next Wave of technology & innovation

A Solution for the Big Data Scientist Shortage Might Be Closer Than You Think

You’ve heard that big data is a big deal. You’ve read all the latest books on the subject, invested in the infrastructure, and secured a solid cloud service provider or your own banks of servers that can help you store and process all that data. Now you realize there’s a big shortage of big data analysts. How are you going to leverage big data without this elusive creature? The solution is to grow your own, and it’s easier than you might think.

How to DIY a Big Data Scientist

Big data scientists aren’t rocket scientists. They’re just harder to find than rocket scientists.

Educational programs are beginning to pop up to train big data scientists, but there are no set rules for skills, training, or experience that a person in this position must have. A big data scientist is essentially a solid statistician with strong mathematical skills plus someone with good understanding of the business. In this sense, a homegrown data scientist is going to be a better choice for you than hiring off the street.

Even if you secure the best mathematician with the strongest statistical skills, you’ll still be hiring someone who knows exactly diddly about the internal workings of your company. By promoting from within, you get someone who already knows what your data holds, and already has a good understanding of what the business needs that data to do.

Good big data scientists also have a few soft skills, including the ability to communicate effectively and collaborate well with others. Ideally, this individual will also have a good rapport with upper management, and will be able to influence their decision making when it comes time to invest in big data innovations.

By hiring from within your own pool of workers, you send a couple of strong messages. First, you’re proving that you value your IT workers, which is a good thing in an era when excellent techies are at a high premium. Second, you are establishing that you are willing to invest long-term with your big data scientist. He or she isn’t someone you pulled in off the sidewalk, and are willing to shove right back out there if things don’t work out. Hiring from within means you value the position and are willing to go the distance with them, their ideas, and sometimes their mistakes.

Yes, whether you hire from within or outside the company, big data isn’t a tried and true science. Anyone you put in this position will need time, patience, and support to find eventual success with analyzing and learning to predict the future using big data.

How Not to DIY a Big Data Scientist

Hire well, and retain better. Big data scientists won’t have a hard time finding another job with your competitor.

What you can’t do is devalue the position of data scientist by thinking that hiring from within is a great way to get good skills for not much money. Whatever you are willing to pay for such an employee coming from outside the company should be extended to the person you choose to promote and groom as your big data scientist. Otherwise, you’ll end up handing over a highly marketable and lucrative set of skills and experience to someone who ends up walking out the door and putting all those goodies to work for your competition. Then the scientist is making good money, your competitor is kicking you into next Sunday, and you’re left without one of the hardest workers to lay your hands on in the marketplace today.

What to Do After DIYing a Big Data Scientist

Home growing your big data scientist means giving them the tools and training they need to make the big data project within your organization a success. Most (if not all) of the vendors that offer big data tools and software also offer training. Make sure your DIY scientist gets all of the training, Hadoop tutorials, and other materials offered by the vendors you use. Ideally, the scientist will have a big hand in selecting which vendors those should be. No scientist — whether experienced or entirely new — can turn coal into gold. If the tools and services aren’t there, none of your big data projects will be a success.

Finally, have patience. Discovering what big data can yield and how to get the goodies out isn’t an overnight proposition. Give whomever you hire to that position the time to make mistakes and figure things out. In the end, it’s a win-win.

Related Posts