You have a big data project. You understand the problem domain, you know what infrastructure to use, and maybe you’ve even decided on the framework you will use to process all that data, but one decision looms large: What language should I choose? (Or perhaps more pointed: What language should I force all my developers and data scientists to suffer?) It’s a question that can be put off for only so long.
Sure, there’s nothing stopping you from doing big data work with, say, XSLT transformations (a good April Fools’ suggestion for tomorrow, simply to see the looks on everybody’s faces). But in general, there are three languages of choice for big data these days — R, Python, and Scala — plus the perennial stalwart enterprise tortoise of Java. What language should you choose and why … or when?