Top Big Data Skills In Demand











Fari Payandeh







Sep 29, 2013

By Fari Payandeh

According to MarketsandMarkets the global Big Data market will reach $46.34 billion by 2018. I wanted to check the local DC Metropolitan job market to verify the hype related to the shortage of professionals with Big Data skills. I went on a job board and searched for the number of job postings that listed Big Data tools as part of the requirements.  I compared the numbers with the numbers that I had recorded on September of 2012.

The  broader IT market didn’t change much in the past year, but there is significant growth in the Big Data related tools (Hadoop, R, MongoDB). I didn’t record the number of postings  for Data Scientists last year, but the number stands at 29 this year– some are posted under Data Analyst because Big Data is new and some folks don’t know the difference between the two; the only way we would know is to read the job description. One interesting fact pertaining to the geographic spread of the Big Data jobs; the jobs are concentrated in few areas. Below is the ranking according to Dice. The sorting was based on Per Capita Ratio.

  • Silicon Valley
  • Boston
  • Seattle
  • Washington, D.C.
  • New York
  • Philadelphia
  • Los Angeles

Big Data will create jobs in three areas: Data Infrastructure, Data Management, and Data Presentation (visualization). Large data sets need storage, Database software to store the data, and Analytics tools to turn data into meaningful information for businesses. Additionally, there are other parts of IT that will be impacted by the surge of Big Data adoption such as Network, Security, and the Cloud.

I divide the required skill sets into two categories: Immediate opportunities and long term opportunities. If you are interested to build your Big Data hot skills for today’s market, you might want to take into consideration this list: Java, Hadoop, Linux, C#, R, Python, Php, Hbase, Cassandra, MongoDB, Amazon ERM, and Mapreduce.

Planning your career for long term engagement will involve several other tools that might be very much in demand in a couple of years. Windows Azure, IBM PureData, Oracle Big Data Appliance, HPCC, Cloud Foundry, Pivotal, Sap HANA, Google BigQuery, OpenStack, Mahout, SAS, Excel 2013, Tableau, Ruby, postgreSQL, MariaDB, Pentaho, MySql, Jaspersoft, Talend, and LAMP.

If you are the leading-edge type and you want to know how to get a hot job in Big Data  with startup companies, I recommend following the developments on these startups. SiSense, Parstream, Skytree, Platfora, VOLTDB, nuodb, Hortonworks, Cloudera, MapR, Splunk, Mu Sigma, Opera Solutions, 1010data, Alteryx, Datameer, and fusion-io

Software Defined Networking, Software Defined Storage, and Cloud Security are among the promising fields in support of Big Data in the coming years.

91 thoughts on “Top Big Data Skills In Demand

  1. Krish

    Very good question , How good is the opportunities for SAP HANA. I’m into SAP CRM. i want to learn this new technology of SAP. I want to know how good is the demand for HANA in the Market. will there be any boost for my carrer growth ?

    1. HANA is the darling of Big Data and I see great future for it. You cannot go wrong there… I highly recommend that you learn it. Not only HANA will be a market winner, it’ll open doors to other Big Data tools for you. Wish you luck!

    1. I think it has a great future and it may become the defacto search utility for Hadoop. Cloudera’s search is powered by the Apache Solr project. MapR’s search is powered by LucidWorks, a commercial search platform based on the Solr and Lucene projects

  2. what about php it have a bright future? currently i am working as a php developer.shall i continue this one or change to big data? please give your suggestions

  3. shaibal das

    Hi Fari,
    Thanks for the informative article on Big data skill sets. I am a Linux system administrator. In which Big Data field should I hone my skill ?

  4. You should keep an eye on the open source Berkeley Data Analytics Stack (BDAS) with Spark & Shark along with supporting start-up Databricks. BDAS is getting a lot of attention in Silicon Valley.

  5. Dear Fari,

    it seems like someone appreciated your work a bit too much – He plagiarized your content at , which was read by the CodeProject Blog Engine and automatically published to
    Take the action you see as needed.


      1. Farid,

        you need not to worry about the CodeProject – The Editors there are quick at preventing Plagiarism and I have directed an email to them.


  6. Sudheer

    Thanks for your wonderful article. I am thinking of moving to big data. I am currently working as a Microstrategy BI Consultant. Please advice.

    1. Sudheer,
      Thank you for your kind words!
      MRST is one of the best places to be for Big Data. It’s keeping up with Predictive Analytics. In my opinion you should stay with MRST, but learn the new tools.

      1. Sudheer


        Thanks! Its gr8 that you spare your valuable time for helping others and am glad I found your article in web!

        Will continue to excel in microstrategy and learn other tools.

        Merry Christmas! 🙂

  7. Somnath Pal Chaudhuri

    I am a Mainframe developer with 6.5 yrs of experience. I want to pursue my career in Big Data using Hadoop.

    Will it be fruitful for me if I undergo a course and then go for a job search? Pl. Advise. I am bit confused.

    1. Somnath,
      Considering your background as a developer, you might want to learn Java first and become certified. The next step would be to learn Hadoop. Once you accomplish that, you can start your job search… sorry but the transition is not easy, but you can do it.

  8. Hakami

    Thank you for your wonderful article. I am oracle forms developer with 4 yrs of experience. I want to move to Big Data but I’m not sure what it is best or the shortest way. Please advice.

    1. Hakami,

      I am sorry about the delay. I wasn’t available. I’d suggest learning BI tools. Since you know how to build GUI application that would be the right place. Business Objects, Oracle OBIEE, Cognos, Microstrategy. Jaspersoft and Pentaho have open source BI tools. Also, learn R language. getting into Infographics is another option you have. Best wishes!

  9. arun

    I m having enty level experiance . i m proficient intrested in data science and and i am profient in pl/sql .kindly suggest which certification to get ? to good career in the feild of data science..

  10. pankaj

    i have 1.2 years of exp in java & 8 months exp in python.i want to switch my job..but m confused about in which prog. language i should make my career..plz suggest something

  11. oyedeji muyiwa

    Hello fari,
    i will like to change my carreer from being hardware engineer to data scienctist, please kindly advice on the step to take to become expert on the field.Thanks in advance.

  12. sameer

    I have 7 yrs exp. last 2 yrs in project management and business analyst.Want to become data scientist. please advice on the steps. is there scope for research in this field ? would MS and PHD in this area help.

    1. Sameer,

      Data Science jobs are more related to hype than reality. I’d recommend staying a business analyst or data analyst. I’d learn Tableau. The public version is free. Also R programming language. This would a great starting point.

  13. Anonymous

    Hello Fari,

    I am a PL/SQL database developer having 8 years experience developing operational data stores & batch applications. I have never had any opportunity to work on Informatica or any other ETL tool, nor any reporting tool. I now find that there are hardly in openings for PL/SQL developers in the market, particularly for people like me exceeding 8+ years. What is your advise for me to be able to break into DWBI market ?


    1. Use your programming background and become Java certified. It won’t be easy but it will guarantee job security. Once you know Java then you can learn Nosql such as MongoDB, Hadoop, etc… Java is your ticket! Best wishes

  14. Vinay

    I have been a developer in Microsoft BI technologies (SSIS and SSRS) for the past 8-10 years. We have lot of jobs but the billing rates are not that high. I want to get in to Big Data as an architect or a manager (Not again as a developer).

    What do you suggest would be the best skills I should acquire. I give my self up to six months to change my domain/skills.

    1. Vinay,

      You have just the right background for getting into Microsoft-Bigdata… I’d suggest learning Windows Azure, power view, powerpivot, and Excel 2013. You can learn it to the degree required to manage a Microsoft/Big Data project. Best wishes!

  15. caezy

    Thanks for your article. I would be grateful if you could guide me with my career path. I am a sap bw consultant with 3 years of exp. Right now I am learning and working on sap hana and bpc. Is it wise to do a master’s on big data? Will sap hana help me get into big data or should I have to learn other tools and databases like hadoop and R.

  16. ak2288

    Hi I am having 4 years of experience in c /c++/unix with database as oracle . i want to switch my technology please suggest

  17. Ananta

    Thanks for your article. I have 8 years of experience in microsoft web development and very good hands on sql server. I want to move to big data technologies. Please guide me.

  18. Abhay

    I have 2.6 years of Experience as VBA Developer and now I want to move in BI. Kindly guide me which technology is better for me to good career and also i want to work in USA. So which technology is better for me to get me in USA.
    Kindly guide me.

  19. Amit

    Hello Fari,
    How are you?
    Could you please let me know in SAP BI and Informatica, which one having excellent future and also take me out of from India. Acutually I am intrested in both of the technology but having little bit confussion about like what to do.


    Hi Fari,

    This is very great article I ever seen for big data and hadoop.

    I am having almost 13+ years of experience in IT also having in depth functional knowledge in retail banking as well as telecom postpaid billing system. Technologically I am good in SQL PLSQL and unix.

    Presently I am senior developer in PLSQL and unix.

    I am planing to move in big data . Requesting you to suggest good course in big data /hadoop

    With Regards,


  21. amit

    Hello Fari,

    i am 9 years exp in IT, including 3 years informatica,3 years SAP BODS ETL tool and also good knowldge in SAP BO.
    so wants a good technology like BIG DATA or HANA.
    Can you please suggest me wich one is best to get job in respective my ETL experience with market situation.



Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.