Friday, November 10, 2017

A shortage of data scientists - really?

One of the standards of our industry at the moment is that there is, right now (and in the future), a crucial  and serious shortage of data scientists. I was at an industry seminar last week (a roadshow that had been all over the country) where a senior executive from a BI company repeated this "axiom". She showed some charts and numbers to back up her comment. All was accepted uncritically by the audience. It's what we want to believe.

... but ...

Her numbers were based on predictions that were relevant to the US., and she'd just made some simple (and reasonable) adjustments for the size and nature of Australia's economy to come up with numbers representing the shortage in Australia. A reasonable approach, but, I have a problem those U.S.-based predictions in the first place. I think - based on the data we collect at Monash (and I say "we", as I'm a guest at Monash now, and not a full-time staff member) - they are way out.

No doubt there is a significant market for data scientists in Australia. Just log on to, say, Seek.com today and you'll see 200 odd jobs. I think - and I didn't collect the data previously, so I'm guessing - that most of the jobs listed aren't "new" in the sense that most people want to believe - that there's a revolution happening. Most are for jobs that only a few years ago would have been called statistician, and econometrician or operational researcher. Just now these roles labelled being labelled "data scientist". If you delve into the jobs, you'll see listed as required qualification, a degree in "Data Science, Analytics, Operational Research, Engineering, Econometrics, Applied Mathematic or Computer Science" - except for "Data Science" in that list (from an actual current role) all those degrees have been around for many years. The job label "data scientist" is new, and that's good, but relabelling a category of job (even it it represents a maturation or appreciation of the role of IT and of data in those roles) isn't a revolution.

Regardless of what they are called, what of the numbers of jobs themselves, are they growing? In 2015 a feature article in the Australian Newspaper agreed with the still prevailing wisdom that the market for data scientists was growing fast. A 77% annual increase is noted in the article. Well - that might be true, I started collecting data on how many jobs there where for roles matching the description "data scientist" at that time, and I think at the time the article was published there was actually about 70 odd jobs listed on-line, so that could be consistent with the numbers quotes if the growth came from a very low base. What about now, 2 years later? Well not much has happened. The growth has been solid but slow. There are 200 odd listed right now (jobs tend to list for 30 days - so the total count is kind of a moving average), but that's up dramatically in the last few days due to a lot of cross posting for roles in the federal govt. Dept. of Human Services (Data Science jobs seem to get cross posted - list with more than one agency - at a much higher rate than other jobs). So there's a bit of a spike right now. So the longer term trend is slower growth from nothing to a couple of hundred in the last few years. Is that the start of a major gap between supply and demand? I don't think so. Universities all over the country now have graduate and undergraduate offerings in data science. These courses are popular, and a lot of graduates are being produced. Graduates with qualifications in computer science, statistics, mathematical modelling are also being still being produced. Most of the students attracted to the graduate courses have come from overseas, so it's not like there has been displacement from one University course to another at the expense of the "older" courses.

The long term job trend for "data science" - well - I think it's pretty flat. Its largely - for the past 18 months - been between 100 and 200 active listed vacancies. To give some context - there are about 700 active listed roles in traditional BI, and about 2,500 for JavaScript programmers (the leading required language skill for programmers). So no, there's not a massive shortage of people with data science qualifications, and there isn't a massive job market for data scientists.

Feel free to check my numbers here: http://dsslab.infotech.monash.edu.au:8080/datajobs/, or follow the Twitter account @MonashBIIndex

7 comments:

doll said...

It's the best time to make some plans for the future and it is time to be happy. I have read this post and if I could I desire to suggest you some interesting things or tips. Maybe you can write next articles referring to this article. I want to read more things about it!
Cryptography and Network Security test bank solutions manual

Jobs said...

Good point

Jobs said...

a good reference for business
https://www.frasercoastchronicle.com.au/find-your-local/koalakrane-claw-machine-franchise/

Yousufzai said...

Thankyou for taking the time to write this it was a great read. Good job! Damien Bromfield

Dylan Eales said...

Thanks for this information....

I am passionate blogger and associate with Sample Assignment bestows over the college going students with online assignment help. It is a consultancy possessing academic experts providing a number of subject-specific assignment helps. Marketing, economics, MATLAB, MySQL, management assignment help, etc. are a few to name. The assignment help packages that they supply the students with ensure the students receive an HD or a full money back. With their assignment consultation services, the students can learn by exposing themselves to out-of-the-box learning when it comes to module related studies. Get assignment help in Australia at cheapest price.

kiruthiprabha said...

This is a really helpful article. It helped me gain more amount of knowledge. I must I have read a lot of articles on this topic but this one caters to my queries in the best way...I hope it’s useful to many individuals for developing their skill to get good career.
Learn Data Science Course | Data Science Online Training

manjula said...

Thank you for allowing me to read it, welcome to the next in a recent article. And thanks for sharing the nice article, keep posting or updating the news article.
Best Android Online Certification |Android Training Course