An experiment on Twitter

For a long time a few of us, and not just at Monash, have wondered how a real time text feed - perhaps in RSS format - might be applied to BI systems. Now that social networking sites like Facebook/Twitter/Friendfeed exist and are becoming widely used the idea has a bit more traction with people we talk too - nothing like concrete experience to help people understand what you are talking about.

I've just set up a Twitter account that I'm going to use to develop a prototype system to demonstrate how a "feed" of text updates might be useful in a BI context. This feed (@monashbiindex) will contain updates and observations on data collected as part of out BI Index project.

Right now, all it will report on is a single number - the number of on-line job ads placed in Australia for positions related to business intelligence and data warehousing. I've been collecting this number (off and on) since 2005. It's quite interesting, there are weekly wobbles (up on Thursday and Friday) and significant seasonal variations too (up before and after the end of the financial year - way down in January). The analysis we are doing will soon expand (just like any dimensional data set) to include more detail like locations, industry sectors, job categories and the like. Later we might extend it to other countries but right now we'll stick to Australia.

So, I've got a nice little automated app, that will at 9:00 Am each morning go to seek.com.au, do a search and grab from the resulting page the total number of jobs. It records that number in a database - along with the data and time. Then a 'reporting' app fires up and does a simple day by day comparison of the number to the previous days and posts a tweet. Much better than the Excel macro's I've been messing with since 2005! Once I'm happy with how that's working, I'll extend the range of topics tweeted on to include a wider range of temporal tweets (end of week, end of month, end of season summaries), link the tweet to reports (no 3D donut charts I promise) and start to build agent style data monitors that look for exceptions, to demonstrate how a twitter style feed might be used for reporting the results of data mining algorithms.

POD

P.S. Oh, later the "Index" will include more measures of health than job ads. We are planning a regular survey of the "industry" and also a stock market index - something again I started a long time ago.

P.P.S The BI Job index is based on the number of jobs that match the search terms "Business Intelligence" or "Data Warehous" that are listed on www.seek.com.au. The index is expressed on points based on a ratio of the number of jobs to the number on the day the index started 23/10/2005. On that day there were 349 jobs - 100 index points.