MIT Researcher Develops Algorithm To Identify Twitter Trends Before Twitter Can

Topics that are trending on Twitter generate huge amounts of traffic and have the potential to generate serious profits for companies able to take advantage of trending terms. A researcher from MIT claims to have developed a new algorithm that is able to accurately predict topics that will trend on Twitter. The researcher claims that his algorithm can identify trending topics even before Twitter can.

The researcher believes that his algorithm can also be put to use in other business markets as well including predicting stock prices, ticket sales, and other dynamically changing items. The researcher who developed the algorithm is MIT associate professor Devavrat Shah. Shah says that his algorithm has been 95% accurate during testing and has been able to predict trends hours before they appear on Twitter's trend list.

The new algorithm uses a unique approach to machine learning allowing the use of real-time data with historical data to predict outcomes based on past events. The algorithm is also able to predict these outcomes based on past events that most closely align with the current situation being analyzed. That means the algorithm can assign an increase weight to topics that have a trending path similar to trends of the past.

Shah's algorithm may have the potential to be used in all sorts of businesses, but it seems Twitter is certainly interested in the algorithm for its own needs. In fact, one of the research assistants helping Shah is a Twitter employee. So far, the algorithm has been trained using a set of 400 topics. Half the topics it has been trained on have trended and half of them have not. Shah says that the algorithm is designed to process data in parallel across systems like those that are used by large web companies and to avoid traditional issues with large data sets.

[via Gigaom]