The App Market’s Big Data Challenge :)

The App Market’s Big Data Challenge
The App Market’s Big Data Challenge

The smartphone revolution has changed our lives. For millions of people, smartphones have become essential, and the actual function of making a phone call with your phone is just another important feature. We now use our phones for a variety of tasks, from managing to-do lists, to passing time playing games or watching movies, to completing daily tasks like comparing prices or locating the nearest ATM.

Over 53% of United States mobile phone users own a smartphone, which translates to 123.3 million people, according to a November 2012 report by comScore. This smart phone revolution, which comScore claims will continue to surge for years to come, has been supercharged by major developments in software (as well as hardware) technology, allowing thousands of independent companies and engineers to bring their ideas and innovations to millions of people through mobile app markets like Google Play and the iPhone App Store.

However, the success and time-to-market has been so great and so rapid, that new challenges have surfaced. The proliferation of apps on the market has made it impossible for a mobile user to review all the apps available to them. To add some perspective, Netflix US has about 14K streamable titles , the average supermarket stocks between 15,000 to 60,000 SKUs — whilst Google Play and iPhone AppStore have over 800,000 apps each ! Moreover, standing in front of a store shelf, we glance through dozens of products, while on the app market we only see a handful of featured and popular that fit on the small screen.

The app markets will therefore have to evolve to help users navigate these thick woods. Specifically, app markets will need to seek out better ways of sorting quality apps from poor apps, differentiating originals from copycats, and introducing additional methods for tailoring the experience and offerings for each individual user based on his or her taste and preferences. This requires the technology to support more accurate personal recommendations, tailor search results, suggest relevant opinion leaders to follow and more. For example, someone looking for a “doctor” for their elderly parent should be directed to download “Doctor At Home” and not the casual game named “Doctor Bubble.”

The Good News: Smartphones Provide App Stores with Plenty of Data

In the same way a typical supermarket may track which cereal a consumer purchases, the app store can track and reveal data on which other cereal boxes (or apps) the same person viewed and picked up from the shelf, how often they eat that cereal, how much is left, how much they like it, how many friends they recommended should buy it, and if there are other cereal types they might like better.

Thanks to smartphones, app stores can tap into a much wider range of data sources including where the user currently resides, what their friends and colleague use and buy, which types of content interests them, what they searched for in the past and more . And perhaps most importantly, app stores can instantly adapt which “stock” gets prime ‘store real-estate’ accordingly, even while the user is still in the store making purchases.

This “big data” available within an app store can significantly help to tailor the user experience and offerings. For example, a user who lives in NYC and just landed in London might be interested in the “TimeOut: London” app or “” app for booking a hotel. A user who posted a video on Facebook of the latest Knicks game may be interested in the “New York Knicks Official App,” and a user who listens to Coldplay a lot, might want to download some Coldplay wallpapers. App stores can leverage dynamic big data that is predictive of future purchases and preferences—if the data is processed efficiently and knowledge gleaned is then applied correctly.

The Bad News: Collecting, Managing and Processing Available Data Can Be a Daunting Task

The amount of data generated by millions of smartphones users, combined with the rapid rate at which this data is created, makes it difficult for standard processing technologies to analyze the massive amount of data. Furthermore, the data is pulled from a variety of sources including the app store, social networks, web reviews, data based on location, actual device content like music and past searches, and more, making it even more daunting to analyze the data. Standard processing technologies simply can’t analyze this high volume, high variety, high velocity data. Only “big data” technologies often used from network analytics to drug research development and beyond , are designed to collect, manage and analyze massive amount of diversified data.

Applying big data technologies to the agile app world raises two main constraints:

1. Massive computing resources. Fortunately, developments in cloud computing, a technology allowing companies to easily rent and use “by the hour” computing resources from the likes of Amazon and Microsoft, have made these resources much more accessible, even to startups and single developers.

2. Talent. Although there are many off-the-shelf tools to assist in coping with the massive amount of data, various situations still pose significant challenges that require domain specific knowledge. Precise experience with data handling, including data-working with distributed, non-relational data bases with terabytes of data, and prior experience with algorithms, including machine learning, natural language processing, and dealing with noisy data, are truly required to make full use of the valuable big data and the insights it can provide.

This is likely the app developer’s biggest challenge, in turn the user’s biggest loss, and ultimately, the App Market’s biggest limitation: Leveraging the insights hidden within all this big data can achieve the Holy Grail – the right users discovering the right apps. However with powerful computing and specialized skills, the app discovery obstacle can be overcome.

So how long it will take before going into the app store will be as enjoyable as strolling through a store? How long before the app store magically pops out that perfect app? It’s hard to predict. Much development is yet to be done, and improvements to the app markets over coming months and years will make better use of the vast amount of available data. In the meantime, app market personalization powered by big data analytics and the valuable insights within, will push us toward this goal of the perfect app shopping environment for each individual.