Analytics Rediscovered

Event Analytics     Event is defined as any “incident” or a “transaction” at a point in time that may have monetary impact or not. For example and order is a transaction event with monetary values while a student attending a class or a web page visits are samples of...

Search and Big Data

With the advent of new data sources, ability of “big data” architectures to handle huge data volumes at reduced cost, powerful search interfaces and machine learning techniques are becoming the beacon based on which of powerful business intelligence and...

Search and big data

With the advent of new data sources, ability of “big data” architectures to handle huge data volumes at reduced cost, powerful search interfaces and machine learning techniques are becoming the beacon based on which of powerful business intelligence and analytics applications are built. Let me explain you a bit more.

By this time you would have known about the 3 or 4 V’s of Big data is referred to but this is made possible because of a tectonic shift in major technologies in areas like capture and connectivity, storage and retrieval, powerful processing and analytical capability.

Capture and connectivity : First of the three major processes and a major reason pushing the boundaries to new frontiers.

More devices are capturing data : The internet of things is in very early stages but there are already more devices connected to the net. Imagine, every smartphone(estimated 2 billion devices), personal health monitors are examples of ‘devices’ that captures and processes both personal and business data.

More data is captured by those devices : Do you know facebook captures 600 terabytes a day? The increased data capture stems from three main areas.

  • More data types : Data is captured in all forms and types from pictures, music to video, text
  • Event Vs process/history : The move is from capturing events and occurrences to continuous capturing of process/activity data. Not just departure and arrival now, every flight’s ‘global position data’ is continuously tracked and provided to travellers 24/7 globally
  • Active Vs passive : Many devices are ‘always on’ — means they are continuously capturing and processing data even when you are not engaging in an active transaction. (no need to open the device). e.g. Location data captured, processed with ‘local suggestion/event push notifications’ weather, restaurant, gas station notifications

Connectivity : Increased connectivity increases sharing and replication of the data much quicker in huge volumes.

Storage and Retrieval : Latest enhancements in technologies like hadoop and Mongo DB helps holding vast stores of data in its native format. These technologies combined with innovative delivery models like cloud computing makes the storage cheaper and faster. These technologies also use nodes and cluster management easy for storing multiple copies of the data for quicker retrieval and increased availability.

Powerful processing and analytical capability : The technologies available today makes processing and analysis of the available data by using one or more of these techniques or concepts. The processing capabilities have increased in every aspects like advanced queries, ease of use, reduced deployment time, speed of results, improved relevancy of results, data availability and failover management to name a few. All these when combined sensibly increases the productivity, engagement, experience of customers leading to increase revenue creation and/or conversion of opportunities.

  • Connect to different data sources
  • Indexing, classification, analyzers
  • Powerful filters, Queries to match the data
  • Natural language processing, suggesters
  • Machine algorithm to understand and “learn” automatically
  • Query Optimization and failover management
  • Search results relevancy scoring
  • Powerful analytics incorporating search results

The combined power of data and search and analytics are opening new possibilities and findings that were not available or possible sometime back smart companies are making it’s part of their strategy to gain insights into their business and get an edge over their competitors. The Use of search and analysis tools are not splurge because any company can leverage the power of cloud and managed services to gain the benefits without having huge upfront investment on IT. Also, these technologies have very quick implementation time such that the tangible gains can be obtained early in the game. It is important to have a strategy, goal and implement solutions that are powerful, quick and more importantly business friendly. Although many of the technologies are emerging and far from mature, wait-and-see is not an option for IT managers and CIOs as they will be left behind the competition.