When the Google Analytics is not enough

The problem

The telecommunication company implements few applications (web and mobile) for managing accounts and services for customers (self-service). The web application is available in many domains (multi-brand application).

A traditional approach based on Google Analytics, Funnel.io or Mixpanel does not fulfill the requirements. Most of the platforms allow export raw event data, but the company needs advanced reports, covering many aspects of the business, including:

  • Customer transaction history (payments, transactions, due dates)
  • Data used for segmentation (profile data, mobile phone)
  • Omni-channel oriented solution (possible to join data from other communication sources)

The future aim for this architecture is to collect events and predict customer next possible action on any channel.

The architecture

User events came from many sources (different web applications, mobile devices). Each source of events uses Javascript or native mobile technologies to send REST messages with user data.

AWS Cloud

The goal is to receive data without any delay and push messages for future transformation. The architecture based on the Amazon Web Services stack is built on following components:

  • Amazon API Gateway – the entry point for requests, which expose REST API for applications and web pages. The REST API can be executed from any software. The API Gateway supports simple anonymous authentication as well as advanced (tokens, API ID’s). The message resized by the API Gateway is passed to Amazon Kinesis Stream
  • Amazon Kinesis – this part of the system provide three different components:
    • Amazon Streams – The role of the component is to receive data messages and build the execution queue. The benefit of this component is scalability. Depends on the traffic the component could be scaled from one to 126 instances (be careful: the pricing depends on how many instances you are using).
    • Amazon Kinesis Analytics – The component transforms messages and pass to data sinks. The idea behind Kinesis Analytics is to provide the quasi-SQL mechanism for stream transformation, filtering, and aggregations.
    • Amazon Firehouse – as a delivery method which pushes data into Amazon S3 and to Redshift in parallel,
  • Amazon Redshift – data warehouse which provides distributed SQL engine which can be scaled from one to 250 instances. The platform is supported by many commercial and Open Source business intelligence platform
  • Amazon QuickSight – Business Intelligence platform for data visualization and analysis.


The overall architecture could be set up on open source stack (Apache Kafka, Postgresql or Green Plum). But you don’t need to care about many aspects on Amazon including:

  • hardware installation, configuration, and problems,
  • operating system licensing, setup and maintenance,
  • core software installation and support

The benefits come from the scalability of the whole solution, which covers many scenarios:

  • when your business is growing
  • when your business needs periodic reports (monthly, annual)
  • when your business getting down (because you are very near to bankrupt or your business is seasonal),
  • when you have a brilliant idea how to use a data to create new business and you need more power to verify it.

How to setup/tutorials

Coming soon

  • How to setup Redshift/
  • How to setup AWS Kinesis (link)/
  • How to setup AWS API Gateway/



Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s