Atlas processes log data in the following way:

1. Data collection endpoint writes data to Amazon Simple Queue Service (Amazon SQS).

2. Data expansion worker takes data from Amazon SQS, attaches a range of records including session data, attributes, and article links managed via

Amazon DynamoDB, and writes the result to Amazon Kinesis.

3. Running on AWS Lambda, a Kinesis consumer takes records from Kinesis and writes each to Elasticsearch on Amazon Elastic Compute Cloud (Amazon EC2) and an in-memory database.

4. Running on AWS Elastic Beanstalk

, a Kinesis consumer stores data for loading to Amazon Simple Storage Service (Amazon S3) and loads it to Amazon Redshift for batch processing.

5. Data users analyze the data through business intelligence and data science tools running on Amazon EC2 and Elastic Beanstalk.

Nikkei also uses

AWS Snowball to rapidly migrate on-premises data to the cloud without putting pressure on communications bandwidth, and Amazon CloudWatch for monitoring.

Using Amazon SQS and Kinesis makes it possible for Atlas to smoothly absorb changing access loads on news articles for which traffic patterns are difficult to predict, processing thousands or even tens of thousands of requests per second with no difficulty.

Clickstream data currently arrives at Kinesis within 200 milliseconds on average. From data transfer to the front end to an analyst query hit takes about one second. This speed means data can be processed in near-real time. Nikkei expects this responsiveness to make a significant contribution to real-time recommendation functionality. "Atlas's processing speeds depend on the availability of a highly reliable messaging system like Amazon SQS. Also, because of the performance and reliability offered by AWS, we can use it in confidence without devoting excessive resources to monitoring and maintenance," says Toshiyuki Isobe, of the Corporate IT Consulting Center at NS Solutions, which provides architectural and operational support.

AWS's managed services not only shortened development time, but also made the solution more cost-effective. "Using AWS's managed services not only let us reduce development time, but also reduced costs to roughly a fifth of their former levels. On the other hand, ten or twenty times as much data can be processed, delivering an ROI of more than 5,000 percent. On top of all this, the system is also more usable than ever," says Sano.