About Monsoon

Monsoon Inc is an eCommerce Software Solution Provider helping hundreds of sellers sell on various marketplaces such as Amazon.com and eBay. Monsoon helps seller manage inventory across multiple markets and price them based on competing offers on marketplaces.

Role of Jothi Software

Our role in this project is to build a cost efficient and functionality enriched version of a pricing pipeline which processes several million price change notifications and price items in the System of Record hosted on the Amazon Web Services (AWS) Cloud infrastructure.

Problem

Monsoon manages a System of Record which consists of millions of items from various sellers. Pricing Parameters are defined in the form of pricing rules and strategies for these items. These items get price change notifications from Amazon marketplaces when the offer information gets changed on the market.

Why AWS?

The number of pricing events received from Amazon is proportional to the volume of inventory the seller sells on Amazon markets and the frequency of changes that take place on offers on the Amazon markets. Processing these high-volume events require a lot of on-premise processing power and storage and is prohibitively expensive for a lot of sellers, especially since there is a lot of fluctuation in seller inventory due to sales seasonality. These variables made AWS workload a perfect choice.

About the Project:

Pricing Events are subscribed to the Amazon SQS Queue from which via a lambda, messages are transformed and written to a kinesis stream. From the kinesis stream these messages are redirected to a Kinesis Analytics Stream where in which similar pricing events from sellers are consolidated and also grouped based on time. Based on the time interval, the pricing events are consolidated and the most recent event is selected and the consolidated batch is sent to the Pricing Request Preparer Kinesis Stream. This stream triggers pricing requests to the preparation lambda which process the messages by reading pricing parameters from the System of Record.

For each Pricing event there could be potentially many items subscribed for the events and for each item there could be multiple pricing variations. The Price Requests are then created and written to another Pricer Kinesis Stream. On an average over 110 million messages per day are expected to be processed in this pipeline and the load will increase based on sales seasonality. Because of the pricing variations the number of pricing requests created could be between 3 to 4 times the number of the notifications. These requests are compressed and written to the Pricer Kinesis Stream. The Pricer Kinesis Stream triggers pricer lambdas which process the messages sent by the kinesis stream, prices and updates the price in the System of Record and records the trace in pricing trace database. There are 3 pricing pipelines which processes the entire workload.

Illustration of Realtime Cloudwatch metrics from November to December 2019

About Monsoon

Monsoon Inc is an eCommerce Software Solution Provider helping hundreds of sellers sell on various marketplaces such as Amazon.com and eBay. Monsoon helps seller manage inventory across multiple markets and price them based on competing offers on marketplaces.

Role of Jothi Software

Our role in this project is to develop a file processing pipeline which reads through files from the S3 bucket which consists of price traces and then write the price traces to price trace database hosted on Amazon Web Services (AWS) Cloud infrastructure.

Problem

Several hundred sellers use Monsoon to manage their inventory across marketplaces and Monsoon price their Amazon listings based on offer information obtained from Amazon markets. Depending on seller’s preference items could be repriced multiple times a day and depending upon the volume of inventory they sell on markets, the volume of pricing events would vary. Each pricing event is associated with a price trace and through Monsoon these price traces can be visualized.

Why AWS?

The number of pricing events priced on the on-premise system could vary based on the volume of inventory the seller lists on various markets and frequency of repricing interval chosen by the Seller. Storing these price traces on an on-premise database requires high storage and server capacity which not all sellers could afford. So AWS was the ideal choice to host a centralized solution.

About the Project:

Monsoon prices inventory items through it’s on-premise software and creates price trace files which are compressed and uploaded to the S3 bucket. The S3 bucket in turn triggers a lambda upon receipt of each file and the files could be from any of the customers supported by the cloud pricing system. The lambda decompresses the file and parses the price trace messages and inserts them to the price trace database. Unexpected failures are handled via writing the messages to a Dead letter Kinesis Stream which triggers a lambda to process these stragglers. The pipeline is expected to process about 100 million files per month from hundreds of on-premise instances which constitutes about 2 billion price trace inserts per month to the database. These price traces will be viewed by the user in the on-premise application via a different end point.

Illustration of Realtime Cloudwatch metrics from September to December 2019