Manage size, limit query time, monitor use, and automate.
I had the opportunity to speak with Dave Mariani, Chief Strategy Officer and Founder of AtScale about how they have helped Rakuten Rewards, formerly Ebates, improve ROI with Snowflake while reducing compute costs. Rakuten is a Japanese retail conglomerate that enables buyers to shop and get rewards. They manage huge data sets including the inventory of multiple retailers, members’ data, promotions, rebates, and marketing campaigns. All of the data comes together in real-time when the customer is buying; as such, data recency is critical, especially during the holiday buying season.
In 2014, Rakuten Rewards was using SQL Server but needed to break their hardware bottleneck that was getting worse as their business, and data expanded. Users were tripping over each other and this resulted in frustration and missed goals. They moved to Hadoop and added more clusters to scale for three years.
As managing Hadoop clusters became more difficult, Rakuten looked to move to the cloud. They tested many options including Big Query and Red Shift before deciding on Snowflake. Snowflake separates storage and compute into discrete clusters to give different departments and user groups different sized clusters without duplicating data. It provides the ultimate control over costs and resource allocation.
AtScale enabled Rakuten Rewards to scale in the cloud without the management and provision of hardware. The move from SQL to Hadoop on Cloudera had taken several weeks. The move from Cloudera to Snowflake only took a few days.
Now, there’s at least one data warehouse per business or engineering team with dedicated warehouses for ETL components and third-party products. Control of warehouse size is centralized and constantly reviewed for potential downsizing and cost management. IO intensive workloads work in smaller clusters while aggregation and join work on bigger clusters.
AtScale provides a semantic layer that simplifies and normalizes data access to reduce query size. Orders of magnitude query improvements are amplified by high user concurrence. This results in smaller compute resources and mitigates unpredictable and unbounded costs for on-demand pricing models.
By using AtScale and Snowflake, Rakuten Rewards has reduced compute by 73% and reduced query complexity by 76% thereby realizing a 270% ROI on their investment while continuing to grow handling even more data and transactions during the COVID-19 pandemic.