Luminate to Lakehouse: Project Lighthouse
The world of CPG and retail is evolving more rapidly than ever, increasing the pressure on suppliers to innovate, adapt and more efficiently harness the insights encapsulated within their data. In response to these demands, Walmart have created a game changing data product, Luminate, to enable brands to access rich, changing datasets to support their decision-making processes. With a fast-moving industry comes even faster moving data, emphasizing the need for a robust, scalable, and reliable platform and solution.
What is Luminate?
Luminate Channel Performance: Walmart Luminate Channel Performance Overview
Walmart Luminate is a comprehensive set of data tools that provide U.S. merchants and suppliers unparalleled access to extensive, consolidated customer insights, facilitating quicker, more informed decision-making. Walmart Luminate offers a conformed view of the omnichannel purchase journey, empowering suppliers to scrutinize shopper actions and performance across different channels, collect direct customer feedback, experiment with new expansion tactics, and evaluate their effectiveness. The triad of modules constituting Walmart Luminate are Shopper Behavior, Channel Performance, and Customer Perception. Luminate is offered in two ways, as a one-click application to get instant access to vital information and as an API data feed. The former gives you quick insight into your customer behaviors, the latter unlocks significant value through enabling integrating Walmart Luminate data with your existing decision support systems – This is where project Lighthouse comes in.
Lighthouse
Project Lighthouse is an initiative developed by Advancing Analytics to optimize the Modern Data Lakehouse architecture to solve challenges faced when implementing Walmart Luminate data feeds.
Luminate presents data via bulk feed endpoints each day and operates on a monthly release cycle, introducing new and enhancing existing endpoints for consumption. Getting access to newly available data is critical for driving day to day decisions, and to accommodate this, Luminate data is ingested “as soon as it’s ready”. But this comes with specific challenges.
· How do you know when all your data will be ready for ingestion?
· What do you do if some datasets are ready, but others have been delayed?
· What if some datasets are available daily, but others are only available weekly?
· What if you want to explore additional datasets without affecting your day-to-day operations?
Lighthouse addresses these challenges from a flexibility first perspective. Being able to easily customize which datasets are business critical, group related datasets and having the ability to rapidly ingest new endpoints for exploration is at the heart of the design. Being able to enrich these endpoints with data from other source systems is key to unlocking Luminate’s potential, such as customizing attribution, defining seasonal periods and optimizing forecast data. All of this comes paired with an intelligent polling and a flexible orchestration solution, giving you the power to decide exactly how, what and when your platform executes.
Trust and Control
To act on the insights your data provides, users must feel secure that the data is accurate and reliable. This means that restatements need to be processed in a timely manner, and that critical metrics must align. Lighthouse comes ready prepared to efficiently handle restatements and is responsive to new data availability to only update downstream systems and reports that have all the data they require. Under the hood, that means having clearly mapped out dependencies, a suite of alerting capabilities to highlight potential data inaccuracies, and efficient error handling.
Scalable and Repeatable
Since its release, Luminate has continued to grow and expand its offering to include more varied and richer datasets; and continues to innovate new ways of delivering data to their partners, such as through the Near Real Time offering. Lighthouse is specifically designed to accelerate new and changing datasets through to production as quickly and reliably as possible. The solution is maximized for repeatability, using the same polling implementation, ingestion pattern, authentication application and processing notebooks for every single endpoint. All of this is underpinned by a comprehensive metadata driven implementation, radically streamlining end to end data processing.
Visibility
No implementation is immune to the unexpected, which is why transparency is key. At every point throughout the platform, status, processing progress and data validation results are exposed to the relevant users so that expectations can be managed with business teams. This includes analytics on the platform itself, so that data availability SLA’s can be monitored and scrutinized, and so that alerts can be generated to communicate business critical events or delays. This ensures that data teams are ahead of any issues occurring within the data before the business teams have visibility of potentially erroneous information.
How to get started?
If you are considering how your business might utilize Walmart Luminate or you are already exploring the one-click solution, get in contact with us to understand how we can reduce upfront development time from months to weeks, to bring the vast wealth of data into your own reports and dashboards.
Lighthouse, the Luminate accelerator, was built on top of our Lakehouse platform accelerator, Hydr8. More information on Hydr8 can be found on the Azure Marketplace or inside our blog on how Hydr8 is compliant with the Azure Well Architected framework patterns.
Lighthouse was initially designed for The Hershey Company, who were early adopters of the Luminate data source. With the help of Lighthouse, Hershey became one of the first organizations to successfully migrate away from a third-party analytics provider to Walmart Luminate. Through our collaboration Hershey were able to save millions of dollars by enhancing in-house data capabilities, consolidate almost 300 spreadsheets into a single executive dashboard and saved thousands of hours through automating previously manual reports.
“By working with Advancing Analytics, we not only connected with the new Walmart platform, but also provided our business team with more value than they had before”
– Jordan Donmoyer, Manager, Customer Data Solutions, Hershey
You can read more about the project undertaken with Hershey via the customer success story written by Databricks or through the Hershey and Walmart case study.
We were thrilled to be selected to co-present the technical implementation and adoption of the solution live in San Francisco at the Databricks Data and AI Summit 2023 , which can be watched here.
Contact the Advancing Analytics team for further information here.