Enhance Shelf Share Calculation in Retail Stores Using AI-Powered Computer Vision

In the retail industry, a frequent issue is how products are positioned on store shelves. Each store must adhere to contractual agreements that mandate a specific percentage of certain brands to be displayed. To address this challenge, certain companies focus on manually counting products to determine the share of different brands on supermarket shelves. This process often involves monitoring prices, identifying missing items, and more.

In this article, we delve into the benefits of LandingLens, a cloud-based computer vision platform developed by Landing AI. By utilizing LandingLens, Marvik.ai, a cutting-edge machine learning solution provider, streamline and enhance the shelf share calculation process, ultimately saving clients’ valuable time and resources.

Problem overview

A product’s share refers to the percentage of the number of products displayed on a shelf that belong to the product in question. In order to calculate the share, it is necessary to have a series of images showing the region of interest (the entire shelf) and then, for each image, count each particular product, avoiding duplicates due to overlapping images, to then determine the share of each brand and its products. The following figure shows an example to illustrate the problem.

Example of product’s share on a shelf

One of the main difficulties of this problem is the scalability of the solution. The more different products there are, the greater the likelihood of errors and the more tedious the manual work becomes. Suppose you are one of the people that calculates the share, and you start with one type of product (e.g. toothpaste). In the beginning, you are slowly identifying all the different products, and get overwhelmed. Once you work with ten, or twenty different shelves, it gets easier to identify different brands and products, and now you become an expert in recognizing all the different products for each brand. Although it took you some time, you are now able to detect all the different brands displayed on the shelves with low effort. When a new product of the same category comes on the market, it is not difficult to learn to recognize it very quickly.

Imagine that now, you have to not only detect toothpaste, but also soap. In this case, you have a lot of new products that you have never seen before, and the learning curve for that category starts from scratch, letting you behind the time it was taking to you counting all the toothpaste products.

It becomes necessary to incorporate automation to improve the user experience and reduce the likelihood of errors.

First, we were asked to develop a solution for a specific brand in the beverage category. In this case, the goal was to calculate the share for one specific brand, so the number of classes was not too large.

Proposed solution

The complexity of this challenge arises from the diverse range of products and use cases, along with the requirement for human-level accuracy. To tackle this issue effectively, machine learning techniques are crucial. Achieving this goal involves a series of intricate steps, which can be categorized into three main stages.

1. Image stitching: Since shelves are large enough to not fit in a single photo, multiple images of the shelf must be taken to get a complete image of the shelf. To solve this problem, the user must take images that overlap and then perform a stitching process to have one image with all the images that fit the shelf.
2. Products detection: Once the image is stitched, all products in the image must be detected and classified to calculate shelf share.
3. Shelf share calculation: Once all of the products are identified, the share calculation for each product and brand is direct.

Using LandingLens to break time records

Ensuring the proper functioning of a model and retraining it when deviations occur can be time-consuming and divert resources from other aspects of the problem. This was further complicated by the small team size, highlighting the need for operational efficiency to meet project deadlines. Fortunately, the inclusion of LandingLens proved to be a valuable asset in effectively addressing this challenge.

LandingLens was used to develop the second stage of the solution, training an object detector specifically for images of shelves. By using this tool, the team was able to concentrate on other components of the solution while still achieving the required performance level.

The experience

Using LandingLens made it possible to show the client preliminary results in a very short time. First of all, there is a section dedicated to labeling. This feature of the tool made it possible to start testing without the need for the client to label a dataset.

Once an adequate number of labeled images were available, a model was trained to distinguish the target product from the rest, with extremely good results from the start. LandingLens is so well designed that it was not necessary to load it with a large amount of data to get promising results.

The image above showcases impressive results. Despite using only ten training images, the model excels at identifying desired products in new images with exceptional accuracy. In this particular instance, the model correctly identified 73 out of 75 “brand” products and 13 out of 16 “other” products, resulting in branded products occupying 85% of the shelf space. It goes without saying that all assigned labels were accurate.

With just a single click, the trained model was effortlessly deployed, enabling end-to-end testing during the early stages of the project. This not only demonstrated professionalism but also showcased agility.

Furthermore, when labeled data was provided by the client, all that was required was to upload the data to the tool to enhance the model’s performance with minimal additional effort.

Conclusions

Despite the complexity of the problem, the team were able to satisfy the customer’s needs by leveraging LandingLens. LandingLens played a critical role in the development process by offering agility and integration. By reducing the workload associated with data processing, model training, and production deployment, LandingLens improved both the work experience and customer satisfaction. The team were able to see results earlier in the project and release updates more efficiently.