How Artificial Intelligence Can Uncover Customer Needs in Real Time While They Shop in the Store

How do you know when a customer has experienced that magic moment of falling in love with a product at your store? This is a huge question for retailers of all types, ranging from high-consideration retailers such as fashion and luxury to chain stores selling detergent and soda pop.

People exploring the aisles in a grocery store or browsing through blouses in a boutique clothing store give off clues about what they want long before they make a purchase. Perhaps they dwell at one place in the aisle for several minutes. Perhaps they handle three different sizes of that blouse or examine price tags carefully for competing brands of soda pop.

These clues are not always easy for store associates and salespeople to notice, especially in stores where employees are handling a variety of other jobs such as stocking shelves.

Fortunately, technology is making some big inroads.

Anonymous Tracking

A number of retailers are embracing the use of tracking anonymous customer behavior via beacons connected to customers’ devices while they browse. This is done through a combination of cameras and WiFi hotspots. When a customer has the WiFi setting activated on their phone, these devices automatically grab the nearest hotspot as the person moves around a location. By using a series of beacons within the shop, the store can track where the person goes in the store without infringing their personal data in any way. This is because each phone has a unique identity that is registered with every hotspot it touches.

This kind of tracking is useful to gain real-time information on what customers buy and where they go in the shop. All of this information can be collected and used to make a better customer experience as well as to improve profits. For instance, tracking behavior can alert a store manager about a surge in interest in a particular product or promotion, which makes it possible for the store manager to act faster to assign store associates to do necessary restocking.

A Better Way: Behavior Anomaly Detection

But the above approach works only for customers connected to WiFi, and it focuses on the what. Device tracking doesn’t tell you as much about how customers feel about the experience they are having. It doesn’t pick up on customers’ subtle physical cues that they need help from someone in the store.

To understand this richer layer of insight, retailers need to invest in behavior anomaly detection.

Behavior anomaly detection is an application of artificial intelligence (AI). It consists of the use of computer vision and pattern recognition to observe human behavior in stores, airports, and anywhere else people gather. The term means what it sounds like: watching for unusual behavior. This technology helps stores in three major ways:

• Customer service: making it easier for store employees to help people. For instance, behavior anomaly detection could alert a store employee that someone is struggling to load a cart with a flat-screen TV (the anomalous behavior would be the customer spending far longer than normal to place a large item in a cart).
• Sales: alerting store employees that a customer might be interested in buying a product, such as the example of a customer handling several different colors of the same blouse. (The anomalous behavior: the customer is lingering in front of a mirror for a lengthy period of time.)
• Safety: catching people who are committing crimes or acting unsafely.

The technology consists of:

• Computer vision, a form of artificial intelligence (AI) that makes it possible for computers to record visual data such as pictures and video. Computer vision gives store managers real-time insight into everything that is going on in a store to a level of detail that typical cameras miss.
• Pattern recognition, a field of machine learning used to identify patterns and regularities in data, and then classify the data based on the information gained from patterns. Pattern recognition is used to extract useful information from given samples, such as speech, images (such as examples cited above), or a stream of text. The idea is to analyze incoming data and try to identify patterns. Humans use pattern recognition whether they realize it or not in fields ranging from art to math. Machines do so much faster, and they can teach themselves to get better just as people can. Instant pattern recognition makes it possible for behavior anomaly detection to function in real time.

Together, computer vision and pattern recognition can help retailers in a countless number of scenarios in the service and sales alone.

Examples of Behavior Anomaly Detection in Action

Let’s look at a few more hypothetical examples that store associates might overlook without the use of behavior anomaly detection:

• A couple in a department store take turns dribbling different basketballs in the aisles for several minutes. Their faces are unsmiling, so apparently they are not just goofing around with store merchandise for fun. So, what is going on? A camera equipped with computer vision captures the activity in a sequencing transcript. A notification alert is sent to the store manager or to store associate via rapid-response mechanisms such as SMS delivered to their store-issued devices. A playback video occurring in real-time helps the store associate identify where in the store the unusual activity is happening and who is doing it. Thus alerted, the associate asks the couple if they need help. And, yes, they do: the couple turns out to be choosing a holiday gift for a niece who is an avid basketball fan, but they are unfamiliar with basketballs. They’ve been inspecting different brands and dribbling them to make sure they have enough air pressure because they lack a pressure pump at home to test them. The store associate, who specializes in sporting goods, asks a few questions and learns that their niece is five years old. The store associate suggests a smaller mini basketball that is a better option for young children who are just starting out and learning to play around with a basketball.
• A shopper lingers for several minutes at the lightbulb section of a home improvement store. The shopper picks up different lightbulb brands, places them back on the shelf, and paces up and down the aisle for several minutes without making a purchase. This scenario might seem more obvious: someone is trying to make a decision. But the store is unusually crowded, and store associates don’t notice the shopper. Fortunately, anomaly detection system alerts them that someone might  need help. And indeed, in this scenario, the shopper has just moved into a new home and is about to make a large purchase of lightbulbs for a track lighting system but is not sure what to buy.
• A shopper in a luxury clothing store emerges from the dressing room wearing an expensive cashmere overcoat. The shopper roams the aisles but does not seem to be very interested in taking the coat off or stopping in front of a mirror. The shopper is sweating, which could be a sign of someone being nervous about something. So, what’s happening here? A store associate, alerted, asks the shopper if they need any assistance. It turns out the shopper is considering the cashmere coat but wonders if the coat might fit too tight. The sales associate realizes that the shopper has not asked for help because they are a bit reluctant to admit they might need to level up to a larger size. So, the shopper has walking around the store to get the feel for the fit, but wearing the coat inside has made them a bit warm. The store associate consults with them about the size and also offers a complementary bottled water – a nice touch that helps land a sale.

In all these examples, the technology alone does not assist the customer. Sales personnel are needed to interpret what behavior anomaly detection tells them. The technology can empower people to get better at their jobs in many ways beyond alerting sales associates about someone who needs help. The technology can also let a store manager spot scenarios such as too many sales associates clustered in one department, which can lead to a customer getting approached by too many employees in one area while other shoppers are neglected.  

Behavior anomaly detection should be used in a very transparent way, with both customers and employees made aware that the store uses cameras to assist in customer service and safeguard the store. Stores typically do so already with proper signage. The legal requirements are not always clear about this, but it’s simply a best practice for a store to be upfront about the use of cameras.

So, how far are we from seeing the adoption of behavior anomaly detection in stores? Closer than you might think. If you’ve ever shopped at an Amazon Go store, you’re no doubt aware that Amazon uses a combination of cameras and sensors to track your every movement. The brilliance of Amazon Go is that Amazon has effectively communicated the value of tracking customers by educating the marketplace about how the technology, when coordinated with your Amazon Go app, ensures a seamless shopping experience free of check-out lines. But Amazon Go is about efficiency in a closed network – getting the shopper from the front door to checkout as quickly as possible. Behavior anomaly detection is about providing better service with human intervention.

At the same time, behavior anomaly detection won’t work effectively without proper training. The AI needs to be trained with data to know what to look for. Moreover, the technology needs to be used in a way that protects consumer privacy and does not unfairly profile anyone. But with the right training by a diverse team of humans, behavior anomaly detection can be a powerful ally to retailers.

How Centific Can Help

Centific helps retailers use this technology our Scout platform. Scout provides personalized and prescriptive analytics in real-time, intuitively alerting a store’s team to events ranging from people needing help to potentially bad behaviors.

Scout's human-in-the-loop foundation mitigates bias through comprehensive AI training data sets,and leverages Centific's global team of risk mitigation experts for real-time situational analysis, decreasing instances of the technology incorrectly identifying a behavior. As Scout's knowledge of patterns grows, they are shared with users through the exclusive and secure pattern recognition network.

Centific Labs

We operate a simulated retail space that makes it possible to re-create scenarios such as shoppers needing help, customers forming long lines that need to be managed with better crowd control, or theft. We do this with real people acting in roles assigned to them. From there, we record behaviors and use the footage to train computer vision accordingly.

Typically, we give our test subjects very minimal guidance on the scenario we want to record (shopping, theft, etc.). That’s because we want to record all the ways people behave naturally in a real store. If our trainers provide too much detailed instruction, they limit the behaviors we want to record.

Whether it’s understanding a particular customer's behavior, detecting fraud at self-checkouts, or identifying on-premises hazards, Scout empowers retailers to act with confidence in real-time.

To learn more, contact Centific.