It was great to be in retailing during the past 15 years. Inflated home values, freely available credit, and low interest rates fueled unprecedented levels of consumer spending. Retailers responded by aggressively adding new stores, launching new concepts, building an online presence, and expanding internationally. While the U.S. economy grew 5% annually from 1996 to 2006, in nominal terms, the retail sector grew at more than double that rate—an eye-popping 12%. Revenues rose sharply, profits ballooned, and share prices soared.

But that’s all gone now. Even before the financial crisis and recession began, retailers were hitting a wall. Same-store sales—or “comps”—have dropped by double digits for many chains, store closures have accelerated, store openings have slowed, and shareholder-value destruction has been massive. Starbucks—an icon of the good times—is a case in point. Last fall, it decided to shutter some 600 stores and cut back new-shop openings after the company suffered a first-ever year-over-year drop in same-store traffic and sales. The result: Its share price collapsed by almost 60% from the fall of 2007 to the summer of 2008, and it continued to slide as the economy worsened in the autumn.

Still, hard times—even a deep recession—can be an opportunity to win the loyalty of more customers, increase productivity, and strengthen market position. In this article, we draw on a study of more than 50 major U.S.-based retailers and over 20 years of global consulting experience and research to show how retail executives can respond to a downturn in their business and emerge from it even stronger than before. By following the recommendations laid out in these pages, companies like Starbucks will discover that a larger universe of growth and productivity opportunities is open to them than they might believe. What’s more, they don’t need to overhaul their entire business model to tap into these opportunities; they just need to alter their operating rules.

Rule 1: Go Where the Headroom Is

In tough times, managers instinctively rush to unleash a host of new programs and initiatives—they extend store hours (or cut them back), implement a new staffing system, reallocate store space, introduce or extend loyalty programs, offer “triple point days” and special promotions for big spenders, reorganize store operations or the merchandise or marketing department—even tinker with the parking lot. But without a clear sense of where the opportunity for profitably retaining market share is most promising—let alone where they can win new share—managers engage in too many initiatives that produce too little impact. That can prove expensive, perhaps fatal, at a time when resources are suddenly more limited and getting the highest return on those resources is paramount.

To avoid that trap, you need to understand where your true headroom lies and use that to guide a measured, targeted response. We define “headroom” asmarket share you don’t have minus market share you won’t get. Customers who are loyal to your competitors represent market share you don’t have and will likely not get. Customers who are loyal to you represent market share you already have. Protecting your most loyal customers is an obvious priority in a downturn. But if they are suddenly spending 25% less, most of that will come directly out of what they spend in your stores. Your headroom, therefore, lies with customers who are loyal neither to you nor to your competitors—we call them “switchers.” You may be collecting only 20% of what they’re spending today; taking that to 30% will represent a net gain even when their total spending drops by 25%.

Let’s see how that applies to Starbucks. Earlier in its history, a high proportion of its customers were loyalists for a simple reason: No one else offered the experience they were seeking—high-quality coffee, individualized service, that comfortable coffeehouse atmosphere. But the company has added things like over-the-counter food, drive-through windows, and cookie-cutter store formats, which have made the Starbucks experience more akin to that of fast-food chains than perhaps was ever intended. And this at a time when those chains have become more direct competitors, since corporate investment by such powerhouses as Dunkin’ Donuts and McDonald’s has allowed franchisees to install new, higher-quality coffee machines in their restaurants. As a result, according to customer research we recently conducted, about half of Starbucks’s customers are now spending an average of only 40% of their coffee-related dollars at the Seattle-based firm’s coffeehouses; they’re taking the rest of their money to competitors. These “switchers” are loyal neither to Starbucks nor to its competitors. While loyalists remain Starbucks’s best customers and have been willing to give it the benefit of the doubt, they are not where its headroom exists (see the exhibit, “The Real Opportunity for Starbucks”).

The Real Opportunity for Starbucks

Starbucks already has almost 90% of the business of its most loyal customers. Not much room for growth there. And it’s not likely to get much business from people who are loyal to competitors. So it needs to focus on the sizable group of “switchers”—those who go both to its shops and to others. By giving these customers more of what they need, Starbucks can dramatically turn around its business, even if its most loyal customers are cutting back.

You can measure headroom in many different ways—identifying switchers by category, by local market, by where or how customers shop, or even by competitor. One electronics retailer found its headroom by examining how customers relate to technology, seeking switchers among early adopters, mainstream users, and late adopters. A camera store chain organized its search by segmenting customers according to their level of product sophistication and the amount of service they require.

Whatever the analysis and measures used, we find, generally speaking, that over two-thirds of any given retailer’s opportunity for new market share is concentrated in only one-third of its business. Yet we also find that many—if not most—of its initiatives are aimed at those parts of the business with the least headroom. That explains why such programs multiply: Because they are not targeted at the true opportunities, they fail, and managers respond by firing off yet more projects.

When companies go where the headroom is, they avoid that vicious circle. The initiatives are more likely to work—or, at any rate, it’s clearer how they can be made to work. In either case, that means successful projects get more funding and attention, and people don’t start clutching at straws.

That lesson was not lost on one specialty retailer we worked with, which had long been a must-shop destination for younger women seeking fashion at a good price. Increasing competition eventually hit its sales, which had a devastating effect on performance. Management and the board could not agree on what to do in response: “Should we reformat our existing stores, invest in our brand, open new stores more quickly, develop new formats, or try something else?”

To answer that question, managers analyzed the customers in each product category and geographic market (using large-sample, panel-based research conducted mainly over the internet) to determine who the switchers were, where they were shopping, what they were buying, and why. They found that their loyalists were mostly the “fun and value shopper,” but their opportunity was with the “everyday-trendy dresser.” These customers were in their stores but not finding what they were looking for—and therefore not spending nearly as much as they were at several other chains, including the retailer’s closest competitor. Managers realized that by capturing more of what those customers spent, the sales and profit potential for their existing stores could be three times what they had previously thought. And by changing specific elements of their total offering—including product assortment, store environment, and space layout—they could do a much better job of attracting these particular customers. As a result, the company managed to sustain flat comps while its competitors suffered double-digit declines, thus strengthening its market position and slowing the effects of a rapidly weakening overall market.

Rule 2: Close the Needs-Offer Gap

In our experience, most retailers have a lot of customers who could be spending more money at their stores than they are. The challenge is to entice them to do so. This is both easier and harder than it would seem. It’s easy because all you need to do is give them what they want. But it’s hard because what they want is not more of what you’re currently providing. And to fill the gap between what they’re looking for and what you’re offering, you must forsake the incremental “last year, plus-or-minus” optimization approach that may have served you well in headier times.

Such “needs-offer gaps” can take any number of different forms. They can reside not only in the makeup of your product mix but also in your service levels, in-store environments, or the brand positioning itself. For Starbucks, the proliferation of new stores, together with the emergence of new competition, has created an enormous gap between the experience customers want from the company and the experience they get. For some, it takes too long to buy a simple cup of coffee. For others, Starbucks’s plain vanilla format, particularly in suburbia, makes it difficult to justify the premium they pay there relative to independent coffeehouses, local coffeehouse chains, and even McDonald’s and Dunkin’ Donuts. Many coffee drinkers want a self-serve food experience much like that offered by such outlets as Pret A Manger. Coffee connoisseurs want the espressos, cappuccinos, and experience that can be found in Italy’s best coffee bars. And many just want their original Starbucks back—the socially responsible “third place” between the office and home. Needs-offer gaps such as these explain not only why half of Starbucks’s customers are now spending more of their coffee-related dollars at competitors than at Starbucks but also how the company can change that.

To survive a downturn, retailers must constantly work to identify and close their needs-offer gaps to win as much of their headroom as they can. This is how they gain share and offset the sales they must inevitably lose when their most loyal customers reduce spending. In our experience, though, many retailers do not do that work. Ironically, this is largely owing to an unintended consequence of the explosion of information technology. Most retailers can track on a daily basis what items are selling in which store—and often even to whom and when during the day. But while this information has led to much greater efficiency in inventory management and purchasing, it conditions merchants and store managers to stock up on what’s selling well and pare down on what’s not. This then leads to big gaps between a retailer’s offer and what customers want precisely where the headroom is greatest, since it says nothing about what customers might be buying elsewhere.

This was a trap profitably avoided by one department store retailer we studied. Its apparel sales had been declining, so space productivity (sales and profit per square foot) had fallen below what it was in the rest of the store. The optimization mind-set—ration space according to what is selling the best—would have suggested reallocating the areas devoted to apparel on the floor and in the stockroom to more productive departments, such as handbags and accessories. However, this retailer’s headroom in apparel was disproportionate to the sales it was realizing. Even its most frequent shoppers were going elsewhere to purchase their clothing. So instead of simply reducing apparel space to make it more productive, which would essentially have resulted in overserving its most loyal customers, the retailer compared its apparel assortment with the attributes its customers most wanted but were going elsewhere to get, namely: clothing for the right occasions, in the right styles, at the right price, and with the right fit. The retailer was able to close these gaps through a few targeted merchandising initiatives, such as offering more wear-to-work clothing at a better price; introducing new in-house and external brands in more modern and expressive styles; and expanding mix-and-match basics at the expense of designer collections and “flair” fashions.

Within nine months, the apparel division’s comps went from negative to positive, inventory turns and margins improved, and record operating profits were generated. In some apparel departments, achieving higher levels of productivity required a significant shift in the merchandise mix to fill the needs-offer gaps. When these gaps were pointed out to one of the top executives, he responded: “I just don’t get it. We plan down what doesn’t sell and stock up on what does. How can we be so far off?” The problem with that method is, of course, that current and recent sales data can tell you only what is selling, not what could be selling. By doing the work required to understand the needs-offer gaps, this retailer was able to turn around a business in a way it could never have done by analyzing historical sales data alone.

Rule 3: Go After Bad Costs

When sales stall, retailers confront a stark choice: Cut costs or face declining margins. Most choose to take out costs to preserve as much of their margins as they can. And who can blame them? But all too often they take out the good with the bad.

If you think about it, it’s obvious what the good costs are—they’re the ones essential to producing what your customers value and are willing to pay for. Perhaps these are costs associated with providing convenience, a particular shopping experience, a distinctive service, or a better range of goods than competitors offer. Taking out good costs might improve margins initially, but sooner or later revenue will begin to suffer and margins will come under further pressure, thus defeating the very purpose of taking out the costs in the first place.

Conversely, bad costs are those that add nothing to what customers are ultimately willing to pay for. Even the best run companies incur a lot of bad costs: These might result from ever-evolving consumer needs or competitors’ innovations that change what customers are willing to pay for. Technological advances and process innovations can turn once-necessary costs into unnecessary ones. Costs can creep in through operational complexity resulting from growth in scale and scope. Starbucks’s bad costs might involve too much seating in stores used primarily by take-out customers, or unnecessarily extended hours in certain local markets, or too much inventory and space dedicated to accessories (those coffeepots, movies, and whatnot) that few customers purchase. Or they could be the systemic costs of complexity arising from a proliferation of blends and flavors that have only an incremental impact on the benefit of the Starbucks experience for the bulk of its customers.

Certainly the retailers that do the best job of going after their bad costs while preserving their good costs will have the best chance of both protecting their sales and margins in a downturn and building for the future. But our experience suggests that most retailers don’t have the tools to do this effectively. Like most companies, retailers tend to manage their costs on either a line-item or an activity basis, a practice widely known as “activity-based costing.” Unfortunately, tracking costs in those ways does little to establish two critical links: the link between cost and each aspect of the offer—the product range, store ambience, service levels, and so on—and the link between each aspect of the offer and the customer benefit it produces (which, after all, is what customers are willing to pay for). Viewing expenditures in this way is what we call “customer-benefit costing.” Without this tool, retailers struggle to work out which—or how much—changes in particular costs affect revenue. This prevents them from knowing how to protect margins in ways that won’t also weaken the top line.

What’s more, retailers typically control their costs through the annual budgeting process and become entrenched in the way they’ve always done things. Worse, costs are thought of monolithically (that is, they are considered all necessary or all bad) and tend to be raised or lowered all together, incrementally, rather than in a targeted fashion. A certain German convenience retailer illustrates our point. Customers were equally aware of this retailer and its competitors; as many people shopped there as elsewhere; and customers bought as much in its stores as they did in competitors’. But they visited this retailer less frequently than others, making its costs per customer visit much higher and its share of the total profit pool available to all convenience retailers much lower. Why? It was overinvesting in some areas and underinvesting in others.

Traditional attitudinal research suggested that having clean facilities was very important to customers, and this explained why all competitors were heavily invested in satisfying customers on this score. What that meant, though, was that having bright, inviting shops was merely the table stakes, which in effect turned incremental investment in one of the most important attributes into a bad cost, since outspending competitors on it would garner no marginal gain in business. A careful analysis of customers’ true switching behavior suggested that the more important driver of their loyalty was how friendly the staff was. For our client, the lack of sufficient investment in friendly staff (good costs) and the costs of exceeding customers’ expectations for clean facilities (bad costs) created a deadly combination of lower margins and lower market share.

Managers found that they could reduce the budget for ensuring clean facilities by 20% with no impact on sales or market share. They then reinvested half the budget savings to establish new staff-training programs, a new time-allocation system, and new in-store standards to dramatically improve customer service—and took the rest in margin improvement. The net result was a triple play: lower total costs, a higher share of visits (from 25% to 30%), and a 20% increase in return on capital.

In tough times, there is often no avoiding the need to take out costs. But with the right level of insight, retailers can tie their costs to the benefits that customers are willing to pay for when shopping in their stores. That gives them an important tool for managing their expenses more precisely.

Rule 4: Cluster Stores

Most retailers will tell you that no location is exactly the same as the next one. This doesn’t matter much when the opportunities for rolling out a successful formula in new locations are plentiful. All that matters is opening as many new stores as possible while the formula is still working. But differences among locations are especially crucial when managing through a downturn.

Merchants have been tailoring stores to local markets for years by adjusting assortment, layout, and overall shopping experience to reflect local peculiarities. But that can add immense operational complexity and overwhelm any benefits. Winning retailers master this benefit-cost equation by clustering their stores. A “cluster” is a group of stores representing a set of communities that are very similar to one another in terms of their competitive situations and their customers’ needs and behavior but very different from the communities (and stores) found in other clusters. The stores in a particular cluster can be found in geographically adjacent local markets, but more often they are not.

Many retailers think that clustering stores is the same thing as segmenting customers. But that’s frequently not the case. If you want to use customer segmentation to cluster stores, your segmentation scheme has to fulfill three requirements.

First, you need to segment in such a way that you can identify the proportion of each segment that shops in each of your stores. Otherwise, you can’t uncover opportunities to tailor the product mix, space allocation, staffing, and so on of your stores according to the different needs of each segment.

Second, to use segmentation to establish clusters of stores that can exploit different opportunities to go after headroom, close needs-offer gaps, and take out bad costs, each cluster must contain substantially different proportions of each customer segment. This was a problem for one retailer we worked with: Once we located where different segments of customers were shopping, it turned out that all of the stores had pretty much the same 40%/35%/25% mix of its three segments. So using that segmentation scheme, the retailer could not identify different clusters of stores that could be profitably treated differently.

Third, your segmentation has to cover just about 100% of your customer base. To see why, consider Best Buy. As part of its “customer centricity” strategy, Best Buy tags each of its 900-plus U.S. stores to one or more of five customer segments: affluent young professional males (“Barry”), young entertainment enthusiasts (“Buzz”), upscale suburban moms (“Jill”), middle-class married men who are on a budget (“Ray”), and small-business owners. Best Buy skews the mix of products and services in each of its stores according to particular customer segments. The problem is that these segments represent substantially less than 100% of Best Buy’s customer base in any one of its stores (and probably overall, as well). Thus, in each store, Best Buy has to retain and gain an unrealistically high market share in its target customer segments to protect that store’s overall sales in a downturn. Clustering would allow Best Buy to tailor its stores according to differences in the total customer population of its local markets, not just an important minority.

Starbucks could certainly benefit from clustering its stores. Local differences in the prevailing reasons why customers have occasion to drink coffee at Starbucks’s 10,000-plus U.S. shops—from that first cup of coffee in the morning, to social meetings, to business meetings, to relaxation time—combined with differences in competitive intensity and other factors mean that there is likely to be, in our estimation, a 20% to 30% variation in the level and nature of Starbucks’s headroom across its outlets. A one-size-fits-all solution would miss the mark in any one coffee shop, since specific adjustments to the offer would be needed to capture that variation. But treating each store as entirely unique would be too hard to manage, confuse customers, and take too long to be effective in turning around the business. Clustering would enable Starbucks to vary its stores—both their local offerings and cost structures—according to local differences among the high-potential switchers in its customer base.

There is no best way for all retailers to cluster stores because the factors that explain differences in customer behavior are different for each company. The specialty retailer we cited earlier clustered according to a combination of three factors: the nature of local competition, mall location, and density of local population. Bigger multicategory retailers may find it best to cluster their stores in different ways for individual categories or departments. Then such a retailer could more easily see the various dynamics underpinning demand in different stores. Income level might determine how to cluster stores in a certain category, for instance, but ethnic makeup might be more important in another category.

One large general-merchandise retailer groups its stores into a dozen clusters, ranging from as few as 50 to as many as hundreds of stores. This retailer discovered that differences in five quantifiable factors explained most of the differences in local customer needs among its stores (and therefore the factors that motivated switchers): ethnicity; location (urban, suburban, or rural); family composition (young single professionals, couples with kids, empty nesters, retirees, and so on); income; and level of competitive intensity. Each store cluster represented a unique combination of these five factors. In fact, the retailer had tested as many as 50 potential factors before landing on those five as the ones that best explained the locally distinctive shopper population in terms of customer needs and behavior for each of its stores.

Category by category, this retailer uncovered dramatic variations in the nature of its headroom and needs-offer gaps and, consequently, in its growth opportunities across the business. For example, in its computer category, it found that stores in high-income areas could use a much richer mix of laptops than it was currently providing, whereas its rural stores needed more desktops. Its suburban stores required a different range of brands (Dell, HP, Compaq, and Gateway) than both the high-income cluster (Toshiba, Sony, IBM, and Apple) and the rural cluster (eMachines, Gateway, Compaq, HP, and Dell). The service needs of customers varied a great deal from cluster to cluster as well: The high-income cluster valued installation options, repair, and warranties. The suburban stores favored internet service packages and detailed in-store product information. The rural stores required hands-on technical assistance. This example throws into stark relief exactly how much this retailer’s value proposition had to be tailored from cluster to cluster in each of its main categories and how poorly a one-size-fits-all approach would have suited the needs of its customers with the highest profit potential.

Rule 5: Retool Core Processes

To find headroom, expose needs-offer gaps, reduce bad costs, and cluster stores correctly requires changes to all four of the processes that are core to managing every retailer’s business: customer research, merchandise planning, performance management, and strategic planning.

When sales slow and margins erode, retailers’ decisions tend to become more inward looking. The customer research process must help to prevent this from happening. Traditionally, such research asks, Who is shopping at our stores? What do they buy from us? How satisfied are they with us? and Who are our most profitable customers? These are fine questions, but it would be much better to ask, Why are customers shopping our stores? What do they buy from other retailers? What are their needs relative to what we offer? and Who are the most profitable customers that we don’t have but could get? Answering these kinds of questions is what will give retailers the information they require to find and exploit their headroom and determine which costs they can cut to protect margins without undermining sales.

The good news is that most retailers don’t have to overhaul their research processes to get the right information on their customers. One supermarket chain we know of, for instance, routinely asks patrons, “Did you find what you need?” at checkout. But when the answer is “no,” the next question clerks ask is, “Did you ask for help in finding it?” In other words, the clerks are focused on determining whether current offerings are in stock. But if, when a shopper said no, the clerks responded by asking, “Is there somewhere else you’d expect to find that item?” and if they also asked, “Is there something you want that we don’t ever carry?” the company would end up with a treasure trove of much more useful customer research. As a bonus, customers would see that the company really does care about meeting their needs.

As we pointed out at the beginning, merchandise planning for most retailers is a process of stocking up on what’s selling well and stocking down on what’s not. But in a recession, the process should be governed by answers to these four questions: Which merchandise lines should be expanded because both their headroom and current productivity (sales and profit per square foot) are high? Which should be shrunk because both their headroom and productivity are low? Which should be fixed (rather than shrunk) because their productivity is low but their headroom is high? And which should remain as they are because their productivity is high but their headroom is low? A retailer’s merchants should be able to produce a merchandise-planning map that lays out the answers to those four questions for each of their categories. The map should also specify the needs-offer gaps that have to be closed to grow, shrink, or fix each category’s merchandise lines. (Having such a map for each store cluster would be even better.) This gives merchandisers a practical way to avoid the incremental decisions that traditional merchandise planning entails.

Performance management typically means monitoring progress against budget, as well as benchmarking stores and categories using such measures as comps, gross margins, and sales and profits per square foot. But in a recession, retailers’ scorecards should also indicate where they stand in capturing headroom, closing needs-offer gaps, and taking out bad costs. And they should track their performance by store cluster to avoid the apples-to-oranges comparisons that inevitably occur when monitoring stores by region, district, or other geographically defined territories. One retailer we know does exactly that and has actually improved its performance since the retailing downturn began to intensify last summer, in part because it has the right information at the right level to manage its performance.

Finally, there is strategic planning. Blue-sky planning doesn’t make a lot of sense when the sky seems to be falling. But that isn’t the only role for strategic planning. Strategic decisions still need to be made regarding space allocation, chain investment, store format, cost structure, and staffing. When facing a downturn, the imperative in every one of these areas must be to go where the headroom is, close the needs-offer gaps, go after bad costs, and exploit the differences among store clusters. Let us be clear: The strategic-planning process must be entirely focused on meeting those imperatives. Otherwise, it is just a distraction from what needs to be done in the short-term to protect and strengthen the business for the long haul.

In all likelihood, the current generation of retail executives will not soon see anything like the prolonged tailwind that steadily propelled their sector over the past 15 years. An era of consumer frugality has begun, shifting that tailwind into a nasty headwind. Some retailers will turn this into an opportunity to strengthen their business and gain market share at the expense of the weaker competition. Follow the rules in this article, and you could be one of them.