Search engine marketing is an ever-changing field, and columnist David Rodnitzky warns that agencies that fail to adapt will be left out in the cold.

Search engine marketing agencies will face many business challenges over the next couple of years, including increased competition, fee pressure, clients bringing SEM in-house and a decline in the prominence of SEM in general. SEM agencies that fail to pivot into broad-based digital agencies will either need to focus exclusively on small business SEM or cease to exist.

Why do companies hire agencies?

There are two vectors against which companies evaluate agencies: 1) the value that the agency provides; and 2) the scarcity of the agency’s services. For agencies, scarcity is determined by two criteria: a) the number of other agencies that offer the same service; and b) a potential client’s ability to perform the same service in-house. When you plot these factors on a chart, it looks like this:

As the chart shows, agencies tend to attract business when they can fulfill a scarce need. As supply increases, agencies lose their leverage with clients, resulting in either price wars between agencies (essentially commoditization) or clients bringing services in-house.

There are two additional points that need to be stated here. First, “scarcity” is a subjective term. For example, a company may not understand the difference between a seasoned marketing pro and a college grad with a year or two of marketing experience. Thus, when comparing agency services versus hiring in-house, the company may mistakenly conclude that there is a plentiful supply of expertise, when in fact they are making an apples-to-oranges comparison.

From an agency perspective, however, client perceptions are far more important than reality. In other words, if a client believes the agency’s services are a commodity, they are a commodity, at least with respect to that client.

Second, scarcity and value are not static values. Rather, both change constantly based on market conditions. For example, there was a time when the value of advertising in the yellow pages was very high, but that value continues to drop every year. Similarly, the scarcity of expertise generally declines as a market matures. Today, there are very few experts at virtual reality marketing. Ten years from now, there may be thousands of experts.

The trend lines for SEM agencies are not encouraging

If you look at the SEM business over the last 10 years, scarcity has decreased, and value has peaked. Scarcity was at its height in the early days of SEM, when SEM was still a “nice to have” for most companies and only cutting-edge direct response marketers cared to become experts. I remember a headhunter contacting me in 2003 with a job description that required “5+ years of SEM experience.” I told her to go find Bill Gross, the founder of GoTo.com, since he invented the SEM space and was the only one with that much experience.

So, in 2006, the chart for SEM agencies looked like this (the shaded area is where SEM agencies operated):

Over the last decade, however, SEM has gone mainstream. Today, there are tens of thousands of SEM experts, and a new crop of SEM managers is being minted yearly. Finding an expert with five, 10, or even 15 years of experience is very possible. While the value of great SEM is still high, the supply of SEM professionals has dramatically increased. Thus, the trend for many companies is to bring SEM in-house:

Obviously, there are still plenty of SEM agencies out there (by my count, there are more than 100 agencies that have at least 10 SEM experts on staff). So there are still thousands of companies that are letting agencies handle their SEM. The trend, however, does not favor agencies.

And it’s unlikely that the situation will improve in the future, for two reasons. First, because the supply of SEM experts will continue to expand. And second, because the value of SEM has peaked. Both Forrester Research and eMarketer are projecting SEM budgets to increase in the next five years but SEM market share to decrease (primarily outpaced by social media spend).

Google is also making changes to AdWords that will further diminish the value of SEM. Over the last several years, Google has changed AdWords to “level the playing field” for advertisers. These changes make it easier for advertisers to drive scale, but also increase CPC prices and reduce opportunities for expert optimization. Examples include:

• enhanced campaigns (removing device-optimization);
• expanded reach of all match types (removing long-tail opportunities);
• Product Listing Ads (reducing the value of keywords);
• fewer ads on each page (increasing CPCs, reducing auctions); and
• Conversion Optimizer, Display Conversion Optimizer and Dynamic Search Ads (enabling automation).

I am not claiming that Google has done this with malice aforethought, I am simply making an observation about the changing landscape of SEM. The bottom line is that SEM has become a zero-sum game, with a few big advertisers at the top of each vertical owning an ever-increasing percentage of clicks (and paying more for each click). This is bad for agencies for two reasons: First, SEM spend is concentrated in a few players, so there are fewer potential clients; and second, as CPC prices increase, SEM becomes less profitable for advertisers.

In sum, declining SEM market share, SEM concentration in fewer advertisers and decreasing value from SEM means that we may see SEM fall to the lowest category on my chart in a few years:

If you’re the CEO of an SEM agency reading this, at this point you are either in denial, panicking or a combination of the two. But despite the gloomy outlook I have presented so far, I do think there are opportunities for SEM agencies to thrive moving forward. There are two broad options: go local, or go broad.

Option #1: Go local

By “go local,” I mean focusing your business on small to medium-sized businesses. One of the consequences of Google’s product changes is that market share will be wrested from mid-sized advertisers and transferred to larger ones. Consider a hypothetical vertical — call it the “widgets” category. For the last few years, 15 advertisers have dominated 95 percent of the spend on Google. In the future, these 15 advertisers are pruned to only eight advertisers:

In other words, about 50 percent of the advertisers will no longer rely on Google AdWords for traffic in the future (even though Google will make the same amount of revenue). Inevitably, this means that — at least for this hypothetical vertical — 50 percent of the agencies will be out of business.

The 50 percent left will be big enterprise clients and small to medium-sized businesses (SMBs). As noted, enterprise clients are more and more likely to move to an in-house model, as they can afford to build these teams in-house and there is no longer a shortage of SEM experts. And enterprise spenders who decide not to hire in-house will likely hire a broad-based digital agency to handle multiple channels and services, instead of hiring numerous agencies. Most enterprise spenders choose the efficiency of “one throat to choke” over the increased performance of many specialty partners.

Hence, SEM agencies that want to remain focused on SEM will need to focus on small to mid-sized potential clients that need help. While Google’s changes will also drive some consolidation in the SMB space, there are always going to be local advertisers who need to geotarget a metro and specific vertical (think lawyers, plumbers, preschools). Granted, these are different types of advertisers from the enterprise clients spending millions a month, but in many respects, they are more secure clients anyway than large clients, since they are unlikely to replace an agency with an in-house team.

To survive on SMB clients requires either significant automation or a high client-to-account-manager ratio, since the fees an SMB can afford are much lower than an enterprise client. Many agencies have proven that they can scale on this model, however, so this is a viable option.

Option #2: Go broad

The second option for SEM agencies is to, frankly, no longer be SEM agencies, but rather, to aspire to be something more than just the SEM buyers for their clients. There are many ways to accomplish this, but I will generally refer to this as becoming a digital agency of record (AOR).

There are a few different setups that might work here, and I’ve organized them in order from the easiest to the hardest transitions for SEM agencies:

1. The digital performance media AOR. An agency that handles all digital performance media buying for a client. This is anything that is bottom-of-funnel (SEM, Facebook, PLAs, retargeting) and could also include mid-funnel channels like content marketing and programmatic display.
2. The digital performance AOR. All of the media buying capabilities of the first category, but with the addition of non-media buying services that complement media buying, like analytics, strategic consulting, conversion rate optimization and so on.
3. The digital full-funnel performance AOR. This level adds top-of-funnel digital media buying, which essentially is any digital media that is designed to drive incremental users who are not being captured by mid- or bottom-of-funnel channels (and as a result, will have much high CPAs or lower ROAS).
4. The performance AOR. Okay, I admit that this one may sound crazy, but the concept here is that the former SEM-only agency manages all performance marketing — direct mail, DRTV, radio — whatever fills the funnel is on the table.

The key to whatever expansion of services an SEM agency elects to offer is that this expansion must meet the dual criteria of providing value to clients and being a scarce resource that isn’t easily replicated. For better or worse, the farther you move down the above list — the more comprehensive your service offerings — the more likely you create scarcity and value.

Build your agency moat

In 2011, a venture capitalist named Bill Gurley described Google’s corporate strategy as an “economic moat,” which amounts to a product roadmap and acquisition philosophy that is designed to protect (create a moat around) Google’s AdWords business. In particular, he points to the Android mobile operating system as a great example of Google building products that they are giving away for free in order to protect AdWords:

Google’s aim is defensive not offensive. They are not trying to make a profit on Android or Chrome. They want to take any layer that lives between themselves and the consumer and make it free (or even less than free).

For most SEM agencies, AdWords is also the economic castle around which a moat must now be built. If you want to continue to provide SEM services to clients, you need to build a suite of services or products that complement AdWords. And here I agree with Andrew Goodman when he notes:

It’s about a culture of expertise and a shared professional mission, as opposed to a culture of pure financial performance.

Some of the largest agencies in the world — ironically, those who can most afford to educate and inspire their people — are the ones skimping the most. The bean-counters have taken hold.

The road to hell is paved with good intentions, after all, and Goodman is absolutely right that many large agencies have set off to build multi-channel, full-funnel service solutions but have lost their culture of expertise along the way. Someone smarter than me once said it another way: an expert at everything is an expert at nothing.

So the choice is not between being experts at SEM or offering a lot of services; successful agencies will have to be both. Future-proof agencies will need to maintain a strong culture of expertise (though I do not believe this mandates that the most senior executives must forever play this role), while hiring or cultivating new experts who can provide expertise in ancillary fields.

I doubt, by the way, that most large agencies will be able to regain a culture of expertise. So this presents an amazing opportunity for SEM agencies to fill a void. Going back to the notion of scarcity and value, in the future, clients will have three choices:

1. SEM agencies: perceived commoditization, often more expensive than in-house;
2. Large agencies: one-stop shops without true digital expertise; or
3. Digital AORs: SEM agencies that maintain a culture of data and expertise and are able to offer complementary services around SEM (the economic moat).

On the chart, it looks like this:

For some SEM agencies, a strict focus on SEM may work in the future — as noted, these agencies will likely need to go downstream and offer services to smaller clients. For others, the digital AOR route seems like the most likely path. It is not an easy path, by the way; if it were, the large agencies would have figured it out by now.

Build your personal moats

Just because SEM agencies are dying, it doesn’t mean that SEM is no longer a great profession to pursue. There are still plenty of job opportunities in SEM — from entry-level to seasoned professional — and I think that having SEM expertise will continue to be a valued, high-paying career path for the foreseeable future. After all, even if a company eschews SEM agencies and hires SEM expertise in-house, the actual SEM need isn’t going away, it is just being shifted from the agency to the client.

Moreover, SEM will continue to be a huge chunk of digital media spend going forward, and SEM experts will be needed to tweak campaigns and software to drive incremental gains. Companies and agencies will continue to fight over the best SEM pros, which means that top performers will still command great salaries.

As with SEM agencies, however, standing still and resting on your SEM bona fides is a risky strategy. SEM experts who also understand the rest of the marketing ecosystem — other media channels, analytics, martech, devices, customer journey, personas and user experience — will be in even higher demand than SEM experts who just know SEM. Put another way, SEM experts who build a moat around their SEM knowledge will be well-positioned moving forward.

Trees, meet forest

I noted previously that value and scarcity are moving targets: today’s value is tomorrow’s commodity. And much as I wish it were true, most people don’t wake up and think, “How can I work with more agencies today?”

Spend too much time “in your business,” and you run the risk of maintaining expertise in a field that is in decline. I’ve always wondered what it must have been like to be a blacksmith during the transition from a horse-centric to a car-centric economy, or the CEO of Kodak at the dawn of digital photography, or a yellow pages salesperson in 2000. Change happens, and a myopic focus on what’s worked for the last 10 years doesn’t guarantee that you’ll be in business for the next 10 years.

Search engine marketing has been very good to me. For one, it gave me an avenue to not be a lawyer, for which I will forever be grateful! It’s also given me a steady income for more than 15 years, the opportunity to build and sell a company and a network of super-smart people.

But the next 15 years are not going to be about who can come up with the best negative keyword list or build the best day-parting algorithm. SEM isn’t going away, it’s changing and morphing into something much bigger. It will still be a lot of fun, but it will be very different, and it probably won’t be called SEM at some point. Agencies that don’t pick up on this trend probably won’t be around.