Better Than Basics: Custom-Tailoring Your SEO Approach

Posted by Laura.Lippay

Just like people, websites come in all shapes and sizes. They’re different ages, with different backgrounds, histories, motivations, and resources at hand. So when it comes to approaching SEO for a site, one-size-fits-all best practices are typically not the most effective way to go about it (also, you’re better than that).

An analogy might be if you were a fitness coach. You have three clients. One is a 105lb high school kid who wants to beef up a little. One is a 65-year-old librarian who wants better heart health. One is a heavyweight lumberjack who’s working to be the world’s top springboard chopper. Would you consider giving each of them the same diet and workout routine? Probably not. You’re probably going to:

1. Learn all you can about their current diet, health, and fitness situations.
2. Come up with the best approach and the best tactics for each situation.
3. Test your way into it and optimize, as you learn what works and what doesn’t.

In SEO, consider how your priorities might be different if you saw similar symptoms — let’s say problems ranking anything on the first page — for:

1. New sites vs existing sites
2. New content vs older content
3. Enterprise vs small biz
4. Local vs global
5. Type of market — for example, a news site, e-commerce site, photo pinning, or a parenting community

A new site might need more sweat equity or have previous domain spam issues, while an older site might have years of technical mess to clean up. New content may need the right promotional touch while old content might just simply be stale. The approach for enterprise is often, at its core, about getting different parts of the organization to work together on things they don’t normally do, while the approach for small biz is usually more scrappy and entrepreneurial.

With the lack of trust in SEO today, people want to know if you can actually help them and how. Getting to know the client or project intimately and proposing custom solutions shows that you took the time to get to know the details and can suggest an effective way forward. And let’s not forget that your SEO game plan isn’t just important for the success of the client — it’s important for building your own successes, trust, and reputation in this niche industry.

How to customize an approach for a proposal

Do: Listen first

Begin by asking questions. Learn as much as you can about the situation at hand, the history, the competition, resources, budget, timeline, etc. Maybe even sleep on it and ask more questions before you provide a proposal for your approach.

Consider the fitness trainer analogy again. Now that you’ve asked questions, you know that the high school kid is already at the gym on a regular basis and is overeating junk food in his attempt to beef up. The librarian has been on a low-salt paleo diet since her heart attack a few years ago, and knows she knows she needs to exercise but refuses to set foot in a gym. The lumberjack is simply a couch potato.

Now that you know more, you can really tailor a proposed approach that might appeal to your potential client and allow you and the client to see how you might reach some initial successes.

Do: Understand business priorities.

What will fly? What won’t fly? What can we push for and what’s off the table? Even if you feel strongly about particular tactics, if you can’t shape your work within a client’s business priorities you may have no client at all.

Real-world example:

Site A wanted to see how well they could rank against their biggest content-heavy SERP competitors like Wikipedia but wanted to keep a sleek, content-light experience. Big-brand SEO vendors working for Site A pushed general, content-heavy SEO best practices. Because Site A wanted solutions that fit into their current workload along with a sleek, content-light experience, they pushed back.

The vendors couldn’t keep the client because they weren’t willing to get into the clients workload groove and go beyond general best practices. They didn’t listen to and work within the client’s specific business objectives.

Site A hired internal SEO resources and tested into an amount of content that they were comfortable with, in sync with technical optimization and promotional SEO tactics, and saw rankings slowly improve. Wikipedia and the other content-heavy sites are still sometimes outranking Site A, but Site A is now a stronger page one competitor, driving more traffic and leads, and can make the decision from here whether it’s worth it to continue to stay content-light or ramp up even more to get top 3 rankings more often.

The vendors weren’t necessarily incorrect in suggesting going content-heavy for the purpose of competitive ranking, but they weren’t willing to find the middle ground to test into light content first, and they lost a big brand client. At its current state, Site A could ramp up content even more, but gobs of text doesn’t fit the sleek brand image and it’s not proven that it would be worth the engineering maintenance costs for that particular site — a very practical, “not everything in SEO is most important all the time” approach.

Do: Find the momentum

It’s easiest to inject SEO where there’s already momentum into a business running full-speed ahead. Are there any opportunities to latch onto an effort that’s just getting underway? This may be more important than your typical best practice priorities.

Real-world example:

Brand X had 12–20 properties (websites) at any given time, but their small SEO team could only manage about 3 at a time. Therefore the SEO team had to occasionally assess which properties they would be working with. Properties were chosen based on:

1. Which ones have the biggest need or opportunities?
2. Which ones have resources that they’re willing to dedicate?
3. Which ones are company priorities?

#2 was important. Without it, the idea that one of the properties might have the biggest search traffic opportunity didn’t matter if they had no resources to dedicate to implement the SEO team’s recommendations.

Similarly, in the first example above, the vendors weren’t able to go with the client’s workflow and lost the client. Make sure you’re able to identify which wheels are moving that you can take advantage of now, in order to get things done. There may be some tactics that will have higher impact, but if the client isn’t ready or willing to do them right now, you’re pushing a boulder uphill.

Do: Understand the competitive landscape

What is this site up against? What is the realistic chance they can compete? Knowing what the competitive landscape looks like, how will that influence your approach?

Real-world example:

Site B has a section of pages competing against old, strong, well-known, content-heavy, link-rich sites. Since it’s a new site section, almost everything needs to be done for Site B — technical optimization, building content, promotion, and generating links. However, the nature of this competitive landscape shows us that being first to publish might be important here. Site B’s competitors oftentimes have content out weeks if not months before the actual content brand owner (Site B). How? By staying on top of Site B’s press releases. The competitors created landing pages immediately after Site B put out a press release, while Site B didn’t have a landing page until the product actually launched. Once this was realized, being first to publish became an important factor. And because Site B is an enterprise site, and changing that process takes time internally, other technical and content optimization for the page templates happened concurrently, so that there was at least the minimal technical optimization and content on these pages by the time the process for first-publishing was shaped.

Site B is now generating product landing pages at the time of press release, with links to the landing pages in those press releases that are picked up by news outlets, giving Site B the first page and the first links, and this is generating more links than their top competitor in the first 7 days 80% of the time.

Site B didn’t audit the site and suggest tactics by simply checking off a list of technical optimizations prioritized by an SEO tool or ranking factors, but instead took a more calculated approach based on what’s happening in the competitive landscape, combined with the top prioritized technical and content optimizations. Optimizing the site itself without understanding the competitive landscape in this case would be leaving the competitors, who also have optimized sites with a lot of content, a leg up because they were cited (linked to) and picked up by Google first.

Do: Ask what has worked and hasn’t worked before

Asking this question can be very informative and help to drill down on areas that might be a more effective use of time. If the site has been around for a while, and especially if they already have an SEO working with them, try to find out what they’ve already done that has worked and that hasn’t worked to give you clues on what approaches might be successful or not..

General example:

Site C has hundreds, sometimes thousands of internal cross-links on their pages, very little unique text content, and doesn’t see as much movement for cross-linking projects as they do when adding unique text.

Site D knows from previous testing that generating more keyword-rich content on their landing pages hasn’t been as effective as implementing better cross-linking, especially since there is very little cross-linking now.

Therefore each of these sites should be prioritizing text and cross-linking tactics differently. Be sure to ask the client or potential client about previous tests or ranking successes and failures in order to learn what tactics may be more relevant for this site before you suggest and prioritize your own.

Do: Make sure you have data

Ask the client what they’re using to monitor performance. If they do not have the basics, suggest setting it up or fold that into your proposal as a first step. Define what data essentials you need to analyze the site by asking the client about their goals, walking through how to measure those goals with them, and then determining the tools and analytics setup you need. Those essentials might be something like:

• Webmaster tools set up. I like to have at least Google and Bing, so I can compare across search engines to help determine if a spike or a drop is happening in both search engines, which might indicate that the cause is from something happening with the site, or in just one search engine, which might indicate that the cause is algo-related.
• Organic search engine traffic. At the very least, you should be able to see organic search traffic by page type (ex: service pages versus product pages). At best, you can also filter by things like URL structure, country, date, referrers/source and be able to run regex queries for granularity.
• User testing & focus groups. Optional, but useful if it’s available & can help prioritization. Has the site gathered any insights from users that could be helpful in deciding on and prioritizing SEO tactics? For example, focus groups on one site showed us that people were more likely to convert if they could see a certain type of content that wouldn’t have necessarily been a priority for SEO otherwise. If they’re more likely to convert, they’re less likely to bounce back to search results, so adding that previously lower-priority content could have double advantages for the site: higher conversions and lower bounce rate back to SERPs.

Don’t: Make empty promises.

Put simply, please, SEOs, do not blanket promise anything. Hopeful promises leads to SEOs being called snake oil salesmen. This is a real problem for all of us, and you can help turn it around.

Clients and managers will try to squeeze you until you break and give them a number or a promised rank. Don’t do it. This is like a new judoka asking the coach to promise they’ll make it to the Olympics if they sign up for the program. The level of success depends on what the judoka puts into it, what her competition looks like, what is her tenacity for courage, endurance, competition, resistance… You promise, she signs up, says “Oh, this takes work so I’m only going to come to practice on Saturdays,” and everybody loses.

Goals are great. Promises are trouble. Good contracts are imperative.

Here are some examples:

• We will get you to page 1. No matter how successful you may have been in the past, every site, competitive landscape, and team behind the site is a different challenge. A promise of #1 rankings may be a selling point to get clients, but can you live up to it? What will happen to your reputation of not? This industry is small enough that word gets around when people are not doing right by their clients.
• Rehashing vague stats. I recently watched a well-known agency tell a room full of SEOs: “The search result will provide in-line answers for 47% of your customer queries”. Obviously this isn’t going to be true for every SEO in the room, since different types of queries have different SERPS, and the SERP UI constantly changes, but how many of the people in that room went back to their companies and their clients and told them that? What happens to those SEOs if that doesn’t prove true?
• We will increase traffic by n%. Remember, hopeful promises can lead to being called snake oil salesmen. If you can avoid performance promises, especially in the proposal process, by all means please do. Set well-informed goals rather than high-risk promises, and be conservative when you can. It always looks better to over-perform than to not reach a goal.
• You will definitely see improvement. Honestly, I wouldn’t even promise this unless you would *for real* bet your life on it. You may see plenty of opportunities for optimization but you can’t be sure they’ll implement anything, they’ll implement things correctly, implementations will not get overwritten, competitors won’t step it up or new ones rise, or that the optimization opportunities you see will even work on this site.

Don’t: Use the same proposal for every situation at hand.

If your proposal is so vague that it might actually seem to apply to any site, then you really should consider taking a deeper look at each situation at hand before you propose.

Would you want your doctor to prescribe the same thing for your (not yet known) pregnancy as the next person’s (not yet known) fungal blood infection, when you both just came in complaining of fatigue?

Do: Cover yourself in your contract

As a side note for consultants, this is a clause I include in my contract with clients for protection against being sued if clients aren’t happy with their results. It’s especially helpful for stubborn clients who don’t want to do the work and expect you to perform magic. Feel free to use it:

“Consultant makes no warranty, express, implied or statutory, with respect to the services provided hereunder, including without limitation any implied warranty of reliability, usefulness, merchantability, fitness for a particular purpose, noninfringement, or those arising from the course of performance, dealing, usage or trade. By signing this agreement, you acknowledge that Consultant neither owns nor governs the actions of any search engine or the Customer’s full implementations of recommendations provided by Consultant. You also acknowledge that due to non-responsibility over full implementations, fluctuations in the relative competitiveness of some search terms, recurring changes in search engine algorithms and other competitive factors, it is impossible to guarantee number one rankings or consistent top ten rankings, or any other specific search engines rankings, traffic or performance.”

Go get 'em!

The way you approach a new SEO client or project is critical to setting yourself up for success. And I believe we can all learn from each other’s experiences. Have you thought outside the SEO standards box to find success with any of your clients or projects? Please share in the comments!

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!

Better Than Basics: Custom-Tailoring Your SEO Approach

Posted by Laura.Lippay

Just like people, websites come in all shapes and sizes. They’re different ages, with different backgrounds, histories, motivations, and resources at hand. So when it comes to approaching SEO for a site, one-size-fits-all best practices are typically not the most effective way to go about it (also, you’re better than that).

An analogy might be if you were a fitness coach. You have three clients. One is a 105lb high school kid who wants to beef up a little. One is a 65-year-old librarian who wants better heart health. One is a heavyweight lumberjack who’s working to be the world’s top springboard chopper. Would you consider giving each of them the same diet and workout routine? Probably not. You’re probably going to:

1. Learn all you can about their current diet, health, and fitness situations.
2. Come up with the best approach and the best tactics for each situation.
3. Test your way into it and optimize, as you learn what works and what doesn’t.

In SEO, consider how your priorities might be different if you saw similar symptoms — let’s say problems ranking anything on the first page — for:

1. New sites vs existing sites
2. New content vs older content
3. Enterprise vs small biz
4. Local vs global
5. Type of market — for example, a news site, e-commerce site, photo pinning, or a parenting community

A new site might need more sweat equity or have previous domain spam issues, while an older site might have years of technical mess to clean up. New content may need the right promotional touch while old content might just simply be stale. The approach for enterprise is often, at its core, about getting different parts of the organization to work together on things they don’t normally do, while the approach for small biz is usually more scrappy and entrepreneurial.

With the lack of trust in SEO today, people want to know if you can actually help them and how. Getting to know the client or project intimately and proposing custom solutions shows that you took the time to get to know the details and can suggest an effective way forward. And let’s not forget that your SEO game plan isn’t just important for the success of the client — it’s important for building your own successes, trust, and reputation in this niche industry.

How to customize an approach for a proposal

Do: Listen first

Begin by asking questions. Learn as much as you can about the situation at hand, the history, the competition, resources, budget, timeline, etc. Maybe even sleep on it and ask more questions before you provide a proposal for your approach.

Consider the fitness trainer analogy again. Now that you’ve asked questions, you know that the high school kid is already at the gym on a regular basis and is overeating junk food in his attempt to beef up. The librarian has been on a low-salt paleo diet since her heart attack a few years ago, and knows she knows she needs to exercise but refuses to set foot in a gym. The lumberjack is simply a couch potato.

Now that you know more, you can really tailor a proposed approach that might appeal to your potential client and allow you and the client to see how you might reach some initial successes.

Do: Understand business priorities.

What will fly? What won’t fly? What can we push for and what’s off the table? Even if you feel strongly about particular tactics, if you can’t shape your work within a client’s business priorities you may have no client at all.

Real-world example:

Site A wanted to see how well they could rank against their biggest content-heavy SERP competitors like Wikipedia but wanted to keep a sleek, content-light experience. Big-brand SEO vendors working for Site A pushed general, content-heavy SEO best practices. Because Site A wanted solutions that fit into their current workload along with a sleek, content-light experience, they pushed back.

The vendors couldn’t keep the client because they weren’t willing to get into the clients workload groove and go beyond general best practices. They didn’t listen to and work within the client’s specific business objectives.

Site A hired internal SEO resources and tested into an amount of content that they were comfortable with, in sync with technical optimization and promotional SEO tactics, and saw rankings slowly improve. Wikipedia and the other content-heavy sites are still sometimes outranking Site A, but Site A is now a stronger page one competitor, driving more traffic and leads, and can make the decision from here whether it’s worth it to continue to stay content-light or ramp up even more to get top 3 rankings more often.

The vendors weren’t necessarily incorrect in suggesting going content-heavy for the purpose of competitive ranking, but they weren’t willing to find the middle ground to test into light content first, and they lost a big brand client. At its current state, Site A could ramp up content even more, but gobs of text doesn’t fit the sleek brand image and it’s not proven that it would be worth the engineering maintenance costs for that particular site — a very practical, “not everything in SEO is most important all the time” approach.

Do: Find the momentum

It’s easiest to inject SEO where there’s already momentum into a business running full-speed ahead. Are there any opportunities to latch onto an effort that’s just getting underway? This may be more important than your typical best practice priorities.

Real-world example:

Brand X had 12–20 properties (websites) at any given time, but their small SEO team could only manage about 3 at a time. Therefore the SEO team had to occasionally assess which properties they would be working with. Properties were chosen based on:

1. Which ones have the biggest need or opportunities?
2. Which ones have resources that they’re willing to dedicate?
3. Which ones are company priorities?

#2 was important. Without it, the idea that one of the properties might have the biggest search traffic opportunity didn’t matter if they had no resources to dedicate to implement the SEO team’s recommendations.

Similarly, in the first example above, the vendors weren’t able to go with the client’s workflow and lost the client. Make sure you’re able to identify which wheels are moving that you can take advantage of now, in order to get things done. There may be some tactics that will have higher impact, but if the client isn’t ready or willing to do them right now, you’re pushing a boulder uphill.

Do: Understand the competitive landscape

What is this site up against? What is the realistic chance they can compete? Knowing what the competitive landscape looks like, how will that influence your approach?

Real-world example:

Site B has a section of pages competing against old, strong, well-known, content-heavy, link-rich sites. Since it’s a new site section, almost everything needs to be done for Site B — technical optimization, building content, promotion, and generating links. However, the nature of this competitive landscape shows us that being first to publish might be important here. Site B’s competitors oftentimes have content out weeks if not months before the actual content brand owner (Site B). How? By staying on top of Site B’s press releases. The competitors created landing pages immediately after Site B put out a press release, while Site B didn’t have a landing page until the product actually launched. Once this was realized, being first to publish became an important factor. And because Site B is an enterprise site, and changing that process takes time internally, other technical and content optimization for the page templates happened concurrently, so that there was at least the minimal technical optimization and content on these pages by the time the process for first-publishing was shaped.

Site B is now generating product landing pages at the time of press release, with links to the landing pages in those press releases that are picked up by news outlets, giving Site B the first page and the first links, and this is generating more links than their top competitor in the first 7 days 80% of the time.

Site B didn’t audit the site and suggest tactics by simply checking off a list of technical optimizations prioritized by an SEO tool or ranking factors, but instead took a more calculated approach based on what’s happening in the competitive landscape, combined with the top prioritized technical and content optimizations. Optimizing the site itself without understanding the competitive landscape in this case would be leaving the competitors, who also have optimized sites with a lot of content, a leg up because they were cited (linked to) and picked up by Google first.

Do: Ask what has worked and hasn’t worked before

Asking this question can be very informative and help to drill down on areas that might be a more effective use of time. If the site has been around for a while, and especially if they already have an SEO working with them, try to find out what they’ve already done that has worked and that hasn’t worked to give you clues on what approaches might be successful or not..

General example:

Site C has hundreds, sometimes thousands of internal cross-links on their pages, very little unique text content, and doesn’t see as much movement for cross-linking projects as they do when adding unique text.

Site D knows from previous testing that generating more keyword-rich content on their landing pages hasn’t been as effective as implementing better cross-linking, especially since there is very little cross-linking now.

Therefore each of these sites should be prioritizing text and cross-linking tactics differently. Be sure to ask the client or potential client about previous tests or ranking successes and failures in order to learn what tactics may be more relevant for this site before you suggest and prioritize your own.

Do: Make sure you have data

Ask the client what they’re using to monitor performance. If they do not have the basics, suggest setting it up or fold that into your proposal as a first step. Define what data essentials you need to analyze the site by asking the client about their goals, walking through how to measure those goals with them, and then determining the tools and analytics setup you need. Those essentials might be something like:

• Webmaster tools set up. I like to have at least Google and Bing, so I can compare across search engines to help determine if a spike or a drop is happening in both search engines, which might indicate that the cause is from something happening with the site, or in just one search engine, which might indicate that the cause is algo-related.
• Organic search engine traffic. At the very least, you should be able to see organic search traffic by page type (ex: service pages versus product pages). At best, you can also filter by things like URL structure, country, date, referrers/source and be able to run regex queries for granularity.
• User testing & focus groups. Optional, but useful if it’s available & can help prioritization. Has the site gathered any insights from users that could be helpful in deciding on and prioritizing SEO tactics? For example, focus groups on one site showed us that people were more likely to convert if they could see a certain type of content that wouldn’t have necessarily been a priority for SEO otherwise. If they’re more likely to convert, they’re less likely to bounce back to search results, so adding that previously lower-priority content could have double advantages for the site: higher conversions and lower bounce rate back to SERPs.

Don’t: Make empty promises.

Put simply, please, SEOs, do not blanket promise anything. Hopeful promises leads to SEOs being called snake oil salesmen. This is a real problem for all of us, and you can help turn it around.

Clients and managers will try to squeeze you until you break and give them a number or a promised rank. Don’t do it. This is like a new judoka asking the coach to promise they’ll make it to the Olympics if they sign up for the program. The level of success depends on what the judoka puts into it, what her competition looks like, what is her tenacity for courage, endurance, competition, resistance… You promise, she signs up, says “Oh, this takes work so I’m only going to come to practice on Saturdays,” and everybody loses.

Goals are great. Promises are trouble. Good contracts are imperative.

Here are some examples:

• We will get you to page 1. No matter how successful you may have been in the past, every site, competitive landscape, and team behind the site is a different challenge. A promise of #1 rankings may be a selling point to get clients, but can you live up to it? What will happen to your reputation of not? This industry is small enough that word gets around when people are not doing right by their clients.
• Rehashing vague stats. I recently watched a well-known agency tell a room full of SEOs: “The search result will provide in-line answers for 47% of your customer queries”. Obviously this isn’t going to be true for every SEO in the room, since different types of queries have different SERPS, and the SERP UI constantly changes, but how many of the people in that room went back to their companies and their clients and told them that? What happens to those SEOs if that doesn’t prove true?
• We will increase traffic by n%. Remember, hopeful promises can lead to being called snake oil salesmen. If you can avoid performance promises, especially in the proposal process, by all means please do. Set well-informed goals rather than high-risk promises, and be conservative when you can. It always looks better to over-perform than to not reach a goal.
• You will definitely see improvement. Honestly, I wouldn’t even promise this unless you would *for real* bet your life on it. You may see plenty of opportunities for optimization but you can’t be sure they’ll implement anything, they’ll implement things correctly, implementations will not get overwritten, competitors won’t step it up or new ones rise, or that the optimization opportunities you see will even work on this site.

Don’t: Use the same proposal for every situation at hand.

If your proposal is so vague that it might actually seem to apply to any site, then you really should consider taking a deeper look at each situation at hand before you propose.

Would you want your doctor to prescribe the same thing for your (not yet known) pregnancy as the next person’s (not yet known) fungal blood infection, when you both just came in complaining of fatigue?

Do: Cover yourself in your contract

As a side note for consultants, this is a clause I include in my contract with clients for protection against being sued if clients aren’t happy with their results. It’s especially helpful for stubborn clients who don’t want to do the work and expect you to perform magic. Feel free to use it:

“Consultant makes no warranty, express, implied or statutory, with respect to the services provided hereunder, including without limitation any implied warranty of reliability, usefulness, merchantability, fitness for a particular purpose, noninfringement, or those arising from the course of performance, dealing, usage or trade. By signing this agreement, you acknowledge that Consultant neither owns nor governs the actions of any search engine or the Customer’s full implementations of recommendations provided by Consultant. You also acknowledge that due to non-responsibility over full implementations, fluctuations in the relative competitiveness of some search terms, recurring changes in search engine algorithms and other competitive factors, it is impossible to guarantee number one rankings or consistent top ten rankings, or any other specific search engines rankings, traffic or performance.”

Go get 'em!

The way you approach a new SEO client or project is critical to setting yourself up for success. And I believe we can all learn from each other’s experiences. Have you thought outside the SEO standards box to find success with any of your clients or projects? Please share in the comments!

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!

Using the Flowchart Method for Diagnosing Ranking Drops - Whiteboard Friday

Posted by KameronJenkins

Being able to pinpoint the reason for a ranking drop is one of our most perennial and potentially frustrating tasks as SEOs. There are an unknowable number of factors that go into ranking these days, but luckily the methodology for diagnosing those fluctuations is readily at hand. In today's Whiteboard Friday, we welcome the wonderful Kameron Jenkins to show us a structured way to diagnose ranking drops using a flowchart method and critical thinking.

Click on the whiteboard image above to open a high-resolution version in a new tab!

Video Transcription

Hey, everyone. Welcome to this week's edition of Whiteboard Friday. My name is Kameron Jenkins. I am the new SEO Wordsmith here at Moz, and I'm so excited to be here. Before this, I worked at an agency for about six and a half years. I worked in the SEO department, and really a common thing we encountered was a client's rankings dropped. What do we do?

This flowchart was kind of built out of that mentality of we need a logical workflow to be able to diagnose exactly what happened so we can make really pointed recommendations for how to fix it, how to get our client's rankings back. So let's dive right in. It's going to be a flowchart, so it's a little nonlinear, but hopefully this makes sense and helps you work smarter rather than harder.

Was it a major ranking drop?: No

The first question I'd want to ask is: Was their rankings drop major? By major, I would say that's something like page 1 to page 5 overnight. Minor would be something like it just fell a couple positions, like position 3 to position 5.

We're going to take this path first. It was minor.

Has there been a pattern of decline lasting about a month or greater?

That's not a magic number. A month is something that you can use as a benchmark. But if there's been a steady decline and it's been one week it's position 3 and then it's position 5 and then position 7, and it just keeps dropping over time, I would consider that a pattern of decline.

So if no, I would actually say wait.

• Volatility is normal, especially if you're at the bottom of page 1, maybe page 2 plus. There's going to be a lot more shifting of the search results in those positions. So volatility is normal.
• Keep your eyes on it, though. It's really good to just take note of it like, "Hey, we dropped. Okay, I'm going to check that again next week and see if it continues to drop, then maybe we'll take action."
• Wait it out. At this point, I would just caution against making big website updates if it hasn't really been warranted yet. So volatility is normal. Expect that. Keep your finger on the pulse, but just wait it out at this point.

If there has been a pattern of decline though, I'm going to have you jump to the algorithm update section. We're going to get there in a second. But for now, we're going to go take the major rankings drop path.

Was it a major ranking drop?: Yes

The first question on this path that I'd want to ask is:

Was there a rank tracking issue?

Now, some of these are going seem pretty basic, like how would that ever happen, but believe me it happens every once in a while. So just before we make major updates to the website, I'd want to check the rank tracking.

I. The wrong domain or URL.

That can be something that happens a lot. A site maybe you change domains or maybe you move a page and that old page of that old domain is still listed in your ranking tracker. If that's the case, then the rank tracking tool doesn't know which URL to judge the rankings off of. So it's going to look like maybe you dropped to position 10 overnight from position 1, and that's like, whoa, that's a huge update. But it's actually just that you have the wrong URL in there. So just check that. If there's been a page update, a domain update, check to make sure that you've updated your rank tracker.

II. Glitches.

So it's software, it can break. There are things that could cause it to be off for whatever reason. I don't know how common that is. It probably is totally dependent on which kind of software you use. But glitches do happen, so I would manually check your rankings.

III. Manually check rankings.

One way I would do that is...

• Go to incognito in Google and make sure you're logged out so it's not personalized. I would search the term that you're wanting to rank for and see where you're actually ranking.
• Google's Ad Preview tool. That one is really good too if you want to search where you're ranking locally so you can set your geolocation. You could do mobile versus desktop rankings. So it could be really good for things like that.
• Crosscheck with another tool, like Moz's tool for rank tracking. You can pop in your URLs, see where you're ranking, and cross-check that with your own tool.

So back to this. Rank tracking issues. Yes, you found your problem. If it was just a rank tracking tool issue, that's actually great, because it means you don't have to make a lot of changes. Your rankings actually haven't dropped. But if that's not the issue, if there is no rank tracking issue that you can pinpoint, then I would move on to Google Search Console.

Problems in Google Search Console?

So Google Search Console is really helpful for checking site health matters. One of the main things I would want to check in there, if you experience a major drop especially, is...

I. Manual actions.

If you navigate to Manual Actions, you could see notes in there like unnatural links pointing to your site. Or maybe you have thin or low-quality content on your site. If those things are present in your Manual Actions, then you have a reference point. You have something to go off of. There's a lot of work involved in lifting a manual penalty that we can't get into here unfortunately. Some things that you can do to focus on manual penalty lifting...

• Moz's Link Explorer. You can check your inbound links and see their spam score. You could look at things like anchor text to see if maybe the links pointing to your site are keyword stuffed. So you can use tools like that.
• There are a lot of good articles too, in the industry, just on getting penalties lifted. Marie Haynes especially has some really good ones. So I would check that out.

But you have found your problem if there's a manual action in there. So focus on getting that penalty lifted.

II. Indexation issues.

Before you move out of Search Console, though, I would check indexation issues as well. Maybe you don't have a manual penalty. But go to your index coverage report and you can see if anything you submitted in your sitemap is maybe experiencing issues. Maybe it's blocked by robots.txt, or maybe you accidentally no indexed it. You could probably see that in the index coverage report. Search Console, okay. So yes, you found your problem. No, you're going to move on to algorithm updates.

Algorithm updates

Algorithm updates happen all the time. Google says that maybe one to two happen per day. Not all of those are going to be major. The major ones, though, are listed. They're documented in multiple different places. Moz has a really good list of algorithm updates over time. You can for sure reference that. There are going to be a lot of good ones. You can navigate to the exact year and month that your site experienced a rankings drop and see if it maybe correlates with any algorithm update.

For example, say your site lost rankings in about January 2017. That's about the time that Google released its Intrusive Interstitials Update, and so I would look on my site, if that was the issue, and say, "Do I have intrusive interstitials? Is this something that's affecting my website?"

If you can match up an algorithm update with the time that your rankings started to drop, you have direction. You found an issue. If you can't match it up to any algorithm updates, it's finally time to move on to site updates.

Site updates

What changes happened to your website recently? There are a lot of different things that could have happened to your website. Just keep in mind too that maybe you're not the only one who has access to your website. You're the SEO, but maybe tech support has access. Maybe even your paid ad manager has access. There are a lot of different people who could be making changes to the website. So just keep that in mind when you're looking into it. It's not just the changes that you made, but changes that anyone made could affect the website's ranking. Just look into all possible factors.

Other factors that can impact rankings

A lot of different things, like I said, can influence your site's rankings. A lot of things can inadvertently happen that you can pinpoint and say, "Oh, that's definitely the cause."

Some examples of things that I've personally experienced on my clients' websites...

I. Renaming pages and letting them 404 without updating with a 301 redirect.

There was one situation where a client had a blog. They had hundreds of really good blog posts. They were all ranking for nice, long tail terms. A client emailed into tech support to change the name of the blog. Unfortunately, all of the posts lived under the blog, and when he did that, he didn't update it with a 301 redirect, so all of those pages, that were ranking really nicely, they started to fall out of the index. The rankings went with it. There's your problem. It was unfortunate, but at least we were able to diagnose what happened.

II. Content cutting.

Maybe you're working with a UX team, a design team, someone who is looking at the website from a visual, a user experience perspective. A lot of times in these situations they might take a page that's full of really good, valuable content and they might say, "Oh, this is too clunky. It's too bulky. It has too many words. So we're going to replace it with an image, or we're going to take some of the content out."

When this happens, if the content was the thing that was making your page rank and you cut that, that's probably something that's going to affect your rankings negatively. By the way, if that's happening to you, Rand has a really good Whiteboard Friday on kind of how to marry user experience and SEO. You should definitely check that out if that's an issue for you.

III. Valuable backlinks lost.

Another situation I was diagnosing a client and one of their backlinks dropped. It just so happened to be like the only thing that changed over this course of time. It was a really valuable backlink, and we found out that they just dropped it for whatever reason, and the client's rankings started to decline after that time. Things like Moz's tools, Link Explorer, you can go in there and see gained and lost backlinks over time. So I would check that out if maybe that might be an issue for you.

IV. Accidental no index.

Depending on what type of CMS you work with, it might be really, really easy to accidentally check No Index on this page. If you no index a really important page, Google takes it out of its index. That could happen. Your rankings could drop.So those are just some examples of things that can happen. Like I said, hundreds and hundreds of things could have been changed on your site, but it's just really important to try to pinpoint exactly what those changes were and if they coincided with when your rankings started to drop.

SERP landscape

So we got all the way to the bottom. If you're at the point where you've looked at all of the site updates and you still haven't found anything that would have caused a rankings drop, I would say finally look at the SERP landscape.

What I mean by that is just Google your keyword that you want to rank for or your group of keywords that you want to rank for and see which websites are ranking on page 1. I would get a lay of the land and just see:

• What are these pages doing?
• How many backlinks do they have?
• How much content do they have?
• Do they load fast?
• What's the experience?

Then make content better than that. To rank, so many people just think avoid being spammy and avoid having things broken on your site. But that's not SEO. That's really just helping you be able to compete. You have to have content that's the best answer to searchers' questions, and that's going to get you ranking.

I hope that was helpful. This is a really good way to just kind of work through a ranking drop diagnosis. If you have methods, by the way, that work for you, I'd love to hear from you and see what worked for you in the past. Let me know, drop it in the comments below.

Thanks, everyone. Come back next week for another edition of Whiteboard Friday.

Video transcription by Speechpad.com

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!

Using the Flowchart Method for Diagnosing Ranking Drops - Whiteboard Friday

Posted by KameronJenkins

Being able to pinpoint the reason for a ranking drop is one of our most perennial and potentially frustrating tasks as SEOs. There are an unknowable number of factors that go into ranking these days, but luckily the methodology for diagnosing those fluctuations is readily at hand. In today's Whiteboard Friday, we welcome the wonderful Kameron Jenkins to show us a structured way to diagnose ranking drops using a flowchart method and critical thinking.

Click on the whiteboard image above to open a high-resolution version in a new tab!

Video Transcription

Hey, everyone. Welcome to this week's edition of Whiteboard Friday. My name is Kameron Jenkins. I am the new SEO Wordsmith here at Moz, and I'm so excited to be here. Before this, I worked at an agency for about six and a half years. I worked in the SEO department, and really a common thing we encountered was a client's rankings dropped. What do we do?

This flowchart was kind of built out of that mentality of we need a logical workflow to be able to diagnose exactly what happened so we can make really pointed recommendations for how to fix it, how to get our client's rankings back. So let's dive right in. It's going to be a flowchart, so it's a little nonlinear, but hopefully this makes sense and helps you work smarter rather than harder.

Was it a major ranking drop?: No

The first question I'd want to ask is: Was their rankings drop major? By major, I would say that's something like page 1 to page 5 overnight. Minor would be something like it just fell a couple positions, like position 3 to position 5.

We're going to take this path first. It was minor.

Has there been a pattern of decline lasting about a month or greater?

That's not a magic number. A month is something that you can use as a benchmark. But if there's been a steady decline and it's been one week it's position 3 and then it's position 5 and then position 7, and it just keeps dropping over time, I would consider that a pattern of decline.

So if no, I would actually say wait.

• Volatility is normal, especially if you're at the bottom of page 1, maybe page 2 plus. There's going to be a lot more shifting of the search results in those positions. So volatility is normal.
• Keep your eyes on it, though. It's really good to just take note of it like, "Hey, we dropped. Okay, I'm going to check that again next week and see if it continues to drop, then maybe we'll take action."
• Wait it out. At this point, I would just caution against making big website updates if it hasn't really been warranted yet. So volatility is normal. Expect that. Keep your finger on the pulse, but just wait it out at this point.

If there has been a pattern of decline though, I'm going to have you jump to the algorithm update section. We're going to get there in a second. But for now, we're going to go take the major rankings drop path.

Was it a major ranking drop?: Yes

The first question on this path that I'd want to ask is:

Was there a rank tracking issue?

Now, some of these are going seem pretty basic, like how would that ever happen, but believe me it happens every once in a while. So just before we make major updates to the website, I'd want to check the rank tracking.

I. The wrong domain or URL.

That can be something that happens a lot. A site maybe you change domains or maybe you move a page and that old page of that old domain is still listed in your ranking tracker. If that's the case, then the rank tracking tool doesn't know which URL to judge the rankings off of. So it's going to look like maybe you dropped to position 10 overnight from position 1, and that's like, whoa, that's a huge update. But it's actually just that you have the wrong URL in there. So just check that. If there's been a page update, a domain update, check to make sure that you've updated your rank tracker.

II. Glitches.

So it's software, it can break. There are things that could cause it to be off for whatever reason. I don't know how common that is. It probably is totally dependent on which kind of software you use. But glitches do happen, so I would manually check your rankings.

III. Manually check rankings.

One way I would do that is...

• Go to incognito in Google and make sure you're logged out so it's not personalized. I would search the term that you're wanting to rank for and see where you're actually ranking.
• Google's Ad Preview tool. That one is really good too if you want to search where you're ranking locally so you can set your geolocation. You could do mobile versus desktop rankings. So it could be really good for things like that.
• Crosscheck with another tool, like Moz's tool for rank tracking. You can pop in your URLs, see where you're ranking, and cross-check that with your own tool.

So back to this. Rank tracking issues. Yes, you found your problem. If it was just a rank tracking tool issue, that's actually great, because it means you don't have to make a lot of changes. Your rankings actually haven't dropped. But if that's not the issue, if there is no rank tracking issue that you can pinpoint, then I would move on to Google Search Console.

Problems in Google Search Console?

So Google Search Console is really helpful for checking site health matters. One of the main things I would want to check in there, if you experience a major drop especially, is...

I. Manual actions.

If you navigate to Manual Actions, you could see notes in there like unnatural links pointing to your site. Or maybe you have thin or low-quality content on your site. If those things are present in your Manual Actions, then you have a reference point. You have something to go off of. There's a lot of work involved in lifting a manual penalty that we can't get into here unfortunately. Some things that you can do to focus on manual penalty lifting...

• Moz's Link Explorer. You can check your inbound links and see their spam score. You could look at things like anchor text to see if maybe the links pointing to your site are keyword stuffed. So you can use tools like that.
• There are a lot of good articles too, in the industry, just on getting penalties lifted. Marie Haynes especially has some really good ones. So I would check that out.

But you have found your problem if there's a manual action in there. So focus on getting that penalty lifted.

II. Indexation issues.

Before you move out of Search Console, though, I would check indexation issues as well. Maybe you don't have a manual penalty. But go to your index coverage report and you can see if anything you submitted in your sitemap is maybe experiencing issues. Maybe it's blocked by robots.txt, or maybe you accidentally no indexed it. You could probably see that in the index coverage report. Search Console, okay. So yes, you found your problem. No, you're going to move on to algorithm updates.

Algorithm updates

Algorithm updates happen all the time. Google says that maybe one to two happen per day. Not all of those are going to be major. The major ones, though, are listed. They're documented in multiple different places. Moz has a really good list of algorithm updates over time. You can for sure reference that. There are going to be a lot of good ones. You can navigate to the exact year and month that your site experienced a rankings drop and see if it maybe correlates with any algorithm update.

For example, say your site lost rankings in about January 2017. That's about the time that Google released its Intrusive Interstitials Update, and so I would look on my site, if that was the issue, and say, "Do I have intrusive interstitials? Is this something that's affecting my website?"

If you can match up an algorithm update with the time that your rankings started to drop, you have direction. You found an issue. If you can't match it up to any algorithm updates, it's finally time to move on to site updates.

Site updates

What changes happened to your website recently? There are a lot of different things that could have happened to your website. Just keep in mind too that maybe you're not the only one who has access to your website. You're the SEO, but maybe tech support has access. Maybe even your paid ad manager has access. There are a lot of different people who could be making changes to the website. So just keep that in mind when you're looking into it. It's not just the changes that you made, but changes that anyone made could affect the website's ranking. Just look into all possible factors.

Other factors that can impact rankings

A lot of different things, like I said, can influence your site's rankings. A lot of things can inadvertently happen that you can pinpoint and say, "Oh, that's definitely the cause."

Some examples of things that I've personally experienced on my clients' websites...

I. Renaming pages and letting them 404 without updating with a 301 redirect.

There was one situation where a client had a blog. They had hundreds of really good blog posts. They were all ranking for nice, long tail terms. A client emailed into tech support to change the name of the blog. Unfortunately, all of the posts lived under the blog, and when he did that, he didn't update it with a 301 redirect, so all of those pages, that were ranking really nicely, they started to fall out of the index. The rankings went with it. There's your problem. It was unfortunate, but at least we were able to diagnose what happened.

II. Content cutting.

Maybe you're working with a UX team, a design team, someone who is looking at the website from a visual, a user experience perspective. A lot of times in these situations they might take a page that's full of really good, valuable content and they might say, "Oh, this is too clunky. It's too bulky. It has too many words. So we're going to replace it with an image, or we're going to take some of the content out."

When this happens, if the content was the thing that was making your page rank and you cut that, that's probably something that's going to affect your rankings negatively. By the way, if that's happening to you, Rand has a really good Whiteboard Friday on kind of how to marry user experience and SEO. You should definitely check that out if that's an issue for you.

III. Valuable backlinks lost.

Another situation I was diagnosing a client and one of their backlinks dropped. It just so happened to be like the only thing that changed over this course of time. It was a really valuable backlink, and we found out that they just dropped it for whatever reason, and the client's rankings started to decline after that time. Things like Moz's tools, Link Explorer, you can go in there and see gained and lost backlinks over time. So I would check that out if maybe that might be an issue for you.

IV. Accidental no index.

Depending on what type of CMS you work with, it might be really, really easy to accidentally check No Index on this page. If you no index a really important page, Google takes it out of its index. That could happen. Your rankings could drop.So those are just some examples of things that can happen. Like I said, hundreds and hundreds of things could have been changed on your site, but it's just really important to try to pinpoint exactly what those changes were and if they coincided with when your rankings started to drop.

SERP landscape

So we got all the way to the bottom. If you're at the point where you've looked at all of the site updates and you still haven't found anything that would have caused a rankings drop, I would say finally look at the SERP landscape.

What I mean by that is just Google your keyword that you want to rank for or your group of keywords that you want to rank for and see which websites are ranking on page 1. I would get a lay of the land and just see:

• What are these pages doing?
• How many backlinks do they have?
• How much content do they have?
• Do they load fast?
• What's the experience?

Then make content better than that. To rank, so many people just think avoid being spammy and avoid having things broken on your site. But that's not SEO. That's really just helping you be able to compete. You have to have content that's the best answer to searchers' questions, and that's going to get you ranking.

I hope that was helpful. This is a really good way to just kind of work through a ranking drop diagnosis. If you have methods, by the way, that work for you, I'd love to hear from you and see what worked for you in the past. Let me know, drop it in the comments below.

Thanks, everyone. Come back next week for another edition of Whiteboard Friday.

Video transcription by Speechpad.com

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!

Rewriting the Beginner's Guide to SEO, Chapter 2: Crawling, Indexing, and Ranking

Posted by BritneyMuller

It's been a few months since our last share of our work-in-progress rewrite of the Beginner's Guide to SEO, but after a brief hiatus, we're back to share our draft of Chapter Two with you! This wouldn’t have been possible without the help of Kameron Jenkins, who has thoughtfully contributed her great talent for wordsmithing throughout this piece.

This is your resource, the guide that likely kicked off your interest in and knowledge of SEO, and we want to do right by you. You left amazingly helpful commentary on our outline and draft of Chapter One, and we'd be honored if you would take the time to let us know what you think of Chapter Two in the comments below.

---

Chapter 2: How Search Engines Work – Crawling, Indexing, and Ranking

First, show up.

As we mentioned in Chapter 1, search engines are answer machines. They exist to discover, understand, and organize the internet's content in order to offer the most relevant results to the questions searchers are asking.

In order to show up in search results, your content needs to first be visible to search engines. It's arguably the most important piece of the SEO puzzle: If your site can't be found, there's no way you'll ever show up in the SERPs (Search Engine Results Page).

How do search engines work?

Search engines have three primary functions:

1. Crawl: Scour the Internet for content, looking over the code/content for each URL they find.
2. Index: Store and organize the content found during the crawling process. Once a page is in the index, it’s in the running to be displayed as a result to relevant queries.
3. Rank: Provide the pieces of content that will best answer a searcher's query. Order the search results by the most helpful to a particular query.

What is search engine crawling?

Crawling, is the discovery process in which search engines send out a team of robots (known as crawlers or spiders) to find new and updated content. Content can vary — it could be a webpage, an image, a video, a PDF, etc. — but regardless of the format, content is discovered by links.

The bot starts out by fetching a few web pages, and then follows the links on those webpages to find new URLs. By hopping along this path of links, crawlers are able to find new content and add it to their index — a massive database of discovered URLs — to later be retrieved when a searcher is seeking information that the content on that URL is a good match for.

What is a search engine index?

Search engines process and store information they find in an index, a huge database of all the content they’ve discovered and deem good enough to serve up to searchers.

Search engine ranking

When someone performs a search, search engines scour their index for highly relevant content and then orders that content in the hopes of solving the searcher's query. This ordering of search results by relevance is known as ranking. In general, you can assume that the higher a website is ranked, the more relevant the search engine believes that site is to the query.

It’s possible to block search engine crawlers from part or all of your site, or instruct search engines to avoid storing certain pages in their index. While there can be reasons for doing this, if you want your content found by searchers, you have to first make sure it’s accessible to crawlers and is indexable. Otherwise, it’s as good as invisible.

By the end of this chapter, you’ll have the context you need to work with the search engine, rather than against it!

Note: In SEO, not all search engines are equal

Many beginners wonder about the relative importance of particular search engines. Most people know that Google has the largest market share, but how important it is to optimize for Bing, Yahoo, and others? The truth is that despite the existence of more than 30 major web search engines, the SEO community really only pays attention to Google. Why? The short answer is that Google is where the vast majority of people search the web. If we include Google Images, Google Maps, and YouTube (a Google property), more than 90% of web searches happen on Google — that's nearly 20 times Bing and Yahoo combined.

Crawling: Can search engines find your site?

As you've just learned, making sure your site gets crawled and indexed is a prerequisite for showing up in the SERPs. First things first: You can check to see how many and which pages of your website have been indexed by Google using "site:yourdomain.com", an advanced search operator.

Head to Google and type "site:yourdomain.com" into the search bar. This will return results Google has in its index for the site specified:

The number of results Google displays (see “About __ results” above) isn't exact, but it does give you a solid idea of which pages are indexed on your site and how they are currently showing up in search results.

For more accurate results, monitor and use the Index Coverage report in Google Search Console. You can sign up for a free Google Search Console account if you don't currently have one. With this tool, you can submit sitemaps for your site and monitor how many submitted pages have actually been added to Google's index, among other things.

If you're not showing up anywhere in the search results, there are a few possible reasons why:

• Your site is brand new and hasn't been crawled yet.
• Your site isn't linked to from any external websites.
• Your site's navigation makes it hard for a robot to crawl it effectively.
• Your site contains some basic code called crawler directives that is blocking search engines.
• Your site has been penalized by Google for spammy tactics.

If your site doesn't have any other sites linking to it, you still might be able to get it indexed by submitting your XML sitemap in Google Search Console or manually submitting individual URLs to Google. There's no guarantee they'll include a submitted URL in their index, but it's worth a try!

Can search engines see your whole site?

Sometimes a search engine will be able to find parts of your site by crawling, but other pages or sections might be obscured for one reason or another. It's important to make sure that search engines are able to discover all the content you want indexed, and not just your homepage.

Ask yourself this: Can the bot crawl through your website, and not just to it?

Is your content hidden behind login forms?

If you require users to log in, fill out forms, or answer surveys before accessing certain content, search engines won't see those protected pages. A crawler is definitely not going to log in.

Are you relying on search forms?

Robots cannot use search forms. Some individuals believe that if they place a search box on their site, search engines will be able to find everything that their visitors search for.

Is text hidden within non-text content?

Non-text media forms (images, video, GIFs, etc.) should not be used to display text that you wish to be indexed. While search engines are getting better at recognizing images, there's no guarantee they will be able to read and understand it just yet. It's always best to add text within the <HTML> markup of your webpage.

Can search engines follow your site navigation?

Just as a crawler needs to discover your site via links from other sites, it needs a path of links on your own site to guide it from page to page. If you’ve got a page you want search engines to find but it isn’t linked to from any other pages, it’s as good as invisible. Many sites make the critical mistake of structuring their navigation in ways that are inaccessible to search engines, hindering their ability to get listed in search results.

Common navigation mistakes that can keep crawlers from seeing all of your site:

• Having a mobile navigation that shows different results than your desktop navigation
• Any type of navigation where the menu items are not in the HTML, such as JavaScript-enabled navigations. Google has gotten much better at crawling and understanding Javascript, but it’s still not a perfect process. The more surefire way to ensure something gets found, understood, and indexed by Google is by putting it in the HTML.
• Personalization, or showing unique navigation to a specific type of visitor versus others, could appear to be cloaking to a search engine crawler
• Forgetting to link to a primary page on your website through your navigation — remember, links are the paths crawlers follow to new pages!

This is why it's essential that your website has a clear navigation and helpful URL folder structures.

Information architecture

Information architecture is the practice of organizing and labeling content on a website to improve efficiency and fundability for users. The best information architecture is intuitive, meaning that users shouldn't have to think very hard to flow through your website or to find something.

Your site should also have a useful 404 (page not found) page for when a visitor clicks on a dead link or mistypes a URL. The best 404 pages allow users to click back into your site so they don’t bounce off just because they tried to access a nonexistent link.

Tell search engines how to crawl your site

In addition to making sure crawlers can reach your most important pages, it’s also pertinent to note that you’ll have pages on your site you don’t want them to find. These might include things like old URLs that have thin content, duplicate URLs (such as sort-and-filter parameters for e-commerce), special promo code pages, staging or test pages, and so on.

Blocking pages from search engines can also help crawlers prioritize your most important pages and maximize your crawl budget (the average number of pages a search engine bot will crawl on your site).

Crawler directives allow you to control what you want Googlebot to crawl and index using a robots.txt file, meta tag, sitemap.xml file, or Google Search Console.

Robots.txt

Robots.txt files are located in the root directory of websites (ex. yourdomain.com/robots.txt) and suggest which parts of your site search engines should and shouldn't crawl via specific robots.txt directives. This is a great solution when trying to block search engines from non-private pages on your site.

You wouldn't want to block private/sensitive pages from being crawled here because the file is easily accessible by users and bots.

Pro tip:

• If Googlebot can't find a robots.txt file for a site (40X HTTP status code), it proceeds to crawl the site.
• If Googlebot finds a robots.txt file for a site (20X HTTP status code), it will usually abide by the suggestions and proceed to crawl the site.
• If Googlebot finds neither a 20X or a 40X HTTP status code (ex. a 501 server error) it can't determine if you have a robots.txt file or not and won't crawl your site.

Meta directives

The two types of meta directives are the meta robots tag (more commonly used) and the x-robots-tag. Each provides crawlers with stronger instructions on how to crawl and index a URL's content.

The x-robots tag provides more flexibility and functionality if you want to block search engines at scale because you can use regular expressions, block non-HTML files, and apply sitewide noindex tags.

These are the best options for blocking more sensitive*/private URLs from search engines.

*For very sensitive URLs, it is best practice to remove them from or require a secure login to view the pages.

WordPress Tip: In Dashboard > Settings > Reading, make sure the "Search Engine Visibility" box is not checked. This blocks search engines from coming to your site via your robots.txt file!

Avoid these common pitfalls, and you'll have clean, crawlable content that will allow bots easy access to your pages.

Once you’ve ensured your site has been crawled, the next order of business is to make sure it can be indexed. That’s right — just because your site can be discovered and crawled by a search engine doesn’t necessarily mean that it will be stored in their index. Read on to learn about how indexing works and how you can make sure your site makes it into this all-important database.

Sitemaps

A sitemap is just what it sounds like: a list of URLs on your site that crawlers can use to discover and index your content. One of the easiest ways to ensure Google is finding your highest priority pages is to create a file that meets Google's standards and submit it through Google Search Console. While submitting a sitemap doesn’t replace the need for good site navigation, it can certainly help crawlers follow a path to all of your important pages.

Google Search Console

Some sites (most common with e-commerce) make the same content available on multiple different URLs by appending certain parameters to URLs. If you’ve ever shopped online, you’ve likely narrowed down your search via filters. For example, you may search for “shoes” on Amazon, and then refine your search by size, color, and style. Each time you refine, the URL changes slightly. How does Google know which version of the URL to serve to searchers? Google does a pretty good job at figuring out the representative URL on its own, but you can use the URL Parameters feature in Google Search Console to tell Google exactly how you want them to treat your pages.

Indexing: How do search engines understand and remember your site?

Once you’ve ensured your site has been crawled, the next order of business is to make sure it can be indexed. That’s right — just because your site can be discovered and crawled by a search engine doesn’t necessarily mean that it will be stored in their index. In the previous section on crawling, we discussed how search engines discover your web pages. The index is where your discovered pages are stored. After a crawler finds a page, the search engine renders it just like a browser would. In the process of doing so, the search engine analyzes that page's contents. All of that information is stored in its index.

Read on to learn about how indexing works and how you can make sure your site makes it into this all-important database.

Can I see how a Googlebot crawler sees my pages?

Yes, the cached version of your page will reflect a snapshot of the last time googlebot crawled it.

Google crawls and caches web pages at different frequencies. More established, well-known sites that post frequently like https://www.nytimes.com will be crawled more frequently than the much-less-famous website for Roger the Mozbot’s side hustle, http://www.rogerlovescupcakes.com (if only it were real…)

You can view what your cached version of a page looks like by clicking the drop-down arrow next to the URL in the SERP and choosing "Cached":

You can also view the text-only version of your site to determine if your important content is being crawled and cached effectively.

Are pages ever removed from the index?

Yes, pages can be removed from the index! Some of the main reasons why a URL might be removed include:

• The URL is returning a "not found" error (4XX) or server error (5XX) – This could be accidental (the page was moved and a 301 redirect was not set up) or intentional (the page was deleted and 404ed in order to get it removed from the index)
• The URL had a noindex meta tag added – This tag can be added by site owners to instruct the search engine to omit the page from its index.
• The URL has been manually penalized for violating the search engine’s Webmaster Guidelines and, as a result, was removed from the index.
• The URL has been blocked from crawling with the addition of a password required before visitors can access the page.

If you believe that a page on your website that was previously in Google’s index is no longer showing up, you can manually submit the URL to Google by navigating to the “Submit URL” tool in Search Console.

Ranking: How do search engines rank URLs?

How do search engines ensure that when someone types a query into the search bar, they get relevant results in return? That process is known as ranking, or the ordering of search results by most relevant to least relevant to a particular query.

To determine relevance, search engines use algorithms, a process or formula by which stored information is retrieved and ordered in meaningful ways. These algorithms have gone through many changes over the years in order to improve the quality of search results. Google, for example, makes algorithm adjustments every day — some of these updates are minor quality tweaks, whereas others are core/broad algorithm updates deployed to tackle a specific issue, like Penguin to tackle link spam. Check out our Google Algorithm Change History for a list of both confirmed and unconfirmed Google updates going back to the year 2000.

Why does the algorithm change so often? Is Google just trying to keep us on our toes? While Google doesn’t always reveal specifics as to why they do what they do, we do know that Google’s aim when making algorithm adjustments is to improve overall search quality. That’s why, in response to algorithm update questions, Google will answer with something along the lines of: “We’re making quality updates all the time.” This indicates that, if your site suffered after an algorithm adjustment, compare it against Google’s Quality Guidelines or Search Quality Rater Guidelines, both are very telling in terms of what search engines want.

What do search engines want?

Search engines have always wanted the same thing: to provide useful answers to searcher’s questions in the most helpful formats. If that’s true, then why does it appear that SEO is different now than in years past?

Think about it in terms of someone learning a new language.

At first, their understanding of the language is very rudimentary — “See Spot Run.” Over time, their understanding starts to deepen, and they learn semantics—- the meaning behind language and the relationship between words and phrases. Eventually, with enough practice, the student knows the language well enough to even understand nuance, and is able to provide answers to even vague or incomplete questions.

When search engines were just beginning to learn our language, it was much easier to game the system by using tricks and tactics that actually go against quality guidelines. Take keyword stuffing, for example. If you wanted to rank for a particular keyword like “funny jokes,” you might add the words “funny jokes” a bunch of times onto your page, and make it bold, in hopes of boosting your ranking for that term:

Welcome to funny jokes! We tell the funniest jokes in the world. Funny jokes are fun and crazy. Your funny joke awaits. Sit back and read funny jokes because funny jokes can make you happy and funnier. Some funny favorite funny jokes.

This tactic made for terrible user experiences, and instead of laughing at funny jokes, people were bombarded by annoying, hard-to-read text. It may have worked in the past, but this is never what search engines wanted.

The role links play in SEO

When we talk about links, we could mean two things. Backlinks or "inbound links" are links from other websites that point to your website, while internal links are links on your own site that point to your other pages (on the same site).

Links have historically played a big role in SEO. Very early on, search engines needed help figuring out which URLs were more trustworthy than others to help them determine how to rank search results. Calculating the number of links pointing to any given site helped them do this.

Backlinks work very similarly to real life WOM (Word-Of-Mouth) referrals. Let’s take a hypothetical coffee shop, Jenny’s Coffee, as an example:

• Referrals from others = good sign of authority
  Example: Many different people have all told you that Jenny’s Coffee is the best in town
• Referrals from yourself = biased, so not a good sign of authority
  Example: Jenny claims that Jenny’s Coffee is the best in town
• Referrals from irrelevant or low-quality sources = not a good sign of authority and could even get you flagged for spam
  Example: Jenny paid to have people who have never visited her coffee shop tell others how good it is.
• No referrals = unclear authority
  Example: Jenny’s Coffee might be good, but you’ve been unable to find anyone who has an opinion so you can’t be sure.

This is why PageRank was created. PageRank (part of Google's core algorithm) is a link analysis algorithm named after one of Google's founders, Larry Page. PageRank estimates the importance of a web page by measuring the quality and quantity of links pointing to it. The assumption is that the more relevant, important, and trustworthy a web page is, the more links it will have earned.

The more natural backlinks you have from high-authority (trusted) websites, the better your odds are to rank higher within search results.

The role content plays in SEO

There would be no point to links if they didn’t direct searchers to something. That something is content! Content is more than just words; it’s anything meant to be consumed by searchers — there’s video content, image content, and of course, text. If search engines are answer machines, content is the means by which the engines deliver those answers.

Any time someone performs a search, there are thousands of possible results, so how do search engines decide which pages the searcher is going to find valuable? A big part of determining where your page will rank for a given query is how well the content on your page matches the query’s intent. In other words, does this page match the words that were searched and help fulfill the task the searcher was trying to accomplish?

Because of this focus on user satisfaction and task accomplishment, there’s no strict benchmarks on how long your content should be, how many times it should contain a keyword, or what you put in your header tags. All those can play a role in how well a page performs in search, but the focus should be on the users who will be reading the content.

Today, with hundreds or even thousands of ranking signals, the top three have stayed fairly consistent: links to your website (which serve as a third-party credibility signals), on-page content (quality content that fulfills a searcher’s intent), and RankBrain.

What is RankBrain?

RankBrain is the machine learning component of Google’s core algorithm. Machine learning is a computer program that continues to improve its predictions over time through new observations and training data. In other words, it’s always learning, and because it’s always learning, search results should be constantly improving.

For example, if RankBrain notices a lower ranking URL providing a better result to users than the higher ranking URLs, you can bet that RankBrain will adjust those results, moving the more relevant result higher and demoting the lesser relevant pages as a byproduct.

Like most things with the search engine, we don’t know exactly what comprises RankBrain, but apparently, neither do the folks at Google.

What does this mean for SEOs?

Because Google will continue leveraging RankBrain to promote the most relevant, helpful content, we need to focus on fulfilling searcher intent more than ever before. Provide the best possible information and experience for searchers who might land on your page, and you’ve taken a big first step to performing well in a RankBrain world.

Engagement metrics: correlation, causation, or both?

With Google rankings, engagement metrics are most likely part correlation and part causation.

When we say engagement metrics, we mean data that represents how searchers interact with your site from search results. This includes things like:

• Clicks (visits from search)
• Time on page (amount of time the visitor spent on a page before leaving it)
• Bounce rate (the percentage of all website sessions where users viewed only one page)
• Pogo-sticking (clicking on an organic result and then quickly returning to the SERP to choose another result)

Many tests, including Moz’s own ranking factor survey, have indicated that engagement metrics correlate with higher ranking, but causation has been hotly debated. Are good engagement metrics just indicative of highly ranked sites? Or are sites ranked highly because they possess good engagement metrics?

What Google has said

While they’ve never used the term “direct ranking signal,” Google has been clear that they absolutely use click data to modify the SERP for particular queries.

According to Google’s former Chief of Search Quality, Udi Manber:

“The ranking itself is affected by the click data. If we discover that, for a particular query, 80% of people click on #2 and only 10% click on #1, after a while we figure out probably #2 is the one people want, so we’ll switch it.”

Another comment from former Google engineer Edmond Lau corroborates this:

“It’s pretty clear that any reasonable search engine would use click data on their own results to feed back into ranking to improve the quality of search results. The actual mechanics of how click data is used is often proprietary, but Google makes it obvious that it uses click data with its patents on systems like rank-adjusted content items.”

Because Google needs to maintain and improve search quality, it seems inevitable that engagement metrics are more than correlation, but it would appear that Google falls short of calling engagement metrics a “ranking signal” because those metrics are used to improve search quality, and the rank of individual URLs is just a byproduct of that.

What tests have confirmed

Various tests have confirmed that Google will adjust SERP order in response to searcher engagement:

• Rand Fishkin’s 2014 test resulted in a #7 result moving up to the #1 spot after getting around 200 people to click on the URL from the SERP. Interestingly, ranking improvement seemed to be isolated to the location of the people who visited the link. The rank position spiked in the US, where many participants were located, whereas it remained lower on the page in Google Canada, Google Australia, etc.
• Larry Kim’s comparison of top pages and their average dwell time pre- and post-RankBrain seemed to indicate that the machine-learning component of Google’s algorithm demotes the rank position of pages that people don’t spend as much time on.
• Darren Shaw’s testing has shown user behavior’s impact on local search and map pack results as well.

Since user engagement metrics are clearly used to adjust the SERPs for quality, and rank position changes as a byproduct, it’s safe to say that SEOs should optimize for engagement. Engagement doesn’t change the objective quality of your web page, but rather your value to searchers relative to other results for that query. That’s why, after no changes to your page or its backlinks, it could decline in rankings if searchers’ behaviors indicates they like other pages better.

In terms of ranking web pages, engagement metrics act like a fact-checker. Objective factors such as links and content first rank the page, then engagement metrics help Google adjust if they didn’t get it right.

The evolution of search results

Back when search engines lacked a lot of the sophistication they have today, the term “10 blue links” was coined to describe the flat structure of the SERP. Any time a search was performed, Google would return a page with 10 organic results, each in the same format.

In this search landscape, holding the #1 spot was the holy grail of SEO. But then something happened. Google began adding results in new formats on their search result pages, called SERP features. Some of these SERP features include:

• Paid advertisements
• Featured snippets
• People Also Ask boxes
• Local (map) pack
• Knowledge panel
• Sitelinks

And Google is adding new ones all the time. It even experimented with “zero-result SERPs,” a phenomenon where only one result from the Knowledge Graph was displayed on the SERP with no results below it except for an option to “view more results.”

The addition of these features caused some initial panic for two main reasons. For one, many of these features caused organic results to be pushed down further on the SERP. Another byproduct is that fewer searchers are clicking on the organic results since more queries are being answered on the SERP itself.

So why would Google do this? It all goes back to the search experience. User behavior indicates that some queries are better satisfied by different content formats. Notice how the different types of SERP features match the different types of query intents.

Query Intent	Possible SERP Feature Triggered
Informational	Featured Snippet
Informational with one answer	Knowledge Graph / Instant Answer
Local	Map Pack
Transactional	Shopping

We’ll talk more about intent in Chapter 3, but for now, it’s important to know that answers can be delivered to searchers in a wide array of formats, and how you structure your content can impact the format in which it appears in search.

Localized search

A search engine like Google has its own proprietary index of local business listings, from which it creates local search results.

If you are performing local SEO work for a business that has a physical location customers can visit (ex: dentist) or for a business that travels to visit their customers (ex: plumber), make sure that you claim, verify, and optimize a free Google My Business Listing.

When it comes to localized search results, Google uses three main factors to determine ranking:

1. Relevance
2. Distance
3. Prominence

Relevance

Relevance is how well a local business matches what the searcher is looking for. To ensure that the business is doing everything it can to be relevant to searchers, make sure the business’ information is thoroughly and accurately filled out.

Distance

Google use your geo-location to better serve you local results. Local search results are extremely sensitive to proximity, which refers to the location of the searcher and/or the location specified in the query (if the searcher included one).

Organic search results are sensitive to a searcher's location, though seldom as pronounced as in local pack results.

Prominence

With prominence as a factor, Google is looking to reward businesses that are well-known in the real world. In addition to a business’ offline prominence, Google also looks to some online factors to determine local ranking, such as:

Reviews

The number of Google reviews a local business receives, and the sentiment of those reviews, have a notable impact on their ability to rank in local results.

Citations

A "business citation" or "business listing" is a web-based reference to a local business' "NAP" (name, address, phone number) on a localized platform (Yelp, Acxiom, YP, Infogroup, Localeze, etc.).

Local rankings are influenced by the number and consistency of local business citations. Google pulls data from a wide variety of sources in continuously making up its local business index. When Google finds multiple consistent references to a business's name, location, and phone number it strengthens Google's "trust" in the validity of that data. This then leads to Google being able to show the business with a higher degree of confidence. Google also uses information from other sources on the web, such as links and articles.

Check a local business' citation accuracy here.

Organic ranking

SEO best practices also apply to local SEO, since Google also considers a website’s position in organic search results when determining local ranking.

In the next chapter, you’ll learn on-page best practices that will help Google and users better understand your content.

[Bonus!] Local engagement

Although not listed by Google as a local ranking determiner, the role of engagement is only going to increase as time goes on. Google continues to enrich local results by incorporating real-world data like popular times to visit and average length of visits...

...and even provides searchers with the ability to ask the business questions!

Undoubtedly now more than ever before, local results are being influenced by real-world data. This interactivity is how searchers interact with and respond to local businesses, rather than purely static (and game-able) information like links and citations.

Since Google wants to deliver the best, most relevant local businesses to searchers, it makes perfect sense for them to use real time engagement metrics to determine quality and relevance.

---

You don’t have to know the ins and outs of Google’s algorithm (that remains a mystery!), but by now you should have a great baseline knowledge of how the search engine finds, interprets, stores, and ranks content. Armed with that knowledge, let’s learn about choosing the keywords your content will target!

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!