Indexing as many pages as possible on your website can be very tempting for marketers trying to strengthen their authority in search engines.

But, while it’s true that posting more pages that are relevant to a particular keyword (assuming they are also of high quality) will improve your rankings for that keyword, sometimes it’s better to keep certain pages on your site. Web. outside of the index of a search engine.

… Can you repeat that please?!

Stick with us, guys. This article will explain why you might want to remove certain web pages from the SERPs (Search Engine Results Pages) and exactly how to do it.

Deindex a page from Google

There are a few occasions when you might want to exclude a web page – or part of a web page – from search engine crawling and indexing, like:

  • TO prevent duplicate content (when there is more than one version of a page indexed by search engines, such as in a printable version of your content) to be indexed
  • Treat administration and login pages for internal use, unless they are intended for use by a community
  • For a thank you page (i.e. the page on which a visitor arrives after conversion on one of your landing pages) where the visitor has access to any offer promised by that landing page, such as a link to a PDF ebook

This is what the thank you page is for our SEO tips ebook looks like, for example:

HubSpot SEO Starter Pack Thank You Page

You want anyone who lands on your thank you pages to get there because they’ve already filled out a form on a landing page – not because they found your thank you page in the search.

Why not? Because anyone who finds your thank you page in search can go straight to your lead-generating offers – without having to provide their information to you to go through your lead capture form. Any marketer who understands the value of landing pages understands how important it is to capture these visitors as leads before they can access your offers.

At the end of the line : If your thank you pages are easily found with a simple Google search, you may be leaving some valuable leads on the table.

Worse yet, you might even find that some of your top ranking pages for some of your long tail keywords could be your thank you pages, meaning you could invite hundreds of potential prospects to bypass your forms. lead capture. This is a pretty compelling reason why you would want to remove some of your web pages from the SERPs.

So how do you go about “de-indexing” certain pages from search engines? Here are three ways to do it.

3 Ways To Deindex A Web Page From Search Engines

Robots.txt to deindex

Use this if: you want more control over what you deindex and have the technical resources to do so.

One way to remove a page from search engine results is to add a robots.txt file to your site. The advantage of using this method is that you can have more control over what you allow bots to index. The result? You can proactively prevent unwanted content from search results.

In a robots.txt file, you can specify whether you want to block robots from a single page, an entire directory, or even a single image or a single file. There is also an option to prevent crawling of your site while still allowing Google AdSense ads to work if you have them.

That being said, of the two options available to you, this one requires the more technical kung fu. To learn how to create a robots.txt file, you must read this article that explains exactly how to do it..

HubSpot Customers: You can learn how to install robots.txt file on your website here, and learn how to customize the content of the Robots.txt file here.

If you don’t need all the control of a robots.txt file and are looking for a simpler, less technical solution, then this second option is for you.

Htaccess No index No deindexing tracking

Use If: Your website is running on Apache and mod_headers is enabled, this is a quick fix.

In this case, you can attach this single line to your .htaccess file:

X-Robots-Tag header set “noindex, nofollow”

To mean that your website can be indexed, but never shown in Google search results.

Meta No Index No Follow to De-Index

Use if: you want a simpler solution to deindex an entire web page and / or deindex links on an entire web page.

Using a meta tag to prevent a page from appearing in SERPs – and / or links on a page – is both simple and effective. It only requires a tiny bit of technical know-how – in fact, it’s really just a copy / paste job if you use the right one. content management system.

The tags that allow you to do these things are called “noindex” and “nofollow”. Before we get into how to add these tags, let’s take a moment to define and distinguish the two. They are, after all, two completely different directives – and they can be used either on their own or side by side.

What is a “noindex” tag?

When you add a “noindex” meta tag to a web page, it tells a search engine that although it can crawl the page, it cannot add the page to its search index.

Thus, any page containing the “noindex” directive will be not go into the search engine’s search index, and therefore cannot be displayed in search engine results pages.

What is a “nofollow” tag?

When you add a “nofollow” meta tag to a web page, it prohibits search engines from crawling the connections on this page. This also means that any page ranking authority on the SERP will be not be transmitted to the pages to which it refers.

Thus, any page containing a “nofollow” directive will have all of its links ignored by Google and other search engines.

As I said before, you can add a “noindex” directive alone or together with a “nofollow” directive. You can also add a single “nofollow” directive.

When to use “noindex” and “nofollow” separately

Add only a “noindex” tag when you not want a search engine to index your webpage in search, but you to do want it to follow the links on that page – thus giving ranking authority to other pages your page is linked to.

Paid landing pages are a prime example. You don’t want search engines to index landing pages that people are supposed to pay to see, but you might want the pages they link to have its authority.

Add only a “nofollow” tag when you to do want a search engine to index your webpage in search, but you not want him to follow the links on this page.

There aren’t many examples of adding a “nofollow” tag to an entire page without also adding a “noindex” tag. When determining what to do on a given page, it’s more a matter of adding your “noindex” tag with or without a “nofollow” tag.

When to use “noindex, nofollow” together

Add both a “noindex” and “nofollow” tag when you not want search engines to index a web page in search, and you don’t want it to follow links on that page.

Thank you pages are a prime example of this situation. You don’t want search engines to index your thank you page, or for them to follow the link to your offer and start indexing that offer’s content.

How to add a “noindex” and / or “nofollow” meta tag

Step 1: Copy one of the following tags.

For “noindex”:

For “nofollow”:

For “noindex” and “nofollow”:

2nd step: Add the tag to the section HTML code for your page, aka the page header.

If you are a HubSpot customer, it’s super easy – click here or scroll down for instructions specific to HubSpot users.

If you are not a HubSpot customer, you will then need to paste this tag into your web page code manually. Don’t worry, it’s pretty straightforward. Here’s how to do it.

First, open the source code of the web page you are trying to deindex. Then paste the full tag into a new line in the section HTML code for your page, known as the page header. The screenshots below will walk you through it.

The tag means the start of your header:

head tag at the start of the header

Here is the meta tag for “noindex” and “nofollow” pasted in the header:

meta tag in the header for noindex nofollow

And the beacon means the end of the header:

end header tag

Boom! That’s it. This tag tells a search engine to turn around and go, leaving the page out of search results.

No index No tracking in HubSpot

Adding the “noindex” and “nofollow” meta tags is even easier. All you have to do is open the HubSpot tool on the page you want to add these tags to and choose the ‘Settings’ tab.

Then under Advanced options and click on “Head HTML”. In the window below, paste the appropriate code snippet. In the example below I have added both a “noindex” tag and a “nofollow” tag because this is a thank you page.

no index no tracking in hubspot

Hit “Save” and you’re golden.

No index successfully No tracking of a page

You just magically wiped your page from the search engine results. Now you can start capturing more of those lost leads again.

Now keep in mind that you won’t see the results instantly. Your changes will only be applied the next time your page is crawled by a search engine. Depending on how often you typically post new pages to your website, this may take a few weeks. The more frequently you post content, the more search engines will crawl your site. The best way to track how often Google visits your website is to review your exploration statistics in Google tools for webmasters.

At the end of the line : If you notice that your page still appears in Google search results even with the “noindex” tag, it’s probably because Google hasn’t crawled your site since you added the tag. You can ask Google to crawl your page again using the Extract as Google Tool.

Also note that some search engine crawlers may interpret these guidelines differently than Google, so your page may still appear in other search engine results. But for Google, it will work just fine – once it’s done crawling your website.

Either way, you’ll be able to sleep a little easier knowing that you’ve finally made your website a better place for your marketing.

Editor’s Note: This article was originally published in July 2016 and has been updated for completeness.

SEO audit


LEAVE A REPLY

Please enter your comment!
Please enter your name here