How to fix indexed, though blocked by Robot.txt

Robots.txt is a simple text file which can be placed in the website’s root directory. This file contains information for search engine bots, such as the Google bot. The information tells the bot which folders/sections on the website it should not crawl.re.

How to Fix? There are many different ways to fix this issue:

Urgent fix – Add a removal request in the Google Search Console

If these pages contain information that may affect your brand financially or legally. You may consider using this removal feature in Google Search Console.

It takes less than 15 min in my experience to get the page removed after you put in the request.

Steps:

1. Go to https://search.google.com/search-console/
2. Verify your domain if it is not already there. You can verify using DNS, HTML, Tag Manager, Google Analytics or other methods. To start the process click on add property and follow the instructions.
  If you need help to verify search console, follow this guide from Google.

1. Under indexing click on Removal.
2. Under Temporary removals > Request a removal, click on new request.

1. Add the URL, and choose to remove this URL-only option.

1. Done

Add a no-index tag to the webpage header

Another way to make sure that the search bot does not index a page is to add a no index tag to the page.

Where to add?

This tag goes in the webpage head.

What’s the tag?

How to add?

For WordPress:

Use your SEO plugin to add noindex to this page, if you are using Yoast SEO plugin you can do this by:

1. Login to backend of your website
2. Find and edit the page you want to hide
3. Under Yoast settings go to the advanced tab
4. And choose no for both these options below:

For Shopify:

For Shopify you will need to add some code to the theme.liquid to search for the page URL and add the no index tag to that page.

Here is the code to add:

{% if handle contains ‘page-url-you-want-to-exclude’ %}

{% endif %}

It goes into your theme.liquid file which can be found under online store > Customise > Edit code.

Other than the two options above you can always make sure there are no internal/external links going to the pages you want to hide. It’s a good chance that Google bot crawled these pages because they were linked from someplace on the internet. You can always add more security to these types of pages like a login page to avoid these issues in the future.

Contact Melbourne SEO Consulting for help on any of the above solutions.

How to fix: Indexed, though blocked by robots.txt