robots.txt - How to prevent Google from indexing the wp-content directory?

admin管理员组

文章数量:1122833

I received an error from google saying that I have mobile usability errors on my site.

the "page" that they are referring to is www.example/wp-content/plugins/ag-admin.

I'm not asking for help with that plugin, but trying to figure out how to make google not index that directory at all.

Is disallawing this in my robots.txt enough or should I even have to do that?

I received an error from google saying that I have mobile usability errors on my site.

the "page" that they are referring to is www.example.com/wp-content/plugins/ag-admin.

I'm not asking for help with that plugin, but trying to figure out how to make google not index that directory at all.

Is disallawing this in my robots.txt enough or should I even have to do that?

• robots.txt

Share Improve this question edited Sep 7, 2020 at 19:22 Jesse Nickles 73577 silver badges1919 bronze badges asked Sep 2, 2020 at 19:19 presswordpressword 37011 gold badge44 silver badges1313 bronze badges

Add a comment  | 

5 Answers 5

Sorted by: Reset to default 2

Here's the thing

There are multiple ways to do what you want, from adding meta tags, to passing headers, but because, you tagged your question with robots.txt So i consider it off-topic to discuss any other solutions.

Considering your demand you need to have this as your robots.txt This disallowed access to wp-admin, but depending on use case, you may need ajax, so I gave it an exception, and then disallowed wp-content, and wp-includes.

Change your robots.txt situated in root directory, like this

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Noindex: /wp-content/
Noindex: /wp-includes/

This shall ask google to not index anything in wp-content and wp-includes, Google is slow and it would take its bot some time, to realize he is going at wrong place, so it would eventually remove it from its index.

The below HTACCESS METHOD is completely optional, the Robots.txt can do the trick with work, but HTACCESS is much more consistent method, therefore I couldnt resist myself from telling it

Incase you are willing to edit ht-access file, then that would be a better approach.

I myself use it on my site. My code below if put in htaccess files, blocks all PHP and backend specific files, but allows all images, videos, pdfs and various similar file formats to be indexed by Google and others.

# Serves only static files
RewriteCond %{REQUEST_FILENAME} -f
RewriteRule ^wp-(content|includes)/([^/]+/)*([^/.]+\.)+ (jp(e?g|2)?|png|gif|bmp|ico|css|js|swf|xml|xsl|html?|mp(eg[34])|avi|wav|og[gv]|xlsx?|docx?|pptx?|gz|zip|rar|pdf|xps|7z|[ot]tf|eot|woff2?|svg|od[tsp]|flv|mov)$ - [L]
RewriteRule ^wp-(content|includes|admin/includes)/ - [R=404,L]

Using robots.txt

Yes you could use a robots.txt file for this, simply add the following into this file:

User-agent: *
Disallow: /wp-content/

Notice that you can have multiple Disallow directives if you would like to restrict indexing of other folders as well.

User-agent: *
Disallow: /wp-content/
Disallow: /wp-admin/

If you would like to allow indexing on a specific file in the folder you could use an Allow directive after Disallow like this:

Disallow: /wp-content/
Allow: /wp-content/plugins/askimet.php

If you only would like to stop google from indexing you could instead add a user agent:

User-agent: Googlebot

Using set Header

You should also be able to use .htaccess for this. Try adding something like this in your .htaccess file:

<Directory wp-content>
  Header set X-Robots-Tag "noindex"
</Directory>

For the above to work you need to have mod_headers module enabled in apache. You can enable this module by executing:

sudo a2enmod headers
sudo apachectl restart

just add this to .htaccess

#Serves only static files
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule ^wp-content/uploads/ - [R=404,L]

This id the correct robot.txt

User-agent: *
Disallow: /wp-content/
Disallow: /wp-admin/

Just an FYI on "NoIndex" rules: Google stopped recognizing "NoIndex" rules for robots.txt and this was announced in 2019.

本文标签： robotstxtHow to prevent Google from indexing the wpcontent directory