Does Wordpress list all pages for crawlers?

admin管理员组

文章数量:1426052

I created a page on a Wordpress site that was for internal use only and triggers some backend code. Within a few days I started seeing hits on that page from "bingbot".

I'm not using any kind of sitemap plugin. How are crawlers finding this page?

I know the robots.txt file can block them but I want to make sure they don't show up for crawlers that don't respect this. I still want to have the page publicly accessible if someone types in the URL.

What needs to be done in Wordpress to make sure a page can't be discovered except by typing in the URL?

I created a page on a Wordpress site that was for internal use only and triggers some backend code. Within a few days I started seeing hits on that page from "bingbot".

I'm not using any kind of sitemap plugin. How are crawlers finding this page?

I know the robots.txt file can block them but I want to make sure they don't show up for crawlers that don't respect this. I still want to have the page publicly accessible if someone types in the URL.

What needs to be done in Wordpress to make sure a page can't be discovered except by typing in the URL?

• pages

Share Improve this question asked Jun 18, 2019 at 1:10 user83358user83358 101 8

• " to make sure a page can't be discovered except by typing in the URL?" - but that is exactly what many crawlers are doing. The way to go is usually either a robots.txt or some restrictions like HTTP Basic Auth (can be a simple user/password) – kero Commented Jun 18, 2019 at 6:21
• Crawlers can't guess the URL, they have to get it from somewhere. That is what I would like to remove. – user83358 Commented Jun 18, 2019 at 22:33
• This is just not correct. I have had several websites in development that suddenly appeared in Google. Even if they do not guess URLs (which I presume they do), if you surf the site in Chrome, they know it exists. The only effective way of blocking them out is by blocking out all random visitors (eg via Basic Auth) – kero Commented Jun 19, 2019 at 6:45
• If you create the url yoursite/dfoimsfg9q743jroaiemf9w84mr09wqnmr09 will it show up in a crawler? Why do you think that is? The URL that I created was only for an internal function, can't use basic auth, and was only accessed through Safari for testing. Since Apple doesn't have a search engine that seems like an unlikely way for it to be discovered. – user83358 Commented Jun 20, 2019 at 15:42
• After creating a page with a completely random URL, and not accessing it at all, it took 4 days to start getting hits from bing bot. Since the server hasn't been hammered with trillions of requests for random URLs, this can only mean that crawlers (and potentially malicious attackers) are getting a list of all pages that exist in Wordpress, whether or not they are linked from anywhere else on the public site. That could be a concern to some people. – user83358 Commented Jun 28, 2019 at 0:51

 |  Show 3 more comments

1 Answer 1

Sorted by: Reset to default -1

Yes, if they are public pages WordPress will let them be indexed unless you tell it not to.

Try adding this to you function.php this:

function allow_no_robots() {

    global $post;

    if ( $post->ID == PAGE_ID_TO_NOT_INDEX ) {
        echo '<meta name="robots" content="noindex, nofollow">';

} add_action( 'wp_head', 'allow_no_robots' );

That sould do it.

本文标签： Does Wordpress list all pages for crawlers