搜索爬行原理_如何阻止搜索引擎爬行WordPress网站

搜索爬行原理

Recently, one of our users asked us how they can stop search engines from crawling and indexing their WordPress site? There are many scenarios when you would want to stop search engines from crawling your website or listing it in search results. In this article, we will show you how to stop search engines from crawling a WordPress site.

最近，我们的一位用户问我们如何阻止搜索引擎对WordPress网站进行爬网和编制索引？ 在许多情况下，您都希望停止搜索引擎搜寻您的网站或将其列出在搜索结果中。 在本文中，我们将向您展示如何阻止搜索引擎抓取WordPress网站。

为什么和谁想要停止搜索引擎 (Why and Who Would Want to Stop Search Engines)

For most websites, search engines are the biggest source of traffic. You may ask, why would anyone want to block search engines?

对于大多数网站而言，搜索引擎是最大的流量来源。 您可能会问，为什么有人要阻止搜索引擎？

When starting out, a lot of people don’t know how to create a local development environment or a staging site. If you’re developing your website live on a publicly accessible domain name, then you likely don’t want Google to index your under construction or maintenance mode page.

刚开始时，很多人都不知道如何创建本地开发环境或暂存站点 。 如果您要使用公开的域名来开发网站，那么您可能不希望Google为您的在建或维护模式页面编制索引。

There are also many people who use WordPress to create private blogs, and they don’t want those indexed in search results because they’re private.

也有许多人使用WordPress创建私人博客 ，并且他们不希望将它们编入搜索结果中，因为它们是私人博客 。

Also some people use WordPress for project management or intranet, and you wouldn’t want your internal documents being publicly accessible.

还有一些人使用WordPress进行项目管理或Intranet ，并且您不希望内部文档可以公开访问。

In all the above situations, you probably don’t want search engines to index your website.

在上述所有情况下，您可能不希望搜索引擎将您的网站编入索引。

A common misconception is that if I do not have links pointing to my domain, then search engines will probably never find my website. This is not completely true.

一个普遍的误解是，如果我没有指向我的域的链接，那么搜索引擎可能永远找不到我的网站。 这不是完全正确的。

There are many ways search engines can find a website linked elsewhere. For example:

搜索引擎可以通过多种方式找到其他地方链接的网站。 例如：

1. Your domain name could have been previously owned by someone else, and they still have some links pointing to your website now.
   您的域名以前可能是其他人拥有的，但他们现在仍有一些指向您网站的链接。
2. Some domain search site’s results could get indexed with your link on them.
   某些域搜索网站的结果可能会在您的链接上被索引。
3. There are literally thousands of pages with just list of domain names, your site can appear on one of those.
   实际上，有数千个页面仅列出了域名列表，您的网站可以显示在其中之一上。

There are a lot of things happening on the web and most of them are not under your control. However, your website is still under your control, and you can instruct search engines to not index or follow your website.

网络上发生了很多事情，其中​​大多数都不在您的控制之下。 但是，您的网站仍处于您的控制之下，并且您可以指示搜索引擎不要为您的网站编制索引或关注您的网站。

影片教学 (Video Tutorial)

演示地址

If you don’t like the video or need more instructions, then continue reading.

如果您不喜欢该视频或需要更多说明，请继续阅读。

阻止搜索引擎爬行和索引您的WordPress网站 (Blocking Search Engines from Crawling and Indexing Your WordPress Site)

WordPress comes with a built-in feature that allows you to instruct search engines not to index your site. All you need to do is visit Settings » Reading and check the box next to Search Engine Visibility option.

WordPress具有内置功能，可让您指示搜索引擎不要为您的网站编制索引。 您所需要做的就是访问“设置»阅读”，然后选中“搜索引擎可见性”选项旁边的框。

When this box is checked, WordPress adds this line to your website’s header:

选中此框后，WordPress将此行添加到您的网站标题中：

<meta name='robots' content='noindex,follow' />

WordPress also modifies your site’s robots.txt file and add these lines to it:

WordPress还修改了您网站的robots.txt文件，并在其中添加了以下几行：

User-agent: *
Disallow: /

These lines ask robots (web crawlers) not to index your pages. However, it is totally up to search engines to accept this request or ignore it. Even though most search engines respect this, some page or random image from your site may get indexed.

这些行要求漫游器(网络搜寻器)不要为您的页面编制索引。 但是，完全由搜索引擎决定是否接受此请求。 即使大多数搜索引擎都尊重这一点，您网站上的某些网页或随机图片也可能会被索引。

如何确保您的网站没有出现在搜索结果中？ (How to Make Sure Your Site Doesn’t Appear in Search Results?)

The default WordPress visibility settings instruct search engines not to index your website. However, search engines may still end up crawling and indexing a page, file, or image from your website.

默认的WordPress可见性设置指示搜索引擎不要为您的网站建立索引。 但是，搜索引擎仍可能最终会爬网和索引您网站上的页面，文件或图像。

The only way to to make sure that search engines don’t index or crawl your website at all is by password protecting your entire WordPress site on the server level.

确保搜索引擎根本不会索引或爬网您的网站的唯一方法是通过密码保护服务器级别的整个WordPress网站。

This means when anyone tries to access your website, they are asked to provide a username and password even before they reach WordPress. This includes search engines as well. Upon login failure, they are shown 401 error and the bots turn away. Here is how to password protect an entire WordPress site.

这意味着当任何人尝试访问您的网站时，甚至在他们到达WordPress之前，都要求他们提供用户名和密码。 这也包括搜索引擎。 登录失败后，将显示401错误，并且漫游器会关闭。 这是密码保护整个WordPress网站的方法。

方法1：使用cPanel密码保护整个站点 (Method 1: Password Protecting an Entire Site using cPanel)

If your WordPress hosting provider offers cPanel access to manage your hosting account, then you can protect your entire site using cPanel. All you have to do is log in to your cPanel dashboard and then click on ‘Password Protect Directories’ icon under the ‘Security’ section.

如果您的WordPress托管服务提供商提供cPanel访问权限来管理您的托管帐户，那么您可以使用cPanel保护整个网站。 您要做的就是登录cPanel仪表板，然后单击“安全性”部分下的“密码保护目录”图标。

Next, you will need to choose the folder where you have installed WordPress. Usually, it is the public_html folder.

接下来，您将需要选择安装WordPress的文件夹。 通常，它是public_html文件夹。

If you have multiple WordPress sites installed under public_html directory, then you need to click on the folder icon to browse and select the folder for the website you want to password protect.

如果在public_html目录下安装了多个WordPress网站，则需要单击文件夹图标以浏览并选择要用密码保护的网站的文件夹。

On the next screen, you need to enter a name for the protected directory and click on the ‘Save’ button.

在下一个屏幕上，您需要输入受保护目录的名称，然后单击“保存”按钮。

Your information will be saved and cPanel will reload the page. After that, you need to add an authorized user by entering a username and password.

您的信息将被保存，cPanel将重新加载页面。 之后，您需要输入用户名和密码来添加授权用户。

That’s all, you have successfully added password protection to your WordPress site.

就是这样，您已经成功地将密码保护添加到了WordPress网站。

Now, whenever a user or search engine visits your website they will be prompted to enter the username and password you created earlier to view the site.

现在，每当用户或搜索引擎访问您的网站时，都会提示他们输入您先前创建的用户名和密码来查看该网站。

方法2：使用插件对WordPress进行密码保护 (Method 2: Password Protecting WordPress with a Plugin)

If you are using a managed WordPress hosting solution, sometimes you don’t have access to cPanel.

如果您使用的是托管WordPress托管解决方案，则有时您无权访问cPanel。

In that case, you have the option to use different WordPress plugins to password protect your website. Below are the two most popular solutions:

在这种情况下，您可以选择使用其他WordPress插件来密码保护您的网站。 以下是两个最受欢迎的解决方案：

1. SeedProd – it’s the #1 coming soon and maintenance mode plugin for WordPress, used on over 800,000+ websites. It comes with complete access control and permissions features that you can use to hide your website from everyone including search engines. We have a step by step how-to guide for SeedProd.
2. SeedProd –这是即将推出的＃1和WordPress维护模式插件，已在超过80万个网站上使用。 它具有完整的访问控制和权限功能，可用于对所有人(包括搜索引擎)隐藏您的网站。 我们为SeedProd提供了逐步指南 。
3. Password Protected密码保护how to password protect a WordPress site.如何密码保护WordPress网站的分步指南。

We hope this article helped you stop search engines from crawling or indexing your WordPress site. You may also want to see our ultimate step by step WordPress security guide for beginners.

我们希望本文能帮助您阻止搜索引擎抓取或索引WordPress网站。 您可能还想看看我们针对初学者的终极逐步WordPress安全指南 。

If you liked this article, then please subscribe to our YouTube Channel for WordPress video tutorials. You can also find us on Twitter and Facebook.

如果您喜欢这篇文章，请订阅我们的YouTube频道 WordPress视频教程。 您也可以在Twitter和Facebook上找到我们。

翻译自: https://www.wpbeginner.com/beginners-guide/how-to-stop-search-engines-from-crawling-a-wordpress-site/

搜索爬行原理