Do you want to remove robots.txt from WordPress? The robots.txt file is essential for blocking search engine bots from indexing specific pages of your website. But if it is not configured correctly, this may cause problems for your site, so you may need to remove it.
So before we dive in with the process of removing it, you need to have a clearer understanding of the robots.txt file and why you may need to remove it.
Table of contents
What is Robots.txt?
The robots.txt is a file on your WordPress website that contains the guidelines for the search engine bots on which areas or pages of your website should be indexed and crawled. It also informs the search engines using specific rules and instructions on whether the pages and sections of your website should be indexed.
Consider a real-world example: Google’s web crawlers, also known as Googlebot, regularly analyze websites to keep their page index up to date. These web crawlers follow the instructions outlined in the robots.txt file as they scan the website.
Then, they index and crawl the pages and areas of your website according to the guidelines mentioned in the robots.txt file.
In the same way, the search engine bots of all the other well-known web crawlers index and crawl your website using the robots.txt file.
The file is handy for restricting bots from accessing specific areas, pages, or directories on a website. But at the same time, you may benefit by removing or disabling the robots.txt file from your WordPress website in some cases.
Why Should You Remove Robots.txt from WordPress?
There are many reasons to consider removing or disabling the robots.txt file from your WordPress website. Here are a few of them:
- Unintended restrictions: The robots.txt file may sometimes contain rules that unintentionally restrict search engine crawlers from accessing essential pages or sections of your website. This can negatively impact your site’s visibility in search engine results and affect your site’s ranking.
- Development or staging sites: If you have a development or staging site for testing purposes, you may not want search engines to index these sites. If you remove the robots.txt file, you can ensure that search engine crawlers won’t index or display these sites in search results. This allows you to focus on testing and refining your site during the early development and testing process without the risk of it being visible to the users publicly.
- Enhanced indexing: Search engine crawlers can freely access and index your website’s pages and content when you remove the robots.txt file. So they can discover and include a broader range of your site’s content in their index. This can be beneficial if you want to ensure maximum visibility and discoverability of your website’s content.
- Plugin Conflicts: Conflicts between specific plugins of your WordPress website and the robots.txt file may sometimes lead to unexpected issues or errors. So, removing the robots.txt file can help resolve such conflicts and ensure the smooth functioning of your WordPress site.
However, it’s important to note that removing the robots.txt file is not a decision to be taken lightly and isn’t usually recommended. Consider the implications and consult with an expert before you continue if you’re unsure about its impact on your website’s SEO performance.
How to Remove Robots.txt from WordPress
You can remove or disable the robots.txt file from WordPress in 2 major ways:
- Using FTP client
- cPanel
We’ll go through these methods in detail with a step-by-step guide to removing robots.txt from WordPress. Both ways are similar to each other. However, you must first access the FTP or cPanel of your website.
Note: Before removing the robots.txt file from your website, please ensure that you make a copy of it. Then, you can upload the original file to your website again if you make any unintentional changes during the procedure, which may lead to errors or bugs.
1. Remove Robots.txt using an FTP Client
Using an FTP client to remove robots.txt from your WordPress website is a straightforward process. You can access FTP for WordPress using an FTP client like FileZilla or Cyberduck. Then, you can start editing the files of your website using it.
We will use the FTP client FileZilla for this tutorial, which you can download and install on your computer from their website. However, you are free to use any FTP client you prefer. The steps are almost identical to one another.
After installing the FTP client, you need to connect your website to it. Open the FileZilla application on your computer and navigate to File > Site Manager from the top-left corner of the menu bar. You can also use the keyboard shortcut Ctrl+S on your keyboard to access the site manager.
Then, click on New Site and enter the name of your website under My Sites.
Now, open the General tab on the right side of your screen, where you’ll need to enter the credentials to connect your site with the FTP client. If you don’t have the required credentials, you can ask your hosting provider for them.
After you enter the credentials, click on Connect.
You will now have access to your website’s file directory from where you can remove the robust.txt file of your WordPress website. The robots.txt file is located in your website’s root directory, also known as the “public_html” folder. You can navigate to it using FileZilla Explorer.
Once you have found the robots.txt file, right-click on it and select ‘Delete’.
That’s it! You have successfully removed the robots.txt file from your WordPress website.
2. Remove Robots.txt from cPanel
If you want a more direct approach to manually removing the robots.txt file from your WordPress website, you can use the cPanel. To access your cPanel account, simply log in using the link provided by your hosting provider. If you don’t have the necessary credentials and details, you can contact your hosting provider for assistance.
After logging in to the cPanel, scroll to the Files section and click on File Manager. You can now access all the files of your WordPress website.
Then, navigate to the root directory or the “public_html” folder, as the robots.txt file is located here. Now, select the robots.txt file and click Delete on the file manager’s menu bar.
Great! Your robots.txt file has been removed from your WordPress website.
How to Disable Robots.txt in WordPress?
The approaches mentioned above can help you remove the robots.txt file from your WordPress website. However, it isn’t usually recommended, as the file is crucial for your SEO. Therefore, if you prefer a safer option, you can disable the robots.txt file on your WordPress website.
As a result, the robots.txt file will be present in your website file directory even if there are no restrictions for the search engine bots to crawl and index your website. There are several ways to disable the robots.txt file in WordPress.
You can do so directly from the WordPress dashboard or edit the file manually using a plugin, FTP, or cPanel.
1. Disable Robots.txt from WordPress Dashboard
It’s straightforward to disable the robots.txt file from your WordPress dashboard. All you have to do is go to Settings > Reading from your WordPress dashboard, uncheck the Search Engine Visibility option, and save the changes.
This will remove all the contents of the robots.txt file in your WordPress website and change it to:
User-agent: *
Disallow:Therefore, this will prevent the search engines from indexing your website. But the decision to honor this request lies with search engines.
2. Disable Robots.txt Using a Plugin
You can also edit and disable the robots.txt file using a plugin if you already use an SEO plugin on your WordPress website. This can save a significant amount of time if you don’t want search engine bots to index your site using the robots.txt file.
Several WordPress SEO plugins allow you to edit and disable the robots.txt file. But we will be using the Rank Math SEO plugin for this demonstration.
It has a very straightforward interface for editing the robots.txt file. This will make it very easy to edit and remove the contents of the robots.txt file of your WordPress website to deactivate it.
If you prefer to use a different SEO plugin, you can do so. The steps are very similar for each plugin. You must first install and activate the plugin before using it.
2.1. Install and Activate the Plugin
To install the plugin, navigate to Plugins > Add New in your WordPress dashboard and search for the plugin’s keywords. As soon as it appears in the search results, click ‘Install Now’.
The installation will be completed in just a few seconds. Then, click on Activate to activate the plugin.
You can also manually upload and install the plugin if you want to use any premium SEO plugins. Please look at our guide on manually installing a WordPress plugin if you need help.
2.2. Edit and Disable Robots.txt File
After you activate the plugin, go to Rank Math > General Settings from your WordPress dashboard and open the Edit robots.txt file tab. This will allow you to edit the codes and instructions you want to include in your robots.txt file. The plugin has even provided a text editor here to add or remove the codes and instructions on your robots.txt file.
If you have edited this file before, you may see several instructions here depending on the pages you don’t want the search engine bots to crawl and index. On the other hand, if this is the default robots.txt file that WordPress created, you will see the following details:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.phpHere, all you have to do is remove all the additional codes in the robots.txt file and leave the file with the following instructions:
User-agent: * Disallow:
After you add the above instruction, click on Save Changes.
That’s it! You have successfully disabled the robots.txt file using a plugin.
3. Disable Robots.txt Using FTP Client or cPanel
The robots.txt file can also be disabled using an FTP client or cPanel. For this approach, you don’t need to remove any content from the robots.txt file on your WordPress website. However, you need to create a new robots.txt file using any text editor and include just the following instructions:
User-agent: *
Disallow:Make sure that you save the file as “robots.txt.”
Now all you have to do is upload and replace the newly created robots.txt file with the older one using an FTP client or cPanel.
If you want to use FTP, you can first connect your website with the FTP client using the steps mentioned in this guide. Then, upload the new robots.txt file to the root directory or the “public_html” folder.
Similarly, if you want to use cPanel to disable the robots.txt file, you can first log in to your cPanel account and open the file directory.
You can follow the steps outlined in the “Remove Robots.txt from cPanel” section of this guide. Then, navigate to the “public_html” folder and upload the newly created robots.txt file here.
If you need help editing this file, look at our detailed guide on editing robots.txt in WordPress.
Frequently Asked Questions
Now, let’s take a look at some of the frequently asked questions regarding this topic.
It’s a file that tells search engine bots which parts of your site they can or can’t crawl. WordPress generates a virtual one by default if a physical file doesn’t exist.
Yes, if you’ve created a physical robots.txt file in your site’s root directory, you can delete it via FTP or File Manager. If it’s the virtual one, it can only be modified, not deleted.
You may remove or modify it if it blocks important pages, interferes with SEO, or if you’re switching to a different method of managing crawl settings (such as in your SEO plugin).
If not configured properly, it might. Removing or misconfiguring it can accidentally allow search engines to crawl sensitive or unnecessary pages. Make sure to handle it carefully.
You can create a new one manually in your root directory or use SEO plugins like Yoast SEO or Rank Math to generate and manage a clean, optimized version.
Conclusion
These are the various methods with which you can disable or remove the robots.txt file from your WordPress website.
It can be helpful if issues arise on your website due to the robots.txt file. Removing it can also benefit you if your website is in the developing stage or if you want advanced indexing altogether.
But to summarize, you can completely remove the robots.txt file from your WordPress website using two major ways:
- Using an FTP client
- cPanel
Both of these approaches are very effective in removing the robots.txt file from your website. But since this isn’t usually recommended, you can also disable it if you want to stay safer. You can disable it using the WordPress dashboard or edit it using a plugin, FTP, or cPanel.
So, can you remove or disable the robots.txt file now? Do let us know if you have ever tried it.
In the meantime, here are some more guides that can help you customize or improve your WordPress website: