Do you want to remove robots.txt from WordPress? The robots.txt file is essential for blocking search engine bots from indexing specific pages of your website. But if it is not configured correctly, this may cause problems for your site, so you may need to remove it.
Before we dive into removing it, you need a clearer understanding of the robots.txt file and why you may need to remove it.
Table of contents
What is Robots.txt?
The robots.txt file on your WordPress website contains guidelines for search engine bots on which areas or pages of your website should be indexed and crawled. It also informs search engines, using specific rules and instructions, on whether pages and sections of your website should be indexed.
Consider a real-world example: Google’s web crawlers, also known as Googlebot, regularly analyze websites to keep their page index up to date. These web crawlers follow the instructions outlined in the robots.txt file as they scan the website.
Then, they index and crawl the pages and areas of your website according to the guidelines mentioned in the robots.txt file.
In the same way, the search engine bots of all the other well-known web crawlers index and crawl your website using the robots.txt file.
The file is handy for restricting bots from accessing specific areas, pages, or directories on a website. But at the same time, you may benefit by removing or disabling the robots.txt file from your WordPress website in some cases.
Why Should You Remove Robots.txt from WordPress?
There are many reasons to remove or disable the robots.txt file on your WordPress website. Here are a few of them:
- Unintended restrictions: The robots.txt file may sometimes contain rules that unintentionally restrict search engine crawlers from accessing essential pages or sections of your website. This can negatively impact your site’s visibility in search results and its ranking.
- Development or staging sites: If you have a development or staging site for testing purposes, you may not want search engines to index these sites. If you remove the robots.txt file, you can ensure that search engine crawlers won’t index or display these sites in search results. This allows you to focus on testing and refining your site during the early development and testing process without risking it being publicly visible to users.
- Enhanced indexing: Search engine crawlers can freely access and index your website’s pages and content when you remove the robots.txt file. So they can discover and index a broader range of your site’s content. This can be beneficial if you want to ensure maximum visibility and discoverability of your website’s content.
- Plugin Conflicts: Conflicts between specific plugins of your WordPress website and the robots.txt file may sometimes lead to unexpected issues or errors. Removing the robots.txt file can help resolve such conflicts and ensure your WordPress site runs smoothly.
However, it’s important to note that removing the robots.txt file is not a decision to be taken lightly and isn’t usually recommended. Consider the implications and consult with an expert before you continue if you’re unsure about its impact on your website’s SEO performance.
How to Remove Robots.txt from WordPress
You can remove or disable the robots.txt file from WordPress in 2 major ways:
- Using an FTP client
- cPanel
We’ll go through these methods in detail with a step-by-step guide to removing robots.txt from WordPress. Both ways are similar. However, you must first access your website’s FTP or cPanel.
Note: Before removing the robots.txt file from your website, please make a copy of it. Then, you can upload the original file to your website again if you make any unintentional changes during the process, which may cause errors or bugs.
1. Remove Robots.txt using an FTP Client
Using an FTP client to remove robots.txt from your WordPress website is straightforward. You can access WordPress via FTP using an FTP client such as FileZilla or Cyberduck. Then you can start editing your website’s files using it.
We will use the FTP client FileZilla for this tutorial; you can download and install it from their website. However, you are free to use any FTP client you prefer. The steps are almost identical.
After installing the FTP client, you need to connect your website to it. Open the FileZilla application on your computer and navigate to File > Site Manager from the top-left corner of the menu bar. You can also use the keyboard shortcut Ctrl+S to open the site manager.
Then, click on New Site and enter the name of your website under My Sites.
Now, open the General tab on the right side of your screen, where you’ll enter the credentials to connect your site to the FTP client. If you don’t have the required credentials, you can ask your hosting provider for them.
After you enter the credentials, click on Connect.
You will now have access to your website’s file directory from where you can remove the robust.txt file of your WordPress website. The robots.txt file is located in your website’s root directory, also known as the “public_html” folder. You can navigate to it using FileZilla Explorer.
Once you have found the robots.txt file, right-click on it and select ‘Delete’.
That’s it! You have successfully removed the robots.txt file from your WordPress website.
2. Remove Robots.txt from cPanel
If you want a more direct approach to manually removing the robots.txt file from your WordPress website, you can use the cPanel. To access your cPanel account, log in using the link provided by your hosting provider. If you don’t have the necessary credentials and details, you can contact your hosting provider for assistance.
After logging in to the cPanel, scroll to the Files section and click on File Manager. You can now access all the files of your WordPress website.
Then navigate to the root directory or the “public_html” folder, where the robots.txt file is located. Now, select the robots.txt file and click Delete on the file manager’s menu bar.
Great! Your robots.txt file has been removed from your WordPress website.
How to Disable Robots.txt in WordPress?
The approaches mentioned above can help you remove the robots.txt file from your WordPress website. However, it isn’t usually recommended, as the file is crucial for your SEO. Therefore, if you prefer a safer option, you can disable the robots.txt file on your WordPress website.
As a result, the robots.txt file will be present in your website file directory even if there are no restrictions for the search engine bots to crawl and index your website. There are several ways to disable the robots.txt file in WordPress.
You can do so directly from the WordPress dashboard or manually using a plugin, FTP, or cPanel.
1. Disable Robots.txt from WordPress Dashboard
It’s straightforward to disable robots.txt in your WordPress dashboard. All you have to do is go to Settings > Reading from your WordPress dashboard, uncheck the Search Engine Visibility option, and save the changes.
This will remove all the contents of the robots.txt file in your WordPress website and change it to:
User-agent: *
Disallow:Therefore, this will prevent the search engines from indexing your website. But the decision to honor this request lies with search engines.
2. Disable Robots.txt Using a Plugin
You can also edit and disable the robots.txt file using a plugin if you already use an SEO plugin on your WordPress website. This can save a significant amount of time if you don’t want search engine bots to index your site, as specified in the robots.txt file.
Several WordPress SEO plugins allow you to edit and disable the robots.txt file. But we will be using the Rank Math SEO plugin for this demonstration.
It has a very straightforward interface for editing the robots.txt file. This will make it very easy to edit and remove the contents of the robots.txt file of your WordPress website to deactivate it.
If you prefer a different SEO plugin, you can use it. The steps are very similar across all plugins. You must first install and activate the plugin before using it.
2.1. Install and Activate the Plugin
To install the plugin, navigate to Plugins > Add New in your WordPress dashboard and search for the plugin’s keywords. As soon as it appears in the search results, click ‘Install Now’.
The installation will be completed in just a few seconds. Then, click on Activate to activate the plugin.
You can also manually upload and install the plugin if you want to use any premium SEO plugins. Please look at our guide on manually installing a WordPress plugin if you need help.
2.2. Edit and Disable Robots.txt File
After you activate the plugin, go to Rank Math > General Settings from your WordPress dashboard and open the Edit robots.txt file tab. This will let you edit the code and instructions you want to include in your robots.txt file. The plugin has even provided a text editor here to add or remove code and instructions in your robots.txt file.
If you have edited this file before, you may see several instructions here depending on the pages you don’t want the search engine bots to crawl and index. On the other hand, if this is the default robots.txt file that WordPress created, you will see the following details:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.phpHere, all you have to do is remove all the additional codes in the robots.txt file and leave the file with the following instructions:
User-agent: * Disallow:
After you add the above instruction, click on Save Changes.
That’s it! You have successfully disabled the robots.txt file using a plugin.
3. Disable Robots.txt Using FTP Client or cPanel
The robots.txt file can also be disabled using an FTP client or cPanel. For this approach, you don’t need to remove any content from the robots.txt file on your WordPress website. However, you need to create a new robots.txt file using any text editor and include just the following instructions:
User-agent: *
Disallow:Make sure that you save the file as “robots.txt.”
Now all you have to do is upload and replace the newly created robots.txt file with the older one using an FTP client or cPanel.
If you want to use FTP, you can first connect to your website using an FTP client by following the steps in this guide. Then, upload the new robots.txt file to the root directory or the “public_html” folder.
Similarly, if you want to use cPanel to disable the robots.txt file, you can first log in to your cPanel account and open the file directory.
You can follow the steps outlined in the “Remove Robots.txt from cPanel” section of this guide. Then, navigate to the “public_html” folder and upload the newly created robots.txt file here.
If you need help editing this file, look at our detailed guide on editing robots.txt in WordPress.
Frequently Asked Questions
Now, let’s take a look at some frequently asked questions about this topic.
It’s a file that tells search engine bots which parts of your site they can or can’t crawl. WordPress generates a virtual one by default if a physical file doesn’t exist.
Yes, if you’ve created a physical robots.txt file in your site’s root directory, you can delete it via FTP or File Manager. If it’s the virtual one, it can only be modified, not deleted.
You may remove or modify it if it blocks important pages, interferes with SEO, or if you’re switching to a different method of managing crawl settings (such as in your SEO plugin).
If not configured properly, it might. Removing or misconfiguring it can accidentally allow search engines to crawl sensitive or unnecessary pages. Make sure to handle it carefully.
You can create a new one manually in your root directory or use SEO plugins like Yoast SEO or Rank Math to generate and manage a clean, optimized version.
Conclusion
These are the methods you can use to disable or remove the robots.txt file on your WordPress website.
It can be helpful if issues arise on your website due to the robots.txt file. Removing it can also benefit you if your website is in the development stage or if you want advanced indexing altogether.
But to summarize, you can completely remove the robots.txt file from your WordPress website using two significant ways:
- Using an FTP client
- cPanel
Both approaches are very effective at removing the robots.txt file from your website. But since this isn’t usually recommended, you can also disable it if you want to stay safer. You can disable it using the WordPress dashboard or edit it using a plugin, FTP, or cPanel.
So, can you remove or disable the robots.txt file now? Do let us know if you have ever tried it.
In the meantime, here are some more guides that can help you customize or improve your WordPress website: