The most important obstacle to the growth of a website is the unwanted errors that occur in it. Timely detection and correction of these errors will improve the performance of your website and help it to continue to grow.
Do you also have trouble recognizing crawling errors? Do not worry! In this article, we explain what crawl errors are. We have also introduced the common types and suggested ways to fix each of these errors. So stay tuned.
Table of Contents
What are crawl errors?
A crawl error occurs when a search engine tries to reach a page on your webpage but fails.
Crawling is the process by which a search engine tries to crawl every page of your website through its bots. The search engine robot first finds a link and a path to your website and then searches all your public pages from there. These bots crawl pages and index all content for use on Google. One of the main tasks of an SEO expert is to make sure that search engine bots can crawl all the pages of your website. If this process fails, Google will show an error, which we call a crawl error.
Google Search Console is the best tool for detecting crawl errors
Google Search Console is a tool that is the best friend of your website to detect crawl errors. You can use this free tool to check how bots crawl on your site.
Crawl errors cause your website to get lost somewhere like a digital vortex and not reach your intended audience, and your efforts to get the site seen on the Google results page will be in vain.
So it’s important to master the Google Search Console tool to deal with crawl errors. This free tool lets you check your sitemaps, crawl errors and website traffic.
Types of crawl errors
Now that we understand what crawl errors are and how they can affect website SEO, it’s time to get acquainted with the types of crawl errors.
Google divides crawl errors into two categories:
- Site error: This error means that all pages of your site are inaccessible to Google bots and can not be crawled.
- URL error: This error only applies to a specific URL. These errors are easier to maintain and fix than before.
Site error
There are many reasons behind the site error. These errors include the following:
DNS error
Search engine crawlers must be connected to your site’s DNS server before they can access your site. If this connection fails, a DNS error occurs.
DNS errors can be categorized as follows:
DNS Timeout
This error occurs when the DNS server fails to respond to a Google request within a specified time frame.
DNS lookup
This error occurs when the DNS server fails to recognize the hostname.
DNS errors should be fixed immediately as this will restrict Google’s connection to your site.
How to fix DNS error
To fix the DNS error, it is recommended that you use the Google Console Search Tool to see how Google bots crawl the page. If this tool also fails to fetch your site, notify your DNS service provider to see where the problem is.
Server error
This error means that Google bots can not access your website address. This lack of access can be due to the fact that your request time has expired or the site is busy. This problem may also mean that your site has so many visitors that your current server is unable to handle all requests.
Before fixing your server error, you should first see what type of server error you are receiving because there are different types:
Timeout
The server responds too long to a request and ends without a response.
Truncated Headers
Google can connect to your server, but disconnect before the header can be sent.
Connection reset
Your server successfully processes a Google request, but does not return any content because the connection to the server has been reset.
Truncated Response
The connection to the server is disconnected before the full response is received and the response is shortened.
Connection Refused
Google could not access your site because your server denied the connection. Your host may be blocking the entry of Google bots, or there may be a problem configuring its firewall.
Connection Failed
Google can not connect to your server because the network is inaccessible or broken.
No Response
Google can connect to your server, but the connection is closed before any data is sent by the server.
Note: Server errors are different from DNS errors. DNS error means that bots cannot search your URL because of DNS problems; While server errors mean that although the bot can connect to your site, it can not load the page due to server errors.
How to fix server error
- Consult your hosting service provider if there is a problem connecting, interrupting or responding to the server.
- Make sure you do not block the entry of Google bots with incorrect firewall, DoS, or CMS configuration.
- Use Google Search Console to see if Google bots can currently navigate your site. If Google Search Console returns your homepage content smoothly, you can assume that Google is generally able to access your site properly.
Robots.txt error
Before crawling web pages, Google bots try to check your robots.txt file to see if there are pages on your website that you do not want to be indexed (pages with no index tags). The problem occurs when the robot can not access the robots.txt file. In this case, Google will delay crawling to the robots.txt file. It is therefore essential to ensure that the robots.txt file is always available.
How to fix Robots.txt error
If there are pages on the website that you do not want to be indexed, each domain and subdomain must have a related robots.txt file. Also, if you want to remove your pages from no index mode, you need to remove the blocked resources from your robots.txt file to make sure that important pages appear in search engine results.
URL error
As mentioned, URL errors are limited to specific pages on your site. There are several types of URL errors that you may encounter:
Soft 404 error
In 2014, webmasters began receiving “Soft 404” errors for some of their content pages.
The “Soft 404” error occurs when Google bots think it’s a 404 page when they enter a page. While in reality this is not the case. In fact, this method of Google wants to warn that the content on this page is poor!
How to fix soft 404 error
If the page with this error is not an important page for you, you can redirect it to a more valuable 301 page.
If the “Soft 404” page URL is one of your actual content pages, then Google considers your content to be of poor quality. So, make sure you put valuable content on your website.
Error 404 or Not found
Error 404 means that Google bots have tried to find a page that does not exist on your site. You may be wondering how Google bots can try to crawl a page that does not exist?
In fact, bots find these pages through the links given to that page.
If the page that received this error is an important page that has been deleted for any reason and there is no replacement, you should fix it.
Otherwise do not worry at all about other worthless pages. Many 404 errors are not worth correcting because 404s do not harm your site indexing or ranking.
How to fix 404 error
If there is an alternative to a page with a 404 error, be sure to redirect its address to 301.
Many times typos in internal and external links cause this error. For example (www.example.com/seo instead of www.example.com/swo) In this case, you can solve this problem by creating a 301 redirect to the correct address.
You can also contact a webmaster who has an incorrect link and request an update or delete link.
Error 403 or Access Denied
Access Denied error occurs when search engine bots are prevented from crawling on a particular page.
Like error 404, if the pages that have this problem are not important to you, and if you do not want these pages to be indexed by Google, do not worry. But if this is a page that needs to be indexed, you can try the following to fix this error:
How to fix 403 error
If you need login details to enter the page, delete it.
Remove the page URL from the site robots.txt file.
Ask your hosting provider if they are preventing search engines from crawling your site.
Not followed Error
An “Not followed” error means that Google could not track that particular URL. “Not followed” error refers to URLs that have been redirected to another address, but this redirect has not been applied correctly.
Check for redirects to fix this error.
500 server and DNS error
This is the same server and DNS error mentioned in the site errors section. The only difference between this error and the previous one is that this error occurs for multiple URLs, not the entire site. Follow the same method as before to fix this error.