We spend a lot of time at SpringTrax focusing on the 404's that the humans visiting your website encounter. After all, it is those 404's that cause you to lose customers.
However, you also have several robotic visitors accessing your website. Google and Bing, for example, send automated computer programs (robots) to explore your website. These robotic visitors can, and often do, encounter 404 errors just as easily as human visitors.
The problem with robots accessing your site's 404 isn't that you'll lose customers (these robotic computer programs weren't planning on buying from your website), but instead that the robots will be distracted by the 404's and miss out on some of the important parts of your website. In other words, you don't want the robotic visitors exploring broken pages, when they could instead be exploring the working pages on your website.
There are three common types of 404's that we've seen robots encounter. All are relatively easy to fix.
A robots.txt file provides robots instructions on what they can and cannot crawl on your website. This file is optional, but most robots check for this program before accessing your website so that they know the rules. However, if you don't have a robots.txt file, the robot will get a 404 error when they attempt to access the robots.txt on your site.
The solution is to create a robots.txt file and place that file in your website's root directory. You can learn more about robots.txt files and set one up for your website using the information provided by Robotstxt.org.
The other advantage to setting up a robots.txt file is that you can use the file to guide robots around your website, ensuring they access the files you want them to find (and that they won't access the files you'd rather they not find).
A sitemap.xml file lists the pages contained on your website. Search engines, like Google and Bing, can use this sitemap to find all the pages that on your site and, from that information, have a better idea of what pages to show people searching the web.
However, if you don't have an XML Sitemap file and Google or Bing attempt to access that file, Google and Bing will reach a 404 error page.
The solution is to create a sitemap.xml file. There are several tools to do so, including XML-Sitemaps.com. Along with avoiding the 404, having an XML sitemap can also help your search marketing efforts by helping Google and Bing find more of the pages on your site.
The third most popular 404 error automated programs encounter are shortcut icons. For instance, in the top of your browser you will see little images associated with each website.
Every time a visitor comes to your website, the browser is requesting a shortcut icon. However, if your website doesn't have a shortcut icon, your server will return a 404 error to the browser. This adds an unnecessary load on your website's server for a 404 that can be easily avoided.
Along with browsers, mobile devices require certain shortcut icons, specific to their environment. Creating shortcut icons for your website is a relatively easy process. Learn how to create shortcut icons for various devices.