404 Checker

About The 404 Checker User Agent

Whenever one of the scripts is used on this website, (either to check HTTP status codes or links), the script identifies itself to the server that it is connecting to. It does this by setting the "User Agent" directive in the header of the request to the server.

The User Agent for all scripts run from this site is "404 Checker".

This means that the destination server knows who is connecting to it and can handle any requests specifically tailored to that agent should it wish.

Additionally any server can pre-empt requests from a particular user agent by means of a robots.txt file which contains details of the Robots Exclusion Protocol for that server (or website). A Robot Exclusion file can specify what user agents can and can not access specific areas of a website.

The HTTP Status Checker and the Full Header Checker do not check for exclusion - this is because they only fetch a single header, and fetching a single header is much less resource intensive than actually checking the robots.txt file.

The Link Checker does check the robots.txt file as an entire page is fetched (plus any additional headers from internal links from that page).

If you do not want the Link Checker to access your site add the user agent "404 Checker" to your robots.txt file.

The 404 Checker
Copyright © Drop Web Solutions 2017 - All Rights Reserved [Contact]