Discuss: A More Useful 404

by Dean Frickey

11 redirect 301

For the search-engine induced broken link, you can help / speed up the searchengine update by making a 301 redirect. either you create the missing page, only including the redirection inside, or if you are more savvy, use your .htaccess to do it( much cleaner).

info here : http://www.webconfs.com/how-to-redirect-a-webpage.php

posted at 08:26 pm on November 18, 2008 by Alexandre Plennevaux

12 Re: Use the logs, Luke!

Kevin, your comment regarding duplication of effort is absolutely correct, however it only deals with half of the problem. This solution gives you the flexibility to handle the error message that appears to users, which is more important.

Honestly, the e-mail part of this is likely unnecessary, as most basic server-based stats software will list these pages. But the increased usability for the end user is stellar, and more sites should be implementing this thought process where possible.

posted at 11:48 pm on November 18, 2008 by Chris Parsons

13 Re: Use the Logs...

I agree with Chris. The idea here is not only to alert the webmaster (which certainly can be done in other ways) but also to provide better feedback to the user. In this light, I’d also be interested in experimenting with the Google 404 script that Jeremy mentioned. If it’s possible to include search results for the most likely page the user would have been looking for, that would add even more value.

posted at 03:05 am on November 19, 2008 by Nathan Walton

14 It's about workflow.

while I understand the comments above, I think this technique has great value to clients – especially those who never go ‘under the hood’ of their CMS. I can see myself coding this option in as a standard feature.

I also appreciate the ever-present consideration of a well maintained site. (there are a lot out there that still are not.)

Good show.

posted at 03:29 am on November 19, 2008 by George Carstensen

15 you forgot the firewalls

one thing that wasn’t mentioned is the slew of personal firewalls/security programs that strip http-referer headers from all requests for privacy reasons. i would make note of that in the 404’s content for the first case, but still wouldn’t send an email.

posted at 04:24 am on November 19, 2008 by Lucas Sandery

16 404 Hijacking

Certain software, such as AVG virus checker:
http://www.avg.com/special-toolbar-404-dns-error-tlbrc.tpl-mcr1
hijacks 404 pages in the browser in order to display their own ad page.
I find it impossible to use the 404 status header because of this

posted at 03:24 pm on November 19, 2008 by Eoghan Murray

17 Replies to several comments

Olaf: You’re certainly correct in that the developer can make some effort to help resolve bad links from search engines and my statement that “there really isn’t anything I can do about it�? is technically not accurate. I have made attempts to remove URLs from search engines, but in my opinion, working through the search engine’s web site, looking for a link to add/remove URLs, then actually going through the process takes too much time and effort. Not to mention that the list of search engines I check against already numbers 150.

Stefan: Personally, I would never automatically re-direct a user from a 404 page back to the home page (I’m assuming that you were referring to doing this automatically). If the user doesn’t realize that they’ve been redirected, they’ll think the index page has the information that he’s looking for, and this probably won’t be the case. However, the message provided to the user could easily contain a link to the site’s home page in addition to the search engine and site map that I showed in my article.

Peter: I can appreciate your concerns. I have had discussions with other developers who suggest that this will generate an inordinate about of e-mail, however, from my experience on a relatively large site, this has not been the case. If a spider follows a link to a page that doesn’t exist, then the e-mail message from that will allow me to correct the link and the 404 goes away. If the error is the result of a missing file that should be available for download, either because it’s moved, or was never uploaded, this becomes very helpful in identifying and resolving those issues. However, if someone, or something, is just hitting the site looking for pages that don’t exist, no e-mail is even generated.

Clerkendweller: Excellent comments. The original intent was to provide immediate feedback to the user, then I thought “well, why not inform the developer of the problem�? and that led to what I have. The idea of logging these errors has been discussed for exactly the reasons you mention. I’m leaving it for phase II.

Kevin: Thanks for your input. I realize that server logs contain this same information, however, I don’t want to look through any log report, and more importantly, I want to know right now that I have a problem on my site. A simple e-mail does the trick.

Daniel, Chris, George: Thanks for your kind words. Yes, a large part of this effort was driven by trying to provide the user with accurate and specific information with links that will help them find what they’re looking for.

posted at 06:17 pm on November 19, 2008 by Dean Frickey

18 Possibly Related Pages

I’ve seen quite a few sites (mostly content focused) that take some of the query string parameters and use them as a search to produce a list of pages that the user might have been trying to reach. Not exactly guaranteed to pull the right page, but might save the user some time, and keep them from leaving the site altogether.

posted at 11:20 pm on November 19, 2008 by Nick Gagne

19 A few missed tricks

First, as other people have said, you ought to give the user what they want: a way to find that content.

Whether it’s a simple search box or a simple site map or the Google 404 script you’re missing a trick if you simply create a “nice looking 404 page”.

Second, you forgot to stress that it should still return a 404 HTTP header (if not, you’re causing a LOAD of issues with Google). And you might also want to use custom Google Analytics code on the page too, to enable logging of everything in a viewable way.

All this is for nothing if your custom 404 page is 512 bytes or less: because, if you’re running Google Toolbar, if it’s 512 bytes or less, Google serves you a rather better 404 error anyway. Overwriting yours.

And I strongly recommend against firing off automated emails: if your site is even mildly attacked by bots trying to find a way into your SQL/membership data/credit card data, then you’ve made your problem a whole lot worse.

posted at 02:30 am on November 20, 2008 by James Cridland