Welcome Back Brett

December 15, 2005

I was doing some test surfing this morning using a new user agent/header checking tool Dax just built. Just for fun, I loaded up WebmasterWorld with a Slurp UA. Suprisingly, I was able to navigate through the site. I was also able to surf the site as Googlebot and MSNbot.

I quick check of the robots.txt with several different UA’s showed that MSN and Yahoo are now given a robots.txt that allows them to crawl. However, Google is still banned, and humans still must login in order to view content.

Apparently, it’s been this way for awhile because both engines already show a dramatic increase in page counts.

MSN 57,000
Yahoo 160,000

I’m not sure why he’s still banning Google. But at least there are a couple places where you can once again search WmW.

IMO, it’s a smart move. But I’m sure there will be those who will complain about not allowing humans to see the page they click on from a SERP without registering.

Update: It appears the BT is allowing visitors coming from Yahoo to view the page listed in the SERP. You aren’t forced to login/register until you try and click on a second page. That seems like a good compromise to me.

Filed Under: Work

Comments

4 Responses to “Welcome Back Brett”

  1. Sebastian on December 15th, 2005 3:08 pm

    >I’m not sure why he’s still banning Google.
    Because it would make no sense during the 180 days suspension period caused by a helpful guy submitting WMW to the removal service?

  2. littleman on December 15th, 2005 4:17 pm

    Or maybe BT has a thing against Google.

  3. WebGuerrilla - aka Greg Boser » Banned from WebmasterWorld on December 16th, 2005 2:36 pm

    [...] I guess Brett didn’t like me posting about the fact that he’s cloaking is robots.txt file? [...]

  4. NFFC on December 21st, 2005 1:47 pm

    I’m getting wmw pages in Google now, this a recent change?

Before You Reply, Please Read My Comment Policy




The Greatest Real Estate Agent in the World