Beware of Google and Apache URL Rewriting

 

I recently launched a site that had a crazy amount of URL rewriting. It involved a huge database and a huge amount of filter options. I was in my hoster's control panel (which hosts around ten sites) and I noticed by bandwidth had gone through the roof! So much so that at its current rate I was due to run out very soon. I checked my Analytics stats and there was nothing untoward. I then checked Webalizer and the in question was using gig upon gig a day (Since Analytics requires JavaScript it would suggest the bandwidth hogger was a bot or a user with JS disabled)! I did a site: query in Google and found my problem - Google was indexing over 150,000 pages a day!

I've since used robots.txt to control Google's thirst for information but let it be a lesson to you if you try to get clever with ModRewrite.

This is all not to mention the duplicate content penalty Google may be putting on the site!

It's a funny little irony that I used URL rewrites in the first places to help my SERPs.

No comments.