Mibew Messenger Community

General => General Discussion => Topic started by: pppplus on September 21, 2009, 08:27:18 AM

Title: How to avoid crawler like google or yahoo ?
Post by: pppplus on September 21, 2009, 08:27:18 AM
When a bot (like google or yahoo) crawls my site, it opens a vistor request.

Is it possible to avoid it ?

Thanks
Title: Re: How to avoid crawler like google or yahoo ?
Post by: Inspirer on September 21, 2009, 03:05:41 PM
Create robots.txt file in the root of your site.

User-Agent: *
Disallow: /webim/
Allow: /
Title: Re: How to avoid crawler like google or yahoo ?
Post by: pppplus on September 21, 2009, 03:24:27 PM
oh yes, simple...