« previous next »
Pages: [1]   Go Down

Author Topic: How to avoid crawler like google or yahoo ?  (Read 13818 times)

0 Members and 1 Guest are viewing this topic.

pppplus

  • Jr. Member
  • **
  • Posts: 3
How to avoid crawler like google or yahoo ?
« on: September 21, 2009, 08:27:18 AM »
When a bot (like google or yahoo) crawls my site, it opens a vistor request.

Is it possible to avoid it ?

Thanks
Logged

Inspirer

  • Mibew Project founder
  • Native
  • *****
  • Posts: 262
    • Mibew Messenger
Re: How to avoid crawler like google or yahoo ?
« Reply #1 on: September 21, 2009, 03:05:41 PM »
Create robots.txt file in the root of your site.

User-Agent: *
Disallow: /webim/
Allow: /
Logged

pppplus

  • Jr. Member
  • **
  • Posts: 3
Re: How to avoid crawler like google or yahoo ?
« Reply #2 on: September 21, 2009, 03:24:27 PM »
oh yes, simple...
Logged
Pages: [1]   Go Up
« previous next »