Translate

Wednesday, September 18, 2013

Robots.txt code to crawl in search engine on wapka

Goto EDIT SITE »
Global Setting »
Head Tags »
Robots.txt »
Paste This Code »

# robots.txt to block all bots except bots from Google , MSN , Yahoo
User-agent: *
Disallow: /fcmedianet.js
Disallow: /__media__/js/templates.js
Disallow: /cmedianet
Disallow: /cmdynet
Disallow: /mediamainlog.php
User-agent: Googlebot
Disallow:
User-agent: Slurp
Disallow:
User-agent: MSNBot
Disallow:
User-agent: ia_archiver
Disallow:
User-agent: *
Disallow: /
Powered By : Fresh-Code.TK

1 comment: