Kumpulan List Kode Pemanggil Robot Web Crawler Lengkap

Selamat Malam sobat, untuk kali ini saya akan membagikan tutorial blog seperti biasanya. Sebelumnya sobat harus mengetahui apa itu Robot Web Crawler.Silakan simak penjelasan dibawah ini

Web crawler adalah suatu program atau script otomat yang relatif simple, yang dengan metode tertentu melakukan scan atau “crawl” ke semua halaman-halaman Internet untuk membuat index dari data yang dicarinya. Nama lain untuk web crawl adalah web spider, web robot, bot, crawl dan automatic indexer.
Web crawl dapat digunakan untuk beragam tujuan. Penggunaan yang paling umum adalah yang terkait dengan search engine. Search engine menggunakan web crawl untuk mengumpulkan informasi mengenai apa yang ada di halaman-halaman web publik. Tujuan utamanya adalah mengumpukan data sehingga ketika pengguna Internet mengetikkan kata pencarian di komputernya, search engine dapat dengan segera menampilkan web site yang relevan.

Gimana ? sudah mengerti bukan ? itu hanya sebagian penjelasan yang saya ambil dari blog milik orang lain. Untuk lebih jelasnya silakan sobat cari sendiri di google.com :D Nah, daripada bertele tele berikut saya sediakan list lengkap beserta tutorial pemasangannya

Tutorial Pemasangan

Buka blog seperti biasa dan masuk ke template > edit html
Coba cek template sobat apakah ada kode semacam ini ? "Follow, All" ?
Contoh kode:
<meta content='follow, all' name='googlebot'/>
<meta content='follow, all' name='Googlebot-Image'/>
<meta content='follow, all' name='msnbot'/>
<meta content='follow, all' name='robots'/>
Jika ada maka lanjut ke nomor 4
Lihat list dibawah ini misalnya "ABACHOBot"
Maka sobat harus merubahnya seperti ini
<meta content='follow, all' name='ABACHOBot'/>
Atau mungkin 008<meta content='follow, all' name='008'/>
Tinggal rubah sesuai namanya sob :D
Gampang "Tinggal menambahkan" saja sob :D

List Crawler

008

http://www.80legs.com/webcrawler.html

ABACHOBot

Abacho's spider. German based portal and search engine. Has localized versions in the following countries: Austria, Switzerland, France, UK, Spain, Italy, Sweden and Turkey.

Accoona-AI-Agent

Accoona's webcrawler

AnyApexBot

http://www.anyapex.com/bot.html

Arachmo

Baiduspider

http://www.baidu.com/search/spider_jp.html
http://www.baidu.com/search/spider.htm

BecomeBot

http://www.become.com/site_owners.html

BeslistBot

http://www.beslist.nl

BillyBobBot

http://www.billybobbot.com/crawler

Bimbot

Bingbot

http://www.bing.com/bingbot.htm

boitho.com-dc

http://www.boitho.com/dcbot.html

boitho.com-robot

boitho.com-robot/1.0

btbot

http://www.btbot.com/btbot.html

CatchBot

http://www.catchbot.com

Charlotte

http://www.searchme.com/support/

ConveraCrawler

http://ews.converasearch.com/crawl.htm

Covario IDS

http://www.covario.com/ids

DataparkSearch

DiamondBot

Discobot

http://discoveryengine.com/discobot.html

Dotbot

http://www.dotnetdotcom.org/#info

EmeraldShield.com WebBot

http://www.emeraldshield.com/webbot.aspx

envolk[ITS]spider

http://www.envolk.com/envolkspider.html

EsperanzaBot

http://www.esperanza.to/bot

Exabot

FAST Enterprise Crawler

http://www.scirus.com/srsapp/contactus

FAST-WebCrawler

http://fast.no/support/crawler.asp

FDSE robot

Search engine of Fluid Dynamics Software Corporation

FindLinks

http://wortschatz.uni-leipzig.de/findlinks/

FurlBot

Furl's crawler. Furl is a social bookmark service from LookSmart

FyberSpider

http://www.fybersearch.com/fyberspider.php

Gaisbot

http://gais.cs.ccu.edu.tw/robot.php

GalaxyBot

http://www.galaxy.com/galaxybot.html

genieBot

Web-indexing robot of GenieKnows Local Search Engine

Gigabot

http://www.gigablast.com/spider.html

Googlebot

http://www.google.com/bot.html

Googlebot-Image

http://www.google.com/bot.html

GurujiBot

http://www.guruji.com/en/WebmasterFAQ.html

HappyFunBot

http://www.happyfunsearch.com/bot.html

Holmes

http://morfeo.centrum.cz/bot

htdig

Crawler of the ht://Dig Group's software package, a system for indexing and searching a finite (not necessarily small) set of sites or intranet. It is not meant to replace any of the many internet-wide search engines. htdig retrieves HTML documents using the HTTP protocol.

iaskspider

Bot for iAsk , chinese search engine from Sina.com

http://iask.com/help/help_index.html

ia_archiver

Alexa Web crawler

http://www.alexa.com/faqs/?p=18031

iCCrawler

http://www.iccenter.net/bot.htm

ichiro

http://help.goo.ne.jp/door/crawler.html

igdeSpyder

http://igde.ru/doc/tech.html

IRLbot

http://irl.cs.tamu.edu/crawler

IssueCrawler

Govcom.org Foundation's web bot. Locates and visualizes networks on the Web. The Issue Crawler is used by NGOs and other researchers to answer questions about specific networks and effective networking more generally. You also may do in-depth research with the software. You need an account to use it.

Jaxified Bot

http://www.jaxified.com/crawler

Jyxobot

Czech Webcrawler for Jyxo

KoepaBot

http://www.koepa.nl/bot.html

L.webis

http://webalgo.iit.cnr.it/index.php?pg=lwebis

LapozzBot

http://robot.lapozz.com

LDSpider

http://code.google.com/p/ldspider/wiki/Robots

LexxeBot

Linguee Bot

http://www.linguee.com/bot

LinkWalker

lmspider

lwp-trivial

mabontland

magpie-crawler

Mediapartners-Google

MJ12bot

Mnogosearch

mogimogi

MojeekBot

http://www.mojeek.com/bot.html

Moreoverbot

http://w.moreover.com

Morning Paper

msnbot

http://search.msn.com/msnbot.htm

MSRBot

http://research.microsoft.com/research/sv/msrbot

MVAClient

mxbot

http://www.chainn.com/mxbot.html

NetResearchServer

loopimprovements.com/robot.html

NetSeer Crawler

NewsGator

http://www.newsgator.com

NG-Search

http://www.ng-search.com)

nicebot

noxtrumbot

Nusearch Spider

www.nusearch.com

NutchCVS

http://www.nutch.org/docs/en/bot.html

Nymesis

http://nymesis.com

obot

oegp

omgilibot

http://www.omgili.com/Crawler.html

OmniExplorer_Bot

http://www.omni-explorer.com

OOZBOT

http://www.setooz.com/oozbot.html

Orbiter

http://www.dailyorbit.com/bot.htm

PageBitesHyperBot

http://www.pagebites.com

Peew

http://www.peew.de/crawler

polybot

http://cis.poly.edu/polybot

Pompos

http://pompos.iliad.fr

PostPost

http://postpo.st/crawlers

Psbot

http://www.picsearch.com/bot.html

PycURL

Qseero

Radian6

www.radian6.com/crawler

RAMPyBot

http://www.giveramp.com/bot.html

RufusBot

http://64.124.122.252/feedback.html

SandCrawler

SBIder

http://support.sitesell.com/contact-support.html

ScoutJet

http://www.scoutjet.com

Scrubby

http://www.scrubtheweb.com/abs/meta-check.html

SearchSight

http://SearchSight.com

Seekbot

http://www.seekbot.net/bot.html

semanticdiscovery

Sensis Web Crawler

SEOChat::Bot

SeznamBot

http://fulltext.sblog.cz/robot

ShopWiki

http://www.shopwiki.com/wiki/Help

Shoula robot

silk

http://www.slider.com/silk.htm

Sitebot

http://www.sitebot.org/robot

Snappy

http://www.urltrends.com

sogou spider

Sosospider

http://help.soso.com/webspider.htm

Speedy Spider

http://www.entireweb.com/about/search_tech/speedy_spider

suggybot

http://blog.suggy.com/was-ist-suggy/suggy-webcrawler

SynooBot

http://www.synoo.de/bot.html

Teoma

http://about.ask.com/en/docs/about/webmasters.shtml

TerrawizBot

http://www.terrawiz.com/bot.html

TheSuBot

www.thesubot.de

Thumbnail.CZ robot

http://thumbnail.cz/why-no-robots-txt.html

TinEye

http://tineye.com/crawler.html

truwoGPS

http://www.lan4lano.net/browser.html

TurnitinBot

http://www.turnitin.com/robot/crawlerinfo.html

TweetedTimes Bot

http://tweetedtimes.com

updated

http://www.updated.com

Urlfilebot

http://urlfile.com/bot.html

Vagabondo

http://webagent.wise-guys.nl

VoilaBot

http://www.voila.com

Vortex

http://marty.anstey.ca/robots/vortex

voyager

http://www.kosmix.com/crawler.html

webcollage

Websquash.com

wf84

http://www.almaden.ibm.com/cs/crawler

WoFindeIch Robot

http://www.search.wofindeich.com/robot.php

WomlpeFactory

http://www.Womple.com/bot.html

Xaldon_WebSpider

yacy

http://yacy.net/bot.html

Yahoo! Slurp

http://help.yahoo.com/help/us/ysearch/slurp

Yahoo! Slurp China

http://misc.yahoo.com.cn/help.html

YahooSeeker

help.yahoo.com/help/us/shop/merchant

YahooSeeker-Testing

http://search.yahoo.com

YandexBot

http://yandex.com/bots

YandexImages

http://yandex.com/bots

Yasaklibot

http://www.Yasakli.com/bot.php

Yeti

http://help.naver.com/robots

YodaoBot

http://www.yodao.com/help/webmaster/spider

yoogliFetchAgent

YoudaoBot

http://www.youdao.com/help/webmaster/spider

Zao

http://www.kototoi.org/zao

Zealbot

zspider

http://feedback.redkolibri.com

ZyBorg

http://www.WISEnutbot.com

INGAT Sob DITAMBAHKAN bukan dihapus atau diganti hehe :D