百度贴吧爬虫

把互联网比喻成一个蜘蛛网，那么Spider就是在网上爬来爬去的蜘蛛。

网络蜘蛛是通过网页的链接地址来寻找网页的。



从网站某一个页面（通常是首页）开始，读取网页的内容，找到在网页中的其它链接地址，



然后通过这些链接地址寻找下一个网页，这样一直循环下去，直到把这个网站所有的网页都抓取完为止。



如果把整个互联网当成一个网站，那么网络蜘蛛就可以用这个原理把互联网上所有的网页都抓取下来。



这样看来，网络爬虫就是一个爬行程序，一个抓取网页的程序。-Likened to a spider web of the Internet, so Spider spider is crawling around on the Internet.

Web spider is to find the page through a web link address.



Starting　a one page website (usually home), read the content of the page to find other links on the page in the address,



Then look through these links address of the next page, this has been the cycle continues, until all the pages of this site are crawled last.



If the entire Internet as a website, so web spiders can use this principle to all the pages on the Internet you have to crawl down.



It would appear that a spider web crawler, a program to crawl pages.