文件名称:WebCollector-master
-
所属分类:
- 标签属性:
- 上传时间:2014-10-16
-
文件大小:3.71mb
-
已下载:0次
-
提 供 者:
-
相关连接:无下载说明:别用迅雷下载,失败请重下,重下不扣分!
介绍说明--下载内容来自于网络,使用问题请自行百度
基于WebCollector内核,可以自己编写爬虫的http请求、链接解析器、爬取信息更新器、抓取器等模块,WebCollector把这些基于内核编写的模块称作 插件 ,通过不同的插件组合,可以在1分钟内,把WebCollector组装成一个全新的爬虫。
WebCollector内置了一套插件(cn.edu.hfut.dmic.webcollector.plugin.redis)。基于这套插件,可以把WebCollector的任务管理放到redis数据库上,这使得WebCollector可以爬取海量的数据(上亿级别)。
用户可以根据自己的需求,去开发适合自己的插件-Based WebCollector kernel, you can write your own http request reptiles, link resolver, crawling updates, spider and other modules, WebCollector write these kernel-based modules called plug-in through a combination of different plug-ins, you can within one minute, the WebCollector assembled into a new reptile.
WebCollector built a set of plug-ins (cn.edu.hfut.dmic.webcollector.plugin.redis). Based on this set of plug-ins, you can put WebCollector task management onto the redis , which makes WebCollector be crawling mass of data (on a million levels).
Users can according to their own needs, to develop their own plug-ins. .
WebCollector内置了一套插件(cn.edu.hfut.dmic.webcollector.plugin.redis)。基于这套插件,可以把WebCollector的任务管理放到redis数据库上,这使得WebCollector可以爬取海量的数据(上亿级别)。
用户可以根据自己的需求,去开发适合自己的插件-Based WebCollector kernel, you can write your own http request reptiles, link resolver, crawling updates, spider and other modules, WebCollector write these kernel-based modules called plug-in through a combination of different plug-ins, you can within one minute, the WebCollector assembled into a new reptile.
WebCollector built a set of plug-ins (cn.edu.hfut.dmic.webcollector.plugin.redis). Based on this set of plug-ins, you can put WebCollector task management onto the redis , which makes WebCollector be crawling mass of data (on a million levels).
Users can according to their own needs, to develop their own plug-ins. .
(系统自动生成,下载前可以参看下载内容)
下载文件列表
WebCollector-master/
WebCollector-master/.gitignore
WebCollector-master/LICENSE
WebCollector-master/README.md
WebCollector-master/README.zh-cn.md
WebCollector-master/WebCollector/
WebCollector-master/WebCollector/pom.xml
WebCollector-master/WebCollector/src/
WebCollector-master/WebCollector/src/main/
WebCollector-master/WebCollector/src/main/java/
WebCollector-master/WebCollector/src/main/java/cn/
WebCollector-master/WebCollector/src/main/java/cn/edu/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/crawler/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/crawler/BreadthCrawler.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/Fetcher.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/generator/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/generator/DbUpdater.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/generator/Generator.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/generator/Injector.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/generator/ParamURLGenerator.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/generator/StandardGenerator.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/generator/filter/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/generator/filter/Filter.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/generator/filter/IntervalFilter.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/generator/filter/URLRegexFilter.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/generator/filter/UniqueFilter.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/handler/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/handler/Handler.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/handler/Message.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/model/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/model/AvroModel.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/model/CommonURL.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatum.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/model/Link.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/output/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/output/FileSystemOutput.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/parser/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/parser/HtmlParser.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/parser/LinkUtils.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/parser/ParseResult.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/parser/Parser.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/ui/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/ui/BreadthCrawlerUI.form
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/ui/BreadthCrawlerUI.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/util/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/util/CharsetDetector.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/util/Config.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/util/ConnectionConfig.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/util/FileUtils.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/util/HttpRetry.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/util/HttpUtils.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/util/Log.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/util/RandomUtils.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/util/Retry.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/util/Task.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/u
WebCollector-master/.gitignore
WebCollector-master/LICENSE
WebCollector-master/README.md
WebCollector-master/README.zh-cn.md
WebCollector-master/WebCollector/
WebCollector-master/WebCollector/pom.xml
WebCollector-master/WebCollector/src/
WebCollector-master/WebCollector/src/main/
WebCollector-master/WebCollector/src/main/java/
WebCollector-master/WebCollector/src/main/java/cn/
WebCollector-master/WebCollector/src/main/java/cn/edu/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/crawler/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/crawler/BreadthCrawler.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/fetcher/Fetcher.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/generator/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/generator/DbUpdater.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/generator/Generator.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/generator/Injector.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/generator/ParamURLGenerator.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/generator/StandardGenerator.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/generator/filter/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/generator/filter/Filter.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/generator/filter/IntervalFilter.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/generator/filter/URLRegexFilter.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/generator/filter/UniqueFilter.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/handler/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/handler/Handler.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/handler/Message.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/model/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/model/AvroModel.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/model/CommonURL.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/model/CrawlDatum.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/model/Link.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/model/Page.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/output/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/output/FileSystemOutput.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/parser/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/parser/HtmlParser.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/parser/LinkUtils.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/parser/ParseResult.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/parser/Parser.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/ui/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/ui/BreadthCrawlerUI.form
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/ui/BreadthCrawlerUI.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/util/
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/util/CharsetDetector.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/util/Config.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/util/ConnectionConfig.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/util/FileUtils.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/util/HttpRetry.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/util/HttpUtils.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/util/Log.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/util/RandomUtils.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/util/Retry.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/util/Task.java
WebCollector-master/WebCollector/src/main/java/cn/edu/hfut/dmic/webcollector/u
本网站为编程资源及源代码搜集、介绍的搜索网站,版权归原作者所有! 粤ICP备11031372号
1999-2046 搜珍网 All Rights Reserved.