Java爬蟲框架之WebMagic實戰
阿新 • • 發佈:2020-12-01
一、介紹
WebMagic是一個簡單靈活的Java爬蟲框架。基於WebMagic,你可以快速開發出一個高效、易維護的爬蟲。
二、如何學習
1.檢視官網
官網地址為:http://webmagic.io/
官網詳細文件:http://webmagic.io/docs/zh/
2.跑通hello world示例(具體可以參考官網,也可以參考部落格)
我下面寫的單元測試案例,可作為Hello World示例。
注意需要匯入Maven依賴:
<dependency> <groupId>us.codecraft</groupId> <artifactId>webmagic-core</artifactId> <version>0.7.3</version> </dependency> <dependency> <groupId>us.codecraft</groupId> <artifactId>webmagic-extension</artifactId> <version>0.7.3</version> </dependency>
3.帶著一個目的
說說我的目的,最近我開發的部落格系統,其中有個匯入第三方部落格的外掛,這個外掛比較簡單就是一個搜尋框,在對應的搜尋框裡面填寫URL,點選搜尋即可匯入到自己的部落格。
以匯入部落格園單篇文章為例:
下面是我的原始碼(單篇文章匯入,我已經將其封裝成一個工具類):
import cn.hutool.core.date.DateUtil; import com.blog.springboot.dto.CnBlogModelDTO; import com.blog.springboot.entity.Posts; import com.blog.springboot.service.PostsService; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.stereotype.Component; import us.codecraft.webmagic.Page; import us.codecraft.webmagic.Site; import us.codecraft.webmagic.Spider; import us.codecraft.webmagic.pipeline.ConsolePipeline; import us.codecraft.webmagic.processor.PageProcessor; import us.codecraft.webmagic.selector.Selectable; import javax.annotation.PostConstruct; /** * 匯入部落格園文章工具類 */ @Component public class WebMagicCnBlogUtils implements PageProcessor { @Autowired private PostsService postService; public static WebMagicCnBlogUtils magicCnBlogUtils; @PostConstruct public void init() { magicCnBlogUtils = this; magicCnBlogUtils.postService = this.postService; } private Site site = Site.me() .setDomain("https://www.cnblogs.com/") .setSleepTime(1000) .setUserAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/67.0.3396.99 Safari/537.36"); @Override public void process(Page page) { Selectable obj = page.getHtml().xpath("//div[@class='post']"); Selectable title = obj.xpath("//h1[@class='postTitle']//a"); Selectable content = obj.xpath("//div[@class='blogpost-body']"); System.out.println("title:" + title.replace("<[^>]*>","")); System.out.println("content:" + content); CnBlogModelDTO blog = new CnBlogModelDTO(); blog.setTitle(title.toString()); blog.setContent(content.toString()); Posts post = new Posts(); String date = DateUtil.date().toString(); post.setPostAuthor(1L); post.setPostTitle(title.replace("<[^>]*>","").toString()); post.setPostContent(content.toString()); post.setPostExcerpt(content.replace("<[^>]*>","").toString()); post.setPostDate(date); post.setPostDate(date); post.setPostModified(date); boolean importPost = magicCnBlogUtils.postService.insert(post); if (importPost) { System.out.println("success"); } else { System.out.println("fail"); } } @Override public Site getSite() { return site; } /** * 匯入單篇部落格園文章資料 * * @param url */ public static void importSinglePost(String url) { Spider.create(new WebMagicCnBlogUtils()) .addUrl(url) .addPipeline(new ConsolePipeline()) .run(); } }
單元測試程式碼:
import com.blog.springboot.dto.CnBlogModelDTO; import us.codecraft.webmagic.Page; import us.codecraft.webmagic.Site; import us.codecraft.webmagic.Spider; import us.codecraft.webmagic.pipeline.ConsolePipeline; import us.codecraft.webmagic.processor.PageProcessor; import us.codecraft.webmagic.selector.Selectable; public class WebMagicJunitTest implements PageProcessor { private Site site = Site.me() .setDomain("https://www.cnblogs.com/") .setSleepTime(1000) .setUserAgent("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML,like Gecko) Chrome/67.0.3396.99 Safari/537.36"); @Override public void process(Page page) { Selectable obj = page.getHtml().xpath("//div[@class='post']"); Selectable title = obj.xpath("//h1[@class='postTitle']//a"); Selectable content = obj.xpath("//div[@class='blogpost-body']"); System.out.println("title:" + title.replace("<[^>]*>","")); System.out.println("content:" + content); } @Override public Site getSite() { return site; } public static void importSinglePost(String url) { Spider.create(new WebMagicJunitTest()) .addUrl(url) .addPipeline(new ConsolePipeline()) .run(); } public static void main(String[] args) { WebMagicJunitTest.importSinglePost("https://www.cnblogs.com/youcong/p/9404007.html"); }
另外我是怎麼知道要爬取哪些資料呢?
需求第一,然後通過Chrome或Firefox瀏覽器檢查元素,如圖:
到此這篇關於Java爬蟲框架之WebMagic實戰的文章就介紹到這了,更多相關Java WebMagic內容請搜尋我們以前的文章或繼續瀏覽下面的相關文章希望大家以後多多支援我們!