Java實現的基於模板的網頁結構化資訊精準抽取元件:HtmlExtractor
阿新 • • 發佈:2019-02-19
HtmlExtractor由2個子專案構成,html-extractor和html-extractor-web。 html-extractor實現了資料抽取邏輯,是從節點,html-extractor-web提供web介面來維護抽取規則,是主節點。 html-extractor是一個jar包,可通過maven引用: <dependency> <groupId>org.apdplat</groupId> <artifactId>html-extractor</artifactId> <version>1.0</version> </dependency> html-extractor-web是一個war包,需要部署到Servlet/Jsp容器上。
單機集中式使用方法:
//1、構造抽取規則 List<UrlPattern> urlPatterns = new ArrayList<>(); //1.1、構造URL模式 UrlPattern urlPattern = new UrlPattern(); urlPattern.setUrlPattern("http://money.163.com/\\d{2}/\\d{4}/\\d{2}/[0-9A-Z]{16}.html"); //1.2、構造HTML模板 HtmlTemplate htmlTemplate = new HtmlTemplate(); htmlTemplate.setTemplateName("網易財經頻道"); htmlTemplate.setTableName("finance"); //1.3、將URL模式和HTML模板建立關聯 urlPattern.addHtmlTemplate(htmlTemplate); //1.4、構造CSS路徑 CssPath cssPath = new CssPath(); cssPath.setCssPath("h1"); cssPath.setFieldName("title"); cssPath.setFieldDescription("標題"); //1.5、將CSS路徑和模板建立關聯 htmlTemplate.addCssPath(cssPath); //1.6、構造CSS路徑 cssPath = new CssPath(); cssPath.setCssPath("div#endText"); cssPath.setFieldName("content"); cssPath.setFieldDescription("正文"); //1.7、將CSS路徑和模板建立關聯 htmlTemplate.addCssPath(cssPath); //可象上面那樣構造多個URLURL模式 urlPatterns.add(urlPattern); //2、獲取抽取規則物件 ExtractRegular extractRegular = ExtractRegular.getInstance(urlPatterns); //注意:可通過如下3個方法動態地改變抽取規則 //extractRegular.addUrlPatterns(urlPatterns); //extractRegular.addUrlPattern(urlPattern); //extractRegular.removeUrlPattern(urlPattern.getUrlPattern()); //3、獲取HTML抽取工具 HtmlExtractor htmlExtractor = HtmlExtractor.getInstance(extractRegular); //4、抽取網頁 String url = "http://money.163.com/08/1219/16/4THR2TMP002533QK.html"; List<ExtractResult> extractResults = htmlExtractor.extract(url, "gb2312"); //5、輸出結果 int i = 1; for (ExtractResult extractResult : extractResults) { System.out.println((i++) + "、網頁 " + extractResult.getUrl() + " 的抽取結果"); for(ExtractResultItem extractResultItem : extractResult.getExtractResultItems()){ System.out.print("\t"+extractResultItem.getField()+" = "+extractResultItem.getValue()); } System.out.println("\tdescription = "+extractResult.getDescription()); System.out.println("\tkeywords = "+extractResult.getKeywords()); }