多執行緒批量檢測未註冊域名
阿新 • • 發佈：2018-12-05
最近想註冊一個域名，使用萬網嘗試了很多域名，基本都已被註冊。之前聽說雙拼域名很火，所以想寫個指令碼，看看哪些雙拼域名還未被註冊。
一、查詢介面
網上搜索了一下，萬網的域名查詢介面比較簡單易用，查詢URL格式為: http://panda.www.net.cn/cgi-bin/check.cgi?area_domain=aaa.com
返回值及含義:
210 : Domain name is available
211 : Domain name is not available
212 : Domain name is invalid
214 : Unknown error
二、程式設計思路
1. DomainGenerator讀取檔案pinyin.txt，獲取所有可用的拼音字母。遍歷拼音字母， 組裝成雙拼域名。這個拼音列表是從網上搜索來的，可能會有紕漏。
2. 建立域名檢測執行緒DomainRunner，每個執行緒採用httpclient呼叫萬網的域名查詢介面。
3. 每個執行緒呼叫DomainValidator檢查返回結果。
4. 執行緒ResultRunner將可用域名寫入domain.txt檔案。
三、核心程式碼
DomainGenerator.java， 啟動類，讀取拼音列表，組裝需要檢測的域名，建立檢測執行緒和結果處理執行緒。
-
package com.learnworld;
-
-
import java.util.List;
-
import
-
import java.io.BufferedWriter;
-
import java.io.FileReader;
-
import java.io.FileWriter;
-
import java.util.ArrayList;
-
import java.util.concurrent.ArrayBlockingQueue;
-
import java.util.concurrent.CountDownLatch;
-
import java.util.concurrent.ExecutorService;
-
import java.util.concurrent.Executors;
-
import java.util.concurrent.LinkedBlockingQueue;
-
import java.util.concurrent.atomic.AtomicInteger;
-
-
import org.apache.http.impl.client.CloseableHttpClient;
-
import org.apache.http.impl.client.HttpClients;
-
import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
-
-
public
class DomainGenerator {
-
-
public static void main(String[] args){
-
// pinyin list, read from pinyin.txt
-
List<String> items =
new ArrayList<String>();
-
// domain list, which need to check
-
ArrayBlockingQueue<String> taskQueue =
new ArrayBlockingQueue<String>(
163620);
-
// available domain list, which need to save into file
-
LinkedBlockingQueue<String> resultQueue =
new LinkedBlockingQueue<String>();
-
// counter, need to count unavailable domain statistical information
-
AtomicInteger count =
new AtomicInteger(
0);
-
-
// Httpclient initialization
-
PoolingHttpClientConnectionManager cm =
new PoolingHttpClientConnectionManager();
-
cm.setMaxTotal(
20);
-
cm.setDefaultMaxPerRoute(
20);
-
CloseableHttpClient httpClient = HttpClients.custom().setConnectionManager(cm).build();
-
-
try {
-
// pinyin.txt, used to save all available pinyin
-
BufferedReader reader =
new BufferedReader(
new FileReader(
"pinyin.txt"));
-
// domain.txt, used to save all available domain result
-
BufferedWriter writer =
new BufferedWriter(
new FileWriter(
"domain.txt"));
-
-
String item =
null;
-
while((item = reader.readLine()) !=
null){
-
items.add(item);
-
}
-
-
// generate domain list
-
for (String item1 : items){
-
for (String item2 : items) {
-
taskQueue.offer(item1 + item2 +
".com");
-
}
-
}
-
-
int domainThreadNum =
3;
-
CountDownLatch downLatch =
new CountDownLatch(domainThreadNum);
-
ExecutorService executor = Executors.newFixedThreadPool(domainThreadNum +
1);
-
-
// start domain check thread
-
for(
int i =
0; i < domainThreadNum; i++){
-
executor.execute(
new DomainRunner(taskQueue, resultQueue, downLatch, count, httpClient));
-
}
-
-
// start result handle thread
-
executor.execute(
new ResultRunner(resultQueue, writer));
-
-
downLatch.await();
-
System.out.println(
"All tasks are done!");
-
-
// TODO, suggest use volatile flag to control ResultRunner
-
executor.shutdownNow();
-
-
reader.close();
-
writer.close();
-
httpClient.close();
-
}
catch (Exception e) {
-
e.printStackTrace();
-
}
-
-
}
-
}
DomainRunner：域名檢測執行緒，從域名domainQueue中讀取域名，呼叫介面進行檢測。 如果域名可用，將結果放入resultQueue中等待寫入檔案。
-
package com.learnworld;
-
-
import java.io.IOException;
-
import java.util.Calendar;
-
import java.util.concurrent.ArrayBlockingQueue;
-
import java.util.concurrent.CountDownLatch;
-
import java.util.concurrent.LinkedBlockingQueue;
-
import java.util.concurrent.atomic.AtomicInteger;
-
-
import org.apache.http.HttpEntity;
-
import org.apache.http.client.config.RequestConfig;
-
import org.apache.http.client.methods.CloseableHttpResponse;
-
import org.apache.http.client.methods.HttpGet;
-
import org.apache.http.impl.client.CloseableHttpClient;
-
import org.apache.http.protocol.BasicHttpContext;
-
import org.apache.http.protocol.HttpContext;
-
import org.apache.http.util.EntityUtils;
-
-
public
class DomainRunner implements Runnable {
-
-
private ArrayBlockingQueue<String> domainQueue;
-
private LinkedBlockingQueue<String> resultQueue;
-
private CountDownLatch downLatch;
-
private AtomicInteger count;
-
private CloseableHttpClient httpClient;
-
-
public DomainRunner(ArrayBlockingQueue<String> domainQueue,
-
LinkedBlockingQueue<String> resultQueue, CountDownLatch downLatch,
-
AtomicInteger count, CloseableHttpClient httpClient) {
-
super();
-
this.domainQueue = domainQueue;
-
this.resultQueue = resultQueue;
-
this.downLatch = downLatch;
-
this.count = count;
-
this.httpClient = httpClient;
-
}
-
-
@Override
-
public void run() {
-
String domain =
null;
-
while ((domain = domainQueue.poll()) !=
null) {
-
boolean isDomainAvailable =
false;
-
-
RequestConfig requestConfig = RequestConfig.custom()
-
.setSocketTimeout(
5000)
-
.setConnectTimeout(
5000)
-
.setConnectionRequestTimeout(
5000)
-
.build();
-
-
HttpGet httpGet =
new HttpGet(
"http://panda.www.net.cn/cgi-bin/check.cgi?area_domain=" + domain);
-
httpGet.setConfig(requestConfig);
-
httpGet.setHeader(
"Connection",
"close");
-
HttpContext context =
new BasicHttpContext();
-
CloseableHttpResponse response =
null;
-
try {
-
response = httpClient.execute(httpGet, context);
-
HttpEntity entity = response.getEntity();
-
int status = response.getStatusLine().getStatusCode();
-
if (status >=
200 && status <
300) {
-
String resultXml = EntityUtils.toString(entity);
-
isDomainAvailable = DomainValidator.isAvailableDomainForResponse(resultXml);
-
EntityUtils.consumeQuietly(entity);
-
}
else {
-
System.out.println(domain +
" check error.");
-
}
-
}
catch (Exception e) {
-
e.printStackTrace();
-
}
finally {
-
try {
-
httpGet.releaseConnection();
-
if (response !=
null) {
-
response.close();
-
}
-
-
}
catch (IOException e) {
-
e.printStackTrace();
-
}
-
}
-
-
// result handle
-
if(isDomainAvailable) {
-
resultQueue.offer(domain);
-
}
else {
-
int totalInvalid = count.addAndGet(
1);
-
if (totalInvalid %
100 ==
0) {
-
System.out.println(totalInvalid +
" " + Calendar.getInstance().getTime());
-
}
-
}
-
}
-
-
downLatch.countDown();
-
-
}
-
-
}
DomainValidator: 對萬網返回結果進行檢查，判斷域名是否可用。
-
package com.learnworld;
-
-
public
class DomainValidator {
-
-
public static boolean isAvailableDomainForResponse(String responseXml){
-
if(responseXml ==
null || responseXml.isEmpty()){
-
return
false;
-
}
-
-
if(responseXml.contains(
"<original>210")){
-
return
true;
-
}
else
if(responseXml.contains(
"<original>211")
-
|| responseXml.contains(
"<original>212")
-
|| responseXml.contains(
"<original>214")){
-
return
false;
-
}
else {
-
System.out.println(
"api callback error!");
-
try {
-
Thread.sleep(
60000);
-
}
catch (InterruptedException e) {
-
e.printStackTrace();
-
}
-
-
return
false;
-
}
-
}
-
-
}
ResultRunner: 結果處理執行緒，將可用域名寫入檔案domain.txt中。
-
<