Avro序列化和RPC實現
阿新 • • 發佈:2019-01-07
序列化和反序列化
- Maven:Pom.xml
<dependencies> <dependency> <groupId>org.apache.avro</groupId> <artifactId>avro</artifactId> <version>1.8.2</version> </dependency> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.12</version> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.apache.avro</groupId> <artifactId>avro-maven-plugin</artifactId> <version>1.8.1</version> <executions> <execution> <phase>generate-sources</phase> <goals> <goal>schema</goal> </goals> <configuration> <sourceDirectory>${project.basedir}/src/main/avro/</sourceDirectory> <outputDirectory>${project.basedir}/src/main/java/</outputDirectory> </configuration> </execution> </executions> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <configuration> <source>1.8</source> <target>1.8</target> </configuration> </plugin> </plugins> </build>
- Avro:MapAvro.avsc
{ "type":"record", "namespace":"com.qidai", "name":"Employee", "doc":"Test Employee bean", "fields":[ {"name":"id","type":["null","int"]}, {"name":"name","type":["null","string"]}, {"name":"age","type":["null","int"]}, {"name":"gender","type":["null","string"]} ] }
- 新增完依賴後直接點選maven外掛install即可,就會產生對應的class
- 序列化和反序列化(使用javaclass)
@Test public void ser() throws Exception { Employee employee = Employee.newBuilder().setAge(12).setGender("NAN").setId(1).setName("tom").build(); DatumWriter<Employee> employeeDatumWriter = new SpecificDatumWriter<>(Employee.class); DataFileWriter<Employee> dataFileWriter = new DataFileWriter<>(employeeDatumWriter); dataFileWriter.create(employee.getSchema(), new File("emp.avro")); dataFileWriter.append(employee); dataFileWriter.close(); } @Test public void deSer() throws Exception { DatumReader<Employee> employeeDatumReader = new SpecificDatumReader<>(Employee.class); DataFileReader<Employee> dataFileReader = new DataFileReader<>(new File("emp.avro"), employeeDatumReader); Employee employee = null; while (dataFileReader.hasNext()) { employee = dataFileReader.next(); System.out.println(employee); } }
- 序列化反序列化(直接使用avro檔案)
@Test
public void ser() throws Exception {
Schema schema = new Schema.Parser().parse(new File("emp.avsc"));
GenericRecord empRecord1 = new GenericData.Record(schema);
empRecord1.put("id", 1);
empRecord1.put("name", "Ben");
empRecord1.put("age", 7);
empRecord1.put("gender", "nv");
File file = new File("empser.avro");
DatumWriter<GenericRecord> datumWriter = new GenericDatumWriter<>(schema);
DataFileWriter<GenericRecord> dataFileWriter = new DataFileWriter<>(datumWriter);
dataFileWriter.create(schema, file);
dataFileWriter.append(empRecord1);
dataFileWriter.close();
}
@Test
public void deSer() throws Exception {
//指定avro格式檔案
Schema schema = new Schema.Parser().parse(new File("emp.avsc"));
//指定序列化好的資料檔案
File file = new File("empser.avro");
DatumReader<GenericRecord> datumReader = new GenericDatumReader<>(schema);
DataFileReader<GenericRecord> dataFileReader = new DataFileReader<>(file, datumReader);
GenericRecord emp = null;
while (dataFileReader.hasNext()) {
emp = dataFileReader.next();
System.out.println(emp);
}
}
RPC實現
- 編寫avsc檔案:user.avsc,作為要傳送的實體類
{
"namespace":"com.qidai.bean",
"name":"User",
"doc":"test rpc class",
"type":"record",
"fields":[
{"name":"name","type":["string","null"]},
{"name":"age","type":["int","null"]},
{"name":"date","type":["string","null"]}
]
}
- 編寫協議檔案:protomes.avdl
@namespace("com.qidai.proto")
protocol UserProtocol{
import schema "user.avsc";
string sendMes(com.qidai.bean.User user); //對應協議中的方法
}
- 編寫完畢之後檢查maven pom檔案
<dependencies>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
<version>1.8.2</version>
</dependency>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro-ipc</artifactId>
<version>1.8.2</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.avro</groupId>
<artifactId>avro-maven-plugin</artifactId>
<version>1.8.1</version>
<executions>
<execution>
<phase>generate-sources</phase>
<goals>
<goal>schema</goal>
<goal>idl-protocol</goal>
</goals>
<configuration>
<sourceDirectory>${project.basedir}/src/main/avro/</sourceDirectory>
<outputDirectory>${project.basedir}/src/main/java/</outputDirectory>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
</plugins>
</build>
- 檢查沒錯之後,直接外掛install生成avro類
- 生成之後會看到一個User的傳輸類,還有一個proto包下的協議介面
- 實現協議介面,重寫自己的邏輯:server
public class UserProtocolImpl implements UserProtocol {
@Override
public CharSequence sendMes(User user) throws AvroRemoteException {
System.out.println("Server --> 接收" + user.toString());
return "Client sendMes " + user.toString();
}
}
- 實現協議介面,重寫自己的邏輯:client
public class UserProtocolImpl implements UserProtocol {
@Override
public CharSequence sendMes(User user) throws AvroRemoteException {
System.out.println("Client --> 傳送" + user.toString());
return "Client sendMes " + user.toString();
}
}
- 編寫serverApp
public class ServerApp {
public static void main(String[] args) throws IOException, InterruptedException {
Responder responder = new SpecificResponder(UserProtocol.class,new UserProtocolImpl());
SaslSocketServer server = new SaslSocketServer(responder,new InetSocketAddress(9999));
server.start();
Thread.sleep(5000000);
}
}
- 編寫clientApp
public class ClientApp {
public static void main(String[] args) throws IOException {
SaslSocketTransceiver transceiver = new SaslSocketTransceiver(new InetSocketAddress(9999));
UserProtocol client = SpecificRequestor.getClient(UserProtocol.class, transceiver);
Scanner scanner = new Scanner(System.in);
while (scanner.next() != null) {
User user = new User();
user.setName("xiaofen");
System.out.println(client.sendMes(user));
}
}
}
-
測試:先啟動serverApp,然後啟動clientApp,這時候在client的console中輸入任意值,會發現服務端和客戶端都會輸出訊息
- server
Server --> {"name": "xiaofen", "age": null, "date": null} Server --> {"name": "xiaofen", "age": null, "date": null} Server --> {"name": "xiaofen", "age": null, "date": null} Server --> {"name": "xiaofen", "age": null, "date": null} Server --> {"name": "xiaofen", "age": null, "date": null} Server --> {"name": "xiaofen", "age": null, "date": null} Server --> {"name": "xiaofen", "age": null, "date": null}
- client
1 Client sendMes {"name": "xiaofen", "age": null, "date": null} 1 Client sendMes {"name": "xiaofen", "age": null, "date": null} 1 Client sendMes {"name": "xiaofen", "age": null, "date": null} 1 Client sendMes {"name": "xiaofen", "age": null, "date": null} 1 Client sendMes {"name": "xiaofen", "age": null, "date": null} 1 Client sendMes {"name": "xiaofen", "age": null, "date": null} 1 Client sendMes {"name": "xiaofen", "age": null, "date": null}
- 對於server的client的實現有如下幾種
1.基於jetty的http實現:HttpServer 和HttpTransceiver
2.基於netty的實現:NettyServer和NettyTransceiver
3.基於TCP的實現:SocketServer和SocketTransceiver
4.基於UDP的實現:DatagramServer和DatagramTransceiver
5.基於加密的TCP實現:SaslSocketServer和SaslSocketTransceiver