1. 程式人生 > >avro序列化例項

avro序列化例項

hadoop中,就是用的avro作為序列化的,現在按照官方文件,我們按照如下步驟來做這個例項。

  1. 構建maven工程,加入apache-avro依賴庫,以及外掛依賴。
  2. 編寫avro檔案
  3. 生成java實體類
  4. 編寫測試程式

pom.xml配置檔案

<dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.9</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.apache.avro</groupId>
      <artifactId>avro</artifactId>
      <version>1.8.2</version>
    </dependency>
    <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-log4j12</artifactId>
      <version>1.7.21</version>
    </dependency>
</dependencies>

<build>
    <plugins>
        <plugin>   
            <groupId>org.apache.avro</groupId>
            <artifactId>avro-maven-plugin</artifactId>
            <version>1.8.2</version>
            <executions>
            <execution>
            <phase>generate-sources</phase>
            <goals>
                <goal>schema</goal>
            </goals>
            <configuration>	
                <sourceDirectory>${project.basedir}/src/main/avro/</sourceDirectory>   
                <outputDirectory>${project.basedir}/src/main/java/</outputDirectory>
            </configuration>
            </execution>
            </executions>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <configuration>
                <source>1.8</source>
                <target>1.8</target>
            </configuration>
        </plugin>
    </plugins>
</build>

src/main/avro目錄下編寫user.avsc,注意字尾是avsc 

{
  "namespace":"com.xxx.avro.entity",
  "type":"record",
  "name":"User",
  "fields":[
        {"name":"id","type":"int"},
        {"name":"name","type":"string"},
        {"name":"age","type":["int","null"]},
        {"name":"mobile","type":["string","null"]}
  ]
}

這是一個用json格式表示的avro物件

因為我們的pom.xml配置檔案中配置了avro生成java的外掛,所以這裡直接執行Run As ->Maven install,構建成功之後,會在com.xxx.avro.entity目錄下生成User.java的檔案。

User.java檔案部分內容截圖:

接下來,編寫測試驗證:

執行結果:

eclipse下,使用avro-maven-plugin外掛,工程中的pom.xml會報錯: Plugin execution not covered by lifecycle configuration...這個錯誤提示上有解決辦法,要麼在pom.xml配置檔案中增加配置,忽略phase,要麼在lifecycle-mapping-metadata.xml配置檔案中增加配置忽略phase,然後重新載入lifecycle-mapping-metadata.xml檔案。這裡兩種解決辦法都貼出來。

1、在pom.xml中增加配置:

<pluginManagement>
   <plugins>
      <plugin>
         <groupId>org.eclipse.m2e</groupId>
         <artifactId>lifecycle-mapping</artifactId>
         <version>1.0.0</version>
         <configuration>
            <lifecycleMappingMetadata>
            <pluginExecutions>
               <pluginExecution>
                  <pluginExecutionFilter>
                  <groupId>org.apache.avro</groupId>
                  <artifactId>avro-maven-plugin</artifactId>
                  <versionRange>[1.8.2,)</versionRange>
                  <goals>
                  	<goal>schema</goal>
                  </goals>
                  </pluginExecutionFilter>
                  <action>
                  	<ignore></ignore>
                  </action>
               </pluginExecution>
            </pluginExecutions>
            </lifecycleMappingMetadata>
         </configuration>
      </plugin>
   </plugins>
</pluginManagement>

2、在lifecycle-mapping-metadata.xml中增加配置:

<lifecycleMappingMetadata>
  <pluginExecutions>
    <pluginExecution>
      <pluginExecutionFilter>
        <groupId>org.apache.avro</groupId>
        <artifactId>avro-maven-plugin</artifactId>
        <versionRange>1.8.2</versionRange>
        <goals>
          <goal>schema</goal>
        </goals>
      </pluginExecutionFilter>
      <action>
        <ignore />
      </action>
    </pluginExecution>
  </pluginExecutions>
</lifecycleMappingMetadata>