dsg_08 serialization system

Serialization

The purpose of serialization is to reduce the load of the network
Serialization technology:

  • google protobuf
  • avro
  • java built-in serialization

Relatively speaking, java serialization is worse than google protobuf and

google protobuf

  1. Download protocol-2.5.0-win32 zip
  2. Design object
  3. Description object
  4. Compile description
  5. Import source code into the project
  6. Use object

Description language of google protobuf

Field type
protobuf type java type
double double
float float
int32 int
int64 long
bool boolean
string String
demo
  1. Define a Person serialization class description

    //Define package name
    option java_package = "cn.com.trueway.pojo";
    option java_outer_classname = "AddressBookProtos";
    message Person{
    	//required indicates that the field must be assigned a value
    	required string username = 1;
    	//optional indicates that the field can be empty
    	optional string xb = 2;
    	//repeated specifies that the field is a collection
    	repeated string hobbyList;
    }
    
  2. Open the command window and compile the description file to generate java serialized objects

    	#protoc. exe --java_ The generation path of the out class is a description file that needs to be edited
    	protoc.exe --java_out . person.proto
    
  3. Copy the generated java serialized object to the project

    /**
     * Serialization
     * @throws Exception
     */
    public static void writeInJava() throws Exception {
        List<String> hobbyList = new ArrayList<>();
        hobbyList.add("Play basketball");
        hobbyList.add("play football");
        hobbyList.add("Swimming");
        AddressBookProtos.Person person = AddressBookProtos.Person.newBuilder().setUsername("Zhang San").setXb("male").addAllHobbyList(hobbyList).build();
    
        person.writeTo(new FileOutputStream("D:/aa/person.data"));
    }
    
    /**
     * Anti concatenation line
     * @throws Exception
     */
    public static void read() throws Exception{
        AddressBookProtos.Person person = AddressBookProtos.Person.parseFrom(new FileInputStream("D:/aa/person.data"));
        System.out.println(person.getUsername());
    }
    

avro

characteristic:

  1. Self description language, data structure and data are stored in files, cross language, using json format to store data
  2. Compressible and cuttable

schema syntax

schema data type
data type describe
null No value type
int 32 byte signed integer
long 64 byte signed integer
float 32-bit floating point number
double 32-bit floating point number
bytes 8-bit byte type
string Unicode character sequence
Inherent attribute
Attribute name describe
name The name at the same level as fields is the object name, and in fields is the attribute name
namespace Namespace
type The type at the same level as fields is the description document, and in fields is the data type
fields Attribute array

Use of avro

Compile mode
  1. Define schema file

    {
     "type":"record",
     "namespace":"cn.com.trueway.pojo",
     "name":"PersonAvro",
     "fields":[
     	{
     		"name":"username", 
     		"type":"string" 
     	},
     	{ 
     		"name":"xb",
     		"type":"string" 
     	},
     	{ 
     		"name":"age",
     		"type":"int" 
     	}
     ]
    }
    
  2. In avro-tools-1.8.0 Open the command window under the jar package directory of jar and enter the following command to compile the schema and generate java classes

    #Format: Java - jar avro-tools-1.8.0 Jar compile schema file name generation path
    java -jar avro-tools-1.8.0.jar compile schema person.avsc .
    
  3. Import the generated java class into the project and import the dependency

    <dependency>
        <groupId>org.apache.avro</groupId>
        <artifactId>avro</artifactId>
        <version>1.8.1</version>
    </dependency>
    
    import org.apache.avro.file.DataFileReader;
    import org.apache.avro.file.DataFileWriter;
    import org.apache.avro.specific.SpecificDatumReader;
    import org.apache.avro.specific.SpecificDatumWriter;
    	/**
         * Serialization
         * @throws Exception
         */
        public static void write() throws Exception {
            //Create writer object
            SpecificDatumWriter<PersonAvro> sdw = new SpecificDatumWriter<>(PersonAvro.class);
            //write file
            DataFileWriter<PersonAvro> dw = new DataFileWriter<>(sdw);
            PersonAvro pa = new PersonAvro();
            pa.setUsername("Zhang San");
            pa.setXb("man");
            pa.setAge(23);
            //Write class description to file
            dw.create(pa.getSchema(),new File("D:\\aa\\person.avro"));
            //Add data to
            dw.append(pa);
            dw.close();
        }
    
        /**
         * Anti serialization
         * @throws Exception
         */
        public static void read() throws Exception {
            SpecificDatumReader<PersonAvro> sdr = new SpecificDatumReader<>(PersonAvro.class);
    
            DataFileReader<PersonAvro> dr = new DataFileReader<>(new File("D:\\aa\\person.avro"), sdr);
    
            Iterator<PersonAvro> iterator = dr.iterator();
    
            while (iterator.hasNext()){
                System.out.println(iterator.next().getUsername());
            }
        }
    
Non compiled mode
  1. Define schema file

    {
     "type":"record",
     "namespace":"cn.com.trueway.pojo",
     "name":"PersonAvro",
     "fields":[
     	{
     		"name":"username", 
     		"type":"string" 
     	},
     	{ 
     		"name":"xb",
     		"type":"string" 
     	},
     	{ 
     		"name":"age",
     		"type":"int" 
     	}
     ]
    }
    
  2. Import dependency

    <dependency>
        <groupId>org.apache.avro</groupId>
        <artifactId>avro</artifactId>
        <version>1.8.1</version>
    </dependency>
    
  3. Serial deserialization

    	/**
         * Serialization
         * @throws Exception
         */
        public static void write2() throws Exception {
            //Specifies the schema file for the definition
            Schema schema = new Schema.Parser().parse(new File("E:\\big data\\05-Big data serial number Avro and Protobuf\\person.avsc"));
            //Creating GenericRecord is equivalent to a compiled object
            GenericRecord gr = new GenericData.Record(schema);
            gr.put("username","Zhang San");
            gr.put("xb","man");
            gr.put("age",23);
            DatumWriter dw = new SpecificDatumWriter (schema);
            DataFileWriter dw2 = new DataFileWriter(dw);
            dw2.create(schema,new File("D:\\aa\\person2.avro"));
            dw2.append(gr);
            dw2.close();
        }
    
        /**
         * Anti serialization
         * @throws Exception
         */
        public static void read2() throws Exception {
            //Specifies the schema file for the definition
            Schema schema = new Schema.Parser().parse(new File("E:\\big data\\05-Big data serial number Avro and Protobuf\\person.avsc"));
            DatumReader dr = new SpecificDatumReader (schema);
            DataFileReader r2 = new DataFileReader(new File("D:\\aa\\person2.avro"),dr);
            while(r2.hasNext()){
                GenericRecord rec = (GenericRecord)r2.next();
                System.out.println(rec.get("username"));
            }
            r2.close();
        }
    

Tags: Big Data

Posted by Xster on Fri, 13 May 2022 05:58:34 +0300