Go launched http2 0 request flow analysis (later) - header compression

Reading suggestions

This is HTTP 2 0 series, the author recommends reading in the following order:

  1. HTTP request in Go -- http1 1 request process analysis
  2. Go launched http2 0 request process analysis (previous)
  3. Go launched http2 0 request flow analysis (Part 2) -- data frame & flow control

review

In the previous article (* http2clientconn) The roundtrip method refers to the write request header, which needs to be encoded before writing the request header (see the source code) github.com/golang/go/b...

In the second part (* http2clientconn) The ReadFrame() method is mentioned in the readloop method. This method will read the data frame. If it is http2FrameHeaders data frame, it will call (* http2framer) Readmetaframe decodes the read data frame (see for source code) github.com/golang/go/b...

Because header compression has high independence, the author realizes a small example that can run independently based on the source code of the above-mentioned encoding / decoding part. This article will analyze the header compression based on the example of its own implementation (for a complete example, see github.com/Isites/go-c...

come to the point

HTTP2 uses HPACK compression format to compress request and response header metadata. This format adopts the following two technologies:

  1. The transmitted header field is encoded by static Huffman code, so as to reduce the size of data transmission.
  2. In a single connection, the client and server jointly maintain the same header field index list (called HPACK index list by the author), which is used as a reference for encoding and decoding in subsequent transmission.

This article does not elaborate too much on Huffman coding, but mainly analyzes the index list maintained by both ends.

The HPACK compression context includes a static table and a dynamic table: the static table is defined in the specification and provides a list of common HTTP header fields that may be used by all connections; The dynamic table is initially empty and will be updated based on the values exchanged within a particular connection.

HPACK index list

To understand static / dynamic tables, you need to first understand the headerFieldTable structure. Both dynamic tables and static tables are implemented based on it.

type headerFieldTable struct {
    // As in hpack, unique ids  are 1-based. The unique id for ents[k] is k + evictCount + 1.
    ents       []HeaderField
    evictCount uint64

    // byName maps a HeaderField name to the unique id of the newest entry with the same name.
    byName map[string]uint64

    // byNameValue maps a HeaderField name/value pair to the unique id of the newest
    byNameValue map[pairNameValue]uint64
}
Copy code

The above fields will be described below:

Entries: the abbreviation of entries, which represents the currently indexed Header data. In the headerFieldTable, each Header has a unique ID. take ents[k] as an example, the unique ID is calculated as k + evictCount + 1.

evictCount: the number of entries that have been deleted from ents.

byName: stores the unique Id of the Header with the same Name. The Name of the latest Header will overwrite the old unique Id.

byNameValue: store the corresponding unique Id with the Name and Value of the Header as the key.

After understanding the meaning of the field, the following describes several important behaviors of the headerFieldTable.

(*headerFieldTable).addEntry: adds a Header entity to the table

func (t *headerFieldTable) addEntry(f HeaderField) {
    id := uint64(t.len()) + t.evictCount + 1
    t.byName[f.Name] = id
    t.byNameValue[pairNameValue{f.Name, f.Value}] = id
    t.ents = append(t.ents, f)
}
Copy code

First, calculate the unique Id of the Header in the headerFieldTable and store it in byName and byNameValue respectively. Finally, save the Header into entries.

Because the append function is used, this means that ents[0] stores the longest surviving Header.

(*headerFieldTable).evictOldest: deletes the specified number of Header entities from the table

func (t *headerFieldTable) evictOldest(n int) {
    if n > t.len() {
        panic(fmt.Sprintf("evictOldest(%v) on table with %v entries", n, t.len()))
    }
    for k := 0; k < n; k++ {
        f := t.ents[k]
        id := t.evictCount + uint64(k) + 1
        if t.byName[f.Name] == id {
            delete(t.byName, f.Name)
        }
        if p := (pairNameValue{f.Name, f.Value}); t.byNameValue[p] == id {
            delete(t.byNameValue, p)
        }
    }
    copy(t.ents, t.ents[n:])
    for k := t.len() - n; k < t.len(); k++ {
        t.ents[k] = HeaderField{} // so strings can be garbage collected
    }
    t.ents = t.ents[:t.len()-n]
    if t.evictCount+uint64(n) < t.evictCount {
        panic("evictCount overflow")
    }
    t.evictCount += uint64(n)
}
Copy code

The subscript of the first for loop starts from 0, that is, the first in first out principle is followed when deleting the Header. The steps to delete Header are as follows:

  1. Delete the mapping of byName and byNameValue.
  2. Move the Header after the n th bit forward.
  3. Empty the last n headers to facilitate garbage collection.
  4. Change the length of ents.
  5. Increase the number of evictCount.

(*headerFieldTable).search: search the specified Header from the current table and return the Index in the current table (the meaning of the Index here is different from that of the subscript in the slice)

func (t *headerFieldTable) search(f HeaderField) (i uint64, nameValueMatch bool) {
    if !f.Sensitive {
        if id := t.byNameValue[pairNameValue{f.Name, f.Value}]; id != 0 {
            return t.idToIndex(id), true
        }
    }
    if id := t.byName[f.Name]; id != 0 {
        return t.idToIndex(id), false
    }
    return 0, false
}
Copy code

If the Name and Value of the Header match, the Index in the current table is returned and nameValueMatch is true.

If only the Name of the Header matches, the Index in the current table is returned and nameValueMatch is false.

If the Name and Value of the Header do not match, 0 is returned and nameValueMatch is false.

(*headerFieldTable).idToIndex: calculate the Index corresponding to the current table through the unique Id in the current table

func (t *headerFieldTable) idToIndex(id uint64) uint64 {
    if id <= t.evictCount {
        panic(fmt.Sprintf("id (%v) <= evictCount (%v)", id, t.evictCount))
    }
    k := id - t.evictCount - 1 // convert id to an index t.ents[k]
    if t != staticTable {
        return uint64(t.len()) - k // dynamic table
    }
    return k + 1
}
Copy code

Static table: when the Index starts from 1 and the Index is 1, the corresponding element is t.ents[0].

Dynamic table: Index also starts from 1, but when Index is 1, the corresponding element is t.ents[t.len()-1].

Static table

The static table contains some headers that each connection may use. The implementation is as follows:

var staticTable = newStaticTable()
func newStaticTable() *headerFieldTable {
    t := &headerFieldTable{}
    t.init()
    for _, e := range staticTableEntries[:] {
        t.addEntry(e)
    }
    return t
}
var staticTableEntries = [...]HeaderField{
    {Name: ":authority"},
    {Name: ":method", Value: "GET"},
    {Name: ":method", Value: "POST"},
  // Code omitted here
    {Name: "www-authenticate"},
}
Copy code

The above t.init function is only used to initialize t.byName and t.byNameValue. The author only shows some predefined headers here. For the complete predefined headers, see github.com/golang/go/b...

Dynamic table

The dynamic table structure is as follows:

type dynamicTable struct {
    // http://http2.github.io/http2-spec/compression.html#rfc.section.2.3.2
    table          headerFieldTable
    size           uint32 // in bytes
    maxSize        uint32 // current maxSize
    allowedMaxSize uint32 // maxSize may go up to this, inclusive
}
Copy code

The implementation of dynamic table is based on headerFieldTable. Compared with the original basic functions, the size limit of the table is increased, and other functions remain unchanged.

Static and dynamic tables form a complete list of HPACK indexes

The internal Index and internal unique Id in the dynamic / static table are introduced earlier. In a connection, the HPACK Index list is composed of static table and dynamic table. What is the HPACK Index in the connection at this time?

With this question in mind, let's look at the following structure:

 

In the above figure, the blue part represents the static table and the yellow part represents the dynamic table.

H1...Hn and H1 HM represents Header elements stored in static tables and dynamic tables, respectively.

In the HPACK index, the index of the static table part is consistent with the internal index of the static table. The index of the dynamic table part is the maximum value of the internal index of the dynamic table plus the static table index. In one connection, the Client and Server identify the unique Header element through the HPACK index.

HPACK coding

As we all know, HTTP2 header compression can reduce the transmission of many data. Next, let's compare the data size before and after coding through the following example:

var (
  buf     bytes.Buffer
  oriSize int
)
henc := hpack.NewEncoder(&buf)
headers := []hpack.HeaderField{
  {Name: ":authority", Value: "dss0.bdstatic.com"},
  {Name: ":method", Value: "GET"},
  {Name: ":path", Value: "/5aV1bjqh_Q23odCf/static/superman/img/topnav/baiduyun@2x-e0be79e69e.png"},
  {Name: ":scheme", Value: "https"},
  {Name: "accept-encoding", Value: "gzip"},
  {Name: "user-agent", Value: "Go-http-client/2.0"},
  {Name: "custom-header", Value: "custom-value"},
}
for _, header := range headers {
  oriSize += len(header.Name) + len(header.Value)
  henc.WriteField(header)
}
fmt.Printf("ori size: %v, encoded size: %v\n", oriSize, buf.Len())
//Output: ori size: 197, encoded size: 111
 Copy code

Note: in HTTP2, the definition of request and response header fields remains unchanged, with only some minor differences: all header field names are lowercase, and the request line is now divided into pseudo header fields: method,: scheme,: authority and: path.

In the above example, we can see that the original 197 byte header data is now only 111 bytes, reducing the amount of data by nearly half!

With a "lying trough, cow force!" My mood began to feel towards henc Writefield method debugging.

func (e *Encoder) WriteField(f HeaderField) error {
    e.buf = e.buf[:0]

    if e.tableSizeUpdate {
        e.tableSizeUpdate = false
        if e.minSize < e.dynTab.maxSize {
            e.buf = appendTableSize(e.buf, e.minSize)
        }
        e.minSize = uint32Max
        e.buf = appendTableSize(e.buf, e.dynTab.maxSize)
    }

    idx, nameValueMatch := e.searchTable(f)
    if nameValueMatch {
        e.buf = appendIndexed(e.buf, idx)
    } else {
        indexing := e.shouldIndex(f)
        if indexing {
            e.dynTab.add(f) // Add to dynamic table
        }

        if idx == 0 {
            e.buf = appendNewName(e.buf, f, indexing)
        } else {
            e.buf = appendIndexedName(e.buf, f, idx, indexing)
        }
    }
    n, err := e.w.Write(e.buf)
    if err == nil && n != len(e.buf) {
        err = io.ErrShortWrite
    }
    return err
}
Copy code

After debugging, it is found that in this example: authority,: path, accept encoding and user agent take the appendIndexedName branch method and scheme: take the appendIndexed branch; The custom header goes to the appendNewName branch. These three branches represent two different coding methods in total.

Since the default value of f.Sensitive in this example is false and the default size given by Encoder to the dynamic table is 4096, according to the logic of e.shouldIndex, indexing in this example has always been true (in the go1.14.2 source code used by the author, the client has not found any code that makes f.Sensitive true).

The reason why the author doesn't mention the above logic related to e.tableSizeUpdate is that the methods to control e.tableSizeUpdate are e.SetMaxDynamicTableSizeLimit and e.SetMaxDynamicTableSize, and the author is in (* http2transport) Such comments are found in the source code related to newclientconn (see the previous article for the relevant logic of this method):

// TODO: SetMaxDynamicTableSize, SetMaxDynamicTableSizeLimit on
// henc in response to SETTINGS frames?
Copy code

When I saw this, I was very excited and had a strong desire to contribute code. However, my ability was limited, I could only look at the opportunity but couldn't grasp it. I had to study hard with hatred (just kidding ~, after all, a wise man said that the less I wrote, the less bugs I wrote 😄).

(*Encoder).searchTable: search Header from HPACK index list and return the corresponding index.

func (e *Encoder) searchTable(f HeaderField) (i uint64, nameValueMatch bool) {
    i, nameValueMatch = staticTable.search(f)
    if nameValueMatch {
        return i, true
    }

    j, nameValueMatch := e.dynTab.table.search(f)
    if nameValueMatch || (i == 0 && j != 0) {
        return j + uint64(staticTable.len()), nameValueMatch
    }

    return i, false
}
Copy code

The search order is: first search the static table. If the static table does not match, search the dynamic table, and finally return.

Index Header notation

The function corresponding to this representation is appendIndexed, and the Header is already in the index list.

This function encodes the index of the Header in the HPACK index list. The original Header can be represented by only a few bytes.

func appendIndexed(dst []byte, i uint64) []byte {
    first := len(dst)
    dst = appendVarInt(dst, 7, i)
    dst[first] |= 0x80
    return dst
}
func appendVarInt(dst []byte, n byte, i uint64) []byte {
    k := uint64((1 << n) - 1)
    if i < k {
        return append(dst, byte(i))
    }
    dst = append(dst, byte(k))
    i -= k
    for ; i >= 128; i >>= 7 {
        dst = append(dst, byte(0x80|(i&0x7f)))
    }
    return append(dst, byte(i))
}
Copy code

According to appendIndexed, when using the index header field representation, the format of the first byte must be 0b1xxxxxx, that is, the 0 bit must be 1, and the lower 7 bits are used to represent the value.

If the index is larger than Uint64 ((1 < < n) - 1), you need to use multiple bytes to store the value of the index. The steps are as follows:

  1. The lowest n bits of the first byte are all 1.
  2. After subtracting Uint64 ((1 < < n) - 1) from index i, take the lower 7 bits or the upper 0b10000000 each time, and then move i 7 bits to the right and compare it with 128 to determine whether to enter the next cycle.
  3. After the loop ends, put the remaining i values directly into buf.

When using this method to represent a Header, only a few bytes are needed to represent a complete Header field. The best case is that one byte can represent a Header field.

Add dynamic table Header representation

This representation corresponds to two situations: first, the Name of the Header has a matching index; 2, There is no matching index for Name and Value of Header. The corresponding processing functions in these two cases are appendIndexedName and appendNewName. In both cases, the Header is added to the dynamic table.

appendIndexedName: Header field with matching code Name.

func appendIndexedName(dst []byte, f HeaderField, i uint64, indexing bool) []byte {
    first := len(dst)
    var n byte
    if indexing {
        n = 6
    } else {
        n = 4
    }
    dst = appendVarInt(dst, n, i)
    dst[first] |= encodeTypeByte(indexing, f.Sensitive)
    return appendHpackString(dst, f.Value)
}

Here, let's first look at the encodeTypeByte function:

func encodeTypeByte(indexing, sensitive bool) byte {
    if sensitive {
        return 0x10
    }
    if indexing {
        return 0x40
    }
    return 0
}
Copy code

As mentioned earlier, in this example, indexing is always true and sensitive is false, so the return value of encodeTypeByte is always 0x40.

Returning to the appendIndexedName function, we know that the format of the first byte of the dynamic table Header representation must be 0xb01xxxxx, that is, the highest two bits must be 01, and the lower six bits are used to represent the index of the Name in the Header.

After encoding the index through appendVarInt, let's take a look at how the appendHpackString function encodes the Value of the Header:

func appendHpackString(dst []byte, s string) []byte {
    huffmanLength := HuffmanEncodeLength(s)
    if huffmanLength < uint64(len(s)) {
        first := len(dst)
        dst = appendVarInt(dst, 7, huffmanLength)
        dst = AppendHuffmanString(dst, s)
        dst[first] |= 0x80
    } else {
        dst = appendVarInt(dst, 7, uint64(len(s)))
        dst = append(dst, s...)
    }
    return dst
}
Copy code

appendHpackString can be encoded in two cases:

When the length after Huffman coding is less than the length of the original Value, use appendVarInt to store the final length after Huffman coding in buf, and then store the real Huffman coding in buf.

When the length of the original Huffman code is greater than the length of the original Huffman code, then store the length of the Huffman code into the original Huffman code, and then store it into the original Huffman code.

It should be noted here that only the lower 7 bits of the byte are used to store the length of Value. The highest bit is 1, indicating that the stored content is Huffman encoded, and the highest bit is 0, indicating that the stored content is the original Value.

appendNewName: code Name and Value have no matching Header field.

func appendNewName(dst []byte, f HeaderField, indexing bool) []byte {
    dst = append(dst, encodeTypeByte(indexing, f.Sensitive))
    dst = appendHpackString(dst, f.Name)
    return appendHpackString(dst, f.Value)
}
Copy code

As mentioned earlier, the return value of encodeTypeByte is 0x40, so the first byte we encode at this time is 0b010000000.

After encoding the first byte, encode the Name and Value of the Header successively through appendHpackString.

HPACK decoding

The encoding process of HPACK has been dealt with before. Next, we will deal with the decoding process through a decoding example.

// The coding example in HPACK coding is omitted here
var (
  invalid    error
  sawRegular bool
  // 16 << 20 from fr.maxHeaderListSize() from
  remainSize uint32 = 16 << 20
)
hdec := hpack.NewDecoder(4096, nil)
// 16 << 20 from fr.maxHeaderStringLen() from fr.maxHeaderListSize()
hdec.SetMaxStringLength(int(remainSize))
hdec.SetEmitFunc(func(hf hpack.HeaderField) {
  if !httpguts.ValidHeaderFieldValue(hf.Value) {
    invalid = fmt.Errorf("invalid header field value %q", hf.Value)
  }
  isPseudo := strings.HasPrefix(hf.Name, ":")
  if isPseudo {
    if sawRegular {
      invalid = errors.New("pseudo header field after regular")
    }
  } else {
    sawRegular = true
    // if !http2validWireHeaderFieldName(hf.Name) {
    //  invliad = fmt.Sprintf("invalid header field name %q", hf.Name)
    // }
  }
  if invalid != nil {
    fmt.Println(invalid)
    hdec.SetEmitEnabled(false)
    return
  }
  size := hf.Size()
  if size > remainSize {
    hdec.SetEmitEnabled(false)
    // mh.Truncated = true
    return
  }
  remainSize -= size
  fmt.Printf("%+v\n", hf)
  // mh.Fields = append(mh.Fields, hf)
})
defer hdec.SetEmitFunc(func(hf hpack.HeaderField) {})
fmt.Println(hdec.Write(buf.Bytes()))
// The output is as follows:
// ori size: 197, encoded size: 111
// header field ":authority" = "dss0.bdstatic.com"
// header field ":method" = "GET"
// header field ":path" = "/5aV1bjqh_Q23odCf/static/superman/img/topnav/baiduyun@2x-e0be79e69e.png"
// header field ":scheme" = "https"
// header field "accept-encoding" = "gzip"
// header field "user-agent" = "Go-http-client/2.0"
// header field "custom-header" = "custom-value"
// 111 <nil>
Copy code

Through the output of the last line, we can know that 197 bytes of original Header data have been decoded from 111 bytes.

This decoding process will be from HDEC Write method begins to analyze and gradually uncover its mystery.

 func (d *Decoder) Write(p []byte) (n int, err error) {
   // Code omitted here
    if d.saveBuf.Len() == 0 {
        d.buf = p
    } else {
        d.saveBuf.Write(p)
        d.buf = d.saveBuf.Bytes()
        d.saveBuf.Reset()
    }

    for len(d.buf) > 0 {
        err = d.parseHeaderFieldRepr()
        if err == errNeedMore {
            // Code omitted here
            d.saveBuf.Write(d.buf)
            return len(p), nil
        }
        // Code omitted here
    }
    return len(p), err
}
Copy code

In the process of debug ging, the author found that the core logic of decoding is mainly in the d.parseHeaderFieldRepr method.

func (d *Decoder) parseHeaderFieldRepr() error {
    b := d.buf[0]
    switch {
    case b&128 != 0:
        return d.parseFieldIndexed()
    case b&192 == 64:
        return d.parseFieldLiteral(6, indexedTrue)
    // Code omitted here
    }
    return DecodingError{errors.New("invalid encoding")}
}
Copy code

There is only one case where the first byte and the upper 128 are not 0, that is, B is the data in 0b1xxxxxx format. Based on the previous coding logic, we can know that the decoding method corresponding to the index Header representation is d.parseFieldIndexed.

There is only one case where the first byte and the upper 192 are 64, that is, B is the data in 0b01xxxxx format. According to the previous coding logic, the decoding method corresponding to adding the dynamic table Header representation is d.parseFieldLiteral.

Index Header notation

By (* decoder) When parsefieldindexed is decoded, the real Header data is already in the static table or dynamic table. As long as the corresponding Header is found through the HPACK index, the decoding is successful.

func (d *Decoder) parseFieldIndexed() error {
    buf := d.buf
    idx, buf, err := readVarInt(7, buf)
    if err != nil {
        return err
    }
    hf, ok := d.at(idx)
    if !ok {
        return DecodingError{InvalidIndexError(idx)}
    }
    d.buf = buf
    return d.callEmit(HeaderField{Name: hf.Name, Value: hf.Value})
}
Copy code

The above method mainly has three steps:

  1. Read the HPACK index through the readVarInt function.
  2. Find the real Header data in the index list through the d.at method.
  3. Pass the Header to the top layer. d.CallEmit will eventually call HDEC Setemitfunc sets the closure to pass the Header to the top layer.

readVarInt: read HPACK index

func readVarInt(n byte, p []byte) (i uint64, remain []byte, err error) {
    if n < 1 || n > 8 {
        panic("bad n")
    }
    if len(p) == 0 {
        return 0, p, errNeedMore
    }
    i = uint64(p[0])
    if n < 8 {
        i &= (1 << uint64(n)) - 1
    }
    if i < (1<<uint64(n))-1 {
        return i, p[1:], nil
    }

    origP := p
    p = p[1:]
    var m uint64
    for len(p) > 0 {
        b := p[0]
        p = p[1:]
        i += uint64(b&127) << m
        if b&128 == 0 {
            return i, p, nil
        }
        m += 7
        if m >= 63 { // TODO: proper overflow check. making this up.
            return 0, origP, errVarintOverflow
        }
    }
    return 0, origP, errNeedMore
}
Copy code

From the above readVarInt function, when the low n of the first byte is not all 1, the low n represents the real HPACK index and can be returned directly.

When the low n of the first byte is all 1, more bytes need to be read to calculate the real HPACK index.

  1. In the first cycle, m is 0, the lower 7 bits of b plus (1 < < Uint64 (n)) - 1 and assigned to i

  2. In subsequent cycles, m is incremented by 7, and the low 7 bits of b will gradually fill the high bit of i.

  3. When b is less than 128, the end speed cycle, at which time the complete HPACK index has been read.

The readVarInt function logic corresponds to the appendVarInt function logic above.

(*Decoder).at: get the real Header data according to the index of HPACK.

func (d *Decoder) at(i uint64) (hf HeaderField, ok bool) {
    if i == 0 {
        return
    }
    if i <= uint64(staticTable.len()) {
        return staticTable.ents[i-1], true
    }
    if i > uint64(d.maxTableIndex()) {
        return
    }
    dt := d.dynTab.table
    return dt.ents[dt.len()-(int(i)-staticTable.len())], true
}
Copy code

When the index is less than the length of the static table, the Header data is obtained directly from the static table.

When the index length is longer than the static table, according to the HPACK index list described above, you can use DT len()-(int(i)-staticTable. Len ()) calculates the real subscript of I in the dynamic table ents, so as to obtain the Header data.

Add dynamic table Header representation

By (* decoder) Two situations need to be considered in parsefieldliteral decoding. 1, The Name of the header has an index. 2, Header's Name and Value have no index. In both cases, the header does not exist in the dynamic table.

The following step-by-step analysis (* decoder) Parsefieldliteral method.

1. Read the HPACK index in buf.

nameIdx, buf, err := readVarInt(n, buf)
Copy code

2. If the index is not 0, get the Name of the Header from the HPACK index list.

ihf, ok := d.at(nameIdx)
if !ok {
  return DecodingError{InvalidIndexError(nameIdx)}
}
hf.Name = ihf.Name
 Copy code

3. If the index is 0, the Name of the Header is read from buf.

hf.Name, buf, err = d.readString(buf, wantStr)
Copy code

4. Read the Value of Header from buf and add the complete Header to the dynamic table.

hf.Value, buf, err = d.readString(buf, wantStr)
if err != nil {
  return err
}
d.buf = buf
if it.indexed() {
  d.dynTab.add(hf)
}
Copy code

(*Decoder).readString: read the real Header data from the encoded byte data.

func (d *Decoder) readString(p []byte, wantStr bool) (s string, remain []byte, err error) {
    if len(p) == 0 {
        return "", p, errNeedMore
    }
    isHuff := p[0]&128 != 0
    strLen, p, err := readVarInt(7, p)
    // Omit check logic
    if !isHuff {
        if wantStr {
            s = string(p[:strLen])
        }
        return s, p[strLen:], nil
    }

    if wantStr {
        buf := bufPool.Get().(*bytes.Buffer)
        buf.Reset() // don't trust others
        defer bufPool.Put(buf)
        if err := huffmanDecode(buf, d.maxStrLen, p[:strLen]); err != nil {
            buf.Reset()
            return "", nil, err
        }
        s = buf.String()
        buf.Reset() // be nice to GC
    }
    return s, p[strLen:], nil
}
Copy code

First, judge whether the byte data is Huffman coded (corresponding to the appendHpackString function above), and then read the length of the data through readVarInt and assign it to strLen.

If it is not Huffman coding, the data of strLen length will be returned directly. If it is Huffman coding, read the data of strLen length, decode it with Huffman algorithm, and then return it.

Verification & summary

Previously, we have learned about the HPACK index list and the encoding / decoding process based on the HPACK index list.

Next, the author finally verifies the size of the codec Header.

// The previous demo of HAPACK encoding and HPACK decoding is omitted here
// try again
fmt.Println("try again: ")
buf.Reset()
henc.WriteField(hpack.HeaderField{Name: "custom-header", Value: "custom-value"}) // Encoded Header
fmt.Println(hdec.Write(buf.Bytes())) // decode
// Output:
// ori size: 197, encoded size: 111
// header field ":authority" = "dss0.bdstatic.com"
// header field ":method" = "GET"
// header field ":path" = "/5aV1bjqh_Q23odCf/static/superman/img/topnav/baiduyun@2x-e0be79e69e.png"
// header field ":scheme" = "https"
// header field "accept-encoding" = "gzip"
// header field "user-agent" = "Go-http-client/2.0"
// header field "custom-header" = "custom-value"
// 111 <nil>
// try again:
// header field "custom-header" = "custom-value"
// 1 <nil>
Copy code

It can be seen from the output of the last line above that only one byte is used for decoding, that is, in this example, only one byte is required to encode an encoded Header.

To sum up: on one connection, the client and server maintain the same HPACK index list. Multiple requests can be divided into two cases when sending and receiving Header data.

  1. In the HPACK index list, the Header can not transmit the real Header data, but only the HPACK index, so as to achieve the purpose of Header compression.
  2. The Header is not in the HPACK index list. For most headers, only the HPACK index of Header Value and Name needs to be transmitted, so as to reduce the transmission of Header data. At the same time, when sending and receiving such Header data, the respective HPACK index list will be updated to ensure that the Header data to be transmitted in the next request is as few as possible.

Finally, heartfelt thanks will be http2 Readers who have finished reading the 0 series sincerely hope that all readers can gain something.

If you have any questions, you can discuss them harmoniously in the comment area. I will reply in time when I see them. I hope you can make progress together.

Note:

  1. When writing this article, the author uses the go version: go1 fourteen point two
  2. The index Header representation and the dynamic table Header representation are independently named by the author, which is mainly convenient for readers to understand.

 

Tags: Python Java Go MySQL Big Data

Posted by brandone on Mon, 16 May 2022 09:17:40 +0300