Implementation principle of APNG playback on Web side

Source of drawings: https://commons.wikimedia.org

Author: Yang Caifang

Write in front

In the live broadcast development of cloud music, we often encounter the demand of animation playback. The application scenarios of each demand are different, and the smaller animation mostly adopts APNG format.

If the animation is only displayed separately, you can use < img > to directly display APNG animation, but there will be compatibility bugs. For example, some browsers do not support APNG playback, and repeated playback of some Android models will fail.

If you need to combine APNG animation and other DOM elements with CSS3 Animation to display animation, APNG needs to be preloaded and controlled. Preloading can prevent APNG parsing from taking time, resulting in the problem of non synchronization between the two. Controlled can help users to carry out some operations at time nodes such as successful APNG parsing or playback conclusion.

These problems apng-canvas Can help us solve it. APNG canvas uses canvas to draw APNG animation, which can be compatible with more browsers, smooth the differences between different browsers, and facilitate the control of APNG playback. The following will introduce the implementation principle of APNG and APNG canvas library and the implementation method of WebGL rendering added on the basis of APNG canvas.

APNG introduction

APNG (Animated Portable Network Graphics, Animated PNG) is a bitmap animation format based on PNG format extension, which increases the support for animation images, and adds the support of 24 bit true color images and 8-bit Alpha transparency. The animation has better quality. APNG is backward compatible with traditional PNG. When the decoder does not support APNG playback, the default image will be displayed.

In addition to APNG, GIF and WebP are common animation formats. From three aspects of browser compatibility, size and image quality, the results are as follows (the size of one image is taken as an example, and the size of other solid or colorful images can be viewed GIF vs APNG vs WebP , in most cases, APNG is smaller). In general, APNG is better, which is why we choose APNG.

APNG structure

APNG is extended based on PNG format. Let's first understand the composition and structure of PNG.

PNG structure composition

Png mainly includes PNG Signature, IHDR, IDAT, IEND and some auxiliary blocks. PNG Signature is the document identifier, which is used to verify whether the document format is PNG; IHDR is a file header data block, which contains the basic information of the image, such as the width and height information of the image; IDAT is an image data block that stores specific image data. A PNG file may have one or more IDAT blocks; IEND is the end data block, indicating the end of the image; The auxiliary block is located after IHDR and before IEND, and the PNG specification does not impose sorting restrictions on it.

The size of PNG Signature block is 8 bytes, as follows:

0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a

The composition structure of each other block is basically as follows:

4 bytes identify the length of data, 4 bytes identify the block type, length bytes are data (if the length of data is 0, there is no such part), and the last 4 bytes are CRC verification.

APNG structure composition

APNG adds three blocks: acTL, fcTL and fdat on the basis of PNG, and their composition structure is shown in the figure below:

  • acTL: animation control block, including the number of frames and cycles of the picture (0 indicates infinite cycles)
  • fcTL: frame control block, which belongs to the auxiliary block in PNG specification, including the sequence number of the current frame, the width and height of the image, the horizontal and vertical offset, the playing time of the frame and the drawing method (dispose_op and blend_op). There is only one fcTL block in each frame
  • fdAT: frame data block, which contains the sequence number and image data of the frame. Only the sequence number of the frame is more than that of IDAT. Each frame can have one or more fcTL blocks. The serial number of fdAT is shared with fcTL to detect the sequence error of APNG and can be selectively corrected

The IDAT block is the default picture for APNG downward compatibility display. If there is fcTL before IDAT, the data of IDAT is regarded as the first frame picture (as shown in the structure above). If there is no fcTL before IDAT, the first frame picture is the first fdAT, as shown in the figure below:

APNG animation playback is mainly controlled by fcTL to render the image of each frame, that is, through dispose_op and blend_op controls how you draw.

  • dispose_op specifies the operation of the buffer before drawing the next frame

    • 0: without emptying the canvas, directly render the new image data to the specified area of the canvas
    • 1: Empty the canvas in the area of the current frame to the default background color before rendering the next frame
    • 2: Restores the current frame area of the canvas to the result of the previous frame before rendering the next frame
  • blend_op specifies the operation on the buffer before drawing the current frame

    • 0: clear the current area and draw again
    • 1: Indicates that the current area is drawn directly without clearing, and the image is superimposed

Implementation principle of apng canvas

After understanding the composition and structure of APNG, we can analyze the implementation principle of APNG canvas, which is mainly divided into two parts: decoding and rendering.

APNG decoding

The process of APNG decoding is shown in the following figure:

First, download the APNG resources in the format of arraybuffer, and operate the binary data through the view; Then check whether the file format is PNG and APNG in turn; Then, each APNG block is split in turn for processing and storage; Finally, the split PNG marking block, header block, other auxiliary blocks, frame image data block and end block of one frame are reconstituted into PNG pictures, and the image resources are loaded. In this process, the browser needs to support Typed Arrays and Blob URLs.

APNG file resources are downloaded through XMLHttpRequest, which is simple to implement and will not be repeated here.

Verify PNG format

Verifying the PNG format is to verify the PNG Signature block. Compare the file resources from the first byte to the contents of the first eight bytes. The key implementation is as follows:

const bufferBytes = new Uint8Array(buffer); // buffer is the downloaded arraybuffer resource
const PNG_SIGNATURE_BYTES = new Uint8Array([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]);
for (let i = 0; i < PNG_SIGNATURE_BYTES.length; i++) {
    if (PNG_SIGNATURE_BYTES[i] !== bufferBytes[i]) {
        reject('Not a PNG file (invalid file signature)');
        return;
    }
}

Verify APNG format

Verifying the APNG format is to determine whether there are blocks of type acTL in the file. Therefore, it is necessary to read each block in the file in order to obtain data such as block type. The reading of the block is processed according to the basic composition structure of the PNG block described above. The process implementation is shown in the following figure:

The initial value of off is 8, that is, the byte size of PNG Signature, and then read each block in order. First read 4 bytes to get the length of the data block, continue to read 4 bytes to get the data block type, and then execute the callback function to process the data of this block. Judge whether to continue reading the next block according to the return value res, block type and off value of the callback function (res value indicates whether to continue reading the next block of data, which is undefined by default). If you continue, the off value is accumulated by 4 + 4 + length + 4, which is offset to the beginning of the next block, and the cycle is executed. Otherwise, it ends directly. The key codes are as follows:

const parseChunks = (bytes, callback) => {
    let off = 8;
    let res, length, type;
    do {
        length = readDWord(bytes, off);
        type = readString(bytes, off + 4, 4);
        res = callback(type, bytes, off, length);
        off += 12 + length;
    } while (res !== false && type !== 'IEND' && off < bytes.length);
};

Call parseChunks to search from the beginning. Once there is a block with type === 'acTL', return false and stop reading. The key implementation is as follows:

let isAnimated = false;
parseChunks(bufferBytes, (type) => {
    if (type === 'acTL') {
        isAnimated = true;
        return false;
    }
    return true;
});
if (!isAnimated) {
    reject('Not an animated PNG');
    return;
}

Process each piece by type

The detailed structure of the core type block in APNG structure is shown in the following figure:

Call parseChunks to read each block in turn, and process and store the data contained in each type of block and its corresponding offset and byte size respectively. When processing fcTL and fdAT blocks, the reading of frame sequence number is skipped. It seems that the problem of sequence number error is not considered. The key realization is as follows:

let preDataParts = [], // Store other auxiliary blocks
    postDataParts = [], // Store IEND block
    headerDataBytes = null; // Store IHDR blocks

const anim = anim = new Animation();
let frame = null; // Store each frame

parseChunks(bufferBytes, (type, bytes, off, length) => {
    let delayN,
        delayD;
    switch (type) {
        case 'IHDR':
            headerDataBytes = bytes.subarray(off + 8, off + 8 + length);
            anim.width = readDWord(bytes, off + 8);
            anim.height = readDWord(bytes, off + 12);
            break;
        case 'acTL':
            anim.numPlays = readDWord(bytes, off + 8 + 4); // Number of cycles
            break;
        case 'fcTL':
            if (frame) anim.frames.push(frame); // Previous frame data
            frame = {}; // New frame
            frame.width = readDWord(bytes, off + 8 + 4);
            frame.height = readDWord(bytes, off + 8 + 8);
            frame.left = readDWord(bytes, off + 8 + 12);
            frame.top = readDWord(bytes, off + 8 + 16);
            delayN = readWord(bytes, off + 8 + 20);
            delayD = readWord(bytes, off + 8 + 22);
            if (delayD === 0) delayD = 100;
            frame.delay = 1000 * delayN / delayD;
            anim.playTime += frame.delay; // Total cumulative playback time
            frame.disposeOp = readByte(bytes, off + 8 + 24);
            frame.blendOp = readByte(bytes, off + 8 + 25);
            frame.dataParts = [];
            break;
        case 'fdAT':
            if (frame) frame.dataParts.push(bytes.subarray(off + 8 + 4, off + 8 + length));
            break;
        case 'IDAT':
            if (frame) frame.dataParts.push(bytes.subarray(off + 8, off + 8 + length));
            break;
        case 'IEND':
            postDataParts.push(subBuffer(bytes, off, 12 + length));
            break;
        default:
            preDataParts.push(subBuffer(bytes, off, 12 + length));
    }
});
if (frame) anim.frames.push(frame); // Each frame of data is stored in sequence

Assemble PNG

After splitting the data block, you can assemble PNG and traverse anim Frames PNG general data block PNG_SIGNATURE_BYTES, headerDataBytes, preDataParts, frame data dataParts of one frame and postDataParts form a PNG image resource (bb) in order. The URL of the picture created by createObjectURL is stored in the frame for subsequent drawing.

const url = URL.createObjectURL(new Blob(bb, { type: 'image/png' }));
frame.img = document.createElement('img');
frame.img.src = url;
frame.img.onload = function () {
    URL.revokeObjectURL(this.src);
    createdImages++;
    if (createdImages === anim.frames.length) { //Complete decoding
        resolve(anim);
    }
};

Here we have finished the decoding work and call apng Parseurl can realize the pre loading function of animation resources: call the loading resources for the first time after page initialization, and call again when rendering to directly return the analysis results for drawing operation.

const url2promise = {};
APNG.parseURL = function (url) {
    if (!(url in url2promise)) {
        url2promise[url] = loadUrl(url).then(parseBuffer);
    }
    return url2promise[url];
};

APNG drawing

After APNG decoding, you can draw and play according to the animation control block and frame control block. Specifically, use requestAnimationFrame to draw each frame of pictures on the canvas in turn to play. APNG canvas adopts Canvas 2D rendering.

const tick = function (now) {
    while (played && nextRenderTime <= now) renderFrame(now);
    if (played) requestAnimationFrame(tick);
};

Canvas 2D drawing is mainly realized by using canvas 2D API drawImage, clearRect, getImageData and putImageData.

const renderFrame = function (now) {
    // fNum records the total number of frames during playback cycle
    const f = fNum++ % ani.frames.length;
    const frame = ani.frames[f];
    // End of animation playback
    if (!(ani.numPlays === 0 || fNum / ani.frames.length <= ani.numPlays)) {
        played = false;
        finished = true;
        if (ani.onFinish) ani.onFinish(); // This line is added by the author to facilitate some operations after the animation is played
        return;
    }

    if (f === 0) {
        // Empty the canvas of the whole animation area before painting the first frame
        ctx.clearRect(0, 0, ani.width, ani.height);  
        prevF = null; // Previous frame
        if (frame.disposeOp === 2) frame.disposeOp = 1;
    }

    if (prevF && prevF.disposeOp === 1) { // Empty the basemap of the previous frame area
        ctx.clearRect(prevF.left, prevF.top, prevF.width, prevF.height);
    } else if (prevF && prevF.disposeOp === 2) { // Restore the basemap before drawing the previous frame
        ctx.putImageData(prevF.iData, prevF.left, prevF.top);
    } // 0 is drawn directly

    const {
        left, top, width, height,
        img, disposeOp, blendOp
    } = frame;
    prevF = frame;
    prevF.iData = null;
    if (disposeOp === 2) { // Store the current drawing base map for recovering the data before drawing the next frame
        prevF.iData = ctx.getImageData(left, top, width, height);
    }
    if (blendOp === 0) { // Clear the basemap of the current frame area
        ctx.clearRect(left, top, width, height);
    }

    ctx.drawImage(img, left, top); // Draw current frame picture

    // Draw time of the next frame
    if (nextRenderTime === 0) nextRenderTime = now;
    nextRenderTime += frame.delay; // delay is the frame interval
};

WebGL drawing

In addition to Canvas 2D, WebGL can also be used for rendering. The rendering performance of WebGL is better than Canvas 2D, but WebGL does not have an API that can directly draw images, and the drawing implementation code is relatively complex. This paper will not show the specific code of drawing images, and the WebGL implementation similar to drawImage API can be used for reference WebGL-drawimageTwo dimensional matrix Wait. The following will introduce the key points of the drawing implementation scheme selected by the author.

Since WebGL does not have API s such as getImageData and putImageData to obtain or copy the image data of the current canvas, multiple textures are initialized when WebGL is initialized, and the texture data of historical rendering is recorded by using the variable glRenderInfo.

// Number of textures
const textureLens = ani.frames.filter(item => item.disposeOp === 0).length;

// Historical rendered texture data
const glRenderInfo = {
    index: 0,
    frames: {},
};

Each frame is rendered according to glRenderInfo Frames uses multiple textures to render in turn, while updating glRenderInfo data.

const renderFrame = function (now) {
    ...
    let prevClearInfo;
    if (f === 0) {
        glRenderInfo.index = 0;
        glRenderInfo.frames = {};
        prevF = null;
        prevClearInfo = null;
        if (frame.disposeOp === 2) frame.disposeOp = 1;
    }
    if (prevF && prevF.disposeOp === 1) { //  You need to clear the area basemap of the previous frame
        const prevPrevClear = glRenderInfo.infos[glRenderInfo.index].prevF;
        prevClearInfo = [
            ...(prevPrevClear || []),
            prevF,
        ];
    }
    if (prevF && prevF.disposeOp === 0) { // Increment the texture subscript sequence number, otherwise directly replace the previous frame picture
        glRenderInfo.index += 1;
    }
    // disposeOp === 2 directly replace the previous picture
    glRenderInfo.frames[glRenderInfo.index] = { // Update glRenderInfo
        frame,
        prevF: prevClearInfo, // Used to clear the previous frame area basemap
    };
    prevF = frame;
    prevClearInfo = null;
    // Draw a picture and clear the basemap, which is implemented inside the glDrawImage interface
    Object.entries(glRenderInfo.frames).forEach(([key, val]) => {
        glDrawImage(gl, val.frame, key, val.prevF);
    });
    ...
}

Summary

This paper introduces the structure and composition of APNG, picture decoding and rendering with Canvas 2D / WebGL. I hope reading this article can help you. Welcome to explore.

reference resources

This article is published from Netease cloud music front end team , the article is prohibited from being reproduced in any form without authorization. We recruit front-end, iOS and Android all year round. If you are ready to change jobs and happen to like cloud music, join us GRP music-fe(at)corp.netease. com!

Tags: Web Development animation gif

Posted by Mathy on Tue, 24 May 2022 14:15:43 +0300