⚡ Front end multi-threaded large file download practice, speed up 10 times, grasp Baidu cloud disk

background

Yes, you're right. It's the front-end multithreading, not the Node. This exploration originated from the recent development. When encountering the development requirements related to video streaming, a special status code was found, his name is 206~

In order to prevent the dryness of this article, first put on the effect drawing. (take a 3.7M size picture as an example).

Animation effect comparison (single thread - left VS 10 threads - right)

Time comparison (single thread VS 10 threads)

Is it a little exciting to see here? Then please continue to listen to me. Let's grab a bag first to see how the whole process happened.

`GET /360_0388.jpg HTTP/1.1
Host: limit.qiufeng.com
Connection: keep-alive
...
Range: bytes=0-102399

HTTP/1.1 206 Partial Content
Server: openresty/1.13.6.2
Date: Sat, 19 Sep 2020 06:31:11 GMT
Content-Type: image/jpeg
Content-Length: 102400
....
Content-Range: bytes 0-102399/3670627

...(Here is the file stream)` 

You can see that there is an additional field Range: bytes=0-102399 in the request, there is also an additional field content range: bytes 0-102399 / 3670627 in the server, and the returned status code is 206

So what is Range? I still remember writing an article about file download a few days ago, which mentioned the download method of large files. There is something called Range, but Last As a systematic overview of file download, range is not introduced in detail.

All of the following codes are in https://github.com/hua1995116/node-demo/tree/master/file-download/example/download-multiple

Basic introduction to Range

The origin of Range

Range is in HTTP/1.1 A new field is added in. This feature is also the core mechanism of Xunlei that supports multi-threaded download and breakpoint download. (introductory copy, excerpt)

First, the client will initiate a request with Range: bytes=0-xxx. If the server supports Range, accept ranges: bytes will be added to the response header to indicate that the request supports Range. Then the client may initiate a request with Range.

The server determines whether the Range processing is performed through the Range: bytes=0-xxx in the request header. If this value exists and is valid, only the requested part of the file content will be sent back. The status code of the response will change to 206, indicating Partial Content, and the content Range will be set. If it is invalid, a 416 status code is returned, indicating request Range not satisfactory. If there is no Range in the request header, the server will respond normally and will not set content Range, etc.

The format of Range is:

Range:(unit=first byte pos)-[last byte pos]

That is, Range: unit (e.g. bytes) = start byte position - end byte position.

Let's take an example. Suppose we enable multi-threaded downloading. We need to divide a 5000 byte file into four threads for downloading.

  • Range: bytes=0-1199 first 1200 bytes
  • Range: bytes=1200-2399 second 1200 bytes
  • Range: bytes=2400-3599 the third 1200 bytes
  • Range: bytes=3600-5000 last 1400 bytes

The server gives a response:

First response

  • Content-Length: 1200
  • Content-Range: bytes 0-1199/5000

2nd response

  • Content-Length: 1200
  • Content-Range: bytes 1200-2399/5000

3rd response

  • Content-Length: 1200
  • Content-Range: bytes 2400-3599/5000

4th response

  • Content-Length: 1400
  • Content-Range: bytes 3600-5000/5000

If each request is successful, the response header returned by the server has a content range field. The content range is used in the response header to tell the client how much data has been sent. It describes the coverage of the response and the length of the whole entity. General format:

Content range: bytes (unit first byte POS) - [last byte POS] / [entity length], that is, content range: byte start byte position - end byte position / file size.

Browser support

Mainstream browsers currently support this feature.

Server support

Nginx

After nginx version 1.9.8, (plus ngx_http_slice_module) is automatically supported by default, and Max can be set_ Set ranges to 0 to cancel this setting.

Node

Node does not provide the processing of Range method by default. You need to write your own code for processing.

router.get('/api/rangeFile', async(ctx) => {
    const { filename } = ctx.query;
    const { size } = fs.statSync(path.join(__dirname, './static/', filename));
    const range = ctx.headers['range'];
    if (!range) {
        ctx.set('Accept-Ranges', 'bytes');
        ctx.body = fs.readFileSync(path.join(__dirname, './static/', filename));
        return;
    }
    const { start, end } = getRange(range);
    if (start >= size || end >= size) {
        ctx.response.status = 416;
        ctx.body = '';
        return;
    }
    ctx.response.status = 206;
    ctx.set('Accept-Ranges', 'bytes');
    ctx.set('Content-Range', `bytes ${start}-${end ? end : size - 1}/${size}`);
    ctx.body = fs.createReadStream(path.join(__dirname, './static/', filename), { start, end });
}) 

Or you can use the koa send library.

https://github.com/pillarjs/send/blob/0.17.1/index.js#L680

Range practice

Architectural Overview

Let's first take a look at the overview of the process architecture diagram. Single thread is very simple. You can download it normally. For those who don't understand, please refer to me Last article. Multithreading will be more troublesome. You need to download by piece. After downloading, you need to merge and then download. (for download methods such as blob, please refer to Last)

Server code

Very simple, it is compatible with the Range.

router.get('/api/rangeFile', async(ctx) => {
    const { filename } = ctx.query;
    const { size } = fs.statSync(path.join(__dirname, './static/', filename));
    const range = ctx.headers['range'];
    if (!range) {
        ctx.set('Accept-Ranges', 'bytes');
        ctx.body = fs.readFileSync(path.join(__dirname, './static/', filename));
        return;
    }
    const { start, end } = getRange(range);
    if (start >= size || end >= size) {
        ctx.response.status = 416;
        ctx.body = '';
        return;
    }
    ctx.response.status = 206;
    ctx.set('Accept-Ranges', 'bytes');
    ctx.set('Content-Range', `bytes ${start}-${end ? end : size - 1}/${size}`);
    ctx.body = fs.createReadStream(path.join(__dirname, './static/', filename), { start, end });
}) 

html

Then write html. There's nothing to say. Write two buttons to show it.

<!-- html -->
<button id="download1">Serial download</button>
<button id="download2">Multithreaded Download</button>
<script src="https://cdn.bootcss.com/axios/0.19.2/axios.min.js"></script> 

js public parameters

const m = 1024 * 520;  // Slice size
const url = 'http://localhost:8888/api/rangeFile?filename=360_0388.jpg'; //  Address to download 

Single threaded part

Single thread download code, directly request to obtain it in the form of blob, and then download it in the form of blobURL.

download1.onclick = () => {
    console.time("Direct download");
    function download(url) {
        const req = new XMLHttpRequest();
        req.open("GET", url, true);
        req.responseType = "blob";
        req.onload = function (oEvent) {
            const content = req.response;
            const aTag = document.createElement('a');
            aTag.download = '360_0388.jpg';
            const blob = new Blob([content])
            const blobUrl = URL.createObjectURL(blob);
            aTag.href = blobUrl;
            aTag.click();
            URL.revokeObjectURL(blob);
            console.timeEnd("Direct download");
        };
        req.send();
    }
    download(url);
} 

Multithreaded part

First, send a head request to obtain the size of the file, and then calculate the sliding distance of each partition according to the length and the set partition size. Via promise In the callback of all, use the concatenate function to merge the fragment buffer into a blob, and then download it in the form of blobURL.

// script
function downloadRange(url, start, end, i) {
    return new Promise((resolve, reject) => {
        const req = new XMLHttpRequest();
        req.open("GET", url, true);
        req.setRequestHeader('range', `bytes=${start}-${end}`)
        req.responseType = "blob";
        req.onload = function (oEvent) {
            req.response.arrayBuffer().then(res => {
                resolve({
                    i,
                    buffer: res
                });
            })
        };
        req.send();
    })
}
// Merge buffer
function concatenate(resultConstructor, arrays) {
    let totalLength = 0;
    for (let arr of arrays) {
        totalLength += arr.length;
    }
    let result = new resultConstructor(totalLength);
    let offset = 0;
    for (let arr of arrays) {
        result.set(arr, offset);
        offset += arr.length;
    }
    return result;
}
download2.onclick = () => {
    axios({
        url,
        method: 'head',
    }).then((res) => {
        // Get the length to split the block
        console.time("Concurrent Download");
        const size = Number(res.headers['content-length']);
        const length = parseInt(size / m);
        const arr = []
        for (let i = 0; i < length; i++) {
            let start = i * m;
            let end = (i == length - 1) ?  size - 1  : (i + 1) * m - 1;
            arr.push(downloadRange(url, start, end, i))
        }
        Promise.all(arr).then(res => {
            const arrBufferList = res.sort(item => item.i - item.i).map(item => new Uint8Array(item.buffer));
            const allBuffer = concatenate(Uint8Array, arrBufferList);
            const blob = new Blob([allBuffer], {type: 'image/jpeg'});
            const blobUrl = URL.createObjectURL(blob);
            const aTag = document.createElement('a');
            aTag.download = '360_0388.jpg';
            aTag.href = blobUrl;
            aTag.click();
            URL.revokeObjectURL(blob);
            console.timeEnd("Concurrent Download");
        })
    })
} 

Complete example

https://github.com/hua1995116/node-demo

`// Enter directory
cd file-download
// start-up
node server.js
// open 
http://localhost:8888/example/download-multiple/index.html` 

Due to Google browser's restrictions on a single domain name in HTTP/1.1, the maximum concurrency of a single domain name is 6.5

This can be reflected in the discussion of source code and official personnel.

Discussion address

https://bugs.chromium.org/p/chromium/issues/detail?id=12066

Chromium source code

// https://source.chromium.org/chromium/chromium/src/+/refs/tags/87.0.4268.1:net/socket/client_socket_pool_manager.cc;l=47
// Default to allow up to 6 connections per host. Experiment and tuning may
// try other values (greater than 0).  Too large may cause many problems, such
// as home routers blocking the connections!?!?  See http://crbug.com/12066.
//
// WebSocket connections are long-lived, and should be treated differently
// than normal other connections. Use a limit of 255, so the limit for wss will
// be the same as the limit for ws. Also note that Firefox uses a limit of 200.
// See http://crbug.com/486800
int g_max_sockets_per_group[] = {
    6,   // NORMAL_SOCKET_POOL
    255  // WEBSOCKET_SOCKET_POOL
}; 

Therefore, in order to cooperate with this feature, I divide the file into six segments, each segment is 520kb (yes, write a code with a number that loves you), that is, open six threads to download.

I downloaded it six times with a single thread and multiple threads respectively. It seems that the speed is the same. So why is it different from what we expected?

Explore the causes of failure

I began to compare the two requests carefully and observe the speed of the two requests.

6 threads concurrent

Single thread

If we calculate according to the speed of 3.7m and 82ms, it will be about 46kb in 1ms, but the actual situation can be seen that 533kb will be downloaded for about 20ms on average (the connection time and pure content download time have been shaved).

I went to find some information and understood that there was something called downlink speed and uplink speed.

The actual transmission speed of the network is divided into uplink speed and downlink speed, Uplink rate Is the speed of sending data, and the downlink is the speed of receiving data. ADSL is a transmission mode realized according to our habit of surfing the Internet and sending out data, which is relatively small compared with downloading data. We said for 4M broadband , then our l theoretical maximum download speed is 512K/S, which is the so-called downlink speed-- Baidu Encyclopedia

What is our current situation?

Compare the server to a big water pipe. Let me use the diagram to simulate the downloading of a single thread and multiple threads. The server side is on the left and the client side is on the right. (all the following cases are considered in the ideal case, just for the purpose of simulating the process, without considering the race state influence of other programs.)

Single thread

Multithreading

Yes, because our server is a large water pipe, the flow rate is certain, and there is no limit on our client. If it is a single thread, it will run at the maximum speed of the user. If it is multithreading, taking three threads as an example, it is equivalent to each thread running one-third of the speed of the original thread. There is no difference between the speed of a single thread and that of a single thread.

Now I'll explain it in several cases. What kind of situation will our multithreading take effect?

The server bandwidth is larger than the user bandwidth, and there is no restriction

In fact, the situation we encounter is similar.

The server bandwidth is much larger than the user bandwidth, limiting the single connection speed

If the server limits the download speed of a single broadband, this is also the case for most of them. For example, baidu cloud is like this. For example, it is clear that you are 10M broadband, but the actual download speed is only 100kb/s. in this case, we can turn on multi threads to download, because it often limits the download of a single TCP. Of course, the online environment does not mean that users can turn on unlimited threads, but there will be restrictions, Will limit the maximum TCP of your current IP. In this case, the upper limit of download is often the maximum speed of your users. According to the above example, if you have reached the maximum speed by opening 10 threads, no matter how large, your entry has been restricted, then each thread will seize the speed, and it is useless to open more threads.

Improvement scheme

Because I haven't found a relatively simple way to control the download speed of Node for the time being, I introduced Nginx.

We control the speed of each TCP connection at 1M/s.

Add configuration limit_rate 1M;

preparation

1.nginx_conf

server {
    listen 80;
    server_name limit.qiufeng.com;
    access_log  /opt/logs/wwwlogs/limitqiufeng.access.log;
    error_log  /opt/logs/wwwlogs/limitqiufeng.error.log;

    add_header Cache-Control max-age=60;
    add_header Access-Control-Allow-Origin *;
    add_header Access-Control-Allow-Methods 'GET, OPTIONS';
    add_header Access-Control-Allow-Headers 'DNT,X-Mx-ReqToken,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Authorization,range,If-Range';
    if ($request_method = 'OPTIONS') {
        return 204;
    }
    limit_rate 1M;
    location / {
        root Your static directory;
        index index.html;
    }
} 

2. Configure local host

`127.0.0.1 limit.qiufeng.com` 

Check the effect. The speed is basically normal. Multi thread download is faster than single thread. The basic speed is 5-6:1, but it is found that if you click several times quickly during the download process, the download using Range will be faster and faster (it is suspected that Nginx has made some cache, so there is no in-depth research for the time being).

Modify the download address in the code
const url = 'http://localhost:8888/api/rangeFile?filename=360_0388.jpg';
become
const url = 'http://limit.qiufeng.com/360_0388.jpg'; 

Test download speed

Remember what I said above? With regard to HTTP/1.1, only 6 requests can be concurrent at the same site, and the redundant requests will be put into the next batch. However, HTTP/2.0 is not subject to this restriction, and multiplexing replaces http / 1.0 X sequence and blocking mechanism. Let's upgrade HTTP/2.0 to test it.

A certificate needs to be generated locally. (method of generating certificate: https://juejin.im/post/6844903556722475021)

server {
    listen 443 ssl http2;
    ssl on;
    ssl_certificate /usr/local/openresty/nginx/conf/ssl/server.crt;
    ssl_certificate_key /usr/local/openresty/nginx/conf/ssl/server.key;
    ssl_session_cache shared:le_nginx_SSL:1m;
    ssl_session_timeout 1440m;

    ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers RC4:HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers on;
    server_name limit.qiufeng.com;
 
    access_log  /opt/logs/wwwlogs/limitqiufeng2.access.log;
    error_log  /opt/logs/wwwlogs/limitqiufeng2.error.log;

    add_header Cache-Control max-age=60;
    add_header Access-Control-Allow-Origin *;
    add_header Access-Control-Allow-Methods 'GET, OPTIONS';
    add_header Access-Control-Allow-Headers 'DNT,X-Mx-ReqToken,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Authorization,range,If-Range';
    if ($request_method = 'OPTIONS') {
        return 204;
    }
    limit_rate 1M;
    location / {
        root The prefix path where you store the item/node-demo/file-download/;
        index index.html;
    }
} 

10 threads

`Modify the size of a single download
const m = 1024 * 400;` 

12 threads

24 threads

Of course, the more threads, the better. After testing, it is found that when the number of threads reaches a certain number, the speed will be slower. The following is a rendering of 36 concurrent requests.

Practical application exploration

What's the use of so many process downloads? Yes, it was also said at the beginning that this fragmentation mechanism is the core mechanism of Xunlei and other download software.

Netease cloud classroom

https://study.163.com/course/courseLearn.htm?courseId=1004500008#/learn/video?lessonId=1048954063&courseId=1004500008

When we open the console, we can easily find the download url and directly a streaking mp4 download address.

Input our test script from the console.

// The test script is too long, and if you read the above article carefully, you should be able to write code. I really can't write the following code.
https://github.com/hua1995116/node-demo/blob/master/file-download/example/download-multiple/script.js 

Direct download

Multithreaded Download

It can be seen that since Netease cloud classroom has no restrictions on the download speed of a single TCP, the improvement speed is not so obvious.

Baidu cloud

Let's test the web version of Baidu cloud.

Take a 16.6M file as an example.

Open the interface of Baidu cloud disk on the web and click download

Click pause at this time to open chrome - > more - > download content - > right click Copy download link

Still use the above Netease cloud course to download the script of the course. It's just that you need to change the parameters.

`url Change to the corresponding Baidu cloud download link
m Change to 1024 * 1024 * 2 Appropriate slice size~` 

Direct download

It took Baidu 217 seconds to connect to the cloud!!! For a 17M file, we have suffered a lot from it at ordinary times. (except VIP players)

Multithreaded Download

Because it is HTTP/1.1, we just need to start 6 or more threads to download. The following is the speed of multi-threaded download, which takes about 46 seconds.

Let's feel the speed difference through this picture.

It's really fragrant. It's free and only depends on our front-end to realize this function. It's too tm fragrant. Why don't you try it quickly??

Scheme defect

1. There are certain restrictions on the upper limit of large files

Due to the upper limit of blob size in major browsers, this method still has some defects.

2. The server limits the speed of a single TCP

Generally, there will be restrictions. At this time, it depends on the width and speed of the user.

ending

The article is written in a hurry, and the expression may not be particularly accurate. If there are mistakes, you are welcome to point out.

Looking back, do you have a web version of Baidu cloud acceleration plug-in? If not, create a web version of Baidu cloud download Plug-in ~.

Series articles

reference

Nginx bandwidth control: https://blog.huoding.com/2015/03/20/423

openresty deploys https and enables http2 support: https://www.gryen.com/articles/show/5.html

Let's talk about the Range of HTTP: https://dabing1022.github.io/2016/12/24/ Talk about the range and content range of HTTP/

last

If my article can help you, I hope you can help me too. Welcome to follow my wechat official account Qiufeng's notes and reply to friends twice. You can add wechat and join the exchange group. Qiufeng's notes will always be with you.

Tags: Javascript node.js Front-end Nginx webkit

Posted by bkanmani on Mon, 16 May 2022 04:30:21 +0300