Vue + WebRTC realize audio and video live broadcast (with custom player style)

1. What is WebRTC

1.1 introduction to webrtc

WebRTC, whose name comes from the abbreviation of Web real time communication (English: Web real time communication), is a real-time communication framework that supports real-time voice dialogue or video dialogue with web browsers, and provides a series of page callable API s.

Reference definition: Google open real-time communication framework

In the last blog Vue +WebSocket + WaveSurferJS realize H5 chat conversation interaction In, the use of WebRTC interface has been involved. The getUserMedia method is used to obtain the device microphone through the browser to collect audio.

The demand in the recent project is to establish instant communication with the server to realize low delay audio and video live broadcasting.

The characteristics of RTC are (reference source: https://www.zhihu.com/question/22301898)

  • High complexity
  • Semi reliable transmission enables lossy transmission of audio and video in specific situations (such as poor network environment) to reduce delay
  • Audio and video friendly: it can be customized and optimized for audio and video
  • Provide end-to-end optimization scheme. For the traditional connection mode, using the C/S architecture, a = > server = > b, while WebRTC uses the peer-to-peer mode, a = > B. once the connection between points is formed, the data transmission between them does not pass through the server, which greatly reduces the pressure on the server.
  • The theoretical delay is low and can be applied to various low delay scenarios.

2. Business description

Function Description:
Realize the management list of camera equipment. When clicking to view the video in the device list, the page floating window will pop up to broadcast the video and audio captured by the camera in real time.
There is a control bar under the video pop-up window to realize playback / pause control, which can display playback time, switch resolution, full screen, etc.

The effect is shown in the figure:

3. Code implementation

3.1 Html template code

<el-dialog ref="videoDialog" title="Video playback" :visible.sync="dialogShowed" :close-on-click-modal="false">
        <div id="dialog-wrap">
            <div id="video-wrap" v-if="isSuccess" v-loading="isLoading" element-loading-text="Video loading" element-loading-spinner="el-icon-loading"
                element-loading-background="rgba(0, 0, 0, 0.8)" />
            <div class="video-onloading" v-else v-loading="isLoading" element-loading-text="Video loading" element-loading-spinner="el-icon-loading"
                element-loading-background="rgba(0, 0, 0, 0.8)">
                <span><i class="el-icon-error" v-if="!isLoading" />{{errorMessage}}</span>
            </div>
            <!-- mask  -->
            <div class="cover" v-if="isSuccess">
                <div class="controls">
                  
                    <i class="el-icon-video-play" v-if="!isPlaying" @click="playOrPauseVideo" />
                    <i class="el-icon-video-pause" v-else @click="playOrPauseVideo" />
                    <div id="currentTime">Playback duration:{{currentTime}}</div>
                    <div class="control-resolution">
                        resolving power:
                        <el-select v-model="selectResolution" @change="changeResolution">
                            <el-option v-for="item in resolutions" :key="item" :value="item">
                                {{item}}
                            </el-option>
                        </el-select>
                    </div>
                    <i class="el-icon-full-screen" @click="onClickFullScreen"></i>
                </div>
            </div>
        </div>
    </el-dialog>

  • The v-loading instruction provided by the element UI framework is used, which determines whether to load the loading animation in the area according to the isLoading attribute

  • If the video loading fails, an error message is displayed

  • Reserved tags for mounting ` video and audio DOM elements
    <div id="video-wrap" ></div>
    Note that it is better not to add other elements in the label, so the subsequent judgment is relatively simple.

3.2 establish connection and receive audio

       getVideo() {
                let that = this;
                that.isLoading = true;
                that.pc = new RTCPeerConnection();
                that.pc.addTransceiver("video");
                that.pc.addTransceiver("audio");
                that.pc.ontrack = function (event) {
                    var el = document.createElement(event.track.kind);
                    el.srcObject = event.streams[0];
                    el.autoplay = true;
                    document.getElementById("video-wrap").appendChild(el);
                    if (el.nodeName === "VIDEO") {
                        el.oncanplay = () => {
                            that.isLoading = false;
                            // Set playback status to true
                            that.isPlaying = true;
                            that.getVideoDuration();
                        };
                    } else if (el.nodeName === "AUDIO") {
                        el.oncanplay = () => {
   
                        };
                    }
                };
                that.pc
                    .createOffer()
                    .then((offer) => {
                        that.pc.setLocalDescription(offer);
                        let req = {
                            webrtc: offer,
                        };
                        console.log(offer);
                        return that.$api.device.getSignaling(
                            that.deviceData.id,
                            that.origin,
                            that.selectResolution,
                            req
                        );
                    })
                    .then((res) => {
                        if (res.code === 0) {
                            that.isSuccess = true;
                            that.pc.setRemoteDescription(res.body.webrtc);
                            that.connId = res.body.connId;
                        } else {
                        
                            that.errorMessage = res.message || "Video loading error";
                        }
                    })
                    .catch(alert);
            }

reference resources https://www.jianshu.com/p/43957ee18f1a , view the process of establishing a connection with Peer Connection.
reference resources https://developer.mozilla.org/zh-CN/docs/Web/API/RTCPeerConnection View the interfaces supported by RTCPeerConnection

createOffer() method: actively establish P2P connection with other peers, sort out their own SDP information, and forward it to other peers through the signaling server.
In the above code, signaling exchange is realized by sending POST request to the back end.

 that.pc.addTransceiver("video");
 that.pc.addTransceiver("audio");

Indicates that audio and video are received at the same time.

 that.pc.ontrack = function(event){
}

This method receives audio and video, and uses the received data to create video and audio elements.
Only monitor the pc status, and you can't monitor the actual video playback status. Therefore, you need to add a monitoring method to the video:

  el.oncanplay = () => {
     that.isLoading = false;
     // Set playback status to true
    that.isPlaying = true;
    that.getVideoDuration();
};

Only when the video can be played can the loading status be cancelled and the video duration be obtained.

3.3 JS code for controlling audio and video

How to obtain video playback duration:

getVideoDuration() {
    var video = document.getElementsByTagName("video")[0];
    //  If no video element is obtained
    if (!video) {
        return;
    }
    let that = this;

    video.addEventListener("timeupdate", () => {
        that.currentTime = getTime(video.currentTime);
    });

    var getTime = function (time) {
        let hour =
            Math.floor(time / 3600) < 10
                ? "0" + Math.floor(time / 3600)
                : Math.floor(time / 3600);
        let min =
            Math.floor((time % 3600) / 60) < 10
                ? "0" + Math.floor((time % 3600) / 60)
                : Math.floor((time % 3600) / 60);
        var sec =
            Math.floor(time % 60) < 10
                ? "0" + Math.floor(time % 60)
                : Math.floor(time % 60);
        return hour + ":" + min + ":" + sec;
    };
}

Method of controlling audio / video synchronization pause:

  playOrPauseVideo() {
    var video = document.getElementsByTagName("video")[0];
    var audio = document.getElementsByTagName("audio")[0];
    if (this.isPlaying) {
        video.pause();
        audio.pause();
    } else {
        // audio
        video.play();
        audio.play();
    }
    this.isPlaying = !this.isPlaying;
}

Full screen method

onClickFullScreen() {
    let dialogElement = document.getElementById("dialog-wrap");
    dialogElement.webkitRequestFullScreen();
}

3.4 style sheet

The style part is relatively simple. It is worth noting the following points:

  • Hide the original video control bar to facilitate customization of the control bar
video::-webkit-media-controls {
    /* Remove the built-in control bar displayed in full screen */
    display: none !important;
}
  • Expand the hover hot area and suspend the display control bar in the lower half of the video (the part with a height of 400px)
    (it is not set to all parts because if it is set to all parts, the control bar cannot be hidden in the full screen state)
    The following complete style sheet (scss):
    $controlFontColor: rgb(136 141 150);
    $backgroundColor: rgba(0, 0, 0, 0.8);
    $height: 60px;

    .el-dialog .el-dialog__body {
        padding: 0 !important;
        margin-bottom: 0 !important;
        width: unset !important;
    }

    .video-onloading {
        min-height: 500px;
        background-color: $backgroundColor;

        span {
            width: 100%;
            display: block;

            line-height: 500px;
            text-align: center;
            color: $controlFontColor;
            i {
                margin-right: 5px;
            }

            i::before {
                font-size: 17px;
            }
        }
    }

  .cover {
        bottom: 0px;
        height: 300px;
        position: absolute;
        width: 100%;
        z-index: 2;
        &:hover,
        &:focus,
        &:focus-within {
            .controls {
                display: flex;
            }
        }
    }
  .controls {
        width: 100%;
        height: $height;
        line-height: $height;
        font-size: 15px;
        display: none;
        z-index: 2;
        background-color: $backgroundColor;
        color: $controlFontColor;
        position: absolute;
        bottom: 0
        justify-content: space-between;

        & > [class^="el-icon-"] {
            &::before {
                font-size: 26px;
                line-height: $height;
                padding: 0 15px;
                cursor: pointer;
            }
        }

        .playStatus {
            width: 64px;
            height: $height;
            position: absolute;
            top: 50%;
            left: 50%;
            transform: translate(-50%, -50%);
        }
        #currentTime {
            width: 140px;
            height: $height;
            text-align: center;
        }

        .control-resolution {
            line-height: $height;
            .el-input__inner {
                background: $backgroundColor;
            }
            .el-input {
                width: 95px;
            }
            input {
                border: none;
                font-size: 15px !important;
                color: $controlFontColor;
                &::-webkit-input-placeholder {
                    color: $controlFontColor;
                }
            }
        }
        #fullScreen {
            width: 32px;
            height: 32px;
            position: relative;
            top: 16px;
         
        }
    }

summary

This front-end business WebRTC has only made a simple understanding and application, only applied the receiving stream, and has not used the push stream. WebRTC has more uses, such as realizing real-time video call, voice call, etc., which may be used in future business, so make an introductory record with this blog~

Tags: Front-end

Posted by rosieraz on Sat, 07 May 2022 15:57:51 +0300