Dynamic watermark design of audio and video RTMP push and GB28181 docking on Android platform

Technical background

With the development of mobile individual soldier, smart vehicle, smart security, smart home, industrial simulation, GB28281 technology docking and other industries, the scene is no longer limited to the collection of video data, coding, packaging and sending or docking to the streaming media server. Most scenes have higher and higher requirements for video watermark, and the demand for video watermark is slowly transitioning from the previous fixed position static text watermark and png watermark to dynamic watermark.

Taking the camera data collected by Android platform as an example, this paper adds different layers in the form similar to PhotoShop layer to realize the effect of dynamic watermark.

Needless to say, the last rendering shows real-time time watermark, text watermark, png watermark and text watermark respectively after Android acquisition terminal obtains camera data. All watermarks support dynamic setting, which can meet the watermark setting requirements of traditional industries such as real-time timestamp superposition, dynamic longitude and latitude setting, png logo and other scenes.

Technical realization

Camera data acquisition, which will not be repeated. After obtaining the data of the front and rear cameras (see onPreviewFrame() processing for details), post the data to jni layer through PostLayerImageNV21ByteArray().

int w = videoWidth, h = videoHeight;
  int y_stride = videoWidth, uv_stride = videoWidth;
  int y_offset = 0, uv_offset = videoWidth * videoHeight;
  int is_vertical_flip = 0, is_horizontal_flip = 0;
  int rotation_degree = 0;

  // The mirror image is only used in the front camera scene
  if (is_mirror && FRONT == currentCameraType) {
      // Vertical screen, (vertical flip - > clockwise rotation 270 degrees) is equivalent to (clockwise rotation 270 degrees - > horizontal flip)
      if (PORTRAIT == currentOrigentation)
          is_vertical_flip = 1;
      else
          is_horizontal_flip = 1;
  }

  if (PORTRAIT == currentOrigentation) {
      if (BACK == currentCameraType)
          rotation_degree = 90;
      else
          rotation_degree = 270;
  } else if (LANDSCAPE_LEFT_HOME_KEY == currentOrigentation) {
      rotation_degree = 180;
  }

  int scale_w = 0, scale_h = 0, scale_filter_mode = 0;

  // Scaling test++
  /*
  if (w >= 1280 && h >= 720) {
      scale_w = align((int)(w * 0.8 + 0.5), 2);
      scale_h = align((int)(h * 0.8 + 0.5), 2);
  } else {
      scale_w = align((int)(w * 1.5 + 0.5), 2);
      scale_h = align((int)(h * 1.5 + 0.5), 2);
  }

  if(scale_w >0 && scale_h >0) {
      scale_filter_mode = 3;
   Log.i(TAG, "onPreviewFrame w:" + w + ", h:" + h + " s_w:" + scale_w + ", s_h:" + scale_h);
  }
  */
  // Scaling test---

  libPublisher.PostLayerImageNV21ByteArray(publisherHandle, 0, 0, 0,
          data, y_offset, y_stride, data, uv_offset, uv_stride, w, h,
          is_vertical_flip, is_horizontal_flip, scale_w, scale_h, scale_filter_mode, rotation_degree);

You may be curious about the design of PostLayerImageNV21ByteBuffer() and PostLayerImageNV21ByteArray(). The interface parameters are very powerful. Like our previous interface for camera2, it is almost a universal interface. The raw data obtained can not only be flipped horizontally and vertically, but also be scaled.

/**
   * Delivery layer NV21 image
   *
   * @param index: Layer index, must be greater than or equal to 0
   *
   * @param left: The coordinates of the upper left corner of the layer superposition are passed to 0 for layer 0
   *
   * @param top: The coordinates of the upper left corner of the layer superposition are passed to 0 for layer 0
   *
   * @param y_plane: y Plane image data
   *
   * @param y_offset: Image offset, which is mainly used for clip, is generally transmitted to 0
   *
   * @param y_row_stride: stride information
   *
   * @param uv_plane: uv Plane image data
   *
   * @param uv_offset: Image offset, which is mainly used for clip, is generally transmitted to 0
   *
   * @param uv_row_stride: stride information
   *
   * @param width: width, Must be greater than 1 and must be an even number
   *
   * @param height: height, Must be greater than 1 and must be an even number
   *
   * @param  is_vertical_flip: Whether to flip vertically, 0 does not flip, 1 flips
   *
   * @param  is_horizontal_flip: Whether to flip horizontally, 0 does not flip, 1 flips
   *
   * @param  scale_width: Scale width, must be even, 0 or negative number does not scale
   *
   * @param  scale_height: Scale high, must be even, 0 or negative number does not scale
   *
   * @param  scale_filter_mode: Scaling quality: the default speed is used when transmitting 0. The optional level range is: [1,3]. The larger the value, the better the scaling quality, but the slower the speed
   *
   * @param  rotation_degree: Clockwise rotation must be 0, 90, 180, 270. Note: the rotation is done after scaling, vertical / water product reversal. Please pay attention to the order
   *
   * @return {0} if successful
   */
  public native int PostLayerImageNV21ByteBuffer(long handle, int index, int left, int top,
                             ByteBuffer y_plane, int y_offset, int y_row_stride,
                               ByteBuffer uv_plane, int uv_offset, int uv_row_stride,
                               int width, int height, int is_vertical_flip,  int is_horizontal_flip,
                             int scale_width,  int scale_height, int scale_filter_mode,
                             int rotation_degree);


  /**
   * Please refer to PostLayerImageNV21ByteBuffer for details of posting layer NV21 image
   *
   * @return {0} if successful
   */
  public native int PostLayerImageNV21ByteArray(long handle, int index, int left, int top,
                           byte[] y_plane, int y_offset, int y_row_stride,
                           byte[] uv_plane, int uv_offset, int uv_row_stride,
                           int width, int height, int is_vertical_flip,  int is_horizontal_flip,
                           int scale_width,  int scale_height, int scale_filter_mode,
                           int rotation_degree);

Dynamic time watermarking

Dynamic time watermark is actually an extension of text watermark. It generates TextBitmap and copies it from bitmap to text_timestamp_buffer_, Post to jni layer through PostLayerImageRGBA8888ByteBuffer().

private int postTimestampLayer(int index, int left, int top) {

    Bitmap text_bitmap = makeTextBitmap(makeTimestampString(), getFontSize(),
            Color.argb(255, 0, 0, 0), true, Color.argb(255, 255, 255, 255),true);

    if (null == text_bitmap)
        return 0;

    if ( text_timestamp_buffer_ != null) {
        text_timestamp_buffer_.rewind();

        if ( text_timestamp_buffer_.remaining() < text_bitmap.getByteCount())
            text_timestamp_buffer_ = null;
    }

    if (null == text_timestamp_buffer_ )
        text_timestamp_buffer_ = ByteBuffer.allocateDirect(text_bitmap.getByteCount());

    text_bitmap.copyPixelsToBuffer(text_timestamp_buffer_);

    int scale_w = 0, scale_h = 0, scale_filter_mode = 0;
    //scale_w = align((int)(bitmapWidth*1.5 + 0.5), 2);
    //scale_h = align((int)(bitmapHeight*1.5 + 0.5),2);
    //scale_filter_mode = 3;

    /*
    if ( scale_w > 0 && scale_h > 0)
        Log.i(TAG, "postTextLayer scale_w:" + scale_w + ", scale_h:" + scale_h + " w:" + bitmapWidth + ", h:" + bitmapHeight) ;
     */

    libPublisher.PostLayerImageRGBA8888ByteBuffer(handle_, index, left, top, text_timestamp_buffer_, 0,
            text_bitmap.getRowBytes(), text_bitmap.getWidth(), text_bitmap.getHeight(),
            0, 0, scale_w, scale_h, scale_filter_mode,0);

    int ret = scale_h > 0? scale_h : text_bitmap.getHeight();

    text_bitmap.recycle();

    return ret;
}

Text watermark

The text watermark will not be repeated. The main attention is the size, color and position of the text.

private int postText1Layer(int index, int left, int top) {
    Bitmap text_bitmap = makeTextBitmap("Text watermark I", getFontSize()+8,
            Color.argb(255, 200, 250, 0),
            false, 0,false);

    if (null == text_bitmap)
        return 0;

    ByteBuffer buffer = ByteBuffer.allocateDirect(text_bitmap.getByteCount());
    text_bitmap.copyPixelsToBuffer(buffer);

    libPublisher.PostLayerImageRGBA8888ByteBuffer(handle_, index, left, top, buffer, 0,
            text_bitmap.getRowBytes(), text_bitmap.getWidth(), text_bitmap.getHeight(),
            0, 0, 0, 0, 0,0);

    int ret = text_bitmap.getHeight();

    text_bitmap.recycle();

    return ret;
}

png watermark

In addition to the conventional location, png watermark also involves the size of logo watermark. Therefore, we add a scaling effect, which can be scaled and pasted to the layer to ensure that it is displayed at the desired location of the layer in a more appropriate proportion.

private int postPictureLayer(int index, int left, int top) {
    Bitmap bitmap = getAssetsBitmap();
    if (null == bitmap) {
        Log.e(TAG, "postPitcureLayer getAssetsBitmap is null");
        return 0;
    }

    if (bitmap.getConfig() != Bitmap.Config.ARGB_8888) {
        Log.e(TAG, "postPitcureLayer config is not ARGB_8888, config:" + Bitmap.Config.ARGB_8888);
        return 0;
    }

    ByteBuffer buffer = ByteBuffer.allocateDirect(bitmap.getByteCount());
    bitmap.copyPixelsToBuffer(buffer);

    final int w = bitmap.getWidth();
    final int h = bitmap.getHeight();
    if ( w < 2 || h < 2 )
        return 0;

    int scale_w = 0, scale_h = 0, scale_filter_mode = 0;

    final float r_w = width_ - left; // Possible negative number
    final float r_h = height_ - top; // Possible negative number

    if (w > r_w || h > r_h) {
        float s_w = w;
        float s_h = h;

        // The 10th power of 0.85 is 0.19687, which is almost scaled to 0.2 times
        for ( int i = 0; i < 10; ++i)  {
            s_w *= 0.85f;
            s_h *= 0.85f;

            if (s_w < r_w && s_h < r_h )
                break;
        }

        if (s_w > r_w || s_h > r_h)
            return 0;

        // If it's less than 16, it's too small to see
        if (s_w < 16.0f || s_h < 16.0f)
            return  0;

        scale_w = align((int)(s_w + 0.5f), 2);
        scale_h = align( (int)(s_h + 0.5f), 2);
        scale_filter_mode = 3;
    }

    /*
    if ( scale_w > 0 && scale_h > 0)
        Log.i(TAG, "postTextLayer scale_w:" + scale_w + ", scale_h:" + scale_h + " w:" + w + ", h:" + h) ; */

    libPublisher.PostLayerImageRGBA8888ByteBuffer(handle_, index, left, top, buffer, 0, bitmap.getRowBytes(), w, h,
            0, 0, scale_w, scale_h, scale_filter_mode,0);

    int ret = scale_h > 0 ? scale_h : bitmap.getHeight();

    bitmap.recycle();

    return ret;
}

The final delivery interface of the above watermarks is designed as follows. The interface will not be repeated. Almost the image processing you expect has been covered:

/**
   * RGBA8888 image delivery layer, if you do not need Aplpha channel, please use RGBX8888 interface, with high efficiency
   *
   * @param index: The layer index must be greater than or equal to 0. Note: if the index is 0, the Alpha channel will be ignored
   *
   * @param left: The coordinates of the upper left corner of the layer superposition are passed to 0 for layer 0
   *
   * @param top: The coordinates of the upper left corner of the layer superposition are passed to 0 for layer 0
   *
   * @param rgba_plane: rgba image data
   *
   * @param offset: Image offset, which is mainly used for clip, is generally transmitted to 0
   *
   * @param row_stride: stride information
   *
   * @param width: width, Must be greater than 1. If it is an odd number, it will be reduced by 1
   *
   * @param height: height, Must be greater than 1. If it is an odd number, it will be reduced by 1
   *
   * @param  is_vertical_flip: Whether to flip vertically, 0 does not flip, 1 flips
   *
   * @param  is_horizontal_flip: Whether to flip horizontally, 0 does not flip, 1 flips
   *
   * @param  scale_width: Scale width, must be even, 0 or negative number does not scale
   *
   * @param  scale_height: Scale high, must be even, 0 or negative number does not scale
   *
   * @param  scale_filter_mode: Scaling quality: the default speed is used when transmitting 0. The optional level range is: [1,3]. The larger the value, the better the scaling quality, but the slower the speed
   *
   * @param  rotation_degree: Clockwise rotation must be 0, 90, 180, 270. Note: the rotation is done after scaling, vertical / water product reversal. Please pay attention to the order
   *
   * @return {0} if successful
   */
  public native int PostLayerImageRGBA8888ByteBuffer(long handle, int index, int left, int top,
                       ByteBuffer rgba_plane, int offset, int row_stride, int width, int height,
                       int is_vertical_flip,  int is_horizontal_flip,
                       int scale_width,  int scale_height, int scale_filter_mode,
                       int rotation_degree);

The display control of the above watermark is encapsulated by layerposthread:

/*
 * LayerPostThread Realize dynamic watermark encapsulation
 * Author: https://daniusdk.com
 */
class LayerPostThread extends Thread
{
  private final int update_interval = 400; // 400 ms
  private volatile boolean is_exit_ = false;
  private long handle_ = 0;
  private int width_  = 0;
  private int height_ = 0;
  private volatile boolean is_text_ = false;
  private volatile boolean is_picture_ = false;
  private volatile boolean clear_flag_ = false;

  private final int timestamp_index_ = 1;
  private final int text1_index_ = 2;
  private final int text2_index_ = 3;
  private final int picture_index_ = 4;
  private final int rectangle_index_ = 5;

  ByteBuffer text_timestamp_buffer_ = null;
  ByteBuffer rectangle_buffer_ = null;

  @Override
  public void run() {
      text_timestamp_buffer_ = null;
      rectangle_buffer_ = null;

      if (0 == handle_)
          return;

      boolean is_posted_pitcure = false;
      boolean is_posted_text1 = false;
      boolean is_posted_text2 = false;

      int rectangle_aplha = 0;

      while(!is_exit_) {
          long t = SystemClock.elapsedRealtime();

          if (clear_flag_) {
              clear_flag_ = false;
              is_posted_pitcure = false;
              is_posted_text1 = false;
              is_posted_text2 = false;

              if (!is_text_ || !is_picture_) {
                  rectangle_aplha = 0;
                  libPublisher.RemoveLayer(handle_, rectangle_index_);
              }
          }

          int cur_h = 8;
          int ret = 0;

          if (!is_exit_ && is_text_) {
              ret = postTimestampLayer(timestamp_index_, 0, cur_h);
              if ( ret > 0 )
                  cur_h = align(cur_h + ret + 2, 2);
          }

          if(!is_exit_&& is_text_&&!is_posted_text1) {
              cur_h += 6;
              ret = postText1Layer(text1_index_, 0, cur_h);
              if ( ret > 0 ) {
                  is_posted_text1 = true;
                  cur_h = align(cur_h + ret + 2, 2);
              }
          }

          if (!is_exit_ && is_picture_ && !is_posted_pitcure) {
              ret = postPictureLayer(picture_index_, 0, cur_h);
              if ( ret > 0 ) {
                  is_posted_pitcure = true;
                  cur_h = align(cur_h + ret + 2, 2);
              }
          }

          if(!is_exit_&& is_text_&&!is_posted_text2) {
              postText2Layer(text2_index_);
              is_posted_text2 = true;
          }

          // This is a demonstration of a rectangle, which can be shielded if not required
          if (!is_exit_ && is_text_ && is_picture_) {
                  postRGBRectangle(rectangle_index_, rectangle_aplha);
                  rectangle_aplha += 8;
                  if (rectangle_aplha > 255)
                      rectangle_aplha = 0;
          }

          waitSleep((int)(SystemClock.elapsedRealtime() - t));
      }

      text_timestamp_buffer_ = null;
      rectangle_buffer_ = null;
  }

We divide watermarks into two categories: one is text watermark and the other is png logo watermark, which can be displayed or hidden by controlling:

public void enableText(boolean is_text) {
      is_text_ = is_text;
      clear_flag_ = true;
      if (handle_ != 0) {
          libPublisher.EnableLayer(handle_, timestamp_index_, is_text_?1:0);
          libPublisher.EnableLayer(handle_, text1_index_, is_text_?1:0);
          libPublisher.EnableLayer(handle_, text2_index_, is_text_?1:0);
      }
  }

  public void enablePicture(boolean is_picture) {
      is_picture_ = is_picture;
      clear_flag_ = true;
      if (handle_ != 0) {
          libPublisher.EnableLayer(handle_, picture_index_, is_picture_?1:0);
      }
  }

To remove a layer, you can also call the RemoveLayer() interface. The specific design is as follows:

/**
   * To enable or disable the video layer, this interface must be called after StartXXX
   *
   * @param index: The layer index must be greater than 0. Note that layer 0 cannot be deactivated
   *
   * @param  is_enable: Enable or not, 0 disabled, 1 enabled
   *
   * @return {0} if successful
   */
  public native int EnableLayer(long handle, int index, int is_enable);


  /**
   * Remove the video layer. This interface must be called after StartXXX
   *
   * @param index: The layer index must be greater than 0. Note that layer 0 cannot be removed
   *
   * @return {0} if successful
   */
  public native int RemoveLayer(long handle, int index);

For starting outer encapsulation such as watermark type:

private LayerPostThread layer_post_thread_ = null;

  private void startLayerPostThread() {
      if (3 == video_opt_) {
          if (null == layer_post_thread_) {
              layer_post_thread_ = new LayerPostThread();
              layer_post_thread_.startPost(publisherHandle, videoWidth, videoHeight, currentOrigentation, isHasTextWatermark(), isHasPictureWatermark());
          }
      }
  }

  private void stopLayerPostThread() {
      if (layer_post_thread_ != null) {
          layer_post_thread_.stopPost();
          layer_post_thread_ = null;
      }
  }

summary

With the increasing requirements of traditional industries for real-time watermarking of video data, dynamic watermarking design is the general trend. There are many implementation modes of watermarking design. For example, in the early stage, we directly realized the processing of static watermarking through jni encapsulation layer. If we want to realize dynamic watermarking more flexibly through layered design, the ideas provided in this paper can be used for reference by developers.

 

Posted by White_Coffee on Wed, 25 May 2022 08:58:54 +0300