javascript - Optimizing transparent video method - Stack Overflow

admin管理员组

文章数量:1387316

Video

Hello, I've been trying to render this video on my site with a transparent background. After some research I managed to stumble upon this resource which let me get my desired effect using a canvas.

Using the B&W mask on the right side of the video, I can set the pixels I'd like to be transparent. Issue is my current method at times stutters or has low framerate. How would I go about optimizing this or taking a different approach entirely? I feel like having a second canvas just to draw then read the pixels seems inefficient.

Also to note, for whatever reason my rendered image seems to be squished compared to the original.

Here is my current code

const canvas = this.$refs.canvas.getContext("2d", { willReadFrequently: true })
const render = this.$refs.render.getContext("2d", { willReadFrequently: true })
const video = this.$refs.video

video.play()

function maskVideo() {
    canvas.drawImage(video, 0, 0, 3840, 1280)
    
    const frame = canvas.getImageData(0, 0, 1920, 1280)
    const frameData = frame.data

    for (var i = 0; i < frameData.length; i += 4) {
        const [r, g, b] = [frameData[i], frameData[i + 1], frameData[i + 2]]

        frameData[i + 3] = (r + g + b)
    }

    render.putImageData(frame, 0, 0)

    window.requestAnimationFrame(maskVideo)
}

window.requestAnimationFrame(maskVideo)

Video

Hello, I've been trying to render this video on my site with a transparent background. After some research I managed to stumble upon this resource which let me get my desired effect using a canvas.

Using the B&W mask on the right side of the video, I can set the pixels I'd like to be transparent. Issue is my current method at times stutters or has low framerate. How would I go about optimizing this or taking a different approach entirely? I feel like having a second canvas just to draw then read the pixels seems inefficient.

Also to note, for whatever reason my rendered image seems to be squished compared to the original.

Here is my current code

const canvas = this.$refs.canvas.getContext("2d", { willReadFrequently: true })
const render = this.$refs.render.getContext("2d", { willReadFrequently: true })
const video = this.$refs.video

video.play()

function maskVideo() {
    canvas.drawImage(video, 0, 0, 3840, 1280)
    
    const frame = canvas.getImageData(0, 0, 1920, 1280)
    const frameData = frame.data

    for (var i = 0; i < frameData.length; i += 4) {
        const [r, g, b] = [frameData[i], frameData[i + 1], frameData[i + 2]]

        frameData[i + 3] = (r + g + b)
    }

    render.putImageData(frame, 0, 0)

    window.requestAnimationFrame(maskVideo)
}

window.requestAnimationFrame(maskVideo)

• javascript
• canvas
• video
• html5-canvas

Share Improve this question Follow asked Mar 17 at 10:35 giannigianni 12511 silver badge1111 bronze badges 1

• 2 Seems like an XY Problem. Why not just render your video with a transparent background (SO)? Most modern browsers handle this already. Maybe I've missed something in the question? – fdomn-m Commented Mar 17 at 10:52

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 1

• You need frameData[i + 3] = ((r + g + b)/3) to avoid "echo" trails.

• Use a smaller video resolution.
  (Does the avatar really need to be 1280 pixels high? or is 800 enough?).

• Use a video format with transparency.
  (Save Webm with VP9 codec, or else, MP4 with HEVC codec).

But if you insist that it must be done manually (with no file re-encoding) then...
For smoother playback of such masked videos (at 3840 x 1280) you need to use the GPU.

Compare your Canvas speed with my example GPU demo:
https://vcone.github.io/public/demos/js/draw_vid_mask/index.html

Is GPU processing (3D) looking smoother than the drawImage() of Canvas (2D)?
You can also check the page's source code to learn from it.
It is based on this other StackOverflow Answer.

In that Answer's code you need to modify the function main(void) to achieve a masking effect.
First create two variables to store the current colours of video and mask pixels separately (because easier to process later).

mediump vec4 my_FragColor_video = vec4(1.0, 1.0, 1.0, 1.0);
mediump vec4 my_FragColor_mask = vec4(1.0, 1.0, 1.0, 1.0);

void main(void) 
{
    if( (vDirection.x * 0.5) < 0.005) //# when GPU is scanning at first half of image size 
    {
        //# read from first half
        my_FragColor_video = texture2D(uSampler, vec2( (vDirection.x * 0.5 ) + 0.5 , (vDirection.y * 0.5 ) + 0.5) );
        
        //# read from second half
        my_FragColor_mask = texture2D(uSampler, vec2( (vDirection.x * 0.5 ) + 1.0 , (vDirection.y * 0.5 ) + 0.5) );
    }
    
    //# first put te video's pixel colour
    gl_FragColor = my_FragColor_video;
    
    //# then alpha is from using "green" of mask, but can use any r/g/b channel of mask 
    gl_FragColor.a = my_FragColor_mask.g; 
    
}

Hope it helps as a starting point for smoother video playback on your project.

Experiment by editing the GPU main() code to work with your screen size.
Expeminent which vars will adjust picture w/h, scaling, reading position, etc.

If you're new to GPU coding then just remember:

• X and Y are replaced by U and V. (eg: Usampler is really just some X-pos reader).

• Position is measured as ratio in range 0 to 1 of Width or Height.

• gl_FragColor is a built-in variable for you to get/set the current GPU "pixel" colour.

本文标签： javascriptOptimizing transparent video methodStack Overflow