admin管理员组文章数量:1287517
Having made a program which streams PNG images to the browser by means of a multipart/x-mixed-replace Content-Type header, I noticed that only the frame before-last is displayed in the <img> tag, as opposed to the most recently sent one.
This behaviour is very annoying, as I'm only sending updates when the image changes to save on bandwidth, which means that the wrong frame will be on screen while I'm waiting for it to update.
Specifically, I am using Brave Browser (based on chromium), but as I have tried with both "shields" up and down, I assume this problem occurs also in other chromium-based browsers at least.
Searching for the problem yields only one relevant result (and many non-relevant ones) which is this HowToForge thread, with no replies. Likewise, I also thought the issue is to do with buffering, but I made sure to flush the buffer to no avail, much alike to the user in the thread. The user does report that it works on one of their servers though and not the other, which then lead me to believe that it may be to do with a specific HTTP header or something along those lines. My first guess was Content-Length because the browser can tell when the image is plete from that, but it didn't seem to have any effect.
So essentially, my question is: Is there a way to tell the browser to show the most recent multipart/x-mixed-replace and not the one before? And, if this isn't standard behaviour, what could the cause be?
And of course, here's the relevant source code, though I imagine this is more of a general HTTP question than one to do with the code:
Server
package routes
import (
"crypto/md5"
"fmt"
"image/color"
"net/http"
"time"
brain "path/to/image/generator/module"
)
func init() {
RouteHandler{
function: func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "multipart/x-mixed-replace; boundary=frame")
w.Header().Set("Cache-Control", "no-cache") // <- Just in case
w.WriteHeader(200)
// If the request contains a token and the token maps to a valid "brain", start consuming frames from
// the brain and returning them to the client
params := r.URL.Query()
if val, ok := params["token"]; ok && len(val) > 0 {
if b, ok := SharedMemory["brains"].(map[string]*brain.Brain)[val[0]]; ok && !b.CheckHasExit() {
// Keep a checksum of the previous frame to avoid sending frames which haven't changed. Frames cannot
// be pared directly (at least efficiently) as they are slices not arrays
previousFrameChecksum := [16]byte{}
for {
if !b.CheckHasExit() {
frame, err := b.GetNextFrame(SharedMemory["conf"].(map[string]interface{})["DISPLAY_COL"].(color.Color))
if err == nil && md5.Sum(frame) != previousFrameChecksum {
// Only write the frame if we succesfully read it and it's different to the previous
_, err = w.Write([]byte(fmt.Sprintf("--frame\r\nContent-Type: image/png\r\nContent-Size: %d\r\n\r\n%s\r\n", len(frame), frame)))
if err != nil {
// The client most likely disconnected, so we should end the stream. As the brain still exists, the
// user can re-connect at any time
return
}
// Update the checksum to this frame
previousFrameChecksum = md5.Sum(frame)
// If possible, flush the buffer to make sure the frame is sent ASAP
if flusher, ok := w.(http.Flusher); ok {
flusher.Flush()
}
}
// Limit the framerate to reduce CPU usage
<-time.After(time.Duration(SharedMemory["conf"].(map[string]interface{})["FPS_LIMITER_INTERVAL"].(int)) * time.Millisecond)
} else {
// The brain has exit so there is no more we can do - we are braindead :P
return
}
}
}
}
},
}.Register("/stream", "/stream.png")
}
Client (start() runs in body onload)
function start() {
// Fetch the token from local storage. If it's empty, the server will automatically create a new one
var token = localStorage.getItem("token");
// Create a session with the server
http = new XMLHttpRequest();
http.open("GET", "/startsession?token="+(token)+"&w="+(parent.innerWidth)+"&h="+(parent.innerHeight));
http.send();
http.onreadystatechange = (e) => {
if (http.readyState === 4 && http.status === 200) {
// Save the returned token
token = http.responseText;
localStorage.setItem("token", token);
// Create screen
var img = document.createElement("img");
img.alt = "main display";
// Hide the loader when it loads
img.onload = function() {
var loader = document.getElementById("loader");
loader.remove();
}
// Start loading
img.src = "/stream.png?token="+token;
// Start capturing keystrokes
document.onkeydown = function(e) {
// Send the keypress to the server as a mand (ignore the response)
cmdsend = new XMLHttpRequest();
cmdsend.open("POST", "/cmd?token="+(token));
cmdsend.send("keypress:"+e.code);
// Catch special cases
if (e.code === "Escape") {
// Clear local storage to remove leftover token
localStorage.clear();
// Remove keypress handler
document.onkeydown = function(e) {}
// Notify the user
alert("Session ended succesfully and the screen is inactive. You may now close this tab.");
}
// Cancel whatever it is the keypress normally does
return false;
}
// Add screen to body
document.getElementById("body").appendChild(img);
} else if (http.readyState === 4) {
alert("Error while starting the session: "+http.responseText);
}
}
}
Having made a program which streams PNG images to the browser by means of a multipart/x-mixed-replace Content-Type header, I noticed that only the frame before-last is displayed in the <img> tag, as opposed to the most recently sent one.
This behaviour is very annoying, as I'm only sending updates when the image changes to save on bandwidth, which means that the wrong frame will be on screen while I'm waiting for it to update.
Specifically, I am using Brave Browser (based on chromium), but as I have tried with both "shields" up and down, I assume this problem occurs also in other chromium-based browsers at least.
Searching for the problem yields only one relevant result (and many non-relevant ones) which is this HowToForge thread, with no replies. Likewise, I also thought the issue is to do with buffering, but I made sure to flush the buffer to no avail, much alike to the user in the thread. The user does report that it works on one of their servers though and not the other, which then lead me to believe that it may be to do with a specific HTTP header or something along those lines. My first guess was Content-Length because the browser can tell when the image is plete from that, but it didn't seem to have any effect.
So essentially, my question is: Is there a way to tell the browser to show the most recent multipart/x-mixed-replace and not the one before? And, if this isn't standard behaviour, what could the cause be?
And of course, here's the relevant source code, though I imagine this is more of a general HTTP question than one to do with the code:
Server
package routes
import (
"crypto/md5"
"fmt"
"image/color"
"net/http"
"time"
brain "path/to/image/generator/module"
)
func init() {
RouteHandler{
function: func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "multipart/x-mixed-replace; boundary=frame")
w.Header().Set("Cache-Control", "no-cache") // <- Just in case
w.WriteHeader(200)
// If the request contains a token and the token maps to a valid "brain", start consuming frames from
// the brain and returning them to the client
params := r.URL.Query()
if val, ok := params["token"]; ok && len(val) > 0 {
if b, ok := SharedMemory["brains"].(map[string]*brain.Brain)[val[0]]; ok && !b.CheckHasExit() {
// Keep a checksum of the previous frame to avoid sending frames which haven't changed. Frames cannot
// be pared directly (at least efficiently) as they are slices not arrays
previousFrameChecksum := [16]byte{}
for {
if !b.CheckHasExit() {
frame, err := b.GetNextFrame(SharedMemory["conf"].(map[string]interface{})["DISPLAY_COL"].(color.Color))
if err == nil && md5.Sum(frame) != previousFrameChecksum {
// Only write the frame if we succesfully read it and it's different to the previous
_, err = w.Write([]byte(fmt.Sprintf("--frame\r\nContent-Type: image/png\r\nContent-Size: %d\r\n\r\n%s\r\n", len(frame), frame)))
if err != nil {
// The client most likely disconnected, so we should end the stream. As the brain still exists, the
// user can re-connect at any time
return
}
// Update the checksum to this frame
previousFrameChecksum = md5.Sum(frame)
// If possible, flush the buffer to make sure the frame is sent ASAP
if flusher, ok := w.(http.Flusher); ok {
flusher.Flush()
}
}
// Limit the framerate to reduce CPU usage
<-time.After(time.Duration(SharedMemory["conf"].(map[string]interface{})["FPS_LIMITER_INTERVAL"].(int)) * time.Millisecond)
} else {
// The brain has exit so there is no more we can do - we are braindead :P
return
}
}
}
}
},
}.Register("/stream", "/stream.png")
}
Client (start() runs in body onload)
function start() {
// Fetch the token from local storage. If it's empty, the server will automatically create a new one
var token = localStorage.getItem("token");
// Create a session with the server
http = new XMLHttpRequest();
http.open("GET", "/startsession?token="+(token)+"&w="+(parent.innerWidth)+"&h="+(parent.innerHeight));
http.send();
http.onreadystatechange = (e) => {
if (http.readyState === 4 && http.status === 200) {
// Save the returned token
token = http.responseText;
localStorage.setItem("token", token);
// Create screen
var img = document.createElement("img");
img.alt = "main display";
// Hide the loader when it loads
img.onload = function() {
var loader = document.getElementById("loader");
loader.remove();
}
// Start loading
img.src = "/stream.png?token="+token;
// Start capturing keystrokes
document.onkeydown = function(e) {
// Send the keypress to the server as a mand (ignore the response)
cmdsend = new XMLHttpRequest();
cmdsend.open("POST", "/cmd?token="+(token));
cmdsend.send("keypress:"+e.code);
// Catch special cases
if (e.code === "Escape") {
// Clear local storage to remove leftover token
localStorage.clear();
// Remove keypress handler
document.onkeydown = function(e) {}
// Notify the user
alert("Session ended succesfully and the screen is inactive. You may now close this tab.");
}
// Cancel whatever it is the keypress normally does
return false;
}
// Add screen to body
document.getElementById("body").appendChild(img);
} else if (http.readyState === 4) {
alert("Error while starting the session: "+http.responseText);
}
}
}
- Sounds like it could be a caching issue. Make the img.src url unique by adding some random junk as a get param. – John Commented Jan 6, 2021 at 22:03
-
@John That crossed my mind, but the image does change, it just changes one frame too late, so it's unlikely that it's caching IMO. On top of that, the token changes between program restarts (and that's a GET param) so that somewhat stops caching, and then also I added the
Cache-Control: no-cacheheader for good measure. – user9123 Commented Jan 6, 2021 at 22:15
4 Answers
Reset to default 5A part inside a multipart MIME message starts with the MIME header and ends with the boundary. There is a single boundary before the first real part. This initial boundary closes the MIME preamble.
Your code instead assumes that a part starts with the boundary. Based on this assumption you first send the boundary, then the MIME header and then the MIME body. Then you stop sending until the next part is ready. Because of this the end of one part will only be detected once you send the next part, since only then you send the end boundary of the previous part.
To fix this your code should initially send one boundary to end the MIME preamble. For each new part it should then send the MIME header, the MIME body and then the boundary to end this part.
I had the same problem: There is 1 frame delay when using multipart/x-mixed-replace
This problem seems to appear in Chrome and it seems related to the fact Chrome no longer support multipart/x-mixed-replace resources. This problem is not present in Firefox.
So, the only way to "trick" Chrome into displaying a video stream is to send every image twice or accept there will be 1 frame delay. As said, problem not present in Firefox.
It is an issue with Chrome. In Firefox it worked as expected.
I worked around it in the following way C# example
var chromeWorkaround = Encoding.UTF8.GetBytes($"\r\n--{Boundary}\r\n\r\n--{Boundary}\r\n");
Append this to your stream and it seems to force chrome to render immediately.
I reported it here: https://bugs.chromium/p/chromium/issues/detail?id=1250396
I managed to make it send image immediately without 1-frame delay with Google Chrome by sending following sequence:
def frame_generator():
image = None
buf = BytesIO()
buf.write(b"Content-Type: image/jpeg\r\n\r\n")
yield buf.getvalue()
while True:
image = self.get_image()
img = Image.fromarray(image)
buf = BytesIO()
img.save(buf, "JPEG")
buf.write(b"\r\n--frame\r\n")
buf.write(b"Content-Type: image/jpeg\r\n\r\n")
yield buf.getvalue()
本文标签： javascriptmultipartxmixedreplace PNG stream always showing frame before lastStack Overflow
版权声明：本文标题：javascript - multipartx-mixed-replace PNG stream always showing frame before last - Stack Overflow 内容由网友自发贡献，该文观点仅代表作者本人， 转载请联系作者并注明出处：http://www.betaflare.com/web/1741273125a2369576.html， 本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌抄袭侵权/违法违规的内容，一经查实，本站将立刻删除。
发表评论