javascript - How to read audio data from a 'MediaStream' object in a C++ addon - Stack Overflow

admin管理员组

文章数量:1279247

After sweating blood and tears I've finally managed to set up a Node C++ addon and shove a web-platform standard MediaStream object into one of its C++ methods for good. For patibility across different V8 and Node.js versions, I'm using Native Abstractions for Node.js (nan):

addon

NAN_METHOD(SetStream)
{
    Nan::HandleScope scope;
    v8::Local<v8::Object> mediaStream = info[0]->ToObject();
}

addon.js

setStream(new MediaStream());

For what it's worth, this works correctly (i.e. it does not obliterate the renderer process on sight), and I can verify the presence of the MediaStream object, e.g. by returning its constructor name from the C++ method:

addon

info.GetReturnValue().Set(mediaStream->GetConstructorName());

When called from JavaScript through setStream, this would return the string MediaStream, so the object is definitely there. I can also return the mediaStream object itself and everything will work correctly, so it's indeed the object I need.

So, how would I read audio data (i.e. audio samples) from this MediaStream object in C++? As a sidenote, the actual data read (and processing) would be done in a separate std::thread.

---

Bounty Update

I understand this would be sort of easier/possible if I were piling Electron and/or Chromium myself, but I'd rather not get involved in that maintenance hell.

I was wondering if it would be possible without doing that, and as far as my research goes, I'm convinced I need 2 things to get this done:

1. The relevant header files, for which I believe blink public should be adequate
2. A chromium/blink library file (?), to resolve external symbols, similarly to the node.dylib file

Also, as I said, I believe I could pile chromium/blink myself and then I would have this lib file, but that would be a maintenance hell with Electron. With this in mind, I believe this question ultimately es down to a C++ linking question. Is there any other approach to do what I'm looking for?

Edit

ScriptProcessorNode is not an option in my case, as its performance makes it nearly unusable in production. This would require to process audio samples on the ui/main thread, which is absolutely insane.

Edit 2

AudioWorklets have been available in Electron for some time now, which, unlike the ScriptProcessorNode (or worse, the AnalyzerNode), is low-latency and very reliable for true C++ backed audio processing even in real time.

If someone wants to go ahead and write an AudioWorklet-based answer, I'll gladly accept, but beware: it's a very advanced field and a very deep rabbit hole, with countless obstacles to get through even before a very simple, general-purpose pass-through prototype (especially so because currently in Electron, Atomics-synced, buffered cross-thread audio processing is required to pull this off because -- although getting a native C++ addon into one audio renderer thread, let alone multiple threads at the same time, is probably equally as challenging).

After sweating blood and tears I've finally managed to set up a Node C++ addon and shove a web-platform standard MediaStream object into one of its C++ methods for good. For patibility across different V8 and Node.js versions, I'm using Native Abstractions for Node.js (nan):

addon

NAN_METHOD(SetStream)
{
    Nan::HandleScope scope;
    v8::Local<v8::Object> mediaStream = info[0]->ToObject();
}

addon.js

setStream(new MediaStream());

For what it's worth, this works correctly (i.e. it does not obliterate the renderer process on sight), and I can verify the presence of the MediaStream object, e.g. by returning its constructor name from the C++ method:

addon

info.GetReturnValue().Set(mediaStream->GetConstructorName());

When called from JavaScript through setStream, this would return the string MediaStream, so the object is definitely there. I can also return the mediaStream object itself and everything will work correctly, so it's indeed the object I need.

So, how would I read audio data (i.e. audio samples) from this MediaStream object in C++? As a sidenote, the actual data read (and processing) would be done in a separate std::thread.

---

Bounty Update

I understand this would be sort of easier/possible if I were piling Electron and/or Chromium myself, but I'd rather not get involved in that maintenance hell.

I was wondering if it would be possible without doing that, and as far as my research goes, I'm convinced I need 2 things to get this done:

1. The relevant header files, for which I believe blink public should be adequate
2. A chromium/blink library file (?), to resolve external symbols, similarly to the node.dylib file

Also, as I said, I believe I could pile chromium/blink myself and then I would have this lib file, but that would be a maintenance hell with Electron. With this in mind, I believe this question ultimately es down to a C++ linking question. Is there any other approach to do what I'm looking for?

Edit

ScriptProcessorNode is not an option in my case, as its performance makes it nearly unusable in production. This would require to process audio samples on the ui/main thread, which is absolutely insane.

Edit 2

AudioWorklets have been available in Electron for some time now, which, unlike the ScriptProcessorNode (or worse, the AnalyzerNode), is low-latency and very reliable for true C++ backed audio processing even in real time.

If someone wants to go ahead and write an AudioWorklet-based answer, I'll gladly accept, but beware: it's a very advanced field and a very deep rabbit hole, with countless obstacles to get through even before a very simple, general-purpose pass-through prototype (especially so because currently in Electron, Atomics-synced, buffered cross-thread audio processing is required to pull this off because https://github./electron/electron/issues/22503 -- although getting a native C++ addon into one audio renderer thread, let alone multiple threads at the same time, is probably equally as challenging).

• javascript
• c++
• electron
• v8
• blink

Share Improve this question Follow edited Apr 15, 2021 at 19:35 John Weisz asked Feb 26, 2018 at 18:03 John WeiszJohn Weisz 32k1414 gold badges9696 silver badges136136 bronze badges 10

• 3 I'm not getting your question. but u can check these two : github./mon-tater/wkwebview-getusermedia-shim/blob/master/… ... realtimeweekly.co/… – nullqube Commented Mar 9, 2018 at 6:04
• @nullqube I've been checking out WebRTC for this task, but it seems it's not possible to stream raw, unpressed audio. If it was possible, I could simply start a new Electron window process, stream audio there, do ScriptProcessorNode-based processing in that separate window process, and stream back. But there's always significant amounts of lossy pression hardcoded into WebRTC, as well as very high latency. – John Weisz Commented Mar 11, 2018 at 9:31
• does this help? discuss.atom.io/t/… – nullqube Commented Aug 12, 2018 at 6:49
• Did you try using WebWorker on the JavaScript side to transfer your work into "background thread" then call the C++ part from it?? – user9335240 Commented Oct 14, 2018 at 20:43
• 3 Its not possible by doing it just with the V8 API. If you're capable of knowing the Blink engine version of your electron or chromium version, then you could add a AudioDestinationConsumer to the source of one of the audio tracks on the media stream. cs.chromium/chromium/src/third_party/blink/renderer/… I've currently not yet tested that, but In a few months I will and post a more detailed answer :) – WolverinDEV Commented Apr 6, 2019 at 14:21

 |  Show 5 more ments

1 Answer 1

Sorted by: Reset to default 1

The MediaStream header is part of Blink's renderer modules, and it's not obvious to me how you could retrieve this from nan plugin.

So, instead let's look at what you do have, namely a v8::Object. I believe that v8::Object exposes all the functionality you need, it has:

• GetPropertyNames()
• Get(context, index)
• Set(context, key, value)
• Has(context, key)

Unless you really need a strictly defined interface, why not avoid the issue altogether and just use the dynamic type that you already have?

For getting audio data out specifically, you would need to call getAudioTracks() on the v8::Object, which probably looks something like this?

Note: I don't think you need a context, v8 seems to be happy with it being empty: v8/src/api/api

Should look something like this, plus some massaging of types in and out of v8.

v8::MaybeLocal<v8::Value> get_audio_tracks = mediaStream->Get("getAudioTracks");
// Maybe needs to be v8::Object or array?
if (!get_audio_tracks.IsEmpty()) {
    v8::Local<v8::Value> audio_tracks = get_audio_tracks.ToLocalChecked()();
}

本文标签： javascriptHow to read audio data from a 39MediaStream39 object in a C addonStack Overflow