javascript - Provide ReadableStream as a download source for streamed download - Stack Overflow

admin管理员组

文章数量:1355128

Searching for "download ReadableStream", I find two questions:

1. Download Readablestream as file
2. How to download a ReadableStream on the browser that has been returned from fetch

Both have answers ending up in collecting all the data in a Blob and using URL.createObjectURL. Yet having a ReadableStream, it is a waste to first create a huge Blob in memory before the browser can start the download operation.

Is there a way to let the client download directly from a ReadableStream, without first collecting all data in memory?

NOTE: this ReadableStream is created by the client from lots of data, so relying on fetch Response objects will hardly be a solution.

Searching for "download ReadableStream", I find two questions:

1. Download Readablestream as file
2. How to download a ReadableStream on the browser that has been returned from fetch

Both have answers ending up in collecting all the data in a Blob and using URL.createObjectURL. Yet having a ReadableStream, it is a waste to first create a huge Blob in memory before the browser can start the download operation.

Is there a way to let the client download directly from a ReadableStream, without first collecting all data in memory?

NOTE: this ReadableStream is created by the client from lots of data, so relying on fetch Response objects will hardly be a solution.

• javascript
• download
• streaming

Share Improve this question Follow edited Mar 29 at 16:21 Harald asked Mar 29 at 16:14 HaraldHarald 5,21588 gold badges3838 silver badges8282 bronze badges 1

• Your question 2 has a "what about this" comment which appears to offer a more direct solution. – James Commented Mar 29 at 18:32

Add a comment  | 

2 Answers 2

Sorted by: Reset to default 1

This is how to use WHATWG Streams ReadableStream from WHATWG Fetch Response on Chromium-based browsers (Chrome, Chromium, Brave, Opera, Edge, et al.) and write that file (it's also possible to write folders) to the filesystem using WICG File System Access API - not be be confused with WHATWG File System (Standard), which share some of the same interfaces, such as FileSystemWritableFileStream

let handle = await showSaveFilePicker({
  startIn: "downloads",
  suggestedName: "qjs"
});

let writable = await handle.createWritable();

let readable = (await fetch(
  "https://corsproxy.io?url=https://github/quickjs-ng/quickjs/releases/latest/download/qjs-linux-x86_64",
)).body;

await readable.pipeTo(writable);

console.log("Done downloading file");

Yes.

Here's how to do that using Deno, from the browser with Native Messaging https://github/guest271314/native-messaging-file-writer

var readable = ... // WHATWG ReadableStream
var fs = new FileWriter({
  fileName: "/home/user/Downloads/node", // File path to write to
  mode: 0o764 // Mode 
}, "/home/user/native-messaging-file-writer"); // Path to unpacked extension directory
fs.write(readable).then(console.log).catch(console.warn);
// Abort writing to the file
fs.abort("reason");

And here's how to do that using QuickJS NG https://github/guest271314/native-messaging-file-writer/tree/quickjs. Still, from the browser, using Native Messaging.

Fetch and write latest QuickJS NG qjs to file system

var {
  externalController,
  progressStream
} = await connectExternalFileWriter(
  "/home/user/native-messaging-file-writer-quickjs",
  "/home/user/Downloads/qjs",
  ["O_RDWR", "O_CREAT", "O_TRUNC"],
  "0o744"
).catch(console.error);
// externalController.error("a reason");
// externalController.close();
progressStream.pipeTo(new WritableStream({
  start() {
    console.groupCollapsed("FileWriter progress");
  },
  write(v) {
    console.log(v);
  },
  close() {
    console.groupEnd("FileWriter progress");
  },
  abort(reason) {
    console.log(reason);
    console.groupEnd("FileWriter progress");
  }
}), ).catch(console.error);

var writeStream =
  fetch(
    "https://corsproxy.io?url=https://github/quickjs-ng/quickjs/releases/latest/download/qjs-linux-x86_64",
  ).then((r) => r.body.pipeTo(new WritableStream({
    write(v) {
      console.log(externalController);
      externalController.enqueue(v);
    },
    close() {
      externalController.close();
    },
  })));

Fetch and write node nightly to file system

const {
  UntarFileStream
} = await import(URL.createObjectURL(new Blob([await (await fetch("https://gist.githubusercontent/guest271314/93a9d8055559ac8092b9bf8d541ccafc/raw/022c3fc6f0e55e7de6fdfc4351be95431a422bd1/UntarFileStream.js")).bytes()], {
  type: "text/javascript"
})));

const cors_api_host = "corsproxy.io/?url=";
const cors_api_url = "https://" + cors_api_host;
let osArch = "linux-x64";
let file;

let [node_nightly_build] = await (await fetch("https://nodejs./download/nightly/index.json")).json();
let {
  version,
  files
} = node_nightly_build;
let node_nightly_url = `https://nodejs./download/nightly/${version}/node-${version}-${osArch}.tar.gz`;
let url = `${cors_api_url}${node_nightly_url}`;
console.log(`Fetching ${node_nightly_url}`);
const request = (await fetch(url)).body.pipeThrough(new DecompressionStream('gzip'));
// Download gzipped tar file and get ArrayBuffer
const buffer = await new Response(request).arrayBuffer();
// Decompress gzip using pako
// Get ArrayBuffer from the Uint8Array pako returns
// const decompressed = await pako.inflate(buffer);
// Untar, js-untar returns a list of files
// (See https://github/InvokIT/js-untar#file-object for details)
const untarFileStream = new UntarFileStream(buffer);
while (untarFileStream.hasNext()) {
  file = untarFileStream.next();
  if (/\/bin\/node$/.test(file.name)) {
    break;
  }
}

var stream = new Blob([file.buffer]).stream();

var {
  externalController,
  progressStream
} = await connectExternalFileWriter(
  "/home/user/native-messaging-file-writer-quickjs",
  "/home/user/Downloads/node",
  ["O_RDWR", "O_CREAT", "O_TRUNC"],
  "0o744"
).catch(console.error);
// externalController.error("a reason");
// externalController.close();
progressStream.pipeTo(new WritableStream({
  start() {
    console.groupCollapsed("FileWriter progress");
  },
  write(v) {
    console.log(v);
  },
  close() {
    console.groupEnd("FileWriter progress");
  },
  abort(reason) {
    console.log(reason);
    console.groupEnd("FileWriter progress");
  }
}), ).catch(console.error);

var writeStream = stream.pipeTo(new WritableStream({
    write(v) {
      externalController.enqueue(v);
    },
    close() {
      externalController.close();
    },
  }));

writeStream.catch(console.error);

Technically, this is also possible using WICG File System Access API in the browser (Chromium-based browsers such as Chrome, Opera, Brave, Edge, Chromium) alone, without using a Web extension or Native Messaging.

Here's one way I fetch the Node.js nightly archive, extract only the node executable, and write that file to the filesystem https://github/guest271314/download-node-nightly-executable/blob/main/index.html#L29-L68. Hint: If you do something like

cd ~/Downloads
touch node
chmod u+x node

to create an empty file named node and set executable permission on that file, the file permission will be retained, see https://issues.chromium./issues/40742294

      try {
        let [node_nightly_build] = await (
          await fetch('https://nodejs./download/nightly/index.json')
        ).json();
        let {
          version,
          files
        } = node_nightly_build;
        let node_nightly_url = `https://nodejs./download/nightly/${version}/node-${version}-${osArch}.tar.gz`;
        let url = `${cors_api_url}${node_nightly_url}`;
        console.log(`Fetching ${node_nightly_url}`);
        const request = (await fetch(url)).body.pipeThrough(
          new DecompressionStream('gzip')
        );
        // Download gzipped tar file and get ArrayBuffer
        const buffer = await new Response(request).arrayBuffer();
        // Decompress gzip using pako
        // Get ArrayBuffer from the Uint8Array pako returns
        // const decompressed = await pako.inflate(buffer);
        // Untar, js-untar returns a list of files
        // (See https://github/InvokIT/js-untar#file-object for details)
        const untarFileStream = new UntarFileStream(buffer);
        while (untarFileStream.hasNext()) {
          file = untarFileStream.next();
          if (/\/bin\/node$/.test(file.name)) {
            break;
          }
        }
        writable = await fileSystemHandle.createWritable();
        writer = writable.getWriter();
        await writer.write(file.buffer);
        await writer.close();
        new Notification('Download complete.', {
          body: `Successfully downloaded node executable ${version}`
        });
      } catch (e) {
        console.log(e);
      } finally {
        console.log('Done');
      }

Now, you can do something like

await readable.pipeTo(writable);

to pipe a WHATWG ReadableStream to the file system on chrome, and that atomic pipe automatically gets rid of the .crswap file chrome writes to originally before moving the data to the actual file.

I wrote the above approaches using deno and qjs to fix the currently won't fix status of the linked Chromium bug https://issues.chromium./issues/40743502.

本文标签： javascriptProvide ReadableStream as a download source for streamed downloadStack Overflow