admin管理员组

文章数量:1291383

I'm writing a script that will transfer large amounts of data to an SFTP server, and I'd like to have some sort of terminal print-out during the long loading time for troubleshooting/debugging. I'm using paramiko for the SFTP connection and file writing.

What I currently have it this:

remote_zip_file = sftp_client.file(file_name_with_path, "wb")

remote_zip_file.write(my_data)

Is there any way to run a while loop (or something similar) that runs while paramiko.sftp_file.SFTPFile is being written?

What I'd like is something like this (pseudo code):

remote_zip_file = sftp_client.file(file_name_with_path, "wb")

while remote_zip_file.write(my_data) == IN_PROGRESS:
    time.sleep(1)
    print('Some print out that shows that the file writing is in progress')

This question is concerning the actual write function, not the put function. Answers to question How to see (log) file transfer progress using paramiko? mention the built-in callback function that can be used, but this callback parameter isn't present in the paramiko.sftp_file.SFTPFile class specified in the original question (paramiko documentation source).

I'm writing a script that will transfer large amounts of data to an SFTP server, and I'd like to have some sort of terminal print-out during the long loading time for troubleshooting/debugging. I'm using paramiko for the SFTP connection and file writing.

What I currently have it this:

remote_zip_file = sftp_client.file(file_name_with_path, "wb")

remote_zip_file.write(my_data)

Is there any way to run a while loop (or something similar) that runs while paramiko.sftp_file.SFTPFile is being written?

What I'd like is something like this (pseudo code):

remote_zip_file = sftp_client.file(file_name_with_path, "wb")

while remote_zip_file.write(my_data) == IN_PROGRESS:
    time.sleep(1)
    print('Some print out that shows that the file writing is in progress')

This question is concerning the actual write function, not the put function. Answers to question How to see (log) file transfer progress using paramiko? mention the built-in callback function that can be used, but this callback parameter isn't present in the paramiko.sftp_file.SFTPFile class specified in the original question (paramiko documentation source).

• python
• while-loop
• paramiko

Share Improve this question Follow edited Feb 14 at 21:21 mkrieger1 23.3k77 gold badges6464 silver badges8181 bronze badges asked Feb 13 at 21:36 JwokJwok 73211 gold badge99 silver badges2727 bronze badges 0

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 0

Here is a suggestion: read and write data in chunks (blocks) instead of the whole content. That way, you can display progress in between blocks. In the following sample, I upload a file from a local host to a remote one.

import contextlib

import fabric
import tqdm


def main():
    hostname = "ssh-sandbox"

    config = fabric.Config()
    if hostname not in config.base_ssh_config.get_hostnames():
        raise SystemExit(f"{hostname} not found in ~/.ssh/config")

    with contextlib.ExitStack() as stack:
        conn = stack.enter_context(fabric.Connection(host=hostname))
        sftp = stack.enter_context(conn.sftp())

        inf = stack.enter_context(open("/tmp/big", "rb"))
        outf = stack.enter_context(sftp.file("/tmp/big", "wb"))
        pbar = stack.enter_context(tqdm.tqdm(total=inf.seek(0, 2)))
        inf.seek(0, 0)

        # Read & write in chunks
        chunk_size = 10 * 1024 * 1024
        while chunk := inf.read(chunk_size):
            outf.write(chunk)
            pbar.update(chunk_size)


if __name__ == "__main__":
    main()

Notes

• I use the tqdm library to provide a progress bar. If you don't want to use it, then replace the pbar.update() call with a print() to show the progress

• I read/write in chunk of 10MB, you can adjust the chunk size for better performance on your system

• The inf.seek(0, 2) will seek to the end of the file and return its position, which is the file's size. After seeking to the end, I called inf.seek(0, 0) to move the read pointer back to the beginning of the file

• I am using the contextlib.ExitStack() to avoid deeply nested with statement. Without it, the code will be deeply nested like this:

        with fabric.Connection(host=hostname) as conn:
            with con.sftp() as sftp:
                with open("/tmp/big", "rb") as inf:
                    with sftp.file("/tmp/big", "wb") as outf:
                        with tqdm.tqdm(total=inf.seek(0, 2))) as pbar:
                            inf.seek(0, 0)
                            # The rest of the code ...
  

Update

Per request, I have updated the code to use paramiko instead of fabric.

import contextlib
import pathlib

import paramiko
import tqdm


def main():
    hostname = "ssh-sandbox"

    config = paramiko.config.SSHConfig()
    config_path = pathlib.Path("~/.ssh/config").expanduser()
    config = paramiko.config.SSHConfig.from_path(config_path)

    if hostname not in config.get_hostnames():
        raise SystemExit(f"{hostname} not found in ~/.ssh/config")
    cfg = config.lookup(hostname=hostname)

    with contextlib.ExitStack() as stack:
        conn = stack.enter_context(paramiko.SSHClient())
        conn.load_system_host_keys()
        conn.connect(cfg["hostname"], username=cfg["user"])
        sftp = stack.enter_context(conn.open_sftp())
        print(sftp)

        inf = stack.enter_context(open("/tmp/big", "rb"))
        outf = stack.enter_context(sftp.file("/tmp/big", "wb"))
        pbar = stack.enter_context(tqdm.tqdm(total=inf.seek(0, 2)))
        inf.seek(0, 0)

        # Read & write in chunks
        chunk_size = 10 * 1024 * 1024
        while chunk := inf.read(chunk_size):
            outf.write(chunk)
            pbar.update(chunk_size)


if __name__ == "__main__":
    main()

本文标签： pythonRun loop while writing to paramiko SFTP file is in progressStack Overflow