Shell script to backup dir from list - Stack Overflow

admin管理员组

文章数量:1333406

I have a backup script which does the job but takes a long time to complete because it tars Users dir against a list of servers in DEV_Servers.txt one by one.

Is there a way the tar command could run for the list of servers all at the same time instead of going through the list one by one?

#!/bin/ksh
print  "==========================================="
print  "Taking backup of Users dir"
print  "==========================================="

serverList="DEV_Servers.txt"
while read -r server;
do
    print
    print "==========================================="
    print "Taking Backup of Users:" $server
    ssh -o StrictHostKeyChecking=no -n $server "cd /opt/test/ ; tar -zcvf 'Users$(date '+%Y%m%d').tar.gz' Users"
done < "$serverList"

I have a backup script which does the job but takes a long time to complete because it tars Users dir against a list of servers in DEV_Servers.txt one by one.

Is there a way the tar command could run for the list of servers all at the same time instead of going through the list one by one?

#!/bin/ksh
print  "==========================================="
print  "Taking backup of Users dir"
print  "==========================================="

serverList="DEV_Servers.txt"
while read -r server;
do
    print
    print "==========================================="
    print "Taking Backup of Users:" $server
    ssh -o StrictHostKeyChecking=no -n $server "cd /opt/test/ ; tar -zcvf 'Users$(date '+%Y%m%d').tar.gz' Users"
done < "$serverList"

• shell
• backup
• ksh

Share Improve this question Follow edited Nov 23, 2024 at 16:20 Incursio_02 11011 silver badge77 bronze badges asked Nov 20, 2024 at 20:09 Engineer83Engineer83 12977 bronze badges 4

• 3 Run the ssh command in the background with & so they all run in parallel. Or use GNU Parallel. – Barmar Commented Nov 20, 2024 at 20:39
• 3 /bin/ksh? Why is your question tagged with bash? – Cyrus Commented Nov 20, 2024 at 20:39
• sorry you are right, I should of put shell. – Engineer83 Commented Nov 20, 2024 at 20:56
• 1 minor nitpick: $(date '+%Y%m%d') is going to return the same result on each pass through the loop (in a worse case scenario you could generate a 2nd/different result when passing midnight); consider running date '+%Y%m%d' once earlier in the script, store in a variable and then reference the variable in the ssh call; if using bash or ksh you can make use of the printf / -v capability, eg: printf -v dt "%(%Y%m%d)T" - the variable dt will now contain the current date (eg, 20241120) – markp-fuso Commented Nov 20, 2024 at 20:58

Add a comment  | 

2 Answers 2

Sorted by: Reset to default 1

One way to run the tar commands in parallel is by putting each SSH call in the background. You can do this by appending an & at the end of each SSH command. Here's how you can modify your script:

#!/bin/ksh
print "==========================================="
print "Taking backup of Users dir"
print "==========================================="

serverList="DEV_Servers.txt"
while read -r server;
do
    print
    print "==========================================="
    print "Taking Backup of Users:" $server
    # Run tar command in the background
    ssh -o StrictHostKeyChecking=no -n $server "cd /opt/test/ ; tar -zcvf 'Users$(date '+%Y%m%d').tar.gz' Users" &
done < "$serverList"

# Wait for all background jobs to finish
wait

• The & at the end of the ssh command tells the shell to run the command in the background.
• The wait command ensures that the script waits for all background jobs to finish before it exits.

Untested:

#!/bin/ksh
print  "==========================================="
print  "Taking backup of Users dir"
print  "==========================================="

serverList="DEV_Servers.txt"

do_one() {
    server="$1"
    print
    print "==========================================="
    print "Taking Backup of Users:" $server
    ssh -o StrictHostKeyChecking=no -n "$server" "cd /opt/test/ ; tar -zcvf 'Users$(date '+%Y%m%d').tar.gz' Users"
}
export -f do_one

parallel -j0 do_one < "$serverList"

It requires that ssh server does not ask for a passphrase. If you have set up ssh-agent this should be fine.

本文标签： Shell script to backup dir from listStack Overflow