Powershell script to delete duplicate files in windows 10 - Stack Overflow

admin管理员组

文章数量:1356932

I am an amateur programmer (let's leave it that) and don't have experience with Powershell.

I have an unwieldy music library where I'm hoping to clear up duplicates. My library has things like:

\Music\Sufjan Stevens\Illinois\01 Concerning The Ufo Sighting Near Highland, Illinois.m4a

\Music\Sufjan Stevens\Illinois\01 Concerning The Ufo Sighting.m4a

...or similar.

I searched google, found a powershell script:

# Specify the folder to search
$FolderPath = "C:\Users\Ben\Music\Apple Music\Media\Music"

# Get all files in the specified folder and subfolders
$Files = Get-ChildItem -Path $FolderPath -Recurse -File

# Group files by their hash (to identify duplicates)
$Files | Get-FileHash | Group-Object -Property Hash | Where-Object { $_.Count -gt 1 } | ForEach-Object {

    # For each group of duplicate files
    foreach ($DuplicateFile in $_.Group) {

        # Skip the first file in the group (keep one copy)
        if ($DuplicateFile -ne $_.Group[0]) {

            # Delete the duplicate file
            Write-Host "Deleting: $($DuplicateFile.FullName)"
            Remove-Item -Path $DuplicateFile.FullName -Force

        }
    }
}
pause

It comes up with these kind of errors:

Resolve-Path : Cannot find path 'C:\Users\Ben\Music\Apple Music\Media\Music\Sufjan Stevens\Illinoise\02 The Black Hawk
War, Or, How To Demolish An Entire Civilization And Still Feel Good About Yourself In The Morning, Or, We Apologize
For The Inconvenience But You're Going To Have To Leave Now, Or....m4a' because it does not exist.
At
C:\Windows\system32\WindowsPowerShell\v1.0\Modules\Microsoft.PowerShell.Utility\Microsoft.PowerShell.Utility.psm1:106
char:36
+ ...     $pathsToProcess += Resolve-Path -LiteralPath $LiteralPath | Forea ...
+                            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (C:\Users\Ben\Mu... Now, Or....m4a:String) [Resolve-Path], ItemNotFoundE
   xception
    + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.ResolvePathCommand

Deleting:
@{Algorithm=SHA256; Hash=97E00B9B1D94DF9FBC7F5FF30D7BFC4BCB4DC938F8FE7DD99530C8A92EEAED10; Path=C:\Users\Ben\Music\Apple Music\Media\Music\Sufjan Stevens\Illinoise\01 Concerning The Ufo Sighting.m4a}.FullName
Remove-Item : Cannot bind argument to parameter 'Path' because it is null.
At C:\Users\Ben\Documents\DeleteDuplicates.ps1:20 char:21
+         Remove-Item -Path $DuplicateFile.FullName -Force
+                           ~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidData: (:) [Remove-Item], ParameterBindingValidationException
    + FullyQualifiedErrorId : ParameterArgumentValidationErrorNullNotAllowed,Microsoft.PowerShell.Commands.RemoveItemC
   ommand

Press Enter to continue...:

This is with the $FolderPath set to a small subdir with just 20-something songs. Normally there are hundreds of these errors (my library has 15k songs in it)

The 'black hawk' song does exist. It's obviously found the 'concerning the UFO sighting' duplicate, but can't get to it.

Anyone see any obvious errors? I am hopefully missing something obvious :P

I am an amateur programmer (let's leave it that) and don't have experience with Powershell.

I have an unwieldy music library where I'm hoping to clear up duplicates. My library has things like:

\Music\Sufjan Stevens\Illinois\01 Concerning The Ufo Sighting Near Highland, Illinois.m4a

\Music\Sufjan Stevens\Illinois\01 Concerning The Ufo Sighting.m4a

...or similar.

I searched google, found a powershell script:

# Specify the folder to search
$FolderPath = "C:\Users\Ben\Music\Apple Music\Media\Music"

# Get all files in the specified folder and subfolders
$Files = Get-ChildItem -Path $FolderPath -Recurse -File

# Group files by their hash (to identify duplicates)
$Files | Get-FileHash | Group-Object -Property Hash | Where-Object { $_.Count -gt 1 } | ForEach-Object {

    # For each group of duplicate files
    foreach ($DuplicateFile in $_.Group) {

        # Skip the first file in the group (keep one copy)
        if ($DuplicateFile -ne $_.Group[0]) {

            # Delete the duplicate file
            Write-Host "Deleting: $($DuplicateFile.FullName)"
            Remove-Item -Path $DuplicateFile.FullName -Force

        }
    }
}
pause

It comes up with these kind of errors:

Resolve-Path : Cannot find path 'C:\Users\Ben\Music\Apple Music\Media\Music\Sufjan Stevens\Illinoise\02 The Black Hawk
War, Or, How To Demolish An Entire Civilization And Still Feel Good About Yourself In The Morning, Or, We Apologize
For The Inconvenience But You're Going To Have To Leave Now, Or....m4a' because it does not exist.
At
C:\Windows\system32\WindowsPowerShell\v1.0\Modules\Microsoft.PowerShell.Utility\Microsoft.PowerShell.Utility.psm1:106
char:36
+ ...     $pathsToProcess += Resolve-Path -LiteralPath $LiteralPath | Forea ...
+                            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (C:\Users\Ben\Mu... Now, Or....m4a:String) [Resolve-Path], ItemNotFoundE
   xception
    + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.ResolvePathCommand

Deleting:
@{Algorithm=SHA256; Hash=97E00B9B1D94DF9FBC7F5FF30D7BFC4BCB4DC938F8FE7DD99530C8A92EEAED10; Path=C:\Users\Ben\Music\Apple Music\Media\Music\Sufjan Stevens\Illinoise\01 Concerning The Ufo Sighting.m4a}.FullName
Remove-Item : Cannot bind argument to parameter 'Path' because it is null.
At C:\Users\Ben\Documents\DeleteDuplicates.ps1:20 char:21
+         Remove-Item -Path $DuplicateFile.FullName -Force
+                           ~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidData: (:) [Remove-Item], ParameterBindingValidationException
    + FullyQualifiedErrorId : ParameterArgumentValidationErrorNullNotAllowed,Microsoft.PowerShell.Commands.RemoveItemC
   ommand

Press Enter to continue...:

This is with the $FolderPath set to a small subdir with just 20-something songs. Normally there are hundreds of these errors (my library has 15k songs in it)

The 'black hawk' song does exist. It's obviously found the 'concerning the UFO sighting' duplicate, but can't get to it.

Anyone see any obvious errors? I am hopefully missing something obvious :P

• windows
• powershell

Share Improve this question Follow asked Mar 28 at 12:38 Benjamin RichBenjamin Rich 2111 silver badge44 bronze badges 2

• The error states "Resolve-Path : Cannot find path .." and there is no Resolve-Path in your code? You should also first group by Length of the files to avoid hashing them when not needed. – Santiago Squarzon Commented Mar 28 at 12:44
• The path is 274 characters long. Windows and PowerShell had a maximum path length limitation of 260 characters. Set this registry key to remove the limit and use PowerShell 7 which supports long paths. – zett42 Commented Mar 28 at 12:54

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 0

The main issue is in $DuplicateFile.FullName, there is no .FullName property in the object that Get-FileHash outputs, the property that contains the file's absolute path is just .Path. Leaving that aside, it's unclear where the first error with Resolve-Path comes from, it's not seen the usage of this cmdlet in your code.

As an improvement, you should also consider grouping the files first by .Length as to avoid hashing them when it may not be needed.

Here is how you could approach your code:

$FolderPath = 'C:\Users\Ben\Music\Apple Music\Media\Music'
Get-ChildItem -Path $FolderPath -Recurse -File |
    # Group by Length and take only the groups with more than 1 item
    Group-Object Length | Where-Object Count -GT 1 |
    # For each group, get their MD5 Hash
    ForEach-Object Group | Get-FileHash -Algorithm MD5 |
    # Group by Hash and take only the groups with more than 1 item
    Group-Object Hash | Where-Object Count -GT 1 |
    # For each group, skip the first item and take the rest
    ForEach-Object { $_.Group | Select-Object -Skip 1 } |
    # Remove them using verbose to get console output on what is being done
    Remove-Item -LiteralPath { $_.Path } -Verbose

本文标签： Powershell script to delete duplicate files in windows 10Stack Overflow