Hello, hope all is well. This issue is actually a duplicate of the main issue mentioned in issue #1 I am separating it out so it is not bundled with any other problem.

Sorry in advance for any bad formatting in this report.

I've been researching it and have come up with an explanation and possible solution.

I ran into this problem when trying to hash a data set containing 25 folders, 166 Gigs of data. Each folder had its own sets of files and subfolders of course, but the problem was that the program expects at least 1 file at the Path location, or the program exits without hashing.

I traced the issue to a strange behavior of the Get-ChildItem cmdlet. If Get-ChildItem is called with BOTH the -Recurse and -File parameters it will return nothing and not traverse any subdirectories unless 1 or more files is present at the working directory level.

As a small example lets say this is my data set:
C:\working_directory\folder1\file1.txt
C:\working_directory\folder2\an empty subfolder
C:\working_directory\folder3\file3.txt

I try to recurse and nothing is returned:
PS C:\working_directory> Get-ChildItem -Path .\* -Filter * -Recurse -Force -File
PS C:\working_directory>

Now I try again without the -File paramenter:
PS C:\working_directory> Get-ChildItem -Path .\* -Filter * -Recurse -Force

Directory: C:\working_directory\folder1

Mode LastWriteTime Length Name

-a---- 1/14/2022 4:39 PM 12 file1.txt

Directory: C:\working_directory\folder2

Mode LastWriteTime Length Name

d----- 1/14/2022 6:12 PM an empty subfolder

Directory: C:\working_directory\folder3

Mode LastWriteTime Length Name

-a---- 1/14/2022 4:39 PM 12 file3.txt

This time the Recursion is successful but a directory is also returned which we do not want.

Finally I try:
PS C:\working_directory> Get-ChildItem -Path .\* -Filter * -Recurse -Force | where { ! $_.PSIsContainer }

Directory: C:\working_directory\folder1

Mode LastWriteTime Length Name

-a---- 1/14/2022 4:39 PM 12 file1.txt

Directory: C:\working_directory\folder3

Mode LastWriteTime Length Name

-a---- 1/14/2022 4:39 PM 12 file3.txt

And here the recursion is successful and the directories are omitted.

Supposedly the $_.PSIsContainer property is only false in Files, not directories or symlinks so this can be used to obtain the File list instead of using the -File parameter itself.

Original:

lightweight proxy function for Get-ChildItem cmdlet

function dirx ([string]$Path, [string]$Filter, [string[]]$Exclude, $Recurse, [switch]$Force) {
Get-ChildItem @PSBoundParameters -File -ErrorAction SilentlyContinue
}

Modified:

lightweight proxy function for Get-ChildItem cmdlet

function dirx ([string]$Path, [string]$Filter, [string[]]$Exclude, $Recurse, [switch]$Force) {
Get-ChildItem @PSBoundParameters -ErrorAction SilentlyContinue | where { ! $_.PSIsContainer }
}

working_directory.zip

Hello, hope all is well. I was testing PsFCIV on some real-world examples, My Documents containing 16,535 Files, 1,050 Folders. All seemed OK at first, "Total files processed" and "Total new added files" both indicated 16,535, and the Verbose output also printed that many lines.

But when I passed DB.xml through a program to count <FILE_ENTRY> tags, 16,534 Files came back. I traced the issue to one file name containing something that seems like a hyphen but is not. Attached is a smaller data set containing the problem file name. As a matter of fact, the github form isn't even displaying the character as I try to paste it, so check the attached .zip

PS C:\test\data_set> Start-PsFCIV -Path .\ -Include * -XML "C:\test\DB.xml" -HashAlgorithm MD5 -Recurse -Verbose
VERBOSE: Perform file 'C:\test\data_set\Barn Owl.txt' checking.
VERBOSE: Perform file 'C:\test\data_set\Barn_Owl.txt' checking.
VERBOSE: Perform file 'C:\test\data_set\Barn­_Owl.txt' checking.

Total files processed : 3
Total new added files : 3
Total good files : 0
Total bad files : 0
Total unknown status files : 0
Total missing files : 0
Total locked files : 0

Total : 3
New : 3
Ok : 0
Bad : 0
Missed : 0
Locked : 0
Unknown : 0
Deleted : 0

Notice only Two <FILE_ENTRY>'s are actually written to DB.xml

<FCIV> <FILE_ENTRY> <name>Barn Owl.txt</name> <Size>12</Size> <TimeStamp>01/18/2022 07:49:28</TimeStamp> <MD5>HIkS05fEaGZwazcMovokbQ==</MD5> </FILE_ENTRY> <FILE_ENTRY> <name>Barn_Owl.txt</name> <Size>16</Size> <TimeStamp>01/18/2022 07:49:41</TimeStamp> <MD5>NYX+pTjx8Ee6bzjBC+hyLQ==</MD5> </FILE_ENTRY> </FCIV>

Thanks again for making this program available, it is still very useful even with current issues.

TEST.zip

The code for actually writing out the XML DB is commented out in the code file for Start-PsFCIV. It looks like someone started working on removing the $host item for that section but did not complete it and lines 450-455 have been commented out.
PSPKI/Client/Start-PsFCIV.ps1

Adding those lines back in allowed the initial DB to be written again.

Hello,
i had different problems with your cmdlet:
when the startfolder has no files and only a subfolder it seems that it doesnt work. (also with -Recurse)
when i specify the xml file in c:\ folder (c:\test.xml) its not working. (with admin rights).
often PsFCIV show no results, it show nothing.

Please help.
Thank you

Not sure how you possibly tested this before publishing in the gallery...no matter what directory I give it after I hash my temp folder, it adds no new files doesn't honor recurse simply doesn't work. Would love this if it worked!

When initially creating the XML database, this program seems to not create the XML database if there is no file in the PATH directory even if -Recurse is selected and there are folders within the directory tree that do have files. Adding even a zero length file to the PATH folder makes it work. This problem does not occur for folders deeper in the directory tree.

Hello Mr. Podans,

I am very interested in migrating away from Microsoft's FCIV and over to your PsFCIV due to its support of more secure hashing algorithms, i.e. SHA256.

I was initially testing PsFCIV on a data set of 130 megabytes, 267 Files, 89 Folders. The DB.xml generated successfully but a second run for Checking the files was crashing. In this case there were Two zero byte files hiding in different locations in the data, not even next to one another.

I'm able to reproduce the crash on a small scaled down example of just 3 text files, one having text, the other two having zero bytes. Below are the commands and output. Attached is tmp.zip containing the samller example files and db.xml

Thank You!

PS C:> Start-PsFCIV -Path "C:\tmp" -XML "C:\Users\Owner\Desktop\DB.xml" -HashAlgorithm SHA1 -Recurse -Verbose

VERBOSE: Perform file 'C:\tmp\file1_does_have_text_inside.txt' checking.
VERBOSE: Perform file 'C:\tmp\file2_ZERO_bytes.txt' checking.
VERBOSE: Perform file 'C:\tmp\file3_ZERO_bytes.txt' checking.

Total files processed : 3
Total new added files : 3
Total good files : 0
Total bad files : 0
Total unknown status files : 0
Total missing files : 0
Total locked files : 0

Total : 3
New : 3
Ok : 0
Bad : 0
Missed : 0
Locked : 0
Unknown : 0
Deleted : 0

PS C:> Start-PsFCIV -Path "C:\tmp" -XML "C:\Users\Owner\Desktop\DB.xml" -HashAlgorithm SHA1 -Recurse -Verbose

VERBOSE: Perform file 'C:\tmp\file1_does_have_text_inside.txt C:\tmp\file2_ZERO_bytes.txt C:\tmp\file3_ZERO_bytes.txt'
checking.
VERBOSE: File file1_does_have_text_inside.txt file2_ZERO_bytes.txt file3_ZERO_bytes.txt is locked. Skipping this file..

PS C:>

tmp.zip

I have a solution that may resolve this, but here is the issue first from a test run:

PS C:\working_directory\Documents> Start-PsFCIV -Path .\ -XML "C:\working_directory\DB.xml" -HashAlgorithm SHA256 -Recur
se -Verbose -Debug -Action Rename
DEBUG: Native PsFCIV mode ON
VERBOSE: Perform file 'Directory1\Another Test File.txt' checking.
DEBUG: Selected hash name : SHA256
DEBUG: Selected hash value: 6978413F9D61E2AA69A31CB7750B80B7FFF5C9F9DF5336A1D0612EFB6C54A58A
DEBUG: File hash: 6978413F9D61E2AA69A31CB7750B80B7FFF5C9F9DF5336A1D0612EFB6C54A58A
VERBOSE: File 'Another Test File.txt' is ok.
VERBOSE: Perform file 'a text file.txt' checking.
DEBUG: Selected hash name : SHA256
DEBUG: Selected hash value: 837CCB607E312B170FAC7383D7CCFD61FA5072793F19A25E75FBACB56539B86B
DEBUG: File hash: 837CCB607E312B170FAC7383D7CCFD61FA5072793F19A25E75FBACB56539B86B
VERBOSE: File 'a text file.txt' is ok.
VERBOSE: Perform file '[Torgo].gif' checking.
DEBUG: Selected hash name : SHA256
DEBUG: Selected hash value: 0201C1264D23B9EA3183CF5AFA953CA470A3A786F79B039C84C0AB05FEF171AC
DEBUG: File '[Torgo].gif' failed hash verification.
                Expected hash: @{HashName=System.String[];
HashValue=0201C1264D23B9EA3183CF5AFA953CA470A3A786F79B039C84C0AB05FEF171AC}.HashValue
                Actual hash: 7C462C54798E9CA1A737A91E7F4B120460EF8754D00B7F7A1FFE09F395F1AFBC
Rename-Item : Cannot rename because item at 'C:\working_directory\Documents\[Torgo].gif' does not exist.
At C:\Program Files\WindowsPowerShell\Modules\PsFCIV\1.1\PsFCIV.psm1:95 char:19
+         "Rename" {Rename-Item $file $($file.FullName + ".bad")}
+                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [Rename-Item], PSInvalidOperationException
    + FullyQualifiedErrorId : InvalidOperation,Microsoft.PowerShell.Commands.RenameItemCommand

-----------------------------------
Total files processed      : 3
Total good files           : 2
Total bad files            : 1
Total unknown status files : 0
Total missing files        : 0
Total locked files         : 0
-----------------------------------


Total   : 3
New     : 0
Ok      : 2
Bad     : 1
Missed  : 0
Locked  : 0
Unknown : 0
Deleted : 0

It would seem that both the Rename-Item and Remove-Item cmdlets require the -LiteralPath paramenter to be able to interpret square brackets or other special characters.

I would suggest modifying this function:

possible changes below. I have also included two calls to Write-Verbose since I feel better knowing when a file is being modified.

# process -Action parameter to perform an action against bad file (if actual file properties do not match the record in XML).
function __takeAction ($file, $Action) {
    switch ($Action) {
        "Rename" {
            Write-Verbose "Action: Rename file '$file' to '$($file.FullName + ".bad")'"
            Rename-Item -LiteralPath $file -NewName $($file.FullName + ".bad")
         }
        "Delete" {
            Write-Verbose "Action: Delete file '$file'"
            Remove-Item -LiteralPath $file -Force
         }
    }
}

Would it be possible to push version 1.2 to the powershell gallery later? It makes managing versions and re-deployment a bit easier. Thank You!

working_directory.zip