Robocopy Broken

[RichardMeadowsTC]

Prerequisites

• Write a descriptive title.
• Make sure you are able to repro it on the latest released version
• Search the existing issues.
• Refer to the FAQ.
• Refer to Differences between Windows PowerShell 5.1 and PowerShell.

Steps to reproduce

PowerShell 7.4.0 Preview 1
Can no longer use robocopy.exe from powershell. As robocopy.exe returns status code other than 0 for successful copies.

Repro:

1. robocopy .\PowerShell-7.4.0-preview.1-win-x64\ .\temp\

Expect: Runs without creating an error object.

Actual: Creates Error Object
NativeCommandExitException: Program "Robocopy.exe" ended with non-zero exit code: 1.

Issue: This will break all of our existing automation. We use robocopy everywhere. And our scripts all use ErrorAction = Stop

Expected behavior

robocopy .\PowerShell-7.4.0-preview.1-win-x64\ .\temp\

# Files are copied. No error is raised when all files are copied.

Actual behavior

robocopy .\PowerShell-7.4.0-preview.1-win-x64\ .\temp\
 # An error is raised even though robocopy detected no errors.
 # 1 indicates all files copied.

Error details

PS > Get-Error

Exception             :
    Type        : System.Management.Automation.NativeCommandExitException
    Path        : C:\Windows\system32\Robocopy.exe
    ExitCode    : 1
    ProcessId   : 17460
    ErrorRecord :
        Exception             :
            Type    : System.Management.Automation.ParentContainsErrorRecordException
            Message : Program "Robocopy.exe" ended with non-zero exit code: 1.
            HResult : -2146233087
        CategoryInfo          : NotSpecified: (:) [], ParentContainsErrorRecordException
        FullyQualifiedErrorId : ProgramExitedWithNonZeroCode
    Message     : Program "Robocopy.exe" ended with non-zero exit code: 1.
    HResult     : -2146233087
TargetObject          : C:\Windows\system32\Robocopy.exe
CategoryInfo          : NotSpecified: (C:\Windows\system32\Robocopy.exe:String) [], NativeCommandExitException
FullyQualifiedErrorId : ProgramExitedWithNonZeroCode

Environment data

$PSVersionTable

Name                           Value
----                           -----
PSVersion                      7.4.0-preview.1
PSEdition                      Core
GitCommitId                    7.4.0-preview.1
OS                             Microsoft Windows 10.0.22621
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Visuals

Steps.zip

[iSazonov]

What is in robocopy log?

[mklement0]

I know you're aware of at least some of this, but let me summarize:

By convention, CLIs use nonzero exit codes to signal error conditions, and that is what the - experimental, at this point - PSNativeCommandUseErrorActionPreference feature and its $PSNativeCommandUseErrorActionPreference preference variable rely on.

Robocopy does not follow this convention and uses nonzero codes as success error codes too (which communicate additional information) - see https://ss64.com/nt/robocopy-exit.html

Robocopy is a prominent example of such an "unconvential" CLI, but there's no telling how many there are, so it's impossible to anticipate all possible CLIs that require special treatment (let alone maintain tables of what nonzero codes, specifically, indicate success), so I don't think maintaining a list of exceptions is practical.

Pragmatically speaking:

• I'm assuming - perhaps @SteveL-MSFT can confirm - that setting $PSNativeCommandUseErrorActionPreference to $true will be opt-in if and when the experimental PSNativeCommandUseErrorActionPreference feature will be made official. Somewhat curiously, in the 7.4 preview versions it is opt-out. If the stable versions didn't make this opt-in, it would amount to a serious breaking change.

• Thus, in stable versions, only users who explicitly set $PSNativeCommandUseErrorActionPreference = $true will be affected (for the preview versions, you can simple disable the experimental feature or run $PSNativeCommandUseErrorActionPreference = $false)

• If the feature is deliberately enabled, a - somewhat cumbersome ad-hoc workaround is to call via a child scope:

  & { $PSNativeCommandUseErrorActionPreference = $false; robocopy .\PowerShell-7.4.0-preview.1-win-x64\ .\temp\ }

  • Note that while 2>$null would silence PowerShell's error, it would also silence the CLI's standard error output, which is undesirable. More importantly, 2>$null wouldn't prevent a script-terminating error if $ErrorActionPreference = 'Stop' were in effect.
    (Tangentially related: Native error handling: $PSNativeCommandUseErrorActionPreference = $true unexpectedly causes non-terminating, not statement-terminating errors #18368)

Robocopy Exit Codes - Windows CMD - SS64.com

[RichardMeadowsTC]

@mklement0 Thank you for the explanation. A another way to prevent the failure would be to shell out to cmd.exe and redirect stderr to stdout. I had to do that 5 to 10 years ago. When scripting in TFS. Ironically there was a tfs exe which wrote things it thought were very important to stderr.

To do that with robocopy will be very complicated if we need to detect real errors.

[mklement0]

@RichardMeadowsTC, shelling out to cmd /c will only help you if place an explicit & exit 0 at the end - by contrast, it is fine not to redirect stderr to stdout, as it is only the exit code that matters.
It is not uncommon for CLIs to write anything that's not data to stderr, which may or may not be error messages.

However, If you shell out to cmd /c, you'll lose Robocopy's exit code, which you need if you want to robustly detect failure .

# OK, but you Robocopy's exit code is lost.
cmd /c 'robocopy .\PowerShell-7.4.0-preview.1-win-x64\ .\temp\ & exit 0'

Note that the & { $PSNativeCommandUseErrorActionPreference = $false; ... } workaround does preserve the exit code.

[RichardMeadowsTC]

I would need to run some prototypes. I am not so sure about the cmd.exe not returning the exit code of the process which was ran. I do know powershell.exe tends to return 0 only. Anyway it is all overly complicated and would require a special wrapper for EXEs.

[mklement0]

I am not so sure about the cmd.exe not returning the exit code of the process which was run

It does - that's precisely why you need & exit 0 to prevent the nonzero Robocopy exit code from being passed through to PowerShell and again triggering the PowerShell-side error message (but, as noted, & exit 0 comes at the expense of losing the specific Robocopy exit code).

A simple wrapper function that avoids the PowerShell error while preserving the exit code:

function Invoke-WithExitCodeIgnored {
  param([Parameter(Mandatory, Position=0)] [scriptblock] $ScriptBlock)
  $PSNativeCommandUseErrorActionPreference = $false
  & $ScriptBlock
}

# Sample call
Invoke-WithExitCodeIgnored { robocopy .\PowerShell-7.4.0-preview.1-win-x64\ .\temp\ }
# Now you can examine $LASTEXITCODE

[RichardMeadowsTC]

There is no need for Exit 0 in cmd script. Only ever needed that when running ps1 script from cmd scripts. Not the other way around. How well last exit code is sent from cmd.exe I need to check.

[mklement0]

I've explained why & exit 0 (a) is necessary if you were to use cmd /c in order to avoid the problem at hand and (b) that doing so would be suboptimal, because Robocopy's specific exit code would be lost in the process - and, by implication, (c) why this attempt at a workaround is best avoided. Finally, (d) I've provided a workaround that is generally viable.
If you have specific feedback about these points, please provide it.

[RichardMeadowsTC]

I don't think you understand. I working on my tuck today. Will get back to this in the morning. But there a thing we can fix right now. The exit code just set it to what robocopy sets. But I think that is not even necessary. As I have never seen that needed in cmd scripts. Only seen that needed in powershell scripts.

All that said, I think what you do have for properly working solution is very good.

[Andrew74L]

A simple wrapper function that avoids the PowerShell error while preserving the exit code:

function Invoke-WithExitCodeIgnored {
  param([Parameter(Mandatory, Position=0)] [scriptblock] $ScriptBlock)
  $PSNativeCommandUseErrorActionPreference = $false
  & $ScriptBlock
}

# Sample call
Invoke-WithExitCodeIgnored { robocopy .\PowerShell-7.4.0-preview.1-win-x64\ .\temp\ }
# Now you can examine $LASTEXITCODE

What about a built-in variable to complement $PSNativeCommandUseErrorActionPreference, such as:

$ZEROEXITCODE

The variable resets to zero after a native command is executed.
Powershell calculates $LASTEXITCODE just prior to processing any errors.
Ex:

$ZEROEXITCODE = 1
robocopy .\PowerShell-7.4.0-preview.1-win-x64\ .\temp\
# $LASTEXITCODE = %ERRORLEVEL% - $ZEROEXITCODE
$LASTEXITCODE
0

This would maintain the benefit of $PSNativeCommandUseErrorActionPreference = $true

[RichardMeadowsTC]

All we have to do is set $PSNativeCommandUseErrorActionPreference = $false
Thankfully, it isn't raising error when the exe simply writes to stderr. Like TFS Workflows were doing a few years ago.

I made an EXE to verify all this. And sample scripts. https://github.com/RichardMeadowsTC/PowerShellCmdInvoke

This is a side note. The cmd.exe will return current error level. This is different than powershell which has always tried to hide script result from app which invokes powershell.exe or pwsh.exe. I can guess they aren't changing it to prevent existing code from breaking. But here clearly lots of code is going to be broken. Much more than was broken with Pwsh.exe version 7.3.

GitHub

GitHub - RichardMeadowsTC/PowerShellCmdInvoke: Place for samples and possibly new wrappers to deal with PowerShell 7.4 raising errors when exes return non-zero.

Place for samples and possibly new wrappers to deal with PowerShell 7.4 raising errors when exes return non-zero. - GitHub - RichardMeadowsTC/PowerShellCmdInvoke: Place for samples and possibly new...

[Andrew74L]

All we have to do is set $PSNativeCommandUseErrorActionPreference = $false

Then set it back to $true after running the command. If not, why have the feature at all?
I think that's fine as a workaround solution, but it's not ideal.

[RichardMeadowsTC]

The reason for not setting it back is that it never should have been enabled. Old scripts that we have are well tested and check $LASTEXITCODE where necessary.

[mklement0]

Please see the following proposal for a possible solution:

• Introduce a new cmdlet for calls to native (external) programs #18991

[SteveL-MSFT]

I don't like the idea of an "ignore list" for the experimental feature, but robocopy isn't going to change and there are also other known tools that don't follow the non-zero exit is error convention. choice.exe is one where it actually makes sense, but currently would also cause a problem.

[Andrew74L]

For tools that don't follow the non-zero exit is error convention, there is the option to set (by default or otherwise) $PSNativeCommandUseErrorActionPreference = $false and continue to use $LASTEXITCODE for error handling. The other option is to employ some method for 'tweaking' exit codes so as to prevent 'false positives' from triggering Powershell's error detection, thereby maintaining consistency of error handling.
I would suggest that a good reason for favoring the second option is that eventually, all CLI executables that still have at least moderate usage, are going to be 'consumed' by Crescendo-based modules, and the authors of those modules can handle whatever 'tweaks' are required to produce the desired response. So I say make $PSNativeCommandUseErrorActionPreference = $true the default setting, and provide a mechanism for altering non-zero, non-error exit codes, that most Powershell users will rarely if ever need to use.

[mklement0]

@SteveL-MSFT, please confirm that $PSNativeCommandUseErrorActionPreference will default to $false in release versions - not doing so would be a massively breaking change, not least because an effective $ErrorActionPreference = 'Stop' setting would then start breaking scripts with external-program calls that happen to return non-zero exit codes (which are currently ignored).

Proceeding on that assumption:

• Setting $PSNativeCommandUseErrorActionPreference = $true is therefore a deliberate act and the user should be assumed to understand the consequences: any non-zero exit code reported reports a non-terminating error; as such, it can be controlled via $ErrorActionPreference, which, however, is just as cumbersome as temporarily setting $PSNativeCommandUseErrorActionPreference = $true

  • What's more, the error shouldn't statement-terminating, not non-terminating, as argued in Native error handling: $PSNativeCommandUseErrorActionPreference = $true unexpectedly causes non-terminating, not statement-terminating errors #18368
  • Doing so would enable a try / catch solution - see below.
  • Those not setting $PSNativeCommandUseErrorActionPreference = $true could continue to happily ignore exit codes or, preferably, always test $LASTEXITCODE afterwards, CLI-appropriately.

• Building exception list for specific CLIs along with the specific non-zero exit codes that constitute success would mean playing an everlasting game of catch-up, as new non-conformant CLIs are discovered; it would lead to the same mess as with $PSNativeCommandArgumentPassing's Windows mode - see Parameter invalid when it contains a colon. Works prior to 7.3.0. #18554 (comment)

  • The only proper place to implement exceptions would be in - invariably _CLI-_specific - Crescendo cmdlets, though it looks like there's currently no mechanism to define success exit codes - so that may be worth adding (the JSON schema could support defining success exit codes, and the generated code could test for these).

• Not everyone will or should be expected to be using Crescendo modules (if available at all), so the best, exception-free solution is to make per-command overrides of $PSNativeCommandUseErrorActionPreference as simple as possible:

  • A try / catch approach is reasonable concise, but requires fixing Native error handling: $PSNativeCommandUseErrorActionPreference = $true unexpectedly causes non-terminating, not statement-terminating errors #18368

    # !! Does NOT work as of v7.4.0-preview.1 
    # Ignore the error, and examine $LASTEXITCODE afterwards.
    try { robocopy .\PowerShell-7.4.0-preview.1-win-x64\ .\temp\ } catch { }

  • The proposed Invoke-NativeCommand - see Introduce a new cmdlet for calls to native (external) programs #18991 - could help too; assuming inc as its built-in alias:

     # !! No such cmdlet yet, as of v7.4.0-preview.1
     # Ignore the error, and examine $LASTEXITCODE afterwards.
     inc -IgnoreExitCode { robocopy .\PowerShell-7.4.0-preview.1-win-x64\ .\temp\ }

  • Arguably, analogous to bash's set -e, no error should be triggered for external-program calls in the non-final link of a pipeline chain, i.e with && and || - this is currently not the case. If this worked, a concise - albeit slightly obscure - solution would be:

    # !! Does NOT work as of v7.4.0-preview.1       
    # Ignore the error, and examine $LASTEXITCODE afterwards.
    # @() is an effective no-op.
    robocopy .\PowerShell-7.4.0-preview.1-win-x64\ .\temp\ || @()

[Andrew74L]

The following assumes $PSNativeCommandUseErrorActionPreference = $true is the default.

Assume "$PSHOME\ZeroErrorCodes.json" store the minimum magnitude non-error exit codes (with sign), for CLI executables that do not follow the non-zero exit code implies error convention. Example content:

{
  "negExitcodes.exe": -1,
  "Robocopy.exe": 1,
  "choice.exe": 0
}

At Powershell startup:

$ZeroErrorCode = 0
$ZeroErrorCodeObj = Get-Content "$PSHOME\ZeroErrorCodes.json" | ConvertFrom-Json

When user inputs a native command, Powershell does this processing:

# get command line into $cmdline, then get exe name
$executable = (Get-Command $cmdline.Split()[0]).Name
if ($null -eq $ZeroErrorCodeObj.$executable) {
    # run command line
    $exitcode = if ($ZeroErrorCode -gt 0) {
        [math]::Max(0, $LASTEXITCODE - $ZeroErrorCode)
    } else {
        [math]::Min(0, $LASTEXITCODE - $ZeroErrorCode)
    }
    $ZeroErrorCode = 0
    # process error based on $exitcode
} elseif ($ZeroErrorCodeObj.$executable -eq 0) {
    $PSNativeCommandUseErrorActionPreference = $false
    # run command line
    $PSNativeCommandUseErrorActionPreference = $true
    # user can interpret outcome by examining $LASTEXITCODE
} else {
    # run command line
    $exitcode = if ($ZeroErrorCodeObj.$executable -gt 0) {
        [math]::Max(0, $LASTEXITCODE - $ZeroErrorCodeObj.$executable)
    } else {
        [math]::Min(0, $LASTEXITCODE - $ZeroErrorCodeObj.$executable)
    }
    # process error based on $exitcode
}

Consider these scenarios.

User runs Robocopy which copies zero files without error:

$exitcode = [math]::Max(0, $LASTEXITCODE - $ZeroErrorCodeObj.$executable)
$exitcode = [math]::Max(0, 0 - 1) # zero

User runs Robocopy which copies files without error:

$exitcode = [math]::Max(0, $LASTEXITCODE - $ZeroErrorCodeObj.$executable)
$exitcode = [math]::Max(0, 1 - 1) # zero

User runs Robocopy which returns error:

$exitcode = [math]::Max(0, $LASTEXITCODE - $ZeroErrorCodeObj.$executable)
$exitcode = [math]::Max(0, 2 - 1) # one. PS error triggered.

CLI whose json value is zero:

$PSNativeCommandUseErrorActionPreference = $false
> choice /C YN /M "Do you like this proposal"
$PSNativeCommandUseErrorActionPreference = $true
# exitcode = 0, $LASTEXITCODE = 1|2

An exe not in json file:

> unknown.exe arg1 arg2 # $LASTEXITCODE = %ERRORLEVEL% = 1
$exitcode = [math]::Max(0, $LASTEXITCODE - $ZeroErrorCode)
$ZeroErrorCode = 0
# PS detects error

Same but user sets ZeroErrorCode immediately prior to execution:

> $ZeroErrorCode = 1
> unknown.exe arg1 arg2 # $LASTEXITCODE = %ERRORLEVEL% = 1
$exitcode = [math]::Max(0, $LASTEXITCODE - $ZeroErrorCode)
$ZeroErrorCode = 0
# PS does NOT detect error

[SteveL-MSFT]

Discussion in the team we agree that this feature is working as expected and we don't want to special case robocopy. Instead, I've opened a doc bug to ensure we have a good example for cases like robocopy to turn it off and on.

[mklement0]

To offer an alternative, let me add solutions that already work / will work with $PSNativeCommandUseErrorActionPreference set to $false.
In other words: for general per-native-program-call error handling.

# With a conformant CLI
# In a script, use `exit` instead of `throw`
cmd /c 'echo hi && exit 5' || $(throw $LASTEXITCODE)

# With special handling for Robocopy
# Only treat exit codes > 3 as errors.
robocopy .\PowerShell-7.4.0-preview.1-win-x64\ .\temp\ || $(if ($LASTEXITCODE -gt 3) { throw $LASTEXITCODE })

Note the unfortunate need to enclose even throw and exit in $(...), otherwise they'll be treated as (usually non-existent command names) - see #10967 for background.

[jazzdelightsme]

Discussion in the team we agree that this feature is working as expected and we don't want to special case robocopy. Instead, I've opened a doc bug to ensure we have a good example for cases like robocopy to turn it off and on.

Hang on... I didn't see a response to mklement's request for confirmation that we would still have existing behavior in release versions. Your statement, @SteveL-MSFT almost sounds like "no, we are turning this on, and doc'ing it better". If that is the case, please allow me to respectfully scream "bloody murder". I've already gotten bug reports from users of my modules for robocopy, and git, and I can't even remember what else. I assumed this was so clearly such a massively breaking change that it surely wouldn't stand...

[SteveL-MSFT]

@jazzdelightsme I believe it was mentioned in a separate issue that $PSNativeCommandUseErrorActionPreference is default to $false and only set to $true in 7.4 previews to get feedback.

[mklement0]

Glad to hear it, @SteveL-MSFT, but it's somewhat disheartening that it took three separate pleas to get you to address this aspect.

[ghost]

This issue has been marked as by-design and has not had any activity for 1 day. It has been closed for housekeeping purposes.

[sba923]

@SteveL-MSFT Is there an explanation for the fact that the NativeCommandExitException disappears if the native command's output is discarded using 2>&1 | Out-Null? (I'm testing with 7.4.0-preview.4)

[mklement0]

@sba923, the current behavior is a bit counter-intuitive:

• A 2> redirection also swallows the NativeCommandExitException, because the latter is implemented as a non-terminating error and is therefore treated the same as the external program's stderr output.

  • That a non-terminating error is emitted is also problematic for other reasons, as discussed in Native error handling: $PSNativeCommandUseErrorActionPreference = $true unexpectedly causes non-terminating, not statement-terminating errors #18368, and if it were converted to a statement-terminating one, it would no longer be swallowed.

• However, if you have $ErrorActionPreference = 'Stop' in effect, the script aborts (as intended) and the NativeCommandException surfaces (but not the native program's stderr output, as intended).

[sba923]

@sba923, the current behavior is a bit counter-intuitive:

• A 2> redirection also swallows the NativeCommandExitException, because the latter is implemented as a non-terminating error and is therefore treated the same as the external program's stderr output.

  • That a non-terminating error is emitted is also problematic for other reasons, as discussed in Native error handling: $PSNativeCommandUseErrorActionPreference = $true unexpectedly causes non-terminating, not statement-terminating errors #18368, and if it were converted to a statement-terminating one, it would no longer be swallowed.

• However, if you have $ErrorActionPreference = 'Stop' in effect, the script aborts (as intended) and the NativeCommandException surfaces (but not the native program's stderr output, as intended).

This is indeed far from intuitive...