ExchMailboxDatabase resource and WaitForFirstDAGMember script issues

[kevindd992002]

Problem description

We've been using the xExchangeDsc (and now ExchangeDsc) module for years now for our Exchange 2016 servers without any issues. We're using State Configuration in Azure Automation as our pull server. We deploy updated DSC conf files once, remove them as nodes in Azure Automation, and just let the drift check take care of any drift on the onprem servers.

Recently, we saw DSC errors in both the first DAG member and additional DAG member/s in both our lab and prod environments (same exact issue). Let's concentrate on the lab for now as I can do whatever there. There are only two Exchange servers there with the one DAG, one mailbox database (two copies, one on each node). So pretty straightforward and basic setup.

On the first dag member, I get this error:

Database must have only a single copy for the DB path or log path to be moved

I checked the psm1 script being used y the ExchMailboxDatabase resource and saw this block for when a database exists already:

# First check if a DB or log move is required
        if (($PSBoundParameters.ContainsKey('EdbFilePath') -and (Compare-StringToString -String1 $db.EdbFilePath.PathName -String2 $EdbFilePath -IgnoreCase) -eq $false) -or
            ($PSBoundParameters.ContainsKey('LogFolderPath') -and (Compare-StringToString -String1 $db.LogFolderPath.PathName -String2 $LogFolderPath -IgnoreCase) -eq $false))
        {
            if ($db.DatabaseCopies.Count -le 1)
            {
                Write-Verbose -Message 'Moving database and/or log path'

                Move-DatabaseOrLogPath @PSBoundParameters
            }
            else
            {
                throw 'Database must have only a single copy for the DB path or log path to be moved'
            }
        }

Here's a snippet of what I have in the DSC conf file:

$dagName = 'USCHIZWDAG4002'
$dbName = "$($dagName)-DB1"

# Create primary database
ExchMailboxDatabase "MDB_$dbName"
{
    Name = $dbName
    Credential = $Credential
    EdbFilePath = "C:\ExchangeDatabases\$($dbName)\$($dbName).db\$($dbName).edb"
    LogFolderPath = "C:\ExchangeDatabases\$($dbName)\$($dbName).log"
    Server = $server.Name
    CircularLoggingEnabled = $true
    DatabaseCopyCount = 2
    IssueWarningQuota = '1946MB'
    ProhibitSendQuota = '2048MB'
    ProhibitSendReceiveQuota = '2253MB'
    AllowServiceRestart = $true
    DependsOn = '[ExchMailboxServer]MailboxServer'
}

And here's the EdbFilePath and LogFolderPath of the actual database in the Exchange server:

[PS] C:\Windows\system32>(Get-MailboxDatabase).EdbFilePath.PathName
C:\ExchangeDatabases\USCHIZWDAG4002-DB1\USCHIZWDAG4002-DB1.db\USCHIZWDAG4002-DB1.edb
[PS] C:\Windows\system32>(Get-MailboxDatabase).LogFolderPath.PathName
C:\ExchangeDatabases\USCHIZWDAG4002-DB1\USCHIZWDAG4002-DB1.log

As you can see, this SHOULD NOT require a move yet the resource still wants to move it (which is why it entered the DB/log move block). I can't really see anything wrong anywhere so I'm asking for support here.

Now on the second/additional DAG member, I get this error:

Database Availability Group does not have at least one member after waiting the specified amount of time.

And this is the WaitForFirstDAGMember block:

# Wait for DAG to have at least one member before adding additional members (to avoid cluster creation race condition issues)
Script WaitForFirstDAGMember
{
    GetScript = {
        # Establish remote PowerShell session
        Get-RemoteExchangeSession -Credential $Credential -CommandsToLoad 'Get-DatabaseAvailabilityGroup' -Verbose:$VerbosePreference

        $dagProperties = Get-DatabaseAvailabilityGroup -Identity $dagName
        $result = @{'CurrentDAGMembers' = $dagProperties.Servers.Name}
        return $result
    }

    TestScript = {
        # Establish remote PowerShell session
        Get-RemoteExchangeSession -Credential $Credential -CommandsToLoad 'Get-DatabaseAvailabilityGroup' -Verbose:$VerbosePreference
        
        $dagProperties = Get-DatabaseAvailabilityGroup -Identity $dagName
        if ($dagProperties.Servers.Name.Count -gt 0)
        {
            return $true
        }
        else
        {
            return $false
        }
    }

    SetScript = {
        # Establish remote PowerShell session
        Get-RemoteExchangeSession -Credential $Credential -CommandsToLoad 'Get-DatabaseAvailabilityGroup' -Verbose:$VerbosePreference
        
        $foundFirstDAGMember = $false

        for ($i = 0; $i -lt 5; $i++)
        {
            $dagProperties = Get-DatabaseAvailabilityGroup -Identity $dagName

            if ($dagProperties.Servers.Name.Count -eq 0)
            {
                Write-Warning "DAG $dagName object has no members yet. Sleeping for 60 seconds."
                Start-Sleep -Seconds 60
            }
            else
            {
                $foundFirstDAGMember = $true
                break
            }
        }

        if (!$foundFirstDAGMember)
        {
            throw 'Database Availability Group does not have at least one member after waiting the specified amount of time.'
        }
    }
    DependsOn = '[ExchWaitForDAG]WaitForDAG'
}

Manually getting the DAG members count, I get this:

[PS] C:\Windows\system32>$dagProperties = Get-DatabaseAvailabilityGroup -Identity "USCHIZWDAG4002"
[PS] C:\Windows\system32>$dagProperties.Servers.Name.Count
2

So again, the script should see that the DAG has "at least one member" and should continue. Any clues?

Verbose logs

First DAG member:

VERBOSE: [USCHIZWXCH4101]: LCM:  [ Start  Resource ]  [[ExchMailboxDatabase]MDB_USCHIZWDAG4002-DB1]
VERBOSE: [USCHIZWXCH4101]: LCM:  [ Start  Test     ]  [[ExchMailboxDatabase]MDB_USCHIZWDAG4002-DB1]
VERBOSE: [USCHIZWXCH4101]:                            [[ExchMailboxDatabase]MDB_USCHIZWDAG4002-DB1] Importing the module DSC_ExchMailboxDatabase in force mode.
VERBOSE: [USCHIZWXCH4101]:                            [[ExchMailboxDatabase]MDB_USCHIZWDAG4002-DB1] Entering function 'Test-TargetResource'. Notable parameters: Name =
'USCHIZWDAG4002-DB1'
VERBOSE: [USCHIZWXCH4101]:                            [[ExchMailboxDatabase]MDB_USCHIZWDAG4002-DB1] Reusing existing Remote PowerShell Session to Exchange
VERBOSE: [USCHIZWXCH4101]:                            [[ExchMailboxDatabase]MDB_USCHIZWDAG4002-DB1] Importing the ExchangeDsc Remote PowerShell Module.
VERBOSE: [USCHIZWXCH4101]:                            [[ExchMailboxDatabase]MDB_USCHIZWDAG4002-DB1] Removing the imported "Add-DatabaseAvailabilityGroupServer" function.
VERBOSE: [USCHIZWXCH4101]:                            [[ExchMailboxDatabase]MDB_USCHIZWDAG4002-DB1] Loading module from path
'C:\Windows\TEMP\DSCExchangeModule\DSCExchangeModule.psm1'.
VERBOSE: [USCHIZWXCH4101]:                            [[ExchMailboxDatabase]MDB_USCHIZWDAG4002-DB1] Importing function 'Get-MailboxDatabase'.
VERBOSE: [USCHIZWXCH4101]:                            [[ExchMailboxDatabase]MDB_USCHIZWDAG4002-DB1] Importing function 'Get-Recipient'.
VERBOSE: [USCHIZWXCH4101]:                            [[ExchMailboxDatabase]MDB_USCHIZWDAG4002-DB1] Importing function 'Set-ADServerSettings'.
VERBOSE: [USCHIZWXCH4101]:                            [[ExchMailboxDatabase]MDB_USCHIZWDAG4002-DB1] Invalid setting 'EdbFilePath'. Expected value:
'C:\ExchangeDatabases\USCHIZWDAG4002-DB1\USCHIZWDAG4002-DB1.db\USCHIZWDAG4002-DB1.edb'. Actual value: ''
VERBOSE: [USCHIZWXCH4101]:                            [[ExchMailboxDatabase]MDB_USCHIZWDAG4002-DB1] Invalid setting 'LogFolderPath'. Expected value:
'C:\ExchangeDatabases\USCHIZWDAG4002-DB1\USCHIZWDAG4002-DB1.log'. Actual value: ''
VERBOSE: [USCHIZWXCH4101]: LCM:  [ End    Test     ]  [[ExchMailboxDatabase]MDB_USCHIZWDAG4002-DB1]  in 0.6920 seconds.
VERBOSE: [USCHIZWXCH4101]: LCM:  [ Start  Set      ]  [[ExchMailboxDatabase]MDB_USCHIZWDAG4002-DB1]
VERBOSE: [USCHIZWXCH4101]:                            [[ExchMailboxDatabase]MDB_USCHIZWDAG4002-DB1] Importing the module DSC_ExchMailboxDatabase in force mode.
VERBOSE: [USCHIZWXCH4101]:                            [[ExchMailboxDatabase]MDB_USCHIZWDAG4002-DB1] Entering function 'Set-TargetResource'. Notable parameters: Name =
'USCHIZWDAG4002-DB1'
VERBOSE: [USCHIZWXCH4101]:                            [[ExchMailboxDatabase]MDB_USCHIZWDAG4002-DB1] Reusing existing Remote PowerShell Session to Exchange
VERBOSE: [USCHIZWXCH4101]:                            [[ExchMailboxDatabase]MDB_USCHIZWDAG4002-DB1] Importing the ExchangeDsc Remote PowerShell Module.
VERBOSE: [USCHIZWXCH4101]:                            [[ExchMailboxDatabase]MDB_USCHIZWDAG4002-DB1] Removing the imported "Set-ADServerSettings" function.
VERBOSE: [USCHIZWXCH4101]:                            [[ExchMailboxDatabase]MDB_USCHIZWDAG4002-DB1] Removing the imported "Get-Recipient" function.
VERBOSE: [USCHIZWXCH4101]:                            [[ExchMailboxDatabase]MDB_USCHIZWDAG4002-DB1] Removing the imported "Get-MailboxDatabase" function.
VERBOSE: [USCHIZWXCH4101]:                            [[ExchMailboxDatabase]MDB_USCHIZWDAG4002-DB1] Loading module from path
'C:\Windows\TEMP\DSCExchangeModule\DSCExchangeModule.psm1'.
VERBOSE: [USCHIZWXCH4101]:                            [[ExchMailboxDatabase]MDB_USCHIZWDAG4002-DB1] Importing function 'Get-MailboxDatabase'.
VERBOSE: [USCHIZWXCH4101]:                            [[ExchMailboxDatabase]MDB_USCHIZWDAG4002-DB1] Importing function 'Mount-Database'.
VERBOSE: [USCHIZWXCH4101]:                            [[ExchMailboxDatabase]MDB_USCHIZWDAG4002-DB1] Importing function 'Move-ActiveMailboxDatabase'.
VERBOSE: [USCHIZWXCH4101]:                            [[ExchMailboxDatabase]MDB_USCHIZWDAG4002-DB1] Importing function 'Move-DatabasePath'.
VERBOSE: [USCHIZWXCH4101]:                            [[ExchMailboxDatabase]MDB_USCHIZWDAG4002-DB1] Importing function 'New-MailboxDatabase'.
VERBOSE: [USCHIZWXCH4101]:                            [[ExchMailboxDatabase]MDB_USCHIZWDAG4002-DB1] Importing function 'Remove-MailboxDatabase'.
VERBOSE: [USCHIZWXCH4101]:                            [[ExchMailboxDatabase]MDB_USCHIZWDAG4002-DB1] Importing function 'Set-ADServerSettings'.
VERBOSE: [USCHIZWXCH4101]:                            [[ExchMailboxDatabase]MDB_USCHIZWDAG4002-DB1] Importing function 'Set-MailboxDatabase'.
VERBOSE: [USCHIZWXCH4101]: LCM:  [ End    Set      ]  [[ExchMailboxDatabase]MDB_USCHIZWDAG4002-DB1]  in 0.6770 seconds.
PowerShell DSC resource DSC_ExchMailboxDatabase  failed to execute Set-TargetResource functionality with error message: Database must have only a single copy for the DB path or log path to be moved
    + CategoryInfo          : InvalidOperation: (:) [], CimException
    + FullyQualifiedErrorId : ProviderOperationExecutionFailure
    + PSComputerName        : localhost


Second/Additional DAG member:
VERBOSE: [USCHIZWXCH4102]: LCM:  [ Start  Resource ]  [[Script]WaitForFirstDAGMember]
VERBOSE: [USCHIZWXCH4102]: LCM:  [ Start  Test     ]  [[Script]WaitForFirstDAGMember]
VERBOSE: [USCHIZWXCH4102]:                            [[Script]WaitForFirstDAGMember] Importing the module MSFT_ScriptResource in force mode.
VERBOSE: [USCHIZWXCH4102]:                            [[Script]WaitForFirstDAGMember] Reusing existing Remote PowerShell Session to Exchange
VERBOSE: [USCHIZWXCH4102]:                            [[Script]WaitForFirstDAGMember] Importing the xExchange Remote PowerShell Module.
VERBOSE: [USCHIZWXCH4102]:                            [[Script]WaitForFirstDAGMember] Removing the imported "Get-DatabaseAvailabilityGroup" function.
VERBOSE: [USCHIZWXCH4102]:                            [[Script]WaitForFirstDAGMember] Loading module from path 'C:\Windows\TEMP\DSCExchangeModule\DSCExchangeModule.psm1'.
VERBOSE: [USCHIZWXCH4102]:                            [[Script]WaitForFirstDAGMember] Importing function 'Get-DatabaseAvailabilityGroup'.
VERBOSE: [USCHIZWXCH4102]: LCM:  [ End    Test     ]  [[Script]WaitForFirstDAGMember]  in 0.4760 seconds.
VERBOSE: [USCHIZWXCH4102]: LCM:  [ Start  Set      ]  [[Script]WaitForFirstDAGMember]
VERBOSE: [USCHIZWXCH4102]:                            [[Script]WaitForFirstDAGMember] Importing the module MSFT_ScriptResource in force mode.
VERBOSE: [USCHIZWXCH4102]:                            [[Script]WaitForFirstDAGMember] Performing the operation "Set-TargetResource" on target "Executing the SetScript with the user supplied credential".
VERBOSE: [USCHIZWXCH4102]:                            [[Script]WaitForFirstDAGMember] Reusing existing Remote PowerShell Session to Exchange
VERBOSE: [USCHIZWXCH4102]:                            [[Script]WaitForFirstDAGMember] Importing the xExchange Remote PowerShell Module.
VERBOSE: [USCHIZWXCH4102]:                            [[Script]WaitForFirstDAGMember] Removing the imported "Get-DatabaseAvailabilityGroup" function.
VERBOSE: [USCHIZWXCH4102]:                            [[Script]WaitForFirstDAGMember] Loading module from path 'C:\Windows\TEMP\DSCExchangeModule\DSCExchangeModule.psm1'.
VERBOSE: [USCHIZWXCH4102]:                            [[Script]WaitForFirstDAGMember] Importing function 'Get-DatabaseAvailabilityGroup'.
WARNING: [USCHIZWXCH4102]:                            [[Script]WaitForFirstDAGMember] DAG  object has no members yet. Sleeping for 60 seconds.
WARNING: [USCHIZWXCH4102]:                            [[Script]WaitForFirstDAGMember] DAG  object has no members yet. Sleeping for 60 seconds.
WARNING: [USCHIZWXCH4102]:                            [[Script]WaitForFirstDAGMember] DAG  object has no members yet. Sleeping for 60 seconds.
WARNING: [USCHIZWXCH4102]:                            [[Script]WaitForFirstDAGMember] DAG  object has no members yet. Sleeping for 60 seconds.
WARNING: [USCHIZWXCH4102]:                            [[Script]WaitForFirstDAGMember] DAG  object has no members yet. Sleeping for 60 seconds.
VERBOSE: [USCHIZWXCH4102]: LCM:  [ End    Set      ]  [[Script]WaitForFirstDAGMember]  in 300.7700 seconds.
PowerShell DSC resource MSFT_ScriptResource  failed to execute Set-TargetResource functionality with error message: Database Availability Group does not have at least one member after waiting the specified amount of time.
    + CategoryInfo          : InvalidOperation: (:) [], CimException
    + FullyQualifiedErrorId : ProviderOperationExecutionFailure
    + PSComputerName        : localhost

DSC configuration

First DAG member:
$dagName = 'USCHIZWDAG4002'
$dbName = "$($dagName)-DB1"

# Create primary database
ExchMailboxDatabase "MDB_$dbName"
{
    Name = $dbName
    Credential = $Credential
    EdbFilePath = "C:\ExchangeDatabases\$($dbName)\$($dbName).db\$($dbName).edb"
    LogFolderPath = "C:\ExchangeDatabases\$($dbName)\$($dbName).log"
    Server = $server.Name
    CircularLoggingEnabled = $true
    DatabaseCopyCount = 2
    IssueWarningQuota = '1946MB'
    ProhibitSendQuota = '2048MB'
    ProhibitSendReceiveQuota = '2253MB'
    AllowServiceRestart = $true
    DependsOn = '[ExchMailboxServer]MailboxServer'
}

Second DAG member:
# Wait for DAG to have at least one member before adding additional members (to avoid cluster creation race condition issues)
Script WaitForFirstDAGMember
{
    GetScript = {
        # Establish remote PowerShell session
        Get-RemoteExchangeSession -Credential $Credential -CommandsToLoad 'Get-DatabaseAvailabilityGroup' -Verbose:$VerbosePreference

        $dagProperties = Get-DatabaseAvailabilityGroup -Identity $dagName
        $result = @{'CurrentDAGMembers' = $dagProperties.Servers.Name}
        return $result
    }

    TestScript = {
        # Establish remote PowerShell session
        Get-RemoteExchangeSession -Credential $Credential -CommandsToLoad 'Get-DatabaseAvailabilityGroup' -Verbose:$VerbosePreference
        
        $dagProperties = Get-DatabaseAvailabilityGroup -Identity $dagName
        if ($dagProperties.Servers.Name.Count -gt 0)
        {
            return $true
        }
        else
        {
            return $false
        }
    }

    SetScript = {
        # Establish remote PowerShell session
        Get-RemoteExchangeSession -Credential $Credential -CommandsToLoad 'Get-DatabaseAvailabilityGroup' -Verbose:$VerbosePreference
        
        $foundFirstDAGMember = $false

        for ($i = 0; $i -lt 5; $i++)
        {
            $dagProperties = Get-DatabaseAvailabilityGroup -Identity $dagName

            if ($dagProperties.Servers.Name.Count -eq 0)
            {
                Write-Warning "DAG $dagName object has no members yet. Sleeping for 60 seconds."
                Start-Sleep -Seconds 60
            }
            else
            {
                $foundFirstDAGMember = $true
                break
            }
        }

        if (!$foundFirstDAGMember)
        {
            throw 'Database Availability Group does not have at least one member after waiting the specified amount of time.'
        }
    }
    DependsOn = '[ExchWaitForDAG]WaitForDAG'
}

Suggested solution

No because I can't find any error in the script unless this is a bug

Operating system the target node is running

OsName               : Microsoft Windows Server 2016 Standard
OsOperatingSystemSKU : StandardServerEdition
OsArchitecture       : 64-bit
WindowsBuildLabEx    : 14393.6167.amd64fre.rs1_release.230802-0927
OsLanguage           : en-US
OsMuiLanguages       : {en-US}

Exchange Server edition and version the target node is running

ServerRole    Edition AdminDisplayVersion
----------    ------- -------------------
   Mailbox Enterprise Version 15.1 (Build 2507.6)
   Mailbox Enterprise Version 15.1 (Build 2507.6)

PowerShell version and build the target node is running

Name                           Value
----                           -----
PSVersion                      5.1.14393.5582
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.14393.5582
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

ExchangeDsc version

Name        Version Path
----        ------- ----
ExchangeDsc 2.0.0   C:\Program Files\WindowsPowerShell\Modules\ExchangeDsc\2.0.0\ExchangeDsc.psd1