Cluster: Add node to primary cluster that is located in another subnet

[mohamednazar]

How to add secondary node to a existing running cluster which is created using xCluster DSC Resource.. I am using xWaitForCluster resource to add secondary node, but its not adding. Please see the below DSC resource code that I tried.

dsc_resource 'test-cluster' do resource :xWaitForCluster module_name 'xFailOverCluster' module_version '1.1' property :name, 'TestCluster1' property :RetryIntervalSec, 30 property :RetryCount, 100 end

So I tried using following code as well but nothing worked.

dsc_resource 'test-cluster' do resource :xCluster module_name 'xFailOverCluster' module_version '1.1' property :name, 'TestCluster1' property :staticipaddress, '10.0.0.3' property :domainadministratorcredential, ps_credential('mydomainname\\mydomainusername', mydomainpassword') end

Your help is much appreciated.

xCluster and xWaitForCluster DSC resource.

Windows 2012 R2 and PowerShell 5.x is running on target machine.

xFailOverCluster version is 1.1

[johlju]

@mohamednazar You need to use xCluster resource on second node after the xWaitForCluster resource, Please see the example 2-JoinAdditionalNodeToFailoverCluster.ps1.

[mohamednazar]

@johlju as you suggested I have tried xWaitForCluster then xCluster on second node but getting same error.

Code that I tried:
`dsc_resource 'wait-cluster' do
resource :xWaitForCluster
module_name 'xFailOverCluster'
module_version '1.8.0.0'
property :name, 'testcluster1'
property :PsDscRunAsCredential, ps_credential("#{node['sqlserver']['Username']}", "#{node['sqlserver']['Password']}")
property :RetryIntervalSec, 30
property :RetryCount, 100
end

dsc_resource 'join-cluster' do
resource :xCluster
module_name 'xFailOverCluster'
module_version '1.8.0.0'
property :name, "#{node['sqlserver']['cluster_name']}"
property :staticipaddress, "#{node['sqlserver']['static_ip']}"
property :domainadministratorcredential, ps_credential("#{node['sqlserver']['Username']}", "#{node['sqlserver']['Password']}")
end`

Error I am getting:

10.241.23.117 Compiling Cookbooks...←[0m
10.241.23.117 Converging 2 resources←[0m
10.241.23.117 Recipe: SqlServer::WaitForCluster←[0m
10.241.23.117 ←[0m Chef::Exceptions::PowershellCmdletException←[0m
10.241.23.117 -------------------------------------------←[0m
10.241.23.117 Powershell Cmdlet failed: Check the spelling of the cluster name. Otherwise, there might be a problem
10.241.23.117 ←[0m with your network. Make sure the cluster nodes are turned on and connected to
10.241.23.117 ←[0m the network or contact your network administrator.
10.241.23.117 ←[0m + CategoryInfo : ConnectionError: (:) [], CimException
10.241.23.117 ←[0m + FullyQualifiedErrorId : ClusterRpcConnection,Microsoft.FailoverClusters.
10.241.23.117 ←[0m PowerShell.AddClusterNodeCommand
10.241.23.117 ←[0m + PSComputerName : localhost
10.241.23.117 ←[0m
10.241.23.117 ←[0m The PowerShell DSC resource '[xCluster]DirectResourceAccess' with SourceInfo
10.241.23.117 ←[0m '' threw one or more non-terminating errors while running the
10.241.23.117 ←[0m Set-TargetResource functionality. These errors are logged to the ETW channel
10.241.23.117 ←[0m called Microsoft-Windows-DSC/Operational. Refer to this channel for more
10.241.23.117 ←[0m details.
10.241.23.117 ←[0m + CategoryInfo : InvalidOperation: (root/Microsoft/...gurationMan
10.241.23.117 ←[0m ager:String) [], CimException
10.241.23.117 ←[0m + FullyQualifiedErrorId : NonTerminatingErrorFromProvider
10.241.23.117 ←[0m + PSComputerName : localhost
10.241.23.117 ←[0m

Please help to resolve it..

Regards,
Mohamed Nazar

[johlju]

This cluster your trying to configure, is it running in an on-premises Active Directory domarn, or in a cloud Active Directory domain? There are others having similar problems with adding a second node, so trying to figure out the scope.
I have not been able to reproduce the problem the others have reported, yet.

[mohamednazar]

@johlju The cluster is running in AWS EC2 and Active Directory is in On-Premises.

[mohamednazar]

@johlju I have tried with xCluster after xWaitForCluster new set of servers and found secondary node successfully added to the cluster. But Primary node IP address only available due to that cluster went offline after I did move Core Cluster Services to "Best Possible Node" or "Select Node". Please find my code below.

`
dsc_resource 'wait-cluster' do
resource :xWaitForCluster
module_name 'xFailOverCluster'
module_version '1.8.0.0'
property :name, "#{node['sqlserver']['cluster_name']}"
property :PsDscRunAsCredential, ps_credential("#{node['sqlserver']['Username']}", "#{node['sqlserver']['Password']}")
property :RetryIntervalSec, 30
property :RetryCount, 100
end

dsc_resource 'join-cluster' do
resource :xCluster
module_name 'xFailOverCluster'
module_version '1.8.0.0'
property :name, "#{node['sqlserver']['cluster_name']}"
property :staticipaddress, '10.0.0.2'
property :domainadministratorcredential, ps_credential("#{node['sqlserver']['Username']}", "#{node['sqlserver']['Password']}")
end
`

Please find the screenshot below.

How to get added secondary node IP address as well to cluster? so that I can move cluster resources to secondary node. When we do manually the IPs are adding fine. See the screenshot below (Yellow marked IP added) after that I added manually.

Please help!
regards,
Mohamed Nazar

[mohamednazar]

@johlju or @kwirkykat or @mbreakey3

Waiting for your assistance. Anyone please help how to get added secondary node IP address to the cluster.

Regads,
Mohamed Nazar

[johlju]

@mohamednazar A cluster has only one or more IP address for the CNO. They are not different for each cluster node. Each node has at least one IP-address on the NIC, and the CNO has at least one IP-address on the same subnet as the IP-address on the NIC.

For example:
Subnet: 192.168.10.0/24

Node 1: 192.168.10.1
Node 2: 192.168.10.2
CNO: 192.168.10.3

[johlju]

@mohamednazar If you have another NIC in you system, with a different subnet, then you can add an IP-address for that subnet to the CNO as well.

Please let me know if I misunderstood your question,

[mohamednazar]

@johlju
after cluster created and added secondary node successfully, I can see only one IP address (Primary server IP) under Cluster Core Resources which is in bottom of "Fail Over Cluster Manager" see the image below. Secondary server IP address not adding.

But it should be like below.

for cluster and SQL always on as per AWS documentation , the both (Primary and Secondary) servers should have 2 additional IPs on default NIC. See the example below.

Primary Server: Subnet A
Default IP: 10.0.0.1
Listener IP: 10.0.0.2
Cluster IP: 10.0.0.3

Secondary Server: Subnet B
Default IP: 10.0.1.1
Listener IP: 10.0.1.2
Cluster IP: 10.0.1.3

The secondary server cluster IP not adding to the cluster. Hope you clear my issue now.

Regards,
Mohamed Nazar

[johlju]

Okay, you are gonna stretch the cluster over two regions? I personally have not stretched a cluster over different subnets. :/ I have to read up on it to know if it is supported. I won’t have time for that in the near future. Maybe someone else in the community familiar with that setup can tell if the resources support it today, or if we in the community need to make changes to the resources.

Do you have a link to the AWS documentation?

[mohamednazar]

I am going to setup cluster with two availability zone in single region. Please view the AWS documentation link

http://docs.aws.amazon.com/quickstart/latest/sql/implementation.html#ip-addressing

Regards,
Mohamed Nazar

[dead8171]

I am not able to join the second node to the cluster, I have 3 nodes 2 SQL nodes and a DC both the nodes are domain joined, and are in an availability set, I was able to create a cluster on Node1 But was not successful in joining the node, I am using Automation Account and both the nodes and the DC are in Azure.

[mohamednazar]

@dead8171 on secondary nodes use xWaitForCluster resource then use xCluster.. Check whether all cluster related ports are not blocked..

[mohamednazar]

@johlju any update on adding secondary node IP address on cluster core services?

[johlju]

@mohamednazar I did a quick google search yesterday, but haven't been able to read any, nor the AWS link you provided. Long days at the normal day job.
But as far as I gather, if it means that one must provide two IP addresses when configuring the cluster, then xCluster does not support that yet. It only takes a single IP address in the StaticIPAddress parameter.

The New-Cluster cmdlet is called when creating the cluster, it's StaticAddress takes a string array. See documentation here New-Cluster. So it might be as simple as changing the StaticIPAddress parameter from a string to a string array. This change should be done regardless since New-Cluster expects an array.

[mohamednazar]

@johlju I think I need to use xClusterNetwork resource to add secondary node IP & Netmask to the cluster.. Correct me if I am wrong. If its correct should I want to apply the script on PrimaryReplica or SecondaryReplica?

dsc_resource 'dsc_clus_net_secondary' do resource :xClusterNetwork property :Address, "#{node['sqlserver']['static_ip']}" property :AddressMask, "#{node['sqlserver']['address_mask']}" property :Name, 'Clustername' property :PsDscRunAsCredential, ps_credential("Username", "Password") end

Regards,
Mohamed Nazar

[johlju]

xClusterNetwork handles the properties of the cluster networks (subnets), please see issue #92. I think issue #151 is on the right track? What do you think?

[mohamednazar]

Yes seems to be same issue. Let me try xClusterNetwork properties...

[mohamednazar]

@johlju

I have executed following xClusterNetwork script on secondaryreplica server and getting error.

Script:

dsc_resource 'dsc_clus_net_secondary' do
resource :xClusterNetwork
property :Address, "IP"
property :AddressMask, "mask"
property :Name, "name_of_the_cluster"
property :Metric, '10'
property :Role, '3'
property :PsDscRunAsCredential, ps_credential("Username", "password")
end

Error:

[2017-11-27T03:59:12+00:00] INFO: Processing dsc_resource[dsc_clus_net_secondary] action run (SqlServer::dsc_clus_net_secondary line 7)
[2017-11-27T03:59:14+00:00] INFO: Running queued delayed notifications before re-raising exception
[2017-11-27T03:59:15+00:00] ERROR: Running exception handlers
[2017-11-27T03:59:15+00:00] ERROR: Exception handlers complete
[2017-11-27T03:59:15+00:00] FATAL: Stacktrace dumped to C:/chef/cache/chef-stacktrace.out
[2017-11-27T03:59:15+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2017-11-27T03:59:15+00:00] FATAL: Chef::Exceptions::PowershellCmdletException: dsc_resource[dsc_clus_net_secondary] (SqlServer::dsc_clus_net_secondary line 7) had an error: Chef::Exceptions::PowershellCmdletException: Powershell Cmdlet failed: The property 'Name' cannot be found on this object. Verify that the property

exists and can be set.

+ CategoryInfo          : InvalidOperation: (:) [], CimException

+ FullyQualifiedErrorId : PropertyNotFound

+ PSComputerName        : localhost

You cannot call a method on a null-valued expression.

+ CategoryInfo          : InvalidOperation: (:) [], CimException

+ FullyQualifiedErrorId : InvokeMethodOnNull

+ PSComputerName        : localhost

The property 'Role' cannot be found on this object. Verify that the property

exists and can be set.

+ CategoryInfo          : InvalidOperation: (:) [], CimException

+ FullyQualifiedErrorId : PropertyNotFound

+ PSComputerName        : localhost

You cannot call a method on a null-valued expression.

+ CategoryInfo          : InvalidOperation: (:) [], CimException

+ FullyQualifiedErrorId : InvokeMethodOnNull

+ PSComputerName        : localhost

The property 'Metric' cannot be found on this object. Verify that the property

exists and can be set.

+ CategoryInfo          : InvalidOperation: (:) [], CimException

+ FullyQualifiedErrorId : PropertyNotFound

+ PSComputerName        : localhost

You cannot call a method on a null-valued expression.

+ CategoryInfo          : InvalidOperation: (:) [], CimException

+ FullyQualifiedErrorId : InvokeMethodOnNull

+ PSComputerName        : localhost

The PowerShell DSC resource '[xClusterNetwork]DirectResourceAccess' with

SourceInfo '' threw one or more non-terminating errors while running the

Set-TargetResource functionality. These errors are logged to the ETW channel

called Microsoft-Windows-DSC/Operational. Refer to this channel for more

details.

+ CategoryInfo          : InvalidOperation: (root/Microsoft/...gurationMan 

ager:String) [], CimException

+ FullyQualifiedErrorId : NonTerminatingErrorFromProvider

+ PSComputerName        : localhost

Please let me know should I want to change any property value or should I want to execute the above script on primaryreplica server?

Regards,
Mohamed Nazar

[mohamednazar]

@johlju or @kwirkykat or @mbreakey3

Waiting for your assistance.

Regards,
Mohamed Nazar

[johlju]

@mohamednazar xClusterNetwork only changes properties on existing networks. It does not have logic to add networks, nor can it have that kind of logic (as mentioned in issue #92). The xClusterNetwork can only change the properties Name, Role and Metric.

[smastrorocco]

Any update on this, have same issue with cluster across AWS subnets.

[codykonior]

I've been working on setting up a multi-subnet cluster where each node only has one subnet so I have some information to add to the discussion.

• You can't modify xCluster to take multiple StaticIPAddress record because New-Cluster will abort if any particular node doesn't have all subnets.

  New-Cluster: Static address 'xxx/24' was not found on any cluster network.

• Add-ClusterNode doesn't take IPAddress so that's out too.

The way I envision you'd get this going is adding logic to xCluster so that when it's doing an Add-ClusterNode it will do more with the StaticIPAddress.

• Add an IP Address resource to the cluster (there's a set naming convention)
• Determine which Cluster Network will be used (based on IP and subnet mask)
• Create an OR dependency on the Cluster Name, adding the new IP address

It looks like this:

Get-Cluster | Add-ClusterResource -Name "IP Address $ip" -Group "Cluster Group" -ResourceType "IP Address"
$network = Get-Cluster | Get-ClusterNetwork | Where-Object { (([Net.IPAddress] $_.Address).Address -band ([Net.IPAddress] $_.AddressMask).Address) -eq (([Net.IPAddress] $ip).Address -band ([Net.IPAddress] $_.AddressMask).Address)}
Get-Cluster | Get-ClusterResource -Name "IP Address $ip" | Set-ClusterParameter -Multiple @{ Address = $ip; Network = $network.Name; SubnetMask = $network.AddressMask; }
$expression = (Get-Cluster | Get-ClusterResourceDependency -Resource "Cluster Name").DependencyExpression
if ($expression -match "^\((.*)\)$") {
    $expression = $Matches[1] + " or [IP Address $ip]"
} else {
    $expression = $expression + " or [IP Address $ip]"
}
Get-Cluster | Set-ClusterResourceDependency -Resource "Cluster Name" -Dependency $expression
# Without this, it won't start automatically on first try
(Get-Cluster | Get-ClusterResource -Name "IP Address $clusterIPAddress").PersistentState = 1

Maybe this could be controlled through an additional flag so that it doesn't break any existing code that might have dummy StaticIPAddress passed in for nodes.

[mgates86]

Has anyone solved this? @mohamednazar @johlju

[mohamednazar]

@mgates86

I used xWaitForCluster resource on secondary node after that it resolved my problem.

On Primary and Secondary I assigned one additional IP (Secondary IP)

1st I created cluster on Primary, after that secondary was able to add to cluster using xWaitForCluster resource.

Thanks

[mgates86]

@mohamednazar thank you. curious were you doing this for a sql always on cluster? im having a similar issue with multiple subnets on the slq ag listener

[codykonior]

Just get all the replicas up and added to the AG before you create the listener. See https://github.com/codykonior/OftenOn/blob/master/OftenOn/Configuration/OftenOn.ps1 for one way to do it.

[mgates86]

@mgates86

I used xWaitForCluster resource on secondary node after that it resolved my problem.

On Primary and Secondary I assigned one additional IP (Secondary IP)

1st I created cluster on Primary, after that secondary was able to add to cluster using xWaitForCluster resource.

Thanks

@mohamednazar any chance you can post a sample. im still struggling with this.

[segoja7]

I have the same problem, I am using 1-Cluster_CreateFirstNodeOfAFailoverClusterConfig.ps1 for creating my cluster in node1, and later I am using 2-Cluster_JoinAdditionalNodeToFailoverClusterConfig.ps1 for join in the cluster my second node. But this not work.

Reviewing the comments @mohamednazar I am trying again in my second node with: 1-WaitForCluster_WaitForFailoverClusterToBePresentConfig.ps1 but the error is the same.

Check the spelling of the cluster name. Otherwise, there might be a problem with your network. Make sure the cluster nodes are turned on and connected to the network or contact your network administrator.
    + CategoryInfo          : ConnectionError: (:) [], CimException
    + FullyQualifiedErrorId : ClusterRpcConnection,Microsoft.FailoverClusters.PowerShell.AddClusterNodeCommand
    + PSComputerName        : sqlfsx2
 
VERBOSE: [SQLFSX2]:                            [[Cluster]JoinSecondNodeToCluster] Added node SQLFSX2 to cluster cluster.
VERBOSE: [SQLFSX2]: LCM:  [ End    Set      ]  [[Cluster]JoinSecondNodeToCluster]  in 31.1600 seconds.
The PowerShell DSC resource '[Cluster]JoinSecondNodeToCluster' with SourceInfo 'C:\Users\Administrator\Documents\WaitForCluster_WaitForFailoverClusterToBePresentConfig.ps1::87::9::Cluster' threw one or more non-terminating errors while running the 
Set-TargetResource functionality. These errors are logged to the ETW channel called Microsoft-Windows-DSC/Operational. Refer to this channel for more details.
    + CategoryInfo          : InvalidOperation: (:) [], CimException
    + FullyQualifiedErrorId : NonTerminatingErrorFromProvider
    + PSComputerName        : sqlfsx2
 
VERBOSE: [SQLFSX2]: LCM:  [ End    Set      ]
The SendConfigurationApply function did not succeed.
    + CategoryInfo          : NotSpecified: (root/Microsoft/...gurationManager:String) [], CimException
    + FullyQualifiedErrorId : MI RESULT 1
    + PSComputerName        : sqlfsx2

My scenario is the following in AWS:

AD: corp.local
Node1: in the domain corp.local with the following IPS: Primary IP x.x.40.104, secondary IPS. x.x.40.105, x.x.40.106
Node2: in the domain corp.local with the following IPS: Primary IP x.x.44.104, secondary IPS. x.x.44.105, x.x.44.106.

Any comment is good!