Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adding Support for Filtering Specific Subscription Id's to Include in ChildResource Discovery #880

Merged
merged 5 commits into from
May 2, 2024
Merged

Adding Support for Filtering Specific Subscription Id's to Include in ChildResource Discovery #880

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
878fdb4
Update
Jefajers Apr 24, 2024
d305b3d
Update
Jefajers Apr 24, 2024
6b4590e
Update
Jefajers Apr 24, 2024
adc897e
Update
Jefajers Apr 24, 2024
a4509d7
UpdateWording
Jefajers Apr 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/wiki/Frequently-Asked-Questions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,8 @@ Yes, ensure the following setting combinations are applied (replace `rgname1`, `

"Core.SkipResourceGroup": false

"Core.SubscriptionsToIncludeChildResource": ["SubscriptionId1","SubscriptionId2"]

"Core.SubscriptionsToIncludeResourceGroups": ["SubscriptionId1","SubscriptionId2"]
```

Expand Down
17 changes: 9 additions & 8 deletions docs/wiki/Settings.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,14 +35,15 @@ The following configuration values can be modified within the `settings.json` fi
| 23 | SkipResourceType | Skip specific [Resource Types](https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/resource-providers-and-types) (only targets Resource Group scoped resources) | `"Core.SkipResourceType": ["Microsoft.VSOnline/plans"]` |
| 24 | SkipRole | Do not include Role types in pull | `"Core.SkipRole": false` |
| 25 | State | Folder to store AzOpsState artefact, defaults to `root` | `"Core.State: "/root"` |
| 26 | SubscriptionsToIncludeResourceGroups | Filter which Subscription IDs should include Resource Groups in pull [Logic Updated in v2.0.0](https://github.com/Azure/AzOps/releases/tag/2.0.0) | `"Core.SubscriptionsToIncludeResourceGroups": ["*"]` |
| 27 | TemplateParameterFileSuffix | Default template file suffix. *Not recommended to change* | `"Core.TemplateParameterFileSuffix": ".json"` |
| 28 | AllowMultipleTemplateParameterFiles | Control multiple parameter file behaviour. *Not recommended to change* | `"Core.AllowMultipleTemplateParameterFiles": false` |
| 29 | DeployAllMultipleTemplateParameterFiles | Control base template deployment behaviour with changes and un-changed multiple corresponding parameter files. | `"Core.DeployAllMultipleTemplateParameterFiles": false` |
| 30 | MultipleTemplateParameterFileSuffix | Multiple parameter file suffix identifier. *Example mytemplate.x1.bicepparam* | `"Core.MultipleTemplateParameterFileSuffix": ".x"` |
| 31 | ParallelDeployMultipleTemplateParameterFiles | Control parallel deployment of MultipleTemplateParameterFiles behaviour | `"Core.ParallelDeployMultipleTemplateParameterFiles": false` |
| 32 | ThrottleLimit | Value declaring number of parallel threads. [Read more](https://github.com/azure/azops/wiki/performance-considerations) | `"Core.ThrottleLimit": 5` |
| 33 | WhatifExcludedChangeTypes | Exclude specific change types from WhatIf operations | `"Core.WhatifExcludedChangeTypes": ["NoChange","Ignore"]` |
| 26 | SubscriptionsToIncludeChildResource | Filter which Subscription IDs should include child resources in pull | `"Core.SubscriptionsToIncludeChildResource": ["*"]` |
| 27 | SubscriptionsToIncludeResourceGroups | Filter which Subscription IDs should include Resource Groups in pull [Logic Updated in v2.0.0](https://github.com/Azure/AzOps/releases/tag/2.0.0) | `"Core.SubscriptionsToIncludeResourceGroups": ["*"]` |
| 28 | TemplateParameterFileSuffix | Default template file suffix. *Not recommended to change* | `"Core.TemplateParameterFileSuffix": ".json"` |
| 29 | AllowMultipleTemplateParameterFiles | Control multiple parameter file behaviour. *Not recommended to change* | `"Core.AllowMultipleTemplateParameterFiles": false` |
| 30 | DeployAllMultipleTemplateParameterFiles | Control base template deployment behaviour with changes and un-changed multiple corresponding parameter files. | `"Core.DeployAllMultipleTemplateParameterFiles": false` |
| 31 | MultipleTemplateParameterFileSuffix | Multiple parameter file suffix identifier. *Example mytemplate.x1.bicepparam* | `"Core.MultipleTemplateParameterFileSuffix": ".x"` |
| 32 | ParallelDeployMultipleTemplateParameterFiles | Control parallel deployment of MultipleTemplateParameterFiles behaviour | `"Core.ParallelDeployMultipleTemplateParameterFiles": false` |
| 33 | ThrottleLimit | Value declaring number of parallel threads. [Read more](https://github.com/azure/azops/wiki/performance-considerations) | `"Core.ThrottleLimit": 5` |
| 34 | WhatifExcludedChangeTypes | Exclude specific change types from WhatIf operations | `"Core.WhatifExcludedChangeTypes": ["NoChange","Ignore"]` |

## Workflow / Pipeline Settings

Expand Down
5 changes: 3 additions & 2 deletions src/internal/configurations/Core.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Set-PSFConfig -Module AzOps -Name Core.CustomJqTemplatePath -Value (Join-Path $p
Set-PSFConfig -Module AzOps -Name Core.SkipCustomJqTemplate -Value $true -Initialize -Validation bool -Description 'Controls usage of CustomJqTemplatePath to search for custom jq template'
Set-PSFConfig -Module AzOps -Name Core.MainTemplate -Value "$script:ModuleRoot\data\template\template.json" -Initialize -Validation string -Description 'Main template json'
Set-PSFConfig -Module AzOps -Name Core.OfferType -Value 'MS-AZR-0017P' -Initialize -Validation string -Description '-'
Set-PSFConfig -Module AzOps -Name Core.PartialMgDiscoveryRoot -Value @() -Initialize -Validation stringarray -Description 'Used in combination with AZOPS_SUPPORT_PARTIAL_MG_DISCOVERY, example value: "Contoso","Tailspin","Management"'
Set-PSFConfig -Module AzOps -Name Core.PartialMgDiscoveryRoot -Value @() -Initialize -Validation stringarray -Description 'Generate folder hierachy for specific Management Groups IDs'
Set-PSFConfig -Module AzOps -Name Core.IncludeResourcesInResourceGroup -Value @('*') -Initialize -Validation stringarray -Description 'Global flag to discover only resources in these resource groups.'
Set-PSFConfig -Module AzOps -Name Core.IncludeResourceType -Value @('*') -Initialize -Validation stringarray -Description 'Global flag to discover only specific resource types.'
Set-PSFConfig -Module AzOps -Name Core.SkipChildResource -Value $true -Initialize -Validation bool -Description 'Global flag to indicate whether child resources should be discovered or not. Requires SkipResourceGroup and SkipResource to be false.'
Expand All @@ -26,7 +26,8 @@ Set-PSFConfig -Module AzOps -Name Core.SkipResourceGroup -Value $false -Initiali
Set-PSFConfig -Module AzOps -Name Core.SkipResourceType -Value @('Microsoft.VSOnline/plans', 'Microsoft.PowerPlatform/accounts', 'Microsoft.PowerPlatform/enterprisePolicies') -Initialize -Validation stringarray -Description 'Global flag to skip discovery of specific Resource types.'
Set-PSFConfig -Module AzOps -Name Core.SkipRole -Value $false -Initialize -Validation bool -Description '-'
Set-PSFConfig -Module AzOps -Name Core.State -Value (Join-Path $pwd -ChildPath "root") -Initialize -Validation string -Description 'Folder to store AzOpsState artefact'
Set-PSFConfig -Module AzOps -Name Core.SubscriptionsToIncludeResourceGroups -Value @('*') -Initialize -Validation stringarray -Description 'Requires SkipResourceGroup to be false. Subscription ID or Display Name that matches the filter. Powershell filter that matches with like operator is supported.'
Set-PSFConfig -Module AzOps -Name Core.SubscriptionsToIncludeChildResource -Value @('*') -Initialize -Validation stringarray -Description 'Requires SkipResourceGroup, SkipResource and SkipChildResource to be false. Subscription ID that matches the filter.'
Set-PSFConfig -Module AzOps -Name Core.SubscriptionsToIncludeResourceGroups -Value @('*') -Initialize -Validation stringarray -Description 'Requires SkipResourceGroup to be false. Subscription ID that matches the filter.'
Set-PSFConfig -Module AzOps -Name Core.TemplateParameterFileSuffix -Value '.json' -Initialize -Validation string -Description 'Parameter file suffix identifier'
Set-PSFConfig -Module AzOps -Name Core.AllowMultipleTemplateParameterFiles -Value $false -Initialize -Validation string -Description 'Global flag to control multiple parameter file behaviour'
Set-PSFConfig -Module AzOps -Name Core.DeployAllMultipleTemplateParameterFiles -Value $false -Initialize -Validation string -Description 'Global flag to control base template deployment behaviour with changes and un-changed multiple corresponding parameter files'
Expand Down
34 changes: 25 additions & 9 deletions src/internal/functions/Get-AzOpsResourceDefinition.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,10 @@
Skip discovery of roles for better performance.
.PARAMETER StatePath
The root folder under which to write the resource json.
.PARAMETER SubscriptionsToIncludeChildResource
Filter which Subscription IDs should include child resources in pull.
.PARAMETER SubscriptionsToIncludeResourceGroups
Filter which Subscription IDs should include Resource Groups in pull.
.EXAMPLE
$TenantRootId = '/providers/Microsoft.Management/managementGroups/{0}' -f (Get-AzTenant).Id
Get-AzOpsResourceDefinition -scope $TenantRootId -Verbose
Expand Down Expand Up @@ -84,7 +88,13 @@

[Parameter(Mandatory = $false)]
[string]
$StatePath = (Get-PSFConfigValue -FullName 'AzOps.Core.State')
$StatePath = (Get-PSFConfigValue -FullName 'AzOps.Core.State'),

[string[]]
$SubscriptionsToIncludeChildResource = (Get-PSFConfigValue -FullName 'AzOps.Core.SubscriptionsToIncludeChildResource'),

[string[]]
$SubscriptionsToIncludeResourceGroups = (Get-PSFConfigValue -FullName 'AzOps.Core.SubscriptionsToIncludeResourceGroups')
)

begin {
Expand Down Expand Up @@ -222,12 +232,15 @@
}
}
else {
if ((Get-PSFConfigValue -FullName 'AzOps.Core.SubscriptionsToIncludeResourceGroups') -ne '*') {
$subscriptionsToIncludeResourceGroups = $subscriptions | Where-Object { $_.Id -in (Get-PSFConfigValue -FullName 'AzOps.Core.SubscriptionsToIncludeResourceGroups') }
}
$query = "resourcecontainers | where type == 'microsoft.resources/subscriptions/resourcegroups' | where managedBy == '' | order by ['id'] asc"
if ($subscriptionsToIncludeResourceGroups) {
$resourceGroups = Search-AzOpsAzGraph -Subscription $subscriptionsToIncludeResourceGroups -Query $query -ErrorAction Stop
if ($SubscriptionsToIncludeResourceGroups -ne '*') {
$newSubscriptionsToIncludeResourceGroups = $subscriptions | Where-Object { $_.Id -in $SubscriptionsToIncludeResourceGroups }
if ($newSubscriptionsToIncludeResourceGroups) {
$resourceGroups = Search-AzOpsAzGraph -Subscription $newSubscriptionsToIncludeResourceGroups -Query $query -ErrorAction Stop
}
else {
Write-AzOpsMessage -LogLevel Debug -LogString 'Get-AzOpsResourceDefinition.Subscription.NotFound' -Target $ScopeObject
}
}
else {
$resourceGroups = Search-AzOpsAzGraph -Subscription $subscriptions -Query $query -ErrorAction Stop
Expand Down Expand Up @@ -280,8 +293,8 @@
}
# Process Policies at Resource Group scope
if (-not $SkipPolicy) {
if ($subscriptionsToIncludeResourceGroups) {
Get-AzOpsPolicy -ScopeObject $scopeObject -Subscription $subscriptions -SubscriptionsToIncludeResourceGroups $subscriptionsToIncludeResourceGroups -ResourceGroup -StatePath $StatePath
if ($newSubscriptionsToIncludeResourceGroups) {
Get-AzOpsPolicy -ScopeObject $scopeObject -Subscription $subscriptions -SubscriptionsToIncludeResourceGroups $newSubscriptionsToIncludeResourceGroups -ResourceGroup -StatePath $StatePath
}
else {
Get-AzOpsPolicy -ScopeObject $scopeObject -Subscription $subscriptions -ResourceGroup -StatePath $StatePath
Expand Down Expand Up @@ -324,8 +337,11 @@
else {
Write-AzOpsMessage -LogLevel Debug -LogString 'Get-AzOpsResourceDefinition.SkippingResources' -Target $ScopeObject
}
# Process resources as scope in parallel, look for childResource
# Process Child resources at resource scope in parallel
if (-not $SkipResource -and -not $SkipChildResource) {
if ($SubscriptionsToIncludeChildResource -ne '*') {
$resources = $resources | Where-Object { $_.subscriptionId -in $SubscriptionsToIncludeChildResource }
}
$resources | Foreach-Object -ThrottleLimit (Get-PSFConfigValue -FullName 'AzOps.Core.ThrottleLimit') -Parallel {
$resource = $_
$runspaceData = $using:runspaceData
Expand Down
3 changes: 2 additions & 1 deletion src/tests/functional/Functional.Tests.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,8 @@ if (Test-Path -Path $generatedRoot) {

# Invoke the Invoke-AzOpsPull function to generate the scope data which can be tested against to ensure structure is correct and data model hasn't changed.

Set-PSFConfig -FullName AzOps.Core.SubscriptionsToIncludeResourceGroups -Value $script:subscriptionId
Set-PSFConfig -FullName AzOps.Core.SubscriptionsToIncludeChildResource -Value @($script:subscriptionId)
Set-PSFConfig -FullName AzOps.Core.SubscriptionsToIncludeResourceGroups -Value @($script:subscriptionId)
Set-PSFConfig -FullName AzOps.Core.SkipChildResource -Value $false
Set-PSFConfig -FullName AzOps.Core.SkipPim -Value $false
$deploymentLocationId = (Get-FileHash -Algorithm SHA256 -InputStream ([IO.MemoryStream]::new([byte[]][char[]](Get-PSFConfigValue -FullName 'AzOps.Core.DefaultDeploymentRegion')))).Hash.Substring(0, 4)
Expand Down
13 changes: 9 additions & 4 deletions src/tests/integration/Repository.Tests.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,8 +55,9 @@ Describe "Repository" {
Write-PSFMessage -Level Verbose -Message "Creating repository test environment" -FunctionName "BeforeAll"
$templateFile = Join-Path -Path $global:testroot -ChildPath "templates/azuredeploy.jsonc"
$templateParameters = @{
"tenantId" = "$script:tenantId"
"subscriptionId" = "$script:subscriptionId"
"tenantId" = $script:tenantId
"subscriptionId" = $script:subscriptionId
"otherSubscriptionId" = $otherSubscription[0].Id
}
$params = @{
ManagementGroupId = "$script:tenantId"
Expand Down Expand Up @@ -164,7 +165,8 @@ Describe "Repository" {
#endregion PartialMgDiscoveryRoot Pull

#region GeneratedRoot Pull
Set-PSFConfig -FullName AzOps.Core.SubscriptionsToIncludeResourceGroups -Value $script:subscriptionId
Set-PSFConfig -FullName AzOps.Core.SubscriptionsToIncludeChildResource -Value @($script:subscriptionId)
Set-PSFConfig -FullName AzOps.Core.SubscriptionsToIncludeResourceGroups -Value @($script:subscriptionId,$otherSubscription[0].Id)
Set-PSFConfig -FullName AzOps.Core.PartialMgDiscoveryRoot -Value @()
Set-PSFConfig -FullName AzOps.Core.State -Value $generatedRoot
Set-PSFConfig -FullName AzOps.Core.SkipLock -Value $false
Expand Down Expand Up @@ -287,7 +289,7 @@ Describe "Repository" {
$script:subscriptionFile = ($script:subscriptionPath).FullName
Write-PSFMessage -Level Debug -Message "SubscriptionFile: $($script:subscriptionFile)" -FunctionName "BeforeAll"

$script:resourceGroupPath = ($filePaths | Where-Object Name -eq "microsoft.resources_resourcegroups-$(($script:resourceGroup.ResourceGroupName).toLower()).json")
$script:resourceGroupPath = ($filePaths | Where-Object { $_.Name -eq "microsoft.resources_resourcegroups-$(($script:resourceGroup.ResourceGroupName).toLower()).json" -and $_.FullName -match $script:subscriptionId })
$script:resourceGroupDirectory = ($script:resourceGroupPath).Directory
$script:resourceGroupFile = ($script:resourceGroupPath).FullName
$script:resourceGroupDeploymentName = "AzOps-{0}-{1}" -f $($script:resourceGroupPath.Name.Replace(".json", '')), $deploymentLocationId
Expand Down Expand Up @@ -1025,6 +1027,9 @@ Describe "Repository" {
$script:ruleCollectionDeployment = Get-AzResourceGroupDeployment -ResourceGroupName 'App1-azopsrg' -Name $script:ruleCollectionDeploymentName
$ruleCollectionDeployment.ProvisioningState | Should -Be "Succeeded"
}
It "Validate SubscriptionsToIncludeChildResource filter by ensuring only one rulegroup child item is pulled back" {
$script:ruleCollectionGroupsPath.Count | Should -Be 1
}
#endregion

#region Scope - logAnalyticsWorkspaceSavedSearchesPath (./root/tenant root group/test/platform/management/subscription-0/App1-azopsrg/thisisalongloganalyticsworkspacename123456789011121314151617181)
Expand Down