Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

2.1 update multi foundation monitoring #466

Open
wants to merge 3 commits into
base: 2.1
Choose a base branch
from
Open

2.1 update multi foundation monitoring #466

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ca76f03
Additional updates for federation and direct scraping [#182276095]
jerrybelmonte Jun 7, 2022
ff51722
Additional updates for federation and direct scraping [#182276095]
jerrybelmonte Jun 7, 2022
661ddcd
Merge branch '2.1' into 2.1-update-multi-foundation-monitoring
jerrybelmonte Jun 7, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
26 changes: 13 additions & 13 deletions architecture.html.md.erb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -325,9 +325,9 @@ describe the most common configuration scenarios:

* [Monitoring TAS for VMs and TKGI on a Single Ops Manager Foundation](#tas-tkgi-single-foundation)

* [Monitoring TAS for VMs on a Different Ops Manager Foundation](#tas-different-foundation)
* [Configure Multi-Foundation Monitoring Using Direct Scraping](#multiple-foundations-direct-scrape)

* [Monitoring TKGI on a Different Ops Manager Foundation](#tkgi-different-foundation)
* [Configure Multi-Foundation Monitoring Using Federation](#multiple-foundations-federation)

### <a id='tas-single-foundation'></a> Monitoring TAS for VMs on a Single Ops Manager Foundation

Expand Down Expand Up @@ -377,22 +377,22 @@ Pipeline](installing/automated-pipeline.html)

* [Configuring Healthwatch Exporter for TKGI](configuring/configuring-exporter-tkgi.html)

### <a id='tas-different-foundation'></a> Monitoring TAS for VMs on a Different Ops Manager Foundation
### <a id='multiple-foundations-direct-scrape'></a> Configure Multi-Foundation Monitoring Using Direct Scraping

You can monitor several Ops Manager foundations that have TAS for VMs installed from a Healthwatch tile that you install on a separate Ops Manager foundation.
You can monitor several Ops Manager foundations from a Healthwatch tile that you install on a separate Ops Manager foundation using direct scrapes.

To configure your Healthwatch deployment to monitor several Ops Manager foundations from a single monitoring Ops Manager foundation,
see [Configure Multi-Foundation Monitoring for TAS for VMs](configuring/multi-foundation-monitoring.html#multiple-tas-foundations) in _Configuring
To configure your Healthwatch deployment to monitor several Ops Manager foundations from a single monitoring Ops Manager foundation using direct scrapes,
see [Configure Multi-Foundation Monitoring Using Direct Scraping](configuring/multi-foundation-monitoring.html#multiple-foundations-direct-scrape) in _Configuring
Multi-Foundation Monitoring_.

### <a id='tkgi-different-foundation'></a> Monitoring TKGI on a Different Ops Manager Foundation
### <a id='multiple-foundations-federation'></a> Configure Multi-Foundation Monitoring Using Federation

You can monitor several Ops Manager foundations that have TKGI installed from a Healthwatch tile that you install on a separate Ops Manager foundation.
You can monitor several Ops Manager foundations from a Healthwatch tile that you install on a separate Ops Manager foundation using federation.

The Healthwatch tile running on a TKGI Ops Manager foundation is able to dynamically find clusters and generate scrape configs for the cluster metrics.
As a result, internal Kubernetes metrics can only be retrieved from a downstream TSDB VM through federation. The Healthwatch TKGI exporters simply provide
supporting metrics for that foundation.
Federation is necessary to get the TKGI cluster metrics as the Healthwatch tile running on a TKGI Ops Manager foundation is able to dynamically find
clusters and generate scrape configs for the cluster metrics. Directly scraping the Healthwatch TKGI exporters will only provide supporting metrics
for that foundation.

To configure your Healthwatch deployment to monitor several Ops Manager foundations from a single monitoring Ops Manager foundation
through federation, see [Configure Multi-Foundation Monitoring for TKGI](configuring/multi-foundation-monitoring.html#multiple-tkgi-foundations) in
To configure your Healthwatch deployment to monitor several Ops Manager foundations from a single monitoring Ops Manager foundation using federation,
see [Configure Multi-Foundation Monitoring Using Federation](configuring/multi-foundation-monitoring.html#multiple-foundations-federation) in
_Configuring Multi-Foundation Monitoring_.
44 changes: 32 additions & 12 deletions configuring/multi-foundation-monitoring.html.md.erb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,19 +13,36 @@ This topic describes how to configure Healthwatch for VMware Tanzu to monitor mu
You can monitor several Ops Manager foundations that have VMware Tanzu Application Service for VMs (TAS for VMs) or VMware Tanzu Kubernetes Grid Integrated
Edition (TKGI) installed from a Healthwatch tile that you install on a separate Ops Manager foundation.

There are two ways to monitor several Ops Manager foundations from a separate Ops Manager foundation:

1. Directly scraping the metric endpoints on the Healthwatch exporter VMs.
* Only requires the `Healthwatch Exporter for TAS` or the `Healthwatch Exporter for TKGI` to be installed on the sub-foundation.
* TKGI cluster metrics requires federation
* Easy to scrape all metrics from a specific exporter.
* All scraped metrics can have a label added to them to specify which foundation they came from.
* Prometheus performance will degrade as additional foundations are monitored due to the overwhelming number of scraped metrics.

1. Federating metrics from the Prometheus instances running on the Ops Manager foundations that will be monitored.
* Required for monitoring `TKGI` clusters as the `Healthwatch` tile dynamically generates scrape configs for Kubernetes clusters.
* Allows for specific metrics to be filtered for efficient scrapes.
* Scrapes a single endpoint which reduces firewall and network complexity.
* All federated metrics can have a label added to them to specify which foundation they came from.
* Recommended to not federate all metrics from a foundation as Prometheus performance will degrade due to the overwhelming number of scraped metrics.

To configure multi-foundation monitoring, see the section for your runtime below:

* [Configure Multi-Foundation Monitoring for TAS for VMs](#multiple-tas-foundations)
* [Configure Multi-Foundation Monitoring Using Direct Scraping](#multiple-foundations-direct-scrape)

* [Configure Multi-Foundation Monitoring for TKGI](#multiple-tkgi-foundations)
* [Configure Multi-Foundation Monitoring Using Federation](#multiple-foundations-federation)


## <a id='multiple-tas-foundations'></a> Configure Multi-Foundation Monitoring for TAS for VMs
## <a id='multiple-foundations-direct-scrape'></a> Configure Multi-Foundation Monitoring Using Direct Scraping

To configure your Healthwatch deployment to monitor TAS for VMs deployments on multiple Ops Manager foundations from a single monitoring Ops Manager
foundation:
The following is an example of how to configure your Healthwatch deployment to monitor multiple Ops Manager foundations
from a single monitoring Ops Manager foundation using direct scrapes. This example references monitoring TAS for VMs
foundations, but the steps can be applied to monitoring downstream TKGI foundations.

1. Install and configure the Healthwatch tile on either an Ops Manager foundation. To install and configure the Healthwatch tile,
1. Install and configure the Healthwatch tile on the monitoring Ops Manager foundation. To install and configure the Healthwatch tile,
see the following topics:
* [Installing a Tile Manually](../installing/installing-manually.html) or [Installing, Configuring, and Deploying a Tile Through an Automated
Pipeline](../installing/automated-pipeline.html)
Expand All @@ -37,16 +54,15 @@ Exporter for TAS for VMs, see the following topics:
Pipeline](../installing/automated-pipeline.html)
* [Configuring Healthwatch Exporter for TAS for VMs](configuring-exporter-tas.html)

1. For each installation of Healthwatch Exporter for TAS for VMs, open the ports for the metric exporter VMs that Healthwatch Exporter for TAS for VMs
1. For each installation of Healthwatch Exporter for TAS for VMs, open the ports in your IAAS for the metric exporter VMs that Healthwatch Exporter for TAS for VMs
deploys. For more information about the ports you must open for each metric exporter VM, see [Networking Rules for Healthwatch Exporter for TAS for
VMs](#network-rules-tas) above.

1. Once you have installed Healthwatch Exporter for TAS for VMs and opened the required ports on each Ops Manager foundation you want to monitor, add a scrape
job for each Healthwatch Exporter for TAS for VMs tile in the **Prometheus** pane of the Healthwatch tile that you installed on your monitoring Ops Manager
foundation. To add a scrape job for a Healthwatch Exporter TAS for VMs tile:
1. Add a scrape job for each Healthwatch Exporter for TAS for VMs tile in the **Prometheus** pane of the Healthwatch tile that you installed on your
monitoring Ops Manager foundation. To add a scrape job for a Healthwatch Exporter TAS for VMs tile:
1. Retrieve the Ops Manager root certificate authority (CA) for the foundation you want to monitor. For more information, see the [Ops Manager
documentation](https://docs.pivotal.io/ops-manager/security/pcf-infrastructure/managing-certificates.html#root-certs).
1. Nagivate to the Ops Manager Installation Dashboard for the foundation you want to monitor.
1. Navigate to the Ops Manager Installation Dashboard for the foundation you want to monitor.
1. Click the **Healthwatch Exporter for Tanzu Application Service** tile.
1. Select the **Credentials** tab.
1. In the row for **Healthwatch Exporter Client Mtls**, click **Link to Credential**.
Expand Down Expand Up @@ -85,7 +101,11 @@ foundation. To add a scrape job for a Healthwatch Exporter TAS for VMs tile:
the metric exporter VMs that Healthwatch Exporter for TAS for VMs deploys.


## <a id='multiple-tkgi-foundations'></a> Configure Multi-Foundation Monitoring for TKGI
## <a id='multiple-foundations-federation'></a> Configure Multi-Foundation Monitoring Using Federation

The following is an example of how to configure your Healthwatch deployment to monitor multiple Ops Manager foundations
from a single monitoring Ops Manager foundation using federation. This example references monitoring TKGI foundations,
but the steps can be applied to monitoring downstream TAS for VMs foundations.

The Healthwatch tile running on a TKGI Ops Manager foundation is able to dynamically find clusters and generate scrape configs for the cluster metrics.
As a result, internal Kubernetes metrics can only be retrieved from a downstream TSDB VM through federation. The Healthwatch TKGI exporters simply provide
Expand Down
24 changes: 10 additions & 14 deletions configuring/optional-config/federation.html.md.erb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,24 +13,19 @@ This topic describes how to configure federation for your Healthwatch for VMware
Healthwatch supports federation. When you configure your Healthwatch installation to federate metrics, the Prometheus instance in the Healthwatch tile on a
monitoring Ops Manager foundation, scrapes a subset of metrics from the Prometheus instances in the Healthwatch tiles installed on the Ops Manager foundations
you monitor. This is useful if you want to monitor a subset of metrics from multiple Ops Manager foundations without storing all metrics from those Ops Manager
foundations in a single Prometheus instance.
foundations in a single Prometheus instance. As federation allows for filtering of the scraped metrics, it preferred over directly scraping metrics in order to
monitor a large number of Ops Manager foundations without overwhelming monitoring Prometheus.

In a typical Healthwatch installation, you install the Healthwatch tile on the monitoring foundation that you use to monitor other foundations, and you
install either the Healthwatch Exporter for VMware Tanzu Application Service for VMs (TAS for VMs) tile or the Healthwatch Exporter for TKGI tile on each
foundation you want to monitor. The Prometheus instance in the Healthwatch tile on your monitoring foundation scrapes metrics from the Healthwatch Exporter
tiles on each foundation you monitor.

To configure federation for your Healthwatch installation, you install the Healthwatch tile on your monitoring foundation and on each foundation you want to
monitor, in addition to the Healthwatch Exporter tile you install on each foundation you want to monitor. You then configure the Healthwatch tile on each
foundation you want to monitor to scrape metrics from the Healthwatch Exporter tile installed on the same foundation, and the Healthwatch tile on your
monitoring foundation to scrape metrics from the Healthwatch tiles installed on the foundations you want to monitor.
To configure federation for your Healthwatch installation, install the Healthwatch tile on your monitoring foundation and on each foundation you want to
monitor, in addition to installing the Healthwatch Exporter tile on each foundation you want to monitor. Then, configure the Healthwatch tile on your
monitoring foundation to federate metrics from the Prometheus installed on the foundations you want to monitor.

For more information about federation, see the [Prometheus documentation](https://prometheus.io/docs/prometheus/latest/federation/).

<p class='note warning'><strong>Warning:</strong> Storing all Loggregator Firehose metrics from more than one large TAS for VMs foundation in a single
Prometheus instance negatively affects the performance of that Prometheus instance, sometimes even causing it to crash. To avoid this, VMware recommends
federating only service level indicator (SLI) metrics from each foundation you monitor to the Prometheus instance in your monitoring foundation. For more
information about SLI metrics for TAS for VMs, see <a href="../../metrics.html#pas-sli-exporter">TAS for VMs SLI Exporter VM</a> in <em>Healthwatch
federating only service level indicator (SLI) metrics from each foundation you want to monitor to the Prometheus instance in your monitoring foundation.
For more information about SLI metrics for TAS for VMs, see <a href="../../metrics.html#pas-sli-exporter">TAS for VMs SLI Exporter VM</a> in <em>Healthwatch
Metrics</em>.</p>


Expand All @@ -46,7 +41,7 @@ Automated Pipeline](../../installing/automated-pipeline.html).
to monitor. For more information, see [Installing a Tile Manually](../../installing/installing-manually.html) or [Installing, Configuring, and Deploying a
Tile Through an Automated Pipeline](../../installing/automated-pipeline.html).

1. For each foundation you want to monitor, expose the Prometheus instance in the Healthwatch tile on port `4450`.
1. For each foundation you want to monitor, open port `4450` in your IAAS for the Prometheus instance in the Healthwatch tile.

1. For each foundation you want to monitor:
1. Navigate to the Ops Manager Installation Dashboard for the foundation you want to monitor.
Expand Down Expand Up @@ -104,7 +99,8 @@ VMs that the Healthwatch Exporter tiles deploy.
When federating metrics, you can configure the Prometheus instance in the Healthwatch tile on your monitoring foundation to scrape both copies of that data
from the Prometheus instance in the Healthwatch tile on each foundation you monitor. To do this, include both VMs in each Prometheus instance from the
foundations you want to monitor in the scrape job configuration parameters. While including both VMs creates duplicate sets of metrics, it also ensures that
you do not lose metric data if one of the two VMs goes down.
you do not lose metric data if one of the two VMs goes down. However, this will negatively affect the performance of Prometheus as the number of scraped metrics
is doubled.

Alternatively, you can create load balancers or DNS entries in your IaaS console for the Prometheus instances on each foundation you monitor, then include the
IP addresses for each load balancer or DNS entry in the targets listed under the `targets` property in your scrape job configuration parameters. For more
Expand Down