…p string

Summary:
So when we first look at the input config, the problem with current logic is that it finds the first instance directly, rather we need to ensure we match via regexnwith whitespace characters

For example:
Validation FAILS:
host all +ldap_service_users 10.0.0.0/8 ldap  ldapurl=ldaps://ldap.dev.schwab.com:636 ldapsearchattribute=""sAMAccountName"" ldapbasedn=""OU=ServiceAccount,DC=csdev,DC=corp""
ldapbinddn=""CN=svc.yb_ldap_dev,OU=ServiceAccount,DC=csdev,DC=corp"" ldapbindpasswd=""Password""

The above is not working as it finds the first instance of ldap being +ldap_service_users which is wrong, the first instance of ldap is one which is the whole word with whitespace characters

Validation SUCCEEDS:
host all +asldwfhhasg 10.0.0.0/8 ldap  ldapurl=ldaps://ldap.dev.schwab.com:636 ldapsearchattribute=""sAMAccountName"" ldapbasedn=""OU=ServiceAccount,DC=csdev,DC=corp"" ldapbinddn=""CN=svc.yb_ldap_dev,OU=ServiceAccount,DC=csdev,DC=corp""
ldapbindpasswd=""Password""

Test Plan:
Please refer to the screenshots
{F170693}

{F170694}

Reviewers: jmak

Reviewed By: jmak

Subscribers: yugaware

Differential Revision: https://phorge.dev.yugabyte.com/D34230

…ount

Summary:
In this issue we see a tserver crash that can be traced to the root cause that
yb-master's catalog version went backwards. Tserver has already seen version
pair of (320, 242) for DB 16429 but later yb-master's catalog version pair for
DB 16429 became (319, 242). Timing wise it coincided with a PITR restore
operation, and we see the following log repeatedly after a PITR restore operation:

```
E0401 18:14:53.567183 31982 tablet_server.cc:917] Ignoring ysql db 16429 catalog
version update: new version too old. New: 319, Old: 320
```

It is unexpected for catalog version to go backwards on a PITR restore operation.

This lasted for more than 20 minutes until a ysql_dump-based restore operation
happened. As part of the restore operation we were running a global-impact DDL
statement:

```
\if :use_roles
    ALTER DATABASE "postgres_88" OWNER TO postgres;
ALTER DATABASE
\endif
```

This DDL will increment catalog versions for all databases, including 16429. Now
the new pair for 16429 becomes (320, 320). Note that the second number is the
breaking version. When it changes it is always equal to the first number. It is
this new pair (320, 320) that caused the tserver to crash because we see the
breaking version associated with 320 is 242 and it is not possible to have
breaking version change given the same current version.

This bug has so far only appeared once in about 80+ runs of the integration
test `test_cross_db_concurrent_ddls`. The root cause is that we have seen master
version going backwards. However we only have a LOG(DFATAL) when that happens.
If it really happens in production environment which has release build,
this can simply go undetected for long time and eventually crashes when a new
breaking DDL statement come to bump up the current catalog version and breaking
version at the same time.

I added a new gflag --ysql_min_new_version_ignored_count that if we see stale
catalog version returned from master for consecutively this many number of
times, then crash the tserver earlier to sync up with master again. This helps
to
* reproduce the bug easier in our integration test (now have --log_ysql_catalog_versions=true)
* avoids running a tserver for too long a time when its catalog version is out
  of sync with master.

This diff is only limited to per-database catalog version mode because we do not
want to change the behavior in global catalog version mode at this time.
Jira: DB-10651

Test Plan:
(1) ./yb_build.sh release --cxx-test pg_catalog_version-test

(2) Manual test
* # create a RF-3 local cluster

  ./bin/yb-ctl create --rf 3

* ./bin/ysqlsh -c "create table foo(id int)"

* # run a DDL that increments DB yugabyte's current_version to 2

  ./bin/ysqlsh -c "alter table foo add column v1 text"

* # verify database yugabyte's current_version is 2

  ./bin/ysqlsh -c "select * from pg_yb_catalog_version"

* # manually force DB yugabyte's version to go back to 1

  ./bin/ysqlsh -c "set yb_non_ddl_txn_for_sys_tables_allowed=1; update pg_yb_catalog_version set current_version = 1"

* Wait and see all 3 tservers crash with the expected log messages:

F0415 23:51:02.940397 29969 tablet_server.cc:924] Ignoring ysql db 13248 catalog version update: new
version too old. New: 1, Old: 2, ignored count: 19
F0415 23:51:05.871065 30011 tablet_server.cc:924] Ignoring ysql db 13248 catalog version update: new
version too old. New: 1, Old: 2, ignored count: 31
F0415 23:51:17.931600 29927 tablet_server.cc:924] Ignoring ysql db 13248 catalog version update: new
version too old. New: 1, Old: 2, ignored count: 48

Reviewers: jason

Reviewed By: jason

Subscribers: ybase, yql

Differential Revision: https://phorge.dev.yugabyte.com/D34146

… SST files only retained for CDC"

Summary:
D33131 introduced a segmentation fault which was  identified in multiple tests.
```
* thread #1, name = 'yb-tserver', stop reason = signal SIGSEGV
  * frame #0: 0x00007f4d2b6f3a84 libpthread.so.0`__pthread_mutex_lock + 4
    frame #1: 0x000055d6d1e1190b yb-tserver`yb::tablet::MvccManager::SafeTimeForFollower(yb::HybridTime, std::__1::chrono::time_point<yb::CoarseMonoClock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>) const [inlined] std::__1::unique_lock<std::__1::mutex>::unique_lock[abi:v170002](this=0x00007f4ccb6feaa0, __m=0x0000000000000110) at unique_lock.h:41:11
    frame yugabyte#2: 0x000055d6d1e118f5 yb-tserver`yb::tablet::MvccManager::SafeTimeForFollower(this=0x00000000000000f0, min_allowed=<unavailable>, deadline=yb::CoarseTimePoint @ 0x00007f4ccb6feb08) const at mvcc.cc:500:32
    frame yugabyte#3: 0x000055d6d1ef58e3 yb-tserver`yb::tablet::TransactionParticipant::Impl::ProcessRemoveQueueUnlocked(this=0x000037e27d26fb00, min_running_notifier=0x00007f4ccb6fef28) at transaction_participant.cc:1537:45
    frame yugabyte#4: 0x000055d6d1efc11a yb-tserver`yb::tablet::TransactionParticipant::Impl::EnqueueRemoveUnlocked(this=0x000037e27d26fb00, id=<unavailable>, reason=<unavailable>, min_running_notifier=0x00007f4ccb6fef28, expected_deadlock_status=<unavailable>) at transaction_participant.cc:1516:5
    frame yugabyte#5: 0x000055d6d1e3afbe yb-tserver`yb::tablet::RunningTransaction::DoStatusReceived(this=0x000037e2679b5218, status_tablet="d5922c26c9704f298d6812aff8f615f6", status=<unavailable>, response=<unavailable>, serial_no=56986, shared_self=std::__1::shared_ptr<yb::tablet::RunningTransaction>::element_type @ 0x000037e2679b5218) at running_transaction.cc:424:16
    frame yugabyte#6: 0x000055d6d0d7db5f yb-tserver`yb::client::(anonymous namespace)::TransactionRpcBase::Finished(this=0x000037e29c80b420, status=<unavailable>) at transaction_rpc.cc:67:7
```
This diff reverts the change to unblock the tests.

The proper fix for this problem is WIP
Jira: DB-10780, DB-10466

Test Plan: Jenkins: urgent

Reviewers: rthallam

Reviewed By: rthallam

Subscribers: ybase, yql

Differential Revision: https://phorge.dev.yugabyte.com/D34245

* Sandbox support for Innovation track

* typo

…from multiple tablets of a tserver leading to undetected deadlocks

Summary:
The local waiting txn registry at the Tablet Server maintains wait-for dependencies arising at all tablet leaders hosted on the node. When a request is processed at the wait-queue, the local registry sends a partial update containing just the wait-for dependencies of the request. The registry keeps accumulating all dependencies, and periodically sends full update requests which comprise of wait-for dependencies of all outstanding requests.

`UpdateTransactionWaitingForStatusRequestPB` proto is used for sending the wait-for dependency info. Each `WaitingTransaction` in `UpdateTransactionWaitingForStatusRequestPB` has a `wait_start_time` which is populated at the local registry and is set to `clock_->Now()`.

The deadlock detector maintains a multi container `waiters_` indexed by key pair <txn_id, tserver_uuid>. In the existing implementation, the detector overwrites the wait-for dependencies of a waiter when it encounters a `WaitingTransaction` with a later timestamp than the existing one. Since multiple requests of the same txn (at the same tablet or at multiple tablets) with different blockers can exist at a given time, it led to incomplete wait-for info at the detector, thus resulting in undetected deadlocks.

This diff addresses the issue by changing the logic at the detector to keep track of all dependencies of the waiter and not overwrite it based on start time. Each time the detector sees a `WaitingTransaction`, it triggers probes for just the new wait-for dependencies contained in the message and appends the blocker info to the existing waiter record (if any). Additionally, we propagate the request id info of the waiter to the deadlock detector, and store a list of request ids blocked on the `{blocker_id, subtxn, status_tablet}` tuple.

Note that the diff doesn't change the periodic deadlock probing algorithm at the detector.

Sample vlogs from the detector for one of the below tests
```
[ts-1] I0416 05:31:20.734838 129157 deadlock_detector.cc:611] vlog4: T 9eed4e1f5b2745aeb8333e8fe0fd9c61 D c78320b1-e14c-44ba-b6fe-1518c4be25d2 Adding new wait-for relationship -- waiter txn id: 631c3948-6816-403c-890a-95560fc4615f blocker id: 6cb37588-3d67-4301-8336-cf944222cdab, status tablet: 9eed4e1f5b2745aeb8333e8fe0fd9c61, blocking subtxn info: [2, 2], waiting_requests (id, start_time): [{11, { days: 19829 time: 05:31:20.733994 }}] received from TS: 25db8deb4335401ab1261c01e8634ab8
[ts-1] I0416 05:31:20.735044 129157 deadlock_detector.cc:629] vlog4: T 9eed4e1f5b2745aeb8333e8fe0fd9c61 D c78320b1-e14c-44ba-b6fe-1518c4be25d2 Updated blocking data -- txn_id_: 631c3948-6816-403c-890a-95560fc4615f, tserver_uuid_: 25db8deb4335401ab1261c01e8634ab8, waiter_data_: [blocker id: b9356bf9-6bb6-4ef6-be4e-96dcfec46ddd, status tablet: a8643356f0e24c1a836b029f1bea0dfa, blocking subtxn info: [4, 4], waiting_requests (id, start_time): [{10, { days: 19829 time: 05:31:20.733203 }}], blocker id: 6cb37588-3d67-4301-8336-cf944222cdab, status tablet: 9eed4e1f5b2745aeb8333e8fe0fd9c61, blocking subtxn info: [2, 2], waiting_requests (id, start_time): [{11, { days: 19829 time: 05:31:20.733994 }}, {9, { days: 19829 time: 05:31:20.727855 }}]]
[ts-1] I0416 05:31:20.735205 129157 deadlock_detector.cc:244] vlog4: T 9eed4e1f5b2745aeb8333e8fe0fd9c61 D c78320b1-e14c-44ba-b6fe-1518c4be25d2 - probe(c78320b1-e14c-44ba-b6fe-1518c4be25d2, 1) AddBlocker: waiting_txn_id: 631c3948-6816-403c-890a-95560fc4615f, blocker id: b9356bf9-6bb6-4ef6-be4e-96dcfec46ddd, status tablet: a8643356f0e24c1a836b029f1bea0dfa, blocking subtxn info: [4, 4], waiting_requests (id, start_time): [{10, { days: 19829 time: 05:31:20.733203 }}], probe_num: 1, min_probe_num: 0
[ts-1] I0416 05:31:20.735318 129157 deadlock_detector.cc:244] vlog4: T 9eed4e1f5b2745aeb8333e8fe0fd9c61 D c78320b1-e14c-44ba-b6fe-1518c4be25d2 - probe(c78320b1-e14c-44ba-b6fe-1518c4be25d2, 1) AddBlocker: waiting_txn_id: 631c3948-6816-403c-890a-95560fc4615f, blocker id: 6cb37588-3d67-4301-8336-cf944222cdab, status tablet: 9eed4e1f5b2745aeb8333e8fe0fd9c61, blocking subtxn info: [2, 2], waiting_requests (id, start_time): [{11, { days: 19829 time: 05:31:20.733994 }}, {9, { days: 19829 time: 05:31:20.727855 }}], probe_num: 1, min_probe_num: 0
```

Test Plan:
Jenkins

./yb_build.sh --cxx-test pgwrapper_pg_wait_on_conflict-test --gtest_filter PgWaitQueueRF1Test.TestDeadlockAcrossMultipleTablets -n 20
./yb_build.sh --cxx-test pgwrapper_pg_wait_on_conflict-test --gtest_filter PgWaitQueueRF1Test.TestDetectorPreservesBlockerSubtxnInfo -n 20
./yb_build.sh --cxx-test='TEST_F(UnsignedIntSetTest, Hash) {'

test 1 fails consistently prior to this diff - w waits on blockers b1 and b2. the test ensures that the detector doesn't erase w->b1 on seeing w->b2)
test 2 ensures that the detector doesn't overwrite blocking subtxn info of a given blocker - w waits on b1(subtxn 2) and b1(subtxn 3). the test asserts that the detector doesn't erase w->b1(subtxn 2) on seeing w->b1(subtxn 3).

Reviewers: rsami

Reviewed By: rsami

Subscribers: yql, ybase

Differential Revision: https://phorge.dev.yugabyte.com/D33641

Summary:
Audit logs were not exported from newly added read replica cluster. We were sending `AuditLogConfig` as null when calling the read replica API. Since the primary cluster already has the `AuditLogConfig`, we should be using that instead when provisioning the new nodes.

The following flow didn't work before, which does now:
```
Create a universe without RR.
Enable DB audit logging.
Add RR to this universe.
Verify --> Audit logs are not visible on DD
```

Test Plan:
Manually tested the following flow:
Tried the following scenarios:
Case 1:
```
Create a universe with primary cluster and RR.
Enable DB audit logs.
Verify. ---> Works as expected. Audit logs from primary cluster nodes and RR nodes are visible on DD.
```

Case 2:
```
Create a universe without RR.
Enable DB audit logging.
Add RR to this universe.
Verify --> Works as expected. Audit logs from primary cluster nodes and RR nodes are visible on DD.
```

Case 3 (Patched this diff on my diff: https://phorge.dev.yugabyte.com/D33949):
```
Create a universe without RR.
Enable DB audit logging.
Add RR to this universe.
Verify --> Works as expected. Audit logs from primary cluster nodes and RR nodes are visible on DD.
Add new node to universe.
Verify --> Works as expected. Audit logs from both primary cluster nodes and RR nodes are visible on DD.
```

Reviewers: amalyshev

Reviewed By: amalyshev

Subscribers: yugaware

Differential Revision: https://phorge.dev.yugabyte.com/D33995

Summary:
In case the AMI is not present for a region in bundle, we were falling
back to retrieve the same from the region.
This diff makes changes as -
1. For YBA managed bundles - we will read the AMIs from the YBA metadata in
case they are not present in the bundle.

2. For Custom bundles - We will fail without having any fall back mechanism.

Removes the dependency on region -> ybImage.

Test Plan:
Created a provider with custom bundle.
Removed the ybImage from the bundle.
Deployed the universe using the same. Verified that it failed.

Created a provider with YBA managed bundles.
Removed the ybImage from the bundle.
Deployed the universe. Verified that it picks up from the YBA's image metadata.

Reviewers: amalyshev, nbhatia

Reviewed By: amalyshev

Subscribers: yugaware

Differential Revision: https://phorge.dev.yugabyte.com/D34119

Summary:
Special characters like `$` get escaped when provided via the flags for passwords. The help text
indicates using `''` quotes to provide these values to be parsed correctly

Test Plan: Test create universe and ysql connection with single quotes

Reviewers: skurapati, rohita.payideti

Reviewed By: rohita.payideti

Subscribers: yugaware

Differential Revision: https://phorge.dev.yugabyte.com/D34264

Test Plan:
Start tserver with rpc bind and http addresses set differently and verify that the display shows this. Tested with --rpc_bind_addresses set to 0.0.0.0 and verified that the hostname shows up instead (this seems to depend on whether ybdb can discover a local hostname)

{F166067}

Reviewers: hsunder

Reviewed By: hsunder

Subscribers: hsunder, esheng, ybase, bogdan

Differential Revision: https://phorge.dev.yugabyte.com/D33726

Summary:
Created new class to exclude base upgrade tasks from unit tests

The general idea is to distinguish checking of basic subtasks (like `WaitForServer` etc) from logic that is specific to particular upgrade.
In this approach each upgrade test will check only actual upgrade subtasks (like `InstanceActions`) and nodes which these actions are applied to.

And there should also be test, that tests basic sequence (but currently is missing in this diff!)

The problem with approach in master - all upgrade tests check both things, and we have to modify all of tests if we alter basic logic.

Test Plan: sbt test

Reviewers: nsingh, sanketh

Reviewed By: nsingh

Subscribers: yugaware

Differential Revision: https://phorge.dev.yugabyte.com/D33032

…rver proxy if not using a distributed transaction

Summary:
Before this revision, every RollbackToSubTransaction operation in PG would lead to a corresponding RPC call to the local tserver. The local tserver used to
return early in case there was no distributed transaction.

This revision adds the logic in the PG layer (pg_session) to skip sending the RPC if the transaction is read-only or a fast-path transaction i.e., has NON_TRANSACTIONAL isolation level. Note that we were already doing that for transaction commit/aborts but weren't skipping the RPC for rollback of sub-transaction.

This change was proposed as part of the implementation of the PG compatible logical replication support. While streaming the changes to the Walsender, it starts and aborts transactions for every transaction that gets streamed. This is required for reading PG catalog tables. As a result, we were seeing a lot of unnecessary RPC calls to the local tserver.
Jira: DB-10402

Test Plan: All tests

Reviewers: asrinivasan, pjain

Reviewed By: pjain

Subscribers: yql

Differential Revision: https://phorge.dev.yugabyte.com/D34162

Summary: This diff enables the OS Patching runtime flag by default.

Test Plan: iTest pipeline

Reviewers: amalyshev, nbhatia, #yba-api-review!

Reviewed By: amalyshev

Subscribers: yugaware

Differential Revision: https://phorge.dev.yugabyte.com/D34184

…record if needed

Summary:
When there's a `GetChanges` request (req_1) and service layer receives a `CacheMissError`, it refetches the enum labels and executes a new internal `GetChanges` request (req_2) for a fresh `GetChangesResponse`.

Now suppose this is the first `GetChanges` request from the connector where it still hasn't received the DDL record, after the service clears the response, it looks at the `cached_schema_details` object while making `req_2` to decide whether or not to publish the DDL record. But since we have already populated the `cached_schema_details` while processing `req_1`, it will mean that we do not populate the DDL record and thus the client will not receive the DDL record in `GetChangesResponse` causing it to fail while decoding further change events.

**Solution:**

This diff implements a simple solution by clearing the `cached_schema_details` while executing `req_2` if the connector/client has indicated that it needs the schema i.e. if `req->need_schema_info() == true`.
Jira: DB-9701

Test Plan:
```
./yb_build.sh --cxx-test cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestPopulationOfDDLRecordUponCacheMiss
```

Reviewers: skumar, asrinivasan, stiwary

Reviewed By: skumar

Subscribers: ycdcxcluster

Differential Revision: https://phorge.dev.yugabyte.com/D34107

Summary:
This change addresses a bug introduced in diff D32566 that caused some tablet metrics to have the wrong metric_type. It also fixes a pre-existing issue with how metric attributes are stored.

**Root Cause:**
D32566 started storing table metrics into the aggregation map for grouping purposes. (Previously, table metrics were flushed directly if they were at the table level as no aggregation needed). Because of this, It also started saving the attributes of these table metrics in an attributes map with their entity_id, which is table_id, as the key.

However, there were two problems:
- Table level Attribute Collision: Both table and tablet metrics used the table_id as the key for storing their attributes, leading to collisions and incorrect attributes when aggregating them at table level.
- Potential Pre-existing Stream level Attribute Collision: Even before D32566, using entity_id as the key wasn't ideal because some metrics like XClusterMetric and CdcsdkMetric have different attribute structures despite having the same entity_id (stream_id in this case).

**Fix:**
This change addresses both issues by storing metric attributes with a composite key consisting of:
- metric_type: Identifies the specific type of metric (e.g., XClusterMetric, CdcsdkMetric).
- entity_id: Identifies the entity the metric belongs to (e.g., table_id, stream_id).
This approach ensures unique keys for storing metric attributes and avoids collisions based solely on entity_id.
Jira: DB-10501

Test Plan:
Jenkins: urgent

To verify the fix addresses both issues, a `DCHECK` was added in `PrometheusWriter::AddAggregatedEntry`. This check compares the stored attribute map with the incoming attribute map. If there's a mismatch, it indicates a collision. This DCHECK effectively covers both scenarios:
- Table level Attribute Collision: Detected by `PrometheusMetricFilterTest.TestV1Default`
- Potential Pre-existing Stream level Attribute Collision: Detected by `MetricsTest.VerifyHelpAndTypeTags`

Reviewers: mlillibridge, rthallam

Reviewed By: mlillibridge

Subscribers: bogdan, ybase

Differential Revision: https://phorge.dev.yugabyte.com/D33396

* partition by region node settings

* screen shot

* update CLI

* update CLI

* minor edit

* review comments

* cli help edits

* update screenshots

* remove redis/yedis references from docs

* remove old realworld apps

* Apply suggestions from code review

Co-authored-by: Dwight Hodge <79169168+ddhodge@users.noreply.github.com>

* fix broken links

* fix external links

* fix link

---------

Co-authored-by: Dwight Hodge <79169168+ddhodge@users.noreply.github.com>
Co-authored-by: Dwight Hodge <ghodge@yugabyte.com>

…tional replication setup

Summary:
This commit modifies the behavior when a user adds a YSQL table to a bidirectional replication. With these changes, the bootstrapping process is always skipped when adding a table to a bidirectional replication, regardless of whether it is required or not.

The detection of bidirectional replication operates at the database granularity. This means that when adding a table to a replication, the replication is considered bidirectional if any sibling table (i.e., other tables within the same database as the table being added) is already part of a bidirectional replication.

Note: because the bootstrapping is skipped, it will be the responsibility of the user to ensure the existing data are copied over.

Test Plan:
- Made sure the user is able to add tables to a bidirectional replication, no matter it requires bootstrapping or not.
- Made sure for unidirectional replication, it does bootstrapping if required (previous behavior).

Reviewers: #yba-api-review, cwang, jmak, sanketh

Reviewed By: #yba-api-review, sanketh

Subscribers: yugaware

Differential Revision: https://phorge.dev.yugabyte.com/D34228

…smatch

Summary:
With changes to send backup code sending backups to followers of lower YBA version is now an error, which meant we weren't syncing the config. As part of config sync we generate version mismatch events. If we try to sync the config to all instances we will both update the config correctly so standby shows correct state even when it is a lower version and the active correctly fires alert to upgrade standby.

moving backup update code when we send backup

simplifying code

Test Plan: Setup HA, upgrade standby, then promote. Ensure that alert fires and config looks correct on standby.

Reviewers: dshubin, sanketh

Reviewed By: sanketh

Subscribers: yugaware

Differential Revision: https://phorge.dev.yugabyte.com/D34125

…elCache failure

Summary:
`CREATE PUBLICATION FOR ALL TABLES` invalidates entire relcache (via CacheInvalidateRelcacheAll) and hence is global impact DDL. This test fails as a result.

This diff fixes the test failure by checking the PG backend version via `SELECT version()` and adjust the test result depending on whether it is PG11 or PG15.
Jira: DB-10952

Test Plan:
In both master (PG11) and PG15 branches, apply the diff patch and run:

./yb_build.sh release --cxx-test pg_catalog_version-test --gtest_filter PgCatalogVersionTest.InvalidateWholeRelCache

Reviewers: aagrawal

Reviewed By: aagrawal

Subscribers: jason, yql

Differential Revision: https://phorge.dev.yugabyte.com/D34262

…@yugabyte-ui-common-component library

Summary:
Handle ASH as special case as it has OUTLIER style buttons in case of OVERALL mode
Ensure graph API call is made when WAIT EVENT or WAIT EVENT CLASS or WAIT EVENT COMPONENT is selected

Test Plan: Tested locally via TS Web UI

Reviewers: amalyshev, cdavid

Reviewed By: cdavid

Subscribers: yugaware

Differential Revision: https://phorge.dev.yugabyte.com/D34243

Summary:
We have timezone related bug in ASH retrieval code, which breaks retrieval.
Also, old code could skip some sample events.
It's fixed  now.
Also it makes universe details retrieval sync with universe metadata update.

Test Plan: Unit tested + tested ASH retrieval manually

Reviewers: rmadhavan, cdavid

Reviewed By: rmadhavan, cdavid

Subscribers: yugaware

Differential Revision: https://phorge.dev.yugabyte.com/D34236

…tIndex

Summary:
Use std::upper_bound instead of std::lower_bound, which allows to find
the answer in one statement.

Move sanity check forward, so error messages are more accurate
Jira: DB-10789

Test Plan: ./yb_build.sh --cxx-test client_client-test

Reviewers: arybochkin, dmitry, mlillibridge, timur

Reviewed By: arybochkin

Subscribers: ybase, yql

Tags: #jenkins-ready

Differential Revision: https://phorge.dev.yugabyte.com/D33935

…eys when yb.cloud.enabled is true

Summary: This is a workaround to let itests pass. The way we are reading YBM enabled is messy. Another pass to make it uniform can be done that may require validating all the paths work.

Test Plan: Itest should pass.

Reviewers: cwang, yshchetinin, sanketh, kvikraman

Reviewed By: yshchetinin

Subscribers: yugaware

Differential Revision: https://phorge.dev.yugabyte.com/D34257

… log anchor session

Summary:
When RBS is done from a non-leader peer, the rbs source creates a session id of the form `<requestor_uuid>-<tablet_id>-<MonoTime::Now()>`. It sends back the same identifier to the destination node, and the rbs destination node uses this for the subsequent calls. The same is used for propagating the log anchor information to the leader peer, i.e we use this session id in `RegisterLogAnchorRequestPB`.

While creating a session for anchoring the log on the leader, the logic was similar to the below
```
auto tablet_peer_result = tablet_peer_lookup_->GetServingTablet(req->tablet_id());
...
auto it = log_anchors_map_.find(req->owner_info());
if (it == log_anchors_map_.end()) {
  ...
} else {
  tablet_peer.reset(it->second->tablet_peer_.get());         // <- this line creates a problem
}
```
When re-using the session, `tablet_peer.reset` takes ownership of the underlying manager `TabletPeer` object and fails to consider the existing shared_ptrs. So once it goes out of the scope (function `RemoteBootstrapServiceImpl::RegisterLogAnchor`), `~TabletPeer()` is called on the underlying object, which leads to the below fatal.
```
../../src/yb/tablet/maintenance_manager.cc:101] Check failed: !manager_.get() You must unregister the LogGCOp(7b72199ed9714f649d10be762212950d) Op before destroying it.
```

This diff addresses the issue by using `=` operator which rightly tracking the existing shared_pts as well, and doesn't destruct the underlying object once `tablet_peer` goes out of scope.

Note: To repro this in a test, `RemoteBootstrapServiceImpl::RegisterLogAnchor` should be called with the same `owner_info` set in `RegisterLogAnchorRequestPB`. But that is only possible when the rbs source re-uses its rbs session, whose session id is computed with a suffix of `MonoTime::Now()`. So wasn't able to simulate the above crash in a test, but this was observed in the logs reported by in the community forum.
Jira: DB-10926

Test Plan: Jenkins

Reviewers: amitanand

Reviewed By: amitanand

Subscribers: ybase

Differential Revision: https://phorge.dev.yugabyte.com/D34205

Summary:
Show release date as empty string in release details
Release date in general for customers will never be empty, it will always have a date, but in case of internal usage, release date for most of the dev builds will be empty

Test Plan:
Please refer to the screenshot
{F171445}

Reviewers: jmak, dshubin

Reviewed By: dshubin

Subscribers: yugaware

Differential Revision: https://phorge.dev.yugabyte.com/D34291

…eUniverseReplicationRequestPB

Summary:
For xCluster DR we want to be able to failover very quickly. DeleteUniverseReplication cleans up the streams on the source, which will timeout since the source is unavailable during failover.
When `skip_producer_stream_deletion` is set on `DeleteUniverseReplicationRequestPB` we will skip the cleanup process.

**Upgrade/Rollback safety:**
New field is optional and false by default, so safe for upgrade and rollbacks.

Fixes yugabyte#22050
Jira: DB-10965

Test Plan: XClusterTest.DeleteWithoutStreamCleanup

Reviewers: slingam, jhe, xCluster

Reviewed By: slingam

Subscribers: ybase, xCluster

Differential Revision: https://phorge.dev.yugabyte.com/D34286

Summary:
All connections that `postgres_fdw` establishes to foreign servers are kept open in the local session for re-use.
With option `use_remote_estimate true` specified during a foreign table's creation, when PG estimates the cost of the foreign table, it executes a SQL statement remotely using the existing open connection to the foreign server where the foreign table resides.
With changes made in commit 9a27aff, open PG connections need to refresh catalog cache because ANALYZE increments catalog version.
Thus, the plan in test `TestPgRegressContribPostgresFdw` changed based on cost because open connections use up-to-date statistics instead of stable statistics.
Jira: DB-10738

Test Plan: ./yb_build.sh --java-test 'org.yb.pgsql.TestPgRegressContribPostgresFdw'

Reviewers: tverona, myang

Reviewed By: myang

Subscribers: jason, yql

Tags: #jenkins-ready

Differential Revision: https://phorge.dev.yugabyte.com/D34071

…d tservers do not have any tablets assigned to them

Summary: Verify that tablet count is zero on the blacklisted nodes after wait for data move. Also, made a fix to host port comparison for YBM dual NIC.

Test Plan:
1. Create a universe.
2. Run full move (2 times for previously blacklisted nodes not in the universe).
3. Verified the log messgaes.

```
2024-04-15T20:51:17.743Z  [debug] 3525d0d2-3c25-4cb7-b717-35f2f2dbeb6a UniverseTaskBase.java:2207 [TaskPool-EditUniverse(f6314ef5-7324-4671-a335-eea42bc4f758)-3] com.yugabyte.yw.commissioner.tasks.UniverseTaskBase Making url request to endpoint: http://10.9.120.245:7000/dump-entities
2024-04-15T20:51:18.891Z  [info]  AsyncYBClient.java:2758 [yb-nio-1] org.yb.client.AsyncYBClient Discovered tablet YB Master for table YB Master with partition ["", "")
2024-04-15T20:51:18.940Z  [debug] 3525d0d2-3c25-4cb7-b717-35f2f2dbeb6a UniverseTaskBase.java:2271 [TaskPool-EditUniverse(f6314ef5-7324-4671-a335-eea42bc4f758)-3] com.yugabyte.yw.commissioner.tasks.UniverseTaskBase Number of tablets on tserver yb-admin-nsingh-test-universe1-n1 is 0 tablets
2024-04-15T20:51:18.940Z  [debug] 3525d0d2-3c25-4cb7-b717-35f2f2dbeb6a UniverseTaskBase.java:2271 [TaskPool-EditUniverse(f6314ef5-7324-4671-a335-eea42bc4f758)-3] com.yugabyte.yw.commissioner.tasks.UniverseTaskBase Number of tablets on tserver yb-admin-nsingh-test-universe1-n2 is 0 tablets
2024-04-15T20:51:18.940Z  [debug] 3525d0d2-3c25-4cb7-b717-35f2f2dbeb6a UniverseTaskBase.java:2271 [TaskPool-EditUniverse(f6314ef5-7324-4671-a335-eea42bc4f758)-3] com.yugabyte.yw.commissioner.tasks.UniverseTaskBase Number of tablets on tserver yb-admin-nsingh-test-universe1-n3 is 0 tablets
2024-04-15T20:51:18.940Z  [debug] 3525d0d2-3c25-4cb7-b717-35f2f2dbeb6a UniverseTaskBase.java:2271 [TaskPool-EditUniverse(f6314ef5-7324-4671-a335-eea42bc4f758)-3] com.yugabyte.yw.commissioner.tasks.UniverseTaskBase Number of tablets on tserver yb-admin-nsingh-test-universe1-n4 is 0 tablets

```

Also tested with on-prem.
1. Create an onprem universe.
2. Run full move. The old nodes are DEAD but not blacklisted.
3. Run ybadmin command to blacklist the old nodes and make sure from the master leader UI that the DEAD node is blacklisted.
4. Run full move again. It completed successfully.

Reviewers: cwang, sanketh, yshchetinin

Reviewed By: cwang, yshchetinin

Subscribers: yugaware

Differential Revision: https://phorge.dev.yugabyte.com/D34140

…view Flag

Summary:
Convert to enable_xcluster_api_v2 to a Preview Flag
Jira: DB-10928

Test Plan: Jenkins

Reviewers: slingam, xCluster

Reviewed By: slingam

Subscribers: ybase

Differential Revision: https://phorge.dev.yugabyte.com/D34295

…y test with Connection Manager enabled

Summary:
In the test `org.yb.pgsql.TestYbPgStatActivity.testMemUsageOfQueryFromPgStatActivity`, we check the RSS memory consumed in a session before and after doing certain operations, with the help of a second connection.
With YSQL Connection Manager enabled, both the connections would use the same physical connection, defeating the purpose of running this test. This patch ensures to skip this test whenever Connection Manager is enabled at the time of running the test.
Jira: DB-10907

Test Plan:
Ensure below test is skip when executed:
```./yb_build.sh --enable-ysql-conn-mgr-test --java-test org.yb.pgsql.TestYbPgStatActivity#testMemUsageOfQueryFromPgStatActivity```

Reviewers: rbarigidad

Reviewed By: rbarigidad

Subscribers: yql

Differential Revision: https://phorge.dev.yugabyte.com/D34096

…n Manager enabled

Summary:
With Ysql Connection Manager, before each transaction set stmts are being executed to set the session variables corresponding to a particular client. In this test (`org.yb.pgsql.TestPgConfiguration#testTimezoneFlag`) TimeZone flag is also being set before the transaction, due to which the source value in pg_settings view is set to `session` instead of `client`.
Jira: DB-10392

Test Plan:
Ensure test is running is successfully.
```./yb_build.sh --enable-ysql-conn-mgr-test --java-test org.yb.pgsql.TestPgConfiguration#testTimezoneFlag```

Reviewers: rbarigidad

Reviewed By: rbarigidad

Subscribers: rbarigidad, nkumar, yql

Differential Revision: https://phorge.dev.yugabyte.com/D34095

…flush to indicate the units

Summary:
Rename gflag min_segment_size_to_rollover_at_flush to indicate the units. Also update the description to indicate the same.
Jira: DB-10726

Test Plan: Jenkins

Reviewers: rthallam

Reviewed By: rthallam

Subscribers: ybase

Differential Revision: https://phorge.dev.yugabyte.com/D34284

…assthrough

Summary:
In pg_backend, during authentication the user's details/credentials are fetched via the `SearchCatCacheInternal` function, which first searches the cache locally then on cache miss fetches this details from the yb-tserver.
In case of normal authentication this cache is empty, thus the user's credentials are fetched from the tserver everytime. But in case of authentication passthrough this cache may not be empty. It may be filled with an outdates user details/credentials.

This diff contains changes to skip local cache lookup during authentication passthrough by invalidating the local catalogue cache.
Alternatives of this design:
- Increase the catalogue version.
- Invalidate the cache specific to the user credentials
Jira: DB-10567

Test Plan:
Ensure that the `org.yb.pgsql.TestPgAuthorization.testLongPasswords` is passing with ysql connection manager enabled in the test.
`./yb_build.sh debug --java-test org.yb.pgsql.TestPgAuthorization#testLongPasswords --enable-ysql-conn-mgr-test`

Reviewers: nkumar, rbarigidad, smishra, tverona, mihnea, jason

Reviewed By: nkumar, jason

Subscribers: mihnea, tverona, yql

Differential Revision: https://phorge.dev.yugabyte.com/D33463

…eta map

Summary:
While shipping non-empty response from Virtual WAL, we were logging number of unacked txns in the VWAL by reading the commit_meta map. But when walsender sends a restart_lsn > last shipped commit's lsn, this map is cleared. Now suppose in the next GetConsistentChanges call, we could only ship DDL records. In this case, since the response is non-empty, we'll try to compute unacked txns by reading the empty commit_meta map, which is leading to ASAN error.
Jira: DB-10202

Test Plan: Jenkins: test regex: .*CDCSDKConsumptionConsistentChangesTest.*

Reviewers: stiwary, asrinivasan

Reviewed By: stiwary

Subscribers: ycdcxcluster

Tags: #jenkins-ready

Differential Revision: https://phorge.dev.yugabyte.com/D34310

…s function for debugging

Summary:
This revision introduces more VLOG statements in the ListReplicationSlots function in pg_client_service. This will aid us in debugging the issues observed
while reading the CDC state table from the tablet server.
Jira: DB-10546

Test Plan:
Jenkins: compile only

Only VLOG statements have been added and I've run `TestPgReplicationSlot` test manually. No need to run Jenkins tests for this change.

Reviewers: asrinivasan

Reviewed By: asrinivasan

Subscribers: ybase, yql

Differential Revision: https://phorge.dev.yugabyte.com/D34320

Removed customer name.

…tream_id & session_id

Summary:
This diff does the following:
1. Prefixes all logs & vlogs in the Virtual WAL with stream_id and session_id in the manner `VWAL [Stream_id:Session_id]`. Following is an example of how the logs will look like:

```
cdcsdk_virtual_wal.cc:521] vlog1: VWAL [d06f997aa6be5dafca4509e4424670cd:9223372036854775807]: Sending non-empty GetConsistentChanges response with total_records: 104, total_txns: 2, min_txn_id: 17, max_txn_id: 18, min_lsn: 782, max_lsn: 885, is_last_txn_fully_sent: true, begin_records: 2, commit_records: 2, insert_records: 100, update_records: 0, delete_records: 0, ddl_records: 0, contains_publication_refresh_record: 0, VWAL lag: 54681 ms, Number of unacked txns in VWAL: 2
```
2. Store stream_id & session_id in the vwal instance and remove passing stream_id as parameters to VWAL methods.
3. Fixes computation of time spent in GetConsistentChangesInternal(). Before this diff, we were using CoarseMonoClock to compute time spent in GetConsistentChangesInternal() but to compute time spent in GetChangesInternal(), we were using GetCurrentTimeMicros(). As per the code flow, time spent in GetConsistentChangesInternal >= time spent in GetChangesInternal. But I observed in logs, this semantics was not followed sometimes while shipping empty response. To fix this, now we'll use GetCurrentTimeMicros() to compute time spent in GetConsistentChangesInternal().

```
vlog1: Total time spent in processing GetConsistentChanges (GetConsistentChangesInternal) is: 0 microseconds, out of which the time spent in GetChangesInternal is: 332 microseconds.
```
Jira: DB-10202

Test Plan: Jenkins: test regex: .*CDCSDKConsumptionConsistentChangesTest.*

Reviewers: stiwary, asrinivasan

Reviewed By: stiwary

Subscribers: ycdcxcluster

Tags: #jenkins-ready

Differential Revision: https://phorge.dev.yugabyte.com/D34299

* incremental backups

* review comment

* screenshot

* review comment

* minor edits

* minor fix

…upporting graph

Summary: Ensure ASH works in case of Main graph in addition to supporting graph

Test Plan: Ensure ASH works in case of Main graph in addition to supporting graph

Reviewers: cdavid, amalyshev

Reviewed By: cdavid

Subscribers: yugaware

Differential Revision: https://phorge.dev.yugabyte.com/D34301

Summary:
A Resource consists of a set of user-defined properties and system-generated properties. These are called as ResourceSpec and ResourceInfo respectively. So a `GET /resource` returns a structure that contains both -
```
{
    ResourceSpec: {
        name: "resource-name",
        …
    }
    ResourceInfo: {
        uuid: "…",
        …
    }
}
```

During creation of a Resource, there may be additional one-time properties to be set by the user. Credentials is one such example. So the ResourceCreateRequest looks like this -
```
{
    ResourceCreateRequest: {
        credentials: {
        },
        another_create_time_property: {
        },
        ResourceSpec: {
        }
    }
}
```

Not all user defined properties of a resource can be edited. It would mostly be a subset of properties of ResourceSpec. So the payload for edit operations on a Resource would look like -
```
{
    ResourceEditSpec: {
        prop1: ...,
        // sub set of properties from ResourceSpec that are editable
    }
}
```

With these guidelines, this change implements the GET payload for Universe resource. The complete structure looks like this.
```
{
  "spec" : {
    "name" : "Test-Universe-2",
    "arch" : null,
    "communication_ports" : {
      "master_http_port" : 7000,
      "master_rpc_port" : 7100,
      "node_exporter_port" : 9300,
      "otel_collector_metrics_port" : 0,
      "redis_server_http_port" : 11000,
      "redis_server_rpc_port" : 6379,
      "tserver_http_port" : 9000,
      "tserver_rpc_port" : 9100,
      "yb_controller_http_port" : 14000,
      "yql_server_http_port" : 12000,
      "yql_server_rpc_port" : 9042,
      "ysql_server_http_port" : 13000,
      "ysql_server_rpc_port" : 5433
    },
    "encryption_at_rest_config" : {
      "kms_config_uuid" : null,
      "type" : "DATA_KEY",
      "op_type" : "UNDEFINED"
    },
    "encryption_in_transit_spec" : {
      "enable_node_to_node_encrypt" : false,
      "enable_client_to_node_encrypt" : false
    },
    "node_exporter_user" : "prometheus",
    "ysql" : {
      "enable" : true,
      "enable_auth" : false
    },
    "ycql" : {
      "enable" : true,
      "enable_auth" : false
    },
    "remote_package_path" : "",
    "override_prebuilt_ami_db_version" : false,
    "otel_collector_enabled" : false,
    "clusters" : [ {
      "uuid" : "24e2efba-9972-4ebe-8551-23977c459064",
      "cluster_type" : "PRIMARY",
      "num_nodes" : 3,
      "replication_factor" : 3,
      "instance_type" : "m3.medium",
      "yb_software_version" : "2.17.0.0-b1",
      "storage_spec" : {
        "volume_size" : 150,
        "num_volumes" : 2,
        "mount_points" : "/mnt/d0",
        "storage_class" : "",
        "storage_type" : "GP2"
      },
      "networking_spec" : {
        "assign_public_ip" : true,
        "assign_static_public_ip" : false,
        "enable_exposing_service" : "UNEXPOSED",
        "enable_ipv6" : false,
        "enable_lb" : false
      },
      "provider_spec" : {
        "provider" : "7154f5af-1148-484a-aa77-88ddf7fcf6fd",
        "region_list" : [ "bd86b4f1-9662-4430-9f24-59bebafa9fdd" ],
        "access_key_code" : "mock-access-code-key"
      },
      "use_time_sync" : false,
      "use_spot_instance" : false,
      "use_systemd" : false,
      "instance_tags" : {
        "tag2" : "value2",
        "tag1" : "value1"
      },
      "gflags" : {
        "tserver" : {
          "tflag2" : "tval2",
          "tflag1" : "tval1"
        },
        "master" : {
          "mflag2" : "mval2",
          "mflag1" : "mval1"
        },
        "az_gflags" : {
          "24e2efba-9972-4ebe-8551-23977c459064" : {
            "tserver" : {
              "tperaz2" : "v2",
              "tperaz1" : "v1"
            },
            "master" : {
              "mperaz2" : "val2",
              "mperaz1" : "val1"
            }
          }
        }
      }
    } ]
  },
  "info" : {
    "universe_uuid" : "7643c96d-a15c-4185-8168-3f1fdd6c8d04",
    "version" : 1,
    "creation_date" : {
      "offset" : {
        "totalSeconds" : 0,
        "id" : "Z",
        "rules" : {
          "fixedOffset" : true,
          "transitions" : [ ],
          "transitionRules" : [ ]
        }
      },
      "year" : 56258,
      "monthValue" : 6,
      "dayOfMonth" : 27,
      "hour" : 10,
      "minute" : 36,
      "second" : 40,
      "nano" : 0,
      "dayOfWeek" : "SUNDAY",
      "dayOfYear" : 178,
      "month" : "JUNE"
    },
    "node_prefix" : "yb-tc-Test-Universe-2",
    "update_in_progress" : false,
    "update_succeeded" : false,
    "universe_paused" : false,
    "software_upgrade_state" : "Ready",
    "is_software_rollback_allowed" : false,
    "nodes_resize_available" : false,
    "is_kubernetes_operator_controlled" : false,
    "x_cluster_info" : {
      "source_x_cluster_configs" : [ ],
      "target_x_cluster_configs" : [ ]
    },
    "clusters" : [ {
      "uuid" : "24e2efba-9972-4ebe-8551-23977c459064",
      "placement_info" : {
        "cloud_list" : [ {
          "uuid" : "7154f5af-1148-484a-aa77-88ddf7fcf6fd",
          "code" : "aws",
          "region_list" : [ {
            "uuid" : "bd86b4f1-9662-4430-9f24-59bebafa9fdd",
            "code" : "us-west-2",
            "name" : "us-west-2",
            "az_list" : [ {
              "uuid" : "a8bad348-4fcf-46eb-b4f4-34769d23056c",
              "name" : "AZ 1",
              "replication_factor" : 3,
              "subnet" : "subnet-1",
              "is_affinitized" : true
            } ]
          } ]
        } ]
      },
      "spot_price" : 0.0,
      "regions" : [ {
        "uuid" : "bd86b4f1-9662-4430-9f24-59bebafa9fdd",
        "name" : "us-west-2",
        "active" : true,
        "code" : "us-west-2",
        "latitude" : 0.0,
        "longitude" : 0.0,
        "details" : {
          "cloud_info" : {
            "aws" : { }
          }
        },
        "zones" : [ {
          "uuid" : "a8bad348-4fcf-46eb-b4f4-34769d23056c",
          "name" : "AZ 1",
          "code" : "az-1",
          "active" : true,
          "subnet" : "subnet-1",
          "details" : {
            "cloud_info" : { }
          }
        } ]
      } ]
    } ],
    "node_details_set" : [ {
      "az_uuid" : "a8bad348-4fcf-46eb-b4f4-34769d23056c",
      "cloud_info" : {
        "az" : "AZ 1",
        "cloud" : "aws",
        "region" : "us-west-2",
        "use_time_sync" : false
      },
      "crons_active" : true,
      "disks_are_mounted_by_uuid" : true,
      "is_master" : false,
      "is_redis_server" : true,
      "is_tserver" : true,
      "is_yql_server" : true,
      "is_ysql_server" : true,
      "master_http_port" : 7000,
      "master_rpc_port" : 7100,
      "node_exporter_port" : 9300,
      "node_idx" : 1,
      "otel_collector_metrics_port" : 0,
      "placement_uuid" : "24e2efba-9972-4ebe-8551-23977c459064",
      "redis_server_http_port" : 11000,
      "redis_server_rpc_port" : 6379,
      "state" : "ToBeAdded",
      "tserver_http_port" : 9000,
      "tserver_rpc_port" : 9100,
      "yb_controller_http_port" : 14000,
      "yb_controller_rpc_port" : 18018,
      "yb_prebuilt_ami" : false,
      "yql_server_http_port" : 12000,
      "yql_server_rpc_port" : 9042,
      "ysql_server_http_port" : 13000,
      "ysql_server_rpc_port" : 5433
    }, {
      "az_uuid" : "a8bad348-4fcf-46eb-b4f4-34769d23056c",
      "cloud_info" : {
        "az" : "AZ 1",
        "cloud" : "aws",
        "region" : "us-west-2",
        "use_time_sync" : false
      },
      "crons_active" : true,
      "disks_are_mounted_by_uuid" : true,
      "is_master" : false,
      "is_redis_server" : true,
      "is_tserver" : true,
      "is_yql_server" : true,
      "is_ysql_server" : true,
      "master_http_port" : 7000,
      "master_rpc_port" : 7100,
      "node_exporter_port" : 9300,
      "node_idx" : 2,
      "otel_collector_metrics_port" : 0,
      "placement_uuid" : "24e2efba-9972-4ebe-8551-23977c459064",
      "redis_server_http_port" : 11000,
      "redis_server_rpc_port" : 6379,
      "state" : "ToBeAdded",
      "tserver_http_port" : 9000,
      "tserver_rpc_port" : 9100,
      "yb_controller_http_port" : 14000,
      "yb_controller_rpc_port" : 18018,
      "yb_prebuilt_ami" : false,
      "yql_server_http_port" : 12000,
      "yql_server_rpc_port" : 9042,
      "ysql_server_http_port" : 13000,
      "ysql_server_rpc_port" : 5433
    }, {
      "az_uuid" : "a8bad348-4fcf-46eb-b4f4-34769d23056c",
      "cloud_info" : {
        "az" : "AZ 1",
        "cloud" : "aws",
        "region" : "us-west-2",
        "use_time_sync" : false
      },
      "crons_active" : true,
      "disks_are_mounted_by_uuid" : true,
      "is_master" : false,
      "is_redis_server" : true,
      "is_tserver" : true,
      "is_yql_server" : true,
      "is_ysql_server" : true,
      "master_http_port" : 7000,
      "master_rpc_port" : 7100,
      "node_exporter_port" : 9300,
      "node_idx" : 3,
      "otel_collector_metrics_port" : 0,
      "placement_uuid" : "24e2efba-9972-4ebe-8551-23977c459064",
      "redis_server_http_port" : 11000,
      "redis_server_rpc_port" : 6379,
      "state" : "ToBeAdded",
      "tserver_http_port" : 9000,
      "tserver_rpc_port" : 9100,
      "yb_controller_http_port" : 14000,
      "yb_controller_rpc_port" : 18018,
      "yb_prebuilt_ami" : false,
      "yql_server_http_port" : 12000,
      "yql_server_rpc_port" : 9042,
      "ysql_server_http_port" : 13000,
      "ysql_server_rpc_port" : 5433
    } ]
  }
}
```
Kindly review the structure and provide your feedback. I have moved a few properties like EIT from within Cluster to a higher level at UniverseSpec making it common across all clusters (primary, readonly and addon clusters). Let me know if you'd like to move around and clean up any other property of Universe.

I am working off this draft doc to move the properties around. So you can give your comments there as well - https://docs.google.com/spreadsheets/d/1dGsdidYVqgiuTo5EnRt6AieqeteAcP6ncgeqvtIhHrE/edit?pli=1#gid=0

Test Plan: New unit test to GET universe

Reviewers: sanketh, anijhawan, dshubin, vpatibandla, #yba-api-review!, amalyshev, svarshney, yshchetinin, dkumar, skurapati, vkumar, vbansal

Reviewed By: amalyshev

Subscribers: yugaware

Differential Revision: https://phorge.dev.yugabyte.com/D33904

…2039)

…wed_preview_flags_csv.

Summary:
`yb_enable_read_committed_isolation` and `ysql_enable_read_request_caching` were included as part of `allowed_preview_flags_csv` because of which these flags were not getting set properly. In this diff, we are removing it as a part of `allowed_preview_flags_csv` and directly enabling it on the `yb-master` and `yb-tserver` processes.
Jira: DB-10983

Test Plan: Manual Testing

Reviewers: nikhil

Reviewed By: nikhil

Subscribers: yugabyted-dev, shikhar.sahay

Differential Revision: https://phorge.dev.yugabyte.com/D34319

…Scan

Summary:
Bitmap scans try to optimize cases where the actual tuples are not required by avoiding fetching the rows from the table.

This was added in PG to support the COUNT(*) use case, but would also optimize returning system attributes that aren't stored in DocDB. In both of these cases, we already have everything we need locally: an exact number of ybctids, or the system attributes. Sending a request is a waste of time.

However, there are a few edge cases / optimization to this that make it a bit more complicated. For now, disable this optimisation so that we can do it right later, at a priority that makes sense. That follow up is tracked by yugabyte#22044
Jira: DB-10901

Test Plan:
```
./yb_build.sh --java-test 'org.yb.pgsql.TestPgRegressYbBitmapScans'
```

Reviewers: amartsinchyk, tnayak

Reviewed By: amartsinchyk

Subscribers: yql

Tags: #jenkins-ready

Differential Revision: https://phorge.dev.yugabyte.com/D34137

Summary:
In the cases where a ReleaseMetadata has both a local ybdb.tgz and a helm chart,
the helm chart would end up being the only
artifact created.
We know correctly parse the ReleaseMetadata and create all necessary artifacts
Because of the bug fix, I am forcing the migration to run again.

In addition, added in a filter to allow importLocalReleases to grab helm
charts

Test Plan:
1. New unit tests to cover the release metadata to releas migration
2. force the migration to re-run and validated it worked as expected
3. Tested the new import workflow and that it correctly grabbed helm charts from
the releases directory

Reviewers: muthu, anijhawan, rmadhavan

Reviewed By: rmadhavan

Subscribers: yugaware

Differential Revision: https://phorge.dev.yugabyte.com/D34199

Summary:
In order to integrate `postgresql_anonymizer` into yugabyte-db, we need `SECURITY LABEL` to be functional.
Jira: DB-10948

Test Plan:
There is a PostgreSQL test for `SECURITY LABEL` in `security_label.sql`, but it's ported to `yb_pg_security_label.sql` which is now included in `yb_pg_misc_independent_serial_schedule`:
`./yb_build.sh --java-test org.yb.pgsql.TestPgRegressPgMiscIndependent`

Reviewers: kramanathan, tnayak, jason

Reviewed By: jason

Subscribers: smishra, yql

Tags: #jenkins-ready

Differential Revision: https://phorge.dev.yugabyte.com/D34251

Summary:
The YB base scans cost model calculates the index tuple width to estimate the
cost of fetching the index from the disk to memory. This change fixes a bug in
the index tuple width calculation.
Jira: DB-10792

Test Plan: ./yb_build.sh --java-test 'org.yb.pgsql.TestPgRegressTAQO#testPgRegressTAQO'

Reviewers: tverona, tnayak

Reviewed By: tnayak

Subscribers: yql

Differential Revision: https://phorge.dev.yugabyte.com/D33947

Summary:
Messenger launches 4 extra reactor threads.
With this change we reuse the messenger from the sever in AutoFlags instead of creating its own messenger.
The derived classes `MasterAutoFlagsManager` and `TserverAutoFlagsManager` are destroyed before the servers messenger.

We continue to create a temp YbClient object when fetching AutoFlags from master since the servers client is started much later. Also this operation only happens the very first time a node is added to a cluster.

Fixes yugabyte#22076
Jira: DB-10998

Test Plan: Jenkins

Reviewers: sergei

Reviewed By: sergei

Subscribers: ybase

Differential Revision: https://phorge.dev.yugabyte.com/D34307

…yted and changing the shebang to use python3.

Summary:
- Deprecating python2 support in yugabyted and changing the shebang to use python3.
- Replacing dsutils package with  shutil as it deprecated in python3.

Test Plan: Manual

Reviewers: djiang

Reviewed By: djiang

Subscribers: yugabyted-dev

Differential Revision: https://phorge.dev.yugabyte.com/D34340

…clude unknownField in the subsequent new configure call payload

Summary:
Allow the userIntent from previous configure call to include unknownField in the subsequent new configure call payload

All scenarios passed:
Create Universe with Audit logging passed
Edit Universe with Audit logging passed
Create RR with Audit logging passed
Edit RR with Audit logging passed

Refer here where @amalyshev tested: https://gist.github.com/anmalysh-yb/4a7c7cec0b22162e05b7802551f8abc4

Test Plan:
Allow the userIntent from previous configure call to include unknownField in the subsequent new configure call payload
UI smoke tests passed as well

Reviewers: amalyshev, lsangappa

Reviewed By: amalyshev, lsangappa

Subscribers: yugaware

Differential Revision: https://phorge.dev.yugabyte.com/D34078

…gabyte-db

Summary:
This revision imports source code of postgresql_anonymizer into the yugabyte-db repository. This revision does not build and bundle postgresql_anonymizer as a yugabyte extension.

There are some modifications to the upstream commit:
1. Note that `postgresql_anonymizer` uses submodules, which are not included in this revision:
`postgresql_anonymizer/pg_dump_anon/tests: bats-assert`
`postgresql_anonymizer/pg_dump_anon/tests: bats-core`
`postgresql_anonymizer/pg_dump_anon/tests: bats-support`
2. Added `src/postgres/third-party-extensions/postgresql_anonymizer/*` to `build-support/common-test-env.sh` to skip `postgresql_anonymizer` python files during YB build's python doctest.

The details of the imported commit from the upstream [[ https://gitlab.com/dalibo/postgresql_anonymizer | postgresql_anonymizer repo ]] are as follows:
Commit: ba2a748905287306349b1b4dd1e2131e2b77def5 [[ https://gitlab.com/dalibo/postgresql_anonymizer/-/releases/1.3.1 | (1.3.1 ]])
Jira: DB-10947

Test Plan: Jenkins: all tests

Reviewers: kramanathan, tnayak, jason

Reviewed By: jason

Subscribers: jason, smishra, yql

Tags: #jenkins-ready

Differential Revision: https://phorge.dev.yugabyte.com/D34249

Summary: Isssue with HA where over time the number of open fds pile up as we are not closing the new WS client we create with overrides every time we perform an HA operation. This diff adds explicit close logic wherever we use the WSClient.

Test Plan: Run HA and validate open files count does not constantly increase.

Reviewers: nsingh, sanketh

Reviewed By: nsingh

Subscribers: yugaware

Differential Revision: https://phorge.dev.yugabyte.com/D34292

Summary:
ALTER TABLE changing primary key or altering column type comes in two
flavors:

- rewrite: the data needs to be updated (along with metadata)
- no rewrite: the data can stay the same and only metadata needs to be
  modified

Within the rewrite flavor, there are two cases for YB:

- old rewrite (yb_enable_alter_table_rewrite = off): the old
  implementation that doesn't use relfilenode
- new rewrite (yb_enable_alter_table_rewrite = on): the new
  implementation which uses relfilenode

The old rewrite tries to check that there are no rules on the table
being altered because that is currently not supported.  This check
incorrectly uses rd_rel->relhasrules, which stays true even if a rule
was created then dropped.  This does not result in any incorrect
behavior, but it does prevent the alter command unnecessarily.  Fix by
using rd_rules instead, similar to upstream postgres
04d61bfe64c9098aa737d0bea428b46a46b358bb.
Jira: DB-10986

Test Plan:
On Almalinux 8:

    ./yb_build.sh fastdebug --gcc11 --java-test TestPgRegressTable \
      -n 10 --tp 1

Close: yugabyte#22064

Reviewers: fizaa

Reviewed By: fizaa

Subscribers: yql

Differential Revision: https://phorge.dev.yugabyte.com/D34338

…d tables

Summary:
The cost model before this change was designed for colocated tables. In this
change the cost model has been enhanced to model non-colocated tables.

Specifically the following modifications have been made,
* Model the cost of extra network round trip for secondary index scans.
* Estimate the seeks and nexts for hash index lookup.
* In case of hash index lookup, if filter is not provided on all hash columns,
  DocDB, the index conditions get ignored. They are not used for index lookup,
  so we do a series of nexts on the index, moreover they index conditions
  are not executed as Storage filters either, so DocDB sends all the rows of the
  table to pggate and the index conditions are executed as local filters on PG
  side. This should be fixed in future, but the current behavior is modeled in
  this change.

To correctly model the network costs, the index scan cost model code had to be
refactored, and this has also been done in this diff.
Jira: DB-10543

Test Plan:
* TAQO workloads complex, cost-validation and cost-validation-single-table show improvements in hash sharded tables.
* `./yb_build.sh --java-test 'org.yb.pgsql.TestPgRegressTAQO#testPgRegressTAQO'`
* `./yb_build.sh --java-test 'org.yb.pgsql.TestPgRegressJoin#testPgRegressJoin'`

Reviewers: tverona, tnayak, mtakahara

Reviewed By: tnayak

Subscribers: mihnea, amartsinchyk, telgersma, yql

Differential Revision: https://phorge.dev.yugabyte.com/D33477

…s is incorrect

Summary:
The gflag 'enable_automatic_tablet_splitting' was introduced as `kNewInstallsOnly` class (class 4).
Later when the upgrade scenarios for automatic tablet splitting were resolved, the flag's class
was change to `kLocalPersisted` (class 2), D28688.
But while trying to Setup XCluster config for the different versions of the source and the target
(where source version < target version), YBA raises the following:
```Create Cluster config request failed: Auto Flags: Configure Replication
[enable_automatic_tablet_splitting] set on universe puppyfood-2024-xcluster-source
are not set on universe puppyfood-2024-xcluster-target.```

This is happening because only flags with class 3 and class 4 are considered for XCluster, but
the target has that flags class set to 4 and thus the source and the target are not compatible.

The change addresses this issue by switching the class of 'enable_automatic_tablet_splitting' gflag
from class 2 (kLocalPersisted) to class 4 (kExternal).
Jira: DB-11007

Test Plan: Jenkins

Reviewers: hsunder, rthallam

Reviewed By: hsunder, rthallam

Subscribers: ybase

Differential Revision: https://phorge.dev.yugabyte.com/D34334

Summary:
LightweightFunction (LWFunction) is intended to represent lambda as callable interface without
extra  memory allocations. LWFunction is a short living object and it is not intent to be
stored anywhere. Currently the implementation allows move copy but having such operation allows
undesired usage of the LWFunction objects. This diff removes move copy.
Jira: DB-10990

Test Plan: Jenkins

Reviewers: sergei

Reviewed By: sergei

Subscribers: ybase, yql

Differential Revision: https://phorge.dev.yugabyte.com/D34321

Summary:
This diff fixes memory leaks in Ysql Connection Manager which were caused due to following reasons:

  # In upstream odyssey io object of a client is never deallocated when client is free, so it is explicitly made to free.
  # In connection manager while reading the packet from socket during authentication (in yb_auth_passthrough.c), a machine msg object is created which is not set free after it's purpose, so explicitly making it in this diff.
  # Memory allocated to store the name of guc variable (in var.h) is never released, it is fixed by doing automatic allocation on stack so it is released automatically as it goes out of scope.

Jira: DB-8241

Test Plan:
  # All Ysql Connection Manager Tests are passing.

  # Ensured any long running app in yb-stress-test doesn't causes ysql conn mgr's memory to grow continuously while app is running, rather after some point of time it gets stabilized.

Reviewers: janand, nkumar, rbarigidad

Reviewed By: janand, rbarigidad

Subscribers: rbarigidad, nkumar, mihnea, yql

Differential Revision: https://phorge.dev.yugabyte.com/D33190

* fix link

* going beyond sql out og SQL features

* removed accidentally added file

* added xcluster to going beyond sql

* landing page details

* improve going beyond sql

* fix merge conflicts correctly

* intro to sharding

* netlify fix

* some edits and fixes

* Apply suggestions from code review

Co-authored-by: Dwight Hodge <79169168+ddhodge@users.noreply.github.com>

* resolved comments

* Adding system catalog

* more tables and views

* minor adjustments

* hstore and other mods

* applied more review comments

Co-authored-by: Dwight Hodge <79169168+ddhodge@users.noreply.github.com>

* added an entry for cluster topology

* some edits to system catalog

* full list of catalog

* fix typo

* add examples for sharding

* gen-ai

* system catalog edits

* added gen-ai intro

---------

Co-authored-by: Premkumar <contactprem+premkumr@gmail.com>
Co-authored-by: Dwight Hodge <79169168+ddhodge@users.noreply.github.com>

Summary:
LoadingCache does not clean up expired entries automatically or  instantly. It performs small amounts of maintenance during write operations, or during occasional read operations if writes are rare. This change makes sure to clean up actively.

https://github.com/google/guava/wiki/CachesExplained#when-does-cleanup-happen

Test Plan:
Tested by creating and destroying universes.

```
2024-04-17T23:43:58.078Z  [debug] cbcbd103-2072-450a-8417-25f0f79cf283 NodeAgentClient.java:133 [application-pekko.actor.default-dispatcher-4170] com.yugabyte.yw.common.NodeAgentClient Channel for
NodeAgent{uuid: 6010a53b-f5a3-49f7-8d56-b2d0959f3dc2, ip: 10.9.92.11} expired

2024-04-17T23:43:58.080Z  [debug] cbcbd103-2072-450a-8417-25f0f79cf283 NodeAgentClient.java:133 [application-pekko.actor.default-dispatcher-4170] com.yugabyte.yw.common.NodeAgentClient Channel for
NodeAgent{uuid: a9c17aa7-3789-4825-845c-fce81677919b, ip: 10.9.72.25} expired
```

Reviewers: amalyshev, sanketh, cwang

Reviewed By: amalyshev

Subscribers: yugaware

Differential Revision: https://phorge.dev.yugabyte.com/D34254

Summary:
when there is no "release_type" in the version_metadata, fallback to
parsing the type from the version instead of defaulting to preview.

Also continued to fill out the release_type map according to
https://docs.google.com/spreadsheets/d/1NICaqVzwkpYIjeVjScUEMzUCcY-Gen1HdYCVJSnpdqg/edit#gid=0

Test Plan:
New Unit test
tested uploading release
tested adding via url
tested adding via importLocalRelease

Reviewers: muthu, anijhawan

Reviewed By: muthu

Subscribers: yugaware

Differential Revision: https://phorge.dev.yugabyte.com/D34337

…Y_KEY

Summary:
Currently the decimal value of YB_AT_REWRITE_ALTER_PRIMARY_KEY is 22, which clashes with
AT_REWRITE_DEFAULT_VAL and AT_REWRITE_COLUMN_REWRITE.
Set YB_AT_REWRITE_ALTER_PRIMARY_KEY to 0x8000 to eliminate the clashing bits and provide
enough room for new flags that could be added in upstream PG.

Backport-to: 2024.1
Jira: DB-11005

Test Plan: Jenkins: test regex: .*TestPgRegressTable.*

Reviewers: jason

Reviewed By: jason

Subscribers: yql

Differential Revision: https://phorge.dev.yugabyte.com/D34347

* Phase-2 Arch restructuring

* new icons

* removing core functions

* modifying cdc and fixing links

* new cdc image

* fix links

* hybrid time/clocks

* Transactions : landing page and ordering

* txns and arch link fixes

* make page private properly

* feedback from patnaik

* fixes based on review from Mark Lillibridge

* change RAFT to Raft

* test push link fix

* edits and typos

* typo fix

---------

Co-authored-by: aishwarya24 <ashchakravarthy@gmail.com>

* added ASH content

* some more edits

* added links

* updated frontmatter

* changes from review

* moved ash to observability

* some edits

* minor edits

* commented out two views as they not ASH specific

---------

Co-authored-by: Dwight Hodge <ghodge@yugabyte.com>

Summary:
Checks are very old checks against master branch, not current release
branch.

Test Plan:
Jenkins: skip
Will test with dev-build-job to go through packaging workflow.

Reviewers: dskorobogaty, jharveysmith

Reviewed By: dskorobogaty, jharveysmith

Subscribers: devops, yugaware

Differential Revision: https://phorge.dev.yugabyte.com/D34375

…#22077)

Update CoreDB contributor instructions, which went out of date some time
ago.

- Add patchelf package to installation instructions.  It was always
  required, but what likely happened is that it used to be implicitly
  installed as part of a dependency of a different required package, and
  that is no longer the case today.
- Note that there may be compilation issues with centos + gcc.

… + API to retrieve list of queries

Summary:
Added ability to query ASH data grouped by client IP/query.
Also adde API to retrieve list of queries with query text, DB name, etc.

Test Plan: Unit tested.

Reviewers: rmadhavan, cdavid

Reviewed By: rmadhavan

Subscribers: yugaware

Differential Revision: https://phorge.dev.yugabyte.com/D34376

…rics collector

Summary:
Basically, we started creating new client for PlatformInstanceClientFactory each time  we need it. We actually close it, but shutdown hook for it remains in place. Which causes memory leak.
Fixing it.
Also this one tunes GC generation sizes + disables process metrics collector - as it's causing growning memory allocation over time, for some reason.

Test Plan: Tested manually on yba-installer portal.

Reviewers: muthu

Reviewed By: muthu

Subscribers: yugaware

Differential Revision: https://phorge.dev.yugabyte.com/D34353

* TA issues added

* added more TAs

* removed details for a ta

* fixed dates

* adding compute/storage separation and other fixes

* feedback from Franck

* removing unneccessary stuff

* Apply suggestions from code review

* fixes, index page

* minor edits

* Update docs/content/preview/explore/going-beyond-sql/_index.md

* removing column details

* fixing the tip properly

* format

---------

Co-authored-by: Dwight Hodge <79169168+ddhodge@users.noreply.github.com>
Co-authored-by: Dwight Hodge <ghodge@yugabyte.com>
Co-authored-by: Aishwarya Chakravarthy <achakravarthy@yugabyte.com>

…queries

Summary:
Query layer retries should be blocked for multi-statement queries.
This is because they might have transaction blocks within them that
had committed before a statement faces a kConflict, and a
query layer retry would redo the whole query and hence the
transaction block too, which might not be idempotent. Something
like the below where the second update faces a kConflict:

"begin ... repeatable read; update ...; commit; update ...;"

This diff also does cosmetic changes to log messages related to
query layer retries and error messages that are sent back to the
external client.
Jira: DB-10258

Test Plan:
Jenkins

Added yb_query_layer_retries_for_a_multi_statement_query to the yb_pg_isolation_schedule

Reviewers: tfoucher, patnaik.balivada, ishan.chhangani, shubhankar.shastri

Reviewed By: tfoucher

Subscribers: yql

Differential Revision: https://phorge.dev.yugabyte.com/D33860

Summary: [PLAT-13088] removing logging of credentials

Test Plan: phabricator

Reviewers: dshubin

Reviewed By: dshubin

Subscribers: yugaware

Differential Revision: https://phorge.dev.yugabyte.com/D34393

Summary:
e6bb62a/D34307 introduced a TSAN race.

Setting TEST_SetOutboundIpBase from the test causing a TSAN race.
This function is not required to be called anymore from `TEST_SetupConnectivity` since the server sets the OutboundIpBase on its messenger as part of process startup.

Fixes yugabyte#22101
Jira: DB-11034

Test Plan:
Jenkins Ready
Jenkins Urgent

Reviewers: esheng, asrivastava

Reviewed By: esheng

Subscribers: ybase

Differential Revision: https://phorge.dev.yugabyte.com/D34394

…ration (yugabyte#21874)

* Add same version requirement for repl -> yba installer migration

* edit

* copy to stable

* edit

---------

Co-authored-by: Dwight Hodge <ghodge@yugabyte.com>

… shared memory

Summary:
If we are using shared memory to do perform calls, there is no associated wait state, as there
isn't an explicit IncomingCall object created.

To track such Perform calls, we explicitly create and adopt a wait-state in PgClientSession::ProcessSharedRequest
Jira: DB-10692

Test Plan: yb_build.sh --cxx-test wait_states-itest --gtest_filter WaitStateITest.AshPg

Reviewers: asaha, hbhanawat

Reviewed By: asaha

Subscribers: yql, ybase, bogdan

Differential Revision: https://phorge.dev.yugabyte.com/D33809

…m underlying YBA instance is used

Summary:
1) During edit, the `gceProject` and `VPC` are not set on the `editProviderReq` body before the validation step and hence the NPE during validation.
2) The permissions for `instanceFetching` should also be checked during `isValidCreds` function.

UI Change:
The errors with the SA are not correctly mapped, so were not visible on the UI. Now, mapped them to `credentialType`
{F168322}

Test Plan:
  - Created a gcp provider
  - Edited the provider => credential type to `Use service account from this YBA host's instance`
  - No NPE and cross-checked the error message

Reviewers: #yba-api-review!, svarshney, kkannan

Reviewed By: svarshney, kkannan

Subscribers: yugaware

Differential Revision: https://phorge.dev.yugabyte.com/D33979

Summary:
This PR fixes the DB audit log capture for multi-line queries. We now use the multiline config in the Filelog Receiver to split based on the audit log regex pattern prefix.

When we run the following query before this PR:
```
CREATE TABLE emp (
  id int
);
```
We get output log in datadog like:
```
2024-04-19 13:59:52.021 UTC [10159] LOG:  AUDIT: SESSION,2,1,DDL,CREATE TABLE,,,"CREATE TABLE emp (
```
Notice the truncated query.

Test Plan:
Manually tested the following scenario for ysql before and after this change:
Flow:
1. Create universe
2. Enable DB audit logging
3. Run single line query:
```
CREATE TABLE emp1 (id int);
```
4. Run multi line query:
```
CREATE TABLE emp2 (
  id int
);
```

Step 3 datadog output:
```
2024-04-19 15:26:08.973 UTC [10011] LOG:  AUDIT: SESSION,2,1,DDL,CREATE TABLE,,,create table emp1(id int);,<none>
```

Step 4 datadog output:
```
2024-04-19 15:26:29.206 UTC [10011] LOG:  AUDIT: SESSION,3,1,DDL,CREATE TABLE,,,"create table emp2(
	  id int
	);",<none>
```

Reviewers: amalyshev

Reviewed By: amalyshev

Subscribers: yugaware

Differential Revision: https://phorge.dev.yugabyte.com/D34333

Summary: Generated docs

Test Plan: -

Reviewers: sneelakantan

Reviewed By: sneelakantan

Subscribers: yugaware

Differential Revision: https://phorge.dev.yugabyte.com/D34178

…mages

Summary:
Kubernetes Image secret name was missing from the CLI request body which failed universe creation as it did not fetch the image from quay registry.
Test diff to see if the changes helps

Test Plan: Test k8s provider and universe creation

Reviewers: svarshney

Reviewed By: svarshney

Subscribers: yugaware

Differential Revision: https://phorge.dev.yugabyte.com/D34405

…upon receipt of commit record

Summary:
We maintain a list of unacked transactions for the calculation of the restart_lsn from the confirmed_flush lsn. Prior to this revision, this list also included a transaction at
the end for which we haven't yet received the COMMIT record from the CDC service. Such a transaction is stored with commit_lsn as InvalidXLogRecPtr (0) and was leading to issues in the calculation in the restart_lsn.

This revision updates the logic to put a transaction into the list only on the receipt of the commit record.
Jira: DB-10866

Test Plan: Jenkins: test regex: .*ReplicationSlot.*

Reviewers: asrinivasan

Reviewed By: asrinivasan

Subscribers: ycdcxcluster, yql

Differential Revision: https://phorge.dev.yugabyte.com/D34327

Summary:
`keep_order` field should not be part of `YbctidGenerator`. This field is not used for
generation, it is used for processing of generated result.
Jira: DB-10861

Test Plan: Jenkins

Reviewers: amartsinchyk, tnayak, jason, myang

Reviewed By: myang

Subscribers: yql

Tags: #jenkins-ready

Differential Revision: https://phorge.dev.yugabyte.com/D34064

…ocks after a table rewrite

Summary:
Currently, after a table rewrite, the "relation" column in `pg_locks` will be populated with the
table's relfilenode oid instead of the table oid. This is because `TabletLockInfoPB` (and `LockInfoPB`
for a colocated table) is populated with the DocDB table's UUID. However, after a table rewrite
the DocDB table UUID no longer matches the table oid (instead, it matches the relfilenode of the table).

This diff fixes the above issue and introduces the following changes:
- Rename `table_id` proto fields in `LockInfoPB` and `TabletLockInfoPB` to `pg_table_id`.
- Populate the `pg_table_id` field with either `pg_table_id` (if non-empty) or `table_id` present in `TableInfo`.
- Additionally, some `TableInfo` constructors don't set `pg_table_id` -- fix this.
- `pg_table_id` was previously not set in tablet metadata for colocated tables, fix this by setting it in `AsyncAddTableToTablet`.

Upgrade/Rollback safety:
This change is upgrade/rollback safe as we are simply renaming existing proto fields (and not adding any new ones).
Jira: DB-11001

Test Plan: ./yb_build.sh --cxx-test pgwrapper_pg_get_lock_status-test --gtest_filter PgGetLockStatusTest.TestPgLocksOutputAfterTableRewrite

Reviewers: pjain, patnaik.balivada, bkolagani

Reviewed By: patnaik.balivada, bkolagani

Subscribers: yql, ybase

Differential Revision: https://phorge.dev.yugabyte.com/D34339

…neImage is used

Summary:
In YBM, we still use the centos7 AMIs that is passed as `machineImage` param during universe create.
imageBundle will have the `ec2-user` configured as the sshUser given the fact that we do not specify the AMI during
provider creation & ends up creating YBA Managed Bundles.

This diff falls back to sshUser configured in the provider in case `machineImage` is passed in the universe create params.

Test Plan: Manually verified

Reviewers: amalyshev

Reviewed By: amalyshev

Subscribers: yugaware

Differential Revision: https://phorge.dev.yugabyte.com/D34408

…abletSplit test

Summary:
With ddl atomicity enabled, this test goes in a deadlock and eventually fails due to test timeout.

The test uses sync points to reproduce the race mentioned [[ https://phorge.dev.yugabyte.com/D29992 | here ]].

The test design is as follows:

1. Drop one of the tables. (keep bg thread executing CleanUpCDCSDKStreamsMetadata() on hold)

2. Partially execute CleanUpCDCSDKStreamsMetadata() till it completes calling GetTablets() on all tables. (keep the test thread on hold)

3. Initiate a tablet split on one of the tablet of the non-dropped table. (keep bg thread on hold)

4. Resume execution of the CleanUpCDCSDKStreamsMetadata() method so that it can delete cdc_state entries.

With ddl atomicity feature enabled, some other task would have been introduced that needs to be executed by the bg thread. But since in step-1, we are holding the bg thread, the test goes into a deadlock and drop table operation doesnt complete, which is why the test doesnt move forward.
Jira: DB-11021

Test Plan: Jenkins: test regex: .*CDCSDKTabletSplitTest.*

Reviewers: skumar, stiwary

Reviewed By: stiwary

Subscribers: ycdcxcluster

Differential Revision: https://phorge.dev.yugabyte.com/D34379

Summary:
Added a new test `yb_partition_prune`  to validate that bitmap scans use partition pruning as expected. I didn't reuse an existing partition test file because `yb_partition_prune_plancache` should test plancache related things, and `yb_pg_partition_prune` is a port of a PG test.

I renamed the schedule from `yb_pg_partition_prune_schedule` to `yb_partition_prune_schedule` because it already contained a test not ported from pg (`yb_partition_prune_plancache`)
Jira: DB-10771

Test Plan:
Jenkins: test regex: .*PgRegressPartitions.*

```lang=sh
./yb_build.sh --java-test 'org.yb.pgsql.TestPgRegressPartitions#pruning'
```

Reviewers: amartsinchyk, tnayak, jason

Reviewed By: amartsinchyk

Subscribers: jason, yql

Differential Revision: https://phorge.dev.yugabyte.com/D33926

…y distribution anomalies + some fixes

Summary:
Added Query lock contention anomaly detection based on ASH data.
Added ASH RCAs for uneven query distribution anomalies .
Change POST to GET for /queries API.
groupBy and currentGroupBy fields in the APIs are lists now.

Test Plan: Tested manually + unit tested

Reviewers: rmadhavan, cdavid

Reviewed By: cdavid

Subscribers: yugaware

Differential Revision: https://phorge.dev.yugabyte.com/D34396