Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[doc] Copy architecture revamp to stable #22026

Closed
wants to merge 2 commits into from
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -255,7 +255,7 @@ In the event that performance suffers due to automatic tablet splitting, the fol
* [YB-TServer flags](../../../reference/configuration/yb-tserver/#sharding-flags):
* `post_split_trigger_compaction_pool_max_threads` indicates the number of threads dedicated to post-split compaction tasks per node. By default, this is limited to 1. Increasing this may complete tablet splits faster, but would require more CPU and disk resources.
* `post_split_trigger_compaction_pool_max_queue_size` indicates the number of outstanding post-split compaction tasks that can be queued at once per node, limited to 16 by default.
* `automatic_compaction_extra_priority` provides additional compaction priorities to [smaller compactions](../../concepts/yb-tserver#compaction-queues) when automatic tablet splitting is enabled. This prevents smaller compactions from being starved for resources by the larger post-split compactions. This is set to 50 by default (the maximum recommended), and can be reduced to 0.
* `automatic_compaction_extra_priority` provides additional compaction priorities to [smaller compactions](../../yb-tserver/#compaction-queues) when automatic tablet splitting is enabled. This prevents smaller compactions from being starved for resources by the larger post-split compactions. This is set to 50 by default (the maximum recommended), and can be reduced to 0.

#### YCSB workload with automatic tablet splitting example

Expand Down
1 change: 0 additions & 1 deletion docs/content/preview/architecture/query-layer/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,4 +93,3 @@ As the executor requests rows from each node, that node fetches or computes the
This process continues recursively until the top node has received all the rows it needs to produce the final result. For a `SELECT` query, these final rows are sent to the client. For data modification queries like `INSERT`, `UPDATE`, or `DELETE`, the rows are used to make the requested changes in the database tables.

The executor is designed to efficiently pull rows through the pipeline defined by the plan tree, processing rows in batches where possible for better performance.

Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ YugabyteDB supports the following types of reads:
- [Follower reads](../../ysql-language-features/going-beyond-sql/follower-reads-ycql/) that enable spreading the read workload across all replicas in the primary cluster.
- Observer reads that use read replicas. Read replicas are created as a separate cluster that may be located in a different region, possibly closer to the consumers of the data which would result in lower-latency access and enhanced support of analytics workloads.

A data center (also known as a [universe](../../../architecture/concepts#universe)) can have one primary cluster and several read replica clusters.
A data center (also known as a [universe](../../../architecture/key-concepts/#universe)) can have one primary cluster and several read replica clusters.

Stale reads are possible with an upper bound on the amount of staleness. Reads are guaranteed to be timeline-consistent. You need to set the consistency level to `ONE` in your application to work with follower reads or observer reads. In addition, you have to set the application's local data center to the read replica cluster's region.

Expand Down
2 changes: 1 addition & 1 deletion docs/content/stable/admin/yb-admin.md
Original file line number Diff line number Diff line change
Expand Up @@ -2374,7 +2374,7 @@ If specified, `des_ts_uuid` becomes the new leader. If the argument is empty (`"

### Rebalancing commands

For information on YB-Master load balancing, see [Data placement and load balancing](../../architecture/concepts/yb-master/#data-placement-and-load-balancing)
For information on YB-Master load balancing, see [Data placement and load balancing](../../architecture/yb-master/#tablet-assignments)

For YB-Master load balancing flags, see [Load balancing flags](../../reference/configuration/yb-master/#load-balancing-flags).

Expand Down
2 changes: 1 addition & 1 deletion docs/content/stable/admin/ycqlsh.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ ycqlsh [flags] [host [port]]
Where
- `host` is the IP address of the host on which [YB-TServer](../../architecture/concepts/universe/#yb-tserver-process) is run. The default is local host at `127.0.0.1`.
- `host` is the IP address of the host on which [YB-TServer](../../architecture/yb-tserver) is run. The default is local host at `127.0.0.1`.
- `port` is the TCP port at which YB-TServer listens for YCQL connections. The default is `9042`.
### Example
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Use the `SHOW` statement to display the value of a run-time parameter.

## Semantics

The parameter values in YSQL can be set and typically take effect the same way as in PostgreSQL. However, because YugabyteDB uses a different storage engine ([DocDB](../../../../../architecture/layered-architecture/#docdb)), many configurations related to the storage layer will not have the same effect in YugabyteDB as in PostgreSQL. For example, configurations related to connection and authentication, query planning, error reporting and logging, run-time statistics, client connection defaults, and so on, should work as in PostgreSQL.
The parameter values in YSQL can be set and typically take effect the same way as in PostgreSQL. However, because YugabyteDB uses a different storage engine ([DocDB](../../../../../architecture/docdb)), many configurations related to the storage layer will not have the same effect in YugabyteDB as in PostgreSQL. For example, configurations related to connection and authentication, query planning, error reporting and logging, run-time statistics, client connection defaults, and so on, should work as in PostgreSQL.

However, configurations related to write ahead log, vacuuming, or replication, may not apply to Yugabyte. Instead related configuration can be set using yb-tserver (or yb-master) [configuration flags](../../../../../reference/configuration/yb-tserver/#configuration-flags).

Expand Down
198 changes: 79 additions & 119 deletions docs/content/stable/architecture/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@ title: Architecture
headerTitle: Architecture
linkTitle: Architecture
description: Learn about the YugabyteDB architecture, including query, transactions, sharding, replication, and storage layers.
image: /images/section_icons/index/architecture.png
headcontent: YugabyteDB architecture including the query, transactions, sharding, replication, and storage layers.
image: fa-sharp fa-thin fa-puzzle
headcontent: Internals of query, transactions, sharding, replication, and storage layers
menu:
stable:
identifier: architecture
Expand All @@ -13,120 +13,80 @@ menu:
type: indexpage
---

<div class="row">
<div class="col-12 col-md-6 col-lg-12 col-xl-6">
<a class="section-link icon-offset" href="design-goals/">
<div class="head">
<img class="icon" src="/images/section_icons/introduction/core_features.png" aria-hidden="true" />
<div class="title">Design goals</div>
</div>
<div class="body">
The YugabyteDB design goals.
</div>
</a>
</div>

<div class="col-12 col-md-6 col-lg-12 col-xl-6">
<a class="section-link icon-offset" href="concepts/">
<div class="head">
<img class="icon" src="/images/section_icons/architecture/concepts.png" aria-hidden="true" />
<div class="articles">3 articles</div>
<div class="title">Key concepts</div>
</div>
<div class="body">
Key concepts of the YugabyteDB architecture as well as the components and processes of a YugabyteDB universe.
</div>
</a>
</div>

<div class="col-12 col-md-6 col-lg-12 col-xl-6">
<a class="section-link icon-offset" href="core-functions/">
<div class="head">
<img class="icon" src="/images/section_icons/architecture/core_functions.png" aria-hidden="true" />
<div class="articles">5 articles</div>
<div class="title">Core functions</div>
</div>
<div class="body">
Universe and table creation, the I/O path reading and writing, and high availability.
</div>
</a>
</div>

<div class="col-12 col-md-6 col-lg-12 col-xl-6">
<a class="section-link icon-offset" href="layered-architecture/">
<div class="head">
<img class="icon" src="/images/section_icons/architecture/concepts.png" aria-hidden="true" />
<div class="title">Layered architecture</div>
</div>
<div class="body">
YugabyteDB architecture overview.
</div>
</a>
</div>

<div class="col-12 col-md-6 col-lg-12 col-xl-6">
<a class="section-link icon-offset" href="query-layer/">
<div class="head">
<img class="icon" src="/images/section_icons/index/api.png" aria-hidden="true" />
<div class="articles">1 article</div>
<div class="title">Query layer</div>
</div>
<div class="body">
YugabyteDB Query Layer (YQL) overview.
</div>
</a>
</div>

<div class="col-12 col-md-6 col-lg-12 col-xl-6">
<a class="section-link icon-offset" href="transactions/">
<div class="head">
<img class="icon" src="/images/section_icons/architecture/distributed_acid.png" aria-hidden="true" />
<div class="articles">6 articles</div>
<div class="title">DocDB transactions layer</div>
</div>
<div class="body">
The transaction model and transactional consistency at various isolation levels.
</div>
</a>
</div>

<div class="col-12 col-md-6 col-lg-12 col-xl-6">
<a class="section-link icon-offset" href="docdb-sharding/">
<div class="head">
<img class="icon" src="/images/section_icons/architecture/distributed_acid.png" aria-hidden="true" />
<div class="articles">3 articles</div>
<div class="title">DocDB sharding layer</div>
</div>
<div class="body">
DocDB sharding overview.
</div>
</a>
</div>

<div class="col-12 col-md-6 col-lg-12 col-xl-6">
<a class="section-link icon-offset" href="docdb-replication/">
<div class="head">
<img class="icon" src="/images/section_icons/architecture/distributed_acid.png" aria-hidden="true" />
<div class="articles">3 articles</div>
<div class="title">DocDB replication layer</div>
</div>
<div class="body">
DocDB replication overview.
</div>
</a>
</div>

<div class="col-12 col-md-6 col-lg-12 col-xl-6">
<a class="section-link icon-offset" href="docdb/">
<div class="head">
<img class="icon" src="/images/section_icons/architecture/distributed_acid.png" aria-hidden="true" />
<div class="articles">3 articles</div>
<div class="title">DocDB storage layer</div>
</div>
<div class="body">
DocDB persistence storage overview.
</div>
</a>
</div>

</div>
YugabyteDB is a distributed database that seamlessly combines the principles of distributed systems, where multiple machines collaborate, with the familiar concepts of traditional databases, where data is organized in tables with standard interfaces for reading and writing data.

Unlike traditional centralized databases, YugabyteDB is designed to manage and process data across multiple nodes or servers, ensuring resiliency, consistency, high availability, scalability, fault tolerance, and other [design goals](design-goals/).

{{<tip>}}
Check out YugabyteDB [key concepts](./concepts) for your quick reference.
{{</tip>}}

## Layered architecture

In general, operations in YugabyteDB are split logically into 2 layers, the query layer and the storage layer. The query layer is responsible for handling user requests and sending the requests to the right data. The storage layer is responsible for optimally storing the data on disk and managing replication and consistency.

![YugabyteDB Layered Architecture](/images/architecture/layered-architecture.png)

## Query layer

For operating (CRUD) on the data that is split and stored across multiple machines, YugabyteDB provides two APIs, YSQL and YCQL. The query layer takes the user query submitted via the API and sends or fetches data to and from the right set of tablets.

{{<tip>}}
To understand how the query layer is designed, see [Query layer](query-layer/).
{{</tip>}}

## Storage layer

The tablet data is optimally stored and managed by DocDB, a document store that has been built on top of RocksDB for higher performance and persistence.

{{<tip>}}
To understand how data storage works in YugabyteDB, see [DocDB](docdb/).
{{</tip>}}

## Sharding

YugabyteDB splits table data into smaller pieces called tablets so that the data can be stored in parts across multiple machines. The mapping of a row to a tablet is deterministic and this process is known as sharding.

{{<tip>}}
To learn more about the various sharding schemes, see [Sharding](docdb-sharding/).
{{</tip>}}

## Replication

Tablets are replicated for resiliency, high availability, and fault tolerance. Each tablet has a leader that is responsible for consistent reads and writes to the data of the tablet and a few followers. The replication is done using the RAFT protocol to ensure consistency of data across the leader and followers.

{{<tip>}}
To understand how replication works, see [Replication](docdb-replication/).
{{</tip>}}

## Transactions

Transactions are a set of operations (CRUD) that are executed atomically with the option to roll back all actions if any operation fails.

{{<tip>}}
To understand how transactions work in YugabyteDB, see [Transactions](transactions/).
{{</tip>}}

## Core functionality

When YugabyteDB is deployed, it first creates an Universe. You can then create databases/namespaces and tables in the universe. After creating a table, you can read from and write to the tables.

{{<tip>}}
To understand how the core functionalities are implemented in YugabyteDB, see [Core functions](./core-functions).
{{</tip>}}

## Master server

The master service acts a catalog manager and cluster orchestrator, and manages many background tasks.

{{<tip>}}
For more details, see [YB-Master](./yb-master).
{{</tip>}}

## TServer

YugabyteDB splits table data into tablets. These tablets are maintained and managed on each node by the TServer.

{{<tip>}}
For more details, see [YB-TServer](./yb-tserver).
{{</tip>}}
53 changes: 0 additions & 53 deletions docs/content/stable/architecture/concepts/_index.md

This file was deleted.

64 changes: 0 additions & 64 deletions docs/content/stable/architecture/concepts/universe.md

This file was deleted.