docs: update autoscaling

docs/source/bentocloud/how-tos/autoscaling.rst

@@ -6,52 +6,68 @@ BentoCloud autoscales your Bento Deployments to efficiently handle varying loads
 
 This document explains the autoscaling feature of BentoCloud.
 
-Configure autoscaling
----------------------
+Replicas
+--------
 
-You can set the :ref:`minimum and maximum values <bentocloud/how-tos/configure-deployments:scaling>` to define the boundaries for scaling, allowing the autoscaler to reduce or increase the number of replicas as needed. This feature supports scaling to zero replica during idle time.
-
-To enable better control over autoscaling, you can set concurrency and external queue.
+You can set the :ref:`minimum and maximum replicas <bentocloud/how-tos/configure-deployments:scaling>` to define the boundaries for scaling, allowing the autoscaler to reduce or increase the number of replicas as needed. This feature supports scaling to zero replica during idle time.
 
 Concurrency
-^^^^^^^^^^^
+-----------
+
+To enable autoscaling, first configure the ``concurrency`` configuration for the service. :doc:`/guides/concurrency` refers to the concurrent number of requests of a BentoML Service is able to to process simultaneously. Setting this parameter means the Service will be automatically scaled on BentoCloud when the concurrent requests per replica exceeds the specified concurrency threshold.
+
+For example, a ``concurrency`` is set to 32, and the Service has 2 replicas, if the Service receives 100 concurrent requests, BentoCloud will automatically scale up to 4 replicas to handle the traffic. Similarly, if the concurrent requests drop below 32, BentoCloud will scale down to 1 replica to save resources.
+
+In general, the autoscaler will scale the number of replicas based on the following formula, permitted by the ``min_replicas`` and ``max_replicas`` settings in the deployment:
 
-:doc:`/guides/concurrency` refers to the ability of a BentoML Service to process multiple requests simultaneously. Setting this parameter means the Service will be automatically scaled on BentoCloud when the average requests per replica exceeds the specified concurrency threshold.
+.. image:: ../../_static/img/guides/autoscaling/hpa.png
 
 Use the ``@bentoml.service`` decorator to set concurrency:
 
 .. code-block:: python
 
     @bentoml.service(
         traffic={
-            "concurrency": 3,  # An integer value
+            "concurrency": 32,  # An integer value
         }
     )
     class MyService:
         ...
 
+.. note::
+
+    If ``concurrency`` is not set, the Service will only be autoscaled based on CPU utilization, which may not be optimal for your Service.
+
+
 External queue
-^^^^^^^^^^^^^^
+--------------
+
+You can also configure to enable an external request queue to buffer incoming requests. This feature is useful when you want to prevent overloading the Service with requests that exceed the concurrency threshold.
+
+When the external queue is enabled, BentoCloud will hold excess requests in the queue until the Service has the capacity to process them. This ensures that the Service never receives more requests simultaneously than the ``concurrency`` setting allows. BentoCloud will automatically scale the number of replicas based on the number of requests in the queue.
+
+.. note::
+
+    Using the external queue increases the latency of the Service because the extra IO operations are needed to handle the requests in the queue.
 
 You can enhance concurrency management with an external request queue on BentoCloud using the ``@bentoml.service`` decorator:
 
 .. code-block:: python
 
     @bentoml.service(
         traffic={
-            "concurrency": 3,  # An integer value
+            "concurrency": 32,  # An integer value
             "external_queue": True, # A BentoCloud-only field. If set to true, BentoCloud will use an external queue to handle excess requests
         }
     )
     class MyService:
         ...
 
-The external request queue is used to moderate incoming traffic, ensuring that a Service instance never receives more requests simultaneously than the ``concurrency`` setting allows. Excess requests are held in the queue until the Service has the capacity to process them, preventing overloading and maintaining efficient operation.
-
 .. note::
 
-    - If you enable ``external_queue`` in the ``@bentoml.service`` decorator, you must specify a ``concurrency`` value.
-    - ``max_concurrency`` does not take effect on BentoCloud. You need to enable ``external_queue`` to handle excess requests.
+    If you enable ``external_queue`` in the ``@bentoml.service`` decorator, you must specify a ``concurrency`` value.
+
+It's worth noting that when external queue is enabled, ``max_concurrency`` will no longer take effect because BentoCloud guarantees the Service will never receive more requests simultaneously than the ``concurrency`` setting allows. Therefore, ``max_concurrency`` is never reached.
 
 Autoscaling policies
 --------------------
@@ -61,13 +77,13 @@ You can customize scaling behavior to match your Service's needs with scaling-up
 Allowed scaling-up policies (``scale_up_behavior``):
 
 - ``fast`` (default): There is no stabilization window, so the autoscaler can increase the number of replicas immediately if necessary. It can increase the number of replicas by 100% or by 4 replicas, whichever is higher, every 15 seconds.
-- ``stable``: The autoscaler can increase the number of replicas, but it will stabilize the number of replicas for 300 seconds (5 minutes) before deciding to scale up further. It can increase the number of replicas by 100% every 15 seconds.
+- ``stable``: The autoscaler can increase the number of replicas, but it will stabilize the number of replicas for 600 seconds (10 minutes) before deciding to scale up further. It can increase the number of replicas by 100% every 15 seconds.
 - ``disabled``: Scaling-up is turned off.
 
 Allowed scaling-down policies (``scale_down_behavior``):
 
 - ``fast``: There is no stabilization window, so the autoscaler can reduce the number of replicas immediately if necessary. It can decrease the number of replicas by 100% or by 4 replicas, whichever is higher, every 15 seconds.
-- ``stable`` (default): The autoscaler can reduce the number of replicas, but it will stabilize the number of replicas for 300 seconds (5 minutes) before deciding to scale down further. It can decrease the number of replicas by 100% every 15 seconds.
+- ``stable`` (default): The autoscaler can reduce the number of replicas, but it will stabilize the number of replicas for 600 seconds (10 minutes) before deciding to scale down further. It can decrease the number of replicas by 100% every 15 seconds.
 - ``disabled``: Scaling-down is turned off.
 
 To set autoscaling policies, you need to configure the above fields in a separate YAML or JSON file. For example: