docs: update getting-started-splunk-setup.md

docs/gettingstarted/getting-started-splunk-setup.md

@@ -1,49 +1,61 @@
 # Splunk setup
-## Create Indexes
-
-SC4S is pre-configured to map each sourcetype to a typical index. For new installations, it is best practice to create them in Splunk when
-using the SC4S defaults. SC4S can be easily customized to use different indexes if desired.
-
-* email
-* epav
-* epintel
-* infraops
-* netauth
-* netdlp
-* netdns
-* netfw
-* netids
-* netlb
-* netops
-* netwaf
-* netproxy
-* netipam
-* oswin
-* oswinsec
-* osnix
-* print
-* _metrics (Optional opt-in for SC4S operational metrics; ensure this is created as a metrics index)
-
-## Configure the Splunk HTTP Event Collector
-
-- Set up the Splunk HTTP Event Collector with the HEC endpoints behind a load balancer (VIP) configured for https round robin *WITHOUT* sticky
-session.  Alternatively, a list of HEC endpoint URLs can be configured in SC4S (native syslog-ng load balancing) if no load balancer is in
-place.  In most scenarios the recommendation is to use an external load balancer, as that makes longer term
-maintenance simpler by eliminating the need to manually keep the list of HEC URLs specified in sc4s current. However, if a LB is not
-available, native load balancing can be used with 10 or fewer Indexers where HEC is used exclusively for syslog.
-
-  In either case, it is _strongly_ recommended that SC4S traffic be sent to HEC endpoints configured directly on the indexers rather than
-an intermediate tier of HWFs.  
-- Create a HEC token that will be used by SC4S and ensure the token has access to place events in main, _metrics, and all indexes used as
-event destinations.
-
-* NOTE: It is recommended that the "Selected Indexes" on the token configuration page be left blank so that the token has access to
-_all_ indexes, including the `lastChanceIndex`.  If this list is populated, extreme care must be taken to keep it up to date, as an attempt to
-send data to an index not in this list will result in a `400` error from the HEC endpoint. Furthermore, the `lastChanceIndex` will _not_ be
-consulted in the event the index specified in the event is not configured on Splunk.  Keep in mind just _one_ bad message will "taint" the
-whole batch (by default 1000 events) and prevent the entire batch from being sent to Splunk.
-* In case you are not using TLS on SC4S- turn off SSL on global settings for HEC in Splunk.
-- Refer to [Splunk Cloud](http://docs.splunk.com/Documentation/Splunk/7.3.1/Data/UsetheHTTPEventCollector#Configure_HTTP_Event_Collector_on_managed_Splunk_Cloud)
-or [Splunk Enterprise](http://dev.splunk.com/view/event-collector/SP-CAAAE6Q) for specific HEC configuration instructions based on your
+To ensure proper integration for SC4S and Splunk, perform the following tasks in your Splunk instance:
+
+1. Create copies of your SC4S indexes in Splunk.
+2. Configure your HTTP event collector.
+
+
+## Step 1: Create indexes within Splunk
+
+SC4S maps each sourcetype to the following indexes by default:
+
+* `email`
+* `epav`
+* `epintel`
+* `fireeye`
+* `gitops`
+* `infraops`
+* `netauth`
+* `netdlp`
+* `netdns`
+* `netfw`
+* `netids`
+* `netlb`
+* `netops`
+* `netwaf`
+* `netproxy`
+* `netipam`
+* `oswin`
+* `oswinsec`
+* `osnix`
+* `print`
+* `_metrics` (Optional opt-in for SC4S operational metrics; ensure this is created as a metrics index)
+
+If you create custom indexes in SC4S you must also create them in Splunk. See [Create custom indexes]( https://docs.splunk.com/Documentation/Splunk/9.2.1/Indexer/Setupmultipleindexes) for more information.
+
+## Step 2: Configure your HTTP event collector
+
+See [Use the HTTP event collector](https://docs.splunk.com/Documentation/Splunk/9.2.1/Data/UsetheHTTPEventCollector) for HEC configuration instructions based on your
 Splunk type.
 
+Keep in mind the following best practices specific to HEC for SC4S:
+* Make sure that the HEC token created for SC4S has permissions to add events to `main`, `_metrics`, and all other event destination indexes.
+* You can leave "Selected Indexes" blank on the token configuration page so that the token has access to
+all indexes, including the `lastChanceIndex`.  If you do populate this field, take extreme care to keep it up to date; an attempt to
+send data to an index that is not in this list results in a `400` error from the HEC endpoint. The `lastChanceIndex` will not be
+consulted if the index specified in the event is not configured on Splunk and the entire batch is then not sent to Splunk.
+* SC4S traffic should be sent to HEC endpoints configured directly on the indexers rather than an intermediate tier of heavy forwarders.
+* SC4S traffic must be sent to HEC endpoints that are configured directly on the indexers.  
+
+### Create a load balancing mechanism
+In some configurations, you should ensure output balancing from SC4S to Splunk indexers. To do this, you create a load balancing mechanism between SC4S and Splunk indexers. See [Set up load balancing](https://docs.splunk.com/Documentation/Splunk/9.2.1/Forwarding/Setuploadbalancingd) for more information. Note that this should not be confused with load balancing between [sources and SC4S](../lb.md). 
+
+When configuring your load balancing mechanism, Keep in mind the following:
+
+* Splunk Cloud provides an internal ELB on TCP 443.
+* For Splunk Enterprise set up your Splunk HTTP Event Collector with the HEC endpoints behind a load balancer. 
+* An external load balancer simplifies long-term maintenance by eliminating the need to manually keep the list of HEC URLs specified in SC4S current. Set up a load balancer using virtual IP and configured for https round-robin without sticky session. 
+* If a load balancer is not available, you can configure a list of HEC endpoint URLs with native syslog-ng load balancing. For internal load balancing of syslog-ng you should:
+    * Load balance ten or fewer indexers.
+    * Use HEC exclusively for syslog.
+    * Have SC4S extract timestamps from messages (default behavior) rather than use the time of receipt for the message.