docs: update getting-started-splunk-setup.md #2417

jenworthington · 2024-04-23T07:12:23Z

I can't quite figure out how the customer is supposed to use this. It reads as steps but they don't seem like actual guided tasks. Is it more of a best practices topic or an overview? Or if it is numbered steps/process, then maybe we add links to each step? I'd love to get your thoughts and maybe we can discuss in our 1:1

mstopa-splunk · 2024-04-23T12:04:46Z

hi @jenworthington sure, that's what this section is about:

Topic: how to setup your Splunk instance to work with SC4S

Steps:

Create default indexes in Splunk
Set up the Splunk HTTP Event Collector

These are the two things that must be done to ensure SC4S-Splunk connection.

Ad 1 Indexes
You can use your custom set of indexes. But make sure that all of them, as well as the default set, are created in Splunk, else you will miss events processed by SC4S

Ad 2 HTTP event collector

Refer to Splunk docs to see how to set it up
But here are best practices to avoid problems:
a. put HEC endpoints of your indexers behind a load balancer. Use native syslog-ng load balancing or, preferably, an external load balancer
b. don't use an intermediate tier of HWFs
c. make sure that the HEC token has permissions to write in the indexes that you'll need
d. make sure that you either don't put any "Selected Indexes" or you carefully keep this list up to date
e. If you're not using TLS on SC4S, turn it off in Splunk's HEC token too.

mstopa-splunk · 2024-04-23T13:32:13Z

partially solves #2358

mstopa-splunk · 2024-04-23T13:33:41Z

@jenworthington can you work on the new file docs/gettingstarted/getting-started-splunk-setup-new.md ? I will replace the old one with this one when we finish

docs/gettingstarted/getting-started-splunk-setup.md

rjha-splunk

Please check the comment.

mstopa-splunk · 2024-04-23T14:03:04Z

@rjha-splunk I left the file that you saw for reference for Jen, but please check docs/gettingstarted/getting-started-splunk-setup-new.md instead. It will replace the old one completely

More edits, added some new links to the Splunk Enterprise docs

jenworthington · 2024-04-24T05:29:58Z

Thanks for the new suggestions for structure, it was really helpful. I think I've captured all of the requested changes, take a look and let me know, happy to work on this one some more as needed.

mstopa-splunk · 2024-04-24T08:33:33Z

docs/gettingstarted/getting-started-splunk-setup.md

-* In case you are not using TLS on SC4S- turn off SSL on global settings for HEC in Splunk.
- Refer to [Splunk Cloud](http://docs.splunk.com/Documentation/Splunk/7.3.1/Data/UsetheHTTPEventCollector#Configure_HTTP_Event_Collector_on_managed_Splunk_Cloud)
-or [Splunk Enterprise](http://dev.splunk.com/view/event-collector/SP-CAAAE6Q) for specific HEC configuration instructions based on your
+To set up syslog processing with SC4S, perform the following tasks in your Splunk instance:


It's a tricky one: SC4S will process syslog regardless Splunk. But if Splunk is not correctly set up, output from SC4S won't be correctly indexed by Splunk. Can me make it more like: "to ensure proper integration/collaboration/setup of SC4S and Splunk" kind of thing?

@jenworthington please check this comment

How about "To ensure proper integration for SC4S and Splunk, perform the following tasks in your Splunk instance:"

mstopa-splunk · 2024-04-24T08:34:23Z

docs/gettingstarted/getting-started-splunk-setup.md

+To set up syslog processing with SC4S, perform the following tasks in your Splunk instance:
+1. Create indexes within Splunk.
+2. Configure your HTTP event collector.
+3. Create a load balancing mechanism.


"Create a load balancing mechanism" is a subsection to "Configure your HTTP event collector"

@jenworthington I think it's better to remove point 3 as a subsection of point 2

mstopa-splunk · 2024-04-24T08:35:54Z

docs/gettingstarted/getting-started-splunk-setup.md

+
+## Step 1: Create indexes within Splunk
+
+SC4S maps each sourcetype to the following indexes by default:


Customers report problems to us because sometimes they don't create those indexes in Splunk, so better to say in the docs that the SC4S's default set of indexes must be created in Splunk

that makes so much more sense!

right, but the call for action for the reader is that they must create those indexes or they will have problems, please compare with the original document

mstopa-splunk · 2024-04-24T08:37:21Z

docs/gettingstarted/getting-started-splunk-setup.md

+* `print`
+* `_metrics` (Optional opt-in for SC4S operational metrics; ensure this is created as a metrics index)
+
+You can also you create your own indexes in Splunk. See [Create custom indexes]( https://docs.splunk.com/Documentation/Splunk/9.2.1/Indexer/Setupmultipleindexes) for more information.


can we say that if you use your own indexes in SC4S you also have to create them in Splunk?

mstopa-splunk · 2024-04-24T08:40:43Z

docs/gettingstarted/getting-started-splunk-setup.md

+send data to an index that is not in this list results in a `400` error from the HEC endpoint. The `lastChanceIndex` will not be
+consulted if the index specified in the event is not configured on Splunk and the entire batch is then not sent to Splunk.
+* If you are not using TLS on SC4S, turn off SSL on global settings for HEC in Splunk.
+* SC4S traffic must be sent to HEC endpoints that are configured directly on the indexers.  


can we restore the alternative and change it to should? SC4S traffic should be sent to HEC endpoints configured directly on the indexers rather than an intermediate tier of heavy forwarders.

mstopa-splunk · 2024-04-24T08:42:57Z

docs/gettingstarted/getting-started-splunk-setup.md

+* If you are not using TLS on SC4S, turn off SSL on global settings for HEC in Splunk.
+* SC4S traffic must be sent to HEC endpoints that are configured directly on the indexers.  
+
+### Step 2: Create a load balancing mechanism


Create a load balancing mechanism is a subsection of Configure your HTTP event collector

i removed it as a numbered step and made it a subtopic instead.

mstopa-splunk · 2024-04-24T08:47:48Z

docs/gettingstarted/getting-started-splunk-setup.md

+* SC4S traffic must be sent to HEC endpoints that are configured directly on the indexers.  
+
+### Step 2: Create a load balancing mechanism
+Create a load balancing mechanism between SC4S and Splunk indexers. See [Set up load balancing](https://docs.splunk.com/Documentation/Splunk/9.2.1/Forwarding/Setuploadbalancingd) for more information. Note that this should not be confused with load balancing between [sources and SC4S](../lb.md). 


splunk docs on load balancing don't apply to this case. Can we make it more in the style of:

In some situations, it is necessary to ensure balancing of the output from SC4S to Splunk indexers. Note that this should not be confused with load balancing between [sources and SC4S](../lb.md).

what we mean here is that:

if you have Splunk Cloud no worry, you're already covered and your SC4S output will be automatically load balanced to Splunk indexers

if you have Splunk Enterprise and a single indexer, you obviously don't need an lb

if you have Splunk Enterprise and mutliple indexers, you should load balance your SC4S output

thanks! Is there another topic we should link to in case they are now to the product and need some guidance for creating this type of load balancing?

Unfortunately there's nothing to link at this point, we don't provide any further recommendations for lbs at this point

"In some configurations, you should ensure output balancing from SC4S to Splunk indexers. To do this, you create a load balancing mechanism between SC4S and Splunk indexers. See Set up load balancing for more information."

mstopa-splunk · 2024-04-24T08:50:13Z

docs/gettingstarted/getting-started-splunk-setup.md

+* An external load balancer simplifies long-term maintenance by eliminating the need to manually keep the list of HEC URLs specified in SC4S current. Set up a load balancer using virtual IP and configured for https round-robin without sticky session. 
+* If a load balancer is not available, you can configure a list of HEC endpoint URLs with native syslog-ng load balancing. For internal load balancing of syslog-ng you should:
+    * Load balance ten or fewer indexers.
+    * Bse HEC exclusively for syslog.


Suggested change

* Bse HEC exclusively for syslog.

* Use HEC exclusively for syslog.

mstopa-splunk · 2024-04-24T08:51:09Z

@jenworthington ready for the next iteration

mstopa-splunk · 2024-04-29T12:37:55Z

docs/gettingstarted/getting-started-splunk-setup.md

- Refer to [Splunk Cloud](http://docs.splunk.com/Documentation/Splunk/7.3.1/Data/UsetheHTTPEventCollector#Configure_HTTP_Event_Collector_on_managed_Splunk_Cloud)
-or [Splunk Enterprise](http://dev.splunk.com/view/event-collector/SP-CAAAE6Q) for specific HEC configuration instructions based on your
+To set up syslog processing with SC4S, perform the following tasks in your Splunk instance:
+1. Create indexes within Splunk.


Please check how mkdocs renders documentation: https://splunk.github.io/splunk-connect-for-syslog/2417/gettingstarted/getting-started-splunk-setup/

in this case we need a newline before the list else we get:

I added a new line before the steps. I don't see it rendering though, am i doing it wrong?

mstopa-splunk · 2024-04-29T12:39:34Z

docs/gettingstarted/getting-started-splunk-setup.md

+To set up syslog processing with SC4S, perform the following tasks in your Splunk instance:
+1. Create indexes within Splunk.
+2. Configure your HTTP event collector.
+3. Create a load balancing mechanism.


Point 3 is not a separate point, it's rather a subsection of configuring HTTP event collectors. And Splunk Cloud users don't have to worry about it at all

mstopa-splunk · 2024-04-29T12:40:30Z

docs/gettingstarted/getting-started-splunk-setup.md

+
+* `email`
+* `epav`
+* `epintel`


can you also add fireeye and gitops?

@jenworthington can you please add these two indexes too?

docs/gettingstarted/k8s-microk8s.md

mstopa-splunk · 2024-05-08T12:43:05Z

@jenworthington something went wrong and your changes to docs/gettingstarted/getting-started-splunk-setup.md from the last pass were not commited. I opened all previous comments again, please go through them and commit the final pass, I'm sorry for that situation

I made these changes a while back but maybe i did something weird with the branching? So I redid them and hopefully second time is the charm. ;)

jenworthington requested a review from mstopa-splunk April 23, 2024 07:12

jenworthington requested a review from rjha-splunk as a code owner April 23, 2024 07:12

Merge branch 'main' into jenworthington-patch-4

900926c

mstopa-splunk changed the title ~~Update getting-started-splunk-setup.md~~ docs: update getting-started-splunk-setup.md Apr 23, 2024

Add new doc proposition

293fb9e

rjha-splunk reviewed Apr 23, 2024

View reviewed changes

docs/gettingstarted/getting-started-splunk-setup.md Outdated Show resolved Hide resolved

rjha-splunk reviewed Apr 23, 2024

View reviewed changes

docs/gettingstarted/getting-started-splunk-setup.md Outdated Show resolved Hide resolved

rjha-splunk reviewed Apr 23, 2024

View reviewed changes

jenworthington added 2 commits April 23, 2024 16:19

Update getting-started-splunk-setup-new.md

f44f355

More edits, added some new links to the Splunk Enterprise docs

Update getting-started-splunk-setup-new.md

9869523

Overwrite getting-started-splunk-setup.md with the updated version

9628cbe

mstopa-splunk reviewed Apr 24, 2024

View reviewed changes

jenworthington and others added 2 commits April 28, 2024 18:56

Update k8s-microk8s.md

b0760aa

Merge branch 'main' into jenworthington-patch-4

3a750d1

mstopa-splunk reviewed Apr 29, 2024

View reviewed changes

mstopa-splunk mentioned this pull request Apr 29, 2024

Update k8s-microk8s.md #2427

Closed

mstopa-splunk and others added 2 commits May 8, 2024 12:30

Revert changes in k8s-microk8s.md

8588c8d

Merge branch 'main' into jenworthington-patch-4

212c0ac

jenworthington added 2 commits May 10, 2024 11:46

Update getting-started-splunk-setup.md

d7c0346

Update getting-started-splunk-setup.md

fcadfc8

I made these changes a while back but maybe i did something weird with the branching? So I redid them and hopefully second time is the charm. ;)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: update getting-started-splunk-setup.md #2417

docs: update getting-started-splunk-setup.md #2417

jenworthington commented Apr 23, 2024

mstopa-splunk commented Apr 23, 2024

mstopa-splunk commented Apr 23, 2024

mstopa-splunk commented Apr 23, 2024

rjha-splunk left a comment

mstopa-splunk commented Apr 23, 2024

jenworthington commented Apr 24, 2024 •

edited

mstopa-splunk Apr 24, 2024

mstopa-splunk May 8, 2024

jenworthington May 10, 2024

mstopa-splunk Apr 24, 2024

mstopa-splunk May 8, 2024

jenworthington May 10, 2024

mstopa-splunk Apr 24, 2024

jenworthington Apr 26, 2024

mstopa-splunk May 8, 2024

mstopa-splunk Apr 24, 2024

mstopa-splunk Apr 24, 2024

mstopa-splunk Apr 24, 2024

jenworthington Apr 26, 2024

mstopa-splunk Apr 24, 2024

jenworthington Apr 26, 2024

mstopa-splunk Apr 29, 2024

jenworthington May 10, 2024

mstopa-splunk Apr 24, 2024

mstopa-splunk commented Apr 24, 2024

mstopa-splunk Apr 29, 2024

jenworthington May 10, 2024

mstopa-splunk Apr 29, 2024

mstopa-splunk Apr 29, 2024

mstopa-splunk May 8, 2024

jenworthington May 10, 2024

mstopa-splunk commented May 8, 2024 •

edited


		## Step 1: Create indexes within Splunk

		SC4S maps each sourcetype to the following indexes by default:

	* Bse HEC exclusively for syslog.
	* Use HEC exclusively for syslog.

docs: update getting-started-splunk-setup.md #2417

Are you sure you want to change the base?

docs: update getting-started-splunk-setup.md #2417

Conversation

jenworthington commented Apr 23, 2024

mstopa-splunk commented Apr 23, 2024

mstopa-splunk commented Apr 23, 2024

mstopa-splunk commented Apr 23, 2024

rjha-splunk left a comment

Choose a reason for hiding this comment

mstopa-splunk commented Apr 23, 2024

jenworthington commented Apr 24, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mstopa-splunk commented Apr 24, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mstopa-splunk commented May 8, 2024 • edited

jenworthington commented Apr 24, 2024 •

edited

mstopa-splunk commented May 8, 2024 •

edited