docs: update getting-started-splunk-setup.md #2417

jenworthington · 2024-04-23T07:12:23Z

I can't quite figure out how the customer is supposed to use this. It reads as steps but they don't seem like actual guided tasks. Is it more of a best practices topic or an overview? Or if it is numbered steps/process, then maybe we add links to each step? I'd love to get your thoughts and maybe we can discuss in our 1:1

mstopa-splunk · 2024-04-23T12:04:46Z

hi @jenworthington sure, that's what this section is about:

Topic: how to setup your Splunk instance to work with SC4S

Steps:

Create default indexes in Splunk
Set up the Splunk HTTP Event Collector

These are the two things that must be done to ensure SC4S-Splunk connection.

Ad 1 Indexes
You can use your custom set of indexes. But make sure that all of them, as well as the default set, are created in Splunk, else you will miss events processed by SC4S

Ad 2 HTTP event collector

Refer to Splunk docs to see how to set it up
But here are best practices to avoid problems:
a. put HEC endpoints of your indexers behind a load balancer. Use native syslog-ng load balancing or, preferably, an external load balancer
b. don't use an intermediate tier of HWFs
c. make sure that the HEC token has permissions to write in the indexes that you'll need
d. make sure that you either don't put any "Selected Indexes" or you carefully keep this list up to date
e. If you're not using TLS on SC4S, turn it off in Splunk's HEC token too.

mstopa-splunk · 2024-04-23T13:32:13Z

partially solves #2358

mstopa-splunk · 2024-04-23T13:33:41Z

@jenworthington can you work on the new file docs/gettingstarted/getting-started-splunk-setup-new.md ? I will replace the old one with this one when we finish

docs/gettingstarted/getting-started-splunk-setup.md

rjha-splunk

Please check the comment.

mstopa-splunk · 2024-04-23T14:03:04Z

@rjha-splunk I left the file that you saw for reference for Jen, but please check docs/gettingstarted/getting-started-splunk-setup-new.md instead. It will replace the old one completely

More edits, added some new links to the Splunk Enterprise docs

jenworthington · 2024-04-24T05:29:58Z

Thanks for the new suggestions for structure, it was really helpful. I think I've captured all of the requested changes, take a look and let me know, happy to work on this one some more as needed.

docs/gettingstarted/getting-started-splunk-setup.md

mstopa-splunk · 2024-04-24T08:35:54Z

docs/gettingstarted/getting-started-splunk-setup.md

+
+## Step 1: Create indexes within Splunk
+
+SC4S maps each sourcetype to the following indexes by default:


Customers report problems to us because sometimes they don't create those indexes in Splunk, so better to say in the docs that the SC4S's default set of indexes must be created in Splunk

that makes so much more sense!

right, but the call for action for the reader is that they must create those indexes or they will have problems, please compare with the original document

can we make it explicit:

Suggested change

SC4S maps each sourcetype to the following indexes by default:

SC4S maps each sourcetype to the following indexes by default. Make sure to create them in Splunk.

docs/gettingstarted/getting-started-splunk-setup.md

mstopa-splunk · 2024-04-24T08:47:48Z

docs/gettingstarted/getting-started-splunk-setup.md

+* SC4S traffic must be sent to HEC endpoints that are configured directly on the indexers.  
+
+### Step 2: Create a load balancing mechanism
+Create a load balancing mechanism between SC4S and Splunk indexers. See [Set up load balancing](https://docs.splunk.com/Documentation/Splunk/9.2.1/Forwarding/Setuploadbalancingd) for more information. Note that this should not be confused with load balancing between [sources and SC4S](../lb.md). 


splunk docs on load balancing don't apply to this case. Can we make it more in the style of:

In some situations, it is necessary to ensure balancing of the output from SC4S to Splunk indexers. Note that this should not be confused with load balancing between [sources and SC4S](../lb.md).

what we mean here is that:

if you have Splunk Cloud no worry, you're already covered and your SC4S output will be automatically load balanced to Splunk indexers

if you have Splunk Enterprise and a single indexer, you obviously don't need an lb

if you have Splunk Enterprise and mutliple indexers, you should load balance your SC4S output

thanks! Is there another topic we should link to in case they are now to the product and need some guidance for creating this type of load balancing?

Unfortunately there's nothing to link at this point, we don't provide any further recommendations for lbs at this point

"In some configurations, you should ensure output balancing from SC4S to Splunk indexers. To do this, you create a load balancing mechanism between SC4S and Splunk indexers. See Set up load balancing for more information."

@jenworthington sounds great but we cannot use this link because it's for heavy forwarders, not sc4s:

In some configurations, you should ensure output balancing from SC4S to Splunk indexers. To do this, you create a load balancing mechanism between SC4S and Splunk indexers.

docs/gettingstarted/getting-started-splunk-setup.md

mstopa-splunk · 2024-04-24T08:51:09Z

@jenworthington ready for the next iteration

mstopa-splunk · 2024-04-29T12:37:55Z

docs/gettingstarted/getting-started-splunk-setup.md

- Refer to [Splunk Cloud](http://docs.splunk.com/Documentation/Splunk/7.3.1/Data/UsetheHTTPEventCollector#Configure_HTTP_Event_Collector_on_managed_Splunk_Cloud)
-or [Splunk Enterprise](http://dev.splunk.com/view/event-collector/SP-CAAAE6Q) for specific HEC configuration instructions based on your
+To set up syslog processing with SC4S, perform the following tasks in your Splunk instance:
+1. Create indexes within Splunk.


Please check how mkdocs renders documentation: https://splunk.github.io/splunk-connect-for-syslog/2417/gettingstarted/getting-started-splunk-setup/

in this case we need a newline before the list else we get:

I added a new line before the steps. I don't see it rendering though, am i doing it wrong?

rendering works great now:

https://splunk.github.io/splunk-connect-for-syslog/2417/gettingstarted/getting-started-splunk-setup/

docs/gettingstarted/getting-started-splunk-setup.md

docs/gettingstarted/k8s-microk8s.md

mstopa-splunk · 2024-05-08T12:43:05Z

@jenworthington something went wrong and your changes to docs/gettingstarted/getting-started-splunk-setup.md from the last pass were not commited. I opened all previous comments again, please go through them and commit the final pass, I'm sorry for that situation

I made these changes a while back but maybe i did something weird with the branching? So I redid them and hopefully second time is the charm. ;)

mstopa-splunk · 2024-05-16T09:58:16Z

docs/gettingstarted/getting-started-splunk-setup.md

@@ -1,9 +1,8 @@
 # Splunk setup
 To ensure proper integration for SC4S and Splunk, perform the following tasks in your Splunk instance:

-1. Create indexes within Splunk.
+1. Create copies of your SC4S indexes in Splunk.


originally Splunk has just a few indexes; SC4S requires way more to be there, so the user needs to create them, not sure what copies mean in this context

mstopa-splunk · 2024-05-16T09:59:29Z

docs/gettingstarted/getting-started-splunk-setup.md

@@ -30,7 +31,7 @@ SC4S maps each sourcetype to the following indexes by default:
 * `print`
 * `_metrics` (Optional opt-in for SC4S operational metrics; ensure this is created as a metrics index)

-You can also you create your own indexes in Splunk. See [Create custom indexes]( https://docs.splunk.com/Documentation/Splunk/9.2.1/Indexer/Setupmultipleindexes) for more information.
+If you create custom indexes in SC4S you must also create them in Splunk. See [Create custom indexes]( https://docs.splunk.com/Documentation/Splunk/9.2.1/Indexer/Setupmultipleindexes) for more information.


We don't create indexes in SC4S so maybe:

Suggested change

If you create custom indexes in SC4S you must also create them in Splunk. See [Create custom indexes]( https://docs.splunk.com/Documentation/Splunk/9.2.1/Indexer/Setupmultipleindexes) for more information.

If you use custom indexes in SC4S you must also create them in Splunk. See [Create custom indexes]( https://docs.splunk.com/Documentation/Splunk/9.2.1/Indexer/Setupmultipleindexes) for more information.

mstopa-splunk · 2024-05-16T10:03:48Z

docs/gettingstarted/getting-started-splunk-setup.md

+
+## Step 1: Create indexes within Splunk
+
+SC4S maps each sourcetype to the following indexes by default:


can we make it explicit:

Suggested change

SC4S maps each sourcetype to the following indexes by default:

SC4S maps each sourcetype to the following indexes by default. Make sure to create them in Splunk.

mstopa-splunk · 2024-05-16T10:05:23Z

docs/gettingstarted/getting-started-splunk-setup.md

 Splunk type.

+Keep in mind the following best practices specific to HEC for SC4S:
+* Make sure that the HEC token created for SC4S has permissions to add events to `main`, `_metrics`, and all other event destination indexes.


_metrics are not events, so maybe:

Suggested change

* Make sure that the HEC token created for SC4S has permissions to add events to `main`, `_metrics`, and all other event destination indexes.

* Make sure that the HEC token created for SC4S has permissions to write to `_metrics` and all event destination indexes.

mstopa-splunk · 2024-05-16T10:14:58Z

@jenworthington ready for the final pass

jenworthington requested a review from mstopa-splunk April 23, 2024 07:12

jenworthington requested a review from rjha-splunk as a code owner April 23, 2024 07:12

Merge branch 'main' into jenworthington-patch-4

900926c

mstopa-splunk changed the title ~~Update getting-started-splunk-setup.md~~ docs: update getting-started-splunk-setup.md Apr 23, 2024

Add new doc proposition

293fb9e

rjha-splunk reviewed Apr 23, 2024

View reviewed changes

docs/gettingstarted/getting-started-splunk-setup.md Outdated Show resolved Hide resolved

rjha-splunk reviewed Apr 23, 2024

View reviewed changes

docs/gettingstarted/getting-started-splunk-setup.md Outdated Show resolved Hide resolved

rjha-splunk reviewed Apr 23, 2024

View reviewed changes

jenworthington added 2 commits April 23, 2024 16:19

Update getting-started-splunk-setup-new.md

f44f355

More edits, added some new links to the Splunk Enterprise docs

Update getting-started-splunk-setup-new.md

9869523

Overwrite getting-started-splunk-setup.md with the updated version

9628cbe

mstopa-splunk reviewed Apr 24, 2024

View reviewed changes

jenworthington and others added 2 commits April 28, 2024 18:56

Update k8s-microk8s.md

b0760aa

Merge branch 'main' into jenworthington-patch-4

3a750d1

mstopa-splunk reviewed Apr 29, 2024

View reviewed changes

mstopa-splunk mentioned this pull request Apr 29, 2024

Update k8s-microk8s.md #2427

Closed

mstopa-splunk and others added 2 commits May 8, 2024 12:30

Revert changes in k8s-microk8s.md

8588c8d

Merge branch 'main' into jenworthington-patch-4

212c0ac

jenworthington added 2 commits May 10, 2024 11:46

Update getting-started-splunk-setup.md

d7c0346

Update getting-started-splunk-setup.md

fcadfc8

I made these changes a while back but maybe i did something weird with the branching? So I redid them and hopefully second time is the charm. ;)

mstopa-splunk reviewed May 16, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: update getting-started-splunk-setup.md #2417

docs: update getting-started-splunk-setup.md #2417

jenworthington commented Apr 23, 2024

mstopa-splunk commented Apr 23, 2024

mstopa-splunk commented Apr 23, 2024

mstopa-splunk commented Apr 23, 2024

rjha-splunk left a comment

mstopa-splunk commented Apr 23, 2024

jenworthington commented Apr 24, 2024 •

edited

mstopa-splunk Apr 24, 2024

jenworthington Apr 26, 2024

mstopa-splunk May 8, 2024

mstopa-splunk May 16, 2024

mstopa-splunk Apr 24, 2024

jenworthington Apr 26, 2024

mstopa-splunk Apr 29, 2024

jenworthington May 10, 2024

mstopa-splunk May 16, 2024 •

edited

mstopa-splunk commented Apr 24, 2024

mstopa-splunk Apr 29, 2024

jenworthington May 10, 2024

mstopa-splunk May 16, 2024

mstopa-splunk commented May 8, 2024 •

edited

mstopa-splunk May 16, 2024

mstopa-splunk May 16, 2024

mstopa-splunk May 16, 2024

mstopa-splunk May 16, 2024

mstopa-splunk commented May 16, 2024


		## Step 1: Create indexes within Splunk

		SC4S maps each sourcetype to the following indexes by default:

	If you create custom indexes in SC4S you must also create them in Splunk. See [Create custom indexes]( https://docs.splunk.com/Documentation/Splunk/9.2.1/Indexer/Setupmultipleindexes) for more information.
	If you use custom indexes in SC4S you must also create them in Splunk. See [Create custom indexes]( https://docs.splunk.com/Documentation/Splunk/9.2.1/Indexer/Setupmultipleindexes) for more information.

	* Make sure that the HEC token created for SC4S has permissions to add events to `main`, `_metrics`, and all other event destination indexes.
	* Make sure that the HEC token created for SC4S has permissions to write to `_metrics` and all event destination indexes.

docs: update getting-started-splunk-setup.md #2417

Are you sure you want to change the base?

docs: update getting-started-splunk-setup.md #2417

Conversation

jenworthington commented Apr 23, 2024

mstopa-splunk commented Apr 23, 2024

mstopa-splunk commented Apr 23, 2024

mstopa-splunk commented Apr 23, 2024

rjha-splunk left a comment

Choose a reason for hiding this comment

mstopa-splunk commented Apr 23, 2024

jenworthington commented Apr 24, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mstopa-splunk May 16, 2024 • edited

Choose a reason for hiding this comment

mstopa-splunk commented Apr 24, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mstopa-splunk commented May 8, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mstopa-splunk commented May 16, 2024

jenworthington commented Apr 24, 2024 •

edited

mstopa-splunk May 16, 2024 •

edited

mstopa-splunk commented May 8, 2024 •

edited