Skip to content

Commit

Permalink
Release 6.5.2 (#303)
Browse files Browse the repository at this point in the history 
* Fix structlog dependency for app (#280)

* zipfile fix (#284)

* Fix bug 286 random token replacement (#287)

* Fix bug 286 random token replacement

* Change perdayvolume generator logic to get random token value replacement

* Versioning scheme (#278)

* [global] perDayVolume (#288)

* exclude global from perDayVolume assignment

* Address comment

* Fix security vulnerability issue (#289)

* Fix custom plugin stale docs (#290)

* Server fix (#293)

* Flag added

* server fix for count and env clean

* Fix bug 285 (#297)

* Add syslogAddHeader config directive (#296)

* Add syslog header to event in syslog mode

* timezone setting bugfix #249

* Using multiprocess pool to address the OOM issue (#301)

* Using multiprocess pool to address the OOM issue

* Fix test case fail

* Remove workerQueue unfinished tasks (#302)

* Bumped version to 6.5.2
  • Loading branch information
Li Wu committed Oct 8, 2019
1 parent 0a9c727 commit 8f9a1c5
Show file tree
Hide file tree
Showing 23 changed files with 296 additions and 180 deletions.
14 changes: 12 additions & 2 deletions docs/CONFIGURE.md
View file
Expand Up @@ -382,8 +382,10 @@ specifically be supported by all plugins. Plugins that write to files like spool
* Set event sourcetype in Splunk to <source> Defaults to 'eventgen' if none specified.

host = <host>
* ONLY VALID WITH outputMode SPLUNKSTREAM
* Set event host in Splunk to <host>. Defaults to 127.0.0.1 if none specified.
* When outputMode is splunkstream, set event host in Splunk to <host>.
* When outputMode is syslogout and syslogAddHeader is set to true, add initial header with hostname <host>,
see syslogAddHeader for details.
* Defaults to 127.0.0.1 if none specified.

host.token = <regular expression>
* PCRE expression used to identify the host name (or partial name) for replacement.
Expand Down Expand Up @@ -420,6 +422,14 @@ specifically be supported by all plugins. Plugins that write to files like spool
* Only supports UDP ports
* Required

syslogAddHeader = true | false
* Controls whether syslog messages should be prefixed with an RFC3164 compliant header
including the host value defined for the sample.
* Useful in situations where you want to output generated events to syslog and make it
possible for the receiving syslog server to use the sample's defined host value instead of
the hostname of the host that eventgen is running on.
* Defaults to false

###### tcpout
tcpDestinationHost = <host>
* Defaults to 127.0.0.1
Expand Down
16 changes: 13 additions & 3 deletions docs/CONTRIBUTE_CODE.md
View file
Expand Up @@ -5,18 +5,26 @@ If you want to contribute code to eventgen, please read over the following guide

## Pull request guidelines


If you want to contribute to an eventgen repo, please use a GitHub pull request. This is the fastest way for us to evaluate your code and to merge it into the code base. Please don’t file an issue with snippets of code. Doing so means that we need to manually merge the changes in and update any appropriate tests. That decreases the likelihood that your code is going to get included in a timely manner. Please use pull requests.


## Release versioning guidelines

Major Release — Increment the first digit by 1 if the new features break backwards compatibility/current features

Minor Release — Increment the middle digit by 1 if the new features don’t break any existing features and are compatible with the app in it’s current state

Patch Release — Increment the last digit by 1 if you’re publishing bug/patch fixes to your app

### Get started

If you’d like to work on a pull request and you’ve never submitted code before, follow these steps:
1. fork eventgen to your github workspace
2. If you want to fix bugs or make enhancement, please make sure there is a issue in eventgen project. Refer [this guide](FILE_ISSUES.md) to create a issue.


After that, you’re ready to start working on code.


### Working on the code

The process of submitting a pull request is fairly straightforward and generally follows the same pattern each time:
Expand Down Expand Up @@ -75,6 +83,7 @@ The message summary should be a one-sentence description of the change, and it m

**Note**: please squash you changes in one commit before firing the pull request. One commit in one PR keeps the git history clean.


#### Step 3: Rebase onto upstream

Before you send the pull request, be sure to rebase onto the upstream source. This ensures your code is running on the latest available code. We prefer rebase instead of merge when upstream changes. Rebase keeps the git history clearer.
Expand All @@ -83,6 +92,7 @@ git fetch upstream
git rebase upstream/master
```


#### Step 4: Run the tests

The is a place holder as well. We should write about
Expand All @@ -101,6 +111,7 @@ Next, push your changes to your clone:
git push origin fix/issue123
```


#### Step 6: Submit the pull request

Before creating a pull request, here are some recommended **check points**.
Expand All @@ -118,7 +129,6 @@ Next, create a pull request from your branch to the eventgen develop branch.
Mark @lephino , @arctan5x , @jmeixensperger , @li-wu , @GordonWang as the reviewers.



## Code style and formatting tools

Since Eventgen is written in python, we apply a coding style based on [PEP8](https://www.python.org/dev/peps/pep-0008/).
Expand Down
40 changes: 20 additions & 20 deletions docs/Gemfile.lock
View file
@@ -1,12 +1,12 @@
GEM
remote: https://rubygems.org/
specs:
activesupport (4.2.10)
activesupport (4.2.11.1)
i18n (~> 0.7)
minitest (~> 5.1)
thread_safe (~> 0.3, >= 0.3.4)
tzinfo (~> 1.1)
addressable (2.5.2)
addressable (2.6.0)
public_suffix (>= 2.0.2, < 4.0)
coffee-script (2.4.1)
coffee-script-source
Expand All @@ -16,7 +16,7 @@ GEM
commonmarker (0.17.13)
ruby-enum (~> 0.5)
concurrent-ruby (1.1.5)
dnsruby (1.61.2)
dnsruby (1.61.3)
addressable (~> 2.5)
em-websocket (0.5.1)
eventmachine (>= 0.12.9)
Expand All @@ -27,13 +27,13 @@ GEM
execjs (2.7.0)
faraday (0.15.4)
multipart-post (>= 1.2, < 3)
ffi (1.10.0)
ffi (1.11.1)
forwardable-extended (2.6.0)
gemoji (3.0.0)
github-pages (197)
activesupport (= 4.2.10)
gemoji (3.0.1)
github-pages (198)
activesupport (= 4.2.11.1)
github-pages-health-check (= 1.16.1)
jekyll (= 3.7.4)
jekyll (= 3.8.5)
jekyll-avatar (= 0.6.0)
jekyll-coffeescript (= 1.1.1)
jekyll-commonmark-ghpages (= 0.1.5)
Expand Down Expand Up @@ -81,13 +81,13 @@ GEM
octokit (~> 4.0)
public_suffix (~> 3.0)
typhoeus (~> 1.3)
html-pipeline (2.10.0)
html-pipeline (2.12.0)
activesupport (>= 2)
nokogiri (>= 1.4)
http_parser.rb (0.6.0)
i18n (0.9.5)
concurrent-ruby (~> 1.0)
jekyll (3.7.4)
jekyll (3.8.5)
addressable (~> 2.4)
colorator (~> 1.0)
em-websocket (~> 0.5)
Expand Down Expand Up @@ -204,39 +204,39 @@ GEM
jekyll-feed (~> 0.9)
jekyll-seo-tag (~> 2.1)
minitest (5.11.3)
multipart-post (2.0.0)
nokogiri (1.10.2)
multipart-post (2.1.1)
nokogiri (1.10.4)
mini_portile2 (~> 2.4.0)
octokit (4.14.0)
sawyer (~> 0.8.0, >= 0.5.3)
pathutil (0.16.2)
forwardable-extended (~> 2.6)
public_suffix (3.0.3)
public_suffix (3.1.1)
rb-fsevent (0.10.3)
rb-inotify (0.10.0)
ffi (~> 1.0)
rouge (2.2.1)
ruby-enum (0.7.2)
i18n
ruby_dep (1.5.0)
rubyzip (1.2.2)
rubyzip (1.2.3)
safe_yaml (1.0.5)
sass (3.7.3)
sass (3.7.4)
sass-listen (~> 4.0.0)
sass-listen (4.0.0)
rb-fsevent (~> 0.9, >= 0.9.4)
rb-inotify (~> 0.9, >= 0.9.7)
sawyer (0.8.1)
addressable (>= 2.3.5, < 2.6)
faraday (~> 0.8, < 1.0)
sawyer (0.8.2)
addressable (>= 2.3.5)
faraday (> 0.8, < 2.0)
terminal-table (1.8.0)
unicode-display_width (~> 1.1, >= 1.1.1)
thread_safe (0.3.6)
typhoeus (1.3.1)
ethon (>= 0.9.0)
tzinfo (1.2.5)
thread_safe (~> 0.1)
unicode-display_width (1.5.0)
unicode-display_width (1.6.0)

PLATFORMS
ruby
Expand All @@ -245,4 +245,4 @@ DEPENDENCIES
github-pages

BUNDLED WITH
2.0.1
2.0.2
90 changes: 48 additions & 42 deletions docs/PLUGINS.md
View file
Expand Up @@ -16,29 +16,37 @@ Plugins inherit from a base plugin class and are placed in their appropriate dir
Let's take a look at the simplest plugin available to us, the Devnull output plugin:

```python
from __future__ import division
from outputplugin import OutputPlugin
import sys
from logging_config import logger


class DevNullOutputPlugin(OutputPlugin):
name = 'devnull'
MAXQUEUELENGTH = 1000
useOutputQueue = True

def __init__(self, sample):
OutputPlugin.__init__(self, sample)
def __init__(self, sample, output_counter=None):
OutputPlugin.__init__(self, sample, output_counter)
self.firsttime = True

def flush(self, q):
logger.info('flush data to devnull')
if self.firsttime:
self.f = open('/dev/null', 'w')
self.firsttime = False
buf = '\n'.join(x['_raw'].rstrip() for x in q)
self.f.write(buf)


def load():
"""Returns an instance of the plugin"""
return DevNullOutputPlugin

```

First, we import the OutputPlugin superclass. For output plugins, they define a constant MAXQUEUELENGTH to determine the maximum amount of items in queue before forcing a queue flush.
First, we import the OutputPlugin superclass. For output plugins, they define a constant `MAXQUEUELENGTH` to determine the maximum amount of items in queue before forcing a queue flush.

`useOutputQueue` is set to `True` here to use the output queue which functions as a reduce step when you need to maintain a single thread or a limited number of threads outputting data

``__init__()`` is very simple. It calls its superclass init and sets one variable, firsttime. ``flush()`` is also very simple.
If it's the first time, open the file /dev/null, otherwise, output the queue by writing it to the already open file.
Expand All @@ -56,34 +64,34 @@ class SplunkStreamOutputPlugin(OutputPlugin):
intSettings = [ 'splunkPort' ]
```

MAXQUEUELENGTH should look normal, but these other class variables need a little explanation.
`MAXQUEUELENGTH` should look normal, but these other class variables need a little explanation.

### Configuration Validation
Config validation is a modular system in Eventgen, and plugins must be allowed to specify additional configuration parameters that the main Eventgen will consider valid and store.
*Note that eventgen.conf.spec generation is not yet automated, which means plugins must ship with the default distribution and eventgen.conf.spec must be maintained manually.*
> Note that `eventgen.conf.spec` generation is not yet automated, which means plugins must ship with the default distribution and eventgen.conf.spec must be maintained manually.
Eventually spec file generation will be automated as well.

The main configuration of Eventgen validates itself by a list of configuration parameters assigned by type, and each of the configuration parameters is validated by that type.
The settings list is required:

* validSettings | Defines the list of valid settings for this plugin
* validSettings: Defines the list of valid settings for this plugin

The following lists are optional and likely to be used by many plugins:

* intSettings | Will validate the settings as integers
* floatSettings | Will validate the settings as floating point numbers
* boolSettings | Will validate the settings as booleans
* jsonSettings | Will validate the settings as a JSON string
* defaultableSettings | Settings which can be specified in the [global] stanza and will pass down to individual stanzas
* complexSettings | A dictionary of lists or function callbacks, containing a setting name with list of valid options or a callback function to validate the setting.
* intSettings: Will validate the settings as integers
* floatSettings: Will validate the settings as floating point numbers
* boolSettings: Will validate the settings as booleans
* jsonSettings: Will validate the settings as a JSON string
* defaultableSettings: Settings which can be specified in the [global] stanza and will pass down to individual stanzas
* complexSettings: A dictionary of lists or function callbacks, containing a setting name with list of valid options or a callback function to validate the setting.

## Methods required per plugin type

Each plugin type will define a different method required.

**Plugin Type** | **Method** | **Returns** | **Notes**
--- | --- | --- | ---
Rater | ``rate()`` | Integer count of events to generate | n/a
Rater | ``rate()`` | Integer count of events to generate | N/A
Generator | ``gen(count, earliest, latest) `` | Success (0) | Events get put into an output queue by calling the Sample's ``send()`` or ``bulksend()`` methods in the output object.
Output | ``flush(q)`` | Success (0) | Gets a deque list q to operate upon and output as configured.

Expand All @@ -92,48 +100,46 @@ Output | ``flush(q)`` | Success (0) | Gets a deque list q to operate upon and ou
We reviewed a simple Output Plugin earlier, let's look at a simple Generator Plugin:

```python
from __future__ import division
import datetime
from datetime import timedelta

from generatorplugin import GeneratorPlugin
import os
import logging
import datetime, time
import itertools
from collections import deque
from logging_config import logger


class WindbagGenerator(GeneratorPlugin):
def __init__(self, sample):
GeneratorPlugin.__init__(self, sample)

# Logger already setup by config, just get an instance
logger = logging.getLogger('eventgen')
globals()['logger'] = logger

from eventgenconfig import Config
globals()['c'] = Config()

def gen(self, count, earliest, latest):
l = [ {'_raw': '2014-01-05 23:07:08 WINDBAG Event 1 of 100000'} for i in xrange(count) ]

self._out.bulksend(l)
def gen(self, count, earliest, latest, samplename=None):
if count < 0:
logger.warning('Sample size not found for count=-1 and generator=windbag, defaulting to count=60')
count = 60
time_interval = timedelta.total_seconds((latest - earliest)) / count
for i in xrange(count):
current_time_object = earliest + datetime.timedelta(0, time_interval * (i + 1))
msg = '{0} -0700 WINDBAG Event {1} of {2}'.format(current_time_object, (i + 1), count)
self._out.send(msg)
return 0


def load():
return WindbagGenerator

```

For this generator plugin, notice we inherit from GeneratorPlugin instead of OutputPlugin. This plugin is also quite simple.
In its ``__init__()`` method, it calls the superclass ``__init__()`` and it sets up two global variables, c, which holds the config
(and is a Singleton pattern which can be instantiated many times) and a copy of the logger which we'll use for logging in most plugins.
For this generator plugin, notice we inherit from `GeneratorPlugin` instead of `OutputPlugin`. This plugin is also quite simple.

Secondly, it defines a gen() method, which generates ``count`` events between ``earliest`` and ``latest`` time. In this case, we ignore the timestamp and return just event text.
Then we call bulksend. This plugin has several performance optimizations: using a list constructor instead of a loop and using bulksend instead of send.
Secondly, it defines a `gen()` method, which generates ``count`` events between ``earliest`` and ``latest`` time. In this case, we ignore the timestamp and return just event text.
Then we call `bulksend`. This plugin has several performance optimizations: using a list constructor instead of a loop and using bulksend instead of send.
Let's see how this could be implemented in a slightly less performant but easier to understand way:

```python
def gen(self, count, earliest, latest):
for x in xrange(count):
self._sample.send({ '_raw': '2014-01-05 23:07:08 WINDBAG Event 1 of 100000' })

def gen(self, count, earliest, latest, samplename=None):
for i in xrange(count):
current_time_object = earliest + datetime.timedelta(0, time_interval * (i + 1))
msg = '{0} -0700 WINDBAG Event {1} of {2}'.format(current_time_object, (i + 1), count)
self._out.send(msg)
return 0
```

Expand All @@ -142,4 +148,4 @@ Here, we use ``send()`` instead of ``bulksend()`` and a loop to make it easier t
# Shipping a Plugin

When you've developed a plugin that you want to use in your app, shipping it with your app is easy.
Place any Eventgen plugin in your Splunk app's ``bin/`` directory and we'll search for and find any plugins referenced by a ``outputMode``, ``generator`` or ``rater`` config statement.
Place any Eventgen plugin in your Splunk app's ``bin/`` directory and we'll search for and find any plugins referenced by a ``outputMode``, ``generator`` or ``rater`` config statement.
3 changes: 3 additions & 0 deletions docs/REFERENCE.md
View file
Expand Up @@ -139,6 +139,9 @@ syslogDestinationPort = <port>
* Defaults to port 1514
* Only supports UDP ports
syslogAddHeader = true | false
* Defaults to false
tcpDestinationHost = <host>
* Defaults to 127.0.0.1
Expand Down
2 changes: 1 addition & 1 deletion splunk_eventgen/eventgen_api_server/eventgen_controller_api.py
View file
Expand Up @@ -35,7 +35,7 @@ def gather_response(response_number_target=0):
if not response_number_target:
response_number_target = int(self.redis_connector.message_connection.pubsub_numsub(self.redis_connector.servers_channel)[0][1])
response_num = 0
countdown = 1.5 / self.interval
countdown = 60 / self.interval
for i in range(0, int(countdown)):
if response_num == response_number_target:
break
Expand Down

0 comments on commit 8f9a1c5

Please sign in to comment.