THIS REPO IS DEPRECATED. PLEASE USE https://github.com/hubblestack/hubble-salt

Note

After syncing a new version of a beacon to salt, the salt-minion must be restarted to pick up the change. See saltstack/salt#35960 for more info

Introduction

Is your infrastructure immutable? Are you sure?

Pulsar is designed to monitor for file system events, acting as a real-time File Integrity Monitoring (FIM) agent. Pulsar is composed of a custom Salt beacon that watches for these events and hooks into the returner system for alerting and reporting.

In other words, you can recieve real-time alerts for unscheduled file system modifications anywhere you want to recieve them.

We've designed Pulsar to be lightweight and not dependent on a Salt Master. It simply watches for events and directly sends them to one of the Pulsar returner destinations (see the Quasar repository for more on these).

Two different installation methods are outlined below. The first method is more stable (and therefore recommended). This method uses Salt's package manager to track versioned, packaged updates to Hubble's components.

The second method installs directly from git. It should be considered bleeding edge and possibly unstable.

Installation

Each of the four HubbleStack components have been packaged for use with Salt's Package Manager (SPM). Note that all SPM installation commands should be done on the Salt Master.

Required Configuration

Salt's Package Manager (SPM) installs files into /srv/spm/{salt,pillar}. Ensure that this path is defined in your Salt Master's file_roots:

file_roots:
  - /srv/salt
  - /srv/spm/salt

Note

This should be the default value. To verify, run: salt-call config.get file_roots

Tip

Remember to restart the Salt Master after making any change to the configuration.

Required Packages

There is a hard requirement on the pyinotify Python library for each minion that will run the Pulsar FIM beacon.

Red Hat / CentOS

salt \* pkg.install python-inotify

Debian / Ubuntu

salt \* pkg.install python-pyinotify

Installation (Packages)

Installation is as easy as downloading and installing a package. (Note: in future releases you'll be able to subscribe directly to our HubbleStack SPM repo for updates and bugfixes!)

wget http://spm.hubblestack.io/pulsar/hubblestack_pulsar-2016.10.3-1.spm
spm local install hubblestack_pulsar-2016.10.3-1.spm

You should now be able to sync the new modules to your minion(s) using the sync_modules Salt utility:

salt \* saltutil.sync_beacons

Copy the pillar.example into your Salt pillar, renaming is as desired (perhaps hubblestack_pulsar.sls) and target it to selected minions.

base:
  '*':
    - hubblestack_pulsar

salt \* saltutil.refresh_pillar

Once these modules are synced you are ready to begin running the Pulsar beacon.

Skip to Usage <pulsar_usage>.

Installation (Manual)

Place _beacons/pulsar.py into your _beacons/ directory, and sync it to the minions.

git clone https://github.com/hubblestack/pulsar.git hubblestack-pulsar.git
cd hubblestack-pulsar.git
mkdir -p /srv/salt/_beacons/
cp _beacons/pulsar.py /srv/salt/_beacons/
mkdir /srv/salt/hubblestack_pulsar
cp hubblestack_pulsar/hubblestack_pulsar_config.yaml /srv/salt/hubblestack_pulsar
cp hubblestack_pulsar/pillar.example /srv/pillar/hubblestack_pulsar.sls
salt \* saltutil.sync_beacons

Target the copied hubblestack_pulsar.sls to selected minions.

base:
  '*':
    - hubblestack_pulsar

salt \* saltutil.refresh_pillar

Installation (GitFS)

This installation method subscribes directly to our GitHub repository, pinning to a tag or branch. This method requires no package installation or manual checkouts.

Requirements: GitFS support on your Salt Master.

/etc/salt/master.d/hubblestack-pulsar.conf

gitfs_remotes:
  - https://github.com/hubblestack/pulsar:
    - base: v2016.10.3

Tip

Remember to restart the Salt Master after applying this change.

Usage

Once Pulsar is fully running there isn't anything you need to do to interact with it. It simply runs quietly in the background and sends you alerts.

Configuration

The default Pulsar configuration (found in <pillar.example>) is meant to act as a template. It works in tandem with the <hubblestack_pulsar_config.yaml> file. Every environment will have different needs and requirements, and we understand that, so we've designed Pulsar to be flexible.

** pillar.example **

beacons:
  pulsar:
    paths:
      - /var/cache/salt/minion/files/base/hubblestack_pulsar/hubblestack_pulsar_config.yaml
schedule:
  cache_pulsar:
    function: cp.cache_file
    seconds: 86400
    args:
      - salt://hubblestack_pulsar/hubblestack_pulsar_config.yaml
    return_job: False

** hubblestack_pulsar_config **

/etc: { recurse: True, auto_add: True }
/bin: { recurse: True, auto_add: True }
/sbin: { recurse: True, auto_add: True }
/boot: { recurse: True, auto_add: True }
/usr/bin: { recurse: True, auto_add: True }
/usr/sbin: { recurse: True, auto_add: True }
/usr/local/bin: { recurse: True, auto_add: True }
/usr/local/sbin: { recurse: True, auto_add: True }
return: slack_pulsar
checksum: sha256
stats: True
batch: False

In order to receive Pulsar notifications you'll need to install the custom returners found in the Quasar repository.

Example of using the Slack Pulsar returner to recieve FIM notifications:

slack_pulsar:
  as_user: true
  username: calculon
  channel: hubble_pulsar
  api_key: xoxo-xxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxx

Tip

If you need to create a Slack bot, see: https://my.slack.com/services/new/bot

Excluding Paths

There may be certain paths that you want to exclude from this real-time FIM tool. This can be done using the exclude: keyword beneath any defined path.

/var:
  recurse: True
  auto_add: True
  exclude:
    - /var/log
    - /var/spool
    - /var/cache
    - /var/lock

Troubleshooting

If inotify is reporting that it can't create watches due to lack of disk space, but you have plenty of disk space and inodes available, then you may have to raise the max number of inotify watches.

To check the max number of inotify watches:

# cat /proc/sys/fs/inotify/max_user_watches

To set the max number of inotify watches:

# echo 20000 | sudo tee -a /proc/sys/fs/inotify/max_user_watches

Under The Hood

Pulsar is written as a Salt beacon, which requires the salt-minion daemon to be running. This then acts as an agent that watches for file system events using Linux's inotify subsystem.

Development

If you're interested in contributing to this project this section outlines the structure and requirements for Pulsar agent module development.

Anatomy of a Pulsar module

# -*- encoding: utf-8 -*-
'''
Pulsar agent

:maintainer: HubbleStack / owner
:maturity: 20160804
:platform: Linux
:requires: SaltStack

'''
from __future__ import absolute_import
import logging

All Pulsar agents should include the above header, expanding the docstring to include full documentation

Any Pulsar agent should be written as a beacon and send its return data directly to the Quasar endpoint(s). No communication with the master is required.

Contribute

If you are interested in contributing or offering feedback to this project feel free to submit an issue or a pull request. We're very open to community contribution.