Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Secure forward #220

Open
wants to merge 9 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 2 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ ALL_IMAGES := \
v0.12/debian-gcs:v0.12.43-debian-gcs,v0.12-debian-gcs,debian-gcs \
v0.12/debian-papertrail:v0.12.43-debian-papertrail,v0.12-debian-papertrail,debian-papertrail \
v0.12/debian-syslog:v0.12.43-debian-syslog,v0.12-debian-syslog,debian-syslog \
v0.12/debian-secureforward:v0.12.43-debian-secureforward,v0.12-debian-secureforward,debian-secureforward,secureforward \
v0.12/debian-graylog:v0.12.43-debian-graylog,v0.12-debian-graylog,debian-stable-graylog,debian-graylog \
v0.12/debian-logzio:v0.12.43-debian-logzio,v0.12-debian-logzio,debian-logzio \
v0.12/debian-kafka:v0.12.43-debian-kafka,v0.12-debian-kafka,debian-kafka \
Expand All @@ -47,6 +48,7 @@ ALL_IMAGES := \
v1.2/debian-stackdriver:v1.2.6-debian-stackdriver,v1.2-debian-stackdriver \
v1.2/debian-s3:v1.2.6-debian-s3,v1.2-debian-s3 \
v1.2/debian-syslog:v1.2.6-debian-syslog,v1.2-debian-syslog \
v1.2/debian-secureforward:v1.2.6-debian-secureforward,v1.2-debian-secureforward \
v1.2/debian-gcs:v1.2.6-debian-gcs,v1.2-debian-gcs \
v1.2/debian-graylog:v1.2.6-debian-graylog,v1.2-debian-graylog,debian-stable-graylog \
v1.2/debian-papertrail:v1.2.6-debian-papertrail,v1.2-debian-papertrail \
Expand Down
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
- `v0.12-debian-gcs` [docker-image/v0.12/debian-gcs/Dockerfile](docker-image/v0.12/debian-gcs/Dockerfile)
- `v0.12-debian-papertrail` [docker-image/v0.12/debian-papertrail/Dockerfile](docker-image/v0.12/debian-papertrail/Dockerfile)
- `v0.12-debian-syslog` [docker-image/v0.12/debian-syslog/Dockerfile](docker-image/v0.12/debian-syslog/Dockerfile)
- `v0.12-debian-secureforward` [docker-image/v0.12/debian-secureforward/Dockerfile](docker-image/v0.12/debian-secureforward/Dockerfile)
- `v0.12-debian-graylog` [docker-image/v0.12/debian-graylog/Dockerfile](docker-image/v0.12/debian-graylog/Dockerfile)
- `v0.12-debian-logzio` [docker-image/v0.12/debian-logzio/Dockerfile](docker-image/v0.12/debian-logzio/Dockerfile)
- `v0.12-debian-kafka` [docker-image/v0.12/debian-kafka/Dockerfile](docker-image/v0.12/debian-kafka/Dockerfile)
Expand All @@ -31,6 +32,7 @@
- `v1.2-debian-stackdriver` [docker-image/v1.2/debian-stackdriver/Dockerfile](docker-image/v1.2/debian-stackdriver/Dockerfile)
- `v1.2-debian-s3` [docker-image/v1.2/debian-s3/Dockerfile](docker-image/v1.2/debian-s3/Dockerfile)
- `v1.2-debian-syslog` [docker-image/v1.2/debian-syslog/Dockerfile](docker-image/v1.2/debian-syslog/Dockerfile)
- `v1.2-debian-secureforward` [docker-image/v1.2/debian-secureforward/Dockerfile](docker-image/v1.2/debian-secureforward/Dockerfile)
- `v1.2-debian-gcs` [docker-image/v1.2/debian-gcs/Dockerfile](docker-image/v1.2/debian-gcs/Dockerfile)
- `v1.2-debian-graylog` [docker-image/v1.2/debian-graylog/Dockerfile](docker-image/v1.2/debian-graylog/Dockerfile)
- `v1.2-debian-papertrail` [docker-image/v1.2/debian-papertrail/Dockerfile](docker-image/v1.2/debian-papertrail/Dockerfile)
Expand All @@ -52,8 +54,8 @@
- `v0.12-alpine-graylog` [docker-image/v0.12/alpine-graylog/Dockerfile](docker-image/v0.12/alpine-graylog/Dockerfile)
- `v0.12-alpine-logzio` [docker-image/v0.12/alpine-logzio/Dockerfile](docker-image/v0.12/alpine-logzio/Dockerfile)
- `v0.12-alpine-kafka` [docker-image/v0.12/alpine-kafka/Dockerfile](docker-image/v0.12/alpine-kafka/Dockerfile)
- `v0.12-alpine-splunkhec` [docker-image/v0.12/alpine-splunkhec/Dockerfile](docker-image/v0.12/alpine-splunkhec/Dockerfile)
- `v0.12-alpine-kinesis` [docker-image/v0.12/alpine-kinesis/Dockerfile](docker-image/v0.12/alpine-kinesis/Dockerfile)
- `v0.12-alpine-splunkhec` [docker-image/v0.12/alpine-splunkhec/Dockerfile](docker-image/v0.12/alpine-splunkhec/Dockerfile)

## What is Fluentd?

Expand Down
1 change: 1 addition & 0 deletions docker-image/v0.12/debian-secureforward/.dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
**/*.gitkeep
53 changes: 53 additions & 0 deletions docker-image/v0.12/debian-secureforward/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# AUTOMATICALLY GENERATED
# DO NOT EDIT THIS FILE DIRECTLY, USE /templates/Dockerfile.erb

FROM fluent/fluentd:v0.12.43-debian

LABEL maintainer="Eduardo Silva <eduardo@treasure-data.com>"
USER root
WORKDIR /home/fluent
ENV PATH /fluentd/vendor/bundle/ruby/2.3.0/bin:$PATH
ENV GEM_PATH /fluentd/vendor/bundle/ruby/2.3.0
ENV GEM_HOME /fluentd/vendor/bundle/ruby/2.3.0
# skip runtime bundler installation
ENV FLUENTD_DISABLE_BUNDLER_INJECTION 1

COPY Gemfile* /fluentd/
RUN buildDeps="sudo make gcc g++ libc-dev ruby-dev libffi-dev" \
&& apt-get update \
&& apt-get upgrade -y \
&& apt-get install \
-y --no-install-recommends \
$buildDeps net-tools \
&& gem install bundler --version 1.16.2 \
&& bundle config silence_root_warning true \
&& bundle install --gemfile=/fluentd/Gemfile --path=/fluentd/vendor/bundle \
&& SUDO_FORCE_REMOVE=yes \
apt-get purge -y --auto-remove \
-o APT::AutoRemove::RecommendsImportant=false \
$buildDeps \
&& rm -rf /var/lib/apt/lists/* \
&& gem sources --clear-all \
&& rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem

# Copy configuration files
COPY ./conf/fluent.conf /fluentd/etc/
COPY ./conf/systemd.conf /fluentd/etc/
COPY ./conf/kubernetes.conf /fluentd/etc/

# Copy plugins
COPY plugins /fluentd/plugins/
COPY entrypoint.sh /fluentd/entrypoint.sh

# Environment variables
ENV FLUENTD_OPT=""
ENV FLUENTD_CONF="fluent.conf"

# jemalloc is memory optimization only available for td-agent
# td-agent is provided and QA'ed by treasuredata as rpm/deb/.. package
# -> td-agent (stable) vs fluentd (edge)
#ENV LD_PRELOAD="/usr/lib/libjemalloc.so.2"


# Run Fluentd
CMD ["/fluentd/entrypoint.sh"]
14 changes: 14 additions & 0 deletions docker-image/v0.12/debian-secureforward/Gemfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# AUTOMATICALLY GENERATED
# DO NOT EDIT THIS FILE DIRECTLY, USE /templates/Gemfile.erb

source "https://rubygems.org"

gem "fluentd", "0.12.43"
gem "oj", "3.5.1"
gem "fluent-plugin-secure-forward"
gem "fluent-plugin-record-reformer"
gem "fluent-plugin-rewrite-tag-filter"
gem "fluent-plugin-secure-forward"
gem "fluent-plugin-kubernetes_metadata_filter"
gem "ffi"
gem "fluent-plugin-systemd"
102 changes: 102 additions & 0 deletions docker-image/v0.12/debian-secureforward/Gemfile.lock
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
GEM
remote: https://rubygems.org/
specs:
activesupport (5.2.1)
concurrent-ruby (~> 1.0, >= 1.0.2)
i18n (>= 0.7, < 2)
minitest (~> 5.1)
tzinfo (~> 1.1)
addressable (2.5.2)
public_suffix (>= 2.0.2, < 4.0)
concurrent-ruby (1.0.5)
cool.io (1.5.3)
domain_name (0.5.20180417)
unf (>= 0.0.5, < 1.0.0)
ffi (1.9.25)
fluent-plugin-kubernetes_metadata_filter (1.2.1)
fluentd (>= 0.12.0)
kubeclient (~> 1.1.4)
lru_redux
fluent-plugin-record-reformer (0.9.1)
fluentd
fluent-plugin-rewrite-tag-filter (1.6.0)
fluentd (>= 0.12.0, < 0.14.0)
fluent-plugin-secure-forward (0.4.5)
fluentd (>= 0.10.46)
proxifier
resolve-hostname
fluent-plugin-systemd (0.0.11)
fluentd (~> 0.12)
systemd-journal (~> 1.3.2)
fluentd (0.12.43)
cool.io (>= 1.2.2, < 2.0.0)
http_parser.rb (>= 0.5.1, < 0.7.0)
json (>= 1.4.3)
msgpack (>= 0.5.11, < 2)
sigdump (~> 0.2.2)
string-scrub (>= 0.0.3, <= 0.0.5)
tzinfo (>= 1.0.0)
tzinfo-data (>= 1.0.0)
yajl-ruby (~> 1.0)
http (0.9.8)
addressable (~> 2.3)
http-cookie (~> 1.0)
http-form_data (~> 1.0.1)
http_parser.rb (~> 0.6.0)
http-cookie (1.0.3)
domain_name (~> 0.5)
http-form_data (1.0.3)
http_parser.rb (0.6.0)
i18n (1.1.1)
concurrent-ruby (~> 1.0)
json (2.1.0)
kubeclient (1.1.4)
activesupport
http (= 0.9.8)
recursive-open-struct (= 1.0.0)
rest-client
lru_redux (1.1.0)
mime-types (3.2.2)
mime-types-data (~> 3.2015)
mime-types-data (3.2018.0812)
minitest (5.11.3)
msgpack (1.2.4)
netrc (0.11.0)
oj (3.5.1)
proxifier (1.0.3)
public_suffix (3.0.3)
recursive-open-struct (1.0.0)
resolve-hostname (0.1.0)
rest-client (2.0.2)
http-cookie (>= 1.0.2, < 2.0)
mime-types (>= 1.16, < 4.0)
netrc (~> 0.8)
sigdump (0.2.4)
string-scrub (0.0.5)
systemd-journal (1.3.3)
ffi (~> 1.9)
thread_safe (0.3.6)
tzinfo (1.2.5)
thread_safe (~> 0.1)
tzinfo-data (1.2018.6)
tzinfo (>= 1.0.0)
unf (0.1.4)
unf_ext
unf_ext (0.0.7.5)
yajl-ruby (1.4.1)

PLATFORMS
ruby

DEPENDENCIES
ffi
fluent-plugin-kubernetes_metadata_filter
fluent-plugin-record-reformer
fluent-plugin-rewrite-tag-filter
fluent-plugin-secure-forward
fluent-plugin-systemd
fluentd (= 0.12.43)
oj (= 3.5.1)

BUNDLED WITH
1.16.6
21 changes: 21 additions & 0 deletions docker-image/v0.12/debian-secureforward/conf/fluent.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@

# AUTOMATICALLY GENERATED
# DO NOT EDIT THIS FILE DIRECTLY, USE /templates/conf/fluent.conf.erb

@include systemd.conf
@include kubernetes.conf

<match **>
@type secure_forward
shared_key "#{ENV['FLUENT_SECURE_FORWARD_SHARED_KEY']}"
self_hostname client.fqdn.local
secure true
ca_cert_path "#{ENV['FLUENT_SECURE_FORWARD_CERT_PATH']}"
config_path /etc/logentries/tokens.yaml
<server>
host "#{ENV['FLUENT_SECURE_FORWARD_REMOTE_HOST']}"
port "#{ENV['FLUENT_SECURE_FORWARD_PORT'] || 24284 }"
</server>

</match>