Grafana Dashboards

[devopstales]

I think it would be helpful if the benji project shared a generic grafana dashboard.json file to help users discover what key things they should be monitoring via prometheus.

[elemental-lf]

Yes, that would be helpful. But to be honest I'm still a little bit of novice with all things Prometheus and Grafana as well and I'm not even sure if the currently exposed metrics are actually useful. It will be some time until I'll look into this more deeply. But I'd accept, review and merge contributions in the mean time.

[allenporter]

Hi, i'm interested in helping on this one. I just installed benji, and it seems to be working well, and now i'd like to keep it that way with some monitoring and alerting.

I'm using benji via the helm chart, so the way that makes sense for me to think about this is:

• Align on the right set of prometheus metrics (if needed, anything in addition to the existing cron job metrics)
• Sketch out some example prometheus alert rules (again if needed to supplement cron job alerts)
• Make it easy to integrate with the prometheus operators with options for the helm chart that can add ServiceMonitor and PrometheusRule
• Make a basic grafana dashboard
• Make it easy to include grafana dashboards in the helm chart via configmaps.

If that makes sense, would you prefer we track these items in separate issues or just use this one?

[allenporter]

I wrote up some initial thoughts in a document and would love feedback.

[elemental-lf]

@allenporter I'm sorry that it took me so long to respond to you. My Grafana knowledge is very limited but I few comments about your text:

• Re backup target metrics: Plotting backup duration by PVC and not only namespace might also be useful. There might be a type of graph where a summary by namespace and individual durations could be integrated into one graph.
• Re storage metrics: When looking at the whole storage the shared value won't be useful as it will always be zero. There is an explanation of the values in the documentation at https://benji-backup.me/statistics.html.

Have you made any progress based on your document in the meantime?

[allenporter]

Hi, I think the first step after this was the issue related to key group. I was waiting to see how review of that PR was recieved before proceeding.

Now I have forgotten all of this, but can re-review and see if there are some ideas here worth persuing.

Is this a project you'd be active on giving feedback/reviewing if I were to push a little on it? (Basically curious how active you'll be going forward. I appreciate you triaging everything from the last 5 months today)

[allenporter]

@elemental-lf do you have any docs or a short writeup about setting up a benji development environment, running tests, etc? It could save me some time in getting started, as the project isn't setup how i'm used to when jumping into python projects. (Not that i'm an expert, i'm just seeing basic things like the tests need to be run in a specific way and i can't tell whats missing)

[elemental-lf]

@allenporter an example of how to setup the tests can be found inside this project's GitHub actions:

benji/.github/workflows/all-in-one.yaml

Lines 90 to 144 in ea0a37d

	tests:
	name: "Run tests"
	needs:
	- gather_info
	runs-on: ubuntu-latest
	strategy:
	matrix:
	python-version:
	- "3.6"
	- "3.7"
	- "3.8"
	- "3.9"
	steps:
	- name: Checkout repository files
	uses: actions/checkout@v2
	with:
	# This is needed so we can determine the version properly from git history.
	fetch-depth: 0
	- name: Set up Python
	uses: actions/setup-python@v2
	with:
	python-version: ${{ matrix.python-version }}
	- name: Install Benji and its dependencies into the virtual environment
	shell: bash
	run: |
	pip install --upgrade setuptools pip
	pip install --upgrade '.[s3,b2,compression,dev,doc]'
	pip freeze
	- name: Set up tests
	shell: bash
	run: |
	make -C tests/minio-setup up
	make -C tests/postgresql-setup up
	- name: Run tests
	shell: bash
	run: |
	pytest src/benji/tests/
	- name: Run packaging test
	shell: bash
	run: |
	python -m build --sdist --wheel --outdir dist/ .
	ls -al dist/
	env:
	BENJI_VERSION_OVERRIDE: "${{ needs.gather_info.outputs.public_version }}"
	- name: Tear down tests
	shell: bash
	run: |
	make -C tests/minio-setup down
	make -C tests/postgresql-setup down

Some tests can be enabled or disabled by setting an environment variable. See:

benji/.github/workflows/all-in-one.yaml

Lines 12 to 16 in ea0a37d

	UNITTEST_QUIET: "1"
	# UNITTEST_SKIP_POSTGESQL: "1"
	# UNITTEST_SKIP_S3: "1"
	UNITTEST_SKIP_B2: "1"
	UNITTEST_SKIP_NBD: "1"