Add InfluxDB V2 support

[gcrevell]

This addresses issue #159, adding support for InfluxDB 2 to Varken.

To use InfluxDB 2, start an instance it. I recommend this docker image. The config is pretty different from version 1, requiring an organization name and token for authentication.

I recommend creating the bucket in the database ahead of time, because then a token can be created that only allows writes to the single bucket. If creating a bucket is needed, varken will need an all-access token, instead of a read-write token.

I've tested this myself and it seems to work pretty well, but let me know if there's any questions or issues!

[Steve0320]

I've pulled this down and it seems to work well! The only issue I've noticed is that the bucket setting in the ini doesn't work - looks like the setter line in iniparser.py is missing.

[gcrevell]

@Steve0320 Thanks! Fixed that up so now its using the correct bucket

[Ornias1993]

Looks pretty solid!

[dirtycajunrice]

I will try to look this evening

[YouveGotMeowxy]

I cant seem to get this working. Did I do it wrong?

I'm using: boerderij/varken:develop with InfluxDB 2.1.1

My .ini:

[influxdb]
url = influxdb
port = 8086
ssl = false
verify_ssl = false
username = 1232132132323231
password = O_TGI_LjlX0tawrr<redacted>BfAFB6bYoz4Q==

where my username is my organization and pass is the token.

I have also tried using the text version of my organization wrapped in double quotes. No matter what I use I just keep getting an unauthorized error in the log.

today at 9:18:42 AM2022-03-19 09:18:42 : DEBUG : connectionpool : Starting new HTTP connection (1): ********luxdb:8086
today at 9:18:42 AM2022-03-19 09:18:42 : DEBUG : connectionpool : http://********luxdb:8086 "GET /ping HTTP/1.1" 204 0
today at 9:18:42 AM2022-03-19 09:18:42 : INFO : dbmanager : Influxdb version: 2.1.1, 2.1.1
today at 9:18:42 AM2022-03-19 09:18:42 : DEBUG : connectionpool : http://********luxdb:8086 "GET /query?q=SHOW+DATABASES&db=varken HTTP/1.1" 401 48
today at 9:18:42 AMTraceback (most recent call last):
today at 9:18:42 AM  File "/app/Varken.py", line 94, in <module>
today at 9:18:42 AM    DBMANAGER = DBManager(CONFIG.influx_server)
today at 9:18:42 AM  File "/app/varken/dbmanager.py", line 25, in __init__
today at 9:18:42 AM    databases = [db['name'] for db in self.influx.get_list_database()]
today at 9:18:42 AM  File "/usr/local/lib/python3.9/site-packages/influxdb/client.py", line 570, in get_list_database
today at 9:18:42 AM    return list(self.query("SHOW DATABASES").get_points())
today at 9:18:42 AM  File "/usr/local/lib/python3.9/site-packages/influxdb/client.py", line 411, in query
today at 9:18:42 AM    response = self.request(
today at 9:18:42 AM  File "/usr/local/lib/python3.9/site-packages/influxdb/client.py", line 286, in request
today at 9:18:42 AM    raise InfluxDBClientError(response.content, response.status_code)
today at 9:18:42 AMinfluxdb.exceptions.InfluxDBClientError: 401: {"code":"unauthorized","message":"Unauthorized"}
today at 9:18:44 AMContainer stopped

Using the same organization number and token to add the datasource in the InfluxDB to connect to the varken bucket connects fine.

What am I screwing up?

[samwiseg0]

@gcrevell Can you look at a few things?

• Rebase from develop
• Set this PR to draft
• Change the merge branch to develop not master
• I would like this to keep a single influx config section. Can you put in a check for /ping and use the v1 config/db vs v2? As well as keep a single config section? This will allow backward compatibility and keep the config clean and easy to use.

[n1nj4888]

Hi @gcrevell. Thanks for looking into this - InfluxDBv2 support would be great in Varken! Are you still planning on addressing the items samwiseg0 mentions above?

[d-mcknight]

I rebased this branch on develop but ran into trouble with the URL validator when I pass my influx URL (https://influx.domain.tld). I bypassed the validation and everything is working for me, but I'm not great with regex so I'm not sure how to fix that check.

If there's still interest I can address the other feedback here and open a PR

[YouveGotMeowxy]

but I'm not great with regex so I'm not sure how to fix that check.

Which line? This one?

If there's still interest I can address the other feedback here and open a PR

I'm 10000000% interested in a working influxV2 version! :D

[d-mcknight]

Which line? This one?

I believe it should be handled as a domain, but I just bypassed the check to get it running.

I'm 10000000% interested in a working influxV2 version! :D

I did add some automation and push my working branch to ghcr.io/d-mcknight/varken:dev to run for myself if you want to test out that image.

[YouveGotMeowxy]

@d-mcknight

but ran into trouble with the URL validator when I pass my influx URL (https://influx.domain.tld).

What if you remove the https:// in your config and only use influx.domain.tld (and remove your bypass).

[noraemsu]

Note I'm using the boerderij/varken:develop branch
If anyone still has issues with getting influxdb 2.0 working with varken then just follow the steps here
#159 (comment)
The only thing I did differently was using -org-id <org-id> instead of -o <your org>
And using an All Access API Token for the -t <admin token>

Like the following lines:
influx bucket list -org-id <org-id> -t <admin token>

influx v1 dbrp create --db varken --rp varken-rp --bucket-id <bucket-id> --default -org-id <org-id> -t <admin token>

influx v1 auth create --read-bucket <bucket-id> --write-bucket <bucket-id> --username varken-user -org-id <org-id> -t <admin token>

For some reason using -o <your org> arg caused some error for me, but using -org-id <org-id> fixed it.

Then in varken.ini
username is in the 3rd command above
password is prompted after the 3rd command

url = ip/url
port = 8086
ssl = false
verify_ssl = false
username = <username>
password = <password>

Hope this helps someone.