alternate way to count asset downloads

[stolarczyk]

raised by @nmagee

The FS used by the container is read-only, so we can't update the config file in order to store download counts:

INFO: ('10.0.3.50', 36128) - "GET /asset/ERCC92/bowtie1/archive HTTP/1.1" 500

ERROR: Exception in ASGI application

Traceback (most recent call last):

 File "/usr/local/lib/python3.7/site-packages/uvicorn/protocols/http/httptools_impl.py", line 368, in run_asgi

   result = await app(self.scope, self.receive, self.send)

 File "/usr/local/lib/python3.7/site-packages/starlette/applications.py", line 133, in __call__

   await self.error_middleware(scope, receive, send)

 File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 172, in __call__

   raise exc from None

 File "/usr/local/lib/python3.7/site-packages/starlette/middleware/errors.py", line 150, in __call__

   await self.app(scope, receive, _send)

 File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 73, in __call__

   raise exc from None

 File "/usr/local/lib/python3.7/site-packages/starlette/exceptions.py", line 62, in __call__

   await self.app(scope, receive, sender)

 File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 585, in __call__

   await route(scope, receive, send)

 File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 207, in __call__

   await self.app(scope, receive, send)

 File "/usr/local/lib/python3.7/site-packages/starlette/routing.py", line 40, in app

   response = await func(request)

 File "/usr/local/lib/python3.7/site-packages/fastapi/routing.py", line 117, in app

   raw_response = await dependant.call(**values)

 File "/app/refgenieserver/main.py", line 67, in download_asset

   update_stats(rgc, genome, asset)

 File "/app/refgenieserver/helpers.py", line 96, in update_stats

   rgc.write()

 File "/usr/local/lib/python3.7/site-packages/yacman/yacman.py", line 34, in write

   with open(filename, 'w') as f:

OSError: [Errno 30] Read-only file system: '/genomes/genomes.yaml'

We need an alternate way to track downloads:

Other suggestions if you need alternate ways to track counts – you could +1 to a specific key in Redis (there is a ‘counter’ data type made for that), which we have available, or to a more robust DB either relational or nosql.

---

To do:

• revert f486b1d

• implement new approach

[nmagee]

So possible new approaches:

The Redis counter approach just involves setting up a key and then hitting it with an INCR request each time you want to up the counter. You could write some basic logic to check for the existence of a key/value pair for any specific archive, and create it if it doesn't exist, or tick up 1 for each time it's called.

redis> SET mykey "10"
"OK"
redis> INCR mykey
(integer) 11
redis> GET mykey
"11"
redis> 

https://redis.io/commands/INCR

pip install redis

...

>>> import redis
>>> r = redis.Redis(host='localhost', port=6379, db=0)
>>> r.set('foo', 'bar')
True
>>> r.get('foo')
'bar'

You could try this out locally by running below -- but it won't have persistent storage after restarts (this is enabled on the server)

docker run -d -p 6379:6379 redis

[nsheff]

can keys have slashes? like, our key would be {genome}/{asset}.

[nmagee]

Yes.

nem2p > redis-cli -h localhost
localhost:6379> set neal/home dick-woods-road
OK
localhost:6379> get neal/home
"dick-woods-road"

[nsheff]

@MichalStolarczyk if you're working on this, make sure to make it optional... I'd rather not introduce this as a requirement since it's sort of a 'nice-to-have' feature. so it shouldn't add any new dependencies, etc. just use it if available

[nmagee]

Also, let me know if you want to go down this particular path. I would just need to make the Redis address available for you from another host.

[nmagee]

Michal, unfortunately with firewall rules I cannot reach the DCOS redis service from our current swarm, so I am running one with persistent storage on host 10.250.3.170 (the Swarm master). This means that in python you can instantiate and use redis with the snippet below.

It also seems wise that if you incorporate Redis-based counters as an option and for portability, it might be best to pass the redis host in as an env variable. Maybe the port as well?

import redis
import os

REDIS_HOST = os.environ['REDIS_HOST']
REDIS_PORT = os.environ['6379']
r = redis.Redis(host=REDIS_HOST, port=REDIS_PORT, db=0)

key = "genome11/asset37"

exists = r.exists(key)
if exists:
  r.incr(key)
  val = r.get(key)
  print(str(val))

else:
  r.set(key, '0')

(One quirk to note: By default, all responses are returned as bytes in Python 3 and str in Python 2. Having started working with redis well before Python3, I often forget to recast into str ;-) )