WIP: Add per TTS config for free disk space

[krisgesling]

Description

This adds configurable values for required free disk space per TTS service.
If free disk space falls below these values, the temporary TTS cache will be curated.

Split from #2930

Problem # 1 - tests failing
This seems to be working in practice but I haven't got the tests to work.
At first I thought it was because the cached files didn't exist, hence didn't take up any space and so the calculation in curate_cache would have seen them as irrelevant. However populating them with some data didn't fix this.

Assuming this is a simple fix and just needs a fresh set of eyes on it to find my mistake...

Problem # 2 - Are we curating caches properly
It also raises a question for me as to whether the curate_cache should actually use an "OR" eg

if percent_free < min_free_percent or space.free < min_free_disk:

rather than the current "AND"

if percent_free < min_free_percent and space.free < min_free_disk:

This I think is partially how this issue arose. We currently have a call to the method that only includes the min_free_percent argument. It would be assumed that this would then be the factor on whether curation happened or not. However it must also fall below the default disk space argument. So even passing in 99% free disk percentage required, the cache will not be curated unless you also have <50MB of disk space as well.

How to test

• wait until unit tests work

Contributor license agreement signed?

• CLA (Whether you have signed a CLA - Contributor Licensing Agreement

[devops-mycroft]

Voight Kampff Integration Test Succeeded (Results)

[forslund]

I think the test issue is that the curate_mock.return_value isn't set in test_curate_cache_low_disk_space so the curation. Since you're creating actual files on disk it also works if I remove the mock completely.

Regarding Problem 2, the reason for this (as I recall it) is that it's designed for both small partitions and large ones. Say a 60 MB RAM Disk, in that case we don't want to start curating until we reach the percentage. On the other hand on a large disk where we can have 1% free and still have 1 GB disk space we shouldn't start to curate the cache.

[pep8speaks]

Hello @krisgesling! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2021-06-28 01:31:37 UTC

[devops-mycroft]

Voight Kampff Integration Test Failed (Results).
Mycroft logs are also available: skills.log, audio.log, voice.log, bus.log, enclosure.log

[devops-mycroft]

Voight Kampff Integration Test Failed (Results).
Mycroft logs are also available: skills.log, audio.log, voice.log, bus.log, enclosure.log

[devops-mycroft]

Voight Kampff Integration Test Failed (Results).
Mycroft logs are also available: skills.log, audio.log, voice.log, bus.log, enclosure.log

[krisgesling]

Was thinking about another approach and it seemed quick enough to warrant some example code. Haven't done any tests for this, or even verified that it runs as expected, interested to hear if I'm on the right path first.

The idea is that we provide a more flexible approach based off a set of limits, again defined per TTS engine. With this new method, all limits should be adhered to at all times. So there will always be the minimum free disk space whether that is defined as a percentage or a fixed MB value. Likewise, the cache will never exceed the max cache size values. A TTS engine or other service may pass as many or as few limits as they like. Passing a single limits dictionary should also make it easier to add any other parameters in the future (eg a whitelist of paths).

There are currently no sanity checks.

• I couldn't think of a scenario where min and max values might conflict.
• If someone sets no min free and their max size exceeds their disk size it will fill it right up.

Worst case scenarios here are that:

• values are too extreme and the cache is constantly cleared
• values are too lax and the disk gets full

In terms of the tts.{engine}.cache_limits dict - I tossed up between cache_limits and cache_config since it may in the future contain things that aren't explicitly limits but do relate to them. Compared to the preloaded_cache path which would certainly fit in a config dict, but not in a limits dict... very open to discussion or suggestions here.

[forslund]

What unit is the disk space given in? MB?

[forslund]

Since this is mainly a per system config. I.e the limits are something that is mainly dictated by the partition size where the cache is stored I think there should preferably be a config for all TTS's that can be overridden by the setting for the specific TTS engine.

(I assume there are cases where certain TTS engines would like to always re-synthetizise the audio and such so the per TTS may be useful in such cases)