Request: add option to control time delta between two --progress outputs

[alexschlueter]

Have you checked borgbackup docs, FAQ, and open GitHub issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

ISSUE (feature request)

System information. For client/server mode post info for both machines.

Your borg version (borg -V).

borg 1.1.16

Operating system (distribution) and version.

Debian 5.10.205-2 (2023-12-31) x86_64 GNU/Linux

Hardware / network configuration, and filesystems used.

Data on Hetzner storage box mounted via SMB on host, backup to SSD connected to a raspberry pi reachable from host over ssh over wireguard tunnel.

How much data is handled by borg?

400 GB

Full borg commandline that lead to the problem (leave away excludes and passwords)

borg --rsh 'ssh -i /root/.ssh/id_rsa_nopass_borg' --lock-wait 600 --info create --show-rc --compression lz4 --stats --progress --files-cache ctime,size ssh://pi@10.0.0.2:50022/media/pi/Crucial2TB/backup::20240111174342.external-folder.data-external-movies-photos-syncthing .

Describe the problem you're observing.

Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

Include any warning/errors/backtraces from the system logs

Please add an option to control the amount of time between two prints to console by --progress.

My use case: I'm calling borg from a larger backup script, which catches borgs output, checks if borg is hanging, and logs the output to console / log files. To check if borg is hanging, I need borg to print status updates periodically for slow actions, e.g. rechunking of a large amount of data. However, none of the current logging options really work for me:

• --progress gives output too often and the carriage return \r, which would normally overwrite the last progress line, doesn't work when catching output, so my log files fill up and other important output is pushed out of view
• --list may also give too much uninteresting output if a large number of files is processed. I don't think the --filter options really help me.

Idea: Introduce something like the restic option into borg:

RESTIC_PROGRESS_FPS                 Frames per second by which the progress bar is updated

I think this could be easily implemented, as the Statistics.show_progress function in archive.py already has a dt argument, which seems to be set to 0.2 currently.

[ThomasWaldmann]

--progress does not mix nicely with --list, they are not intended to be used together.

did you check the json based log and progress options already? they are intended for wrappers.

btw, I am not sure you can use the progress or list output to detect whether there is something hanging. timing depends on a lot of things and also there are parts in borg with more and with less progress output.

[alexschlueter]

Thank you for your reply!

--progress does not mix nicely with --list, they are not intended to be used together.

I'm not mixing them, just listing the options I have tried.

did you check the json based log and progress options already? they are intended for wrappers.

I did not know about those, I will look into it. Thank you for the suggestion.

btw, I am not sure you can use the progress or list output to detect whether there is something hanging. timing depends on a lot of things and also there are parts in borg with more and with less progress output.

Do you know of a better way? I had some disk problems on the backup target, resulting in borg getting stuck without output and I needed to detect that in my backup wrapper so I can at least try my other backup targets. --progress prints every dt secs for create, which works well enough for me with a large "stuck timeout".

Would a pull request for a BORG_PROGRESS_FPS env var have a chance to be accepted?

[alexschlueter]

Okay, I'm now using --log-json and filtering the progress events myself in the wrapper. However, this results in a lot of duplicated / copied code to bring the messages I do want to print into the same readable format as the messages formatted by borg (which I was happy with, just not with the high frequency). So I still feel like BORG_PROGRESS_FPS would be worthwhile to add.