Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OneDrive Client Version v2.5.0-rc2 #2709

Open
wants to merge 132 commits into
base: master
Choose a base branch
from

Conversation

abraunegg
Copy link
Owner

@abraunegg abraunegg commented Apr 28, 2024

Summary of Changes

Important

OneDrive Client for Linux v2.5.0 contains a database schema change to support the additional functionality and bug fixes in the v2.5.x branch. As such, on your first use of this client version, the database schema will be updated and the client will perform an entire re-validation of your data - both locally and online which will take an extended period of time.

Caution

OneDrive Client for Linux v2.5.0 is not backwards compatible. Do not use v2.5.0 and v2.4.x applications at the same time against the same Microsoft OneDrive account. If using v2.5.0, you must upgrade all your clients to v2.5.0

Caution

OneDrive Business Shared Folder Sync has been 100% re-written in v2.5.0. If you are using this feature, please read the new documentation carefully.

Feature Requests

  • Implement Feature Request: Multi-threaded uploading/downloading of files FR-232
  • Implement Feature Request: Renaming/Relocation of OneDrive Business shared folders FR-1251
  • Implement Feature Request: Support the syncing of individual business shared files FR-1300
  • Implement Feature Request: Log when manual Authorization is required when using --auth-files FR-2475
  • Implement Feature Request: Add cmdline parameter to display (human readable) quota status FR-2359
  • Implement Feature Request: Ability to set --disable-download-validation from Docker environment variable FR-2559
  • Implement Feature Request: Ability to set --sync-shared-files from Docker environment variable FR-2680

Bug Fixes

  • Fix Bug: Business shared folders with same basename get merged BUG-2434
  • Fix Bug: --dry-run prevents authorization BUG-2451
  • Fix Bug: Log timestamps lacking trailing zeros, leading to poor log file output alignment BUG-2479
  • Fix Bug: Subscription ID already exists when using webhooks BUG-2501
  • Fix Bug: Not all files being downloaded when API data includes HTML ASCII Control Sequences BUG-2562
  • Fix Bug: vim backups when editing files cause edited file to be deleted rather than the edited file being uploaded BUG-2586
  • Fix Bug: skip_dir does not always work as intended for all directory entries BUG-2590
  • Fix Bug: Online date being changed in download-only mode BUG-2591
  • Fix Bug: Resolve that upload session are not canceled with resync option BUG-2624
  • Fix Bug: Local files should be safely backed up when the item is not in sync locally to prevent data loss when they are deleted online BUG-2625
  • Fix Bug: Synced file is removed when updated on the remote while being processed by onedrive BUG-2699

Other Changes | Fixes

  • Implement buffered logging so that all logging from all upload & download activities are handled correctly
  • Replace polling monitor loop with blocking wait
  • Update how the application utilises curl to fix socket reuse
  • Various performance enhancements with many thanks to @JC-comp
  • Implement refactored OneDrive API logic with many thanks to @JC-comp
  • Enforcement of operational conflicts
  • Enforcement of application configuration defaults and minimums
  • Implement application output to detail upload|download failures at the end of a sync process
  • Utilise threadsafe sqlite DB access methods
  • Various code cleanup and optimisations

Documentation

  • Most documentation has been totally overhauled

Documentation List

* Initial commit of onedrive-v2.5.0-alpha-5
* Delete documents again as POSIX rename failure
* Re-add documents
* Delete readme & changelog due to POSIX issue
Re-add documents
* Fix zero byte file handling
* Just sync the triggered file, no scan of parent directory
* Cleanup array memory before we start adding files
* Update inotify event handling where files might be temporarily moved by applications. This scenario is common with certain text editors (like Vim with specific configurations), which can lead to misleading file deletion detections.
* Add logging output for what IP protocol is going to be used to access Microsoft OneDrive
Implement refactored and optimised applicationChangeWhereResyncRequired() function
* Re-introduce CURLOPT_FORBID_REUSE setting from 'alpha-2'
Fix that 'config' does not load when a deprecated option is at the beginning of the config file
* Update 'root:' removal for 'skip_dir' path checking
Malformed config line should cause the application to exit. This then prevents malformed lines which might be valid configuration items, from not being read in, thus, causing incorrect application function based on expectations.
* If requesting --help, dont attempt to load any 'config' file
* Remove potentially redundant applicability check of a path if this is already in the database. This is a manual merge of #2525
* Implement Docker options to utilise --disable-download-validation and --disable-upload-validation
* Ensure that we load the 'skip_file' defaults when attempting to compare any configuration file entry changes
Update comment for libcurl forbid_reuse
Fix reported bug where the online date is being changed in download-only mode
Fix that --dry-run prevents authorization
* If timestamp needs to be corrected, return false so that the correct actions can be taken
* If in a --download-only scenario, do not update the timestamp online
* Fix debug output
Reinstate safeRename for online item moves
* Add further timestamp qualifications
* Add UTC note when detailing that there is a time difference
* Add PR 2527v2 to 'alpha-5': Fix edge case file upload bugs when using --local-first
* Clarify CURLOPT_CONNECTTIMEOUT in documentation
* Clarify CURLOPT_TIMEOUT in documentation
* Add 'threads' as a config file option to control the number of threads used, default of 8, max of 16
* Change CURLOPT_CONNECTTIMEOUT value to v2.4.x value default of 10 seconds
* Configure libcurl to reuse connections
* Update code based on fixing valgrind issues
Add specific example around using 'sync_list' and only including a child of a parent in the 'sync_dir'
* Update spacing
* Release files for 2.5.0-rc2
* Code changes from 2.5.0-rc1 --> 2.5.0-rc2 (#2686)
* Update docs regarding Ubuntu 24.04
abraunegg and others added 24 commits April 28, 2024 20:26
* Fix TaskPool compatibility with minimum DMD and LDC versions
* Update Debian Docker file to not specify compiler
* Fix arm64 compiler issue when calculating transfer ETA
* Update docker.md to clarify where to install Docker from
* Potentially resolve #2699 - needs testing by reporter
* Update Alpine to 3.19, GO 1.22, GOSU 1.17
* Update Fedora to 40, GO 1.22, GOSU 1.17
* Add reusable cache for storing the deltaLink between sync loops when using --monitor
* Add comments around the TaskPool shutdown process
* Cleanup CurlEngine logging output
* Align various items to coding style
* Fix up processing dot output when in non-verbose mode
* Add JSON Processing Elapsed Time so performance of JSON element processing can be tracked if required
Update URLs: http:// → https://
* Fix misspellings found by codespell
* Fix merge conflict

---------

Co-authored-by: abraunegg <alex.braunegg@gmail.com>
* Update UML diagrams based on #2715 changes
* Fix suppression
* Fix local deletion of a OneDrive Business Shared Folder so that the online deletion is of the 'link' on our account, not the remote data as this should remain untouched
* Fix that new local folders + files in OneDrive Business Shared Folders cannot be created (regression)
* As reported by @bpozdena , @aothmane-control - a segfault exists on exit, which has been tracked back to the threaded logging functions. This commit should resolve this segfault on exit.

This commit has been developed and tested using Arch Linux:

NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="38;2;23;147;209"
HOME_URL="https://archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://gitlab.archlinux.org/groups/archlinux/-/issues"
PRIVACY_POLICY_URL="https://terms.archlinux.org/docs/privacy-policy/"
LOGO=archlinux-logo

Installed using archlinux-2024.05.01-x86_64.iso

The version of 'ldc' that was installed, as detected by 'configure' was 1.37.0

-----------------------------------------
checking for a BSD-compatible install... /usr/bin/install -c
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for dmd... no
checking for ldc2... ldc2
checking version of D compiler... 1.37.0
checking for curl... yes
checking for sqlite... yes
checking for notify... yes
configure: creating ./config.status
config.status: creating Makefile
config.status: creating contrib/pacman/PKGBUILD
config.status: creating contrib/spec/onedrive.spec
config.status: creating onedrive.1
config.status: creating contrib/systemd/onedrive.service
config.status: creating contrib/systemd/onedrive@.service
-----------------------------------------
* Restore destructor that was missed
* Add debug logging for HTTP Response Headers
* Add debug logging for HTTP Status Line
* Ensure all logging is flushed correctly and log buffers are flushed
* Use forceExit(); in a consistent manner, which shuts down logging and flushes any remaining logs in buffer
* Remove log output that was erroneously left in
* When CURLOPT_NOSIGNAL is set to 1, libcurl will not use any code that installs signal handlers nor will it use any code that might cause signals to be sent to the process by libcurl. This is safer for multi-threaded applications.
* Update wording as to what is being accessed
* Ensure that syncEngineInstance is shutdown the correct way, checking that it has been initialised before attempting a shutdown
* Cleanup code where required in preparation for further testing on other platforms
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants