feat(analytics): Add client-side functionality for analytics collection #5249
Conversation
rashil2000
force-pushed
the
analytics
branch
from
ffba1b4
to
d86b7c7
Compare
| # - Manifest count | ||
|
|
||
| . "$PSScriptRoot\..\lib\json.ps1" # 'ConvertToPrettyJson' | ||
|
|
There was a problem hiding this comment.
| if(!(Resolve-DnsName analytics.scoop.sh -ErrorAction SilentlyContinue)) { | |
| Write-Host "Could not resolve analytics.scoop.sh" | |
| exit 0 | |
| } | |
Just skip the whole script if analytics.scoop.sh can't be resolved (because it will get added to public blocklists 😄)
There was a problem hiding this comment.
Hmm, good idea. But I notice that it takes 10 seconds to time out, isn't that a bit much?
There was a problem hiding this comment.
Hmm, it returns instantly for me. Maybe because I use a local Adguard in my network. 🤔
There was a problem hiding this comment.
I noticed that it takes time when connected to my Azure VPN server. On other networks it returns instantly.
It's useful regardless, so I'll add it.
Could you tell why? I'll make necessary changes here if possible. |
|
Just don't like telemetry 😄 Of course, this analysis could help users pick up useful apps and help app developers know how popular their apps are. Apps tend to have looooooooooong EULA to say they don't collect user-identifiable data but who knows. BTW, why not generate random |
| 'TEAMCITY_VERSION' = 'TeamCity' | ||
| 'TRAVIS' = 'Travis CI' | ||
| }.GetEnumerator()) { | ||
| if (-not [String]::IsNullOrEmpty((Get-Item "Env:/$($ci_env.Key)" -ErrorAction Ignore).Value)) { |
There was a problem hiding this comment.
| if (-not [String]::IsNullOrEmpty((Get-Item "Env:/$($ci_env.Key)" -ErrorAction Ignore).Value)) { | |
| if ((Get-Item "Env:/$($ci_env.Key)" -ErrorAction Ignore).Value) { |
❯ if ('') { $true } else { $false }
False
And why there's / after Env:?
| } | ||
| } | ||
| foreach ($ci_env in 'BUILD_ID', 'CI') { | ||
| if (-not [String]::IsNullOrEmpty((Get-Item "Env:/$($ci_env)" -ErrorAction Ignore).Value)) { |
| if ([String]::IsNullOrEmpty((get_config ANALYTICS_ID))) { | ||
| set_config ANALYTICS_ID (New-Guid).Guid | Out-Null | ||
| } |
| # Known sources | ||
| if ($source -in $known_sources) { | ||
| return $true | ||
| } | ||
| # Local file paths, SSH remotes, and remotes with usernames | ||
| if ($source -match '^/[A-Za-z]:/|^[A-Za-z]:/|^\./|^\.\./|file:/|ssh:/|@') { | ||
| return $false | ||
| } | ||
| return $true |
There was a problem hiding this comment.
| # Known sources | |
| if ($source -in $known_sources) { | |
| return $true | |
| } | |
| # Local file paths, SSH remotes, and remotes with usernames | |
| if ($source -match '^/[A-Za-z]:/|^[A-Za-z]:/|^\./|^\.\./|file:/|ssh:/|@') { | |
| return $false | |
| } | |
| return $true | |
| # Local file paths, SSH remotes, and remotes with usernames | |
| if ($source -match '^/[A-Za-z]:/|^[A-Za-z]:/|^\./|^\.\./|file:/|ssh:/|@') { | |
| return $false | |
| } else { | |
| return $true | |
| } |
| set_config ANALYTICS_ID (New-Guid).Guid | Out-Null | ||
| } | ||
| $def_arch = Get-DefaultArchitecture | ||
| $known_sources = foreach ($item in (known_bucket_repos).PSObject.Properties) { $item.Value } |
There was a problem hiding this comment.
| $known_sources = foreach ($item in (known_bucket_repos).PSObject.Properties) { $item.Value } |
| $stats.machine = [ordered]@{ | ||
| Build = [System.Environment]::OSVersion.Version.ToString() | ||
| Arch = $def_arch | ||
| Desktop = (Get-ItemProperty -Path HKLM:\SOFTWARE\Microsoft\PowerShell\3\PowerShellEngine -Name 'PowerShellVersion').PowerShellVersion |
There was a problem hiding this comment.
All that information is also available in $PSVersionTable
| Arch = $def_arch | ||
| Desktop = (Get-ItemProperty -Path HKLM:\SOFTWARE\Microsoft\PowerShell\3\PowerShellEngine -Name 'PowerShellVersion').PowerShellVersion | ||
| Core = if (Get-Command pwsh -ErrorAction Ignore) { | ||
| (Get-Item (Get-Command pwsh).Source).VersionInfo.ProductVersionRaw.ToString() |
There was a problem hiding this comment.
| (Get-Item (Get-Command pwsh).Source).VersionInfo.ProductVersionRaw.ToString() | |
| (Get-Command pwsh).Version.ToString() |
| $branch = (Get-Content "$PSScriptRoot\..\.git\HEAD").Replace('ref: ', '') | ||
| "$(Get-Content (Join-Path "$PSScriptRoot\..\.git" $branch)) ($($branch.Split('/')[-1]))" | ||
| } elseif (Test-Path "$PSScriptRoot\..\CHANGELOG.md") { | ||
| (Select-String '^## .*([\d]{4}-[\d]{2}-[\d]{2})' "$PSScriptRoot\..\CHANGELOG.md").Matches.Groups[1].Value |
There was a problem hiding this comment.
| (Select-String '^## .*([\d]{4}-[\d]{2}-[\d]{2})' "$PSScriptRoot\..\CHANGELOG.md").Matches.Groups[1].Value | |
| (Select-String -Pattern '^## \[v([\d.]+)\].*?([\d-]+)$' -Path "$PSScriptRoot\..\CHANGELOG.md").Matches.Groups[1].Value |
The version number is better. I don't think there's user used v0.1.0 and lower.
| } elseif (Test-Path "$PSScriptRoot\..\CHANGELOG.md") { | ||
| (Select-String '^## .*([\d]{4}-[\d]{2}-[\d]{2})' "$PSScriptRoot\..\CHANGELOG.md").Matches.Groups[1].Value | ||
| } else { | ||
| "" |
There was a problem hiding this comment.
See above, the CHANGELOG.md should be there, so this will be never reached.
| $stats.apps += $newitem | ||
| } | ||
|
|
||
| $payload = $stats | ConvertToPrettyJSON |
There was a problem hiding this comment.
The payload needn't be prettified, right?
|
I have some internal apps and buckets hosted on secret but publicly accessible https sites, would this script expose it? I think it would be better to only report on officially maintained apps and buckets |
|
I also want to point out that even though I am personally fine with some level of analytics, it is still a bit odd to add them without much warning down the line. I would suggest an opt-in, but is on by default with sufficient notice on new installs. I could also see tracking private usage would be useful for detecting malware using scoop. In which case this data should not be made public and rather sent to security researchers if something looks unusual. |
"machine": {
"Build": "10.0.22621.0",
"Arch": "64bit",
"Desktop": "5.1.22621.1",
"Core": "7.3.2.0",
"Scoop": "d86b7c72e281375e8209f8f23fdedfee790cd9a6 (5249--analytics)"
},Commit hash could be used to link the data to a Github user via the search function. (e.g. data is send from a user working on a PR) |
TODO: Remove analytics_id and analytics_timestamp from scoop-export.ps1
Description
This PR adds the client-side functionality for sending anonymous usage statistics to a remote server.
Script is disabled for now and will be enabled after the server-side code has been written, open-sourced (under the ScoopInstaller org) and deployed to analytics.scoop.sh where stats will be available for everyone to see (similar to https://formulae.brew.sh/analytics).
Motivation and Context
Relates to #3781
How Has This Been Tested?
Analytics are sent only:
ANALYTICS_DISABLEconfig setting is not trueThe analytics script is triggered on each scoop subcommand (except
helpsubcommands), and only after the above conditions are satisfied. The script runs asynchronously in background after the issued subcommand is complete, not disturbing it in any way.Following buckets (and their apps) are not sent:
Click here to see a sample payload (from my own machine)
{ "id": "a346c2ad-40ec-492c-b413-92e48aa9bfca", "machine": { "Build": "10.0.25236.0", "Arch": "64bit", "Desktop": "5.1.25236.1000", "Core": "7.2.7.0", "Scoop": "d7bfe52122ed0e9a237b5dc068fba44268780991 (develop)" }, "buckets": [ { "Name": "extras", "Source": "https://github.com/ScoopInstaller/Extras", "Updated": "2022-11-13T01:58:30+05:30", "Manifests": 1730 }, { "Name": "main", "Source": "https://github.com/ScoopInstaller/Main", "Updated": "2022-11-13T01:58:03+05:30", "Manifests": 1115 }, { "Name": "versions", "Source": "https://github.com/ScoopInstaller/Versions", "Updated": "2022-11-13T02:05:04+05:30", "Manifests": 394 } ], "apps": [ { "Name": "7zip", "Version": "22.01", "Source": "main", "Updated": "2022-07-18T16:29:38.4104603+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "adb", "Version": "33.0.3", "Source": "main", "Updated": "2022-08-26T18:47:20.6517042+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "autohotkey", "Version": "1.1.35.00", "Source": "extras", "Updated": "2022-11-03T10:45:46.9934122+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "bat", "Version": "0.22.1", "Source": "main", "Updated": "2022-09-11T21:51:27.848777+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "bottom", "Version": "0.6.8", "Source": "main", "Updated": "2022-02-02T20:19:01.2676394+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "busybox-lean", "Version": "4784-g5507c8744", "Source": "main", "Updated": "2022-11-12T08:36:05.7609229+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "clangd", "Version": "15.0.3", "Source": "main", "Updated": "2022-10-25T15:29:14.6106689+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "clink", "Version": "1.4.0", "Source": "main", "Updated": "2022-11-12T08:36:07.8455872+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "clink-completions", "Version": "0.4.1", "Source": "main", "Updated": "2022-09-05T02:32:24.5270678+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "dark", "Version": "3.11.2", "Source": "main", "Updated": "2021-12-06T01:48:10.1904677+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "delta", "Version": "0.14.0", "Source": "main", "Updated": "2022-09-05T02:32:28.1011936+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "dog", "Version": "0.1.0", "Source": "main", "Updated": "2022-10-29T22:59:46.6813256+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "fd", "Version": "8.5.2", "Source": "main", "Updated": "2022-11-12T08:36:11.2443471+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "fzf", "Version": "0.35.0", "Source": "main", "Updated": "2022-11-12T08:36:14.6452925+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "gdu", "Version": "5.20.0", "Source": "main", "Updated": "2022-10-25T15:29:26.5753313+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "gh", "Version": "2.20.0", "Source": "main", "Updated": "2022-11-12T08:36:20.9086765+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "gitify", "Version": "4.3.1", "Source": "extras", "Updated": "2021-12-15T01:30:46.8572359+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "gsudo", "Version": "2.0.2", "Source": "main", "Updated": "2022-11-12T08:36:33.7199305+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "hyperfine", "Version": "1.15.0", "Source": "main", "Updated": "2022-09-07T22:40:20.0246336+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "imagemagick-lean", "Version": "7.1.0", "Source": "main", "Updated": "2022-02-27T19:53:50.6609166+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "innounp", "Version": "0.50", "Source": "main", "Updated": "2022-01-12T12:00:06.4527548+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "jcpicker", "Version": "5.6", "Source": "extras", "Updated": "2021-11-08T21:51:52.1650152+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "kate", "Version": "22.08.0", "Source": "extras", "Updated": "2022-09-25T11:45:19.0748957+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "lessmsi", "Version": "1.10.0", "Source": "main", "Updated": "2022-01-05T00:03:13.5737302+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "links", "Version": "2.28", "Source": "main", "Updated": "2022-10-02T14:02:13.8829189+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "micaforeveryone", "Version": "1.2.0.1", "Source": "extras", "Updated": "2022-09-05T02:32:39.4182987+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "mpv", "Version": "0.34.0", "Source": "extras", "Updated": "2022-03-24T23:08:57.4505682+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "mpxplay", "Version": "1.66", "Source": "main", "Updated": "2021-11-08T21:51:52.8000909+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "ncspot", "Version": "0.11.2", "Source": "main", "Updated": "2022-10-25T15:29:50.9748914+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "neovim", "Version": "0.8.0", "Source": "main", "Updated": "2022-10-02T14:02:27.6418235+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "nodejs-lts", "Version": "18.12.0", "Source": "main", "Updated": "2022-10-28T18:34:18.1685793+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "PowerSession", "Version": "1.4.7", "Source": "main", "Updated": "2022-06-09T19:59:13.9960795+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "python", "Version": "3.11.0", "Source": "main", "Updated": "2022-10-25T15:31:08.123046+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "ripgrep", "Version": "13.0.0", "Source": "main", "Updated": "2021-08-08T00:20:47.8968202+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "scrcpy", "Version": "1.24", "Source": "main", "Updated": "2022-05-01T16:02:58.4764196+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "screentogif", "Version": "2.37.1", "Source": "extras", "Updated": "2022-08-23T15:49:38.641573+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "sd", "Version": "0.7.5", "Source": "main", "Updated": "2021-07-11T19:49:01.4241453+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "speedtest-cli", "Version": "1.2.0", "Source": "main", "Updated": "2022-08-12T16:56:35.1612707+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "spicetify-cli", "Version": "2.14.1", "Source": "main", "Updated": "2022-10-28T18:34:22.5286442+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "spotify", "Version": "1.1.98.691.gf759311c", "Source": "extras", "Updated": "2022-11-12T08:13:03.7349411+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "spotify-tui", "Version": "0.25.0", "Source": "main", "Updated": "2022-10-08T21:59:09.9354848+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "starship", "Version": "1.11.0", "Source": "main", "Updated": "2022-10-25T15:32:10.0603874+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "ttyd", "Version": "1.7.2", "Source": "main", "Updated": "2022-11-02T11:26:56.0439679+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "vifm", "Version": "0.12.1", "Source": "extras", "Updated": "2022-09-25T11:45:57.6355513+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "vncviewer", "Version": "6.22.826", "Source": "extras", "Updated": "2022-09-30T10:36:32.3383899+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "win-gpg-agent", "Version": "1.6.3", "Source": "extras", "Updated": "2022-04-27T18:11:42.320004+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "yedit", "Version": "1.80", "Source": "main", "Updated": "2022-07-06T12:54:01.8532661+05:30", "Global": false, "Arch": "64bit", "Status": "OK" }, { "Name": "yt-dlp", "Version": "2022.11.11", "Source": "main", "Updated": "2022-11-12T08:36:54.3443086+05:30", "Global": false, "Arch": "64bit", "Status": "OK" } ] }As you can see, the data is completely anonymous and contains no user-identifiable information.
Checklist:
developbranch.