Merge pull request #7051 from TheThingsNetwork/merge/v3.30-bookmarks

Merge v3.30

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -87,6 +87,13 @@ body:
         - label: I can help by doing more research.
         - label: I can help by implementing a fix after the proposal above is approved.
         - label: I can help by testing the fix before it's released.
+  - type: checkboxes
+    attributes:
+      label: Validation
+      description: Please check these boxes to confirm completion before closing the issue, or comment if not applicable.
+      options:
+        - label: The fix is tested in a staging environment.
+        - label: The fix is documented in [The Things Stack Documentation](https://www.thethingsindustries.com/docs/)
   - type: checkboxes
     attributes:
       label: Code of Conduct

.github/ISSUE_TEMPLATE/feature_request.yml

@@ -37,6 +37,13 @@ body:
         - label: I can help by doing more research.
         - label: I can help by implementing the feature after the proposal above is approved.
         - label: I can help by testing the feature before it's released.
+  - type: checkboxes
+    attributes:
+      label: Validation
+      description: Please check these boxes to confirm completion before closing the issue, or comment if not applicable.
+      options:
+        - label: The feature is tested in a staging environment.
+        - label: The feature is documented in [The Things Stack Documentation](https://www.thethingsindustries.com/docs/)
   - type: checkboxes
     attributes:
       label: Code of Conduct

.github/PULL_REQUEST_TEMPLATE.md

@@ -7,7 +7,9 @@ otherwise we will not be able to process this pull request.
 
 <!--
 A short summary, referencing related issues:
-Closes #0000, References #0000, etc.
+References #0000, etc.
+Pull requests may close issues (using Closes #0000) only for minor changes not needing tests on a staging environment.
+Typically, issues should be closed manually after validation.
 -->
 
 ...
@@ -58,6 +60,7 @@ implementation proposal in the referenced issues.
 - [ ] Scope: The referenced issue is addressed, there are no unrelated changes.
 - [ ] Compatibility: The changes are backwards compatible with existing API, storage, configuration and CLI, according to the compatibility commitments in `README.md` for the chosen target branch.
 - [ ] Documentation: Relevant documentation is added or updated.
-- [ ] The steps/process to test this feature are clearly explained including testing for regressions.
+- [ ] Testing: The steps/process to test this feature are clearly explained including testing for regressions.
+- [ ] Infrastructure: If infrastructural changes (e.g., new RPC, configuration) are needed, a separate issue is created in the infrastructural repositories.
 - [ ] Changelog: Significant features, behavior changes, deprecations and fixes are added to `CHANGELOG.md`.
 - [ ] Commits: Commit messages follow guidelines in `CONTRIBUTING.md`, there are no fixup commits left.

.github/workflows/go.yml

@@ -40,9 +40,9 @@ jobs:
       - name: Check for diff
         run: tools/bin/mage git:diff
       - name: Lint code
-        uses: golangci/golangci-lint-action@3cfe3a4abbb849e10058ce4af15d205b6da42804
+        uses: golangci/golangci-lint-action@v4
         with:
-          version: v1.55.2
+          version: v1.56.2
           only-new-issues: true
           skip-pkg-cache: true # Caching is set up by install-go-and-deps
           skip-build-cache: true # Caching is set up by install-go-and-deps

CHANGELOG.md

@@ -11,8 +11,6 @@ For details about compatibility between different releases, see the **Commitment
 
 ### Added
 
-- Fine grained NbTrans overrides for the dynamic ADR mode. See the `--mac-settings.adr.mode.dynamic.overrides.data-rate-[index].min-nb-trans` and `--mac-settings.adr.mode.dynamic.overrides.data-rate-[index].max-nb-trans` family of parameters.
-
 ### Changed
 
 ### Deprecated
@@ -23,6 +21,39 @@ For details about compatibility between different releases, see the **Commitment
 
 ### Security
 
+## [3.30.1] - unreleased
+
+### Added
+
+- Support fine-grained NbTrans controls while using Dynamic ADR mode in the Console.
+- User bookmark listing now supports filtering bookmarks by entity type. 
+  - This can be specified by setting `entity_types` field in `ListUserBookmarksRequest`.
+
+### Fixed
+
+- Show both AFcntDown and NFcntDown in the Device overview in the Console.
+- Fixes the keys displayed in the session information section of overview tab of an End Device in the Console - for LW 1.1.x, replaces NwkSKey with FNwkSIntKey. For LX 1.0.x display only the NwkSKey and AppSKey.
+
+## [3.30.0] - 2024-04-11
+
+### Added
+
+- Fine grained NbTrans overrides for the dynamic ADR mode. See the `--mac-settings.adr.mode.dynamic.overrides.data-rate-[index].min-nb-trans` and `--mac-settings.adr.mode.dynamic.overrides.data-rate-[index].max-nb-trans` family of parameters.
+- Support for storing user's bookmarks in the database.
+  - This requires a database schema migration (`ttn-lw-stack is-db migrate`) because of the new `user_bookmarks` table.
+- Support for storing user's Console Preferences in the database.
+  - This requires a database schema migration (`ttn-lw-stack is-db migrate`) because of the new `console_preferences` column in the `users` table.
+
+### Changed
+
+- Searching for collaborators displays only accounts that are not already attached to the entity's collaborator list.
+
+## [3.29.2] - 2024-03-26
+
+### Fixed
+
+- Memory leak in components which heavily use HTTP clients, such as the Application Server.
+
 ## [3.29.1] - 2024-03-05
 
 ### Added
@@ -2775,7 +2806,10 @@ For details about compatibility between different releases, see the **Commitment
 NOTE: These links should respect backports. See https://github.com/TheThingsNetwork/lorawan-stack/pull/1444/files#r333379706.
 -->
 
-[unreleased]: https://github.com/TheThingsNetwork/lorawan-stack/compare/v3.29.1...v3.29
+[unreleased]: https://github.com/TheThingsNetwork/lorawan-stack/compare/v3.30.1...v3.30
+[3.30.1]: https://github.com/TheThingsNetwork/lorawan-stack/compare/v3.30.0...v3.30.1
+[3.30.0]: https://github.com/TheThingsNetwork/lorawan-stack/compare/v3.29.2...v3.30.0
+[3.29.2]: https://github.com/TheThingsNetwork/lorawan-stack/compare/v3.29.1...v3.29.2
 [3.29.1]: https://github.com/TheThingsNetwork/lorawan-stack/compare/v3.29.0...v3.29.1
 [3.29.0]: https://github.com/TheThingsNetwork/lorawan-stack/compare/v3.28.2...v3.29.0
 [3.28.2]: https://github.com/TheThingsNetwork/lorawan-stack/compare/v3.28.1...v3.28.2

api/ttn/lorawan/v3/api.md

@@ -1054,7 +1054,7 @@ Application is the message that defines an Application in the network.
 
 ### <a name="ttn.lorawan.v3.ApplicationAccess">Service `ApplicationAccess`</a>
 
-The ApplicationAcces service, exposed by the Identity Server, is used to manage
+The ApplicationAccess service, exposed by the Identity Server, is used to manage
 API keys and collaborators of applications.
 
 | Method Name | Request Type | Response Type | Description |
@@ -2505,7 +2505,7 @@ The OAuth2 flows an OAuth client can use to get an access token.
 
 ### <a name="ttn.lorawan.v3.ClientAccess">Service `ClientAccess`</a>
 
-The ClientAcces service, exposed by the Identity Server, is used to manage
+The ClientAccess service, exposed by the Identity Server, is used to manage
 collaborators of OAuth clients.
 
 | Method Name | Request Type | Response Type | Description |
@@ -2994,7 +2994,8 @@ in a future version of The Things Stack.
 | `target_gateway_id` | [`string`](#string) |  | Gateway ID for the target gateway. This must be a unique value. If this is not set, the target ID for the target gateway will be set to `<gateway-eui>`. |
 | `target_gateway_server_address` | [`string`](#string) |  | Target Gateway Server Address for the target gateway. |
 | `cups_redirection` | [`CUPSRedirection`](#ttn.lorawan.v3.CUPSRedirection) |  | Parameters to set CUPS redirection for the gateway. |
-| `target_frequency_plan_id` | [`string`](#string) |  | Frequency plan ID of the target gateway. This equals the first element of the frequency_plan_ids field. |
+| `target_frequency_plan_id` | [`string`](#string) |  | Frequency plan ID of the target gateway. TODO: Remove this field (https://github.com/TheThingsIndustries/lorawan-stack/issues/4024) DEPRECATED: Use target_frequency_plan_ids instead. |
+| `target_frequency_plan_ids` | [`string`](#string) | repeated | Frequency plan IDs of the target gateway. |
 
 #### Field Rules
 
@@ -3005,6 +3006,7 @@ in a future version of The Things Stack.
 | `target_gateway_id` | <p>`string.max_len`: `36`</p><p>`string.pattern`: `^[a-z0-9](?:[-]?[a-z0-9]){2,}$|^$`</p> |
 | `target_gateway_server_address` | <p>`string.pattern`: `^(?:(?:[a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\-]*[a-zA-Z0-9])\.)*(?:[A-Za-z0-9]|[A-Za-z0-9][A-Za-z0-9\-]*[A-Za-z0-9])(?::[0-9]{1,5})?$|^$`</p> |
 | `target_frequency_plan_id` | <p>`string.max_len`: `64`</p> |
+| `target_frequency_plan_ids` | <p>`repeated.min_items`: `0`</p><p>`repeated.max_items`: `8`</p><p>`repeated.items.string.max_len`: `64`</p> |
 
 ### <a name="ttn.lorawan.v3.ClaimGatewayRequest.AuthenticatedIdentifiers">Message `ClaimGatewayRequest.AuthenticatedIdentifiers`</a>
 
@@ -3114,6 +3116,8 @@ in a future version of The Things Stack.
 
 ### <a name="ttn.lorawan.v3.EndDeviceBatchClaimingServer">Service `EndDeviceBatchClaimingServer`</a>
 
+The EndDeviceBatchClaimingServer service support claiming and managing batches of end devices on external Join Servers.
+
 | Method Name | Request Type | Response Type | Description |
 | ----------- | ------------ | ------------- | ------------|
 | `Unclaim` | [`BatchUnclaimEndDevicesRequest`](#ttn.lorawan.v3.BatchUnclaimEndDevicesRequest) | [`BatchUnclaimEndDevicesResponse`](#ttn.lorawan.v3.BatchUnclaimEndDevicesResponse) | Unclaims multiple end devices on an external Join Server. All devices must have the same application ID. Check the response for devices that could not be unclaimed. |
@@ -3128,8 +3132,7 @@ in a future version of The Things Stack.
 
 ### <a name="ttn.lorawan.v3.EndDeviceClaimingServer">Service `EndDeviceClaimingServer`</a>
 
-The EndDeviceClaimingServer service configures authorization to claim end devices registered in an application,
-and allows clients to claim end devices.
+The EndDeviceClaimingServer service support claiming and managing end devices on external Join Servers.
 
 | Method Name | Request Type | Response Type | Description |
 | ----------- | ------------ | ------------- | ------------|
@@ -3153,6 +3156,8 @@ and allows clients to claim end devices.
 
 ### <a name="ttn.lorawan.v3.GatewayClaimingServer">Service `GatewayClaimingServer`</a>
 
+The GatewayClaimingServer service support claiming and managing gateway claims.
+
 | Method Name | Request Type | Response Type | Description |
 | ----------- | ------------ | ------------- | ------------|
 | `Claim` | [`ClaimGatewayRequest`](#ttn.lorawan.v3.ClaimGatewayRequest) | [`GatewayIdentifiers`](#ttn.lorawan.v3.GatewayIdentifiers) | Claims a gateway by claim authentication code or QR code and transfers the gateway to the target user. |
@@ -3679,7 +3684,7 @@ CreateClientEmailMessage is used as a wrapper for handling the email regarding t
 
 ### <a name="ttn.lorawan.v3.EmailValidationRegistry">Service `EmailValidationRegistry`</a>
 
-The EmailValidationRegistry service, exposed by the Identity Server, is used for validating an user's primary email.
+The EmailValidationRegistry service, exposed by the Identity Server, is used for validating a user's primary email.
 
 | Method Name | Request Type | Response Type | Description |
 | ----------- | ------------ | ------------- | ------------|
@@ -4735,6 +4740,8 @@ NsEndDeviceRegistry, the AsEndDeviceRegistry and the JsEndDeviceRegistry.
 
 ### <a name="ttn.lorawan.v3.EndDeviceTemplateConverter">Service `EndDeviceTemplateConverter`</a>
 
+The EndDeviceTemplateRegistry service is used to convert end devices information between formats.
+
 | Method Name | Request Type | Response Type | Description |
 | ----------- | ------------ | ------------- | ------------|
 | `ListFormats` | [`.google.protobuf.Empty`](#google.protobuf.Empty) | [`EndDeviceTemplateFormats`](#ttn.lorawan.v3.EndDeviceTemplateFormats) | Returns the configured formats to convert from. |
@@ -5395,6 +5402,8 @@ Identifies an end device model with version information.
 
 ### <a name="ttn.lorawan.v3.GatewayConfigurationService">Service `GatewayConfigurationService`</a>
 
+The GatewayConfigurationService fetches the configuration of a gateway.
+
 | Method Name | Request Type | Response Type | Description |
 | ----------- | ------------ | ------------- | ------------|
 | `GetGatewayConfiguration` | [`GetGatewayConfigurationRequest`](#ttn.lorawan.v3.GetGatewayConfigurationRequest) | [`GetGatewayConfigurationResponse`](#ttn.lorawan.v3.GetGatewayConfigurationResponse) |  |
@@ -5512,6 +5521,8 @@ gateway registrations in batches.
 
 ### <a name="ttn.lorawan.v3.GatewayConfigurator">Service `GatewayConfigurator`</a>
 
+The GatewayConfigurator service is used to fetch gateway configuration.
+
 | Method Name | Request Type | Response Type | Description |
 | ----------- | ------------ | ------------- | ------------|
 | `PullConfiguration` | [`PullGatewayConfigurationRequest`](#ttn.lorawan.v3.PullGatewayConfigurationRequest) | [`Gateway`](#ttn.lorawan.v3.Gateway) _stream_ |  |
@@ -5616,6 +5627,8 @@ GatewayUp may contain zero or more uplink messages and/or a status message for t
 
 ### <a name="ttn.lorawan.v3.Gs">Service `Gs`</a>
 
+The Gs service returns information about the Gateway Server and gateways connected to it.
+
 | Method Name | Request Type | Response Type | Description |
 | ----------- | ------------ | ------------- | ------------|
 | `GetGatewayConnectionStats` | [`GatewayIdentifiers`](#ttn.lorawan.v3.GatewayIdentifiers) | [`GatewayConnectionStats`](#ttn.lorawan.v3.GatewayConnectionStats) | Get statistics about the current gateway connection to the Gateway Server. This is not persisted between reconnects. |
@@ -5969,6 +5982,8 @@ OrganizationOrUserIdentifiers contains either organization or user identifiers.
 
 ### <a name="ttn.lorawan.v3.EntityAccess">Service `EntityAccess`</a>
 
+The EntityAccess service provides information about the access rights of an entity.
+
 | Method Name | Request Type | Response Type | Description |
 | ----------- | ------------ | ------------- | ------------|
 | `AuthInfo` | [`.google.protobuf.Empty`](#google.protobuf.Empty) | [`AuthInfoResponse`](#ttn.lorawan.v3.AuthInfoResponse) | AuthInfo returns information about the authentication that is used on the request. |
@@ -5981,6 +5996,8 @@ OrganizationOrUserIdentifiers contains either organization or user identifiers.
 
 ### <a name="ttn.lorawan.v3.Is">Service `Is`</a>
 
+The Is service provides information about the Identity Server.
+
 | Method Name | Request Type | Response Type | Description |
 | ----------- | ------------ | ------------- | ------------|
 | `GetConfiguration` | [`GetIsConfigurationRequest`](#ttn.lorawan.v3.GetIsConfigurationRequest) | [`GetIsConfigurationResponse`](#ttn.lorawan.v3.GetIsConfigurationResponse) | Get the configuration of the Identity Server. The response is typically used to enable or disable features in a user interface. |
@@ -6351,6 +6368,8 @@ The AsJs service connects an Application Server to a Join Server.
 
 ### <a name="ttn.lorawan.v3.Js">Service `Js`</a>
 
+The Js service returns configuration for a Join Server.
+
 | Method Name | Request Type | Response Type | Description |
 | ----------- | ------------ | ------------- | ------------|
 | `GetJoinEUIPrefixes` | [`.google.protobuf.Empty`](#google.protobuf.Empty) | [`JoinEUIPrefixes`](#ttn.lorawan.v3.JoinEUIPrefixes) | Request the JoinEUI prefixes that are configured for this Join Server. |
@@ -9000,6 +9019,8 @@ The NsRelayConfigurationService provides configuration management capabilities f
 
 ### <a name="ttn.lorawan.v3.NotificationService">Service `NotificationService`</a>
 
+The NotificationService is used to send notifications.
+
 | Method Name | Request Type | Response Type | Description |
 | ----------- | ------------ | ------------- | ------------|
 | `Create` | [`CreateNotificationRequest`](#ttn.lorawan.v3.CreateNotificationRequest) | [`CreateNotificationResponse`](#ttn.lorawan.v3.CreateNotificationResponse) | Create a new notification. Can only be called by internal services using cluster auth. |
@@ -9999,6 +10020,8 @@ The Pba service allows clients to manage peering through Packet Broker.
 
 ### <a name="ttn.lorawan.v3.EndDeviceQRCodeGenerator">Service `EndDeviceQRCodeGenerator`</a>
 
+The EndDeviceQRCodeGenerator service provides functionality to generate and parse QR codes.
+
 | Method Name | Request Type | Response Type | Description |
 | ----------- | ------------ | ------------- | ------------|
 | `GetFormat` | [`GetQRCodeFormatRequest`](#ttn.lorawan.v3.GetQRCodeFormatRequest) | [`QRCodeFormat`](#ttn.lorawan.v3.QRCodeFormat) | Return the QR code format. |
@@ -10806,6 +10829,7 @@ Secret contains a secret value. It also contains the ID of the Encryption key us
 | `page` | [`uint32`](#uint32) |  | Page number for pagination. 0 is interpreted as 1. |
 | `order` | [`string`](#string) |  | Order the results by this field path. Default ordering is by ID. Prepend with a minus (-) to reverse the order. |
 | `deleted` | [`bool`](#bool) |  | Only return recently deleted bookmarks. |
+| `entity_types` | [`string`](#string) | repeated | Specifies that only bookmarks that reference the given entity type should be returned. If empty the filter is not applied. |
 
 #### Field Rules
 
@@ -10814,6 +10838,7 @@ Secret contains a secret value. It also contains the ID of the Encryption key us
 | `user_ids` | <p>`message.required`: `true`</p> |
 | `limit` | <p>`uint32.lte`: `1000`</p> |
 | `order` | <p>`string.in`: `[ user_id -user_id entity_type -entity_type entity_id -entity_id created_at -created_at]`</p> |
+| `entity_types` | <p>`repeated.unique`: `true`</p><p>`repeated.items.string.in`: `[application client end device gateway organization user]`</p> |
 
 ### <a name="ttn.lorawan.v3.ListUserSessionsRequest">Message `ListUserSessionsRequest`</a>
 
@@ -11111,7 +11136,7 @@ DashboardLayout is a set of possible layout values to be used in the Console.
 
 ### <a name="ttn.lorawan.v3.UserAccess">Service `UserAccess`</a>
 
-The UserAcces service, exposed by the Identity Server, is used to manage
+The UserAccess service, exposed by the Identity Server, is used to manage
 API keys of users.
 
 | Method Name | Request Type | Response Type | Description |
@@ -11161,6 +11186,8 @@ API keys of users.
 
 ### <a name="ttn.lorawan.v3.UserInvitationRegistry">Service `UserInvitationRegistry`</a>
 
+The UserOrganizationRegistry service, exposed by the Identity Server, is used to manage user invitations.
+
 | Method Name | Request Type | Response Type | Description |
 | ----------- | ------------ | ------------- | ------------|
 | `Send` | [`SendInvitationRequest`](#ttn.lorawan.v3.SendInvitationRequest) | [`Invitation`](#ttn.lorawan.v3.Invitation) | Invite a user to join the network. |