New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Jobs Refactor #391
Jobs Refactor #391
Changes from 11 commits
da147fd
63c24d9
16a77c2
d1600a9
6a33bd4
da0f9e6
284223d
3afb86b
8a898a6
5026e63
217ab3d
f3eff76
f130b57
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,30 @@ | ||
//===----------------------------------------------------------------------===// | ||
// | ||
// This source file is part of the Hummingbird server framework project | ||
// | ||
// Copyright (c) 2024 the Hummingbird authors | ||
// Licensed under Apache License v2.0 | ||
// | ||
// See LICENSE.txt for license information | ||
// See hummingbird/CONTRIBUTORS.txt for the list of Hummingbird authors | ||
// | ||
// SPDX-License-Identifier: Apache-2.0 | ||
// | ||
//===----------------------------------------------------------------------===// | ||
|
||
/// Job definition type | ||
public struct HBJobDefinition<Parameters: Codable & Sendable>: Sendable { | ||
public let id: HBJobIdentifier<Parameters> | ||
let maxRetryCount: Int | ||
let _execute: @Sendable (Parameters, HBJobContext) async throws -> Void | ||
|
||
public init(id: HBJobIdentifier<Parameters>, maxRetryCount: Int = 0, execute: @escaping @Sendable (Parameters, HBJobContext) async throws -> Void) { | ||
self.id = id | ||
self.maxRetryCount = maxRetryCount | ||
self._execute = execute | ||
} | ||
|
||
func execute(_ parameters: Parameters, context: HBJobContext) async throws { | ||
try await self._execute(parameters, context) | ||
} | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,42 @@ | ||
//===----------------------------------------------------------------------===// | ||
// | ||
// This source file is part of the Hummingbird server framework project | ||
// | ||
// Copyright (c) 2024 the Hummingbird authors | ||
// Licensed under Apache License v2.0 | ||
// | ||
// See LICENSE.txt for license information | ||
// See hummingbird/CONTRIBUTORS.txt for the list of Hummingbird authors | ||
// | ||
// SPDX-License-Identifier: Apache-2.0 | ||
// | ||
//===----------------------------------------------------------------------===// | ||
|
||
/// Identifier for a Job type | ||
/// | ||
/// The identifier includes the type of the parameters required by the job to ensure | ||
/// the wrong parameters are not passed to this job | ||
/// | ||
/// Extend this type to include your own job identifiers | ||
/// ``` | ||
/// extension HBJobIdentifier<String> { | ||
/// static var myJob: Self { .init("my-job") } | ||
/// } | ||
/// ``` | ||
public struct HBJobIdentifier<Parameters>: Sendable, Hashable, ExpressibleByStringLiteral { | ||
let name: String | ||
/// Initialize a HBJobIdentifier | ||
/// | ||
/// - Parameters: | ||
/// - name: Unique name for identifier | ||
/// - parameters: Parameter type associated with Job | ||
public init(_ name: String, parameters: Parameters.Type = Parameters.self) { self.name = name } | ||
|
||
/// Initialize a HBJobIdentifier from a string literal | ||
/// | ||
/// This can only be used in a situation where the Parameter type is defined elsewhere | ||
/// - Parameter string: | ||
public init(stringLiteral string: String) { | ||
self.name = string | ||
} | ||
} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,42 @@ | ||
//===----------------------------------------------------------------------===// | ||
// | ||
// This source file is part of the Hummingbird server framework project | ||
// | ||
// Copyright (c) 2021-2024 the Hummingbird authors | ||
// Licensed under Apache License v2.0 | ||
// | ||
// See LICENSE.txt for license information | ||
// See hummingbird/CONTRIBUTORS.txt for the list of Hummingbird authors | ||
// | ||
// SPDX-License-Identifier: Apache-2.0 | ||
// | ||
//===----------------------------------------------------------------------===// | ||
|
||
import Foundation | ||
import Logging | ||
|
||
/// Job queue protocol. | ||
/// | ||
/// Defines how to push and pop job data off a queue | ||
public protocol HBJobQueueDriver: AsyncSequence, Sendable where Element == HBQueuedJob<JobID> { | ||
associatedtype JobID: CustomStringConvertible & Sendable | ||
|
||
/// Called when JobQueueHandler is initialised with this queue | ||
func onInit() async throws | ||
/// Push Job onto queue | ||
/// - Returns: Identifier of queued job | ||
func push(data: Data) async throws -> JobID | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is there a reason we're using There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Just the fact we are moving back and forward between the two types. Should probably change to ByteBuffer. |
||
/// This is called to say job has finished processing and it can be deleted | ||
func finished(jobId: JobID) async throws | ||
/// This is called to say job has failed to run and should be put aside | ||
func failed(jobId: JobID, error: any Error) async throws | ||
/// stop serving jobs | ||
func stop() async | ||
/// shutdown queue | ||
func shutdownGracefully() async | ||
} | ||
|
||
extension HBJobQueueDriver { | ||
// default version of onInit doing nothing | ||
public func onInit() async throws {} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would it be possible to allow drivers to encode these parameters (or the request as a whole)? In MongoDB, it would be really unfortunate to encode this to JSON, then add the JSON into MongoDB. Since MongoDB already has (recursive) object support. So it would be able to store this as-is in BSON. Encoding to JSON would prevent indexes from being applies to
id
, and inspecting the data in MongoDB would require quite a lot of work from the developer.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The idea is that the driver only stores data blobs. Do you see an advantage to storing the full object structure? I guess debugging job issues might be easier.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Debugging mainly, yes
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I had moved away from the allowing drivers to do the encoding because I was repeating the same code across all of the drivers. It also meant the driver needed to know about the job registry so they could be sure to decode the correct job type. Data blobs meant the driver had a clean split from the job decoding. I'll have another look though.