New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[WIP] AI video prototype #2959
base: master
Are you sure you want to change the base?
[WIP] AI video prototype #2959
Conversation
feb3f3e
to
563d199
Compare
d32b1b5 temporarily disables Linux arm64 builds because they fail due to an error related to not being able to find zlib during ffmpeg compilation. This error doesn't occur for amd64 builds so I suspect the issue is related to the amd64 -> arm64 cross-compilation process. I noticed that we compile an arm64 specific version of x264 before compiling ffmpeg - perhaps we have to do something similar with zlib? zlib is currently required as a dependency as of 133050d#diff-4ae778054809274731b9da0c6a5a869c0bd214e92f954a5c9c39181748c2f175 which enabled the png decoder and image2 muxer which are used to demux + decode a sequence of PNG files so they can be encoded into an mp4 file. Ideally, we would replace the PNG demux/decode component by passing tensors (that represent frames) outputted by a model directly from GPU memory to NVENC using torchaudio.StreamWriter, but torchaudio.StreamWriter doesn't support RGB -> YUV conversion on the GPU yet - it can still encode a larger, less-streaming friendly (my understanding is yuv420p is preferred for streaming) RGB output, but I didn't jump to implement this yet due to current limitations. Until this replacement happens, zlib would be a required dependency to support the temporary PNG demux + decode component. |
@yondonfu Weird - on release go-livepeer right now zlib is dynamically linked. I'll have a look. |
* feat(ai): add pipelines optimization flags This commit adds a new `OptimizationFlags` field to the `aiModels` config so that users can forward optimization environment variables to the [ai-worker](git@github.com:livepeer/ai-worker.git) for more information see livepeer/ai-worker#61. * chore: update ai-worker to latest commit This commit ensures that the https://github.com/livepeer/ai-worker dependency is on the latest commit that includes the new optimization flags feature (see livepeer/ai-worker#61). * refactor: improve OptFlags logging This commit ensures that the `optimzation flag not supported` warning is shown for each model that is not loaded warm.
This commit adds a new section explaining the new `optimization_flags` that were enabled #3013.
This commit temporary changes the push branch of the `build.yml` to the `ai-video` branch since the `ai-video` branch has conflicts with the `master` branch preventing the containers to be build.
This commit temporary changes the push branch of the `docker.yml` to the `ai-video` branch since the `ai-video` branch has conflicts with the `master` branch preventing the containers to be build.
This commit gets rid of the Pull request labeler configuration file warning.
This commit flushes the data in the image writer to ensure that all data gets written to the PNG.
This commit ensures that the labeler action also runs on a 'pull_request_target' to ensure pull requests from forks are correctly labeled.
This commit ensures that all AI related issues and feature requests are assigned to the AI team.
* feat(ai): enable AI orchestrator discovery This commit incorporates the AIServiceRegistry contract address, superseding the conventional ServiceRegistry contract address. This strategic alteration streamlines the discovery process of AI Orchestrators within the AI Subnet, thereby bolstering network accessibility and interaction. While this approach serves as a swift workaround to enable the feature without extensive code modification, it's important to note that it may disrupt the existing transcoding discovery mechanism. We have to fix this if we want to merge the two networks in the future. * docs(ai): improve discovery documentation This commit ensures that people are aware that they have to interact with the `AIServiceRegistry` using their main Orch wallet. * fix: fix 'AIServiceRegistry' devnet and testnet issue This commit ensure that the hardcoded `AIServiceRegistry` contract doesn't break the go-livepeer binary on local devnets or testnets.
This commit adds extra devtool input arguments allowing developers to spin up multiple Os on the ETH devnet.
This commit improves the devtool documentation and adds a helpful script if developers want to create multiple Os at the same time.
This commit logs the advertised capabilities and price on startup if users have their logging verbosity level set to 6 or higher.
This commit ensures that an error is thrown when users don't specify the 'aiModels' flag but have the 'aiWorker' flag set.
* fix(ai): improve selection algorithm This commit modifies the selection algorithm to continue retrying for a duration of one second instead of stopping after four attempts. This change addresses issues encountered with the current algorithm's performance in environments with 15 nodes on the network, ensuring more robust and reliable operation until further optimizations can be implemented. * refactor(ai): enhance selection algorithm retry logic This commit replaces the time-based for-loop in the selection algorithm's retry logic with a more context-aware approach.
This commit refines context handling in the orchestrator selection loop for idiomatic Go and enhanced propagation of parent cancellations.
This commit improves the orchestrator selection retry ctx timeout msg.
* fix(ai): handle insufficient capacity payments This commit enhances the Orchestrator's capacity handling by returning an error prior to processing payments when capacity is insufficient. This prevents that the Gateway overpays for requests. * chore(ai): update ai-worker dependency This commit updates the ai-worker dependency to the latest version.
This commit ensures that the `upload_build.sh` script uploads the latest binary that is deployed to the `ai-video` branch under one url. This is done to simplify binary installation.
This commit removes the tempoary AI subnet docs now that the final docs have been deployed on https://docs.livepeer.ai/ai/introduction.
This commit prevents the orchestrator selection go routine from staying in a infinite loop when no Orchestrators can be found.
) This commit introduces latency consideration into the orchestrator selection process, addressing two key issues. Firstly, it resolves a minor bug where the algorithm consistently selected known orchestrators due to a condition that never evaluated to true (see [this condition](https://github.com/livepeer/go-livepeer/blob/1239b4e56133003fe6a98a863cce6bdd6b5f2532/server/selection.go#L110)). Secondly, this change ensures that, once all orchestrators have been evaluated, the one with the fastest response time for a specific job is chosen. While the current method for calculating latency is somewhat basic, it sets the foundation for more sophisticated enhancements in the future. Co-authored-by: Brad P <0xb79orch@gmail.com>
This commit updates the 'ai-worker' dependency to the latest commit.
This commit adds the `gateway` flag and deprecates the `broadcaster` flag per core team decision (details: https://discord.com/channels/423160867534929930/1051963444598943784/1210356864643109004).
* Remove -pricePerUnit requirement for orchestrator with -AIWorker flag * refactor: add PricePerUnit comment This commit reintroduces the previously omitted comment for the PricePerUnit variable, improving code readability and maintainability. * refactor: simplify PricePerUnit flag check condition This commit simplifies the conditional check used to check if the `PricePerUnit` flag is needed. --------- Co-authored-by: Rick Staa <rick.staa@outlook.com>
This commit updates the https://github.com/livepeer/ai-worker to the latest version so that Orchestrators can enable the [DeepCache](https://github.com/horseee/DeepCache) optimization. This optimization will provide a 50% speedup for multi-step inference requests.
Opening this draft PR to kick off CI build process for the WIP
ai-video
branch. This PR should not be merged as-is and the code on this branch will likely be refactored + cleaned up separately later on.