New Release? #31585
Comments
|
A new release is overdue but depends on solving certain cross-platform issues in the CI build process. Meanwhile for the main platforms, see #31530 for update hints. |
|
What are those issues that need sorting out? Is there an issue/PR tracking this? Creating a PR could bring in people that are willing to help with the process (like myself) |
|
#30644, but it should probably be refreshed now. This is very far down my stack today, but as I recall there were various possible blockers:
Obviously I need to dive back into this especially in view of yt-dlp/yt-dlp#6220, which I would also like to acquire, if only for more regular versioned releases. |
|
But isn't the auto-update installer already there, depending on the how the yt-dl instance was installed? Once a build has been deployed to the release page and to PyPi, it can be pulled by almost all installation types. The problem being addressed is really to get the small number of builds in the release tested on all the different platforms. |
This comment was marked as off-topic.
This comment was marked as off-topic.
|
Will #30644 keep tracking this and should I post my remarks on there? I think maybe instead of perfecting the release process it might make sense to release something so as to get updates to the end user. If releases were not automated, whats problematic about building like that, then passing those executables to the user to be tried on various platforms? The related release can be marked as I would also advise to look into the These are the repository configuration options of the release and nighly action: vars.PUSH_VERSION_COMMIT: Update master branch with a push
vars.BUILD_NIGHTLY: Build a nightly release on every commit
# Both need to be set to push build to archive repo as release
vars.ARCHIVE_REPO: repo like `yt-dlp/yt-dlp-builds`
secrets.ARCHIVE_REPO_TOKEN: token with contents:write
# Push to PyPI and brew if each is set
secrets.PYPI_TOKEN: Token for PyPI
secrets.BREW_TOKEN: Token for brew
Are the test adjustments needed for the release for now? Is that preparation to be testing if the builds would work as expected? If so, I'd think that can be moved back for now?
Do we need to supply
This should be handled by 6220. I can help with the remaining blockers there.
Yes, understandable. If we encounter anything in particular we might need to push that back until somebody with experience on that system or with those dependencies can chime in. I think those issues will become apparent once we create/try creating the actual The steps that I would propose for now are:
Feel free to contact me via E-Mail if you would like to move to a different communication channel for this. Alternatively let me know of anything else I can do. |
This comment was marked as off-topic.
This comment was marked as off-topic.
|
Just to show that a release might appear quite soon, and not in the far future. From #31530 (comment)
No, although it's true that the GH workflow ecosystem forces you into an upgrade race as various dependencies are moved to new versions, much as in the rest of the s/w world. Actually, for yt-dl itself, Py2.7 is fine and dandy for the most part: the #31530 fix did reveal some 2.6 hold-outs but also surprisingly many verbose logs showing 2.7 still in use. The previous build and release system, not well documented, depended on some local setup that had been passed down through the former maintainers. Lured by the gleaming promise of online builds, I completely failed to reproduce the necessary local setup. That brings us to last month. Now, I find that GH is about to junk ("deprecated") versions of Linux that support the Pythons needed for yt-dl's CI testing. So we'll have to rework that completely using our own runners, etc. This is too hard for now and the existing tests will have to do: boo to Python >= 3.10 (though I don't expect significant issues there). Therefore the build and release workflow doesn't depend on the test workflow. I took the yt-dlp scheme as a pattern, as suggested, rather than building directly on the excellent work in #30644. Thus Users of new macOSes that have no Python will still have to install a Python and arrange for it to be found by the shebang of the POSIX build. Currently a tricky part is print((Changelog if args.markdown else PlainChangelog)(commits.groups(), args.repo))Less simple, the 200 commits since the last release aren't as disciplined as the yt-dlp script expected. I think the only practical design for grouping these commits automatically is to use git commands to identify what each commit changed and then map back from there. For instance:
Given this, the solution is to take some output of an as-good-as-can-be-expected hack on the existing script and modify the CI process to accept some manually tweaked changelog delta that will be prepended with its version ID to the Obviously any other suggestions would be happily considered. |
|
Good to hear there is a path forward!
Yes, I believe the 18.04 runner is being removed in April if I remember correctly. I assume you're aware of this action to setup Python? https://github.com/actions/setup-python ... Example: https://github.com/LizardByte/Themerr-plex/blob/master/.github/workflows/CI.yml#L36-L49 Ubuntu 20.04 and 22.04 also still have python 2.7 that can be manually installed. Here's a build in an Ubuntu 22.04 dockerfile that uses Python 2.7 for reference: https://github.com/LizardByte/PlexyGlass/blob/nightly/Dockerfile#L4-L24 ... Debian/Ubuntu has too many packages that still rely on Python 2.x to completely remove it. I didn't see a mention of publishing to PyPi in your comment. Are you still going to publish there? |
|
Is there a new PR, some branch or similar that I can take a look at and implement missing factors to help out?
This is why I suggested shimming it for now, as manual mapping will be required anyways. Regarding future commits, it might make sense to adjust the commit message to be closer to dlp format OR (I thought about it before but scrapped it since its easier to just rely on message only) search for the files using file or text search if the format should be kept the same (
Assuming that this is building on the release workflow of dlp the PyPI publish is integrated in those steps It might make sense to diff with dlp build workflow as some new changes were coming. A relevant branch can be found here (though many changes were directly related to nightly and |
|
Homebrew project leader here 👋🏻. I think even getting a tagged prerelease here with failing CI will allow Homebrew to update to a version that will work for downloading from YouTube and stop you getting so many issues filed by our users. |
|
Thanks for your interest! Almost any commit to master is meant to be reliable and its CI test will have succeeded, having been exercised on another branch first. If you would like a special tag applied to some commit, that's easy. If you're able to take an unofficial nightly build (from a downstream), which has the advantage of reporting a different version and showing a disclaimer, that's even easier. |
Upcoming.
I suppose that a manually adjusted changelog delta could be offered to the workflow as a var, even if it's ~15kB?
For now I commented out the package management interfaces. The existing PyPi update process would have been manual AFAIK with any account details held by former maintainers; presumably we can regenerate them if they can't be recalled.
The workflow files all started out from the yt-dlp equivalents in a meld session. So yes. |
Thanks for your great software that I personally use!
Yes, our hope would be any sort of tag applied to a release that would show up in https://github.com/ytdl-org/youtube-dl/releases that we could update to 🙇🏻 |
|
Which release asset is used to create the Homebrew package? |
|
What's the issue with opening the mirror on any non-eu hosing? see no
problem
ср, 2 авг. 2023 г. в 18:32, Vangelis66 ***@***.***>:
… Since when is yt-dl.org blocked?
... I can only assume, going by the "update check" breakage I reported 😉
, that the court order was *finally enforced* (to the *German hoster*
hosting yt-dl.org) on *Aug 1st, 2023* 😭 ...
FWIW, this doesn't seem to have arrived "out of the blue" 😞 :
http://web.archive.org/web/20230727220633/https://yt-dl.org/
*We would also like to heartily thank* our main website
<http://web.archive.org/web/20230727220633/https://yt-dl.org/> *hoster*
Uberspace <https://uberspace.de/en/> *who is currently being sued in
Germany for hosting our essentially business card website and who have
already spent thousands of Euros in their legal defense.*
I don't want to be pessimistic, but, since Germany is a member of the *EU*,
what comes next for "us", *EU-based*, users of yt-dl?
—
Reply to this email directly, view it on GitHub
<#31585 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABY467NWLXHIKMVBMWJZIMLXTJXINANCNFSM6AAAAAAVARTTYU>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
IIANM, My initial report above served the purpose of making the "breakage" (more) widely known and that "breakage" has been acknowledged by the maintainer 😉 ; the exact way this new "issue" is going to be mitigated by dirkf is probably his own call 😄 ; despite my meanderings, this is still the "New Release?" thread and the internal update feature of |
Indeed, prior to 20230801, below redirection was taking place when an "update check" was invoked:
(that screenshot was taken by me on 20230719, but I had forgotten saving it - early signs of dementia, probably? ...) So, I suppose, youtube-dl/youtube_dl/update.py Line 37 in 2efc8de should now become: UPDATE_URL = 'https://ytdl-org.github.io/youtube-dl/update/'Then, indeed: youtube-dl -vU
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-vU']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.08.01
[debug] Lazy loading extractors enabled
[debug] Single file build
[debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015
[debug] exe versions: none
[debug] Proxy map: {}
youtube-dl is up to date (2023.08.01) |
Sorry I am not up to speed on what is blocking the release, but if it is the update feature it may be worth to release without it, and until the update backend is fixed it could return something like "Not available yet, please update manually from: (link...)". Personally I never used the auto update and updated manually, while having a signed release is much better than not having it. |
|
I would also like to see the library updated in PyPi so @dependabot could keep it updated, which does not work when specifying a git tag/commit for pip requirements.txt. |
|
With the somewhat recently implemented support for the For Brotli-1.0.9-cp34-cp34m-win32.zip Earlier this month, Google ( 😡 ) released In that tracker, I was given some hints by a kind person as to how to modify the https://raw.githubusercontent.com/google/brotli/v1.1/setup.py ? I intend to, at least, build a py3.4 wheel of 1.1.0 with your help, someone else would have to build (if there's any need for it 😜 ) the py2.7-based wheel of it... Thanks in advance 😄 ... |
This `setup.py` doesn't use `f''` and also supplies an explicit exception class at l.14 which the _flake8_ checker likes to see.# Copyright 2015 The Brotli Authors. All rights reserved.
#
# Distributed under MIT license.
# See file LICENSE for detail or copy at https://opensource.org/licenses/MIT
import os
import platform
import re
import unittest
try:
from setuptools import Extension
from setuptools import setup
except ImportError:
from distutils.core import Extension
from distutils.core import setup
from distutils.command.build_ext import build_ext
from distutils import errors
from distutils import dep_util
from distutils import log
CURR_DIR = os.path.abspath(os.path.dirname(os.path.realpath(__file__)))
def read_define(path, macro):
""" Return macro value from the given file. """
with open(path, 'r') as f:
for line in f:
m = re.match(r'#define\s{0}\s+(.+)'.format(macro), line)
if m:
return m.group(1)
return ''
def get_version():
""" Return library version string from 'common/version.h' file. """
version_file_path = os.path.join(CURR_DIR, 'c', 'common', 'version.h')
major = read_define(version_file_path, 'BROTLI_VERSION_MAJOR')
minor = read_define(version_file_path, 'BROTLI_VERSION_MINOR')
patch = read_define(version_file_path, 'BROTLI_VERSION_PATCH')
if not major or not minor or not patch:
return ''
return '{0}.{1}.{2}'.format(major, minor, patch)
def get_test_suite():
test_loader = unittest.TestLoader()
test_suite = test_loader.discover('python', pattern='*_test.py')
return test_suite
class BuildExt(build_ext):
def get_source_files(self):
filenames = build_ext.get_source_files(self)
for ext in self.extensions:
filenames.extend(ext.depends)
return filenames
def build_extension(self, ext):
if ext.sources is None or not isinstance(ext.sources, (list, tuple)):
raise errors.DistutilsSetupError(
"in 'ext_modules' option (extension '%s'), "
"'sources' must be present and must be "
"a list of source filenames" % ext.name)
ext_path = self.get_ext_fullpath(ext.name)
depends = ext.sources + ext.depends
if not (self.force or dep_util.newer_group(depends, ext_path, 'newer')):
log.debug("skipping '%s' extension (up-to-date)", ext.name)
return
else:
log.info("building '%s' extension", ext.name)
c_sources = []
for source in ext.sources:
if source.endswith('.c'):
c_sources.append(source)
extra_args = ext.extra_compile_args or []
objects = []
macros = ext.define_macros[:]
if platform.system() == 'Darwin':
macros.append(('OS_MACOSX', '1'))
elif self.compiler.compiler_type == 'mingw32':
# On Windows Python 2.7, pyconfig.h defines "hypot" as "_hypot",
# This clashes with GCC's cmath, and causes compilation errors when
# building under MinGW: http://bugs.python.org/issue11566
macros.append(('_hypot', 'hypot'))
for undef in ext.undef_macros:
macros.append((undef,))
objs = self.compiler.compile(
c_sources,
output_dir=self.build_temp,
macros=macros,
include_dirs=ext.include_dirs,
debug=self.debug,
extra_postargs=extra_args,
depends=ext.depends)
objects.extend(objs)
self._built_objects = objects[:]
if ext.extra_objects:
objects.extend(ext.extra_objects)
extra_args = ext.extra_link_args or []
# when using GCC on Windows, we statically link libgcc and libstdc++,
# so that we don't need to package extra DLLs
if self.compiler.compiler_type == 'mingw32':
extra_args.extend(['-static-libgcc', '-static-libstdc++'])
ext_path = self.get_ext_fullpath(ext.name)
# Detect target language, if not provided
language = ext.language or self.compiler.detect_language(c_sources)
self.compiler.link_shared_object(
objects,
ext_path,
libraries=self.get_libraries(ext),
library_dirs=ext.library_dirs,
runtime_library_dirs=ext.runtime_library_dirs,
extra_postargs=extra_args,
export_symbols=self.get_export_symbols(ext),
debug=self.debug,
build_temp=self.build_temp,
target_lang=language)
NAME = 'Brotli'
VERSION = get_version()
URL = 'https://github.com/google/brotli'
DESCRIPTION = 'Python bindings for the Brotli compression library'
AUTHOR = 'The Brotli Authors'
LICENSE = 'MIT'
PLATFORMS = ['Posix', 'MacOS X', 'Windows']
CLASSIFIERS = [
'Development Status :: 4 - Beta',
'Environment :: Console',
'Intended Audience :: Developers',
'License :: OSI Approved :: MIT License',
'Operating System :: MacOS :: MacOS X',
'Operating System :: Microsoft :: Windows',
'Operating System :: POSIX :: Linux',
'Programming Language :: C',
'Programming Language :: C++',
'Programming Language :: Python',
'Programming Language :: Python :: 2',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.3',
'Programming Language :: Python :: 3.4',
'Programming Language :: Python :: 3.5',
'Programming Language :: Unix Shell',
'Topic :: Software Development :: Libraries',
'Topic :: Software Development :: Libraries :: Python Modules',
'Topic :: System :: Archiving',
'Topic :: System :: Archiving :: Compression',
'Topic :: Text Processing :: Fonts',
'Topic :: Utilities',
]
PACKAGE_DIR = {'': 'python'}
PY_MODULES = ['brotli']
EXT_MODULES = [
Extension(
'_brotli',
sources=[
'python/_brotli.c',
'c/common/constants.c',
'c/common/context.c',
'c/common/dictionary.c',
'c/common/platform.c',
'c/common/shared_dictionary.c',
'c/common/transform.c',
'c/dec/bit_reader.c',
'c/dec/decode.c',
'c/dec/huffman.c',
'c/dec/state.c',
'c/enc/backward_references.c',
'c/enc/backward_references_hq.c',
'c/enc/bit_cost.c',
'c/enc/block_splitter.c',
'c/enc/brotli_bit_stream.c',
'c/enc/cluster.c',
'c/enc/command.c',
'c/enc/compound_dictionary.c',
'c/enc/compress_fragment.c',
'c/enc/compress_fragment_two_pass.c',
'c/enc/dictionary_hash.c',
'c/enc/encode.c',
'c/enc/encoder_dict.c',
'c/enc/entropy_encode.c',
'c/enc/fast_log.c',
'c/enc/histogram.c',
'c/enc/literal_cost.c',
'c/enc/memory.c',
'c/enc/metablock.c',
'c/enc/static_dict.c',
'c/enc/utf8_util.c',
],
depends=[
'c/common/constants.h',
'c/common/context.h',
'c/common/dictionary.h',
'c/common/platform.h',
'c/common/shared_dictionary_internal.h',
'c/common/transform.h',
'c/common/version.h',
'c/dec/bit_reader.h',
'c/dec/huffman.h',
'c/dec/prefix.h',
'c/dec/state.h',
'c/enc/backward_references.h',
'c/enc/backward_references_hq.h',
'c/enc/backward_references_inc.h',
'c/enc/bit_cost.h',
'c/enc/bit_cost_inc.h',
'c/enc/block_encoder_inc.h',
'c/enc/block_splitter.h',
'c/enc/block_splitter_inc.h',
'c/enc/brotli_bit_stream.h',
'c/enc/cluster.h',
'c/enc/cluster_inc.h',
'c/enc/command.h',
'c/enc/compound_dictionary.h',
'c/enc/compress_fragment.h',
'c/enc/compress_fragment_two_pass.h',
'c/enc/dictionary_hash.h',
'c/enc/encoder_dict.h',
'c/enc/entropy_encode.h',
'c/enc/entropy_encode_static.h',
'c/enc/fast_log.h',
'c/enc/find_match_length.h',
'c/enc/hash.h',
'c/enc/hash_composite_inc.h',
'c/enc/hash_forgetful_chain_inc.h',
'c/enc/hash_longest_match64_inc.h',
'c/enc/hash_longest_match_inc.h',
'c/enc/hash_longest_match_quickly_inc.h',
'c/enc/hash_rolling_inc.h',
'c/enc/hash_to_binary_tree_inc.h',
'c/enc/histogram.h',
'c/enc/histogram_inc.h',
'c/enc/literal_cost.h',
'c/enc/memory.h',
'c/enc/metablock.h',
'c/enc/metablock_inc.h',
'c/enc/params.h',
'c/enc/prefix.h',
'c/enc/quality.h',
'c/enc/ringbuffer.h',
'c/enc/static_dict.h',
'c/enc/static_dict_lut.h',
'c/enc/utf8_util.h',
'c/enc/write_bits.h',
],
include_dirs=[
'c/include',
]),
]
TEST_SUITE = 'setup.get_test_suite'
CMD_CLASS = {
'build_ext': BuildExt,
}
setup(
name=NAME,
description=DESCRIPTION,
version=VERSION,
url=URL,
author=AUTHOR,
license=LICENSE,
platforms=PLATFORMS,
classifiers=CLASSIFIERS,
package_dir=PACKAGE_DIR,
py_modules=PY_MODULES,
ext_modules=EXT_MODULES,
test_suite=TEST_SUITE,
cmdclass=CMD_CLASS) |
|
Sorry, isn't the brotli update from 1.0.9 to 1.1.0 out of scope? Also py2.7 and py3.4 are severely out of date receiving no security updates or anything. If supporting decade-old Python versions is blocking this release, perhaps this is a good opportunity to reevaluate the support strategy for old Python versions? |
... It isn't!
How so?
... But there exists H/W that can't (for a variety of reasons) be updated to more "modern" versions of Python and it's those cases that the current development strives to also cater to - this has been discussed/explained/analysed to death elsewhere in this repo... Please, direct your current frustration elsewhere 😉 ; since, it seems, you won't feel comfortable (and more secure?) with nothing but the PSF-supported versions of Python, "downstream" (i.e. Kind regards. |
... It turns out I was too optimistic to begin with 😞 , basically fueled by this comment in the
@nonatomiclabs, are you on Windows and if yes, what version? I tried building Failed attempt at installing brotli-1.1.0(mod) under py3.4, on Win x86python -m pip install "Brotli-1.1.0-mod.tar.gz" =>
Processing c:\python3410-32\Brotli-1.1.0-mod.tar.gz
Installing build dependencies ... done
Getting requirements to build wheel ... done
Installing backend dependencies ... done
Preparing wheel metadata ... done
Building wheels for collected packages: Brotli
Building wheel for Brotli (PEP 517) ... error
ERROR: Complete output from command 'C:\Python3410-32\python.exe' 'C:\Python3410-32\lib\site-packages\pip\_vendor\pep517\_in_process.py' build_wheel 'C:\Users\<redacted>\AppData\Local\Temp\tmppbrr9dw6':
ERROR: running bdist_wheel
running build
running build_py
creating bin
creating bin\lib.win32-3.4
copying python\brotli.py -> bin\lib.win32-3.4
running build_ext
building '_brotli' extension
creating bin\temp.win32-3.4
creating bin\temp.win32-3.4\Release
creating bin\temp.win32-3.4\Release\python
creating bin\temp.win32-3.4\Release\c
creating bin\temp.win32-3.4\Release\c\common
creating bin\temp.win32-3.4\Release\c\dec
creating bin\temp.win32-3.4\Release\c\enc
C:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -Ic/include -IC:\Python3410-32\include -IC:\Python3410-32\include /Tcpython/_brotli.c /Fobin\temp.win32-3.4\Release\python/_brotli.obj
_brotli.c
python/_brotli.c(72) : error C2054: expected '(' to follow 'inline'
python/_brotli.c(74) : error C2085: 'BlocksOutputBuffer_InitAndGrow' : not in formal parameter list
python/_brotli.c(74) : error C2143: syntax error : missing ';' before '{'
python/_brotli.c(108) : error C2054: expected '(' to follow 'inline'
python/_brotli.c(110) : error C2085: 'BlocksOutputBuffer_Grow' : not in formal parameter list
python/_brotli.c(110) : error C2143: syntax error : missing ';' before '{'
python/_brotli.c(155) : error C2054: expected '(' to follow 'inline'
python/_brotli.c(157) : error C2085: 'BlocksOutputBuffer_Finish' : not in formal parameter list
python/_brotli.c(157) : error C2143: syntax error : missing ';' before '{'
python/_brotli.c(203) : error C2054: expected '(' to follow 'inline'
python/_brotli.c(204) : error C2085: 'BlocksOutputBuffer_OnError' : not in formal parameter list
python/_brotli.c(204) : error C2143: syntax error : missing ';' before '{'
python/_brotli.c(224) : error C2143: syntax error : missing ';' before 'type'
python/_brotli.c(225) : error C2065: 'mode_value' : undeclared identifier
python/_brotli.c(229) : error C2065: 'mode_value' : undeclared identifier
python/_brotli.c(291) : error C2059: syntax error : '.'
python/_brotli.c(294) : warning C4013: 'BlocksOutputBuffer_InitAndGrow' undefined; assuming extern returning int
python/_brotli.c(310) : warning C4013: 'BlocksOutputBuffer_Grow' undefined; assuming extern returning int
python/_brotli.c(320) : warning C4013: 'BlocksOutputBuffer_Finish' undefined; assuming extern returning int
python/_brotli.c(320) : warning C4047: '=' : 'PyObject *' differs in levels of indirection from 'int'
python/_brotli.c(326) : warning C4013: 'BlocksOutputBuffer_OnError' undefined; assuming extern returning int
python/_brotli.c(604) : error C2059: syntax error : '.'
python/_brotli.c(634) : warning C4047: '=' : 'PyObject *' differs in levels of indirection from 'int'
python/_brotli.c(856) : error C2059: syntax error : '.'
python/_brotli.c(906) : warning C4047: '=' : 'PyObject *' differs in levels of indirection from 'int'
python/_brotli.c(922) : error C2061: syntax error : identifier 'brotli_methods'
python/_brotli.c(922) : error C2059: syntax error : ';'
python/_brotli.c(922) : error C3409: empty attribute block is not allowed
python/_brotli.c(922) : error C2513: '/*global*/ ' : no variable declared before '='
python/_brotli.c(940) : error C2065: 'brotli_methods' : undeclared identifier
python/_brotli.c(940) : error C2099: initializer is not a constant
python/_brotli.c(978) : error C2143: syntax error : missing ';' before 'type'
python/_brotli.c(979) : error C2275: 'uint32_t' : illegal use of this type as an expression
C:\Program Files\Microsoft Visual Studio 10.0\VC\INCLUDE\stdint.h(23): see declaration of 'uint32_t'
python/_brotli.c(979) : error C2146: syntax error : missing ';' before identifier 'decoderVersion'
python/_brotli.c(979) : error C2065: 'decoderVersion' : undeclared identifier
python/_brotli.c(980) : error C2065: 'version' : undeclared identifier
python/_brotli.c(980) : warning C4047: 'function' : 'char *' differs in levels of indirection from 'int'
python/_brotli.c(980) : warning C4024: '_snprintf' : different types for formal and actual parameter 1
python/_brotli.c(980) : error C2065: 'version' : undeclared identifier
python/_brotli.c(981) : error C2065: 'decoderVersion' : undeclared identifier
python/_brotli.c(981) : error C2065: 'decoderVersion' : undeclared identifier
python/_brotli.c(981) : error C2065: 'decoderVersion' : undeclared identifier
python/_brotli.c(982) : error C2065: 'version' : undeclared identifier
python/_brotli.c(982) : warning C4047: 'function' : 'const char *' differs in levels of indirection from 'int'
python/_brotli.c(982) : warning C4024: 'PyModule_AddStringConstant' : different types for formal and actual parameter 3
error: command 'C:\\Program Files\\Microsoft Visual Studio 10.0\\VC\\BIN\\cl.exe' failed with exit status 2
----------------------------------------
ERROR: Failed building wheel for Brotli
Running setup.py clean for Brotli
Failed to build Brotli
ERROR: Could not build wheels for Brotli which use PEP 517 and cannot be installed directlyThus, I'm even more perplexed by the fact someone else did report installation success under py2.7 which, on Windows, requires the VS2008 (MS Visual C++ 9.0) based VCPython27 compiler (???) ... So, for me, when on py<3.6, |
|
Probably the way to make the C compile is to diff the latest Brotli against the one that works with VC2010 and reapply any substantive code changes, ignoring compiler directives, etc. Since it The |
|
... The following code was committed by Google on July 19th 2023 (a little more than two months ago): https://github.com/google/brotli/blob/acc265655d6dde4ff1bce65308e9e152fbf3381a/python/bro.py#L4-L6 How magnanimous of them, I say 👍 ; still, on Aug 31st, tag
Going back in time, I found that the last (py3.4.10 x86 building environment) python -m pip install "https://codeload.github.com/google/brotli/legacy.tar.gz/509d4419bd" =>
Collecting https://codeload.github.com/google/brotli/legacy.tar.gz/509d4419bd
Downloading https://codeload.github.com/google/brotli/legacy.tar.gz/509d4419bd
/ 1.1MB 939kB/s
Building wheels for collected packages: Brotli
Building wheel for Brotli (setup.py) ... done
Stored in directory: C:\Users\<redacted>\AppData\Local\Temp\pip-ephem-wheel-cache-b009j45f\wheels\bb\80\3b\d13a072b35f01f199e1e2e4aa6d1a0df65909db7732a499fb1
Successfully built Brotli
Installing collected packages: Brotli
Found existing installation: brotli 1.0.9
Uninstalling brotli-1.0.9:
Successfully uninstalled brotli-1.0.9
Successfully installed Brotli-1.0.9The "breaking" change is Python: use a new output buffer code That one changed both files Well, C/C++ is Greek to me (😜 ), but it looks like this "new output buffer code" was essentially a performance optimisation (and security patch?) feature for the upcoming |
|
G changed the module from C++ to C. In principle this would have been a Good Thing, as the author hoped, but the This fragment somewhere at the top of the file might help: /* 1700 == VC2012 (https://dev.to/yumetodo/list-of-mscver-and-mscfullver-8nd) */
#if defined(_MSC_VER) && (_MSC_VER <= 1700)
#define inline __inline
#endifOr modify |
Indebted for your continuous help on this ❤️ ; file Failed attempt at building brotli-v1.0.9-52-gc8df4b3-mod under py3.4, on Win x86python -m pip wheel "google-brotli-v1.0.9-52-gc8df4b3.tar.gz" =>
Processing c:\python3410-32\google-brotli-v1.0.9-52-gc8df4b3.tar.gz
Building wheels for collected packages: Brotli
Building wheel for Brotli (setup.py) ... error
ERROR: Complete output from command 'C:\Python3410-32\python.exe' -u -c 'import setuptools, tokenize;__file__='"'"'C:\\Users\\<redacted>\\AppData\\Local\\Temp\\pip-req-build-a3le9hya\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\<redacted>\AppData\Local\Temp\pip-wheel-z7_iyuty':
ERROR: running bdist_wheel
running build
running build_py
creating bin
creating bin\lib.win32-3.4
copying python\brotli.py -> bin\lib.win32-3.4
running build_ext
building '_brotli' extension
creating bin\temp.win32-3.4
creating bin\temp.win32-3.4\Release
creating bin\temp.win32-3.4\Release\python
creating bin\temp.win32-3.4\Release\c
creating bin\temp.win32-3.4\Release\c\common
creating bin\temp.win32-3.4\Release\c\dec
creating bin\temp.win32-3.4\Release\c\enc
C:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -Ic/include -IC:\Python3410-32\include -IC:\Python3410-32\include /Tcpython/_brotli.c /Fobin\temp.win32-3.4\Release\python/_brotli.obj
_brotli.c
python/_brotli.c(229) : error C2143: syntax error : missing ';' before 'type'
python/_brotli.c(230) : error C2065: 'mode_value' : undeclared identifier
python/_brotli.c(234) : error C2065: 'mode_value' : undeclared identifier
python/_brotli.c(296) : error C2059: syntax error : '.'
python/_brotli.c(609) : error C2059: syntax error : '.'
python/_brotli.c(861) : error C2059: syntax error : '.'
python/_brotli.c(927) : error C2061: syntax error : identifier 'brotli_methods'
python/_brotli.c(927) : error C2059: syntax error : ';'
python/_brotli.c(927) : error C3409: empty attribute block is not allowed
python/_brotli.c(927) : error C2513: '/*global*/ ' : no variable declared before '='
python/_brotli.c(945) : error C2065: 'brotli_methods' : undeclared identifier
python/_brotli.c(945) : error C2099: initializer is not a constant
python/_brotli.c(983) : error C2143: syntax error : missing ';' before 'type'
python/_brotli.c(984) : error C2275: 'uint32_t' : illegal use of this type as an expression
C:\Program Files\Microsoft Visual Studio 10.0\VC\INCLUDE\stdint.h(23): see declaration of 'uint32_t'
python/_brotli.c(984) : error C2146: syntax error : missing ';' before identifier 'decoderVersion'
python/_brotli.c(984) : error C2065: 'decoderVersion' : undeclared identifier
python/_brotli.c(985) : error C2065: 'version' : undeclared identifier
python/_brotli.c(985) : warning C4047: 'function' : 'char *' differs in levels of indirection from 'int'
python/_brotli.c(985) : warning C4024: '_snprintf' : different types for formal and actual parameter 1
python/_brotli.c(985) : error C2065: 'version' : undeclared identifier
python/_brotli.c(986) : error C2065: 'decoderVersion' : undeclared identifier
python/_brotli.c(986) : error C2065: 'decoderVersion' : undeclared identifier
python/_brotli.c(986) : error C2065: 'decoderVersion' : undeclared identifier
python/_brotli.c(987) : error C2065: 'version' : undeclared identifier
python/_brotli.c(987) : warning C4047: 'function' : 'const char *' differs in levels of indirection from 'int'
python/_brotli.c(987) : warning C4024: 'PyModule_AddStringConstant' : different types for formal and actual parameter 3
error: command 'C:\\Program Files\\Microsoft Visual Studio 10.0\\VC\\BIN\\cl.exe' failed with exit status 2
----------------------------------------
ERROR: Failed building wheel for Brotli
Running setup.py clean for Brotli
Failed to build Brotli
ERROR: Failed to build one or more wheelsIf you or someone else can come up with more ideas, chime in - remember, I have to be spoon-fed in this case, as C/C++/Python are totally alien to me 😊 ... FWIW, I'm not pursuing this anymore with determination 😉 ; the previous Brotli tag Brotli TESTyoutube-dl -v "https://www.extra.cz/cauky-lidi-70-dil-babis-predstavil-pohadky-prymulanek-nebo-andrejovy-nove-saty-ac867" -o brotli_TEST.mp4 =>
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'https://www.extra.cz/cauky-lidi-70-dil-babis-predstavil-pohadky-prymulanek-nebo-andrejovy-nove-saty-ac867', '-o', 'brotli_TEST.mp4']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.09.03.2215
[debug] Lazy loading extractors enabled
[debug] Single file build
[debug] Python 3.4.10 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2k 26 Jan 2017
[debug] exe versions: none
[debug] Proxy map: {}
[generic] cauky-lidi-70-dil-babis-predstavil-pohadky-prymulanek-nebo-andrejovy-nove-saty-ac867: Requesting header
WARNING: Could not send HEAD request to https://www.extra.cz/cauky-lidi-70-dil-babis-predstavil-pohadky-prymulanek-nebo-andrejovy-nove-saty-ac867: HTTP Error 405: Method Not Allowed
[generic] cauky-lidi-70-dil-babis-predstavil-pohadky-prymulanek-nebo-andrejovy-nove-saty-ac867: Downloading webpage
WARNING: Falling back on generic information extractor.
[generic] cauky-lidi-70-dil-babis-predstavil-pohadky-prymulanek-nebo-andrejovy-nove-saty-ac867: Extracting information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'https://stream.eom.cz/storage-prod/videos/D1yA3uml-57xeMHzN.mp4'
[download] Destination: brotli_TEST.mp4
[download] 8.1% of 86.81MiB at 881.40KiB/s ETA 01:40 |
|
Two further issues for C90:
That should resolve everything except the problem at l.927 and the consequent errors at l.945:
Maybe it's a weird side-effect of other errors, since all the syntax cases here are used elsewhere without error, except the expression This patch against the 1.1 code includes changes at l.920 and l.951 that would help if the compiler really couldn't use that expression, but should be omitted in a first test.--- <unnamed>
+++ <unnamed>
@@ -1,3 +1,8 @@
+/* 1700 == VC2012 (https://dev.to/yumetodo/list-of-mscver-and-mscfullver-8nd) */
+#if defined(_MSC_VER) && (_MSC_VER <= 1700)
+ #define inline __inline
+#endif
+
#define PY_SSIZE_T_CLEAN 1
#include <Python.h>
#include <bytesobject.h>
@@ -24,6 +29,13 @@
/* Number of whole allocated size. */
Py_ssize_t allocated;
} BlocksOutputBuffer;
+
+/* initialize a new BlocksOutputBuffer in C<99 */
+static inline BlocksOutputBuffer BlocksOutputBuffer_New() {
+ BlocksOutputBuffer buffer;
+ buffer.list = NULL;
+ return buffer;
+}
static const char unable_allocate_msg[] = "Unable to allocate output buffer.";
@@ -219,19 +231,20 @@
if (!PyInt_Check(o)) {
PyErr_SetString(BrotliError, "Invalid mode");
return 0;
- }
-
- int mode_value = -1;
- if (!as_bounded_int(o, &mode_value, 0, 255)) {
- PyErr_SetString(BrotliError, "Invalid mode");
- return 0;
- }
- *mode = (BrotliEncoderMode) mode_value;
- if (*mode != BROTLI_MODE_GENERIC &&
- *mode != BROTLI_MODE_TEXT &&
- *mode != BROTLI_MODE_FONT) {
- PyErr_SetString(BrotliError, "Invalid mode");
- return 0;
+ } else { /* declarations precede statements in C<99 */
+
+ int mode_value = -1;
+ if (!as_bounded_int(o, &mode_value, 0, 255)) {
+ PyErr_SetString(BrotliError, "Invalid mode");
+ return 0;
+ }
+ *mode = (BrotliEncoderMode) mode_value;
+ if (*mode != BROTLI_MODE_GENERIC &&
+ *mode != BROTLI_MODE_TEXT &&
+ *mode != BROTLI_MODE_FONT) {
+ PyErr_SetString(BrotliError, "Invalid mode");
+ return 0;
+ }
}
return 1;
@@ -288,7 +301,7 @@
size_t available_out;
uint8_t* next_out;
- BlocksOutputBuffer buffer = {.list=NULL};
+ BlocksOutputBuffer buffer = BlocksOutputBuffer_New();
PyObject *ret;
if (BlocksOutputBuffer_InitAndGrow(&buffer, &available_out, &next_out) < 0) {
@@ -601,7 +614,7 @@
size_t available_out;
uint8_t* next_out;
- BlocksOutputBuffer buffer = {.list=NULL};
+ BlocksOutputBuffer buffer = BlocksOutputBuffer_New();
PyObject *ret;
if (BlocksOutputBuffer_InitAndGrow(&buffer, &available_out, &next_out) < 0) {
@@ -853,7 +866,7 @@
uint8_t* next_out;
size_t available_out;
- BlocksOutputBuffer buffer = {.list=NULL};
+ BlocksOutputBuffer buffer = BlocksOutputBuffer_New();
PyObject *ret;
static const char *kwlist[] = {"string", NULL};
@@ -920,7 +933,7 @@
}
static PyMethodDef brotli_methods[] = {
- {"decompress", (PyCFunction)brotli_decompress, METH_VARARGS | METH_KEYWORDS, brotli_decompress__doc__},
+ {"decompress", (PyCFunction)brotli_decompress, METH_VARARGS /* | METH_KEYWORDS */, brotli_decompress__doc__},
{NULL, NULL, 0, NULL}
};
@@ -951,7 +964,9 @@
#endif
PyMODINIT_FUNC INIT_BROTLI(void) {
- PyObject *m = CREATE_BROTLI;
+ PyObject *m;
+ brotli_methods[0].ml_flags |= METH_KEYWORDS
+ m = CREATE_BROTLI;
BrotliError = PyErr_NewException((char*) "brotli.error", NULL, NULL);
if (BrotliError != NULL) {
@@ -975,11 +990,13 @@
PyModule_AddIntConstant(m, "MODE_TEXT", (int) BROTLI_MODE_TEXT);
PyModule_AddIntConstant(m, "MODE_FONT", (int) BROTLI_MODE_FONT);
- char version[16];
- uint32_t decoderVersion = BrotliDecoderVersion();
- snprintf(version, sizeof(version), "%d.%d.%d",
- decoderVersion >> 24, (decoderVersion >> 12) & 0xFFF, decoderVersion & 0xFFF);
- PyModule_AddStringConstant(m, "__version__", version);
+ do { /* declarations precede statements in C<99 */
+ char version[16];
+ uint32_t decoderVersion = BrotliDecoderVersion();
+ snprintf(version, sizeof(version), "%d.%d.%d",
+ decoderVersion >> 24, (decoderVersion >> 12) & 0xFFF, decoderVersion & 0xFFF);
+ PyModule_AddStringConstant(m, "__version__", version);
+ } while (0);
RETURN_BROTLI;
} |
The test is failing on CI since some time now Upstream has not made a release since Dec. 2021, so 2 years ago now. Making a new release seems stuck, see: ytdl-org/youtube-dl#32570 ytdl-org/youtube-dl#31585 ytdl-org/youtube-dl#31067 Upstream has more than 3800 open issues, and around 500 open pull requests. We have to draw a line somewhere: I am aware that this is downloaded 4500/month, but seeing upstream not being really reactive worries me. They do have some low activity with a few commits lately, but that's not enough to save the project in my opinion.
|
Dear @ytdl-org team, I think it is time to create a new release build with all recent improvements. The latest released "2021.12.17" from 2021-12-16, more two years and one month: Thanks in advance. |
Checklist
Question
Preface:
There have been a number of fixes done since the last release (2021-12-17) {over a years worth} The error messages refer using youtube-dl -U (which is already in place). All the errors I am experiencing are already closed issues, but no no release so youtube-dl -U won't fix it. Issue #31583
Question:
Can either the instructions be updated (and the error message refering to youtube-dl -U be dropped) to "install" via a github pull from Master
or
Can a new release be cut?
Thanks
The text was updated successfully, but these errors were encountered: