Unable to build packages `assert next_token == "("` #650

alistair23 · 2022-11-16T07:27:33Z

When trying to build any toltec package I am seeing this error:

$  make koreader
./scripts/package_build.py  "koreader"
Traceback (most recent call last):
  File "/var/home/alistair/sources/remarkable/toltec/./scripts/package_build.py", line 44, in <module>
    repo = Repo(paths.RECIPE_DIR, paths.REPO_DIR)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/alistair/sources/remarkable/toltec/scripts/toltec/repo.py", line 56, in __init__
    self.generic_recipes[name] = GenericRecipe.from_file(
                                 ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/alistair/sources/remarkable/toltec/scripts/toltec/recipe.py", line 45, in from_file
    return GenericRecipe(name, path, recipe.read())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/alistair/sources/remarkable/toltec/scripts/toltec/recipe.py", line 58, in __init__
    variables, functions = bash.get_declarations(definition)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/alistair/sources/remarkable/toltec/scripts/toltec/bash.py", line 147, in get_declarations
    assert next_token == "("
AssertionError
make: *** [Makefile:60: koreader] Error 1

It doesn't matter what package I build, I see the same error.

$ python3 --version
Python 3.11.0

bash --version
GNU bash, version 5.2.2(1)-release (x86_64-redhat-linux-gnu)
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Eeems · 2022-11-16T13:46:10Z

Have you tried with python 3.8
https://github.com/toltec-dev/toltec/blob/stable/.github/workflows/pr.yml#L31

matteodelabre · 2022-11-16T15:08:08Z

I think this has more to do with the bash version, since the error is coming from the bash “parser”. It works on my computer running Bash 5.1.16, perhaps they changed their output format in 5.2.2? @alistair23 Could you add a print(declarations) after line 123 of scripts/toltec/bash.py and give us some of the output? I get the following output for one recipe:

declare -- LINENO
declare -- MACHTYPE="x86_64-pc-linux-gnu"
declare -x OLDPWD
declare -- OPTERR="1"
declare -i OPTIND="1"
declare -- OSTYPE="linux-gnu"
declare -x PATH="..."
declare -a PIPESTATUS=([0]="0")
declare -ir PPID="4130"
declare -- PS4="+ "
declare -x PWD=".../toltec"
declare -i RANDOM
declare -- SECONDS
declare -- SHELL="/bin/zsh"
declare -r SHELLOPTS="braceexpand:hashall:interactive-comments"
declare -x SHLVL="1"
declare -i SRANDOM
declare -- TERM="dumb"
declare -ir UID="1000"
declare -- _="-f"
declare -- arch="rmall"
declare -a flags=([0]="patch_rm2fb")
declare -- image="python:v2.1"
declare -a installdepends=([0]="display")
declare -- license="MIT"
declare -- maintainer="raisjn <of.raisjn@gmail.com>"
declare -a makedepends=()
declare -a noextract=()
declare -- pkgdesc="Wordle clone"
declare -- pkgname="wordlet"
declare -a pkgnames=([0]="bufshot" [1]="dumbskull" [2]="genie" [3]="harmony" [4]="iago" [5]="lamp" [6]="mines" [7]="nao" [8]="remux" [9]="rpncalc" [10]="simple" [11]="wordlet")
declare -- pkgver="0.0.1-2"
declare -- section="games"
declare -a sha256sums=([0]="e7f0379d5ad9de61c32a296ab831780cfea73352fbccfe2472422165943457ef" [1]="SKIP" [2]="SKIP")
declare -a source=([0]="https://github.com/rmkit-dev/rmkit/archive/070d5df29a71e55050a5e993f047f1590d69ec1c.zip" [1]="remux.service" [2]="genie.service")
declare -- timestamp="2022-06-23T20:03:10Z"
declare -- url="https://rmkit.dev/apps/wordlet"

build ()
{
    cmake -B build -DCMAKE_TOOLCHAIN_FILE="/usr/share/cmake/$CHOST.cmake" -DCMAKE_INSTALL_PREFIX="/opt";
    cmake --build build
}
package ()
{
    install -D -m 755 -t "$pkgdir"/opt/bin "$srcdir"/build/vnsee "$srcdir"/vnsee-gui;
    install -D -m 644 -t "$pkgdir"/opt/etc/draft/icons "$srcdir"/vnsee.png;
    install -D -m 644 -t "$pkgdir"/opt/etc/draft "$srcdir"/vnsee.draft;
    install -D -m 644 -t "$pkgdir"/opt/usr/share/applications "$srcdir"/VNSee.oxide
}
prepare ()
{
    patch -d "$srcdir" -p1 < "$srcdir"/vnsee-gui-set-path.patch;
    bsdtar --strip-components 1 -xf "$srcdir"/"$_boost_pp_ver".zip -C "$srcdir"/vendor/boost-preprocessor
}

alistair23 · 2022-11-17T09:49:46Z

This is the output

$ make koreader
./scripts/package_build.py  "koreader"
build () 
{ 
    rm .cargo/config;
    cargo build --release --bin appmarkable
}
package () 
{ 
    install -D -m 755 -t "$pkgdir"/opt/bin "$srcdir"/target/armv7-unknown-linux-gnueabihf/release/appmarkable
}
declare -- BASH="/usr/bin/bash"
declare -r BASHOPTS="checkwinsize:cmdhist:complete_fullquote:extquote:force_fignore:globasciiranges:globskipdots:hostcomplete:interactive_comments:patsub_replacement:progcomp:promptvars:sourcepath"
declare -i BASHPID
declare -A BASH_ALIASES=()
declare -a BASH_ARGC=()
declare -a BASH_ARGV=()
declare -- BASH_ARGV0
declare -A BASH_CMDS=()
declare -- BASH_COMMAND
declare -a BASH_LINENO=()
declare -- BASH_LOADABLES_PATH="/usr/local/lib/bash:/usr/lib/bash:/opt/local/lib/bash:/usr/pkg/lib/bash:/opt/pkg/lib/bash:."
declare -a BASH_SOURCE=()
declare -- BASH_SUBSHELL
declare -ar BASH_VERSINFO=([0]="5" [1]="2" [2]="2" [3]="1" [4]="release" [5]="x86_64-redhat-linux-gnu")
declare -- BASH_VERSION="5.2.2(1)-release"
declare -- COMP_WORDBREAKS
declare -a DIRSTACK=()
declare -- EPOCHREALTIME
declare -- EPOCHSECONDS
declare -ir EUID="1000"
declare -a FUNCNAME
declare -a GROUPS=()
declare -i HISTCMD
declare -- HOSTNAME="als-thinkpad-x1"
declare -- HOSTTYPE="x86_64"
declare -- IFS=$' \t\n'
declare -- LINENO
declare -- MACHTYPE="x86_64-redhat-linux-gnu"
declare -x OLDPWD
declare -- OPTERR="1"
declare -i OPTIND="1"
declare -- OSTYPE="linux-gnu"
declare -x PATH="/var/home/alistair/.cargo/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/var/home/alistair/.local/bin"
declare -a PIPESTATUS=([0]="0")
declare -ir PPID="39382"
declare -- PS4="+ "
declare -x PWD="/var/home/alistair/sources/remarkable/toltec"
declare -i RANDOM
declare -- SECONDS
declare -- SHELL="/bin/bash"
declare -r SHELLOPTS="braceexpand:hashall:interactive-comments"
declare -x SHLVL="1"
declare -i SRANDOM
declare -- TERM="dumb"
declare -ir UID="1000"
declare -- _="-f"
declare -a flags=([0]="patch_rm2fb")
declare -- image="rust:v2.1"
declare -a installdepends=([0]="display")
declare -- license="MIT"
declare -- maintainer="Linus K. <linus@cosmos-ink.net>"
declare -- pkgdesc="Front-end for apps that do not have a graphical user interface"
declare -a pkgnames=([0]="appmarkable")
declare -- pkgver="0.0.0-11"
declare -- section="devel"
declare -a sha256sums=([0]="76e151aeae0f18b206dd3c6258bf74bcb5256ee2f803e1ed2073278831158f60")
declare -a source=([0]="https://github.com/LinusCDE/appmarkable/archive/c44ee87ea2b1f1e41c9592476c076150c9a1acf4.zip")
declare -- timestamp="2021-03-10T18:36Z"
declare -- url="https://github.com/LinusCDE/appmarkable"

Traceback (most recent call last):
  File "/var/home/alistair/sources/remarkable/toltec/./scripts/package_build.py", line 44, in <module>
    repo = Repo(paths.RECIPE_DIR, paths.REPO_DIR)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/alistair/sources/remarkable/toltec/scripts/toltec/repo.py", line 56, in __init__
    self.generic_recipes[name] = GenericRecipe.from_file(
                                 ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/alistair/sources/remarkable/toltec/scripts/toltec/recipe.py", line 45, in from_file
    return GenericRecipe(name, path, recipe.read())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/alistair/sources/remarkable/toltec/scripts/toltec/recipe.py", line 58, in __init__
    variables, functions = bash.get_declarations(definition)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/alistair/sources/remarkable/toltec/scripts/toltec/bash.py", line 149, in get_declarations
    assert next_token == "("
AssertionError
make: *** [Makefile:60: koreader] Error 1

with this diff applied:

diff --git a/scripts/toltec/bash.py b/scripts/toltec/bash.py
index cb194e2..3617024 100644
--- a/scripts/toltec/bash.py
+++ b/scripts/toltec/bash.py
@@ -122,6 +122,8 @@ declare -p
 
     declarations = declarations_subshell.stdout.decode()
 
+    print(declarations)
+
     # Parse `declare` statements and function statements
     lexer = shlex.shlex(declarations, posix=True)
     lexer.wordchars = lexer.wordchars + "-"

matteodelabre · 2022-11-17T15:21:13Z

Thanks! After some investigation, the problem seems to stem from the way the IFS variable contents are printed by bash. Our primitive parser doesn’t know how to handle C-quoted strings, i.e., string literals that have a $ sign in front of them.

matteodelabre · 2022-11-26T18:25:46Z

shfmt’s upcoming 3.6.0 release will include both a way to build an AST for a Bash program (--to-json), and a way to generate a Bash program from an AST (--from-json). I think I’ll replace our primitive parser with that.

alistair23 · 2023-01-19T10:51:26Z

3.6 has been released, is it possible to update to use that?

Eeems · 2023-03-15T17:24:40Z

@alistair23 We would happily accept a PR

Eeems · 2023-08-17T19:33:12Z

At some point we need to switch over to using toltecmk to build the repo instead of maintaining two versions of the build code.

Eeems · 2023-08-21T22:53:50Z

@alistair23 you indicated that you are still having issues with c-quoted strings when using the latest toltecmk version? Could you provide some further context?

alistair23 · 2023-08-22T01:36:31Z

Yeah, so running make doesn't work

$ make linux-mainline
./scripts/package_build.py  "linux-mainline"
Traceback (most recent call last):
  File "/var/home/alistair/sources/remarkable/toltec/./scripts/package_build.py", line 44, in <module>
    repo = Repo(paths.RECIPE_DIR, paths.REPO_DIR)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/alistair/sources/remarkable/toltec/scripts/toltec/repo.py", line 60, in __init__
    self.generic_recipes[name] = GenericRecipe.from_file(
                                 ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/alistair/sources/remarkable/toltec/scripts/toltec/recipe.py", line 46, in from_file
    return GenericRecipe(name, path, recipe.read())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/alistair/sources/remarkable/toltec/scripts/toltec/recipe.py", line 59, in __init__
    variables, functions = bash.get_declarations(definition)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/home/alistair/sources/remarkable/toltec/scripts/toltec/bash.py", line 147, in get_declarations
    assert next_token == "("
           ^^^^^^^^^^^^^^^^^
AssertionError
make: *** [Makefile:60: linux-mainline] Error 1

Running toltecmk directly does work though

Eeems · 2023-08-22T03:10:25Z

Right, totlec doesn't use toltecmk yet, which was the comment I left. We still have to port the changes over to this repo. It's good to hear that toltecmk itself is working, though.

matteodelabre added bug Something isn't working tooling Set of scripts and configuration files for building the packages labels Nov 16, 2022

Eeems mentioned this issue May 14, 2023

Handle c-quoted strings toltec-dev/build#31

Merged

Eeems closed this as completed in toltec-dev/build#31 Aug 17, 2023

Eeems reopened this Aug 17, 2023

Eeems linked a pull request Dec 6, 2023 that will close this issue

Add support for limiting packages to OS versions #759

Merged

Eeems linked a pull request Dec 16, 2023 that will close this issue

Update the build process to use toltecmk #789

Merged

5 tasks

Eeems added this to the 2024-W23 Merge Window milestone Jun 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to build packages `assert next_token == "("` #650

Unable to build packages `assert next_token == "("` #650

alistair23 commented Nov 16, 2022

Eeems commented Nov 16, 2022

matteodelabre commented Nov 16, 2022 •

edited

alistair23 commented Nov 17, 2022

matteodelabre commented Nov 17, 2022

matteodelabre commented Nov 26, 2022

alistair23 commented Jan 19, 2023

Eeems commented Mar 15, 2023

Eeems commented Aug 17, 2023

Eeems commented Aug 21, 2023

alistair23 commented Aug 22, 2023

Eeems commented Aug 22, 2023 •

edited

Unable to build packages assert next_token == "(" #650

Unable to build packages assert next_token == "(" #650

Comments

alistair23 commented Nov 16, 2022

Eeems commented Nov 16, 2022

matteodelabre commented Nov 16, 2022 • edited

alistair23 commented Nov 17, 2022

matteodelabre commented Nov 17, 2022

matteodelabre commented Nov 26, 2022

alistair23 commented Jan 19, 2023

Eeems commented Mar 15, 2023

Eeems commented Aug 17, 2023

Eeems commented Aug 21, 2023

alistair23 commented Aug 22, 2023

Eeems commented Aug 22, 2023 • edited

Unable to build packages `assert next_token == "("` #650

Unable to build packages `assert next_token == "("` #650

matteodelabre commented Nov 16, 2022 •

edited

Eeems commented Aug 22, 2023 •

edited