Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Timeout on QML file #536

Open
timangus opened this issue Nov 19, 2020 · 15 comments
Open

Timeout on QML file #536

timangus opened this issue Nov 19, 2020 · 15 comments

Comments

@timangus
Copy link

I get a timeout when invoking cloc on a particular QML file. The weird thing is, if I add a comment line as follows, it works fine:

--- a/source/updater/editor/main.qml
+++ b/source/updater/editor/main.qml
@@ -797,6 +797,7 @@ ApplicationWindow
 
                     function generateJSON(onComplete)
                     {
+                        /*abc*/
                         updateObject = {};
                         updateObject.version = version.text;
                         updateObject.targetVersionRegex = targetVersionRegex.text;
@AlDanial
Copy link
Owner

Wow, that's messed up! I had to bump up the timeout value to 3 seconds for it to go through:

/tmp> cloc --timeout 3 main.qml 
       1 text file.
       1 unique file.                              
       0 files ignored.

github.com/AlDanial/cloc v 1.89  T=2.48 s (0.4 files/s, 368.5 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
QML                              1            185             20            710
-------------------------------------------------------------------------------

(this actually points out another bug: I tried giving --timeout a float and it complained). A few experiments revealed the location of the delay: the Regexp::Common module. This is easy to demonstrate by commenting out line 9087 in the current development version, ie, where the language filters for QML are defined, including the call to Regexp::Common:

 9084     'QML'                => [
 9085                                 [ 'rm_comments_in_strings', '"', '/*', '*/' ],
 9086                                 [ 'rm_comments_in_strings', '"', '//', '' ],
 9087      #                          [ 'call_regexp_common'  , 'C++'    ],
 9088                                 [ 'remove_inline'       , '//.*$'  ],
 9089                             ],

then cloc runs quickly--with incorrect results though.

The Regexp::Common module bundled with cloc is from 2013. Do you know if your system perl has Regexp::Common installed or if uses the bundled one? It isn't straightforward to figure this out; basically have to temporarily add a debug statement as with line 78 below:

   71 BEGIN {
   72     if (eval "use Regexp::Common;") {
   73         $HAVE_Rexexp_Common = 1;
   74     } else {
   75         $HAVE_Rexexp_Common = 0;
   76     }
   77 }
   78 die "HAVE_Rexexp_Common=$HAVE_Rexexp_Common";

I should look for a newer version--but no guarantees that will fix this.

@timangus
Copy link
Author

This sent me down a bit of a rabbit hole 😁. First of all, to answer your question:

HAVE_Rexexp_Common=0 at ./cloc line 78.

Fair enough, I'll install the system version, and see if that works any better:

$ sudo apt install libregexp-common-perl 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
libregexp-common-perl is already the newest version (2017060201-1).
0 to upgrade, 0 to newly install, 0 to remove and 0 not to upgrade.

Hmm, that's weird. I do a bit of googling and change the test for RegExpCommon:

--- a/cloc
+++ b/cloc
@@ -69,7 +69,7 @@ my $HAVE_Rexexp_Common;
 # Regexp::Common isn't in the standard distribution.  It will
 # be installed in a temp directory if necessary.
 BEGIN {
-    if (eval "use Regexp::Common;") {
+    if (eval{require Regexp::Common}) {
         $HAVE_Rexexp_Common = 1;
     } else {
         $HAVE_Rexexp_Common = 0;

I am not a perl hacker (IANAPH?) so I don't really know the ins and outs of this, but it seems to me like the test is broken? (I also note that the variable is named HAVE_ReXexp_Common, not HAVE_ReGexp_Common, but that's unimportant.) Anyway, with the test changed to the above (so using the system Regexp::Common), cloc now works on the problematic QML:

$ ./cloc ~/sources/graphia/source/updater/editor/main.qml 
       1 text file.
       1 unique file.                              
Use of uninitialized value in regexp compilation at ./cloc line 12234.
       0 files ignored.

github.com/AlDanial/cloc v 1.89  T=0.02 s (60.3 files/s, 55193.8 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
QML                              1            185              0            730
-------------------------------------------------------------------------------

...albeit with a warning, perhaps related to the timeout? Who knows.

AlDanial added a commit that referenced this issue Nov 21, 2020
Update logic deciding if Regexp::Common is already installed.
Min Perl version now 5.10.0 (for Regexp::Common)
@AlDanial
Copy link
Owner

Thanks for doing the background research. I agree the existence test for Regexp::Common was broken; should be fixed now.
What bugs me is that even with that fix plus an updated Regexp::Common, cloc still times out on your file. I'm still puzzled as to what's happening.

@timangus
Copy link
Author

Hmm yes. Having forced it to use the system library, I previously thought I had managed to stop it timing out, but now I've tried it again, it does indeed still time out. Perhaps I'd accidentally managed to test with a copy of main.qml that included the comment line.

@loreb
Copy link

loreb commented Nov 1, 2021

Not sure if it's the same bug, but anyway...
I get timeouts counting the lines of the dependencies for the latest release of https://git.sr.ht/~sircmpwn/aerc/refs, both using cloc from my distro's package manager and the latest from this repo:

5 errors:
Line count, exceeded timeout:  ./pkg/mod/github.com/go-ini/ini@v1.52.0/ini_test.go
Line count, exceeded timeout:  ./pkg/mod/golang.org/x/text@v0.3.3/internal/export/idna/tables10.0.0.go
Line count, exceeded timeout:  ./pkg/mod/golang.org/x/text@v0.3.3/internal/export/idna/tables11.0.0.go
Line count, exceeded timeout:  ./pkg/mod/golang.org/x/text@v0.3.3/internal/export/idna/tables12.0.0.go
Line count, exceeded timeout:  ./pkg/mod/golang.org/x/text@v0.3.3/internal/export/idna/tables9.0.0.go

Regexp::Common is installed; those files are automatically generated and make heavy use of unicode, but I have no idea if that is the reason.

@AlDanial
Copy link
Owner

AlDanial commented Nov 3, 2021

aerc and scdoc present no issues for me but I take it I'm not cloning the thing that produces the problem you see. What is the link to either clone or download a tarball for "aerc/refs" ?

> git clone https://git.sr.ht/~sircmpwn/aerc
> cloc aerc/
     174 text files.
     174 unique files.                                          
      56 files ignored.

github.com/AlDanial/cloc v 1.91  T=0.12 s (1257.5 files/s, 171729.2 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Go                             145           2381            663          17221
make                             1             19              1            102
Markdown                         1             27              0             43
YAML                             1              0              0             13
Bourne Shell                     2              0              2             12
-------------------------------------------------------------------------------
SUM:                           150           2427            666          17391
-------------------------------------------------------------------------------

> git clone https://git.sr.ht/~sircmpwn/scdoc
> cloc scdoc
      30 text files.
      29 unique files.                              
      11 files ignored.

github.com/AlDanial/cloc v 1.91  T=0.02 s (1394.8 files/s, 95599.8 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C                                9             51             17            990
Bourne Shell                     9             81              6            312
C/C++ Header                     3             16             20             53
make                             1             12              0             50
Markdown                         1             12              0             15
YAML                             1              0              0             10
-------------------------------------------------------------------------------
SUM:                            24            172             43           1430
-------------------------------------------------------------------------------

@loreb
Copy link

loreb commented Nov 3, 2021

Sorry I didn't make it clear the first time... It's not aerc itself, it's the dependencies that are downloaded when you build it - "go build" will download its dependencies to $HOME/go, so try something like this in a temporary $HOME:

$ mkdir /tmp/$$;cd /tmp/$$
$ export HOME=$PWD
$ git clone https://git.sr.ht/~sircmpwn/aerc
Cloning into 'aerc'...
(...)
$ cd aerc
$ make
go build  \
	-ldflags "-X main.Prefix=/usr/local \
	-X main.ShareDir=/usr/local/share/aerc \
	-X main.Version=0.5.2.r50.g0b19b5e" \
	-o aerc
go: downloading git.sr.ht/~sircmpwn/getopt v0.0.0-20190808004552-daaf1274538b
go: downloading github.com/mattn/go-isatty v0.0.12
go: downloading github.com/gdamore/tcell/v2 v2.0.0
go: downloading github.com/go-ini/ini v1.52.0
go: downloading github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510
go: downloading github.com/mitchellh/go-homedir v1.1.0
go: downloading github.com/riywo/loginshell v0.0.0-20190610082906-2ed199a032f6
go: downloading github.com/emersion/go-message v0.13.1-0.20201112194930-f77964fe28bd
go: downloading github.com/emersion/go-sasl v0.0.0-20200509203442-7bfe0ed36a21
go: downloading github.com/emersion/go-smtp v0.12.1
go: downloading github.com/miolini/datacounter v1.0.2
go: downloading github.com/pkg/errors v0.9.1
go: downloading golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d
go: downloading github.com/imdario/mergo v0.3.8
go: downloading github.com/kyoh86/xdg v1.2.0
go: downloading github.com/emersion/go-imap v1.0.6-0.20201014170015-5a03a09eba6d
go: downloading github.com/emersion/go-pgpmail v0.0.0-20200303213726-db035a3a4139
go: downloading github.com/ProtonMail/crypto v0.0.0-20200420072808-71bec3603bf3
go: downloading github.com/mattn/go-runewidth v0.0.8
go: downloading github.com/creack/pty v1.1.10
go: downloading github.com/danwakefield/fnmatch v0.0.0-20160403171240-cbb64ac3d964
go: downloading github.com/ddevault/go-libvterm v0.0.0-20190526194226-b7d861da3810
go: downloading golang.org/x/sys v0.0.0-20200302150141-5c8b2ff67527
go: downloading github.com/gdamore/encoding v1.0.0
go: downloading github.com/lucasb-eyer/go-colorful v1.0.3
go: downloading golang.org/x/text v0.3.4-0.20201021145329-22f1617af38e
go: downloading github.com/martinlindhe/base36 v1.1.0
go: downloading github.com/emersion/go-textwrapper v0.0.0-20200911093747-65d896831594
go: downloading github.com/mattn/go-pointer v0.0.0-20190911064623-a0a44394634f
go: downloading golang.org/x/net v0.0.0-20200301022130-244492dfa37a
go: downloading github.com/emersion/go-imap-idle v0.0.0-20190519112320-2704abd7050e
go: downloading github.com/emersion/go-imap-sortthread v1.1.1-0.20201009054724-d020d96306b3
go: downloading github.com/emersion/go-maildir v0.2.0
go: downloading github.com/fsnotify/fsnotify v1.4.7
sed -e 's:@SHAREDIR@:/usr/local/share/aerc:g' > aerc.conf < config/aerc.conf.in
scdoc < doc/aerc.1.scd > aerc.1
scdoc < doc/aerc-search.1.scd > aerc-search.1
scdoc < doc/aerc-config.5.scd > aerc-config.5
scdoc < doc/aerc-imap.5.scd > aerc-imap.5
scdoc < doc/aerc-maildir.5.scd > aerc-maildir.5
scdoc < doc/aerc-sendmail.5.scd > aerc-sendmail.5
scdoc < doc/aerc-notmuch.5.scd > aerc-notmuch.5
scdoc < doc/aerc-smtp.5.scd > aerc-smtp.5
scdoc < doc/aerc-tutorial.7.scd > aerc-tutorial.7
scdoc < doc/aerc-templates.7.scd > aerc-templates.7
scdoc < doc/aerc-stylesets.7.scd > aerc-stylesets.7
$ cd ~/go
$ exec cloc .
     100 files
     200 files
     300 files
     400 files
     500 files
     600 files
     700 files
     800 files
     900 files
    1000 files
    1100 files
    1200 files
    1300 files
    1400 files
    1500 files
    1600 files
    1700 files
    1800 files
    1900 files
    2000 files
    2100 files
    2200 files
    2300 files
    2400 files
    2500 files
    2600 files
    2700 files
    2800 files
    2900 files
    3000 files
    3089 text files.
classified 3075 files
Duplicate file check 3075 files (2346 known unique)
Unique:      100 files                                          
Unique:      200 files                                          
Unique:      300 files                                          
Unique:      400 files                                          
Unique:      500 files                                          
Unique:      600 files                                          
Unique:      700 files                                          
Unique:      800 files                                          
Unique:      900 files                                          
Unique:     1000 files                                          
Unique:     1100 files                                          
Unique:     1200 files                                          
Unique:     1300 files                                          
Unique:     1400 files                                          
Unique:     1500 files                                          
Unique:     1600 files                                          
Unique:     1700 files                                          
Unique:     1800 files                                          
Unique:     1900 files                                          
Unique:     2000 files                                          
Unique:     2100 files                                          
Unique:     2200 files                                          
Unique:     2300 files                                          
    2940 unique files.                              
Counting:  100
Counting:  200
Counting:  300
Counting:  400
Counting:  500
Counting:  600
Counting:  700
Counting:  800
Counting:  900
Counting:  1000
Counting:  1100
Counting:  1200
Counting:  1300
Counting:  1400
Counting:  1500
Counting:  1600
Counting:  1700
Counting:  1800
Counting:  1900
Counting:  2000
Counting:  2100
Counting:  2200
Counting:  2300
Counting:  2400
Counting:  2500
Counting:  2600
Counting:  2700
Counting:  2800
Counting:  2900
     567 files ignored.

7 errors:
Line count, exceeded timeout:  ./pkg/mod/github.com/go-ini/ini@v1.52.0/ini_test.go
Line count, exceeded timeout:  ./pkg/mod/golang.org/x/net@v0.0.0-20200301022130-244492dfa37a/idna/tables12.00.go
Line count, exceeded timeout:  ./pkg/mod/golang.org/x/text@v0.3.4-0.20201021145329-22f1617af38e/internal/export/idna/tables10.0.0.go
Line count, exceeded timeout:  ./pkg/mod/golang.org/x/text@v0.3.4-0.20201021145329-22f1617af38e/internal/export/idna/tables11.0.0.go
Line count, exceeded timeout:  ./pkg/mod/golang.org/x/text@v0.3.4-0.20201021145329-22f1617af38e/internal/export/idna/tables12.0.0.go
Line count, exceeded timeout:  ./pkg/mod/golang.org/x/text@v0.3.4-0.20201021145329-22f1617af38e/internal/export/idna/tables13.0.0.go
Line count, exceeded timeout:  ./pkg/mod/golang.org/x/text@v0.3.4-0.20201021145329-22f1617af38e/internal/export/idna/tables9.0.0.go

github.com/AlDanial/cloc v 1.90  T=33.02 s (77.4 files/s, 38734.1 lines/s)
--------------------------------------------------------------------------------
Language                      files          blank        comment           code
--------------------------------------------------------------------------------
Go                             2320          58850         118445        1063411
Assembly                         70           1259           1430          10426
XML                               6            228             20           5802
C                                18           1231            358           5657
HTML                             10            441             16           2214
Markdown                         48            790              0           2119
JSON                             16              0              0           1788
Bourne Shell                      8            127            436            857
C/C++ Header                      6            175            162            605
YAML                             32             57             14            396
AsciiDoc                          2             95              0            254
Perl                              3             53             12            197
make                              7             60              4            173
INI                               4             25              5             92
Bourne Again Shell                3             16              6             87
Dockerfile                        3             18             20             81
Jupyter Notebook                  1              0            253             74
--------------------------------------------------------------------------------
SUM:                           2557          63425         121181        1094233
--------------------------------------------------------------------------------

At least I get this result on my computers (plural), maybe I just need to buy something faster :D

@AlDanial
Copy link
Owner

AlDanial commented Nov 4, 2021

scdoc < doc/aerc.1.scd > aerc.1
/bin/sh: 1: scdoc: not found
make: *** [Makefile:52: aerc.1] Error 127

Starting to go down a rabbit hole here. Can you just post the first of the problem files,
./pkg/mod/github.com/go-ini/ini@v1.52.0/ini_test.go ?

@loreb
Copy link

loreb commented Nov 4, 2021

Here's all of them just in case
all.tar.gz

(fun fact: if it failed at scdoc you should already have everything in ~/go, remember to clean it up if you didn't use a temporary directory as $HOME)

@AlDanial
Copy link
Owner

AlDanial commented Nov 5, 2021

Here's what I found: cloc doesn't time out on me but is super-slow

github.com/AlDanial/cloc v 1.91  T=25.05 s (0.3 files/s, 1170.1 lines/s)  
------------------------------------------------------------------------------- 
Language                     files          blank        comment           code 
------------------------------------------------------------------------------- 
Go                               7            228          29085              0 
------------------------------------------------------------------------------- 
SUM:                             7            228          29085              0 
------------------------------------------------------------------------------- 

Counting only 1k lines/second is pretty terrible.

I took a look in one of the files, tables12.00.go, and was struck by the strings of hex characters. I copied the file to a temp location, deleted definitions for mappings and xorData, then ran cloc again. Normal speed this time; instead of 4.01 seconds for the original, took only 0.02 seconds with hex strings removed.

I'm guessing regular expressions from the Regexp::Common module freak out with these strings and do a ton of back-tracking. No work-arounds come to mind.

@loreb
Copy link

loreb commented Nov 5, 2021

ok, thanks

@andreaplanet
Copy link

andreaplanet commented Jul 25, 2022

The attached C++ file generates a timeout too. The file was a bit bigger, I reduced it to 300 lines.
cloc1.94_timeout.zip
with cloc 1.64 there is no timeout.

@wcjcn
Copy link

wcjcn commented Jul 25, 2022 via email

@AlDanial
Copy link
Owner

@andreaplanet : Which platform are you running on? I have no issues on Linux:

» cloc cloc1.94_timeout.cpp  
       1 text file.
       1 unique file.                              
       0 files ignored.

github.com/AlDanial/cloc v 1.94  T=0.96 s (1.0 files/s, 313.1 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C++                              1             79             38            183
-------------------------------------------------------------------------------

Nonetheless I have a pretty good idea about the cause of your timeout: the three occurrences of the string "-/*+^", specifically, the /* characters. cloc is regex-based and might see these as the beginning of a conventional C comment. There are no matching */ characters so the regex gets stuck in a loop. Ref Limitation #2, this is a known problem.

However: work-around logic exists in cloc to attempt to correct for this. For what ever reason, the work-around succeeds for me but not you. For starters, try running with the --strip-str-comments switch.

@andreaplanet
Copy link

I used the cloc1.94.exe downloaded from here under Windows 10.
I tried now under Fedora with cloc 1.90, same result:

cloc cloc1.94_timeout.cpp 
       1 text file.
       1 unique file.                              
       0 files ignored.

1 error:
Line count, exceeded timeout:  cloc1.94_timeout.cpp

github.com/AlDanial/cloc v 1.90  T=1.01 s (1.0 files/s, 297.2 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C++                              1             79            221              0
-------------------------------------------------------------------------------

Nice the catch "-/*+^" ... lol....
But why did Version 1.64 not had the same issue?
Maybe the " character should be part of the RegExp for C++ to fix that?

It's not an issue for me. Just reporting. :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants