Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

utils(dep_tree2sql): Update to valid syntax, address shellcheck warnings #3547

Open
wants to merge 13 commits into
base: main
Choose a base branch
from
Open

utils(dep_tree2sql): Update to valid syntax, address shellcheck warnings #3547

wants to merge 13 commits into from

Conversation

echoix
Copy link
Member

@echoix echoix commented Mar 30, 2024

I incrementally updated the utils/dep_tree2sql.sh script first to address some easy shellcheck suggestions, but it ended up being a bit more touchy, so I made this a PR by itself.

The initial state of this script was that it wasn't completely working anymore in modern shells (bash in Ubuntu 22.04).
The way I proceeded: I made a single change, dumped the database to a SQL script (pg_dump grass > dumpfile.sql), checked the diff, and continued to a further change. I also explored the various example queries in the file at first, plus exploring the tables (since I was surprised some were empty tables, 16 in this case (see below)).
Hence, it might be easier to review by checking each commit one by one instead of understanding the complete diff all at once.

The couple first commits didn't change the resulting database, as the first three find commands were erroring out because of the long deprecated find test -perm +111. Once I transformed it to the equivalent -perm /a+x ("any file that has the executable bit set for anybody"), the resulting SQL dump grew from 4.93 MB to 7.27 MB, since more deps were found. The number of empty tables is now 11.

There is only one commit that effectively makes a small difference in the results outputted: replacing find * with find ./*. This makes all the paths for the .a, .o, and .so found with find start with ./ instead of nothing. I found the difference between find . and find *, but couldn't find the reason why 20 years ago (the furthest I could go back with the grass-legacy repo https://github.com/OSGeo/grass-legacy/blob/66914c9eb661b32031f493845c6694819cd41f89/tools/sql.sh) the author chose find . for the first 3, and find * for the others. find . looks for hidden files, while find * looks for visible contents. So I kept the *.

A remaining change that I intentionally left out, is the subshell surrounding the first three finds. I assume it was there since the LD_LIBRARY_PATH was assigned to, so to prevent affecting the current shell. But it has been commented out

The last commit of all is uniformizing the shell formatting with

shfmt -w -s -i 4 -ci -bn -sr utils/dep_tree2sql.sh

My couple last shell contributions were made with that too, and some other PR is coming too.

In the end of this PR, I get the following output

gitpod /workspace/grass (dep_tree2sql_shell_fixes) $ utils/dep_tree2sql.sh `pwd`
nm: 'a.out': No such file
nm: 'a.out': No such file
nm: 'linux-vdso.so.1': No such file
nm: 'not': No such file
gitpod /workspace/grass (dep_tree2sql_shell_fixes) $ psql -d grass
grass=# \dt+
                            List of relations
 Schema |        Name         | Type  | Owner  |    Size    | Description 
--------+---------------------+-------+--------+------------+-------------
 public | ansi                | table | gitpod | 8192 bytes | 
 public | defined             | table | gitpod | 256 kB     | 
 public | depends             | table | gitpod | 160 kB     | 
 public | duplicates          | table | gitpod | 8192 bytes | 
 public | duplicates2         | table | gitpod | 8192 bytes | 
 public | imports             | table | gitpod | 224 kB     | 
 public | ldd                 | table | gitpod | 288 kB     | 
 public | lib_deps            | table | gitpod | 16 kB      | 
 public | lib_deps_1          | table | gitpod | 24 kB      | 
 public | lib_deps_2          | table | gitpod | 32 kB      | 
 public | lib_deps_trans      | table | gitpod | 24 kB      | 
 public | lib_exp             | table | gitpod | 376 kB     | 
 public | lib_imp             | table | gitpod | 224 kB     | 
 public | libc                | table | gitpod | 0 bytes    | 
 public | libs                | table | gitpod | 1488 kB    | 
 public | nonansi             | table | gitpod | 0 bytes    | 
 public | nonansi_counts      | table | gitpod | 0 bytes    | 
 public | nonansi_lib_counts  | table | gitpod | 0 bytes    | 
 public | nonansi_libs        | table | gitpod | 0 bytes    | 
 public | nonansi_prog_counts | table | gitpod | 0 bytes    | 
 public | nonansi_progs       | table | gitpod | 0 bytes    | 
 public | obj_exp             | table | gitpod | 1728 kB    | 
 public | obj_imp             | table | gitpod | 4272 kB    | 
 public | prog_exp            | table | gitpod | 912 kB     | 
 public | prog_imp            | table | gitpod | 1392 kB    | 
 public | prog_libs           | table | gitpod | 88 kB      | 
 public | shlib_exp           | table | gitpod | 400 kB     | 
 public | shlib_imp           | table | gitpod | 248 kB     | 
 public | stlib_exp           | table | gitpod | 0 bytes    | 
 public | stlib_imp           | table | gitpod | 0 bytes    | 
 public | undefined           | table | gitpod | 32 kB      | 
 public | undefined_1         | table | gitpod | 32 kB      | 
 public | undefined_2         | table | gitpod | 0 bytes    | 
 public | used                | table | gitpod | 80 kB      | 
(34 rows)

grass=# select n.nspname as table_schema,
grass-#        c.relname as table_name,
grass-#        c.reltuples as rows
grass-# from pg_class c
grass-# join pg_namespace n on n.oid = c.relnamespace
grass-# where c.relkind = 'r'
grass-#       and n.nspname not in ('information_schema','pg_catalog')
grass-# order by c.reltuples desc;
 table_schema |     table_name      | rows  
--------------+---------------------+-------
 public       | obj_imp             | 43843
 public       | prog_imp            | 21704
 public       | libs                | 18156
 public       | obj_exp             | 15247
 public       | prog_exp            | 12008
 public       | lib_exp             |  5350
 public       | shlib_exp           |  5350
 public       | defined             |  5288
 public       | shlib_imp           |  3126
 public       | imports             |  3126
 public       | lib_imp             |  3126
 public       | ldd                 |  3045
 public       | depends             |  1730
 public       | used                |  1511
 public       | prog_libs           |  1207
 public       | undefined           |   560
 public       | undefined_1         |   465
 public       | lib_deps_trans      |   251
 public       | lib_deps_1          |   217
 public       | lib_deps_2          |   213
 public       | lib_deps            |   113
 public       | ansi                |   100
 public       | duplicates2         |    70
 public       | nonansi_counts      |     0
 public       | stlib_imp           |     0
 public       | duplicates          |     0
 public       | undefined_2         |     0
 public       | libc                |     0
 public       | nonansi             |     0
 public       | nonansi_progs       |     0
 public       | nonansi_libs        |     0
 public       | nonansi_prog_counts |     0
 public       | nonansi_lib_counts  |     0
 public       | stlib_exp           |     0
(34 rows)

grass=#

Before this PR, I get the following:

gitpod /workspace/grass (dep_tree2sql_shell_fixes) $ psql -d grass
psql (12.18 (Ubuntu 12.18-1.pgdg22.04+1))
Type "help" for help.

grass=# \dt+
                            List of relations
 Schema |        Name         | Type  | Owner  |    Size    | Description 
--------+---------------------+-------+--------+------------+-------------
 public | ansi                | table | gitpod | 8192 bytes | 
 public | defined             | table | gitpod | 256 kB     | 
 public | depends             | table | gitpod | 160 kB     | 
 public | duplicates          | table | gitpod | 8192 bytes | 
 public | duplicates2         | table | gitpod | 8192 bytes | 
 public | imports             | table | gitpod | 224 kB     | 
 public | ldd                 | table | gitpod | 8192 bytes | 
 public | lib_deps            | table | gitpod | 16 kB      | 
 public | lib_deps_1          | table | gitpod | 24 kB      | 
 public | lib_deps_2          | table | gitpod | 32 kB      | 
 public | lib_deps_trans      | table | gitpod | 24 kB      | 
 public | lib_exp             | table | gitpod | 376 kB     | 
 public | lib_imp             | table | gitpod | 224 kB     | 
 public | libc                | table | gitpod | 0 bytes    | 
 public | libs                | table | gitpod | 8192 bytes | 
 public | nonansi             | table | gitpod | 0 bytes    | 
 public | nonansi_counts      | table | gitpod | 0 bytes    | 
 public | nonansi_lib_counts  | table | gitpod | 0 bytes    | 
 public | nonansi_libs        | table | gitpod | 0 bytes    | 
 public | nonansi_prog_counts | table | gitpod | 0 bytes    | 
 public | nonansi_progs       | table | gitpod | 0 bytes    | 
 public | obj_exp             | table | gitpod | 1696 kB    | 
 public | obj_imp             | table | gitpod | 4184 kB    | 
 public | prog_exp            | table | gitpod | 0 bytes    | 
 public | prog_imp            | table | gitpod | 0 bytes    | 
 public | prog_libs           | table | gitpod | 0 bytes    | 
 public | shlib_exp           | table | gitpod | 400 kB     | 
 public | shlib_imp           | table | gitpod | 248 kB     | 
 public | stlib_exp           | table | gitpod | 0 bytes    | 
 public | stlib_imp           | table | gitpod | 0 bytes    | 
 public | undefined           | table | gitpod | 32 kB      | 
 public | undefined_1         | table | gitpod | 40 kB      | 
 public | undefined_2         | table | gitpod | 0 bytes    | 
 public | used                | table | gitpod | 80 kB      | 
(34 rows)

grass=# select n.nspname as table_schema,
grass-#        c.relname as table_name,
grass-#        c.reltuples as rows
grass-# from pg_class c
grass-# join pg_namespace n on n.oid = c.relnamespace
grass-# where c.relkind = 'r'
grass-#       and n.nspname not in ('information_schema','pg_catalog')
grass-# order by c.reltuples desc;
 table_schema |     table_name      | rows  
--------------+---------------------+-------
 public       | obj_imp             | 43843
 public       | obj_exp             | 15247
 public       | shlib_exp           |  5350
 public       | lib_exp             |  5350
 public       | defined             |  5288
 public       | shlib_imp           |  3126
 public       | imports             |  3126
 public       | lib_imp             |  3126
 public       | depends             |  1730
 public       | used                |  1511
 public       | undefined_1         |   560
 public       | undefined           |   560
 public       | lib_deps_trans      |   251
 public       | lib_deps_1          |   217
 public       | lib_deps_2          |   213
 public       | lib_deps            |   113
 public       | ansi                |   100
 public       | duplicates2         |    70
 public       | nonansi_counts      |     0
 public       | stlib_imp           |     0
 public       | prog_imp            |     0
 public       | prog_exp            |     0
 public       | libs                |     0
 public       | ldd                 |     0
 public       | duplicates          |     0
 public       | undefined_2         |     0
 public       | prog_libs           |     0
 public       | libc                |     0
 public       | nonansi             |     0
 public       | nonansi_progs       |     0
 public       | nonansi_libs        |     0
 public       | nonansi_prog_counts |     0
 public       | nonansi_lib_counts  |     0
 public       | stlib_exp           |     0
(34 rows)

grass=#

@echoix
utils(dep_tree2sql): Exit with exit code 1 when changing directory fails
73a6f68
@echoix
utils(dep_tree2sql): Use $(...) notation instead of legacy backticked…
95706ae 
… command substitution
@echoix
utils(dep_tree2sql): Double quoting some variables
5ed5e92
@echoix
utils(dep_tree2sql): Drop database only if exists, supported on postg…
4b889c5 
…res 9.2 and up (released september 2012)
@echoix
utils(dep_tree2sql): Replace fgrep with grep -F
eba19b9
@echoix
utils(dep_tree2sql): Double quoting some variables
71516a1
@echoix
utils(dep_tree2sql): Replace egrep with grep -E
3f71f1b
@echoix
utils(dep_tree2sql): Use read -r to prevent backslash interpretation
f42f5bf
@echoix
utils(dep_tree2sql): Use 'find .. -print0 | xargs -0 ..' to allow non…
dec5bf7 
…-alphanumeric filenames
@echoix
utils(dep_tree2sql): Replace find * with find ./*
8221bc0 
This has a change in the paths returned, as they start with ./ instead
@echoix
utils(dep_tree2sql): Replace find -perm +111 with find -perm /a+x
fe9d7e4 
-perm +mode was deprecated in favor of -perm /mode. It checks if any of the permissions bits in mode is set. It is thus replaced by the equivalent check of "any file that has the executable bit set for anybody"
@echoix
utils(dep_tree2sql): Use mapfile to assign libs variable with an arra…
e229ac4 
…y, that can be properly quoted in the call below
@echoix
utils(dep_tree2sql): Uniformize shell script formatting
3e509aa 
Using `shfmt -w -s -i 4 -ci -bn -sr`
@echoix
Copy link
Member Author

echoix commented Mar 30, 2024

This will need a shell/bash-knowledgeable reviewer.

@echoix echoix mentioned this pull request Mar 31, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
1 participant
@echoix