Add support for locale and encoding, fix #406

[mscherer]

No description provided.

[centos-ci]

Can one of the admins verify this patch?

[centos-ci]

Can one of the admins verify this patch?

[centos-ci]

Can one of the admins verify this patch?

[mscherer]

Tagged as draft, since the tests are missing.

[hhorak]

Thanks for this contribution. It makes sense to me to have such options.

[hhorak]

[test]

[hhorak]

What I'm thinking about is whether this shouldn't be set earlier, for the whole DB dir, as https://www.postgresql.org/docs/13/multibyte.html says: "The default character set is selected while initializing your PostgreSQL database cluster using initdb".

[hhorak]

@fila43 what do you think?

[mscherer]

oups, seems I pushed the wrong branch and removed my own code...

[mscherer]

12:06:00  + run_locales_test
12:06:00  + local name=pg-test-locales
12:06:00  + DOCKER_ARGS='-e POSTGRESQL_ENCODING=ISO_8859_6 -e POSTGRESQL_LOCALE=C'
12:06:00  + create_container pg-test-locales
12:06:00  + local name=pg-test-locales
12:06:00  + shift
12:06:00  + local 'cargs=-e POSTGRESQL_ENCODING=ISO_8859_6 -e POSTGRESQL_LOCALE=C'
12:06:00  ++ echo '-e POSTGRESQL_ENCODING=ISO_8859_6 -e POSTGRESQL_LOCALE=C'
12:06:00  ++ tr '\n' ' '
12:06:00  + cargs='-e POSTGRESQL_ENCODING=ISO_8859_6 -e POSTGRESQL_LOCALE=C '
12:06:00  + cidfile=/tmp/tmp.oNB18ocbyupostgresql_test_cidfiles/pg-test-locales
12:06:00  + eval 'docker run -e POSTGRESQL_ENCODING=ISO_8859_6 -e POSTGRESQL_LOCALE=C  --cidfile $cidfile -d $IMAGE_NAME "$@"'
12:06:00  ++ docker run -e POSTGRESQL_ENCODING=ISO_8859_6 -e POSTGRESQL_LOCALE=C --cidfile /tmp/tmp.oNB18ocbyupostgresql_test_cidfiles/pg-test-locales -d f31/postgresql:0
12:06:00  7cb14cec8112f6789df61a5cd5a1f1240f0077946da4bc497f421de838334773
12:06:01  ++ cat /tmp/tmp.oNB18ocbyupostgresql_test_cidfiles/pg-test-locales
12:06:01  Created container 7cb14cec8112f6789df61a5cd5a1f1240f0077946da4bc497f421de838334773
12:06:01  + echo 'Created container 7cb14cec8112f6789df61a5cd5a1f1240f0077946da4bc497f421de838334773'
12:06:01  + wait_ready pg-test-locales
12:06:01  ++ get_cid pg-test-locales
12:06:01  ++ local id=pg-test-locales
12:06:01  ++ shift
12:06:01  +++ cat /tmp/tmp.oNB18ocbyupostgresql_test_cidfiles/pg-test-locales
12:06:01  ++ echo 7cb14cec8112f6789df61a5cd5a1f1240f0077946da4bc497f421de838334773
12:06:01  + docker exec 7cb14cec8112f6789df61a5cd5a1f1240f0077946da4bc497f421de838334773 /usr/libexec/check-container
12:06:01  rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:1122: getting init process 7285 start time caused "read /proc/7285/stat: no such process"
12:06:01

I am not sure exactly what to do with that error message

[mscherer]

So using LANG, one can influence the default encoding and collate. However, this doesn't work that well.

I tried to create a database with LANG=C.UTF8, and it set the server encoding correctly, but not LC_COLLATE.

So I guess we need initdb + the createdb options.

[mscherer]

And also, postgresql crash if I set LC_COLLATE to UTF8, so I suspect that using LANG and similar hacks is out of question, as this seems quite fragile and obscure.

[mscherer]

initdb will derive the encoding from the locales (here ). And this code answer a hardcoded SQL_ASCII answer when the locale is C which mean that I can't use LANG only to get what I need for Synapse (encoding = UTF8, locale=C).

[hhorak]

12:06:01 rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:1122: getting init process 7285 start time caused "read /proc/7285/stat: no such process"
I am not sure exactly what to do with that error message

I read this as the container's process ended before the exec was done, which might be what you refer to with "postgresql crash if I set LC_COLLATE".

[mscherer]

So I pushed a fix, but as the CI is manually triggered, if someone could trigger for me, it would help (as I have no idea how to do it locally)

[hhorak]

Thanks.
[test]

[mscherer]

So I fixed my tests (and my code), now it fail on:

ERROR: File /help.1 does not include 'POSTGRESQL\_ADMIN\_PASSWORD'.

It also fail on the same error for the master branch, so I guess that's not my code causing that.

[mscherer]

The issue reported in my last comment is tracked on #421

[phracek]

[test-all]

[phracek]

@mscherer Can you please rebased it against master and also run these two commands:

$ make clean-versions
$ make generate-all

And commit all changes?

[mscherer]

Done

[phracek]

[test-all]

[phracek]

@fila43 @hhorak Please folks, go through this pull request and let me know if everything is fine, you are experts in PostgreSQL then /me.

[phracek]

Personally, I went through your code and did not hit any issue. Let's wait on the tests. Thanks for adding also test for it. Tests are more then welcome.

[mscherer]

I rebased (before rediffing the other PR)