Incorrect UTF-8 character display in stdout logs #20

KhAlena · 2022-06-29T12:18:05Z

Russian search results are correct.
When logging to a file, the display of Russian letters is also correct.
But when redirecting query.log to /dev/stdout characters looks like on image:

English letters displayed also correctly.

sanikolaev · 2022-06-29T15:36:35Z

Please provide more details on how to reproduce the issue and make sure there's no locale issue in the environment. So far I can't reproduce it like this:

msnikolaev@dev:~/issue_20$ cat docker-compose.yml
version: '2.2'

services:
  manticore:
    container_name: manticore
    image: manticoresearch/manticore
    restart: always
    ports:
      - 127.0.0.1:9315:9306
    ulimits:
      nproc: 65535
      nofile:
         soft: 65535
         hard: 65535
      memlock:
        soft: -1
        hard: -1
    environment:
      - QUERY_LOG_TO_STDOUT=true
      - searchd_query_log_format=plain
    volumes:
      - ./data:/var/lib/manticore

mysql> create table t(f text);
Query OK, 0 rows affected (0.01 sec)
mysql> select * from t where match('абв');
Empty set (0.00 sec)

snikolaev@dev:~/issue_20$ docker-compose up
Recreating manticore ... done
Attaching to manticore
manticore    | [Wed Jun 29 14:59:23.670 2022] [1] using config file '/etc/manticoresearch/manticore.conf' (9168 chars)...
manticore    | starting daemon version '5.0.2 348514c86@220530 dev' ...
manticore    | listening on all interfaces for mysql, port=9306
manticore    | listening on UNIX socket /var/run/mysqld/mysqld.sock
manticore    | listening on 192.168.240.2:9312 for sphinx and http(s)
manticore    | listening on all interfaces for sphinx and http(s), port=9308
manticore    | Manticore 5.0.2 348514c86@220530 dev
manticore    | Copyright (c) 2001-2016, Andrew Aksyonoff
manticore    | Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
manticore    | Copyright (c) 2017-2022, Manticore Software LTD (https://manticoresearch.com)
manticore    |
manticore    | precaching index 't'
manticore    | accepting connections
manticore    | prereading 1 indexes
manticore    | prereaded 1 indexes in 0.000 sec
manticore    | [Wed Jun 29 14:59:25.970 2022] 0.000 sec 0.000 sec [ext2/0/ext 0 (0,20)] [t] абв

KhAlena · 2022-06-30T12:08:06Z

docker-compose

  # Search Service
  tamaranga-manticore:
    container_name: tamaranga-manticore
    image: tamaranga-manticore:${TAG:-latest}
    build:
      context: ./docker/manticore/
      dockerfile: Dockerfile
    restart: unless-stopped
    tty: true
    ulimits:
      nproc: 65535
      nofile:
        soft: 65535
        hard: 65535
      memlock:
        soft: -1
        hard: -1
    volumes:
      - tamaranga-manticore:/var/lib/manticoresearch
    networks:
      - internal-network
    env_file:
      - .env

Dockerfile

FROM manticoresearch/manticore

ARG RU_PACK_URL=https://repo.manticoresearch.com/repository/morphology/ru.pak.tgz
ARG EN_PACK_URL=https://repo.manticoresearch.com/repository/morphology/en.pak.tgz
# Locales installation
RUN apt-get update
RUN apt-get install -y locales-all locales curl xz-utils manticore-columnar-lib
RUN locale-gen ru_RU.UTF-8

# Copy conf file
COPY manticore.conf /etc/manticoresearch/manticore.conf

# Prepare folder for Index files
RUN mkdir -pv /var/lib/manticoresearch
RUN chmod -R 777 /var/lib/manticoresearch

# Install ru.pack
RUN curl -LJO ${RU_PACK_URL}
RUN tar -xvzf ru.pak.tgz -C /usr/share/manticore
RUN rm ru.pak.tgz

# Install en.pack
RUN curl -LJO ${EN_PACK_URL}
RUN tar -xvzf en.pak.tgz -C /usr/share/manticore
RUN rm en.pak.tgz


CMD ["searchd", "--nodetach", "--config", "/etc/manticoresearch/manticore.conf"]

KhAlena · 2022-06-30T12:12:31Z

environment:
- QUERY_LOG_TO_STDOUT=true
- searchd_query_log_format=plain

with your parameters i have no query logs in stdout

sanikolaev · 2022-07-01T04:26:55Z

I still can't reproduce the issue based on your docker-compose and your Dockerfile. Unfortunately you didn't provide a fully reproducible example, so I had to guess. Please modify this so the problem starts manifesting. Otherwise it's hard/impossible to fix it:

snikolaev@dev:~/issue_20/repro$ cat docker-compose.yml
version: '2.2'
services:
  tamaranga-manticore:
    container_name: tamaranga-manticore
    image: tamaranga-manticore:${TAG:-latest}
    build:
      context: ./docker/manticore/
      dockerfile: Dockerfile
    restart: unless-stopped
    tty: true
    privileged: true
    ulimits:
      nproc: 65535
      nofile:
        soft: 65535
        hard: 65535
      memlock:
        soft: -1
        hard: -1
    volumes:
      - tamaranga-manticore:/var/lib/manticoresearch
    networks:
      - internal-network
    env_file:
      - .env
volumes:
   tamaranga-manticore:
networks:
   internal-network:

snikolaev@dev:~/issue_20/repro$ cat .env
TAG=a

snikolaev@dev:~/issue_20/repro$ ls -la docker
total 12
drwxrwxr-x 3 snikolaev snikolaev 4096 Jul  1 06:00 .
drwxrwxr-x 3 snikolaev snikolaev 4096 Jul  1 06:14 ..
drwxrwxr-x 2 snikolaev snikolaev 4096 Jul  1 06:19 manticore

snikolaev@dev:~/issue_20/repro$ ls -la docker/manticore/
total 16
drwxrwxr-x 2 snikolaev snikolaev 4096 Jul  1 06:19 .
drwxrwxr-x 3 snikolaev snikolaev 4096 Jul  1 06:00 ..
-rw-rw-r-- 1 snikolaev snikolaev  899 Jul  1 06:19 Dockerfile
-rw-rw-r-- 1 snikolaev snikolaev  453 Jul  1 06:18 manticore.conf

snikolaev@dev:~/issue_20/repro$ cat docker/manticore/Dockerfile
FROM manticoresearch/manticore

ARG RU_PACK_URL=https://repo.manticoresearch.com/repository/morphology/ru.pak.tgz
ARG EN_PACK_URL=https://repo.manticoresearch.com/repository/morphology/en.pak.tgz
# Locales installation
RUN apt-get update
RUN apt-get install -y locales-all locales curl xz-utils manticore-columnar-lib
RUN locale-gen ru_RU.UTF-8

# Copy conf file
COPY manticore.conf /etc/manticoresearch/manticore.conf

# Prepare folder for Index files
RUN mkdir -pv /var/lib/manticoresearch
RUN chmod -R 777 /var/lib/manticoresearch

# Install ru.pack
RUN curl -LJO ${RU_PACK_URL}
RUN tar -xvzf ru.pak.tgz -C /usr/share/manticore
RUN rm ru.pak.tgz

# Install en.pack
RUN curl -LJO ${EN_PACK_URL}
RUN tar -xvzf en.pak.tgz -C /usr/share/manticore
RUN rm en.pak.tgz

RUN ln -sf /dev/stdout /var/log/manticore/query.log

CMD ["searchd", "--nodetach", "--config", "/etc/manticoresearch/manticore.conf"]

snikolaev@dev:~/issue_20/repro$ cat docker/manticore/manticore.conf
common {
    plugin_dir = /usr/local/lib/manticore
    lemmatizer_base = /var/lib/manticore/
}

searchd {
    listen = 127.0.0.1:9312
    listen = 127.0.0.1:9306:mysql
    listen = 127.0.0.1:9308:http
    log = /var/log/manticore/searchd.log
    query_log = /var/log/manticore/query.log
    pid_file = /var/run/manticore/searchd.pid
    data_dir = /var/lib/manticore
    query_log_format = plain
    binlog_max_log_size = 1M
    max_open_files = max
}

build:

snikolaev@dev:~/issue_20/repro$ docker-compose build tamaranga-manticore
Building tamaranga-manticore
Step 1/17 : FROM manticoresearch/manticore
 ---> dda3734281a2
Step 2/17 : ARG RU_PACK_URL=https://repo.manticoresearch.com/repository/morphology/ru.pak.tgz
 ---> Using cache
 ---> 47bad5f977b1
Step 3/17 : ARG EN_PACK_URL=https://repo.manticoresearch.com/repository/morphology/en.pak.tgz
 ---> Using cache
 ---> 8d37a8b799f0
Step 4/17 : RUN apt-get update
 ---> Using cache
 ---> 61a7fb504a52
Step 5/17 : RUN apt-get install -y locales-all locales curl xz-utils manticore-columnar-lib
 ---> Using cache
 ---> 5bdbfc630d5b
Step 6/17 : RUN locale-gen ru_RU.UTF-8
 ---> Using cache
 ---> c5b4fc5f2973
Step 7/17 : COPY manticore.conf /etc/manticoresearch/manticore.conf
 ---> Using cache
 ---> 62911604dd72
Step 8/17 : RUN mkdir -pv /var/lib/manticoresearch
 ---> Using cache
 ---> 2cda4ae78be2
Step 9/17 : RUN chmod -R 777 /var/lib/manticoresearch
 ---> Using cache
 ---> e3aa5d3fecbc
Step 10/17 : RUN curl -LJO ${RU_PACK_URL}
 ---> Using cache
 ---> e2a6ab644e09
Step 11/17 : RUN tar -xvzf ru.pak.tgz -C /usr/share/manticore
 ---> Using cache
 ---> 29fb544c7c0e
Step 12/17 : RUN rm ru.pak.tgz
 ---> Using cache
 ---> 726b4d2dcf06
Step 13/17 : RUN curl -LJO ${EN_PACK_URL}
 ---> Using cache
 ---> 0caec4307a6c
Step 14/17 : RUN tar -xvzf en.pak.tgz -C /usr/share/manticore
 ---> Using cache
 ---> 0110a174c015
Step 15/17 : RUN rm en.pak.tgz
 ---> Using cache
 ---> abe86d6a656c
Step 16/17 : RUN ln -sf /dev/stdout /var/log/manticore/query.log
 ---> Using cache
 ---> 94f0405d7b65
Step 17/17 : CMD ["searchd", "--nodetach", "--config", "/etc/manticoresearch/manticore.conf"]
 ---> Using cache
 ---> bdc1b4250ff3
Successfully built bdc1b4250ff3
Successfully tagged tamaranga-manticore:a

up:

snikolaev@dev:~/issue_20/repro$ docker-compose up
Starting tamaranga-manticore ... done
Attaching to tamaranga-manticore
tamaranga-manticore    | Manticore 5.0.2 348514c86@220530 dev (columnar 1.15.4 2fef34e@220522) (secondary 1.15.4 2fef34e@220522)
tamaranga-manticore    | Copyright (c) 2001-2016, Andrew Aksyonoff
tamaranga-manticore    | Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
tamaranga-manticore    | Copyright (c) 2017-2022, Manticore Software LTD (https://manticoresearch.com)
tamaranga-manticore    |
tamaranga-manticore    | [24:12.139] [1] using config file '/etc/manticoresearch/manticore.conf' (454 chars)...
tamaranga-manticore    | [24:12.141] [1] Set max_open_files to 65535 (previous was 65535), hardlimit is 65535.
tamaranga-manticore    | starting daemon version '5.0.2 348514c86@220530 dev (columnar 1.15.4 2fef34e@220522) (secondary 1.15.4 2fef34e@220522)' ...
tamaranga-manticore    | listening on 127.0.0.1:9312 for sphinx and http(s)
tamaranga-manticore    | listening on 127.0.0.1:9306 for mysql
tamaranga-manticore    | listening on 127.0.0.1:9308 for sphinx and http(s)
tamaranga-manticore    | precaching index 't'
tamaranga-manticore    | precached 1 indexes in 0.001 sec
tamaranga-manticore    | accepting connections
tamaranga-manticore    | prereading 1 indexes
tamaranga-manticore    | prereaded 1 indexes in 0.000 sec
tamaranga-manticore    | [Fri Jul  1 04:24:48.887 2022] 0.000 sec 0.000 sec [ext2/0/ext 0 (0,20)] [t] абв

query:

snikolaev@dev:~/issue_20/repro$ docker-compose exec tamaranga-manticore bash
root@6dbb6590beb8:/var/lib/manticore# mysql -P9306 -h0
mysql> create table t(f text);
ERROR 1064 (42000): index 't': CREATE TABLE failed: index 't' already exists
mysql> select * from t where match('абв');

host's locale:

snikolaev@dev:~/issue_20/repro$ locale
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=en_US.UTF-8

sanikolaev added the waiting for reply Waiting for reply from the author of the issue label Jun 29, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Incorrect UTF-8 character display in stdout logs #20

Incorrect UTF-8 character display in stdout logs #20

KhAlena commented Jun 29, 2022

sanikolaev commented Jun 29, 2022

KhAlena commented Jun 30, 2022

KhAlena commented Jun 30, 2022

sanikolaev commented Jul 1, 2022

Incorrect UTF-8 character display in stdout logs #20

Incorrect UTF-8 character display in stdout logs #20

Comments

KhAlena commented Jun 29, 2022

sanikolaev commented Jun 29, 2022

KhAlena commented Jun 30, 2022

KhAlena commented Jun 30, 2022

sanikolaev commented Jul 1, 2022