Can't open quarantine after upgrade from 1.9.0 to 1.10.1 #108

kryskool · 2018-04-07T11:47:04Z

From @lmfrazao on January 29, 2018 17:18

Impacted versions

Modoboa: 1.10.1
installer used: Yes ?
Webserver: Nginx

OS: centos 7.4

Steps to reproduce

Upgrade from 1.9.0

Current behavior

Can't open quarantine - Internal Error
I have put Debug on, should open the debug log on modoboa ? can't see nothing on log files.

Copied from original issue: modoboa/modoboa#1377

kryskool · 2018-04-07T11:47:06Z

From @fyfe on January 29, 2018 17:22

In settings.py set ADMINS and it'll send error messages to that e-mail address:

# A list of all the people who get code error notifications. When DEBUG=False
# and a view raises an exception, Django will email these people with the full
# exception information.
# See https://docs.djangoproject.com/en/dev/ref/settings/#admins
ADMINS = [('Administrator', 'admin@example.com')]

# The email address that error messages come from, such as those sent to ADMINS
SERVER_EMAIL = 'notifications@example.com'

kryskool · 2018-04-07T11:47:06Z

From @lmfrazao on January 29, 2018 17:37

I have done it, but still no emails with debug information, is there any way I can force another error ?

kryskool · 2018-04-07T11:47:06Z

From @fyfe on January 29, 2018 17:38

Try closing you're browser and reopening the quarantine page.

kryskool · 2018-04-07T11:47:07Z

From @lmfrazao on January 29, 2018 17:43

already did it, cleared cache, and tryed other browsers

kryskool · 2018-04-07T11:47:07Z

From @fyfe on January 29, 2018 17:47

Can you access the quarantine page now or does it still show the Internal Error message?

kryskool · 2018-04-07T11:47:08Z

From @lmfrazao on January 29, 2018 17:48

Jan 29 17:47:00 xxxxxxx uwsgi: xxxxxxxx.xxx [pid: 1919|app: 2|req: 3/8] 192.168.33.8 () {54 vars in 1093 bytes} [Mon Jan 29 17:47:00 2018] GET /quarantine/listing/?sort_order=-date&reset_page=true&_=1517248020399 => generated 18132 bytes in 377 msecs (HTTP/1.1 500) 5 headers in 169 bytes (1 switches on core 0)

Best I can get when I access it .

kryskool · 2018-04-07T11:47:08Z

From @lmfrazao on January 29, 2018 17:50

full lines:
Jan 29 17:49:15 xxxxx uwsgi: xxxxx.xxx [pid: 1919|app: 2|req: 4/18] 192.168.33.8 () {54 vars in 993 bytes} [Mon Jan 29 17:49:15 2018] GET /quarantine/ => generated 9562 byt es in 461 msecs (HTTP/1.1 200) 5 headers in 163 bytes (1 switches on core 0)
Jan 29 17:49:15 xxxxx uwsgi: xxxxx.xxx [pid: 1919|app: 2|req: 5/19] 192.168.33.8 () {52 vars in 890 bytes} [Mon Jan 29 17:49:15 2018] GET /jsi18n/ => generated 3217 bytes i n 100 msecs (HTTP/1.1 200) 5 headers in 154 bytes (1 switches on core 0)
Jan 29 17:49:16 xxxxx uwsgi: xxxxx.xxx [pid: 1919|app: 2|req: 6/20] 192.168.33.8 () {54 vars in 1093 bytes} [Mon Jan 29 17:49:16 2018] GET /quarantine/listing/?sort_order=- date&reset_page=true&_=1517248155949 => generated 18132 bytes in 456 msecs (HTTP/1.1 500) 5 headers in 169 bytes (1 switches on core 0)

kryskool · 2018-04-07T11:47:09Z

From @fyfe on January 29, 2018 17:54

Is it only the quarantine page that isn't working?

kryskool · 2018-04-07T11:47:09Z

From @lmfrazao on January 29, 2018 17:55

yes

kryskool · 2018-04-07T11:47:09Z

From @fyfe on January 29, 2018 17:59

can you try opening /quarantine/listing/?sort_order=-date&reset_page=true you should get a load of JSON.
can you also post your settings.py
which database server (and version) are you using?

kryskool · 2018-04-07T11:47:10Z

From @lmfrazao on January 29, 2018 18:6

I'm using mariadb 5.5.56

Just noticed the database is in latin1, shall I convert it to utf-8 ?

It was working before this update, this "The string that could not be encoded/decoded was: o ve�culo." it's "o veículo"

error:

UnicodeDecodeError at /quarantine/listing/

'utf8' codec can't decode byte 0xed in position 12: invalid continuation byte

Request Method: GET
Request URL: https://xxx.xxx/quarantine/listing/?sort_order=-date&reset_page=true
Django Version: 1.11.9
Exception Type: UnicodeDecodeError
Exception Value:

'utf8' codec can't decode byte 0xed in position 12: invalid continuation byte

Exception Location: /srv/modoboa/env/lib64/python2.7/encodings/utf_8.py in decode, line 16
Python Executable: /usr/sbin/uwsgi
Python Version: 2.7.5
Python Path:

['.',
'',
'/srv/modoboa/env/lib64/python27.zip',
'/srv/modoboa/env/lib64/python2.7',
'/srv/modoboa/env/lib64/python2.7/plat-linux2',
'/srv/modoboa/env/lib64/python2.7/lib-tk',
'/srv/modoboa/env/lib64/python2.7/lib-old',
'/srv/modoboa/env/lib64/python2.7/lib-dynload',
'/usr/lib64/python2.7',
'/usr/lib/python2.7',
'/srv/modoboa/env/lib/python2.7/site-packages']

Server time: Mon, 29 Jan 2018 18:00:14 +0000
Unicode error hint

The string that could not be encoded/decoded was: o ve�culo

kryskool · 2018-04-07T11:47:10Z

From @lmfrazao on January 29, 2018 18:7

Traceback:
Environment:

Request Method: GET
Request URL: https://xxxxxxxx.xx/quarantine/listing/?sort_order=-date&reset_page=true

Django Version: 1.11.9
Python Version: 2.7.5
Installed Applications:
('django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.sites',
'django.contrib.staticfiles',
'reversion',
'ckeditor',
'ckeditor_uploader',
'rest_framework',
'rest_framework.authtoken',
'modoboa',
'modoboa.core',
'modoboa.lib',
'modoboa.admin',
'modoboa.transport',
'modoboa.relaydomains',
'modoboa.limits',
'modoboa.parameters',
'modoboa_amavis',
'modoboa_pdfcredentials',
'modoboa_postfix_autoreply',
'modoboa_sievefilters',
'modoboa_stats',
'modoboa_contacts',
'modoboa_dmarc',
'modoboa_radicale',
'webpack_loader')
Installed Middleware:
('x_forwarded_for.middleware.XForwardedForMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.locale.LocaleMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
'modoboa.core.middleware.LocalConfigMiddleware',
'modoboa.lib.middleware.AjaxLoginRedirect',
'modoboa.lib.middleware.CommonExceptionCatcher',
'modoboa.lib.middleware.RequestCatcherMiddleware')

Traceback:

File "/srv/modoboa/env/lib/python2.7/site-packages/django/core/handlers/exception.py" in inner
41. response = get_response(request)

File "/srv/modoboa/env/lib/python2.7/site-packages/django/core/handlers/base.py" in _get_response
187. response = self.process_exception_by_middleware(e, request)

File "/srv/modoboa/env/lib/python2.7/site-packages/django/core/handlers/base.py" in _get_response
185. response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/srv/modoboa/env/lib/python2.7/site-packages/django/utils/decorators.py" in inner
185. return func(*args, **kwargs)

File "/srv/modoboa/env/lib/python2.7/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
23. return view_func(request, *args, **kwargs)

File "/srv/modoboa/env/lib/python2.7/site-packages/modoboa_amavis/views.py" in _listing
105. context = get_listing_pages(request, connector)

File "/srv/modoboa/env/lib/python2.7/site-packages/modoboa_amavis/views.py" in get_listing_pages
66. email_list += connector.fetch(page.id_start, page.id_stop)

File "/srv/modoboa/env/lib/python2.7/site-packages/modoboa_amavis/sql_connector.py" in fetch
177. "subject": fix_utf8_encoding(qm["mail__subject"]),

File "/srv/modoboa/env/lib/python2.7/site-packages/modoboa_amavis/utils.py" in fix_utf8_encoding
61. return value.encode("raw_unicode_escape").decode("utf-8")

File "/srv/modoboa/env/lib64/python2.7/encodings/utf_8.py" in decode
16. return codecs.utf_8_decode(input, errors, True)

Exception Type: UnicodeDecodeError at /quarantine/listing/
Exception Value: 'utf8' codec can't decode byte 0xed in position 12: invalid continuation byte

kryskool · 2018-04-07T11:47:11Z

From @fyfe on January 29, 2018 18:15

Can you run the following SQL query on the amavis database:

SELECT character_set_name FROM information_schema.`COLUMNS` 
WHERE table_schema = "schemaname"
  AND table_name = "msgs"
  AND column_name = "subject";

mysql -u [username] -p [database] 'SELECT character_set_name, collate_name FROM information_schema.`COLUMNS` WHERE table_schema = "schemaname" AND table_name = "msgs" AND column_name = "subject";'

kryskool · 2018-04-07T11:47:11Z

From @lmfrazao on January 29, 2018 18:26

0 rows returned

kryskool · 2018-04-07T11:47:12Z

From @lmfrazao on January 29, 2018 18:28

in module instance.settings on the error I got this:

AMAVIS_DEFAULT_DATABASE_ENCODING | u'LATIN1'

should be utf-8, since the database is on utf-8 ?

where can I change this ?

kryskool · 2018-04-07T11:47:12Z

From @fyfe on January 29, 2018 18:35

Yes if you're database uses utf-8 encoding add AMAVIS_DEFAULT_DATABASE_ENCODING = 'utf8' to settings.py

Before you change it can you run the next command, there is a deployment check that should checks this is set to the correct value for your database:

python manage.py checks --deploy

If that doesn't work can you run the following SQL on the amavis database

SHOW FULL COLUMNS FROM msgs;

kryskool · 2018-04-07T11:47:13Z

From @lmfrazao on January 29, 2018 18:36

the result is: utf8mb4

kryskool · 2018-04-07T11:47:13Z

From @lmfrazao on January 29, 2018 18:41

I get this:

Traceback (most recent call last):
File "manage.py", line 22, in
execute_from_command_line(sys.argv)
File "/usr/lib64/python2.7/site-packages/django/core/management/init.py", line 367, in execute_from_command_line
utility.execute()
File "/usr/lib64/python2.7/site-packages/django/core/management/init.py", line 341, in execute
django.setup()
File "/usr/lib64/python2.7/site-packages/django/init.py", line 27, in setup
apps.populate(settings.INSTALLED_APPS)
File "/usr/lib64/python2.7/site-packages/django/apps/registry.py", line 85, in populate
app_config = AppConfig.create(entry)
File "/usr/lib64/python2.7/site-packages/django/apps/config.py", line 123, in create
import_module(entry)
File "/usr/lib64/python2.7/importlib/init.py", line 37, in import_module
import(name)
ImportError: No module named transport

kryskool · 2018-04-07T11:47:13Z

From @fyfe on January 29, 2018 18:43

Did you do all the upgrade steps, you're probably missing modoboa.transport from MODOBOA_APPS in settings.py.

kryskool · 2018-04-07T11:47:14Z

From @lmfrazao on January 29, 2018 18:52

I folowed the steps, yes.

I wasn't using modoboa user.

error:

Unknown command: 'checks'

kryskool · 2018-04-07T11:47:14Z

From @fyfe on January 29, 2018 18:54

Sorry that should have been check (no s)

 python manage.py check --deploy

kryskool · 2018-04-07T11:47:15Z

From @lmfrazao on January 29, 2018 18:55

?: (modoboa-amavis.W001) AMAVIS_DEFAULT_DATABASE_ENCODING does not match the character encoding used by the Amavis database.

this should be the error

kryskool · 2018-04-07T11:47:15Z

From @fyfe on January 29, 2018 18:59

AMAVIS_DEFAULT_DATABASE_ENCODING should match the output from this SQL (change the database name if it's not amavis)

SELECT DEFAULT_CHARACTER_SET_NAME
            FROM INFORMATION_SCHEMA.SCHEMATA
            WHERE SCHEMA_NAME = "amavis";

kryskool · 2018-04-07T11:47:15Z

From @lmfrazao on January 29, 2018 19:0

kryskool · 2018-04-07T11:47:16Z

From @lmfrazao on January 29, 2018 19:1

on settings.py:
AMAVIS_DEFAULT_DATABASE_ENCODING = 'utf8'

kryskool · 2018-04-07T11:47:16Z

From @lmfrazao on January 29, 2018 19:2

I have other errors btw, but I think it's shouldn't be related to this:

?: (modoboa-amavis.W001) AMAVIS_DEFAULT_DATABASE_ENCODING does not match the character encoding used by the Amavis database.
HINT: Check your database character encoding and set/update AMAVIS_DEFAULT_DATABASE_ENCODING.
?: (security.W001) You do not have 'django.middleware.security.SecurityMiddleware' in your MIDDLEWARE so the SECURE_HSTS_SECONDS, SECURE_CONTENT_TYPE_NOSNIFF, SECURE_BROWSER_XSS_FILTER, and SECURE_SSL_REDIRECT settings will have no effect.
?: (security.W012) SESSION_COOKIE_SECURE is not set to True. Using a secure-only session cookie makes it more difficult for network traffic sniffers to hijack user sessions.
?: (security.W016) You have 'django.middleware.csrf.CsrfViewMiddleware' in your MIDDLEWARE, but you have not set CSRF_COOKIE_SECURE to True. Using a secure-only CSRF cookie makes it more difficult for network traffic sniffers to steal the CSRF token.
?: (security.W019) You have 'django.middleware.clickjacking.XFrameOptionsMiddleware' in your MIDDLEWARE, but X_FRAME_OPTIONS is not set to 'DENY'. The default is 'SAMEORIGIN', but unless there is a good reason for your site to serve other parts of itself in a frame, you should change it to 'DENY'.

kryskool · 2018-04-07T11:47:16Z

From @fyfe on January 29, 2018 19:8

AMAVIS_DEFAULT_DATABASE_ENCODING = 'utf8' is correct for you're database.

The other warnings are caused because we don't use django.middleware.security.SecurityMiddleware it's dealt with by nginx.

This looks like an issue with python not liking what's in the subject field, can you post the result for the following

SELECT subject FROM msgs

kryskool · 2018-04-07T11:47:17Z

From @lmfrazao on January 29, 2018 19:12

it's a long list with 42k lines

but enconding is fine, the characters appear correctly

kryskool · 2018-04-07T11:47:17Z

From @lmfrazao on January 29, 2018 19:22

just a sample

kryskool · 2018-04-07T11:47:18Z

From @fyfe on January 29, 2018 19:43

I see the problem, unicode emoji, I hate them 😠 The utf8 encoding in MySQL (and MariaDB) doesn't support the full range of unicode character (ie unicode emoji) that's why they added the utf8mb4 encoding.

I'll need to go away and look a this again I might have to revert my original fix for this.
https://github.com/modoboa/modoboa-amavis/blob/master/modoboa_amavis/utils.py#L48

kryskool · 2018-04-07T11:47:18Z

From @fyfe on January 29, 2018 19:51

Ok some quick research it's a python 2 issue, nothing to do with the database encoding.

kryskool · 2018-04-07T11:47:19Z

From @fyfe on January 29, 2018 20:39

@lmfrazao in python could you run the following

>>> import sys
>>> 'UCS4' if sys.maxunicode >= 0x10000 else 'UCS2'

kryskool · 2018-04-07T11:47:19Z

From @lmfrazao on January 30, 2018 0:6

it returns 'UCS4'
same errors with python manage.py check --deploy

kryskool · 2018-04-07T11:47:20Z

From @lmfrazao on January 30, 2018 0:19

oww wait, the error with the check --deploy is gone

but what to do now ?

kryskool · 2018-04-07T11:47:20Z

From @lmfrazao on January 30, 2018 14:17

@fyfe you figured out how to fix this ?

kryskool added help-needed question labels Apr 7, 2018

kryskool mentioned this issue Apr 7, 2018

Can't open quarantine after upgrade from 1.9.0 to 1.10.1 modoboa/modoboa#1377

Closed

Can't open quarantine after upgrade from 1.9.0 to 1.10.1 #108

Can't open quarantine after upgrade from 1.9.0 to 1.10.1 #108

Comments

kryskool commented Apr 7, 2018

Impacted versions

Steps to reproduce

Current behavior

kryskool commented Apr 7, 2018

kryskool commented Apr 7, 2018

kryskool commented Apr 7, 2018

kryskool commented Apr 7, 2018

kryskool commented Apr 7, 2018

kryskool commented Apr 7, 2018

kryskool commented Apr 7, 2018

kryskool commented Apr 7, 2018

kryskool commented Apr 7, 2018

kryskool commented Apr 7, 2018

kryskool commented Apr 7, 2018

kryskool commented Apr 7, 2018

kryskool commented Apr 7, 2018

kryskool commented Apr 7, 2018

kryskool commented Apr 7, 2018

kryskool commented Apr 7, 2018

kryskool commented Apr 7, 2018

kryskool commented Apr 7, 2018

kryskool commented Apr 7, 2018

kryskool commented Apr 7, 2018

kryskool commented Apr 7, 2018

kryskool commented Apr 7, 2018

kryskool commented Apr 7, 2018

kryskool commented Apr 7, 2018

kryskool commented Apr 7, 2018

kryskool commented Apr 7, 2018

kryskool commented Apr 7, 2018

kryskool commented Apr 7, 2018

kryskool commented Apr 7, 2018

kryskool commented Apr 7, 2018

kryskool commented Apr 7, 2018

kryskool commented Apr 7, 2018

kryskool commented Apr 7, 2018

kryskool commented Apr 7, 2018

kryskool commented Apr 7, 2018