Japanese encoding 7-bit iso-2022-jp is processed incorrectly #2498

lsergei · 2019-02-27T09:20:09Z

Infos:

Used Zammad version: latest version 2.9.x
Installation method (source, package, ..): package
Operating system: CentOS 7.1
Database + version: Postgres
Elasticsearch version:
Browser + version: All browsers
Ticket-ID: #1042716 (1047950, 1076276)

Expected behavior:

Normal processing of all japanese language email messages.

Actual behavior:

While processing email messages encoded with 7-bi iso-2022-jp charset, message is decoded incorrectly.

Assuming issue with mail parser.

Other encoding/charset combinations seem to be working just fine.

Steps to reproduce the behavior:

Send email to mailbox connected to Zammad queue in 7bit/iso-2022-jp

As already explained, combination of 7-bit encoding and iso-2022-jp charset in email message results in garbled text in corresponding ticket.

For example, below is plaintext of email:

--_000_SL2P216MB032969A22C6D22226A7B3613EF740SL2P216MB0329KORP_
Content-Type: text/plain; charset="iso-2022-jp"
Content-Transfer-Encoding: quoted-printable

=1B$B$3$N%"%I%l%9$X$N%a%k%^%,$r2r=3D|$7$F$/$@$5$$!#=1B(B

--_000_SL2P216MB032969A22C6D22226A7B3613EF740SL2P216MB0329KORP_
Content-Type: text/html; charset="iso-2022-jp"
Content-Transfer-Encoding: quoted-printable

<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-2022-=
jp">
<style type=3D"text/css" style=3D"display:none;"> P {margin-top:0;margin-bo=
ttom:0;} </style>
</head>
<body dir=3D"ltr">
<div style=3D"font-family: Meiryo, =1B$B%a%$%j%*=1B(B, &quot;Hiragino Sans&=
quot;, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
=1B$B$3$N%"%I%l%9$X$N%a%k%^%,$r2r=3D|$7$F$/$@$5$$!#=1B(B</div>
</body>
</html>

email client will show it as properly formatted HTML message:

このアドレスへのメルマガを解除してください。

This is how Zammad will show the ticket body:

$B$3$N%"%I%l%9$X$N%a%k%^%,$r2r=|$7$F$/$@$5$$!#(B

The text was updated successfully, but these errors were encountered:

thorsteneckel · 2019-02-27T09:22:00Z

Konnichiwa @lsergei - could you please provide a complete example mail that we then can use for our test suite? That would help a lot. Thanks!

lsergei · 2019-02-27T09:25:00Z

Konnichiwa @thorsteneckel, thank you for the fast response.

How you'd like me to forward you a problematic message? Just forwarding it might (and most likely will) replace headers

thorsteneckel · 2019-02-27T09:27:18Z

Usually the mail program has an option to save a raw mail to .eml format. For Apple Mail e.g. it's just File -> Save As. Please make sure to open that file in a regular text editor and remove all sensitive data before uploading it here.

lsergei · 2019-02-27T09:39:07Z

@thorsteneckel thank you for the explanation. Please see the message enclosed.
As GitHub doesn't support uploading of eml files, file extension was changed to txt.

メルマガ解除.txt

MrGeneration · 2019-02-27T10:45:49Z

Domo arigato!
Issue is verified, we'll update this issue as soon as we're ablet to work on it.

thorsteneckel · 2019-07-02T08:41:36Z

So the root cause of this issue is that the dependency we're using for parsing mails does not support ISO 2022 JP encoded mails. There is an additional gem that brings that support to older versions but is not compatible with the (latest) version we're using.
There is a 6 year old, stale and pending Pull Request on which I added a comment to ask for instructions on how to pick up from there. Waiting for feedback...

MrGeneration · 2020-05-27T15:08:15Z

Can we please re-evaluate if there possibly is a working solution by now?

thorsteneckel · 2020-05-27T15:11:40Z

There have been no relevant changes in the mail gem and my last comment is therefore (unfortunately) still valid.

MrGeneration · 2020-05-27T15:19:23Z

Mantas, can you please have a look if there's maybe a solution to work around this without hacking over 7 year old hacked gems? Maybe a conversion would be possible before or so.

thorsteneckel · 2020-06-02T09:37:17Z

Hi @lsergei - the great @mantas found a workaround for the mail gem issue so that we can handle it in our code instead. A sustainable solution in the gem itself would be much nicer but a huge task we can't accomplish right now. However, feel free to test it in the upcoming Zammad 3.4 release in about 2 weeks.

lsergei · 2020-06-02T09:44:46Z

Hi @thorsteneckel and @mantas - thank you, appreciate your efforts! Will test and give you feedback.

Looking forward to new release

hiromayu · 2020-12-28T07:32:32Z

Hi. I'm new at zammad.
I think this problem is not fixed at 3.6.x
my zammad server show similar garbled characters.

MrGeneration · 2020-12-28T07:48:40Z

@hiromayu Would it be possible to receive an original eml file of a email with above encodings that still does fail within Zammad 3.6? You can send this eml to support [at] zammad [dot] com with referencing this issue. We'll have a look then.

Thank you.

hiromayu · 2020-12-28T08:28:25Z

Hi MrGeneration. I send eml file to support .

lsergei · 2020-12-29T00:31:02Z

@hiromayu @MrGeneration this problem is not fixed yet indeed. I kind of gave up (have someone to fix those emails manually)

MrGeneration · 2020-12-31T07:38:29Z

Just as a small follow up:
Thanks to @hiromayu and the sample mail we could pin down the issue.

It's indirectly covering this issue and issue #3096

hiromayu · 2021-01-07T00:44:13Z

Hello. Thank you All.
I think this problem is fixed by applying patch.
https://github.com/kuroda/mail-iso-2022-jp.
my Zammad server handle the mail that I had sent to support perfectly.

lsergei · 2021-01-07T00:48:43Z

@hiromayu thank you for the update! May I know how you applied the patch? Just installed the gem as described in github article?

hiromayu · 2021-01-07T01:19:53Z

I forgot to tell that I could not distinguish the reason that fixing this problem.
I applied patch and updated zammad server.so one of them or both fixed the problem.
I applied the patch by command.
sudo zammad run gem install mail-iso-2022-jp
and reboot.

lsergei · 2021-01-07T01:59:43Z

Thank you @hiromayu . Just installed the gem but supposedly just install wouldn't fix the issue. May I know what did you change in server.so?

hiromayu · 2021-01-07T02:29:07Z

I'm sorry. I didn't do anything to server.so. I just install that patch and update zammad server introduced by ubuntu updater.
if just installing the patch wouldn't work. it's too first to decide problem fixed. I will check my email at several day.

lsergei · 2021-01-07T02:36:10Z

@hiromayu わかりました。I did the same thing and will monitor if the problem is gone.

どもありがとう！

MrGeneration · 2021-01-07T08:08:26Z

Just an important side note: Keep in mind that this action is not update safe.
So if you really want to stick with the solution and bugfix #3096 is not installed on your system we can't stop you.

However, all bugs that are appearing because of this no longer validate for bug fixing.

lsergei · 2021-01-07T08:22:19Z

@MrGeneration from what @hiromayu suggested it is actually update safe since there is no code modification.

Perhaps closing the ticket is too early, but for today I have no issues after installing said gem.

MrGeneration · 2021-01-07T08:24:13Z

You did change the enviroment.
This being said, if you don't have a specific source code installation setup, this action is not update safe.

thorsteneckel · 2021-01-07T09:18:01Z

Hey there! To provide some technical background: Installing a gem has no effect on its own. It won't get loaded without further action/code changes. Therefore I'd suggest to remove the installed gem again to restore a vanilla environment. I suspect that the provided fix of #3096 created 7 days ago resolved your issue @hiromayu 🤔

hiromayu · 2021-01-07T10:05:17Z

Thank you for all. So, update of Zammad was the reason that fix this problem.
I'm sorry for incorrect information.
I will reinstall Zammad server and check the result. and report it's really fixed at several days after.

hiromayu · 2021-01-15T08:34:29Z

Hello. I got another example of this problem.
but, I think small difference before update of zammad.
before update, all of japanese character was garbled. but now subject of mail is correct. only main text of mail is garbled.
What should I do?

MrGeneration · 2021-01-15T08:38:03Z

Please create a new issue.
This is a complex issue and may appear to be the same but also might be of slightly different nature.

MrGeneration added the waiting for feedback label Feb 27, 2019

MrGeneration added bug verified mail processing and removed waiting for feedback labels Feb 27, 2019

MrGeneration added the prioritised by payment label May 10, 2019

thorsteneckel mentioned this issue Jul 2, 2019

Iso 2022 jp - implement charset codec mikel/mail#595

Open

thorsteneckel added third party issue waiting for feedback labels Jul 2, 2019

MrGeneration added the (probably/currently) unfixable 🤯 label May 7, 2020

MrGeneration assigned mantas May 27, 2020

zammad-sync closed this as completed in d64ea29 Jun 2, 2020

thorsteneckel removed the (probably/currently) unfixable 🤯 label Jun 2, 2020

thorsteneckel added this to the 3.4.0 milestone Jun 2, 2020

MrGeneration mentioned this issue Jun 24, 2020

Japanese emails incorrectly converted #3096

Closed

hiromayu mentioned this issue Jan 15, 2021

Japanese character is garbled. #3368

Closed

awh-tokyo mentioned this issue Jun 23, 2023

Japanese emails importing incorrectly #4680

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Japanese encoding 7-bit iso-2022-jp is processed incorrectly #2498

Japanese encoding 7-bit iso-2022-jp is processed incorrectly #2498

lsergei commented Feb 27, 2019 •

edited by MrGeneration

thorsteneckel commented Feb 27, 2019

lsergei commented Feb 27, 2019

thorsteneckel commented Feb 27, 2019

lsergei commented Feb 27, 2019

MrGeneration commented Feb 27, 2019

thorsteneckel commented Jul 2, 2019 •

edited

MrGeneration commented May 27, 2020

thorsteneckel commented May 27, 2020

MrGeneration commented May 27, 2020

thorsteneckel commented Jun 2, 2020

lsergei commented Jun 2, 2020

hiromayu commented Dec 28, 2020

MrGeneration commented Dec 28, 2020

hiromayu commented Dec 28, 2020

lsergei commented Dec 29, 2020

MrGeneration commented Dec 31, 2020

hiromayu commented Jan 7, 2021

lsergei commented Jan 7, 2021

hiromayu commented Jan 7, 2021

lsergei commented Jan 7, 2021

hiromayu commented Jan 7, 2021

lsergei commented Jan 7, 2021

MrGeneration commented Jan 7, 2021

lsergei commented Jan 7, 2021

MrGeneration commented Jan 7, 2021

thorsteneckel commented Jan 7, 2021 •

edited

hiromayu commented Jan 7, 2021

hiromayu commented Jan 15, 2021

MrGeneration commented Jan 15, 2021

Japanese encoding 7-bit iso-2022-jp is processed incorrectly #2498

Japanese encoding 7-bit iso-2022-jp is processed incorrectly #2498

Comments

lsergei commented Feb 27, 2019 • edited by MrGeneration

Infos:

Expected behavior:

Actual behavior:

Steps to reproduce the behavior:

thorsteneckel commented Feb 27, 2019

lsergei commented Feb 27, 2019

thorsteneckel commented Feb 27, 2019

lsergei commented Feb 27, 2019

MrGeneration commented Feb 27, 2019

thorsteneckel commented Jul 2, 2019 • edited

MrGeneration commented May 27, 2020

thorsteneckel commented May 27, 2020

MrGeneration commented May 27, 2020

thorsteneckel commented Jun 2, 2020

lsergei commented Jun 2, 2020

hiromayu commented Dec 28, 2020

MrGeneration commented Dec 28, 2020

hiromayu commented Dec 28, 2020

lsergei commented Dec 29, 2020

MrGeneration commented Dec 31, 2020

hiromayu commented Jan 7, 2021

lsergei commented Jan 7, 2021

hiromayu commented Jan 7, 2021

lsergei commented Jan 7, 2021

hiromayu commented Jan 7, 2021

lsergei commented Jan 7, 2021

MrGeneration commented Jan 7, 2021

lsergei commented Jan 7, 2021

MrGeneration commented Jan 7, 2021

thorsteneckel commented Jan 7, 2021 • edited

hiromayu commented Jan 7, 2021

hiromayu commented Jan 15, 2021

MrGeneration commented Jan 15, 2021

lsergei commented Feb 27, 2019 •

edited by MrGeneration

thorsteneckel commented Jul 2, 2019 •

edited

thorsteneckel commented Jan 7, 2021 •

edited