LDAP/Exchange UTF-8 Status Code 500 #2140

e311 · 2018-07-24T07:57:08Z

Infos:

Used Zammad version:2.6.x
Installation method (source, package, ..): DEP
Operating system: ubuntu 18.04
Database + version: psql (PostgreSQL) 10.4 (Ubuntu 10.4-0ubuntu0.18.04)
Elasticsearch version:6.6.1
Browser + version:
Chrome

Expected behavior:

LDPA frame displayed (in config menue System --> Integration-->LDAP) without error
*

Actual behavior:

LDAP frame is not displayed. Frame remains empty with error message
Also API and Monitoring

*
StatusCode 500
{"error":""\xC5" from ASCII-8BIT to UTF-8"}

Steps to reproduce the behavior:

only Configure LDAP.

My Config:
adapter: postgresql
database: zammad
pool: 50
timeout: 5000
encoding: utf8
username: zammad

log/production.log:

/opt/zammad/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.5/lib/active_support/tagged_logging.rb:69:in `block in tagged'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.5/lib/active_support/tagged_logging.rb:26:in `tagged'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.5/lib/active_support/tagged_logging.rb:69:in `tagged'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/railties-5.1.5/lib/rails/rack/logger.rb:24:in `call'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.5/lib/action_dispatch/middleware/remote_ip.rb:79:in `call'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.5/lib/action_dispatch/middleware/request_id.rb:25:in `call'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/rack-2.0.5/lib/rack/method_override.rb:22:in `call'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/rack-2.0.5/lib/rack/runtime.rb:22:in `call'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.5/lib/active_support/cache/strategy/local_cache_middleware.rb:27:in `call'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.5/lib/action_dispatch/middleware/executor.rb:12:in `call'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/actionpack-5.1.5/lib/action_dispatch/middleware/static.rb:125:in `call'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/rack-2.0.5/lib/rack/sendfile.rb:111:in `call'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/railties-5.1.5/lib/rails/engine.rb:522:in `call'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/puma-3.11.0/lib/puma/configuration.rb:225:in `call'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/puma-3.11.0/lib/puma/server.rb:624:in `handle_request'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/puma-3.11.0/lib/puma/server.rb:438:in `process_client'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/puma-3.11.0/lib/puma/server.rb:302:in `block in run'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/puma-3.11.0/lib/puma/thread_pool.rb:120:in `block in spawn_thread'
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
I, [2018-07-24T09:47:49.326279 #12796]  INFO -- : Completed 500 Internal Server Error in 1121ms (Views: 0.3ms | ActiveRecord: 43.9ms)
I, [2018-07-24T09:47:50.981956 #12794]  INFO -- : 2018-07-24T09:47:50+0200: [Worker(host:support pid:12794)] Job Observer::UserDeviceLogJob (id=3437) RUNNING
I, [2018-07-24T09:47:51.081012 #12794]  INFO -- : 2018-07-24T09:47:51+0200: [Worker(host:support pid:12794)] Job Observer::UserDeviceLogJob (id=3437) COMPLETED after 0.0988

Yes I'm sure this is a bug and no feature request or a general question.

The text was updated successfully, but these errors were encountered:

thorsteneckel · 2018-07-24T09:33:01Z

Hi @e311 - could you please provide the upper part of the log, too? This is where the required information is printed out and which is currently missing. Additionally: Could you please describe the steps that you performed to get to the issue? This is currently not clear. Thanks!

e311 · 2018-07-24T10:08:22Z

Hi,
i will try it.
i open the (in admin --> system -->Integarion) LDAP. I start the configuration All fine.
Domain DN User Password.

I used the standart mapping.

Ad the LDAP Config sreen ist look fine.

I press safe. And i get error message.

After i reload the Page the Frame is emty.

and here is the full log file:

log.txt

thorsteneckel · 2018-07-24T10:52:17Z

Hi @e311 - thanks for the comprehensive writeup! The interesting line in the log is this one:

E, [2018-07-24T11:39:30.491795 #973] ERROR -- : "\xC5" from ASCII-8BIT to UTF-8 (Encoding::UndefinedConversionError)
/opt/zammad/vendor/bundle/ruby/2.4.0/gems/activesupport-5.1.5/lib/active_support/core_ext/object/json.rb:36:in `encode'
...
/opt/zammad/app/controllers/application_controller/renders_models.rb:70:in `model_update_render_item'
/opt/zammad/app/controllers/application_controller/renders_models.rb:66:in `model_update_render'
/opt/zammad/app/controllers/settings_controller.rb:31:in `update'

Which basically says that there are specially encoded chars in the LDAP settings which will get/got stored. I can see those in the log, too.

I think this is sufficiently enough to reproduce the issue. Thanks for your support so far. We will fix the issue.

thorsteneckel · 2018-07-27T17:16:09Z

Please note that the Exchange integration is affected, too: #2152

rlue · 2018-08-06T09:53:35Z

Hi @e311,

We've been digging into this issue and are pretty confident we understand what's going on, but we need to ask for your help again to be completely certain.

Step 1: Confirm the cause of the bug

Try the following line in your terminal, in the Zammad directory. If our suspicions are correct, then it should fail (i.e., raise an error and show a traceback):

$ rails r "puts Setting.get('ldap_config').to_json"

If you saw a traceback for the same Encoding::UndefinedConversionError that you got in the initial bug report, then great — onto Step 2! If not, please confirm that you haven't reset your database or altered your LDAP configuration since the last time this bug occurred. If you're still not getting an error, please let us know so that we can continue to investigate.

Step 2: Send us one last bit of crucial information

Run the following command in your terminal:

$ rails r "pp Setting.get('ldap_config').except('bind_pw')" > zammad_2140_debug_output.txt

Then, send the resulting file (zammad_2140_debug_output.txt) to support@zammad.com. Be sure to mention that the message is for Thorsten Eckel, and for bug #2140.

With this, we should be able to get a fix working right away.

Thanks again for all your help!

e311 · 2018-08-07T11:53:26Z

This is the First massege serveradmin@support:/opt/zammad$ rails r "puts Setting.get('ldap_config').to_json" Traceback (most recent call last): 4: from bin/rails:3:in `<main>' 3: from bin/rails:3:in `require_relative' 2: from /opt/zammad/config/boot.rb:3:in `<top (required)>' 1: from /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require' /usr/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require': cannot load such file -- bundler/setup (LoadError) Gesendet: Montag, 06. August 2018 um 11:53 Uhr Von: "Ryan Lue" <notifications@github.com> An: zammad/zammad <zammad@noreply.github.com> Cc: e311 <falk_wantzen@gmx.de>, Mention <mention@noreply.github.com> Betreff: Re: [zammad/zammad] LDAP/Exchange UTF-8 Status Code 500 (#2140) Hi @e311, We've been digging into this issue and are pretty confident we understand what's going on, but we need to ask for your help again to be completely certain. Step 1: Confirm the cause of the bug Try the following line in your terminal, in the Zammad directory. If our suspicions are correct, then it should fail (i.e., raise an error and show a traceback): $ rails r "puts Setting.get('ldap_config').to_json" If you saw a traceback for the same Encoding::UndefinedConversionError that you got in the initial bug report, then great — onto Step 2! If not, please confirm that you haven't reset your database or altered your LDAP configuration since the last time this bug occurred. If you're still not getting an error, please let us know so that we can continue to investigate. Step 2: Send us one last bit of crucial information Run the following command in your terminal: $ rails r "pp Setting.get('ldap_config').except('bind_pw')" > zammad_2140_debug_output.txt Then, send the resulting file (zammad_2140_debug_output.txt) to support@zammad.com. Be sure to mention that the message is for Thorsten Eckel, and for bug #2140. With this, we should be able to get a fix working right away. Thanks again for all your help! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

e311 · 2018-08-08T05:16:52Z

rails r "pp Setting.get('ldap_config').except('bind_pw')" > zammad_2140_debug_output.txt
prodruce a emty file (i can not upload because it is emty)

thorsteneckel · 2018-08-08T06:37:34Z

Hi @e311 - Please try the following:

zammad run rails r "pp Setting.get('ldap_config').except('bind_pw')" > zammad_2140_debug_output.txt

OR:

bundle exec rails r "pp Setting.get('ldap_config').except('bind_pw')" > zammad_2140_debug_output.txt

but should be the first line.

e311 · 2018-08-08T11:19:21Z

I try the first Command:
The same as "Serveradmin" with sudo or with out sudo

Bundel my be not installed?

rlue · 2018-08-09T11:37:22Z

@e311, that's very strange. Let's ignore the second command (bundle exec rails ...) for now — I don't think that's the right path here. Could you maybe try to back up the log file and create a new empty one?

/opt/zammad:# gzip production.log
/opt/zammad:# mv production.log.gz production.log.15.gz
/opt/zammad:# touch production.log
/opt/zammad:# chown zammad.zammad production.log
/opt/zammad:# chmod 0644 production.log
/opt/zammad:# zammad run rails r "pp Setting.get('ldap_config').except('bind_pw')" > zammad_2140_debug_output.txt

Thanks for all your help working through this issue with us.

MrGeneration · 2018-08-09T11:42:06Z

@rlue we did the commands yesterday with @e311 .

THe log has perfectly fine access rights, I double checked that.
I'll send you the ldap-config output in mattermost, in case you don't have it yet.

rlue · 2018-08-09T23:52:26Z

Whoops, you totally sent me the log at the start of the day yesterday. Don't know how I missed it!

rlue · 2018-08-09T23:58:43Z

@e311, I didn't see what I was expecting to see in your debug output. Now that @MrGeneration was able to help you get your zammad run rails r ... commands working, could you try the following again and tell me if it's raising the Encoding::UndefinedConversionError that this issue is for?

$ rails r "puts Setting.get('ldap_config').to_json"

e311 · 2018-08-13T14:12:31Z

I get the Message "Your Ruby version is 2.5.1, but your Gemfile specified 2.4.4"
I sorry it will not work (i try all the thinks that Google say to me)

monotek · 2018-08-13T14:21:48Z

Your Ruby & your Elasticsearch version are not compatible with Zammad.
Use Ruby 2.4.4 and Elasticsearch up to 5.6.

rlue · 2018-08-14T04:07:48Z

@e311 sorry actually, totally my fault. Forgot to add zammad r again. -_-'

I meant

$ zammad run rails r "puts Setting.get('ldap_config').to_json"

e311 · 2018-08-14T05:21:03Z

Ther is the message :

Neues Textdokument (3).txt

hublux · 2018-08-31T09:17:04Z

Hello,

Just to clarify, I'm not sure if this is obvious since I have not seen anybody mentioning this.
As a LDAP connection I use Active Directory with German as Systemlanguage. So I installed a new system and a seperate Domain. The only difference from the previous system where the issue occured is that on the new system the language is set to U.S English instead of German. With this setup there is no error to be found, everything working as expected. So I took my original zammad system where I had the problem and connected it with newly installed Domain and the problem is gone.

Running following on the working setup give me following output:
sudo zammad run rails r "puts Setting.get('ldap_config').to_json"

workingsetup.txt

If I run the command to check the ldap config with the original setup it fails and receive following error:

Hope this helps you.
Regards!

rlue · 2018-08-31T10:32:21Z

Thanks @hublux! We've got a fix in the works thanks to your input; it should be merged to develop shortly!

#2140)

tidet · 2018-09-03T05:13:56Z

After Updating still get the same error. (But without Exchange-setup, only LDAP-Sync)

rlue · 2018-09-03T07:48:20Z

Hi @tidet, thanks for the feedback. Please follow the directions in this comment; your sample data will go a long way in helping us resolve this issue conclusively.

hublux · 2018-09-03T08:03:11Z

Hi,
Just to inform I had the same issue again but the old config did still exist. So please proceed as following if you experince this issue again.

Delete the old config with following command:
sudo RAILS_ENV=production zammad run rails r "Setting.set('ldap_config', {})"

stop zammad service
update zammad
start the zammad service again and try to reconfigure the ldap connection

After this I did not face any issue.

Regards

tidet · 2018-09-03T08:21:08Z

@rlue should I try solution above or stay at current error for debugging purposes?
E-Mail with logfile is out to support@

rlue · 2018-09-03T08:41:07Z

@tidet, now that we have your sample LDAP data, feel free to go ahead and try the above. I did find one value which might still cause problems anyway, but it's probably fine. ~~And whether or not the directions above fix it for you, there'll be another patch in the pipeline coming later today.~~

EDIT: Changed my mind. Since the fix is working for you, I'm going to refrain from modifying the code and adding any unnecessary complexity until it's actually broken for someone.

tidet · 2018-09-03T09:12:23Z

@hublux Fix worked for me, too.

bnalpay · 2018-09-03T11:05:08Z

@hublux fixed as well

tbeitter · 2018-09-05T08:56:25Z

we still face this issue. I did was @hublux wrote, deleted the config, updated zammad (2.6.0-1534939663.6d23dae9.stretch now), restarted zammad and configured our ldap connection. Do I have to manually patch code? Thanks in advance.

I was on holiday, sorry for the late reply here and if I missed anything :)

hublux · 2018-09-05T09:35:30Z

Hi @tbeitter,
If you run this command do you receive any output:
sudo zammad run rails r "puts Setting.get('ldap_config').to_json"

If so check this out:
#2140 (comment)

Regards

tbeitter · 2018-09-05T09:39:35Z

Yes, I get an error similar to yours:

"\xC2" from ASCII-8BIT to UTF-8 (Encoding::UndefinedConversionError)

hublux · 2018-09-05T09:49:36Z

Hi,

I just checked my version of Zammad I updated this morning:

It differs from yours.

Could you please procceed again with these steps:

Delete LDAP config
stop zammad service
update zammad again
start zammad service

Regards

tbeitter · 2018-09-05T10:51:34Z

@hublux that worked for me! Thanks!

rlue · 2018-09-12T00:49:39Z

Big shout out to @hublux for all your help getting other users sorted out. e07f41e now includes a database migration that should automatically clear faulty LDAP configurations so that any users who encounter this issue in the future should no longer have to deal with these manual steps.

thorsteneckel added bug import LDAP needs verification labels Jul 24, 2018

thorsteneckel assigned rlue Jul 24, 2018

thorsteneckel mentioned this issue Jul 27, 2018

Exchange import fails if folder names can't be converted to Unicode #2152

Closed

thorsteneckel changed the title ~~LDAP UTF-8 Staus Code 500~~ LDAP/Exchange UTF-8 Staus Code 500 Jul 27, 2018

rlue added this to To do in OLD Workflow Jul 31, 2018

thorsteneckel changed the title ~~LDAP/Exchange UTF-8 Staus Code 500~~ LDAP/Exchange UTF-8 Status Code 500 Aug 2, 2018

rlue moved this from To do to In progress in OLD Workflow Aug 2, 2018

rlue moved this from In progress to Blocked in OLD Workflow Aug 7, 2018

rlue added waiting for feedback and removed needs verification labels Aug 7, 2018

thorsteneckel added this to the 2.6.1 milestone Aug 31, 2018

thorsteneckel moved this from QA to Ready for merge (develop) in OLD Workflow Aug 31, 2018

zammad-sync closed this as completed in 81a6fc3 Aug 31, 2018

thorsteneckel moved this from Ready for merge (develop) to Merged to develop, pending CI for stable in OLD Workflow Aug 31, 2018

rlue moved this from Merged to develop, pending CI for stable to Done in OLD Workflow Sep 1, 2018

zammad-sync pushed a commit that referenced this issue Sep 1, 2018

Stop converting binary user attributes to text during LDAP setup (fixes

3773449

#2140)

rlue reopened this Sep 3, 2018

rlue moved this from Done to Blocked in OLD Workflow Sep 3, 2018

rlue added the waiting for feedback label Sep 3, 2018

rlue moved this from Blocked to In progress in OLD Workflow Sep 3, 2018

rlue removed the waiting for feedback label Sep 3, 2018

rlue closed this as completed Sep 3, 2018

rlue moved this from In progress to Done in OLD Workflow Sep 3, 2018

zammad-sync pushed a commit that referenced this issue Sep 12, 2018

Automate clearing of broken LDAP configurations (fixes #2140)

355bc82

zammad-sync pushed a commit that referenced this issue Sep 12, 2018

Automate clearing of broken LDAP configurations (fixes #2140)

e07f41e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

LDAP/Exchange UTF-8 Status Code 500 #2140

LDAP/Exchange UTF-8 Status Code 500 #2140

e311 commented Jul 24, 2018 •

edited by thorsteneckel

thorsteneckel commented Jul 24, 2018

e311 commented Jul 24, 2018

thorsteneckel commented Jul 24, 2018

thorsteneckel commented Jul 27, 2018

rlue commented Aug 6, 2018 •

edited

e311 commented Aug 7, 2018 via email

e311 commented Aug 8, 2018

thorsteneckel commented Aug 8, 2018

e311 commented Aug 8, 2018

rlue commented Aug 9, 2018

MrGeneration commented Aug 9, 2018

rlue commented Aug 9, 2018

rlue commented Aug 9, 2018

e311 commented Aug 13, 2018

monotek commented Aug 13, 2018

rlue commented Aug 14, 2018

e311 commented Aug 14, 2018

hublux commented Aug 31, 2018

rlue commented Aug 31, 2018

tidet commented Sep 3, 2018 •

edited

rlue commented Sep 3, 2018

hublux commented Sep 3, 2018

tidet commented Sep 3, 2018

rlue commented Sep 3, 2018 •

edited

tidet commented Sep 3, 2018

bnalpay commented Sep 3, 2018

tbeitter commented Sep 5, 2018 •

edited

hublux commented Sep 5, 2018

tbeitter commented Sep 5, 2018 •

edited

hublux commented Sep 5, 2018

tbeitter commented Sep 5, 2018

rlue commented Sep 12, 2018

LDAP/Exchange UTF-8 Status Code 500 #2140

LDAP/Exchange UTF-8 Status Code 500 #2140

Comments

e311 commented Jul 24, 2018 • edited by thorsteneckel

Infos:

Expected behavior:

Actual behavior:

Steps to reproduce the behavior:

thorsteneckel commented Jul 24, 2018

e311 commented Jul 24, 2018

thorsteneckel commented Jul 24, 2018

thorsteneckel commented Jul 27, 2018

rlue commented Aug 6, 2018 • edited

Step 1: Confirm the cause of the bug

Step 2: Send us one last bit of crucial information

e311 commented Aug 7, 2018 via email

e311 commented Aug 8, 2018

thorsteneckel commented Aug 8, 2018

e311 commented Aug 8, 2018

rlue commented Aug 9, 2018

MrGeneration commented Aug 9, 2018

rlue commented Aug 9, 2018

rlue commented Aug 9, 2018

e311 commented Aug 13, 2018

monotek commented Aug 13, 2018

rlue commented Aug 14, 2018

e311 commented Aug 14, 2018

hublux commented Aug 31, 2018

rlue commented Aug 31, 2018

tidet commented Sep 3, 2018 • edited

rlue commented Sep 3, 2018

hublux commented Sep 3, 2018

tidet commented Sep 3, 2018

rlue commented Sep 3, 2018 • edited

tidet commented Sep 3, 2018

bnalpay commented Sep 3, 2018

tbeitter commented Sep 5, 2018 • edited

hublux commented Sep 5, 2018

tbeitter commented Sep 5, 2018 • edited

hublux commented Sep 5, 2018

tbeitter commented Sep 5, 2018

rlue commented Sep 12, 2018

e311 commented Jul 24, 2018 •

edited by thorsteneckel

rlue commented Aug 6, 2018 •

edited

tidet commented Sep 3, 2018 •

edited

rlue commented Sep 3, 2018 •

edited

tbeitter commented Sep 5, 2018 •

edited

tbeitter commented Sep 5, 2018 •

edited