SMTP authentication does not occur

[barf]

Steps to reproduce

1. Install Nextcloud following documentation.
2. Configure SMTP with authentication in Settings -> Basic Settings
3. Click 'Send email' to test configuration

Please note this is very easy to reproduce and probably the reason so many SMTP issues are floating around.

Expected behaviour

SMTP authentication attempt occurs

Actual behaviour

No SMTP authentication attempt occurs

Server configuration

Operating system:
Ubuntu 18.04

Web server:
Apache

Database:
MariaDB

PHP version:
7.2

Nextcloud version:
nextcloud-14.0.1

Postfix Log excerpt:
Oct 2 00:10:35 mail postfix/smtpd[14736]: NOQUEUE: reject: RCPT from nextcloud.example.com[10.146.234.197]: 554 5.7.1 foo.barf@example.com: Recipient address rejected: SMTP AUTH is required for users under this sender domain; from=nextcloud-outgoing@example.com to=foo.bar@example.com proto=ESMTP helo=<nextcloud.example.com>

[nextcloud-bot]

GitMate.io thinks possibly related issues are #11219 (SMTP ), #5240 (“Authentication Required” prompt after upgrade to 12.0.0 does not work), #3279 (user_external should log error message if authentication over imap does not work), #11085 (Relation does not exist), and #3158 (authentication prompt seems not to work).

[rullzer]

Please note this is very easy to reproduce and probably the reason so many SMTP issues are floating around.

Not so easy as it works for me @MorrisJobke just fine...

Could you try with another e-mail host?

[barf]

Same result on two different Postfix systems.

SMTP authentication is not attempted. I used debug_peer in Postfix to review the SMTP sessions to confirm that no auth is being attempted. No 'AUTH PLAIN' command is sent by the client. The chatter is as follows:

250-mail.example.com
250-PIPELINING
250-SIZE 10240000
250-VRFY
250-ETRN
250-STARTTLS
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
MAIL FROM:<nextcloud-outgoing@example.com>

Per authenticated SMTP, the MAIL FROM: command should not occur at this time -- the AUTH PLAIN command should be sent, followed by the MAIL and RCPT commands.

Can you give me more information about your environment @rullzer ? Do you need any more information from me?

[kesselb]

Could you share php occ config:list?

server/lib/private/Mail/Mailer.php

Lines 253 to 257 in c2ef47e

	if ($this->config->getSystemValue('mail_smtpauth', false)) {
	$transport->setUsername($this->config->getSystemValue('mail_smtpname', ''));
	$transport->setPassword($this->config->getSystemValue('mail_smtppassword', ''));
	$transport->setAuthMode($this->config->getSystemValue('mail_smtpauthtype', 'LOGIN'));
	}

Credentials for smtp are not passed when mail_smtpauth is not true. This would be my starting point for further investigation.

[barf]

Thanks @danielkesselberg -- I checked config.php to assert the SMTP credentials were correct in there.

FWIW unauthenticated SMTP works OK but authenticated is necessary in my environment. Also FYI the LDAP credentials were not redacted in this config dump, so I did that.

{
    "system": {
        "debug": true,
        "mail_smtpdebug": true,
        "app.mail.smtplog.enabled": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "nextcloud.example.com"
        ],  
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "14.0.1.1",
        "overwrite.cli.url": "https:\/\/nextcloud.example.com\/nextcloud",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "htaccess.RewriteBase": "\/nextcloud",
        "ldapIgnoreNamingRules": false,
        "ldapProviderFactory": "OCA\\User_LDAP\\LDAPProviderFactory",
        "mail_smtpmode": "smtp",   
        "mail_smtpauthtype": "PLAIN",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "25",
        "mail_smtpauth": 1,
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 6379
        },
        "mail_from_address": "***REMOVED SENSITIVE VALUE***"
    },
    "apps": {
        "accessibility": {
            "enabled": "yes",
            "installed_version": "1.0.1",
            "types": ""
        },
        "activity": {
            "enabled": "yes",
            "installed_version": "2.7.0",
            "types": "filesystem"
        },
        "backgroundjob": {
            "lastjob": "303"
        },
        "cloud_federation_api": {
            "enabled": "yes",
            "installed_version": "0.0.1",
            "types": "filesystem"
        },
        "comments": {
            "enabled": "yes",
            "installed_version": "1.4.0",
            "types": "logging"
        },
        "core": {
            "backgroundjobs_mode": "cron",
            "installedat": "1538162135.7035",
            "lastcron": "1538497813",
            "lastupdateResult": "[]",
            "lastupdatedat": "1538424426",
            "oc.integritycheck.checker": "[]",
            "public_files": "files_sharing\/public.php",
            "public_webdav": "dav\/appinfo\/v1\/publicwebdav.php",
            "scss.variables": "17469a460e0ff7dc6fbbf67b5806a9c4",
            "vendor": "nextcloud"
        },
        "dav": {
            "enabled": "yes",
            "installed_version": "1.6.0",
            "types": "filesystem"
        },
        "federatedfilesharing": {
            "enabled": "yes",
            "installed_version": "1.4.0",
            "types": ""
        },
        "federation": {
            "enabled": "yes",
            "installed_version": "1.4.0",
            "types": "authentication"
        },
        "files": {
            "cronjob_scan_files": "500",
            "enabled": "yes",
            "installed_version": "1.9.0",
            "types": "filesystem"
        },
        "files_pdfviewer": {
            "enabled": "yes",
            "installed_version": "1.3.2",
            "types": ""
        },
        "files_sharing": {
            "enabled": "yes",
            "installed_version": "1.6.2",
            "types": "filesystem"
        },
        "files_texteditor": {
            "enabled": "yes",
            "installed_version": "2.6.0",
            "types": ""
        },
        "files_trashbin": {
            "enabled": "yes",
            "installed_version": "1.4.1",
            "types": "filesystem,dav"
        },
        "files_versions": {
            "enabled": "yes",
            "installed_version": "1.7.1",
            "types": "filesystem,dav"
        },
        "files_videoplayer": {
            "enabled": "yes",
            "installed_version": "1.3.0",
            "types": ""
        },
        "firstrunwizard": {
            "enabled": "yes",
            "installed_version": "2.3.0",
            "types": "logging"
        },
        "gallery": {
            "enabled": "yes",
            "installed_version": "18.1.0",
            "types": ""
        },
        "logreader": {
            "enabled": "yes",
            "installed_version": "2.0.0",
            "ocsid": "170871",
            "types": ""
        },
        "lookup_server_connector": {
            "enabled": "yes",
            "installed_version": "1.2.0",
            "types": "authentication"
        },
        "mail": {
            "enabled": "yes",
            "installed_version": "0.10.0",
            "types": ""
        },
        "nextcloud_announcements": {
            "enabled": "yes",
            "installed_version": "1.3.0",
            "pub_date": "Sat, 10 Dec 2016 00:00:00 +0100",
            "types": "logging"
        },
        "notifications": {
            "enabled": "yes",
            "installed_version": "2.2.1",
            "types": "logging"
        },
        "oauth2": {
            "enabled": "yes",
            "installed_version": "1.2.1",
            "types": "authentication"
        },
        "password_policy": {
            "enabled": "yes",
            "installed_version": "1.4.0",
            "types": ""
        },
        "provisioning_api": {
            "enabled": "yes",
            "installed_version": "1.4.0",
            "types": "prevent_group_restriction"
        },
        "serverinfo": {
            "enabled": "yes",
            "installed_version": "1.4.0",
            "types": ""
        },
        "sharebymail": {
            "enabled": "yes",
            "installed_version": "1.4.0",
            "types": "filesystem"
        },
        "support": {
            "enabled": "yes",
            "installed_version": "1.0.0",
            "types": ""
        },
        "survey_client": {
            "enabled": "yes",
            "installed_version": "1.2.0",
            "types": ""
        },
        "systemtags": {
            "enabled": "yes",
            "installed_version": "1.4.0",
            "types": "logging"
        },
        "theming": {
            "enabled": "yes",
            "installed_version": "1.5.0",
            "types": "logging"
        },
        "twofactor_backupcodes": {
            "enabled": "yes",
            "installed_version": "1.3.1",
            "types": ""
        },
        "updatenotification": {
            "enabled": "yes",
            "installed_version": "1.4.1",
            "types": "",
            "update_check_errors": "0"
        },
        "user_ldap": {
            "background_sync_interval": "43200",
            "background_sync_offset": "0",
            "background_sync_prefix": "s01",
            "cleanUpJobOffset": "0",
            "enabled": "yes",
            "installed_version": "1.4.0",
            "s01_lastChange": "1538423752",
            "s01has_memberof_filter_support": "0",
            "s01home_folder_naming_rule": "",
            "s01last_jpegPhoto_lookup": "0",
            "s01ldap_agent_password": "xxx",
            "s01ldap_attributes_for_group_search": "",
            "s01ldap_attributes_for_user_search": "",
            "s01ldap_backup_host": "",
            "s01ldap_backup_port": "",
            "s01ldap_base": "domainName=example.com,o=domains,dc=example,dc=com",
            "s01ldap_base_groups": "ou=Users,domainName=example.com,o=domains,dc=example,dc=com",
            "s01ldap_base_users": "ou=Users,domainName=example.com,o=domains,dc=example,dc=com",
            "s01ldap_cache_ttl": "600",
            "s01ldap_configuration_active": "1",
            "s01ldap_default_ppolicy_dn": "",
            "s01ldap_display_name": "cn",
            "s01ldap_dn": "cn=nextcloud,dc=example,dc=com",
            "s01ldap_dynamic_group_member_url": "",
            "s01ldap_email_attr": "mail",
            "s01ldap_experienced_admin": "0",
            "s01ldap_expert_username_attr": "",
            "s01ldap_expert_uuid_group_attr": "",
            "s01ldap_expert_uuid_user_attr": "",
            "s01ldap_gid_number": "gidNumber",
            "s01ldap_group_display_name": "cn",
            "s01ldap_group_filter": "(&(|(objectclass=inetOrgPerson)))",
            "s01ldap_group_filter_mode": "0",
            "s01ldap_group_member_assoc_attribute": "uniqueMember",
            "s01ldap_groupfilter_groups": "",
            "s01ldap_groupfilter_objectclass": "inetOrgPerson",
            "s01ldap_host": "mail.example.com",
            "s01ldap_login_filter": "(&(|(objectclass=inetOrgPerson))(|(mailPrimaryAddress=%uid)(mail=%uid)))",
            "s01ldap_login_filter_mode": "0",
            "s01ldap_loginfilter_attributes": "",
            "s01ldap_loginfilter_email": "1",
            "s01ldap_loginfilter_username": "0",
            "s01ldap_nested_groups": "0",
            "s01ldap_override_main_server": "",
            "s01ldap_paging_size": "500",
            "s01ldap_port": "389",
            "s01ldap_quota_attr": "",
            "s01ldap_quota_def": "",
            "s01ldap_tls": "0",
            "s01ldap_turn_off_cert_check": "0",
            "s01ldap_turn_on_pwd_change": "0",
            "s01ldap_user_avatar_rule": "default",
            "s01ldap_user_display_name_2": "",
            "s01ldap_user_filter_mode": "0",
            "s01ldap_userfilter_groups": "",
            "s01ldap_userfilter_objectclass": "inetOrgPerson",
            "s01ldap_userlist_filter": "(|(objectclass=inetOrgPerson))",
            "s01use_memberof_to_detect_membership": "1",
            "types": "authentication"
        },
        "workflowengine": {
            "enabled": "yes",
            "installed_version": "1.4.0",
            "types": "filesystem"
        }
    }
}

[kesselb]

When I understand http://www.huschi.net/archiv/esmtp-dialog-smtp-auth.html and https://www.fehcom.de/qmail/smtpauth.html##FRAMEWORK correctly your mailserver has to announce auth to client. Could you try with ssl or tls? It's common that mailserver don't allow unprotected auth on port 25.

[barf]

OK this might be the right direction, thanks!
SSL and STARTTLS are not working for me, the SMTP client rejects the certificate (LetsEncrypt).
I'll investigate that issue and close this if adding the LetsEncrypt CA to the certificate store resolves it.

[kesselb]

adding the LetsEncrypt CA to the certificate store resolves

I guess this could work 👍

[barf]

Can you please give me any hints about how to trust a CA?

I tried like this but it's not trusting LetsEncrypt yet:
# sudo -u www-data php /var/www/nextcloud/occ security:certificates:import /tmp/lets-encrypt-x3-cross-signed.pem

The SMTP client is not trusting the LetsEncrypt CA (from tshark):
TLSv1.2 73 Alert (Level: Fatal, Description: Unknown CA)

[kesselb]

I usually import these to the global certificate store. Then it should be available to all processes.

[barf]

Me too, I put the LetsEncrypt CA (PKCS#7) file in the /etc/ssl/certs directory (although it was already there), I confirmed the chain using 'openssl s_client' from the NextCloud LXD container too.

Does this SMTP client library have it's own trust store (like with Java and the JKS)?

[barf]

I got it! This was related to Ubuntu not trusting LetsEncrypt by default.

Thanks for your help @danielkesselberg