Some remote smtp servers cannot negotiate TLS with 9front smtpd server

Wed, 7 Oct 2015 23:45:47 EDT
sl@[REDACTED]

A day or two ago gmail stopped forwarding mail to my 9front system. Eventually, I received the following bounce message in my gmail inbox:

This is an automatically generated Delivery Status Notification

THIS IS A WARNING MESSAGE ONLY.

YOU DO NOT NEED TO RESEND YOUR MESSAGE.

Delivery to the following recipient has been delayed:

    
sl@[REDACTED] Message will be retried for 2 more day(s) Technical details of temporary failure: TLS Negotiation failed: generic::failed_precondition: starttls error (0): protocol error

I complained about this on IRC, where mischief helpfully ran a couple of tests.

This is an example of a failure:

mischief@delta ~ $ openssl s_client -host stanleylieber.com -port 587
CONNECTED(00000003)
139661759612560:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:794:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 318 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
---
mischief@delta ~ $ socat - tcp:stanleylieber.com:587
220 9front.org ESMTP
EHLO libcore.so
250-9front.org you are libcore.so
250-ENHANCEDSTATUSCODES
250-STARTTLS
250 AUTH CRAM-MD5
QUIT
221 2.0.0 Successful termination
mischief@delta ~ $ 

And this is an example of a success:

Oct  7 20:18:03 thx1138 smtpd[11752]: smtp-in: New session 7bf509896ee7b80b from host thx1138.mindlock.us [local]
Oct  7 20:18:03 thx1138 smtpd[11752]: smtp-in: Accepted message 16642aea on session 7bf509896ee7b80b: 
from=<mischief@[REDACTED]>,
to=<sl@[REDACTED]>, size=320, ndest=1, proto=ESMTP Oct 7 20:18:03 thx1138 smtpd[11752]: smtp-in: Closing session 7bf509896ee7b80b Oct 7 20:18:03 thx1138 smtpd[11752]: smtp-out: Connecting to smtp+tls://107.191.126.29:25 (ur.inri.net) on session 7bf5098c68156826... Oct 7 20:18:03 thx1138 smtpd[11752]: smtp-out: Connected on session 7bf5098c68156826 Oct 7 20:18:04 thx1138 smtpd[11752]: smtp-out: Started TLS on session 7bf5098c68156826: version=TLSv1/SSLv3, cipher=AES256-SHA256, bits=256 Oct 7 20:18:04 thx1138 smtpd[11752]: smtp-out: Server certificate verification failed on session 7bf5098c68156826 Oct 7 20:18:05 thx1138 smtpd[11752]: smtp-in: New session 7bf5098dbbc8caf8 from host ur.inri.net [107.191.126.29] Oct 7 20:18:05 thx1138 smtpd[11752]: smtp-in: Closing session 7bf5098dbbc8caf8 Oct 7 20:18:06 thx1138 smtpd[11752]: relay: Ok for 16642aea2ca3020a: session=7bf5098c68156826,
from=<mischief@[REDACTED]>,
to=<sl@[REDACTED]>, rcpt=<->, source=192.235.78.212, relay=107.191.126.29 (ur.inri.net), delay=3s, stat=250 2.5.0 sent Oct 7 20:18:17 thx1138 smtpd[11752]: smtp-out: Closing session 7bf5098c68156826: 1 message sent.

In my server logs, I see lots of remote systems successfully negotiating TLS connections and delivering mail to my system. However, over the last couple of days, there are hundreds of failures from gmail that all look like this:

ur Oct  7 23:20:05 TLS start-up failed with mail-lb0-f174.google.com illegal parameter

sl


> A day or two ago gmail stopped forwarding mail to my 9front system. > Eventually, I received the following bounce message in my gmail > inbox: > > This is an automatically generated Delivery Status Notification >
> THIS IS A WARNING MESSAGE ONLY. >
> YOU DO NOT NEED TO RESEND YOUR MESSAGE. >
> Delivery to the following recipient has been delayed: >
>
sl@[REDACTED] >
> Message will be retried for 2 more day(s) >
> Technical details of temporary failure: > TLS Negotiation failed: generic::failed_precondition: starttls error (0): protocol error

Apparently, I’m not alone:

http://serverfault.com/questions/653664/cant-receive-mails-from-gmail

sl


that error isnt very helpfull. it could be that google has removed some cipher suits and now requires ecdh or dhe now, which we do not implement at the server side.

what i’d try is to get a pcap packet capture from the tls conversation. snoopy can write pcap files with the -D flag.

so run:

snoopy -Df ‘tcp(sd=25)’ > /tmp/smtp.pcap

then try to send mail from a gmail account to your system. then we can see what the handshake looks like.

cinap


> that error isnt very helpfull. it could be that google has > removed some cipher suits and now requires ecdh or dhe now, > which we do not implement at the server side. > > what i’d try is to get a pcap packet capture from the tls > conversation. snoopy can write pcap files with the -D flag. > > so run: > > snoopy -Df ‘tcp(sd=25)’ > /tmp/smtp.pcap > > then try to send mail from a gmail account to your system. > then we can see what the handshake looks like.

/n/bugs/open/some_remote_smtp_servers_cannot_negotiate_tls_with_9front_smtpd_server/smtp.pcap

Right now this file can only be read by members of the disk group, bugs.

sl


got it.

wireshark complains about the certificate issuer/subject fields. i’ll investigate.

cinap


ok, its not clear what the problem is. the warning from wireshark about the certificate can be fixed by changing /sys/src/libsec/port/x509.c and generate new cert:

static Elem mkstring(char *s) {

Elem e;

e.tag.class = Universal;

but really not sure if thats the problem.

the pcap doesnt contain any sensitive information, so you could try to send it to someone at google and ask what the problem is. alternatively, you might just disable TLS in the meantime.

cinap


> ok, its not clear what the problem is. the warning from wireshark > about the certificate can be fixed by changing /sys/src/libsec/port/x509.c > and generate new cert: > > static Elem > mkstring(char *s) > { > Elem e; > > e.tag.class = Universal; > – e.tag.num = IA5String; > + e.tag.num = PrintableString; > e.val.tag = VString; > e.val.u.stringval = estrdup(s); > return e; > } > > but really not sure if thats the problem.

I’ll try this tomorrow. However:

> the pcap doesnt contain any sensitive information, so you could > try to send it to someone at google and ask what the problem is. > alternatively, you might just disable TLS in the meantime.

In the meantime I disabled TLS on port 25, but left TLS enabled on port 587. Google is now happily delivering messages to port 25 without any complaint.

Maybe it was stupid all along to enable TLS on port 25, but I’m not aware of this ever causing any problem in the past.

sl


>> ok, its not clear what the problem is. the warning from wireshark >> about the certificate can be fixed by changing /sys/src/libsec/port/x509.c >> and generate new cert: >> >> static Elem >> mkstring(char *s) >> { >> Elem e; >> >> e.tag.class = Universal; >> – e.tag.num = IA5String; >> + e.tag.num = PrintableString; >> e.val.tag = VString; >> e.val.u.stringval = estrdup(s); >> return e; >> } >> >> but really not sure if thats the problem. > > I’ll try this tomorrow. However:

i'v looked into it and its a bit more complicated. the various x509 fields require different string encoding based on the type. i pushed the changes to libsec now. you need to pull, and rebuild libsec and auth/rsa2x509, then generate new .pem file.

cinap


> i'v looked into it and its a bit more complicated. the various x509 fields > require different string encoding based on the type. i pushed the changes > to libsec now. you need to pull, and rebuild libsec and auth/rsa2x509, then > generate new .pem file.

Updated my server with all the latest changes.

Google still fails to negotiate a connection:

ur Oct  9 15:34:04 TLS start-up failed with mail-lb0-f171.google.com illegal parameter

Forensics here:

/n/bugs/open/some_remote_smtp_servers_cannot_negotiate_tls_with_9front_smtpd_server/smtp.pcap.2

sl


> Updated my server with all the latest changes. > > Google still fails to negotiate a connection: > > ur Oct 9 15:34:04 TLS start-up failed with mail-lb0-f171.google.com illegal parameter

ok, now cert is all fine. my next best bet would be that the rc4 cipher suit is a trap. the client might just offer it to to see if we pick it and when we do it aborts because it is expecting us to use ecdhe because we'r tls1.2. this might be some “lets make the internet more safe” campain they just recently implemented or something. but i dont know. its really hard to tell. this is a valid server response as far as i can see.

i wonder if its possible to file a ticket with google to see at least whats going on.

cinap


Date: Fri, 9 Oct 2015 18:13:44 -0400 From: Kurt H Maier
<khm@[REDACTED]> To:
9front-bugs@[REDACTED] Subject: Re: [9front-bugs] Some remote smtp servers cannot negotiate TLS with 9front smtpd server Reply-To:
9front-bugs@[REDACTED]

On Fri, Oct 09, 2015 at 10:17:06PM +0200,
cinap_lenrek@[REDACTED] wrote: > > Updated my server with all the latest changes. > > > > Google still fails to negotiate a connection: > > > > ur Oct 9 15:34:04 TLS start-up failed with mail-lb0-f171.google.com illegal parameter > > ok, now cert is all fine. my next best bet would be that the > rc4 cipher suit is a trap. the client might just offer it to > to see if we pick it and when we do it aborts because it is > expecting us to use ecdhe because we'r tls1.2. this might be > some “lets make the internet more safe” campain they just > recently implemented or something. but i dont know. its really > hard to tell. this is a valid server response as far as i > can see. > > i wonder if its possible to file a ticket with google to see > at least whats going on.

https://googleonlinesecurity.blogspot.com/2015/09/disabling-sslv3-and-rc4.html “we expect to disable both SSLv3 and RC4 support at Google’s frontend servers and, over time, across our products in general, including Chrome, Android, our webcrawlers and our SMTP servers. ”

then again, it says “problems will only occur if you don’t support anything but RC4.”

Do we rely on anything needing rc4? Can we just remove it? RFC7465 tells us to stop using it anyway.

khm


Date: Sat, 10 Oct 2015 01:15:34 +0200 From:
cinap_lenrek@[REDACTED] To:
9front-bugs@[REDACTED] Subject: Re: [9front-bugs] Some remote smtp servers cannot negotiate TLS with 9front smtpd server Reply-To:
9front-bugs@[REDACTED]

pushed. someone test.

— cinap


Date: Fri, 9 Oct 2015 19:54:14 -0400 From:
sl@[REDACTED] To:
9front-bugs@[REDACTED] Subject: Re: [9front-bugs] Some remote smtp servers cannot negotiate TLS with 9front smtpd server

> pushed. someone test.

Same failure. More forensics:

/n/mars2/usr/bugs/open/some_remote_smtp_servers_cannot_negotiate_tls_with_9front_smtpd_server/smtp.pcap.3

sl