The unreliability of AF_UNIX datagram sockets

[Ken Brown]

On 4/29/2021 7:05 AM, Corinna Vinschen wrote:
> On Apr 27 11:47, Ken Brown wrote:
>> This is a follow-up to
>>
>>   https://cygwin.com/pipermail/cygwin/2021-April/248383.html
>>
>> I'm attaching a test case slightly simpler than the one posted by the OP in
>> that thread.  This is a client/server scenario, with non-blocking AF_UNIX
>> datagram sockets.  The client writes COUNT messages while the server is
>> playing with his toes.  Then the server reads the messages.
>>
>> If COUNT is too big, the expectation is that the client's sendto call will
>> eventually return EAGAIN.  This is what happens on Linux.  On Cygwin,
>> however, there is never a sendto error; the program ends when recv fails
>> with EAGAIN, indicating that some messages were dropped.
>>
>> I think what's happening is that WSASendTo is silently dropping messages
>> without returning an error.  I guess this is acceptable because of the
>> documented unreliability of AF_INET datagram sockets.  But AF_UNIX datagram
>> sockets are supposed to be reliable.
>>
>> I can't think of anything that Cygwin can do about this (but I would love to
>> be proven wrong).  My real reason for raising the issue is that, as we
>> recently discussed in a different thread, maybe it's time for Cygwin to
>> start using native Windows AF_UNIX sockets.  But then we would still have to
>> come up with our own implementation of AF_UNIX datagram sockets, and it
>> seems that we can't simply use the current implementation.  AFAICT, Mark's
>> suggestion of using message queues is the best idea so far.
>>
>> I'm willing to start working on the switch to native AF_UNIX sockets.  (I'm
>> frankly getting bored with working on the pipe implementation, and this
>            ^^^^^^^^^^^^^
> I not really surprised, Windows pipe semantics are annoying.
> 
>> doesn't really seem like it has much of a future.)  But I'd like to be
>> confident that there's a good solution to the datagram problem before I
>> invest too much time in this.
> 
> Summary of our short discussion on IRC:
> 
> - Switching to SOCK_STREAM under the hood adds the necessary reliabilty
>    but breaks DGRAM message boundaries.
> 
> - There appears to be no way in Winsock to handle send buffer overflow
>    gracefully so that user space knows that messages have been discarded.
>    Strange enoug there's a SIO_ENABLE_CIRCULAR_QUEUEING ioctl, but that
>    just makes things worse, by dropping older messages in favor of the
>    newer ones :-P
> 
> I think it should be possible to switch to STREAM sockets to emulate
> DGRAM semantics.  Our advantage is that this is all local.  For all
> practical purposes there's no chance data gets really lost.  Windows has
> an almost indefinite send buffer.
> 
> If you look at the STREAM as a kind of tunneling layer for getting DGRAM
> messages over the (local) line, the DGRAM content could simply be
> encapsulated in a tunnel packet or frame, basically the same way the
> new, boring AF_UNIX code does it.  A DGRAM message encapsulated in a
> STREAM message always has a header which at least contains the length of
> the actual DGRAM message.  So when the peer reads from the socket, it
> always only reads the header until it's complete.  Then it knows how
> much payload is expected and then it reads until the payload has been
> received.

This should work.  We could even use MSG_PEEK to read the header and then 
MSG_WAITALL to read the whole packet.

I'd be happy to try to implement this.  Do you want to create a branch (maybe 
topic/dgram or something like that) for working on it?

> Ultimately this would even allow to emulate DGRAMs when using native
> Windows AF_UNIX sockets.  Then we'd just have to keep the old code for
> backward compat.

Yep.

> There's just one problem with this entire switch to non-pipes: Sending
> descriptors between peers running under different accounts requires to
> be able to switch the user context.  You need this if the sender is a
> non-admin account to call ImpersonateNamedPipeClient in the receiver.
> So we might need to keep the pipes even if just for the purpose of being
> able to call ImpersonateNamedPipeClient...
> 
> 
> Thoughts?

Sounds great.  Thanks.

Ken