[Outages-discussion] [outages] Discord: Sever outages and Increased API errors

[Gary Josack]

In our case etcd (and go's net/http) would accept a PUT/POST request from
an HTTP request with no body despite there being a Content-Length header.
This combined with our python client sending headers as a searate packet
from the body allowed this to trigger. We changed the way we use our python
client to send the full request in a single packet where possible and
updated out consumers to handle corrupt entries. We'll also be filing bugs
with the upstreams soon.

On Sun, Mar 22, 2020 at 9:30 AM Carsten Bormann <cabo at tzi.org> wrote:

> On 2020-03-21, at 05:04, Jay Ashworth via Outages <outages at outages.org>
> wrote:
> >
> > Do I correctly understand, then, that the root cause was the broken
> golang library function / routine?
>
> I wouldn’t put it this way.  It seems the root cause was a protocol that
> relied on a TCP connection always completely transporting a message, and
> not checking that transported message for completeness before putting it to
> use.  No library function on the receiving end can check that the sending
> application did not crash in mid-transmission and seemingly did an orderly
> close.
>
> The message for protocol design 101: Do not use a TCP connection as the
> outer envelope, always use something transported inside that connection to
> properly delimit its end.
> (Ironically, the JSON encoding used in this instance does provide such
> delimiting in most ways of using it; apparently it just wasn’t checked here
> before relying on a successful transfer.)
>
> That message has been learned the hard way in multiple spaces; e.g., look
> up how TLS gained the closed_notify “closure alert”.
>
> Grüße, Carsten
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20200322/4ca6ea93/attachment.htm>