HTTP/3, Safari, & Google App Engine

Just as an FYI since I wasn’t able to find this information elsewhere…

Our team spent the last ~24h frantically debugging a gnarly issue where iOS Safari was sporadically having trouble fetching information from our Google App Engine backend (using Apollo GraphQL).

The error in question showed “The network connection was lost” in Safari Web Inspector.

We found this issue challenging to debug for a few different reasons:

1. The issue wasn’t consistently reproducible on different devices.
2. The issue was only on Safari iOS (14.2.1) and not Safari macOS (10.15.7).
3. Some requests succeeded and some failed. Later it was helpful to dig in and realize it was always the first request that failed.
4. The backend logs showed the server emitting a “200 OK” HTTP response; from the server’s perspective everything seemed fine.
5. Catalina (10.15.7) Safari (14.0.1) Web Inspector showed a failure for the request but did not show the full network logs for the request, instead just showing that the request was a failure (despite showing the response code of “200 OK” and showing the timing information of the request and the response), giving no more information about why Safari iOS considered the request a failure. This was the most infuriating part because if Safari had simply recorded/surfaced the detailed reason for the failure it would have been very helpful. Instead, we were left to guess.

We tried several things to mitigate and understand what might have gone wrong:

• We disabled HTML preconnect instructions.
• We ensured that no special content/ad blockers were installed.
• We reset all iOS network settings and website data/cookies.
• We learned that Safari ignores a response and hangs up on HTTP/2 responses that include a HTTP Keep-Alive header. But curl didn’t show a Keep-Alive header being sent by the server and neither did Safari.

Finally, we found that on the phone having the issues, HTTP/3 was enabled in Safari iOS (in Experimental Settings). Disabling HTTP/3 on Safari iOS fixed the problem.

Things to look into:

• Does nodeJS Express+GraphQL on AppEngine emit a Keep-Alive over HTTP/3 that Safari is choking on?
• Is there some weird compatibility issue between AppEngine and Safari at the HTTP/3 packet level?

Your inputs, ideas, and feedback are welcome. It’s still unclear to me which party is at fault, though Safari certainly could use more detailed error logging…!

[Apple friends; this is filed in Feedback as FB8924034]