It will only take one hour… (On why programmers suck at estimating and the peril...

(If you are in a hurry and not interested in the details, feel free to jump to Obstacles and speed-ups)

The task was simple: instead of calling a middleware service directly from the browser client, we wanted to proxy the request through our backend application (which also talked to it already). After a quick search and review of a few StackOverflow posts, we had a draft of the solution within an hour. So far so good, but now the troubles came.

First we needed to figure out the URL of our new endpoint. That should be trivial, right? However this was a (normally) complex Spring Boot application so it took a few experiments to get right. I am still not sure where one part of the path comes from. And it also took a few iterations to find out what does the path information in the request look like (do we get the complete path? or are some prefixes removed from it?…) and to remove our local prefix(/myapp/api/middleware-proxy) so that only the middleware path would remain (more on that, sadly, later).

Next we needed how to obtain an authorization token, required to be allowed to talk to the middleware. We already had a RestClient class that handled retrieving this authorization automatically, so it was logical to reuse it - logical and wrong. We wasted about an hour trying to fit it for our purpose, until we gave up and re-implemented the authorization part ourselves within 15 minutes. Moreover this whole thing was actually “nice to have” but not necessary since the client already handled obtaining the token, as we eventually discovered. In effect it was thus a waste of time (given that our purpose wasn’t to build the best solution but to find out whether it was at all feasible). At the root was a wrong assumption. I knew we needed to authorize with the middleware so I just went to implement it, without checking whether it really was necessary, without checking what the client did. It would have sufficed to look carefully at the headers the client was sending.

A major obstacle proved to be authorization between the client and our backend. All our controllers do that so we followed their lead. Later we discovered that the middleware did its own authorization so it might not have been necessary, though more security is always better :-). At least we correctly focused on getting it working in whatever way possible, postponing the consideration of what roles and permissions were needed to access what. Anyway it took a few hours to sort out because of two bugs/limitations in the authorization mechanism, which did not support our use case with a wildcard in the path mapping. After discovering these bugs/limitations and probing unsuccessfully for possible workarounds, we settled on the third, lowest-level authorization mechanism, which finally worked.

All was well and we were finally able to forward the request to the middleware - only to get back 404, which we confirmed after a while was indeed coming from the middleware and not our server. It took a while to discover that we made a typo while copying the RegExp removing our prefix from the request path. We fixed it and finally we were getting through (though getting just another error because of our test call lacked valid data, but it was at least reaching the target service). Time to clean up and refactor a little and then we were ready to integrated this into our frontend and try to proxy a real request. We were hopeful for a moment - until the only response we got back was java.net.SocketException: Connection reset. What the …? Was something wrong with the VPN, proxy setting, ..? After 1-2 hours experimenting and fighting with the tools, we were able to extract the critical piece of code and run it separately to discover that we forgot to update and thus broke (again!) the RegExp during the refactoring. Why did the middleware load balancer shut down the connection instead of returning 404 as it did before I have no idea (well, the path was a little different this time, but still…).

So finally we were able to receive the request from the client, forward it to the middleware and get its response. Unfortunately, the only response that arrived to the client was 500 Internal Server Error - HttpMediaTypeNotAcceptableException: Could not find acceptable representation. What the … again? We got a string from the middleware, returned the string from our method, and copied the correct (as I have verified after some effort) middleware’s Content-Type header, which matched the client’s Accept header. So what was wrong and why was there any conversion involved? I did not ask for or want any! Few hours of debugging Spring internals and searching StackOverflow later, we arrived to marking our method with produces=*/* and registering the built-in StringHttpMessageConverter with Spring so that it was finally able to understand that we could produce what the client asked for and to convert the string we gave it into a string (that must not have been too difficult!).

A little cleanup and - two days later, and some 10 hours over the estimate - we were done. (But, of course, not done done; we still needed to review, merge, deploy, test, release, but let’s be generous and not count that.)

(Sadly, this wasn’t really the end of the story. A while later I discovered that one of the middleware services was returning transfer-encoding: chunked, which resulted in forever pending request in Chrome, curl at least gave me a cryptic error. 1-2 hours later I fixed the headers and all was well again…)

(Update: It actually took yet two days more. The original solution did not support binary content so I just rewrote everything using low-level, reliable constructs. Now I hope it really is the end….)