Request Queue ShouldQueue; server stops responding

Dec 10, 2009 at 1:58 PM

Hi,

I am using this wonderful webserver and am hammering it with Web Service calls, handled by a custom Web Services module I've created.  You're probably wondering why I want to do it like this, but rest assured, there's a reason.

I noticed after a period of time, quantifiable based on the number of concurrent request the web server is handing (throughput, number of requesting clients) the server simply stops fulfilling new http requests (the socket answers but no data), specifically, my modules no longer respond.  However, if you enter a URI not handled by a module the system does come to life, in my case with a 404, for example.

 

Thankfully after hours of trying to "break it" sending continuous web service requests from multiple clients, I managed to replicate the issue with the debugger attached.  It was down to the HttpServer.RequestQueue... My system was so busy that when a new request had come in (inside HttpServer.HttpServer.OnRequest), the _requestQueue.ShouldQueue property returned true.  The current context and request was being enqueued to the _requestQueue, as is expected.  On further investigation, it would appear that the RequestQueue class isn't finished; it's Trigger() body is empty and the ThreadQueue() runner causes an infinite block.  Even tinkering with the block wait period (setting to a number of seconds) the server returned a 404 or similar (can't recall now) for processed queued requests.  So I guess this block of code was never finished, which is kinda fair enough given the scale of the project.

From memory I think my _currentRequestCount and _queue sizes were in the hundreds if not thousands when I broke the process... so clearly _currentRequestCount was way in excess of _maxRequestCount.  Importantly, I managed to resurrect the HTTP Server by clearing the _queue and resetting the _currentRequestCount.

My workaround is as follows (in HttpServer.HttpServer.OnRequest):

if (_requestQueue.ShouldQueue)// Return an HTTP Error code 503 Service Unavailable
   // TODO: log an error
   context.Respond("HTTP/1.0", HttpStatusCode.ServiceUnavailable, HttpStatusCode.ServiceUnavailable.ToString(), "RequestQueue is full", "text/plain");
}

There's probably a MUCH better way of doing this (comments?), but at least the caller gets a 503, which is better than nothing.  In my implementation, whenever I send a negative http status code to the sender, I close the connection to the client by setting the connection: close header (and the socket is then dropped for free by supporting code already present).  If you're interested, I implemented this in 'HttpServer.HttpResponse.SendHeaders()' to close on:

            bool statusDropConnection = false;
            if ((Status == HttpStatusCode.BadRequest) ||
                (Status == HttpStatusCode.RequestTimeout) ||
                (Status == HttpStatusCode.LengthRequired) ||
                (Status == HttpStatusCode.RequestEntityTooLarge) ||
                (Status == HttpStatusCode.RequestUriTooLong) ||
                (Status == HttpStatusCode.InternalServerError)||
                (Status == HttpStatusCode.NotImplemented) ||
                (Status == HttpStatusCode.ServiceUnavailable))
                statusDropConnection = true;

            if ((statusDropConnection) || (Connection != ConnectionType.KeepAlive))
                _headers["Connection"] = "close";
            else.......

Whether these codes are accurate close conditions, you decide.  I did have a look at the Apache source and seem to think I followed suit.
I must've done it this way for a reason, presuably not having access to ConnectionType.Close.

Anyway, hope some or all of this comes in useful.

Simon

Coordinator
Dec 15, 2009 at 7:25 AM

Yes. I really appreciate it. It will also be nice if you could provide a patch.

 

Dec 16, 2009 at 11:03 AM

I forgot to mention, the _requestQueue.CurrentRequestCount needs to be thread safe for this to work effectively.
As jgauffin quite rightly suggested, a patch is on its way to the patch repo.