Re: Getstream starts to become unresponsive to HTTP requests after a few days
Hi, Seems I got the copy/paste wrong: wget localhost:8000/test --2009-11-09 18:25:13-- http://localhost:8000/test Resolving localhost... 127.0.0.1 Connecting to localhost|127.0.0.1|:8000... connected. HTTP request sent, awaiting response... lsof does show 900+ connections to port 8000, so it would seem getstream isn't able to detect when someone disconnects from a http stream? (There is at most, 5 concurrent viewers at a time). Thanks, James On Mon, Nov 9, 2009 at 5:48 PM, Florian Lohoff <flo@rfc822.org> wrote:
On Mon, Nov 09, 2009 at 05:45:41PM +0000, James Creese wrote:
Hi,
Thanks for your reply - it's appreciated.
I'll try to get some data for you when it happens again. The behaviour I see is:
Current HTTP connections appear to still work New HTTP connections, say if I use wget:
wget localhost:8000/test --2009-11-09 17:44:06-- http://localhost:8000/test Resolving localhost... 127.0.0.1 Connecting to localhost|127.0.0.1|:8000...
It will then hang at this point, until I restart getstream.
Okay - an lsof for getstream might be interesting too - Probably it has eaten all its filedescriptors - Looks like the connect gets not accepted - strange ...
Flo -- Florian Lohoff flo@rfc822.org "Es ist ein grobes Missverständnis und eine Fehlwahrnehmung, dem Staat im Internet Zensur- und Überwachungsabsichten zu unterstellen." - - Bundesminister Dr. Wolfgang Schäuble -- 10. Juli in Berlin
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux)
iD4DBQFK+FXRUaz2rXW+gJcRAuBfAJ92/+s+7kzMtm9B2T0/zqhetjp18wCUDm10 Nj/yE9Qc+saDuCfCgLes7A== =tOHk -----END PGP SIGNATURE-----
On Mon, Nov 09, 2009 at 07:32:07PM +0000, James Creese wrote:
Hi,
Seems I got the copy/paste wrong:
wget localhost:8000/test --2009-11-09 18:25:13-- http://localhost:8000/test Resolving localhost... 127.0.0.1 Connecting to localhost|127.0.0.1|:8000... connected. HTTP request sent, awaiting response...
lsof does show 900+ connections to port 8000, so it would seem getstream isn't able to detect when someone disconnects from a http stream? (There is at most, 5 concurrent viewers at a time).
Are they ESTABLISHED or something like CLOSE_WAIT or else? When running with debugging you should see the http connection being droped ... Flo -- Florian Lohoff flo@rfc822.org "Es ist ein grobes Missverständnis und eine Fehlwahrnehmung, dem Staat im Internet Zensur- und Überwachungsabsichten zu unterstellen." - - Bundesminister Dr. Wolfgang Schäuble -- 10. Juli in Berlin
Hi, After doing a bit more investigation, I had 15 VLC instances pulling video streams in from Getstream. I added --loop to the VLC command so it would keep reconnecting if getstream goes down. Whilst I had --loop in the command, getstream became unresponsive within 10 minutes. Removing the --loop command seems to stablise getstream again. Therefore I'm curious as to if the http issue is something along the lines of if there's a burst of http requests, getstream can't handle the requests quickly enough and gets hung up trying to deal with them all and thus won't handle new requests? Thanks, James On Mon, Nov 9, 2009 at 8:06 PM, Florian Lohoff <flo@rfc822.org> wrote:
On Mon, Nov 09, 2009 at 07:32:07PM +0000, James Creese wrote:
Hi,
Seems I got the copy/paste wrong:
wget localhost:8000/test --2009-11-09 18:25:13-- http://localhost:8000/test Resolving localhost... 127.0.0.1 Connecting to localhost|127.0.0.1|:8000... connected. HTTP request sent, awaiting response...
lsof does show 900+ connections to port 8000, so it would seem getstream isn't able to detect when someone disconnects from a http stream? (There is at most, 5 concurrent viewers at a time).
Are they ESTABLISHED or something like CLOSE_WAIT or else?
When running with debugging you should see the http connection being droped ...
Flo -- Florian Lohoff flo@rfc822.org "Es ist ein grobes Missverständnis und eine Fehlwahrnehmung, dem Staat im Internet Zensur- und Überwachungsabsichten zu unterstellen." - - Bundesminister Dr. Wolfgang Schäuble -- 10. Juli in Berlin
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux)
iD8DBQFK+HZZUaz2rXW+gJcRAm94AKCujelIKY+yEQadU9YcXTd3An2+zQCgrNy5 H0s3nmE8wpPwL8Ml6rKmHpg= =0upd -----END PGP SIGNATURE-----
Some time ago I found a fd leak in mcgw and forget to test if it exists in getstream too. Check the number of open file descriptors in /proc/PID/fd when getstream becomes unresponsitive (you may hit the default 1024 open files limit). -- .: Jakub Paweł Głazik, .: email & jabber: zytek%nuxi.pl
Hi, It became unresponsive yesterday, so I checked the number of FDs and indeed it was at the 1024 open files limit, so for now I've just increased the number of fds and hopefully this should help until someone is able to fix the problem (I'd fix it but I'm still a novice in C/C++ programming) Best Regards, James 2009/12/24 Jakub Paweł Głazik <zytek-lists@nuxi.pl>
Some time ago I found a fd leak in mcgw and forget to test if it exists in getstream too. Check the number of open file descriptors in /proc/PID/fd when getstream becomes unresponsitive (you may hit the default 1024 open files limit).
-- .: Jakub Paweł Głazik, .: email & jabber: zytek%nuxi.pl
On Fri, Dec 25, 2009 at 10:41:09AM +0000, James Creese wrote:
Hi,
It became unresponsive yesterday, so I checked the number of FDs and indeed it was at the 1024 open files limit, so for now I've just increased the number of fds and hopefully this should help until someone is able to fix the problem (I'd fix it but I'm still a novice in C/C++ programming)
Just had a quick look at the http code - as i remembered you were doing http streaming. The only way to loose fd's by not closing http connections might be due to urls queried which do not exist, together with http keepalive. The 404 handler triggers http_request_end which in turn only drops the connection in case of non keepalive. You might want to try this patch - it simply ignores the keepalive state. The real fix would be to start a timer and drop the connection after some seconds/minutes if no new request arrives. But i'd rather drop my own http implementation and use the libevent http implementation or libmicrohttp or stuff - There should be tons of implementations out there, more complete, more bugfree than mine. diff --git a/libhttp.c b/libhttp.c index 449cd5a..f4702cb 100644 --- a/libhttp.c +++ b/libhttp.c @@ -444,14 +444,7 @@ void http_drop_connection(struct http_connection *hc) { * */ void http_request_end(struct http_connection *hc) { - if (!hc->keepalive) { - http_drop_connection(hc); - } else { - http_request_free(hc); - hc->status=HC_STATUS_HEAD; - hc->url_handler=&http_url_handler_404; - hc->hsize=0; - } + http_drop_connection(hc); } /* Flo -- Florian Lohoff flo@rfc822.org "Es ist ein grobes Missverständnis und eine Fehlwahrnehmung, dem Staat im Internet Zensur- und Überwachungsabsichten zu unterstellen." - - Bundesminister Dr. Wolfgang Schäuble -- 10. Juli in Berlin
IIRC I fixed it in mcgw using this (this is common code with getstream: durex001:/iptv/proxy# diff -u m/mcgw/libhttp.c mcgw/libhttp.c --- m/mcgw/libhttp.c 2009-03-01 17:34:25.000000000 +0100 +++ mcgw/libhttp.c 2009-10-05 16:39:10.000000000 +0200 @@ -429,7 +429,8 @@ bufferevent_free(hc->bev); http_request_free(hc); - + + shutdown(hc->fd, 2); close(hc->fd); evbuffer_free(hc->evb); durex001:/iptv/proxy# diff -u m/mcgw/socket.c mcgw/socket.c --- m/mcgw/socket.c 2009-03-01 19:39:42.000000000 +0100 +++ mcgw/socket.c 2009-10-05 16:39:10.000000000 +0200 @@ -15,6 +15,7 @@ #include "simplebuffer.h" void socket_close(int sock) { + shutdown(sock, 2); close(sock); } -- .: Jakub Paweł Głazik, .: email & jabber: zytek%nuxi.pl
On Mon, Dec 28, 2009 at 04:28:15PM +0100, Jakub Paweł Głazik wrote:
durex001:/iptv/proxy# diff -u m/mcgw/libhttp.c mcgw/libhttp.c --- m/mcgw/libhttp.c 2009-03-01 17:34:25.000000000 +0100 +++ mcgw/libhttp.c 2009-10-05 16:39:10.000000000 +0200 @@ -429,7 +429,8 @@ bufferevent_free(hc->bev);
http_request_free(hc); - + + shutdown(hc->fd, 2); close(hc->fd);
evbuffer_free(hc->evb);
Getstream does not have a function called shutdown.
durex001:/iptv/proxy# diff -u m/mcgw/socket.c mcgw/socket.c --- m/mcgw/socket.c 2009-03-01 19:39:42.000000000 +0100 +++ mcgw/socket.c 2009-10-05 16:39:10.000000000 +0200 @@ -15,6 +15,7 @@ #include "simplebuffer.h"
void socket_close(int sock) { + shutdown(sock, 2); close(sock); }
And this looks like a layering violation. Flo -- Florian Lohoff flo@rfc822.org "Es ist ein grobes Missverständnis und eine Fehlwahrnehmung, dem Staat im Internet Zensur- und Überwachungsabsichten zu unterstellen." - - Bundesminister Dr. Wolfgang Schäuble -- 10. Juli in Berlin
On Mon, Dec 28, 2009 at 09:31:34PM +0100, Florian Lohoff wrote:
evbuffer_free(hc->evb);
Getstream does not have a function called shutdown.
Aeh - sorry - just found the libc shutdown - but what on earth does shutdown do more than closing the socket - The bug was that the filedescriptor still stays around and getstream eats up all filedescriptors which means they are not closed.
durex001:/iptv/proxy# diff -u m/mcgw/socket.c mcgw/socket.c --- m/mcgw/socket.c 2009-03-01 19:39:42.000000000 +0100 +++ mcgw/socket.c 2009-10-05 16:39:10.000000000 +0200 @@ -15,6 +15,7 @@ #include "simplebuffer.h"
void socket_close(int sock) { + shutdown(sock, 2); close(sock); }
And this looks like a layering violation.
I take this back - its an additional signal - but what for? A closed socket is closed, closed, closed - nothing could happen on it. Flo -- Florian Lohoff flo@rfc822.org "Es ist ein grobes Missverständnis und eine Fehlwahrnehmung, dem Staat im Internet Zensur- und Überwachungsabsichten zu unterstellen." - - Bundesminister Dr. Wolfgang Schäuble -- 10. Juli in Berlin
On Mon, Dec 28, 2009 at 09:39:05PM +0100, Florian Lohoff wrote:
On Mon, Dec 28, 2009 at 09:31:34PM +0100, Florian Lohoff wrote:
evbuffer_free(hc->evb);
Getstream does not have a function called shutdown.
Aeh - sorry - just found the libc shutdown - but what on earth does shutdown do more than closing the socket - The bug was that the filedescriptor still stays around and getstream eats up all filedescriptors which means they are not closed.
Okay - google answers a bit but the outcome says we dont need it. The shutdown will send FIN regardless of the filedescriptor state e.g. close. The assumption is that we have multiple references to the same socket so a close will not close as its not the last ref but the socket will stay around. As we are not dealing with forking or dup there can only be one reference so the shutdown should be unnecessary and the close will drop it. Flo -- Florian Lohoff flo@rfc822.org "Es ist ein grobes Missverständnis und eine Fehlwahrnehmung, dem Staat im Internet Zensur- und Überwachungsabsichten zu unterstellen." - - Bundesminister Dr. Wolfgang Schäuble -- 10. Juli in Berlin
participants (3)
-
Florian Lohoff -
Jakub Paweł Głazik -
James Creese