[erlang-questions] socket_closed_remotely errors with httpc

[jim rosenblum]

In http://erlang.org/pipermail/erlang-questions/2014-February/077788.html and
elsewhere, users observed that httpc would occasionally encounter
`{error,socket_closed_remotely}` when querying a web service of some sort.
The workaround was to use https 1.0, as opposed to 1.1, which is somewhat of
an unsatisfying resolution.

I have this (I believe) same problem and have tried to create a minimal
example that reproduces the issue using a public url to post a simple JSON
body (my 'real' code posts data to a ChicagoBoss server). I am hoping that
someone more qualified than I can take this bit of “research” and maybe
figure out this bug? Alternatively, if someone can nudge me in the right
direction, I *might* be able to take this further


My code is as follows:


-module(httpc_bug).

-export([test/0]).


test()->

httpc:set_options([{max_keep_alive_length, 0},

{max_pipeline_length, 0},

{max_sessions, 0}]),

test_put(0).


test_put(N) ->

Url = "https://posttestserver.com/post.php",

ContentType= "application/json",

Body = <<"{\"type\":\"body type\",\"value\":2}">>,

Header = [],

HttpOptions = [],

Options = [],

case httpc:request(post, {Url, Header, ContentType, Body}, HttpOptions,
Options) of

{ok, {{_V, _Cd, _R}, _RespHeader, B }} ->

io:format("~p[~p]:~p~n”, [self(), N, B]),

test_put(N+1);

{error, Reason} ->

io:format("~p: died after ~p iterations with: ~p. ~n.",

[self(), N, Reason])

end.


The problem manifests itself regardless of the options (httpc:setoptions/1)
used - or not used. I chose the ones that I did to try to eliminate
code-paths involved.

A typical run looks like this:


Erlang/OTP 17 [erts-6.2] [source-aaaefb3] [64-bit] [smp:8:8]
[async-threads:10] [hipe] [kernel-poll:false]


Eshell V6.2 (abort with ^G)

1> inets:start().

ok

2> ssl:start().

ok

3> code:load_file(httpc_bug).

{module,httpc_bug}

4> httpc_bug:test(1).

[<0.15201.0>]

<0.15201.0>[0]:"Successfully dumped 0 post variables.\nView it at
http://www.posttestserver.com/data/2015/03/13/19.21.45157935823\nPost body
was 30 chars long."

<0.15201.0>[1]:"Successfully dumped 0 post variables.\nView it at
http://www.posttestserver.com/data/2015/03/13/19.21.451784952425\nPost body
was 30 chars long."

. SNIP

<0.15201.0>: died after 102 iterations with: socket_closed_remotely.


I just did five runs, four died in under 410 iterations, and the fifth took
809 iterations.


By examining the contents of the http_manager__session_db before each call
to the httpc:request, I am pretty sure that the flow goes:


httpc_manager.erl line 759 — within a handle_request clause

handle_request(Request, State = #state{options = Options}) ->

NewRequest = handle_cookies(generate_request_id(Request), State),

SessionType = session_type(Options),

case select_session(Request#request.method,

Request#request.address,

Request#request.scheme, SessionType, State) of

{ok, HandlerPid} ->

*pipeline_or_keep_alive(NewRequest, HandlerPid, State);*

. SNIP



To

httpc_handler.erl line 323 — within a handle_call where keep_alive requests
are

handled.


I don’t have the sophistication to take this any further. Is this enough
information to enable a more qualified individual to pursue this issue.


Thoughts much appreciated.


JR0

[jim rosenblum]

New work around...

Post by jim rosenblum
httpc:set_options([{max_keep_alive_length, 0},
{max_pipeline_length, 0},
{max_sessions, 0}]),
The problem manifests itself regardless of the options

(httpc:setoptions/1) used - or not used. I chose the
ones that I did to try to eliminate code-paths involved.
The problem disappeared when I add {"connection", "close"} to
the header which:

1) reinforces my belief that the offending code path is the
one outlined in my original post; and,

2) gives an alternative to backing down to http 1.0