"localhost: no such host" after 250 connections in Go when using ResponseWriter.Write

I have the following http client/server code:

Server

func main() {

    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        fmt.Println("Req: ", r.URL)
        w.Write([]byte("OK")) // <== PROBLEMATIC LINE
        // w.WriteHeader(200) // Works as expected
    })

    log.Fatal(http.ListenAndServe(":5008", nil))
}

Client

func main() {

    client := &http.Client{}

    for i := 0; i < 500; i++ {
        url := fmt.Sprintf("http://localhost:5008/%02d", i)
        req, _ := http.NewRequest("GET", url, nil)
        _, err := client.Do(req)

        if err != nil {
            fmt.Println("error: ", err)
        } else {
            fmt.Println("success: ", i)
        }

        time.Sleep(10 * time.Millisecond)
    }
}

When I run the client above against the server, then after 250 connections I get the following error from client.Do: error: Get http://localhost:5008/250: dial tcp: lookup localhost: no such host and no more connections will succeed.

If I change the line in server from w.Write([]byte("OK")) ==> w.WriteHeader(200) however then there is no limit to the amount of connections and it works as expected.

What am I missing here?

You are not closing the body. When you do any writes from the server, the connection is left open because the response has not been read yet. When you just WriteHeader, the response is done and the connection can be reused or closed.

To be completely honest, I do not know why leaving open connections causes domain lookups to fail. Based on the fact that 250 is awfully close to the round number 256, I would guess there is an artificial limitation placed by the OS that you are hitting. Perhaps the max FDs allowed is 256? Seem low, but it would explain the problem.

func main() {
    client := &http.Client{}

    for i := 0; i < 500; i++ {
        url := fmt.Sprintf("http://localhost:5008/%02d", i)
        req, _ := http.NewRequest("GET", url, nil)
        resp, err := client.Do(req)

        if err != nil {
            fmt.Println("error: ", err)
        } else {
            fmt.Println("success: ", i)
            resp.Body.Close()
        }

        time.Sleep(10 * time.Millisecond)
    }
}

The application must close the response body on the client as described at the beginning of the net/http package docmentation.

func main() {

  client := &http.Client{}

  for i := 0; i < 500; i++ {
    url := fmt.Sprintf("http://localhost:5008/%02d", i)
    req, _ := http.NewRequest("GET", url, nil)
    resp, err := client.Do(req)
    if err != nil {
        fmt.Println("error: ", err)
    } else {
        resp.Body.Close() // <---- close is required
        fmt.Println("success: ", i)
    }
    time.Sleep(10 * time.Millisecond)
  }
}

If the application does not close the response body, then the underlying network connection may not be closed or returned to the the client's connection pool. In this case, each new request creates a new network connection. The process eventually hits the file descriptor limit and anything that requires a file descriptor will fail. This includes name lookups and opening new connections.

The default limit for number of open file descriptors on OS X is 256. I'd expect the client application to fail just short of that limit.

Because each connection to the server uses a file descriptor on the server, the server may also have reached its file descriptor limit.

The response body has zero length when w.Write([]byte("OK")) is removed from the server code. This triggers an optimization in the client for zero length response bodies where the connection is closed or returned to the pool before the application closes the response body.

Also using MAC OSX having the same problem when doing post request concurrently:

After 250 requests it will have that error;

Using go1.8.3.

The fix for my problem is to close both req and res body:

for i := 0; i < 10; i++ {
    res, err := client.Do(req)
    if err == nil {
        globalCounter.Add(1)
        res.Body.Close()
        req.Body.Close()
        break
    } else {
        log.Println("Error:", err, "retrying...", i)
    }
}

I am using go version go1.9.4 linux/amd64 I tried different ways to solve this problem. Nothing helped but http://craigwickesser.com/2015/01/golang-http-to-many-open-files/

Along with resp.Body.Close(), I had to add req.Header.Set("Connection", "close")

func PrettyPrint(v interface{}) (err error) {
  b, err := json.MarshalIndent(v, "", "  ")
  if err == nil {
    fmt.Println(string(b))
  }
  return
}

func Json(body []byte, v interface{}) error {
  return json.Unmarshal(body, v)
}

func GetRequests(hostname string, path string) []map[string]interface{} {
transport := &http.Transport{
    TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
    Dial: (&net.Dialer{
        Timeout:   0,
        KeepAlive: 0,
    }).Dial,
    TLSHandshakeTimeout: 10 * time.Second,
}
httpClient := &http.Client{Transport: transport}

req, reqErr := http.NewRequest("GET", "https://"+hostname+path, nil)
req.SetBasicAuth("user", "pwd")
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Connection", "close")
if reqErr != nil {
    fmt.Println("error in making request", reqErr)
}

resp, err := httpClient.Do(req)
if err != nil {
    fmt.Println("error in calling request", err)
}
defer resp.Body.Close()

content, err := ioutil.ReadAll(resp.Body)
//  fmt.Println(string(content))
var json_resp_l []map[string]interface{}
Json(content, &json_resp_l)
if len(json_resp_l) == 0 {
    var json_resp map[string]interface{}
    Json(content, &json_resp)
    if json_resp != nil {
        json_resp_l = append(json_resp_l, json_resp)
    }
}
PrettyPrint(json_resp_l)
return json_resp_l
}

func main() {
   GetRequests("server.com", "/path")
}

I think the key reason is you should use a sharing http.Transport for each http.Client, the http.Transport will be pooling the connections and reuse them for better performance.

Comments
  • This is a problem with your local Go setup, I can't reproduce with neither Go version 1.3.3 nor devel +d4904f349bc8.
  • Running on OSX 10.9.5
  • You should report that, seems like a bug.
  • It really doesn't seem that strange that a header is required (all valid HTTP responses include a header) .. although the result could be more graceful. It sounds like something is getting backlogged.
  • I agree, though w.Write also appears in examples in golang.org/pkg/net/http
  • Mac os x -> default limit for opened file descriptor is 256. So that is "artifical limitation" which you mentoned
  • Thanks, didn't know OS X did that. My guess that max FDs was set to 256 seems much more realistic now.
  • I actually close req and res, but still hit the issue again and again. $ulimit -n 256