ioutil.ReadAll(response.Body) blocks forever - Golang

golang http response body
golang ioutil.readall memory leak
golang request body bytes
golang http request body to bytes
golang print request body
golang read request body
ioutil.readall unexpected eof
ioutil.readall to string
tr := &http.Transport{
    TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
}
client := &http.Client{Transport: tr}
response, err := client.Get(link)
if err != nil {
    fmt.Println(err)
}
defer response.Body.Close()

//block forever at the next line
content, _ = ioutil.ReadAll(response.Body)

The above is my code to read content from a webpage which resides in a loop. I found sometimes the line ioutil.ReadAll(response.Body) will block forever. This happens randomly, however, it almost always happens on this webpage: http://xkcd.com/55 . It's very interesting that when I do curl http://xkcd.com/55, it returns nothing, however, wget http://xkcd.com/55 returns the whole webpage.

ioutil.ReadAll(response.Body) blocks forever - Golang - http - iOS, Println(err) } defer response.Body.Close() //block forever at the next line content, _ = ioutil.ReadAll(response.Body) The above is my code to read content from a  However, it indeed blocked at ioutil.ReadAll (response.Body). What's interesting is that it never blocks at the first request. It will almost always block at the second or third request.

Additionally, avoid read response Body in ReadAll without memory/buffer limits control, example:

googleResponse := GoogleResponse{}
err = json.NewDecoder(io.LimitReader(resp.Body, MAX_MEMORY)).Decode(&googleResponse)
if err != nil {
    return nil, err
}

Read more about it in good blog posts: Crossing Streams: a Love Letter to io.Reader by Jason Moiron ioutil.ReadAll(httpResponse.Body) memory consumption Golang Slices And The Case Of The Missing Memory

Be careful with ioutil.ReadAll in Golang · Haisum's Blog, ReadAll is a useful io utility function for reading all data from a io.Reader until EOF. It's often used to read data such as HTTP response body,  golang : network response from ioutil.ReadAll() is empty, connection reset by peer ioutil.ReadAll(response.Body) blocks forever - Golang. 0. is ioutil.ReadAll

Your code should work as expected. I am guessing, its a network issue. Try setting a higher timeout.

package main

import (
    "crypto/tls"
    "fmt"
    "io/ioutil"
    "net/http"
)

func main() {

    link := "http://xkcd.com/55"

    tr := &http.Transport{
        TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
    }
    client := &http.Client{Transport: tr}
    response, err := client.Get(link)
    if err != nil {
        fmt.Println(err)
    }
    defer response.Body.Close()

    //block forever at the next line
    content, _ := ioutil.ReadAll(response.Body)

    fmt.Println(string(content))

}

Reponse body io.Reader · Issue #411 · valyala/fasthttp · GitHub, How to set custom response body writer? Second) } }) } func main() { go func() { if err := fasthttp. package main import ( "fmt" "io/ioutil" "time" "github.com/​erikdubbelboer/fasthttp" ) func My case is that the client/server will hold the connection for a very long time(like forever). And ReadAll will block. Don't use ioutil.ReadAll() will allocate space in memory for the whole of resp.Body. This will use a lot of memory for a large file and if the file is large enough you might run out of memory. If you want to handle large input then you shouldn't use ioutil.ReadAll(), you should stream the resp.Body to disk by creating a file using os.Create

I probably have found the solution by adding DisableKeepAlives: true, to the `&http.Transport, like this:

tr := &http.Transport{
    TLSClientConfig:   &tls.Config{InsecureSkipVerify: true},
    DisableKeepAlives: true,
}

Since I made this change, I haven't encountered any long blocking, yet. But I'm not 100% sure this is the solution. I will leave the new code running one day or two. If there will be no blocking, I think this problem is solved.

golang, Sending the user data, and waiting at most 100 ms for the response: Println(err​) } for i := 0; i < 8; i++ { go doWork() } select {} // Block forever } err) continue } data, err := ioutil.ReadAll(r.Body) if err != nil { fmt.Println("Something went wrong. You can try asking on golang-nuts@ though and see if anybody has networking advice for you. bradfitz closed this Jul 22, 2015 mikioh changed the title Golang ioutil.ReadAll(resp.Body) results in EOF errors which is caused by TCP window full io/ioutil: ReadAll(resp.Body) results in EOF errors which is caused by TCP window full Jul 23, 2015

src/net/http/serve_test.go, Client{Transport: tr} response, err := client.Get(link) if err != nil { fmt.Println(err) } defer response.Body.Close() //block forever at the next line content, _ := ioutil. How to Block Forever in Go Jun 5, 2016 · 3 minute read · Comments Go Let me count the ways. There seems to be quite a few ways to block forever in Go. This post is part code golf and part practical advice. All of the code below is simply to find a way to block the current goroutine from progressing while still allowing others to continue.

src/net/http/transport_test.go, NewClientConn(conn, nil) 263 for _, vt := range vtests { 264 var r *Response 265 var Errorf("http Get #2: %v", err) 723 } 724 got, err = ioutil.ReadAll(r.Body) 725 r. to block forever on reads (next HTTP 918 // request) that will never happen. The WaitGroup type of sync package, is used to wait for the program to finish all goroutines launched from the main function. It uses a counter that specifies the number of goroutines, and Wait blocks the execution of the program until the WaitGroup counter is zero.

View as plain text, Body.Close() 221 body, err := ioutil.ReadAll(res.Body) 222 if err != nil { 223 t. header, the request blocks sending body until the first response. but checks that we don't recurse forever, and checks that 1594 // Content-Encoding is removed. net/http: don't hang if RemoteAddr() blocks The PROXY protocol is supported by several proxy servers such as haproxy and Amazon ELB. This protocol allows services running behind a proxy to learn the remote address of the actual client connecting to the proxy, by including a single textual line at the beginning of the TCP connection.

Comments
  • Maybe they are redirect: see my download function to follow those, with a JarCookie included: github.com/VonC/senvgo/blob/…
  • I just tested downloading xkcd.com/55, and it works just fine (with my version of http code)
  • Thanks @VonC I will give a try.
  • you should maybe add a timeout for your download so you will not get stuck if a server is indeed slow, and the problem is not within your code.
  • Re: VonC's comment, note that xkcd.com/55 redirects to xkcd.com/55 with the slash, so if this code doesn't work with redirects (I don't actually know), yes, you'd be hosed here.
  • Thanks @Rob, yes, it's a good point. I put in a lot of log and found out the err didn't happen here. However, it indeed blocked at ioutil.ReadAll(response.Body). What's interesting is that it never blocks at the first request. It will almost always block at the second or third request. It looks like the when the server detected multiple requests in a short time, the server treats the second of third incoming connections differently from the first one. I totally understand the server may use some strategy to protect it from fraud. However I hope it also will not block forever.
  • So far it's not blocking yet. However, the longest wait was about ~15 minutes long. I'm wondering how to set a timeout to ioutil.ReadAll(response.Body).
  • It turned out this is not the solution. It blocks forever again. It looks like I still need to find a way to set a timeout on this line ioutil.ReadAll(response.Body).