Building A Powerful Golang HTTP Client With net/http

In this blog, we’re going to talk about the Go standard library net/http and focusing on building HTTP clients. We will discuss how to send requests, create custom headers, handle timeouts, and understand responses. Along the way, we’ll also talk about some internal details.

>> Read more:

• Understanding Golang Ordered Map with Code Examples
• Instruction For Using Golang Generics With Code Examples
• Detailed Guide for Simplifying Testing with Golang Testify
• An In-Depth Guide for Using Go sync.Map with Code Sample

The net/http Package

The net/http package makes the process of creating HTTP requests and handling HTTP responses. If you need to retrieve data from a URL, one way to go about it is by sending an HTTP GET request.

package main

import (
	"io"
	"log"
	"net/http"
)

func main() {
	response, err := http.Get("https://jsonplaceholder.typicode.com/posts")
	if err != nil {
		log.Fatal(err)
	}
	defer response.Body.Close()

	body, err := io.ReadAll(response.Body)
	if err != nil {
		log.Fatal(err)
	}

	println(response.Status, string(body))
}

Sendind A GET Request

Check out this line: http.Get("<https://jsonplaceholder.typicode.com/posts>"). It’s the easiest way to start a GET request. We just give it a URL, and it fetches the data for us. What we get is twofold: the server’s response *http.Response and an error, in case things don’t go right.

Using http.Get is actually like taking a shortcut. It makes sending a GET request easier by doing the work of setting up a http.Client and then using it to send the request for us.

Hang on, and I’ll walk us through what happens under the hood in a moment.

Avoiding Resource Leaks

When we get a response from the server, it includes the information we want. This comes in the form of an http.Response struct, which has details like response status, headers, and the body of the response.

The response.Body is known as an io.ReadCloser. We can use functions from the io package to read from it and it’s important to close the response body when we’re finished with it to save resources.

We use the defer statement to get this done quickly, it makes sure that the body is closed neatly when the function is all done.

Reading the Response Body with io.ReadAll

Since the body comes as an io.ReadCloser, we need a good method to read it. io.ReadAll is a simple way to pull out data from small responses. It lets us collect all the data from the body at once.

You might see ioutil.ReadAll used in many online guides, but be aware, that approach is now deprecated.

After we get the data from the response.Body, we can start working with it to fit our app. If it’s JSON data, it helps to turn into a Golang struct, but for plain text, changing it to a string can be the best move.

“Sending a GET request is easy, but what about other methods like POST, PUT, DELETE, etc?”

For a POST action, we can move ahead using thehttp.Post(url, contentType, body) method. But for PUT, DELETE, and other web methods, we’ll need to create a http.Request object and then, send it using a http.Client.

But before diving in http.Client, let’s start with how to do a POST action using http.Post with a simple example:

func main() {
    response, err := http.Post("http://example.com", "application/json", strings.NewReader(`{"key": "value"}`))
    if err != nil {
        log.Fatal(err)
    }
    defer response.Body.Close()

    body, err := io.ReadAll(response.Body)
    if err != nil {
        log.Fatal(err)
    }

    println(response.Status, string(body))
}

In this case, remember that the http.Post function needs two extra things besides the URL:

• The content type (“application/json”) is a string that tells what kind of data we’re sending.
• The body, as an io.Reader, is where the data we’re sending comes from.

“Why do we use an io.Reader for the request body, and not send the data straight up?”

Well, the content we’re sending in a request can vary widely. It could be a file, perhaps a very large file, a simple string, or a JSON object, among other possibilities.

The io.Reader interface provides a convenient way to efficiently read data from various sources, allowing us to easily handle different types of request bodies, their sizes, and where they come from.

package io 

type Reader interface {
	Read(p []byte) (n int, err error)
}

HTTP Client

Before we dive into the HTTP client, let’s consider why http.Post() and http.Get() might not be the best choice. Behind it all, these functions use a http.Client object to make requests.

If we peek at the http package source code, we can see what’s happening with http.Get():

func http.Get(url string) (resp *Response, err error) {
	return DefaultClient.Get(url)
}

func (c *Client) DefaultClient.Get(url string) (resp *Response, err error) {
	req, err := NewRequest("GET", url, nil)
	if err != nil {
		return nil, err
	}
	return c.Do(req)
}

http.Get() sets up a new http.Request with http.NewRequest() and then uses http.Client.Do() to send it off. It uses DefaultClient, and that’s typically not what we choose for live apps.

We’re going to move in the same direction, but we’ll create our own http.Client and http.Request, then we’ll add some special “touches” as we go.

“So why pick http.Client? Is it just for actions like PUT, PATCH, DELETE,…?”

There’s more to it than that.

For tasks that need a bit more work, like adding custom headers, setting how long to wait (tiemout), or working with proxies, http.Client is really important. http.Get and http.Post are good for simple things, but for more detailed tasks, they might not have what we need.

I can show you how to use a http.Client to do a POST action. It might seem like a lot at first, but don’t stress, we’ll break it down one step at a time:

func main() {
	client := http.Client{}

	requestData := []byte(`{"title": "foo", "body": "bar", "userId": 1}`)
	request, err := http.NewRequest("POST", "https://jsonplaceholder.typicode.com/posts", bytes.NewBuffer(requestData))
	if err != nil {
		log.Fatal(err)
	}
	request.Header.Set("Content-Type", "application/json")

	response, err := client.Do(request)
	if err != nil {
		log.Fatal(err)
	}
	defer response.Body.Close()

	body, err := io.ReadAll(response.Body)
	if err != nil {
		log.Fatal(err)
	}

	println(string(body))
}

Here’s where we start, and by using http.Client, we can change our request as needed.

Setting Up the Request

First, we prepare our request. We have the body of the request, which is a JSON object with a title, body, and userId. To make a new http.Request, we use http.NewRequest(..) and it needs three things:

• The HTTP method we’re using, like POST, PUT, DELETE.
• The URL of the resource we want to interact with.
• The request body, which should be an io.Reader.

Because the third part wants an io.Reader and not just a string or bytes, we turn our byte slice into an io.Reader with bytes.NewBuffer(). This is for the same reasons we talked about before with the response body.

If it’s easier, we don’t have to write out method names like “GET” or “POST” by hand since the http package gives us things like http.MethodPost, http.MethodPut, http.MethodDelete, which can be really handy.

Let’s look at how to put this into action:

request, err := http.NewRequest(
					http.MethodPost, 
					"https://jsonplaceholder.typicode.com/posts", 
					bytes.NewBuffer(requestData)
)

Once our request is ready, we should let it know what kind of content we are sending, which is a JSON object in this case. We set the “Content-Type” header to “application/json” using request.Header.Set(..).

It might look a little tricky to get the data ready and seems like it could go wrong, but this is just to show how it’s done. Normally, we’d use the json.Marshal function to turn a struct into JSON format, then we use bytes.NewBuffer to change it into an io.Reader:

type Post struct {
    Title  string `json:"title"`
    Body   string `json:"body"`
    UserID int    `json:"userId"`
}

bytes, err := json.Marshal(Post{Title: "foo", Body: "bar", UserID: 1})
if err != nil {
    log.Fatal(err)
}

request, err := http.NewRequest(http.MethodPost, "https://jsonplaceholder.typicode.com/posts", bytes.NewBuffer(bytes))

And that’s how we get everything setup for our POST request.

Sending Off the Request

Now we’re ready to send our request to the server. We use the client.Do(..) to make this happen, pass our http.Request to this method, and it returns a http.Response and any errors we might run into.

We then handle the response just like we talked about before the start of this article.

Customizing the HTTP Client

We can change the http.Client OR the http.Request to shape how HTTP requests act.

When we change the http.Client, every request that comes after will follow the new rules. For example, if we set a 10-second wait time on http.Client A, then every request from Client A will wait up to 10 seconds.

Why Timeouts Matter

Setting how long to wait is key, if the server we’re trying to talk to is slow or doesn’t respond, our app might wait too long. That’s not good for us or for the clients using our app.

Here’s how to set a wait time on an http.Client:

client := http.Client{
	Timeout: 5 * time.Second, // 5 seconds
}

This includes the time to connect, go through any redirects, and get the response; here’s what happens:

1. The moment we start an HTTP request with http.Client, the wait timer begins.
2. If connecting to the server, handling redirects, and getting the response takes longer than this time, the request stops.

“What does the wait time really cover?”

The wait time covers the full journey of the request, which means:

• Looking up the DNS (if we need to).
• Setting up a TCP connection.
• Completing the TLS handshake (for secure HTTPS requests).
• Sending the HTTP request through the connection.
• Waiting to hear back with the HTTP response.
• Getting the response’s headers and body.

Redirect Settings

The http.Client deals with HTTP redirects on its own by default. If we don’t set a CheckRedirect function, the client will handle up to 10 redirects before it stops. This is to avoid redirects that just keep going on and on.

But sometimes, we want to control how redirects are managed, maybe to limit them:

client := http.Client{
    CheckRedirect: func(req *http.Request, via []*http.Request) error {
        if len(via) >= 3 {
            return http.ErrUseLastResponse
        }
        return nil // Follow the redirect
    },
}

With this setup, we tell the client to stop after 3 redirects, let’s discuss what’s happening.

Each time a redirect happens, our CheckRedirect function is called, the function gets two things: req, which is the next request, and via, a list of all the requests that came before.

“Why does this function give back an error?”

When CheckRedirect returns an error, it means “don’t follow the redirect.” Instead, it gives us the last response we got, plus any error that CheckRedirect might have given.

But, if the error is http.ErrUseLastResponse, it’s like a sign. It tells the client to stop looking for more redirects and gives us the last response right away, without treating it as an error.

CookieJar

When we set up a CookieJar in the HTTP client, it takes core of cookies for us. This is really helpful for keeping a session going across many requests.

To get this to work, we give the client a CookieJar. The http.Client then handles the cookies, sending them with our requests and changing them when the server responds:

jar, err := cookiejar.New(nil)
if err != nil { 
	// ...
}

client := &http.Client{
    Jar: jar,
}

This is the basic way to set it up.

If we’re using cookiejar.New(...), sometimes we might need to add a list of public suffixes to avoid security issues, like one domain setting a cookie for another by mistake.

“How does the cookie handling work automatically?”

Imagine we’re signing in and need to keep a session:

• Sign in: we enter our login info, and the server sends back a session cookie.
• CookieJar saves the cookie: It keeps this session cookie without us doing anything.
• More requests: When we make a new request, the CookieJar adds the session cookie to the request headers.
• Server’s answer: The server sees the session cookie, knows we’re still signed in, and might change the cookie.
• CookieJar updates: After getting a response, the CookieJar checks for any cookie changes, making sure our session stays active and current.

How Transport Works

The Transport part inside the http.Client is important for how the client acts. Most of the time, we don’t need to change this unless we want to make very low-level changes.

The client usually uses http.DefaultTransport and this default setting is made for many uses and takes care of things like reusing connections, timing out, and other network settings well for most needs.

But if we must meet specific needs, like choosing a certain TLS version or using a proxy, we would set up a special Transport like so:

client := &http.Client{
    Transport: &http.Transport{
        Proxy: http.ProxyURL(proxyURL),
        TLSClientConfig: &tls.Config{MinVersion: tls.VersionTLS12},
    },
}

HTTP Request

After we’ve learned how to tweak the HTTP client, which changes how all requests it sends work, let’s talk about making changes to individual requests. This is done by adjusting the http.Request object.

Setting headers is a usual thing to do, and we’ve already looked at how to use request.Header.Set for setting the “Content-Type” header.

Let’s go over it again, but this time we’ll add an Authorization header too:

func main() {
	client := http.Client{Timeout: 10 * time.Second}

	requestData := []byte(`{"title": "foo", "body": "bar", "userId": 1}`)
	request, err := http.NewRequest("POST", "https://jsonplaceholder.typicode.com/posts", bytes.NewBuffer(requestData))
	if err != nil {
		log.Fatal(err)
	}
	request.Header.Set("Content-Type", "application/json")
	request.Header.Set("Authorization", "Bearer your-token")

	// ...
}

Now, let’s think about this: we don’t have a direct way to set a timeout for just one request. Let’s say the client usually waits for 10 seconds, but we want it to wait only 5 seconds for a specific request. Can we do that?

Absolutely, we can set that up, but it requires a neat little trick. We use http.NewRequestWithContext to make it happen. Here's the method:

ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()

request, err := http.NewRequestWithContext(ctx, "POST", "https://...", bytes.NewBuffer(requestData))

By choosing http.NewRequestWithContext instead of just http.NewRequest, we’re giving it a context as the first thing. The rest of what we put in is the same as withhttp.NewRequest.

“What if the HTTP client and the request each have different timeouts?”

Well, the request’s timeout doesn’t actually change the client’s timeout. This confusion is pretty common, so let’s straighten it out with an example showing how both the client and the request act when their timeouts end:

client := http.Client{
	Timeout: 10 * time.Second,
}
// context deadline exceeded (Client.Timeout exceeded while awaiting headers)

request, err := http.NewRequestWithContext(ctx, "POST", "...", bytes.NewBuffer(requestData))
// context deadline exceeded

Here’s what we need to know: the request stops as soon as the context’s time is up or the http.Client’s timeout is reached, whichever happens first. Remember, it doesn’t make much sense to set a request timeout that’s longer than the client’s timeout. The request timeout should be shorter and it fine-tunes the timing within the client’s larger timeout frame.

Now, look at the error messages closely, especially the one about Client.Timeout. They show us that, behind the scenes, both the client and the request use a context to keep track of timeouts.

Summary

Just to wrap things up, to send a request or set up an HTTP client in Go language, we only need the built-in net/http package. We don’t need outside libraries. Here are the key points:

• The net/http package makes it easy to create and handle HTTP requests and responses. It’s got built-in methods like http.Get for simple tasks, and http.Client for more control and custom options.
• With http.Client, we can really change how requests work — setting timeouts, managing how redirects are handled, taking care of cookies, and changing transport settings for things like TLS or proxies.
• Always use defer to close response bodies to keep things clean and avoid wasting resources.
• We can control the timeout for a specific request with http.NewRequestWithContext, but that request’s timeout can’t be more than what the client’s timeout is set to.