Advanced JSON Customization

Chapter 3.5.

By using struct tags, adding whitespace and enveloping the data, we’ve been able to add quite a lot of customization to our JSON responses already. But what happens when these things aren’t enough, and you need the freedom to customize your JSON even more?

To answer this question, we first need to talk some theory about how Go handles JSON encoding behind the scenes. The key thing to understand is this:

When Go is encoding a particular type to JSON, it looks to see if the type has a MarshalJSON() method implemented on it. If it has, then Go will call this method to determine how to encode it.

That language is a bit fuzzy, so let’s be more exact.

Strictly speaking, when Go is encoding a particular type to JSON it looks to see if the type satisfies the json.Marshaler interface, which looks like this:

type Marshaler interface {
    MarshalJSON() ([]byte, error)
}

If the type does satisfy the interface, then Go will call its MarshalJSON() method and use the []byte slice that it returns as the encoded JSON value.

If the type doesn’t have a MarshalJSON() method, then Go will fall back to trying to encode it to JSON based on its own internal set of rules.

So, if we want to customize how something is encoded, all we need to do is implement a MarshalJSON() method on it which returns a custom JSON representation of itself in a []byte slice.

Customizing the Runtime field

To help illustrate this, let’s look at a concrete example in our application.

When our Movie struct is encoded to JSON, the Runtime field (which is an int32 type) is currently formatted as a JSON number. Let’s change this so that it’s encoded as a string with the format "<runtime> mins" instead. Like so:

{
    "id": 123,
    "title": "Casablanca",
    "runtime": "102 mins",      ← This is now a string
    "genres": [
        "drama",
        "romance",
        "war"
    ],
    "version":1
}

There are a few ways we could achieve this, but a clean and simple approach is to create a custom type specifically for the Runtime field, and implement a MarshalJSON() method on this custom type.

To prevent our internal/data/movie.go file from getting cluttered, let’s create a new file to hold the logic for the Runtime type:

$ touch internal/data/runtime.go

And then go ahead and add the following code:

File: internal/data/runtime.go

package data

import (
    "fmt"
    "strconv"
)

// Declare a custom Runtime type, which has the underlying type int32 (the same as our
// Movie struct field).
type Runtime int32

// Implement a MarshalJSON() method on the Runtime type so that it satisfies the 
// json.Marshaler interface. This should return the JSON-encoded value for the movie 
// runtime (in our case, it will return a string in the format "<runtime> mins").
func (r Runtime) MarshalJSON() ([]byte, error) {
    // Generate a string containing the movie runtime in the required format.
    jsonValue := fmt.Sprintf("%d mins", r)

    // Use the strconv.Quote() function on the string to wrap it in double quotes. It 
    // needs to be surrounded by double quotes in order to be a valid *JSON string*.
    quotedJSONValue := strconv.Quote(jsonValue)

    // Convert the quoted string value to a byte slice and return it.
    return []byte(quotedJSONValue), nil
}

There are two things I’d like to emphasize here:

If your MarshalJSON() method returns a JSON string value, like ours does, then you must wrap the string in double quotes before returning it. Otherwise it won’t be interpreted as a JSON string and you’ll receive a runtime error similar to this:
```
json: error calling MarshalJSON for type data.Runtime: invalid character 'm' after top-level value
```
We’re deliberately using a value receiver for our MarshalJSON() method rather than a pointer receiver like func (r *Runtime) MarshalJSON(). This gives us more flexibility because it means that our custom JSON encoding will work on both Runtime values and pointers to Runtime values. As Effective Go mentions:

The rule about pointers vs. values for receivers is that value methods can be invoked on pointers and values, but pointer methods can only be invoked on pointers.

Hint: If you’re not confident about the difference between pointer and value receivers, then this blog post provides a good summary.

OK, now that the custom Runtime type is defined, open your internal/data/movies.go file and update the Movie struct to use it like so:

File: internal/data/movies.go

package data

import (
    "time"
)

type Movie struct {
    ID        int64     `json:"id"`
    CreatedAt time.Time `json:"-"`
    Title     string    `json:"title"`
    Year      int32     `json:"year,omitempty"`
    // Use the Runtime type instead of int32. Note that the omitempty directive will
    // still work on this: if the Runtime field has the underlying value 0, then it will
    // be considered empty and omitted -- and the MarshalJSON() method we just made 
    // won't be called at all.
    Runtime Runtime  `json:"runtime,omitempty"`
    Genres  []string `json:"genres,omitempty"`
    Version int32    `json:"version"`
}

Let’s try this out by restarting the API and making a request to the GET /v1/movies/:id endpoint. You should now see a response containing the custom runtime value in the format "<runtime> mins", similar to this:

$ curl localhost:4000/v1/movies/123
{
    "movie": {
        "id": 123,
        "title": "Casablanca",
        "runtime": "102 mins",
        "genres": [
            "drama",
            "romance",
            "war"
        ],
        "version": 1
    }
}

All in all, this is a pretty nice way to generate custom JSON. Our code is succinct and clear, and we’ve got a custom Runtime type that we can use wherever and whenever we need it.

But there is a downside. It’s important to be aware that using custom types can sometimes be awkward when integrating your code with other packages, and you may need to perform type conversions to change your custom type to and from a value that the other packages understand and accept.

Additional Information

There are a couple of alternative approaches that you could take to reach the same end result here, and I’d like to quickly describe them and talk through their pros and cons. If you’re coding-along with the build, don’t make any of these changes (unless you’re curious, of course!).

Alternative #1 - Customizing the Movie struct

Instead of creating a custom Runtime type, we could have implemented a MarshalJSON() method on our Movie struct and customized the whole thing. Like this:

// Note that there are no struct tags on the Movie struct itself.
type Movie struct {
    ID        int64
    CreatedAt time.Time
    Title     string
    Year      int32
    Runtime   int32
    Genres    []string
    Version   int32
}

// Implement a MarshalJSON() method on the Movie struct, so that it satisfies the
// json.Marshaler interface.
func (m Movie) MarshalJSON() ([]byte, error) {
    // Declare a variable to hold the custom runtime string (this will be the empty 
    // string "" by default).
    var runtime string

    // If the value of the Runtime field is not zero, set the runtime variable to be a
    // string in the format "<runtime> mins".
    if m.Runtime != 0 {
        runtime = fmt.Sprintf("%d mins", m.Runtime)
    }

    // Create an anonymous struct to hold the data for JSON encoding. This has exactly
    // the same fields, types and tags as our Movie struct, except that the Runtime
    // field here is a string, instead of an int32. Also notice that we don't include
    // a CreatedAt field at all (there's no point including one, because we don't want
    // it to appear in the JSON output).
    aux := struct {
        ID      int64    `json:"id"`
        Title   string   `json:"title"`
        Year    int32    `json:"year,omitempty"`
        Runtime string   `json:"runtime,omitempty"` // This is a string.
        Genres  []string `json:"genres,omitempty"`
        Version int32    `json:"version"`
    }{
        // Set the values for the anonymous struct.
        ID:      m.ID,
        Title:   m.Title,
        Year:    m.Year,
        Runtime: runtime, // Note that we assign the value from the runtime variable here.
        Genres:  m.Genres,
        Version: m.Version,
    }

    // Encode the anonymous struct to JSON, and return it.
    return json.Marshal(aux)
}

Let’s quickly take stock of what’s going on here.

In the MarshalJSON() method we create a new ‘anonymous’ struct and assign it to the variable aux. This anonymous struct is basically identical to our Movie struct, except for the fact that the Runtime field has the type string instead of int32. We then copy all the values from the Movie struct directly into the anonymous struct, except for the Runtime value which we convert to a string in the format "<runtime> mins" first. Then finally, we encode the anonymous struct to JSON — not the original Movie struct — and return it.

It’s also worth pointing out that this is designed so the omitempty directive still works with our custom encoding. If the value of the Runtime field is zero, then the local runtime variable will remain equal to "", which (as we mentioned earlier) is considered ‘empty’ for the purpose of encoding.

Alternative #2 - Embedding an alias

The downside of the approach above is that the code feels quite verbose and repetitive. You might be wondering: is there a better way?

To reduce duplication, instead of writing out all the struct fields long-hand it’s possible to embed an alias of the Movie struct in the anonymous struct. Like so:

// Notice that we use the - directive on the Runtime field, so that it never appears 
// in the JSON output.
type Movie struct {
    ID        int64     `json:"id"`
    CreatedAt time.Time `json:"-"`
    Title     string    `json:"title"`
    Year      int32     `json:"year,omitempty"`
    Runtime   int32     `json:"-"`
    Genres    []string  `json:"genres,omitempty"`
    Version   int32     `json:"version"`
}

func (m Movie) MarshalJSON() ([]byte, error) {
    // Create a variable holding the custom runtime string, just like before.
    var runtime string

    if m.Runtime != 0 {
        runtime = fmt.Sprintf("%d mins", m.Runtime)
    }

    // Define a MovieAlias type which has the underlying type Movie. Due to the way that
    // Go handles type definitions (https://golang.org/ref/spec#Type_definitions) the
    // MovieAlias type will contain all the fields that our Movie struct has but, 
    // importantly, none of the methods. 
    type MovieAlias Movie

    // Embed the MovieAlias type inside the anonymous struct, along with a Runtime field 
    // that has the type string and the necessary struct tags. It's important that we 
    // embed the MovieAlias type here, rather than the Movie type directly, to avoid 
    // inheriting the MarshalJSON() method of the Movie type (which would result in an 
    // infinite loop during encoding).
    aux := struct {
        MovieAlias
        Runtime string `json:"runtime,omitempty"`
    }{
        MovieAlias: MovieAlias(m),
        Runtime:    runtime,
    }

    return json.Marshal(aux)
}

On one hand, this approach is nice because it drastically cuts the number of lines of code and reduces repetition. And if you have a large struct and only need to customize a couple of fields it can be a good option. But it’s not without some downsides.

In particular:

This technique feels like a bit of a ‘trick’, which hinges on the fact that newly defined types do not inherit methods. Although it’s still idiomatic Go, it’s more clever and less clear than the first approach. That’s not always a good trade-off… especially if you have new Gophers working with you on a codebase.
You lose granular control over the ordering of fields in the JSON response. In the above example, the runtime key will always now be the last item in the JSON object, like so:
```
{
    "id": 123,
    "title": "Casablanca",
    "genres": [
        "drama",
        "romance",
        "war"
    ],
    "version": 1,
    "runtime": "102 mins"
}
```
From a technical point of view, this shouldn’t matter as the JSON RFC states that JSON objects are “unordered collections of zero or more name/value pairs”. But it still might be dissatisfactory from an aesthetic or UI point of view, or be problematic if you need to maintain a precise field order for backward-compatibility purposes.