Using Redis for HTTP Api Inter process Communication

First, solve the problem. Then, write the code.

John Johnson

One of the most exciting aspects of the software development process is experiencing the steps leading to a pleasing solution to your problem. That moment when, after some time of thoroughly brainstorming, everything falls into place. I had such a moment today, and I would like to share my story.

The Problem

This morning I was faced with a simple dilemma. I needed to perform a GET request containing a large payload to the server, but I didn’t want to show it in the URL. The reason for the GET request is that I wanted to give the user the ability to download a file with a click of a button. The purpose of the large payload, the requirements for this file. You see, this file is a zip archive that the service will dynamically construct and deliver to the user. The issue is it can potentially contain thousands of files inside, and I didn’t want to clutter the URL with this payload.

That is when it hit me! I have been using Redis for some time now, and I thought this would be a great use of it. With that in mind, I set up to develop the following idea.

Solve it

The plan is to POST the payload and store it in Redis. Then I can send a unique identifier for this payload to the client so they can reference it later on the GET request.

Great! Let’s get to it.

I chose Golang for this since it is remarkably versatile and I could quickly develop the solution. For those of you that do not know what Redis is, let’s talk a bit about it.

Taken from their site https://redis.io/ “Redis is an open source (BSD licensed), in-memory data structure store, used as a database cache and message broker.” In other words, it is a service that allows you to store information in memory RAM in a key-value fashion. One of the keys to take away from this is the fact that the data is stored in RAM. This means, obviously, that when the system is taken down, the info is lost. However, this also means that Redis is extremely fast too. I have found many uses for Redis especially in the area of Interprocess Communication.

Redis contains a large set of instructions (https://redis.io/commands); we will be focusing on only two commands for this project: SETEX (sets a value with an expiration time) and GET (get the value based on a given key). There are many tutorials on how to install Redis, and I encourage you to take a look at them if you need to do so. In fact, it is quite simple to install and use so we will go straight to the fun part (coding).

Back to Kenny G. and my headsets. It’s focus time!

Get to Coding

// file service.go
// This file contains the api endpoints and handlers

package main

import (
	"encoding/json"
	"io/ioutil"
	"net/http"

	"github.com/gorilla/mux"
)

const REDIS_CONNECTION string = "localhost:6379"

type Payload struct {
	Files []string
}

func main() {
	router := mux.NewRouter()
	// Make sure payload only accepts POST requests
	router.HandleFunc("/payload", payload).Methods("POST")
	// Make sure download/{id} only accepts GET requests
	router.HandleFunc("/download/{id}", download).Methods("GET")
	http.ListenAndServe(":8000", router)
}

// Process the endpoint paylaod
func payload(w http.ResponseWriter, r *http.Request) {
	b, _ := ioutil.ReadAll(r.Body)

	ref, err := storePayload(string(b))
	if err != nil {
		http.Error(w, "Error connecting to Redis", http.StatusInternalServerError)
		return
	}
	result, _ := json.Marshal(struct {
		Success bool
		Ref     string
	}{
		true,
		ref,
	})
	w.WriteHeader(http.StatusOK)
	w.Header().Set("Content-Type", "application/json")
	w.Write(result)

}

func download(w http.ResponseWriter, r *http.Request) {
	vars := mux.Vars(r)
	payload, _ := getPayload(vars["id"])

	result, _ := json.Marshal(struct {
		Success bool
		Payload *Payload
	}{
		true,
		payload,
	})
	w.WriteHeader(http.StatusOK)
	w.Header().Set("Content-Type", "application/json")
	w.Write(result)
}

func storePayload(p string) (string, error) {
	redisWrapper := NewRedisWrapper(REDIS_CONNECTION)
	defer redisWrapper.Close()
	return redisWrapper.StorePayload(p)
}

func getPayload(ref string) (*Payload, error) {
	redisWrapper := NewRedisWrapper(REDIS_CONNECTION)
	defer redisWrapper.Close()
	return redisWrapper.GetPayload(ref)
}

Taking all the setup aside for the HTTP handlers, you can see later how easy it is to establish a Redis connection and start using it. As you can see in the code, we defer the Redis functionality to a redisWrapper structure. The following will be the file containing the Redis functionality.

// file redis_wrapper.go
// This file contains the redis functionality

package main

import (
	"encoding/json"
	"errors"
	"time"

	redigo "github.com/garyburd/redigo/redis"
	"github.com/google/uuid"
)

const FIELD_INDEX = "Files:"
const EXPIRATION_TIME = 60

type RedisWrapper struct {
	redisPool *redigo.Pool
}

func NewRedisWrapper(connection string) *RedisWrapper {
	return &RedisWrapper{
		redisPool: initRedis(connection),
	}
}

func initRedis(connection string) *redigo.Pool {
	return &redigo.Pool{
		MaxIdle:     10,
		IdleTimeout: 1 * time.Second,
		Dial: func() (redigo.Conn, error) {
			return redigo.Dial("tcp", connection)
		},
		TestOnBorrow: func(c redigo.Conn, t time.Time) (err error) {
			_, err = c.Do("PING")
			if err != nil {
				panic("Error connecting to redis")
			}
			return
		},
	}
}

func (this *RedisWrapper) GetPayload(ref string) (*Payload, error) {
	redis := this.redisPool.Get()
	// Get the value from Redis
	result, err := redis.Do("GET", FIELD_INDEX+ref)
	if err != nil || result == nil {
		err = errors.New("Reference not found")
		return nil, err
	}
	// Decode the JSON
	var resultByte []byte
	var ok bool
	if resultByte, ok = result.([]byte); !ok {
		err = errors.New("Error reading from redis")
		return nil, err
	}
	var payload Payload
	err = json.Unmarshal(resultByte, &payload)
	if err != nil {
		err = errors.New("Error decoding files redis data")
		return nil, err
	}
	return &payload, nil
}

func (this *RedisWrapper) StorePayload(payload string) (string, error) {
	redis := this.redisPool.Get()
	uniqueId := uuid.New()

	redis.Do("SETEX", FIELD_INDEX+uniqueId.String(), EXPIRATION_TIME, payload)
	return uniqueId.String(), nil
}

func (this *RedisWrapper) Close() {
	this.redisPool.Close()
}

The NewRedisWrapper function will use initRedis which accept a connection string in the form (host: port ) to return a New RedisWrapper. The RedisWrapper will contain a Pool struct from the redigo library initialized, and it will make sure the connection exists before it returns it.

StorePayload will save the given payload into Redis using the SETEX function. SETEX will store a value with a given expiration time. We try to use a unique identifier for the key. For this unique identifier, we use a UUID, since they are pretty much guaranteed to be quite unique on subsequent calls.

The GetPayload function will retrieve this value from Redis and return it to the client. I decided to return it for simplicity, but this is the point in which you retrieve the payload and use it as you see fit :).

In general, Redis is an excellent tool to establish these kinds of communication. Be sure to take a look at Redis since I am sure you will be using it a lot.

Happy coding.

Leave a Reply

Your email address will not be published. Required fields are marked *