Intro to Lua and Openresty, Part 2: JSON/POST Processing in Openresty

Posted on March 3, 2017

In Part 1 of this series we covered the canonical Hello World! with Lua and Openresty. That established the first few bits of our foundation:

handle a GET request with Openresty
return some arbitrary JSON data in the HTTP response
Makefile with various targets helpful for development

Here in Part 2 we will:

handle a POST request with Openresty
capture and process JSON data included with that request
install a lua module, build a docker image, and expand our Makefile

“Processing” the JSON data means:

encoding/parsing the data into JSON, validating it along the way
returning the message received as a JSON response

Capturing POST data

The “new” bit in all of this is retrieving/validating the request data. One of the modules discovered in my initial research appears to be “the way” to do this in Openresty. That module is lua-resty-reqargs.

reqargs greatly simplifies this task. In fact, the workhorse in this example is the following one liner:

local get, post, files = require "resty.reqargs"()

Eg, if we are to POST some arbitrary JSON and use reqargs to capture it, post would be the validated JSON object (or empty, if the JSON was invalid). Note that this is slightly different from the last example in that we’re “importing” and “calling” reqargs in one fell swoop.

What this does for us is pretty simple, but I won’t try to explain it, as the docs say it best:

This function will return three (3) return values, and they are called get, post, and files. These are Lua tables containing the data that was (HTTP) requested. get contains HTTP request GET arguments retrieved with ngx.req.get_uri_args. post contains either HTTP request POST arguments retrieved with ngx.req.get_post_args, or in case of application/json (as a content type header for the request), it will read the request body and decode the JSON, and the post will then contain the decoded JSON structure presented as Lua tables. The last return value files contains all the files uploaded. The files return value will only contain data when there are actually files uploaded and that the request content type is set to multipart/form-data. files has the same structure as get and post for the keys, but the values are presented as a Lua tables, that look like this (think about PHP’s $_FILES)…

`nginx.conf`

OK, so now that we know how to capture that data, let’s see the whole thing together in our nginx.conf:

worker_processes  1;
error_log error.log;
events {
    worker_connections 1024;
}
http {
    server {
        listen        8000;
        charset       utf-8;
        charset_types application/json;
        default_type  application/json;
        location / {
            content_by_lua '
                local cjson = require "cjson"
                local get, post, files = require "resty.reqargs"()
                ngx.status  = ngx.HTTP_OK
                ngx.say(cjson.encode(post))
                return ngx.exit(ngx.HTTP_OK)
            ';
        }
    }
}

While this doesn’t cover all the corner cases, it exemplifies the important bits for this example.

If you are following along with the git repo, we’re in 03-echo-post-json.

Installing the `reqargs` module

The nginx.conf above would work as-is with Openresty, except that the reqargs module needs to be installed.

The Openresty docs I read seemed to recommend using opm to install modules, so I first attempted to use opm, but it didn’t seem to work right:

/ # /usr/local/openresty/bin/opm get bungle/lua-resty-reqargs
* Fetching bungle/lua-resty-reqargs
  Downloading https://opm.openresty.org/api/pkg/tarball/bungle/lua-resty-reqargs-1.4.opm.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  5585  100  5585    0     0   3777      0  0:00:01  0:00:01 --:--:--  3986
Package lua-resty-upload-0.09 already installed.
ERROR: openresty is required but is not available according to resty:

I’m not sure why opm is unable to “find” openresty, but nginx also failed to find reqargs with this method. I would have dug deeper into this, but installing the module with luarocks worked just fine:

ᐅ docker run -it --entrypoint /bin/sh openresty/openresty:alpine-fat
/ #
/ # /usr/local/openresty/luajit/bin/luarocks install lua-resty-reqargs
Installing https://luarocks.org/lua-resty-reqargs-1.4-1.src.rock...
Using https://luarocks.org/lua-resty-reqargs-1.4-1.src.rock... switching to 'build' mode
Updating manifest for /usr/local/openresty/luajit/lib/luarocks/rocks
No existing manifest. Attempting to rebuild...
lua-resty-reqargs 1.4-1 is now built and installed in /usr/local/openresty/luajit (license: BSD)

Note that the openresty:alpine image does not include luarocks, so we need to use the openresty:alpine-fat image here.

`Dockerfile`

Given that the base openresty Docker image needs to be “updated” before we can use it for this example, it makes sense to create a Dockerfile and build an image to use when we want to run our code.

Here is the Dockerfile:

FROM openresty/openresty:alpine-fat

EXPOSE 8000
RUN /usr/local/openresty/luajit/bin/luarocks install lua-resty-reqargs
ADD nginx.conf /usr/local/openresty/nginx/conf/nginx.conf

We can now build that image with:

ᐅ docker build --tag=app:3 --rm=true ./

Note that I’m tagging the image with app:3 as part of the series of examples, other images will have app:4, db:4 and so on.

`Makefile`

I created the following Makefile while working thru this example:

build:
        docker build --tag=app:3 --rm=true ./

run:
        docker run -d --name app --net host -p 127.0.0.1:8000:8000 app:3

dev:
        docker run -d --name app --net host -p 127.0.0.1:8000:8000 -v `pwd`:/usr/local/openresty/nginx/conf/ app:3

clean:
        docker stop app || true
        docker rm   app || true

reload:
        docker exec -it app /usr/local/openresty/nginx/sbin/nginx -s reload

logs:
        docker exec -it app tail -f /usr/local/openresty/nginx/error.log

get:
        curl -i localhost:8000

post:
        curl -H "Content-Type: application/json" -X POST -d '{"id": 1, "username":"xyz","pass":"foobar"}' localhost:8000/

post-invalid:
        curl -H "Content-Type: application/json" -X POST -d '{"id": 1, "username":"xyz","pass:}' localhost:8000/

Build the Docker Image

ᐅ make build
docker build --tag=app:3 --rm=true ./
Sending build context to Docker daemon  5.12 kB
Step 1 : FROM openresty/openresty:alpine-fat
 ---> 366babf2b04d
Step 2 : EXPOSE 8000
 ---> Using cache
 ---> 35a8c6e42825
Step 3 : RUN /usr/local/openresty/luajit/bin/luarocks install lua-resty-reqargs
 ---> Using cache
 ---> d5c51f61f244
Step 4 : ADD nginx.conf /usr/local/openresty/nginx/conf/nginx.conf
 ---> 226f6ade413b
Removing intermediate container 232aa9f27193
Successfully built 226f6ade413b

I have run the build a few times here, so docker is using the cache, otherwise we would see Docker install reqargs with luarocks.

Let’s See it in Action!

ᐅ make run
docker run -d --name app --net host -p 127.0.0.1:8000:8000 app:3
2e0a97eb565d5820f307f06e8e6a1438d886b4c6cbd4f319030b55d571e0814c

We’ve done nothing to differentiate between GET/POST/etc requests, so, technically speaking, this will respond to a GET with no data:

ᐅ make get
curl -i localhost:8000
HTTP/1.1 200 OK
Server: openresty/1.11.2.2
Date: Fri, 3 Mar 2017 14:28:21 GMT
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive

{}

Let’s see what happens when we POST invalid JSON:

ᐅ make post-invalid
curl -H "Content-Type: application/json" -X POST -d '{"id": 1, "username":"xyz","pass:}' localhost:8000/
{}

If we post invalid JSON, reqargs will validate the JSON and post will be empty (we could add logic to handle this as an error, but haven’t in this example).

Let’s POST valid JSON, we should see it echoed back to us:

ᐅ make post
curl -H "Content-Type: application/json" -X POST -d '{"id": 1, "username":"xyz","pass":"foobar"}' localhost:8000/
{"username":"xyz","pass":"foobar","id":1}

In case things go wrong, we have the logs make target to help us see what the problem is:

ᐅ make logs
docker exec -it app tail -f /usr/local/openresty/nginx/error.log
2017/03/03 14:30:53 [error] 5#5: *1 failed to load inlined Lua code: content_by_lua(nginx.conf:20):4: '=' expected near 'post', client: 127.0.0.1, server: , request: "POST / HTTP/1.1", host: "localhost:8000"
^Cmake: *** [logs] Error 130

To clean up (stop/remove the docker container):

ᐅ make clean
docker stop app || true
app
docker rm   app || true
app

Intro to Lua and Openresty, Part 2: JSON/POST Processing in Openresty

Capturing POST data

nginx.conf

Installing the reqargs module

Dockerfile