Intro to Lua and Openresty, Part 10: Pull it all together!

Posted on March 10, 2017

Let’s review our progress as we’ve prepared for this demo:

Part 1:
- use the openresty docker image, the alpine variant
- handle a GET request with Openresty
- use nginx content_by_lua
- use ngx.say to respond with a hello world from nginx/lua, in HTML
- require()
- cjson.encode() to create a JSON object in lua, hello world with JSON
- nginx HTTP_OK response code
- hello world from stand-alone lua script
- basic Makefile
Part 2:
- use resty.reqargs to validate/process a POST request with Openresty
- install a lua module
- process / validate / use JSON object from POST
- include build / run / test / debug / clean workflow in Makefile
Part 3:
- automate db init with postgres docker image
- use pgmoon to connect to postgres
- assertions and handling IO failure with assert()
- write JSON to postgres
- improve JSON HTTP response
Part 4:
- using os.getenv() to access and use environment variables in nginx/lua and luascript
- parametize the database connection
Part 5:
- use resty.redis to connect to redis
- conditionals, error handling, assertions and basic failure messages
- use lpush to write a JSON message to a table in redis
Part 6:
- use hiredis to connect to redis
- playing ping/pong with redis, and basic redis commands
- basic queues and data processing pipelines
- use os.date() and date formats to create a timestamp
- using for loops
- creating our own functions
- simulate “safe” work queue with RPOPLPUSH and friends
- worker scripts to populate our queue/datastore
Part 7:
- conditional processing based on HTTP methods
- exit with HTTP_NOT_ALLOWED response code
Part 8:
- querying Postgres and returning JSON messages from the database
Part 9:
- using httpclient to explore GET and POST requests as an HTTP client
- print the response body, code, headers, status, errors
- use cliargs to process cli parameters/args
- more functions, program structure
- math.random()
- batch POST to generate and send many requests into the system
- print list of top posts from the system

In this post, we will use all of these capabilities together to run a demo of a basic data processing sink based on redis.

If you are following along in the code, we are here.

Components in the System

      User                                                                         User
    +------+                                                                        +
    | JSON |                                                                        |
    +------+                                                                        |
       ||                                                                           |
       ||                                                                           |
       ||                                                                           |
+------------------------------------------------------------------------------------------------------+
|      ||                                                                           |                  |
|      VV                                                                           |                  |
| +------------+                                                           +--------+--------+         |
| | location: /|                   Openresty / NGINX Webapp Server         | location: /list |         |
| | POST       |                                                           | GET             | <-+     |
| +-----+------+                                                           |                 |   |     |
|       |                                                                  +-------------+---+   |     |
|       |                                                                                |       |     |
+------------------------------------------------------------------------------------------------------+
        |                                                                                |       |
        |                                                                                |       |
        |        +----------------------------------------------------------+            |       |
        |        |      ENQUEUED (list)                 PROCESSING (list)   |            |       |  SELECT
        |  LPUSH |    +---+---+---+---+    RPOPLPUSH    +---+---+---+---+   |            |       |  data FROM
        +-----------> | d | c | b | a +->-->---+--->--->+ a |   |   |   |   |            |       |  posts
                 |    +---+---+---+---+        |        +---+---+---+---+   |            |       |  ORDER BY
                 |                             |                            |            |       |  id
                 |    Redis Queue              |                            |            |       |  DESC
                 |                             |                            |            |       |  LIMIT 10
                 +----------------------------------------------------------+            |       |
                                               |                                         |       |
                                               |                                         |       |
                                               |                                         |       |
                                   +------------------------------+                      v       |
                                   |           |                  |                              |
                                   V           V a                V                  +-----------+--+
                                +------+    +------+           +------+              |              |
                                |      |    |      |           |      |              |  Postgres    |
                                | sink |    | sink |     ...   | sink |              |  Database    |
                                |      |    |      |           |      |              |              |
                                +--+---+    +--+---+           +--+---+              +--------------+
                                   |           |                  |
                                   |           |                  |                         ^
                                   |           |                  |                         |
                                   |           |                  |                         |
                                   +-----------+------------------+-------------------------+

The system has the following in it:

nginx API server
- accepts POST with arbitrary JSON, writes that JSON to a queue on redis
  - responds with a confirmation the msg was received and queued, and an echo of the message
- responds to GET with the last 100 messages posted
postgres database
- long-term datastore for messages
redis
- temporary storage for in-flight messages
worker instances
- poll redis for items on the queue
- for each item found on the queue, process the item (write it to postgres)
- run multiple instances

Makefile

Here is our Makefile, rather substantial:

build:
	docker build --tag=db:demo   --rm=true ./db
	docker build --tag=app:demo  --rm=true ./app
	docker build --tag=sink:demo --rm=true ./sink

run:
	docker run -d --name db    --net host -p 127.0.0.1:5342:5432 db:demo
	docker run -d --name app   --net host -p 127.0.0.1:8000:8000 app:demo
	docker run -d --name redis --net host -p 127.0.0.1:6379:6379 redis:alpine
	docker run -d --name sink  --net host --entrypoint /usr/local/openresty/luajit/bin/luajit sink:demo worker.lua

dev:
	docker run -d --name redis --net host -p 127.0.0.1:6379:6379 redis:alpine
	docker run -d --name db    --net host -p 127.0.0.1:5342:5432 db:demo
	docker run -d --name app   --net host -p 127.0.0.1:8000:8000 -v `pwd`/app/producer.lua:/src/producer.lua -v `pwd`/app/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf app:demo

shell-app:
	docker exec -it app /bin/sh

shell-sink:
	docker exec -it sink /bin/sh

shell-redis:
	docker exec -it redis redis-cli

clean:
	docker stop db     || true
	docker stop app    || true
	docker stop sink   || true
	docker stop redis  || true
	docker rm   db     || true
	docker rm   app    || true
	docker rm   sink   || true
	docker rm   redis  || true

reload:
	docker exec -it app /usr/local/openresty/nginx/sbin/nginx -s reload

logs-app:
	docker exec -it app tail -f /usr/local/openresty/nginx/error.log

cat-posts:
	docker exec -it db psql -U postgres -d lua-app -c 'SELECT * FROM posts;'

cat-queue:
	docker exec -it redis redis-cli -c LRANGE enqueued 0 -1

post-msgs:
	curl -i -H "Content-Type: application/json" -X POST -d '{"id": 1, "username":"xyz","password":"xyz"}' localhost:8000/
	curl -i -H "Content-Type: application/json" -X POST -d '{"id": 2, "username":"foo","password":"foo"}' localhost:8000/
	curl -i -H "Content-Type: application/json" -X POST -d '{"id": 3, "username":"bar","password":"bar"}' localhost:8000/

curl-msgs:
	curl -i -H "Content-Type: application/json" localhost:8000/list

Openresty App

Dockerfile:

FROM openresty/openresty:alpine-fat

EXPOSE 8000
RUN /usr/local/openresty/luajit/bin/luarocks install pgmoon
RUN /usr/local/openresty/luajit/bin/luarocks install lua-resty-reqargs
RUN /usr/local/openresty/luajit/bin/luarocks install lua-cjson
RUN /usr/local/openresty/luajit/bin/luarocks install luasocket
ADD nginx.conf /usr/local/openresty/nginx/conf/nginx.conf
ENV REDIS_HOST 127.0.0.1
ENV DB_HOST    127.0.0.1
ENV DB_USER    postgres
ENV DB_PASS    password
ENV DB_NAME    lua-app

nginx.conf:

worker_processes  1;                                                                                                                                                 [26/376]
env DB_HOST;
env DB_USER;
env DB_PASS;
env DB_NAME;
env REDIS_HOST;
error_log error.log;
events {
    worker_connections 1024;
}
http {
    server {
        listen        8000;
        charset       utf-8;
        charset_types application/json;
        default_type  application/json;
        location / {
            content_by_lua '
              local cjson = require "cjson"
              local http_method = ngx.var.request_method
              if http_method == "POST" then
                local redis = require "resty.redis"
                local r     = redis:new()
                local ok, err = r:connect(os.getenv("REDIS_HOST"), 6379)
                if not ok then
                  emsg = "failed to connect to queue: "
                  ngx.say(cjson.encode({status = "error", msg = emsg .. err}))
                  return ngx.exit(ngx.HTTP_SERVICE_UNAVAILABLE)
                end
                local get, post, files = require "resty.reqargs"()
                assert(r:lpush("enqueued", cjson.encode(post)))
                r = nil
                ngx.status = ngx.HTTP_OK
                ngx.say(cjson.encode({status = "saved", msg=post}))
                return ngx.exit(ngx.HTTP_OK)
              else
                ngx.status = ngx.HTTP_NOT_ALLOWED
                ngx.say(cjson.encode({method = http_method , status = "denied"}))
                return ngx.exit(ngx.HTTP_NOT_ALLOWED)
              end
            ';
        }
        location /list {
            content_by_lua '
              local cjson = require "cjson"
              local http_method = ngx.var.request_method
              if http_method == "GET" then
                local pgmoon = require "pgmoon"
                local pg = pgmoon.new({
                  host     = os.getenv("DB_HOST"),
                  port     = "5432",
                  user     = os.getenv("DB_USER"),
                  password = os.getenv("DB_PASS"),
                  database = os.getenv("DB_NAME")
                })
                assert(pg:connect())
                local get, post, files = require "resty.reqargs"()
                top = pg:query("SELECT data FROM posts ORDER BY id DESC LIMIT 10;")
                pg:keepalive()
                pg = nil
                ngx.status = ngx.HTTP_OK
                ngx.say(cjson.encode({ msg = top }))
                return ngx.exit(ngx.HTTP_OK)
              else
                ngx.status = ngx.HTTP_NOT_ALLOWED
                ngx.say(cjson.encode({method = http_method , status = "denied"}))
                return ngx.exit(ngx.HTTP_NOT_ALLOWED)
              end
            ';
        }

    }
}

Database

Dockerfile:

FROM postgres:alpine
ENV  POSTGRES_PASSWORD password
ENV  POSTGRES_DB       lua-app
ADD  init.sql /docker-entrypoint-initdb.d/

init.sql :

CREATE TABLE posts (ID SERIAL PRIMARY KEY, data JSONB);

Data Processing Sink

Dockerfile:

FROM openresty/openresty:alpine-fat

ENV REDIS_HOST 127.0.0.1
ENV DB_HOST    127.0.0.1
ENV DB_USER    postgres
ENV DB_PASS    password
ENV DB_NAME    lua-app
RUN /usr/local/openresty/luajit/bin/luarocks install pgmoon
RUN /usr/local/openresty/luajit/bin/luarocks install luasocket
RUN /usr/local/openresty/luajit/bin/luarocks install lua-hiredis
RUN /usr/local/openresty/luajit/bin/luarocks install lua-cjson
ADD worker.lua /src/
WORKDIR /src/
ENTRYPOINT /bin/sh

worker.lua:

local cjson  = require "cjson"
local redis  = require "hiredis"
local pgmoon = require "pgmoon"
local pg = pgmoon.new({
  host     = os.getenv("DB_HOST"),
  port     = "5432",
  user     = os.getenv("DB_USER"),
  password = os.getenv("DB_PASS"),
  database = os.getenv("DB_NAME")
})
assert(pg:connect())
local encode_json = require("pgmoon.json").encode_json

-- names for our lists in redis
local q     = "enqueued"
local p     = "processing"
-- length of time (seconds) to sleep
local delay = 0.001

-- return redis client, or fail and exit
credis = function (host)
  local rc, err, err_code = hiredis.connect(host, 6379)
  if not rc then
    print("failed to connect to redis..")
    print("error: " .. err)
    print("code:  " .. err_code)
    os.exit(1)
  else
    return rc
  end
end
-- retrieve work from redis, store it in "processing" table
get_work = function()
  return rc:command("RPOPLPUSH", q, p)
end
-- write data to postgres
write_post = function(data)
  assert(pg:query("INSERT INTO posts (data) VALUES(" .. encode_json(data) .. ");"))
end
-- "do" the work
process = function (i)
  write_post(i)
  print(i)
end
-- work is done, drop it from the processing table
dondrop = function (i)
  return rc:command("LREM", p, 1, i)
end
-- pause for a moment..
-- could also use socket.sleep(sec) from the "socket" library
sleep = function(t)
  os.execute("sleep " .. tonumber(t))
end
--
-- MAIN
rc = credis(os.getenv("REDIS_HOST"))
assert(rc)
-- loop doing work until you can't
while true do
  item, err, code = get_work(q, p)
  if item.name == "NIL" then
    -- pass
  else
    --print("got item!")
    process(item)
    dondrop(item)
  end
  sleep(delay)
end

rc:close()

Make the Images

ᐅ make build
docker build --tag=db:demo   --rm=true ./db
Sending build context to Docker daemon 3.072 kB
Step 1 : FROM postgres:alpine
 ---> f0476a087b97
Step 2 : ENV POSTGRES_PASSWORD password
 ---> Using cache
 ---> 30d766415bf6
Step 3 : ENV POSTGRES_DB lua-app
 ---> Using cache
 ---> 449a34987810
Step 4 : ADD init.sql /docker-entrypoint-initdb.d/
 ---> Using cache
 ---> 7ac365dda47a
Successfully built 7ac365dda47a
docker build --tag=app:demo  --rm=true ./app
Sending build context to Docker daemon 18.43 kB
Step 1 : FROM openresty/openresty:alpine-fat
 ---> 366babf2b04d
Step 2 : EXPOSE 8000
 ---> Using cache
 ---> 35a8c6e42825
Step 3 : RUN /usr/local/openresty/luajit/bin/luarocks install pgmoon
 ---> Using cache
 ---> effd23d59e55
Step 4 : RUN /usr/local/openresty/luajit/bin/luarocks install lua-resty-reqargs
 ---> Using cache
 ---> 0cde38c767ed
Step 5 : RUN /usr/local/openresty/luajit/bin/luarocks install lua-cjson
 ---> Using cache
 ---> e36fd7404d14
Step 6 : RUN /usr/local/openresty/luajit/bin/luarocks install luasocket
 ---> Using cache
 ---> c637b4563598
Step 7 : ADD nginx.conf /usr/local/openresty/nginx/conf/nginx.conf
 ---> Using cache
 ---> bc33cf10bf01
Step 8 : ENV REDIS_HOST 127.0.0.1
 ---> Using cache
 ---> 9f78043fdfc7
Step 9 : ENV DB_HOST 127.0.0.1
 ---> Using cache
 ---> c0e0abcf87d5
Step 10 : ENV DB_USER postgres
 ---> Using cache
 ---> f2f16dfddb1f
Step 11 : ENV DB_PASS password
 ---> Using cache
 ---> 7e0b6d86ae27
Step 12 : ENV DB_NAME lua-app
 ---> Using cache
 ---> b0a48a3d9595
Successfully built b0a48a3d9595
docker build --tag=sink:demo --rm=true ./sink
Sending build context to Docker daemon 4.608 kB
Step 1 : FROM openresty/openresty:alpine-fat
 ---> 366babf2b04d
Step 2 : ENV REDIS_HOST 127.0.0.1
 ---> Using cache
 ---> de5f965dde03
Step 3 : ENV DB_HOST 127.0.0.1
 ---> Using cache
 ---> b06582dae81d
Step 4 : ENV DB_USER postgres
 ---> Using cache
 ---> f3df1ddede03
Step 5 : ENV DB_PASS password
 ---> Using cache
 ---> 402db0ba86ba
Step 6 : ENV DB_NAME lua-app
 ---> Using cache
 ---> 030fc5a44f2e
Step 7 : RUN /usr/local/openresty/luajit/bin/luarocks install pgmoon
 ---> Using cache
 ---> f8423c4e9542
Step 8 : RUN /usr/local/openresty/luajit/bin/luarocks install luasocket
 ---> Using cache
 ---> d2e0bec18540
Step 9 : RUN /usr/local/openresty/luajit/bin/luarocks install lua-hiredis
 ---> Using cache
 ---> 64d42d338134
Step 10 : RUN /usr/local/openresty/luajit/bin/luarocks install lua-cjson
 ---> Using cache
 ---> 899c82d7d9dc
Step 11 : ADD worker.lua /src/
 ---> Using cache
 ---> e08cab28d016
Step 12 : WORKDIR /src/
 ---> Using cache
 ---> d46f7aee15f9
Step 13 : ENTRYPOINT /bin/sh
 ---> Using cache
 ---> f461ed471c63
Successfully built f461ed471c63
docker build --tag=load:demo --rm=true ./load
Sending build context to Docker daemon 15.87 kB
Step 1 : FROM openresty/openresty:alpine
 ---> 984d503b1bd8
Step 2 : ADD load-test.lua /src/
 ---> ad623a0dfb04
Removing intermediate container 21a2b3186d98
Step 3 : WORKDIR /src/
 ---> Running in 715d14831bd5
 ---> c709f22bdc16
Removing intermediate container 715d14831bd5
Step 4 : ENTRYPOINT /bin/sh
 ---> Running in 23f3389007ba
 ---> ecf7b4f08462
Removing intermediate container 23f3389007ba
Successfully built ecf7b4f08462

Run the Demo!

Start ’em up..

ᐅ make run
docker run -d --name db    --net host -p 127.0.0.1:5342:5432 db:demo
9141dc3fb9788d73be7971dd44a3d140306695914b5422191f5e9de2f42c7f01
docker run -d --name app   --net host -p 127.0.0.1:8000:8000 app:demo
46cde00714668131e32fbcebc4afb417415ef3437fa5bd669c536bc21ba03a11
docker run -d --name redis --net host -p 127.0.0.1:6379:6379 redis:alpine
139def3c153a845d8d3728e6c7de167ed7ab0a6045a32b72d84cd431e3487e53
docker run -d --name sink  --net host --entrypoint /usr/local/openresty/luajit/bin/luajit sink:demo worker.lua
c3ad3e737997437ec82d6b2899d8746d1de1a9cef72fdc7b1ad63395e8c4774f

Wait, there should be 4!

ᐅ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
139def3c153a        redis:alpine        "docker-entrypoint.sh"   11 seconds ago      Up 10 seconds                           redis
46cde0071466        app:demo            "/usr/local/openresty"   11 seconds ago      Up 10 seconds                           app
9141dc3fb978        db:demo             "docker-entrypoint.sh"   12 seconds ago      Up 11 seconds                           db

…when the sink starts up before postgres is available, the sink fails hard. But, interestingly, this is helpful for stepping through the demo. Let’s leave the sink offline for a moment.

Let’s load up the queue with a few messages:

ᐅ make post-msgs
curl -i -H "Content-Type: application/json" -X POST -d '{"id": 1, "username":"xyz","password":"xyz"}' localhost:8000/
HTTP/1.1 200 OK
Server: openresty/1.11.2.2
Date: Sun, 05 Mar 2017 07:41:15 GMT
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive

{"status":"saved","msg":{"password":"xyz","username":"xyz","id":1}}
curl -i -H "Content-Type: application/json" -X POST -d '{"id": 2, "username":"foo","password":"foo"}' localhost:8000/
HTTP/1.1 200 OK
Server: openresty/1.11.2.2
Date: Sun, 05 Mar 2017 07:41:15 GMT
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive

{"status":"saved","msg":{"password":"foo","username":"foo","id":2}}
curl -i -H "Content-Type: application/json" -X POST -d '{"id": 3, "username":"bar","password":"bar"}' localhost:8000/
HTTP/1.1 200 OK
Server: openresty/1.11.2.2
Date: Sun, 05 Mar 2017 07:41:15 GMT
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive

{"status":"saved","msg":{"password":"bar","username":"bar","id":3}}

The sink is offline, so requesting the list of lastest posts will return empty:

ᐅ make curl-msgs
curl -i -H "Content-Type: application/json" localhost:8000/list
HTTP/1.1 200 OK
Server: openresty/1.11.2.2
Date: Sun, 05 Mar 2017 07:44:20 GMT
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive

{"msg":{}}

…and the posts table in the database would be empty:

ᐅ make cat-posts
docker exec -it db psql -U postgres -d lua-app -c 'SELECT * FROM posts;'
 id | data
----+------
(0 rows)

But the messages should be in the queue..

ᐅ make cat-queue
docker exec -it redis redis-cli -c LRANGE enqueued 0 -1
1) "{\"password\":\"bar\",\"username\":\"bar\",\"id\":3}"
2) "{\"password\":\"foo\",\"username\":\"foo\",\"id\":2}"
3) "{\"password\":\"xyz\",\"username\":\"xyz\",\"id\":1}"

Let’s restart the sink and get those messages moved:

ᐅ make rerun-sink
docker rm sink
sink
docker run -d --name sink  --net host --entrypoint /usr/local/openresty/luajit/bin/luajit sink:demo worker.lua
188d6516936f974c2a96da9b25d2182bb7390d79b978562f109045a1732cca5a

Anything still in the queue?

ᐅ make cat-queue
docker exec -it redis redis-cli -c LRANGE enqueued 0 -1
(empty list or set)

Anything in the database?

ᐅ make cat-posts
docker exec -it db psql -U postgres -d lua-app -c 'SELECT * FROM posts;'
 id |                          data
----+--------------------------------------------------------
  1 | "{\"password\":\"xyz\",\"username\":\"xyz\",\"id\":1}"
  2 | "{\"password\":\"foo\",\"username\":\"foo\",\"id\":2}"
  3 | "{\"password\":\"bar\",\"username\":\"bar\",\"id\":3}"
(3 rows)

OK, let’s request them through the API:

ᐅ make curl-msgs
curl -i -H "Content-Type: application/json" localhost:8000/list
HTTP/1.1 200 OK
Server: openresty/1.11.2.2
Date: Sun, 05 Mar 2017 07:48:12 GMT
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive

{"msg":[{"data":"{\"password\":\"bar\",\"username\":\"bar\",\"id\":3}"},{"data":"{\"password\":\"foo\",\"username\":\"foo\",\"id\":2}"},{"data":"{\"password\":\"xyz\",\"username\":\"xyz\",\"id\":1}"}]}

can also view thru jq:

ᐅ make curl-msgs | tail -n 1 | jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   202    0   202    0     0  24363      0 --:--:-- --:--:-- --:--:-- 25250
{
  "msg": [
    {
      "data": "{\"password\":\"bar\",\"username\":\"bar\",\"id\":3}"
    },
    {
      "data": "{\"password\":\"foo\",\"username\":\"foo\",\"id\":2}"
    },
    {
      "data": "{\"password\":\"xyz\",\"username\":\"xyz\",\"id\":1}"
    }
  ]
}

That confirms we have the basics functional. Let’s see what we can do for load tests next.

Time Tracking: since last check: 2 hours; total: 17 hours