Introduction to Webapp Development with Lua and Openresty

This is a series of posts that aim to explore the basics of webapp development with Lua and Openresty. While I have read a bit about Lua and Openresty in the past, I have no real-world experience with this stack. The purpose of these posts is to document my explorations, and to do so in a way that might help you explore similar topics.

This initial post will provide an introduction to the series, and will include various notes taken from my initial research. The simplest hello world examples will follow in a separate post, and future posts after that will get into more specific problems to solve.

Posts in the Series

1. Hello World Examples
2. JSON/POST Processing in Openresty

What is our target?

I have a difficult time learning how to use a new language, stack, or framework if the examples are not following topics you find in the real-world of webapp development. While this series will be introductory, we will aim to produce a small cluster of services that demonstrate the basic components of a meaningful data transaction/processing workflow.

In pursuit of that demo, each post in the series will focus on small, specific goals that contribute features to the demo. The series will build up from very basic to more meaningful capabilities, eventually coming together to form the demo itself.

Disclaimer

While I have reviewed some of the architectural options or prior art that might be applicable to the requirements of the demo, it has been minimal, and not all details have been scoped out. Being a newb to openresty and lua in general, I will likely run into surprises and need to review a few libraries or options before finding a reasonable solution, and even then, there will be much room for improvement.

Demo Overview

Openresty is built on nginx, so we’ll be using lua with nginx. We’ll use postgres as a database, and may explore redis, rabbitmq, or similar services. For simplicity, we’ll build Docker images and run the exercises in containers.

We’ll run load tests on the system, sending in thousands (maybe millions?) of messages. If we have time, we’ll get into stats/metrics to introspect the system while it runs.

Requirements for the demo

• Set up a basic message queue processing system with the following components:
  • service/app in Nginx and Lua
  • a message queue of some sort
  • a sink to process messages in the queue
  • a datastore for long-term persistence (Postgres)
• The service should work as follows:
  • There are two primary actions:
    1. write a message (POST)
    2. retrieve N of the most recent messages (GET)
  • Nginx has separate location endpoints for the GET and POST actions.
  • The format of the POST data does not matter.
  • The POST data should be put into the message queue, and eventually persisted to the database.
  • The response to GET requests should contain the last N messages received.
  • The application logic is written in Lua and runs inside nginx.
• A sink should be connected to the message queue which processes the items and stores them in the database.
  • We could write the sink in any language, but we’ll use Lua for this as well (it is a good opportunity to learn more Lua in a slightly different context).
  • Processing the message queue should be highly available (two or more nodes) and should gracefully handle a single node failure.
• Assume the nginx server, message queue and database servers all run on separate hosts - don’t assume all services can connect to localhost at the application layer.

OK, let’s get started!

Scoping out the initial tasks

While the requirements of the demo are simple, I have zero experience with Lua, so it makes sense to break up the demo into small challenges to figure out separately.

Here are my initial tasks:

• research my open questions and the specifics of various features:
  • lua + nginx, executing arbitrary code for an HTTP URL path
  • lua as a stand-alone executable or calling luascripts in general
  • how to connect lua to [redis, rabbit, ..], and postgres
    • initial connection + read/write data
  • how to respond to new messages in the queue - run foo() when there’s a new message
  • which {redis,rabbitmq,etc} is most often connected to lua?
    • which seem to have the strongest client libraries?
  • read envvars from lua
  • pass cli params/args to a stand-alone script
  • queues and lua (any prior art?)
• hello world examples
  • nginx + lua for an endpoint /foo
  • lua script
• stand-alone script using CLI args/envvars to set variables
  • for hostnames/credentials/etc
• cli/envvars in nginx+lua
• parse POST data (JSON) in nginx + lua
• connect to postgres and write data to a table
• connect to postgres and read data from a table
  • not using https://github.com/FRiCKLE/ngx_postgres, but do so with lua from nginx
  • https://github.com/leafo/pgmoon#handling-json - retrieve JSON and get it in lua
• connect to queue and read data
• connect to a queue and write data
• keep it easy to swap one queue for another
• Makefile to build and run docker images / etc
• script DB init (create table for messages)
  • can just use the postgres image’s support for auto-running .sql/.sh in db init path

Notes from initial research

• openresty has a docker image built on alpine: openresty/openresty:alpine
  • there is the alpine-fat tag which includes luarocks
• example Dockerfile to build alpine docker image with lua
• Penlight - library for standard stuff that isn’t in lua core
  • TIL: lua core maps to c stdlib (that’s it)
• lua_cliargs - library for parsing cli args/params
• envvars:
  • http://stackoverflow.com/questions/7633397/print-list-of-all-environment-variables
  • “use luaex”
  • maybe easier to use cliargs.. we’ll see
• can build custom static executable with luastatic
• lapis - django-like webapp framework building on openresty

general openresty/nginx and “programming in lua” resources

• Programming in Lua (PIL)
• Lua Users, and wiki
• Openresty/nginx Tutorials
• Programming Openresty

Interesting Openresty Modules

• lua-resty-redis - redis client, actively maintained :)
• lua-redis-parser - a parser for redis responses?
• lua-resty-reqargs - form and JSON processing
• lua-resty-validation
• pgmoon - postgres client
• nginx-json-proxy
• awesome resty - HUGE list of awesome modules

Plan of Attack

After a bunch more research and reading, I settled on the following points:

• use docker for all exploration, development, and running the demo
  • use openresty/nginx bundle for simplicity of development
• start with redis as our datastore for the message queue, fallback to rabbitmq if working with redis does not go well
  • it should be easy to swap queues and queue-processing strategy
• use lua for a stand-alone script (the sink) to process messages in the queue
• the sink will watch for new messages in the queue
• guard for race conditions and data-loss during processing of items on the queue
  • if a sink dies while processing a message, do not silently lose the message
  • if a sink fails to write a message to the database, then write the message to log/stdout (this error condition is separate from losing a sink)
  • ensure we can add multiple sinks to process messages in parallel without race conditions
  • start with RPOPLPUSH for a simple but reliable queue
• use JSON everywhere (processing POST/GET actions, in the queue, and in the datastore)
• format POST messages in JSON with: {"msg": "...."}

Options for the Queue

Overall, the most open-ended questions still lingering are how to implement the queue itself. Here are some notes from my research into this topic.

Redis Queue Bindings / Options

• redis-queue - queue built on redis
  • promising, but hasn’t been updated in 3 years…
• redis-lua - redis client
  • hasn’t been updated in 5 years…
• redis2-nginx-module
  • not really what I’m looking for..
  • nor https://www.nginx.com/resources/wiki/modules/redis/

RabbitMQ Lua Bindings

• lua-resty-rabbitmqstomp
  • “opinionated”, but it’s moved a lot of traffic…
• amqp.lua

Or combine Redis + RabbitMQ…

http://engineering.wingify.com/posts/scaling-with-queues/

Consider putting the lua script right into redis?

See Ch 11 in the Redis Book, scripting redis with Lua.

While this is interesting, it does not seem to be the best fit. It is probably possible to put lua script in redis that responds to the additional messages added to the queue.. but I’m not too excited about this route for a few reasons:

• there’s a bunch to learn with scripting in redis, higher initial entrance fee (at least for now)
• if the code to respond to the queue is in redis, we have to add redis nodes to add workers, which isn’t graceful and doesn’t scale all that well

That said, this post is really interesting and even includes some lua to embed in redis and demonstrates a distributed, scheduled queue.

Next Steps

With the various problems enumerated and some initial research complete, it is time to move on to the canonical Hello World! examples.