Introduction to Webapp Development with Lua and Openresty

Posted on March 2, 2017

This is a series of posts that aim to explore the basics of webapp development with Lua and Openresty. While I have read a bit about Lua and Openresty in the past, I have no real-world experience with this stack. The purpose of these posts is to document my explorations, and to do so in a way that might help you explore similar topics.

This initial post will provide an introduction to the series, and will include various notes taken from my initial research. The simplest hello world examples will follow in a separate post, and future posts after that will get into more specific problems to solve.

Posts in the Series

  1. Part 1: Hello World Examples
  2. Part 2: JSON/POST Processing in Openresty
  3. Part 3: Write to Postgres
  4. Part 4: Using Envvars
  5. Part 5: Redis
  6. Interlude 1: Issues with Luarocks and Alpine
  7. Part 6: Data Processing Sink
  8. Interlude 2: Exploring Luas Package Path
  9. Part 7: Limit HTTP Methods
  10. Part 8: Reading from Postgres
  11. Part 9: HTTP Clients
  12. Part 10: Pull it all Together

What is our target?

I have a difficult time learning how to use a new language, stack, or framework if the examples are not following topics you find in the real-world of webapp development. While this series will be introductory, we will aim to produce a small cluster of services that demonstrate the basic components of a meaningful data transaction/processing workflow.

In pursuit of that demo, each post in the series will focus on small, specific goals that contribute features to the demo. The series will build up from very basic to more meaningful capabilities, eventually coming together to form the demo itself.

Disclaimer

While I have reviewed some of the architectural options or prior art that might be applicable to the requirements of the demo, it has been minimal, and not all details have been scoped out. Being a newb to openresty and lua in general, I will likely run into surprises and need to review a few libraries or options before finding a reasonable solution, and even then, there will be much room for improvement.

Demo Overview

Openresty is built on nginx, so we’ll be using lua with nginx. We’ll use postgres as a database, and may explore redis, rabbitmq, or similar services. For simplicity, we’ll build Docker images and run the exercises in containers.

We’ll run load tests on the system, sending in thousands (maybe millions?) of messages. If we have time, we’ll get into stats/metrics to introspect the system while it runs.

Requirements for the demo

OK, let’s get started!

Scoping out the initial tasks

While the requirements of the demo are simple, I have zero experience with Lua, so it makes sense to break up the demo into small challenges to figure out separately.

Here are my initial tasks:

Notes from initial research

general openresty/nginx and “programming in lua” resources

Interesting Openresty Modules

Plan of Attack

After a bunch more research and reading, I settled on the following points:

Options for the Queue

Overall, the most open-ended questions still lingering are how to implement the queue itself. Here are some notes from my research into this topic.

Redis Queue Bindings / Options

RabbitMQ Lua Bindings

Or combine Redis + RabbitMQ…

http://engineering.wingify.com/posts/scaling-with-queues/

Consider putting the lua script right into redis?

See Ch 11 in the Redis Book, scripting redis with Lua.

While this is interesting, it does not seem to be the best fit. It is probably possible to put lua script in redis that responds to the additional messages added to the queue.. but I’m not too excited about this route for a few reasons:

That said, this post is really interesting and even includes some lua to embed in redis and demonstrates a distributed, scheduled queue.

Next Steps

With the various problems enumerated and some initial research complete, it is time to move on to the canonical Hello World! examples.