Around 5 minutes (959 words). Published July 3, 2020.

gRPC-Web through Envoy with nginx

This is a tutorial (and a memo for me) on how to set up gRPC-Web to proxy through nginx into Envoy and from there into a gRPC server. With TLS.

This is on Ubuntu 20.04. Update your package lists to start:

sudo apt-get update

I assume you have a domain pointed to your box, and from now I’ll refer to it as $DOMAIN.

Create basic gRPC service definition

// sample.proto
syntax = "proto3";

package sample;

service SampleRPC {
  rpc Ping(Msg) returns (Msg);
}

message Msg {
  int64 nonce = 1;
}

Take note of the package name, sample, and the service name, SampleRPC.

Install gRPC and compile service

Get Python 3 and pip:

sudo apt-get install python3 python3-pip

You can do this in a virtualenv if you wish, and it works:

pip3 install grpcio grpcio-tools
python3 -m grpc.tools.protoc -I. --python_out=. --grpc_python_out=. sample.proto

This will generate sample_pb2.py and sample_pb2_grpc.py.

Create a simple Python gRPC server

Create a server implementation, note the names from the section above.

# server.py
from concurrent import futures

import grpc
import sample_pb2, sample_pb2_grpc


class Servicer(sample_pb2_grpc.SampleRPCServicer):
    def Ping(self, request, context):
        print(f"Received request with nonce={request.nonce}")
        return sample_pb2.Msg(nonce=request.nonce)


server = grpc.server(futures.ThreadPoolExecutor(2))
sample_pb2_grpc.add_SampleRPCServicer_to_server(Servicer(), server)
server.add_insecure_port("[::]:50051")
server.start()
server.wait_for_termination()

You can run it with python3 server.py.

Install Docker

Instructions here, or short version:

sudo apt-get install apt-transport-https ca-certificates curl gnupg-agent software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io
# optional, allows running docker without sudo
sudo usermod -aG docker ubuntu

You need to log out and back in for the last command to take effect.

Set up Envoy

Note: this config is using the outdated Envoy v2 config instead of the new v3 config, see the Envoy JSON to gRPC transcoding post post for a newer config sample.

Create an envoy.yaml:

# envoy.yaml
admin:
  access_log_path: /tmp/admin_access.log
  address:
    socket_address: { address: 0.0.0.0, port_value: 9901 }

static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address: { address: 0.0.0.0, port_value: 5000 }
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        config:
          codec_type: auto
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              routes:
              - match: { prefix: "/" }
                route:
                  cluster: sample_cluster
                  max_grpc_timeout: 0s
              cors:
                allow_origin_string_match:
                - prefix: "*"
                allow_methods: GET, PUT, DELETE, POST, OPTIONS
                allow_headers: keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web,grpc-timeout
                max_age: "1728000"
                expose_headers: grpc-status,grpc-message
          http_filters:
          - name: envoy.filters.http.grpc_web
          - name: envoy.filters.http.cors
          - name: envoy.filters.http.router
  clusters:
  - name: sample_cluster
    connect_timeout: 0.25s
    type: logical_dns
    http2_protocol_options: {}
    lb_policy: round_robin
    hosts: [{ socket_address: { address: localhost, port_value: 50051 }}]

Then a Dockerfile:

# Dockerfile
FROM envoyproxy/envoy:v1.14.3

COPY ./envoy.yaml /etc/envoy/envoy.yaml

EXPOSE 5000
EXPOSE 9901

CMD /usr/local/bin/envoy -c /etc/envoy/envoy.yaml

Build the container:

docker build -t sample/envoy .

Run it:

docker run -d --net=host sample/envoy

This will launch in the background, forwarding ports 5000 (where Envoy is listening for gRPC-Web traffic) and 9901 (Envoy admin page) to your box. When you deploy this, it’s probably good to disable/block port 9901, otherwise anyone can go poke at your proxy settings.

Install nginx and get a TLS cert

Get nginx:

sudo apt-get install nginx

You need a domain name, Let’s Encrypt won’t hand your IP address a cert.

sudo apt-get install certbot python3-certbot-nginx
sudo certbot --nginx --redirect -n -d $DOMAIN --agree-tos -m certs@$DOMAIN

The --redirect flag causes certbot to set up a redirect from HTTP to HTTPS. -n means non-interactive, -m is the email to receive notifications to.

Install Node and Yarn

Instructions here, short version (this will also install Node):

curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
sudo apt update
sudo apt install yarn

Install `protoc` with the JavaScript plugin and `protoc-gen-grpc-web`

Okay, so apparently the Python stuff doesn’t come with the JavaScript stuff, so here we go:

export GRPC_WEB_VERSION=1.2.0
wget -O protoc-gen-grpc-web https://github.com/grpc/grpc-web/releases/download/$GRPC_WEB_VERSION/protoc-gen-grpc-web-$GRPC_WEB_VERSION-linux-x86_64
chmod +x protoc-gen-grpc-web
export PROTOC_VERSION=3.12.3
wget -O protoc.zip https://github.com/protocolbuffers/protobuf/releases/download/v$PROTOC_VERSION/protoc-$PROTOC_VERSION-linux-x86_64.zip
sudo apt-get install unzip
unzip protoc.zip

Then generate the JavaScript code:

./bin/protoc -I. --js_out="import_style=commonjs,binary:." --plugin=protoc-gen-grpc-web=./protoc-gen-grpc-web --grpc-web_out="import_style=commonjs,mode=grpcweb:." sample.proto

Creating a client

I’m not a frontend guy, but here’s Aapeli’s Quick Intro to the JavaScript World™. Set up a basic yarn environment:

yarn add webpack webpack-cli grpc-web google-protobuf

You need webpack as gRPC-Web has no way of generating code without imports.

Now create a basic script:

// index.js
const { Msg } = require("./sample_pb.js")
const { SampleRPCPromiseClient } = require("./sample_grpc_web_pb.js")

const client = new SampleRPCPromiseClient("https://$DOMAIN/rpc")
const req = new Msg()
req.setNonce(42)
client.ping(req, null).then(res => {
    document.getElementById("out").innerHTML = "Got response from server, nonce: " + res.getNonce()
}).catch(console.error)

Note the $DOMAIN! Then create a page:

<!-- index.html -->
<!DOCTYPE html>
<html>
    <head>
        <meta charset="UTF-8">
        <script src="dist/main.js"></script>
    </head>
    <body>
        <div id="out">Pinging server...</div>
    </body>
</html>

Finally compile it:

yarn webpack index.js

Now drop it somewhere where it can be served.

Configure nginx

These edits go in /etc/nginx/sites-enabled/default. Add the following section to your server:

# /etc/nginx/sites-enabled/default, server section
location /rpc/ {
    proxy_http_version 1.1;
    proxy_pass http://127.0.0.1:5000/;
}

Then restart nginx:

sudo systemctl restart nginx

Serving the web stuff from your home folder

To serve your home folder with nginx, modify the root directive in the server section:

# /etc/nginx/sites-enabled/default, server section
root /home/ubuntu;

Note that this is probably not a very bright idea in the long run.

Also responding to vanilla gRPC queries

If you also want to service vanilla gRPC queries, you need to do two things. Add http2 to the listen directives like this:

# /etc/nginx/sites-enabled/default, server section
listen [::]:443 ssl http2 ipv6only=on; # managed by Certbot
listen 443 ssl http2; # managed by Certbot

Then add a section in that same server block:

# /etc/nginx/sites-enabled/default, same server section as above
location /sample. {
    grpc_pass 127.0.0.1:50051;
}

Note the sample which is your protobuf package name.

Restart nginx again:

sudo systemctl restart nginx

Testing it out

Make sure the Python server, the Envoy proxy, and nginx are all running.

Now if you visit that site, the page body should change to something like Got response from server, nonce: 42.

Other bits

If you enabled vanilla gRPC queries, then this will connect through nginx where TLS is terminated directly to the Python server, should work from the internet:

# secure_client.py
import grpc
import sample_pb2, sample_pb2_grpc

stub = sample_pb2_grpc.SampleRPCStub(grpc.secure_channel("$DOMAIN:443", credentials=grpc.ssl_channel_credentials()))

response = stub.Ping(sample_pb2.Msg(nonce=5))
print(f"Received response from server with nonce={response.nonce}")

This will connect directly to the Python server and will only succeed from your server box:

# insecure_client.py
import grpc
import sample_pb2, sample_pb2_grpc

stub = sample_pb2_grpc.SampleRPCStub(grpc.insecure_channel("127.0.0.1:50051"))

response = stub.Ping(sample_pb2.Msg(nonce=5))
print(f"Received response from server with nonce={response.nonce}")