Customize the Default Log Format in Kong API Gateway

This is a continuation of my article series on Kong API gateway and this one will go though the steps to change the default log format of the Kong API gateway with the desired style.

Role of Nginx

Kong is built on NGINX, and uses Lua to implement its API functionality, whereas the API management module relies completely on native, high‑performance capabilities that are implemented as NGINX Plus modules. So the format of the log lines in Kong is determined by corresponding configuration in the NGINX service.

# Sample Log Line from Default log Pattern
172.10.24.50 - - [05/May/2021:08:19:40 +0000] "POST /api/service/getinfo HTTP/1.1" 200 2313 "-" "curl/7.61.1"

# Parameters in the above log line
$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"

As you can see in the above default log line, it only contain few basic information. But you might need additional parameters for your use-case. But one important point to remember is including unnecessary information in the logs or defining a bulky log structure will hamper the performance of the API gateway.

There are few options for us to change the default log format with our intended style.

Injecting Nginx directives

Tweaking the Nginx configuration of your Kong instances allows you to update the default configurations. When Kong starts, it builds an Nginx configuration file. You can inject custom Nginx directives to this file directly via your Kong configuration.

a) Define the desired log format using the nginx_http_log_format parameter in the kong.conf;

nginx_http_log_format=show_everything '\$time_iso8601 - \$bytes_sent - \$request - \$status - \$remote_addr'

b) Then point the updated log pattern to the default access log configuration of the kong.conf;

proxy_access_log=/dev/stdout show_everything

You can read more information on this method from here.

Environment variables

This is my favorite method and one of the reasons which make Kong super easy to use. When loading properties out of a configuration file, Kong will also look for environment variables of the same name. This allows you to fully configure Kong via environment variables, which is very convenient for container-based infrastructures, for example:

To override a setting using an environment variable, declare an environment variable with the name of the setting, prefixed with KONG_ and capitalized.

a) In Docker, you would use environment variables as per the usual Docker configuration;

-e "KONG_NGINX_HTTP_LOG_FORMAT=show_everything '\$time_iso8601 - \$bytes_sent - \$request - \$status - \$remote_addr'"

b) Then inject this configuration to Kong configuration via the below environment variables;

-e "KONG_PROXY_ACCESS_LOG=/dev/stdout show_everything"

When you define your own log pattern, this documentation will be useful to understand the variables available in NGINX log module.

My Log Format

For my case, I wanted a very simple log format with minimal information required to do any troubleshooting. So what I wanted to see for the successful API requests were,

• Request source IP
• Request time
• Response time
• API Request information (HTTP method, Context path etc )
• API Response code
• User Agent Header (to identify the requestor)

$remote_addr - $remote_user [$time_local] $start_time "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"

Additionally, for a failed API request by default Kong will print extra information about the error details .

The $time_local parameter in the default log line is the response time for the API request. So for me to obtain the request time, I had to configure the below additional environment variable

# To get the request time, add the below env variable
KONG_NGINX_PROXY_SET_BY_LUA=$$start_time 'return ngx.now()'

If you also need to print the request duration time, you ca

So below are all the environment variables I had to configure for my docker image.

# To get the request time, add the below env variable
KONG_NGINX_PROXY_SET_BY_LUA=$$start_time 'return ngx.now()'
# Below env variables are to define your log pattern:
KONG_PROXY_ACCESS_LOG=/dev/stdout show_everything
KONG_NGINX_HTTP_LOG_FORMAT=show_everything '$remote_addr - $remote_user [$time_local] $start_time - "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"'

The resultant log pattern will be like,

172.18.240.50 - - [05/May/2021:08:59:49 +0000] 1620205189.259 - "POST /api/service/getinfo HTTP/1.1" 200 2313 "-" "curl/7.61.1"
# Format of the log line:
# {Source IP} - {Remote User} [{Response time}] {Request time} .....

Additionally if you need to log the request duration, then you can use the variable $request_time when defining the log format. So the resulting log format will be like

KONG_NGINX_PROXY_SET_BY_LUA=$$start_time 'return ngx.now()'
KONG_PROXY_ACCESS_LOG=/dev/stdout show_everything
KONG_NGINX_HTTP_LOG_FORMAT=show_everything '$remote_addr - $remote_user [$time_local - $msec] $start_time - $request_time "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"'

# Resulting Log Format:
# {Source IP} - {Remote User} [{Response time} - {Response time in milliseconds}] {Request time} - {Duration of the request} .....
172.18.240.50 - - [05/May/2021:08:59:49 +0000 - 1620205189.300] 1620205189.259 - 0.041 "POST /api/service/getinfo HTTP/1.1" 200 2313 "-" "curl/7.61.1"

Time stamps in the above log lines are in the unix epoch time. So we need to use the epoch convertor to covert the ms time to UTC or required time. https://www.epochconverter.com

Below is a sample converted timestamp from the above log line,

1620205189.300:

• Wednesday, May 5, 2021 8:59:49.300 AM
• Wednesday, May 5, 2021 4:59:49.300 PM GMT+08:00

1620205189.259:

• Wednesday, May 5, 2021 8:59:49.259 AM
• Wednesday, May 5, 2021 4:59:49.259 PM GMT+08:00

If you subtract these two values, it will notice that it’s giving a value of 0.041 which is the same value of request duration time as in the above log line.

Thanks!