Kong v3.0.0 Release Notes

  • ๐Ÿš€ > Released 2022/09/12

    ๐Ÿš€ This major release adds a new router written in Rust and a tracing API that is compatible with the OpenTelemetry API spec. Furthermore, ๐ŸŽ various internal changes have been made to improve Kong's performance ๐Ÿš€ and memory consumption. As it is a major release, users are advised to review the list of braking changes to determine whether โฌ†๏ธ configuration changes are needed when upgrading.

    ๐Ÿ’ฅ Breaking Changes

    ๐Ÿš€ Deployment
    • Kong schema library's process_auto_fields function will not any more make a deep copy of data that is passed to it when the given context is "select". This was done to avoid excessive deep copying of tables where we believe the data most of the time comes from a driver like pgmoon or lmdb. If a custom plugin relied on process_auto_fields not overriding the given table, it must make its own copy before passing it to the function now. #8796
    • ๐Ÿšš The deprecated shorthands field in Kong Plugin or DAO schemas was removed in favor or the typed shorthand_fields. If your custom schemas still use shorthands, you need to update them to use shorthand_fields. #8815
    • ๐Ÿšš The support for legacy = true/false attribute was removed from Kong schemas and Kong field schemas. #8958
    • The deprecated alias of Kong.serve_admin_api was removed. If your custom Nginx templates still use it, please change it to Kong.admin_content. #8815
    • ๐Ÿšš The Kong singletons module "kong.singletons" was removed in favor of the PDK kong.*. #8874
    • ๐Ÿšš The dataplane config cache was removed. The config persistence is now done automatically with LMDB. #8704
    • โž• ngx.ctx.balancer_address does not exist anymore, please use ngx.ctx.balancer_data instead. #9043
    • We have changed the normalization rules for route.path: Kong stores the unnormalized path, but regex path always pattern matches with the normalized URI. We used to replace percent-encoding in regex path pattern to ensure different forms of URI matches. That is no longer supported. Except for reserved characters defined in rfc3986, we should write all other characters without percent-encoding. #9024
    • Kong will no longer use an heuristic to guess whether a route.path is a regex pattern. From now 3.0 onwards, all regex paths must start with the "~" prefix, and all paths that don't start with "~" will be considered plain text. The migration process should automatically convert the regex paths when upgrading from 2.x to 3.0 #9027
    • Bumping version number (_format_version) of declarative configuration to "3.0" for changes on route.path. Declaritive configuration with older version are upgraded to "3.0" on the fly. #9078
    • โœ‚ Removed deprecated config.functions from serverless-functions plugin's schema, please use config.access phase instead. #8559
    • ๐Ÿท Tags may now contain space characters. #9143
    • ๐Ÿš€ The Secrets Management feature, which has been in beta since release 2.8.0, is now included as a regular feature. #8871 #9217
    Admin API
    • โšก๏ธ POST requests on Targets endpoint are no longer able to update existing entities, they are only able to create new ones. #8596, #8798. If you have scripts that use POST requests to modify Targets, you should change them to PUT requests to the appropriate endpoints before updating to Kong 3.0.
    • โšก๏ธ Insert and update operations on duplicated Targets returns 409. #8179, #8768
    • ๐Ÿ”Œ The list of reported plugins available on the server now returns a table of metadata per plugin instead of a boolean true. #8810
    • The kong.request.get_path() PDK function now performs path normalization on the string that is returned to the caller. The raw, non-normalized version of the request path can be fetched via kong.request.get_raw_path(). #8823
    • pdk.response.set_header(), pdk.response.set_headers(), pdk.response.exit() now ignore and emit warnings for manually set Transfer-Encoding headers. #8698
    • The PDK is no longer versioned #8585
    • The JavaScript PDK now returns Uint8Array for kong.request.getRawBody, kong.response.getRawBody and kong.service.response.getRawBody. The Python PDK returns bytes for kong.request.get_raw_body, kong.response.get_raw_body, kong.service.response.get_raw_body. All these funtions used to return strings in the past. #8623
    ๐Ÿ”Œ Plugins
    • ๐Ÿ”Œ DAOs in plugins must be listed in an array, so that their loading order is explicit. Loading them in a hash-like table is no longer supported. #8988
    • ๐Ÿ”Œ Plugins MUST now have a valid PRIORITY (integer) and VERSION ("x.y.z" format) field in their handler.lua file, otherwise the plugin will fail to load. #8836
    • ๐Ÿšš The old kong.plugins.log-serializers.basic library was removed in favor of the PDK function kong.log.serialize, please upgrade your plugins to use PDK. #8815
    • ๐Ÿšš The support for deprecated legacy plugin schemas was removed. If your custom plugins still use the old (0.x era) schemas, you are now forced to upgrade them. #8815
    • ๐Ÿ”Œ Some plugins received new priority values. This is important for those who run custom plugins as it may affect the sequence your plugins are executed. Note that this does not change the order of execution for plugins in a standard kong installation. List of plugins and their old and new priority value:
      • acme changed from 1007 to 1705
      • basic-auth changed from 1001 to 1100
      • hmac-auth changed from 1000 to 1030
      • jwt changed from 1005 to 1450
      • key-auth changed from 1003 to 1250
      • ldap-auth changed from 1002 to 1200
      • oauth2 changed from 1004 to 1400
      • rate-limiting changed from 901 to 910
    • ๐ŸŒฒ HTTP-log: headers field now only takes a single string per header name, where it previously took an array of values #6992
    • AWS Lambda: aws_region field must be set through either plugin config or environment variables, allow both host and aws_region fields, and always apply SigV4 signature. #8082
    • Serverless Functions Removed deprecated config.functions, please use config.access instead. #8559
    • Serverless Functions: The pre-functions plugin changed priority from +inf to 1000000. #8836
    • JWT: The authenticated JWT is no longer put into the nginx context (ngx.ctx.authenticated_jwt_token). Custom plugins which depend on that value being set under that name must be updated to use Kong's shared context instead (kong.ctx.shared.authenticated_jwt_token) before upgrading to 3.0
    • Prometheus: The prometheus metrics have been reworked extensively for 3.0.
      • Latency has been split into 4 different metrics: kong_latency_ms, upstream_latency_ms and request_latency_ms (http) /tcp_session_duration_ms (stream). Buckets details below.
      • Separate out Kong Latency Bucket values and Upstream Latency Bucket values.
      • consumer_status removed.
      • request_count and consumer_status have been merged into just http_requests_total. If the per_consumer config is set false, the consumer label will be empty. If the per_consumer config is true, it will be filled.
      • http_requests_total has a new label source, set to either exit, error or service.
      • New Metric: node_info. Single gauge set to 1 that outputs the node's id and kong version.
      • All Memory metrics have a new label node_id
      • nginx_http_current_connections merged with nginx_stream_current_connection into nginx_current_connections #8712
    • Prometheus: The plugin doesn't export status codes, latencies, bandwidth and upstream healthcheck metrics by default. They can still be turned on manually by setting status_code_metrics, latency_metrics, bandwidth_metrics and upstream_health_metrics respectively. Enabling those metrics will impact the performance if you have a large volume of Kong entities, we recommend using the statsd plugin with the push model if that is the case. And now prometheus plugin new grafana dashboard updated #9028
    • ACME: allow_any_domain field added. It is default to false and if set to true, the gateway will ignore the domains field. #9047
    • Statsd:
      • The metric name that is related to the service has been renamed by adding a service. prefix. e.g. kong.service.<service_identifier>.request.count #9046
      • The metric kong.<service_identifier>.request.status.<status> and kong.<service_identifier>.user.<consumer_identifier>.request.status.<status> has been renamed to kong.service.<service_identifier>.status.<status> and kong.service.<service_identifier>.user.<consumer_identifier>.status.<status> #9046
      • The metric *.status.<status>.total from metrics status_count and status_count_per_user has been removed #9046
    • Proxy-cache: The plugin does not store the response data in ngx.ctx.proxy_cache_hit anymore. Logging plugins that need the response data must read it from kong.ctx.shared.proxy_cache_hit from Kong 3.0 on. #8607
    • Rate-limiting: The default policy is now local for all deployment modes. #9344
    • Response-rate-limiting: The default policy is now local for all deployment modes. #9344

    ๐Ÿ—„ Deprecations

    • ๐Ÿ”Œ The go_pluginserver_exe and go_plugins_dir directives are no longer supported. #8552. If you are using Go plugin server, please migrate your plugins to use the Go PDK before upgrading.
    • The migration helper library (mostly used for Cassandra migrations) is no longer supplied with Kong #8781
    • ๐Ÿ—„ The path_handling algorithm v1 is deprecated and only supported when router_flavor config option is set to traditional. #9290
    ๐Ÿ”ง Configuration
    • The Kong constant CREDENTIAL_USERNAME with value of X-Credential-Username was removed. Kong plugins in general have moved (since #5516) to use constant CREDENTIAL_IDENTIFIER with value of X-Credential-Identifier when setting the upstream headers for a credential. #8815
    • Change the default of lua_ssl_trusted_certificate to system #8602 to automatically load trusted CA list from system CA store.
    • โœ‚ Remove a warning of AAAA being experimental with dns_order.
    • It is no longer possible to use a .lua format to import a declarative config from the kong command-line tool, only json and yaml are supported. If your update procedure with kong involves executing kong config db_import config.lua, please create a config.json or config.yml and use that before upgrading. #8898
    • We bumped the version number (_format_version) of declarative configuration to "3.0" because of changes on route.path. Declarative configuration with older version shoudl be upgraded to "3.0" on the fly. #9078
    • Postgres migrations can now have an up_f part like Cassandra migrations, designating a function to call. The up_f part is invoked after the up part has been executed against the database for both Postgres and Cassandra.
    • A new CLI command, kong migrations status, generates the status on a JSON file.


    • โฌ†๏ธ Bumped OpenResty from to #8850
    • โฌ†๏ธ Bumped pgmoon from 1.13.0 to 1.15.0 #8908 #8429
    • โฌ†๏ธ Bumped OpenSSL from 1.1.1n to 1.1.1q #9074 #8544 #8752 #8994
    • โฌ†๏ธ Bumped resty.openssl from 0.8.8 to 0.8.10 #8592 #8753 #9023
    • โฌ†๏ธ Bumped inspect from 3.1.2 to 3.1.3 #8589
    • โฌ†๏ธ Bumped resty.acme from 0.7.2 to 0.8.1 #8680 #9165
    • โฌ†๏ธ Bumped luarocks from 3.8.0 to 3.9.1 #8700 #9204
    • โฌ†๏ธ Bumped luasec from 1.0.2 to 1.2.0 #8754 #8754
    • โฌ†๏ธ Bumped resty.healthcheck from 1.5.0 to 1.6.1 #8755 #9018 #9150
    • โฌ†๏ธ Bumped resty.cassandra from 1.5.1 to 1.5.2 #8845
    • โฌ†๏ธ Bumped penlight from 1.12.0 to 1.13.1 #9206
    • โฌ†๏ธ Bumped lua-resty-mlcache from 2.5.0 to 2.6.0 #9287

    โž• Additions

    ๐ŸŽ Performance
    • Do not register unnecessary event handlers on Hybrid mode Control Plane nodes #8452.
    • ๐ŸŽ Use the new timer library to improve performance, except for the plugin server. #8912
    • 0๏ธโƒฃ Increased use of caching for DNS queries by activating additional_section by default #8895
    • pdk.request.get_header changed to a faster implementation, not to fetch all headers every time it's called #8716
    • ๐Ÿ”Œ Conditional rebuilding of router, plugins iterator and balancer on DP #8519, #8671
    • Made config loading code more cooperative by yielding #8888
    • ๐Ÿ‘‰ Use LuaJIT encoder instead of JSON to serialize values faster in LMDB #8942
    • ๐Ÿšš Move inflating and JSON decoding non-concurrent, which avoids blocking and makes DP reloads faster #8959
    • Stop duplication of some events #9082
    • ๐Ÿ‘Œ Improve performance of config hash calculation by using string buffer and tablepool #9073
    • โฌ‡๏ธ Reduce cache usage in dbless by not using the kong cache for Routes and Services in LMDB #8972
    • Implemented delayed response in stream mode #6878
    • โž• Added cache_key on target entity for uniqueness detection. #8179
    • Introduced the tracing API which compatible with OpenTelemetry API spec and add build-in instrumentations. The tracing API is intend to be used with a external exporter plugin. Build-in instrumentation types and sampling rate are configuable through opentelemetry_tracing and opentelemetry_tracing_sampling_rate options. #8724
    • Added path, uri_capture, and query_arg options to upstream hash_on for load balancing. #8701
    • Introduced unix domain socket based lua-resty-events to replace shared memory based lua-resty-worker-events. #8890
    • Introduced a new router implementation atc-router, which is written in Rust. #8938
    • Introduce a new field for entities table_name that allows to specify a table name. Before the name was deduced by the entity name attribute. #9182
    • โž• Added headers on active healthcheck for upstreams. #8255
    • Target entities using hostnames were resolved when they were not needed. Now when a target is removed or updated, the DNS record associated with it is removed from the list of hostnames to be resolved. #8497 9265
    • ๐Ÿ‘Œ Improved error handling and debugging info in the DNS code #8902
    • Kong will now attempt to recover from an unclean shutdown by detecting and removing dangling unix sockets in the prefix directory #9254
    Admin API
    • โž• Added a new API /timers to get the timer statistics. #8912 and worker info #8999
    • ๐Ÿ”Œ / endpoint now includes plugin priority #8821
    Hybrid Mode
    • โž• Add wRPC protocol support. Now configuration synchronization is over wRPC. wRPC is an RPC protocol that encodes with ProtoBuf and transports with WebSocket. #8357
    • To keep compatibility with earlier versions, add support for CP to fall back to the previous protocol to support old DP. #8834
    • โž• Add support to negotiate services supported with wRPC protocol. We will support more services than config sync over wRPC in the future. #8926
    • Declarative config exports happen inside a transaction in Postgres #8586
    ๐Ÿ”Œ Plugins
    • ๐Ÿ”€ Sync all plugin versions to the Kong version #8772
    • Introduced the new OpenTelemetry plugin that export tracing instrumentations to any OTLP/HTTP compatible backend. opentelemetry_tracing configuration should be enabled to collect the core tracing spans of Kong. #8826
    • Zipkin: add support for including HTTP path in span name through configuration property http_span_name. #8150
    • Zipkin: add support for socket connect and send/read timeouts through configuration properties connect_timeout, send_timeout, and read_timeout. This can help mitigate ngx.timer saturation when upstream collectors are unavailable or slow. #8735
    • AWS-Lambda: add support for cross account invocation through configuration properties aws_assume_role_arn and aws_role_session_name.#8900 #8900
    • AWS-Lambda: accept string type statusCode as valid return when working in proxy integration mode. #8765
    • AWS-Lambda: separate aws credential cache by IAM role ARN #8907
    • Statsd: :fireworks: Newly open-sourced plugin capabilities: All capabilities of Statsd Advanced are now bundled in Statsd. #9046
    ๐Ÿ”ง Configuration
    • ๐Ÿ”ง A new configuration item (openresty_path) has been added to allow developers/operators to specify the OpenResty installation to use when running Kong (instead of using the system-installed OpenResty) #8412
    • Add ipv6only to listen options (e.g. KONG_PROXY_LISTEN) #9225
    • Add so_keepalive to listen options (e.g. KONG_PROXY_LISTEN) #9225
    • โž• Add LMDB dbless config persistence and removed the JSON based config cache for faster startup time #8670
    • nginx_events_worker_connections=auto has a lower bound of 1024 #9276
    • nginx_main_worker_rlimit_nofile=auto has a lower bound of 1024 #9276
    • Added new PDK function: kong.request.get_start_time() #8688
    • kong.db.*.cache_key() falls back to .id if nothing from cache_key is found #8553

    ๐Ÿ›  Fixes

    • The schema validator now correctly converts null from declarative configurations to nil. #8483
    • โฑ Only reschedule router and plugin iterator timers after finishing previous execution, avoiding unnecessary concurrent executions. #8567
    • ๐Ÿ”Œ External plugins now handle returned JSON with null member correctly. #8611
    • ๐Ÿ›  Fixed an issue where the address of the environ variable could change but the code didn't assumed it was fixed after init #8581
    • ๐Ÿ›  Fix issue where the Go plugin server instance would not be updated after a restart (e.g., upon a plugin server crash). #8547
    • ๐Ÿ›  Fixed an issue on trying to reschedule the DNS resolving timer when Kong was being reloaded. #8702
    • ๐Ÿ›ฐ The private stream API has been rewritten to allow for larger message payloads #8641
    • ๐Ÿ›  Fixed an issue that the client certificate sent to upstream was not updated when calling PATCH Admin API #8934
    • Fixed an issue where the CP and wRPC modules would cause Kong to crash when calling export_deflated_reconfigure_payload without a pcall #8668
    • ๐Ÿšš Moved all .proto files to /usr/local/kong/include and ordered by priority. #8914
    • ๐Ÿ›  Fixed an issue that cause unexpected 404 error on creating/updating configs with invalid options #8831
    • ๐Ÿ›  Fixed an issue that causes crashes when calling some PDK APIs #8604
    • ๐Ÿ›  Fixed an issue that cause crashes when go PDK calls return arrays #8891
    • ๐Ÿ”Œ Plugin servers now shutdowns gracefully when Kong exits #8923
    • 0๏ธโƒฃ CLI now prompts with [y/n] instead of [Y/n], as it does not take y as default #9114
    • ๐Ÿ‘Œ Improved the error message when Kong cannot connect to Cassandra on init #8847
    • ๐Ÿ›  Fixed an issue where Vault Subschema wasn't loaded in off strategy #9174
    • The Schema now runs select transformations before process_auto_fields #9049
    • ๐Ÿ›  Fixed an issue where Kong would use too many timers to keep track of upstreams when worker_consistency=eventual #8694, #8858
    • ๐Ÿ›  Fixed an issue where it wasn't possible to set target status using only a hostname for targets set only by their hostname #8797
    • ๐Ÿ›  Fixed pagination issue when getting to the second page while iterationg over a foreign key field using the DAO #9255
    • ๐Ÿ›  Fixed an issue where cache entries of some entities were not being properly invalidated after a cascade delete #9261
    • โš™ Running kong start when Kong is already running will no longer clobber the existing .kong_env file #9254
    Admin API
    • ๐Ÿ‘Œ Support HTTP/2 when requesting /status #8690
    ๐Ÿ”Œ Plugins
    • ๐Ÿ”Œ Plugins with colliding priorities have now deterministic sorting based on their name #8957
    • ๐Ÿ”Œ External Plugins: better handling of the logging when a plugin instance loses the instances_id in an event handler #8652
    • ACME: auth_method default value is set to token #8565
    • ACME: Added cache for domains_matcher #9048
    • syslog: conf.facility default value is now set to user #8564
    • AWS-Lambda: Removed proxy_scheme field from schema #8566
    • AWS-Lambda: Change path from request_uri to upstream_uri, fix uri can not follow the rule defined in the request-transformer configuration #9058 #9129
    • hmac-auth: Removed deprecated signature format using ngx.var.uri #8558
    • โœ‚ Remove deprecated blacklist/whitelist config fields from bot-detection, ip-restriction and ACL plugins. #8560
    • Zipkin: Correct the balancer spans' duration to include the connection time from Nginx to the upstream. #8848
    • Zipkin: Correct the calculation of the header filter start time #9230
    • Zipkin: Compatibility with the latest Jaeger header spec, which makes parent_id optional #8352
    • LDAP-Auth: Refactored ASN.1 parser using OpenSSL API through FFI. #8663
    • Rate-Limiting and Response-ratelimiting: Fix a disordered behaviour caused by pairs function which may cause Postgres DEADLOCK problem #8968
    • Response-rate-Limiting: Fix a disordered behaviour caused by pairs function which may cause Postgres DEADLOCK problem #8968
    • gRPC gateway: Fix the handling of boolean fields from URI arguments #9180
    • Serverless Functions: Fix problem that could result in a crash #9269
    • Azure-functions: Support working without dummy service #9177
    • The cluster listener now uses the value of admin_error_log for its log file instead of proxy_error_log #8583
    • ๐Ÿ›  Fixed a typo in some business logic that checks the Kong role before setting a value in cache at startup #9060
    • ๐Ÿ›  Fixed DP get zero size config while service with plugin-enabled route is disabled #8816
    • Localize config_version to avoid a race condition from the new yielding config loading code #8188
    • ๐Ÿ”Œ kong.response.get_source() now return an error instead of an exit when plugin throws runtime exception on access phase #8599
    • kong.tools.uri.normalize() now does escaping of reserved and unreserved characters more correctly #8140