OpenTelemetry No Traces When Auth Plugin Rejects Request

May 9, 2025 by ADMIN 57 views

Introduction

OpenTelemetry is a popular open-source observability framework that provides a unified way to collect and manage telemetry data from various sources. When used in conjunction with authentication plugins in Kong, it's essential to ensure that OpenTelemetry traces are generated even when authentication fails or Kong returns an early response. In this article, we'll explore the issue of OpenTelemetry not generating traces when an authentication plugin rejects a request and discuss possible solutions.

Background

Kong is a popular open-source API gateway that provides a robust set of features for managing APIs, including authentication, rate limiting, and caching. OpenTelemetry, on the other hand, is a framework for collecting and managing telemetry data from various sources, including APIs. When used together, Kong and OpenTelemetry provide a powerful way to monitor and analyze API traffic.

The Issue

When an authentication plugin in Kong rejects a request, OpenTelemetry does not generate a trace. This is because the authentication plugin is executed before the OpenTelemetry plugin, and the request is terminated early, preventing the OpenTelemetry plugin from generating a trace.

Expected Behavior

The expected behavior is that the OpenTelemetry plugin should have a higher priority than the authentication plugins, so that it can generate a trace even when authentication fails or Kong returns an early response. This would provide valuable insights into API traffic and help identify issues that may not be immediately apparent.

Current Behavior

The current behavior is that the OpenTelemetry plugin is executed at a lower priority (14) than the authentication plugins. This means that when an authentication plugin rejects a request, the OpenTelemetry plugin is not executed, and no trace is generated.

Comparison with Zipkin

For comparison, the Zipkin plugin, which is a similar tracing plugin to OpenTelemetry, has a much higher priority (100000) than the OpenTelemetry plugin. This suggests that the OpenTelemetry plugin should also have a higher priority to ensure that it can generate traces even when authentication fails or Kong returns an early response.

Dynamic Plugin Ordering

Kong provides a feature called dynamic plugin ordering, which allows plugins to be reordered based on certain conditions. However, this feature is enterprise-only and has its downsides. Therefore, it's not a viable solution for this issue.

Possible Solutions

Increase the priority of the OpenTelemetry plugin: The first possible solution is to increase the priority of the OpenTelemetry plugin to a higher value, such as 100000, to ensure that it is executed before the authentication plugins.
Modify the authentication plugins: Another possible solution is to modify the authentication plugins to allow the OpenTelemetry plugin to generate a trace even when authentication fails or Kong returns an early response.
Use a different tracing plugin: If the above solutions are not feasible, a different tracing plugin, such as Zipkin, can be used instead of OpenTelemetry.

Conclusion

In conclusion, the issue of OpenTelemetry not generating traces when an authentication plugin rejects a request is a significant one. By increasing the priority of the OpenTelemetry plugin, modifying the plugins, or using a different tracing plugin, it's possible to ensure that OpenTelemetry traces are generated even when authentication fails or Kong returns an early response.

Recommendations

Increase the priority of the OpenTelemetry plugin: This is the recommended solution, as it ensures that the OpenTelemetry plugin is executed before the authentication plugins.
Modify the authentication plugins: This solution requires modifications to the authentication plugins, but it can provide valuable insights into API traffic.
Use a different tracing plugin: If the above solutions are not feasible, a different tracing plugin, such as Zipkin, can be used instead of OpenTelemetry.

Future Work

In the future, it would be beneficial to explore ways to improve the dynamic plugin ordering feature in Kong, so that plugins can be reordered based on certain conditions without requiring enterprise-only features.

References

OpenTelemetry documentation
Kong documentation
Zipkin documentation
OpenTelemetry No Traces When Auth Plugin Rejects Request: Q&A ===========================================================

Introduction

In our previous article, we discussed the issue of OpenTelemetry not generating traces when an authentication plugin rejects a request in Kong. In this article, we'll provide a Q&A section to address some of the common questions and concerns related to this issue.

Q: What is the root cause of the issue?

A: The root cause of the issue is that the authentication plugin is executed before the OpenTelemetry plugin, and the request is terminated early, preventing the OpenTelemetry plugin from generating a trace.

Q: Why is the OpenTelemetry plugin not executed before the authentication plugins?

A: The OpenTelemetry plugin is executed at a lower priority (14) than the authentication plugins. This is because the authentication plugins are typically executed before the OpenTelemetry plugin in the plugin chain.

Q: Can I increase the priority of the OpenTelemetry plugin?

A: Yes, you can increase the priority of the OpenTelemetry plugin to a higher value, such as 100000, to ensure that it is executed before the authentication plugins.

Q: What are the implications of increasing the priority of the OpenTelemetry plugin?

A: Increasing the priority of the OpenTelemetry plugin will ensure that it is executed before the authentication plugins, but it may also have some implications, such as:

The OpenTelemetry plugin may be executed before other plugins that rely on the authentication plugins.
The OpenTelemetry plugin may not be able to access the authentication plugins' data.

Q: Can I modify the authentication plugins to allow the OpenTelemetry plugin to generate a trace?

A: Yes, you can modify the authentication plugins to allow the OpenTelemetry plugin to generate a trace. However, this may require significant changes to the authentication plugins and may have some implications, such as:

The authentication plugins may need to be modified to allow the OpenTelemetry plugin to access their data.
The authentication plugins may need to be modified to allow the OpenTelemetry plugin to generate a trace even when the authentication fails.

Q: What are the implications of modifying the authentication plugins?

A: Modifying the authentication plugins to allow the OpenTelemetry plugin to generate a trace may have some implications, such as:

The authentication plugins may need to be modified to allow the OpenTelemetry plugin to access their data.
The authentication plugins may need to be modified to allow the OpenTelemetry plugin to generate a trace even when the authentication fails.

Q: Can I use a different tracing plugin instead of OpenTelemetry?

A: Yes, you can use a different tracing plugin instead of OpenTelemetry. However, this may require significant changes to your Kong configuration and may have some implications, such as:

The new tracing plugin may not be compatible with your existing Kong configuration.
The new tracing plugin may require significant changes to your application code.

Q: What are the implications of using a different tracing plugin?

A: Using a different tracing plugin instead of OpenTelemetry may have implications, such as:

The new tracing plugin may not be compatible with your existing Kong configuration.
The new tracing plugin may require significant changes to your application code.

Conclusion

In conclusion, the issue of OpenTelemetry not generating traces when an authentication plugin rejects a request is a significant one. By increasing the priority of the OpenTelemetry plugin, modifying the authentication plugins, or using a different tracing plugin, it's possible to ensure that OpenTelemetry traces are generated even when authentication fails or Kong returns an early response.

Recommendations

Increase the priority of the OpenTelemetry plugin: This is the recommended solution, as it ensures that the OpenTelemetry plugin is executed before the authentication plugins.
Modify the authentication plugins: This solution requires modifications to the authentication plugins, but it can provide valuable insights into API traffic.
Use a different tracing plugin: If the above solutions are not feasible, a different tracing plugin, such as Zipkin, can be used instead of OpenTelemetry.