[Feature] Support Service Account To Use AWS IAM Role

by ADMIN 54 views

Introduction

In today's cloud computing landscape, security and access control are crucial aspects of any application or service. Apache Doris, a popular open-source, distributed SQL database, has been gaining traction in the industry due to its high-performance capabilities and scalability. However, one of the limitations of Doris is its inability to support service accounts using AWS IAM roles. This feature request aims to address this limitation and provide a more secure and efficient way to access AWS resources.

Problem Statement

Currently, when querying data from S3 using Doris, users are required to set their access key and secret key explicitly. This approach has several drawbacks, including:

  • Security Risks: Hardcoding access keys and secret keys in the query can lead to security breaches if the credentials are compromised.
  • Inflexibility: Users are tied to a specific set of credentials, making it difficult to manage access control and rotate credentials.
  • Scalability Issues: As the number of users and queries increases, managing access keys and secret keys becomes a complex task.

Use Case

To demonstrate the benefits of supporting service accounts using AWS IAM roles, let's consider a typical use case:

Suppose we have a data warehouse application that uses Doris to query data from S3. We want to grant access to a specific service account to read data from a particular bucket. Without this feature, we would need to set the access key and secret key explicitly in the query, as shown below:

SELECT * FROM S3 (
  "URI" = "s3://my-bucket/my.parquet",
  "format" = "parquet",
  "s3.endpoint" = "https://s3.us-west-2.amazonaws.com",
  "s3.region" = "us-west-2",
  "s3.access_key" = "xxx",
  "s3.secret_key" = "xxx"
);

However, with the support for service accounts using AWS IAM roles, we can simply specify the service account's ARN (Amazon Resource Name) in the query, eliminating the need for explicit access keys and secret keys.

Benefits

The proposed feature offers several benefits, including:

  • Improved Security: By using AWS IAM roles, we can ensure that access to S3 resources is granted based on the service account's permissions, reducing the risk of security breaches.
  • Increased Flexibility: Users can easily manage access control and rotate credentials without affecting the query.
  • Scalability: The feature enables efficient management of access control and credentials, making it easier to scale the application.

Implementation

To implement this feature, we will need to:

  1. Integrate with AWS IAM: We will need to integrate Doris with AWS IAM to enable the use of service accounts and IAM roles.
  2. Modify the S3 Query: We will need to modify the S3 query to accept the service account's ARN instead of explicit access keys and secret keys.
  3. Implement Access Control: We will need to implement access control mechanisms to ensure that the service account has the necessary permissions to access the S3 resources.

Related Issues

While there are no directly related issues, this feature request is closely related to the following issues:

Are You Willing to Submit a PR?

Yes, I am willing to submit a PR to implement this feature.

Code of Conduct

I agree to follow this project's Code of Conduct.

Conclusion

Introduction

In our previous article, we discussed the importance of supporting service accounts using AWS IAM roles in Apache Doris. This feature will improve security, flexibility, and scalability for users accessing S3 resources. In this article, we will address some frequently asked questions (FAQs) related to this feature.

Q: What is the current limitation of Apache Doris when it comes to accessing S3 resources?

A: Currently, Apache Doris requires users to set their access key and secret key explicitly when querying data from S3. This approach has several drawbacks, including security risks, inflexibility, and scalability issues.

Q: How does the proposed feature address these limitations?

A: The proposed feature will enable users to access S3 resources using service accounts and AWS IAM roles. This will eliminate the need for explicit access keys and secret keys, improving security, flexibility, and scalability.

Q: What are the benefits of using service accounts and AWS IAM roles?

A: Using service accounts and AWS IAM roles offers several benefits, including:

  • Improved Security: Access to S3 resources is granted based on the service account's permissions, reducing the risk of security breaches.
  • Increased Flexibility: Users can easily manage access control and rotate credentials without affecting the query.
  • Scalability: The feature enables efficient management of access control and credentials, making it easier to scale the application.

Q: How will the S3 query be modified to support service accounts and AWS IAM roles?

A: The S3 query will be modified to accept the service account's ARN (Amazon Resource Name) instead of explicit access keys and secret keys.

Q: What is the implementation plan for this feature?

A: The implementation plan involves:

  1. Integrating with AWS IAM: We will integrate Apache Doris with AWS IAM to enable the use of service accounts and IAM roles.
  2. Modifying the S3 Query: We will modify the S3 query to accept the service account's ARN instead of explicit access keys and secret keys.
  3. Implementing Access Control: We will implement access control mechanisms to ensure that the service account has the necessary permissions to access the S3 resources.

Q: Are there any related issues that need to be addressed?

A: Yes, there are related issues that need to be addressed, including:

Q: Can you provide an example of how the modified S3 query will look like?

A: Yes, here is an example of how the modified S3 query will look like:

SELECT * FROM S3 (
  "URI" = "s3://my-bucket/my.parquet",
  "format" = "parquet",
  "s3.endpoint" = "https://s3.us-west-2.amazonaws.com",
  "s3.region" = "us-west-2",
  "s3.service_account_arn" = "arn:aws:iam::123456789012:service-role/MyServiceRole"
);

Q: Are you willing to submit a PR to implement this feature?

A: Yes, I am willing to submit a PR to implement this feature.

Q: What is the Code of Conduct for this project?

A: I agree to follow this project's Code of Conduct.

Conclusion

In conclusion, supporting service accounts using AWS IAM roles is a crucial feature for Apache Doris to improve security, flexibility, and scalability. By addressing the FAQs related to this feature, we can better understand the benefits and implementation plan for this feature. I am willing to submit a PR to implement this feature and contribute to the growth of the Apache Doris community.