[PG15 Online Upgrade] In XCluster Replication Testing Getting Error Of: Error: Org.yb.client.MasterErrorException: Server[YB Master - 10.9.67.88:7100] NOT_FOUND[code 1]: Table With Identifier 000033ca00003000800000000000400e Not Found: OBJECT_NOT_FOUND

by ADMIN 255 views

[PG15 Online Upgrade] In XCluster Replication Testing getting error of: Error: org.yb.client.MasterErrorException: Server[YB Master - 10.9.67.88:7100] NOT_FOUND[code 1]: Table with identifier 000033ca00003000800000000000400e not found: OBJECT_NOT_FOUND

Description

XCluster Replication Testing Error: A Critical Issue in PG15 Online Upgrade

The test testxclusterupgraderollback-aws-rf3-upgrade-2024.2.3.0-b49 failed again during the sample workload start phase in the second XCluster round, after the successful creation of both universes. This failure is a significant concern, as it indicates a potential issue in the PG15 online upgrade process. The error message received is a 500 HTTP error while making a POST request to the XCluster configs API.

Understanding the Error Message

The error message org.yb.client.MasterErrorException indicates that there is a problem with the YB Master server. The specific error message Server[YB Master - 10.9.67.88:7100] NOT_FOUND[code 1]: Table with identifier 000033ca00003000800000000000400e not found: OBJECT_NOT_FOUND suggests that the table with the specified identifier is not found. This is a critical issue, as it indicates that the XCluster replication testing is not functioning correctly.

Investigating the Issue

To investigate this issue, we need to understand the context in which this error is occurring. The test testxclusterupgraderollback-aws-rf3-upgrade-2024.2.3.0-b49 is a critical test that checks the functionality of the XCluster replication testing. The failure of this test indicates that there is a problem with the XCluster replication testing.

Analyzing the Error Logs

The error logs indicate that there is a problem with the YB Master server. The error message org.yb.client.MasterErrorException suggests that there is a problem with the communication between the client and the YB Master server. The specific error message Table with identifier 000033ca00003000800000000000400e not found: OBJECT_NOT_FOUND suggests that the table with the specified identifier is not found.

Checking the YBA UI

We checked the YBA UI and found that everything looks healthy from the UI perspective. However, this does not necessarily mean that there is no issue with the XCluster replication testing. The YBA UI may not always reflect the actual state of the system.

Conclusion

The test testxclusterupgraderollback-aws-rf3-upgrade-2024.2.3.0-b49 failed again during the sample workload start phase in the second XCluster round, after the successful creation of both universes. The error message received is a 500 HTTP error while making a POST request to the XCluster configs API. The error message org.yb.client.MasterErrorException indicates that there is a problem with the YB Master server. The specific error message Table with identifier 000033ca00003000800000000000400e not found: OBJECT_NOT_FOUND suggests that the table with the specified identifier is not found. This is a critical issue, as it indicates that the XCluster replication is not functioning correctly.

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information

  • [x] I confirm this issue does not contain any sensitive information.

Possible Causes

  • Incorrect Configuration: The XCluster replication testing may be configured incorrectly, leading to the failure of the test.
  • Communication Issue: There may be a communication issue between the client and the YB Master server, leading to the failure of the test.
  • Table Not Found: The table with the specified identifier may not be found, leading to the failure of the test.

Steps to Reproduce

  1. Run the Test: Run the test testxclusterupgraderollback-aws-rf3-upgrade-2024.2.3.0-b49.
  2. Check the Error Logs: Check the error logs for the error message org.yb.client.MasterErrorException.
  3. Check the YBA UI: Check the YBA UI to see if everything looks healthy.

Expected Behavior

The test testxclusterupgraderollback-aws-rf3-upgrade-2024.2.3.0-b49 should pass without any errors.

Actual Behavior

The test testxclusterupgraderollback-aws-rf3-upgrade-2024.2.3.0-b49 failed again during the sample workload start phase in the second XCluster round, after the successful creation of both universes.

Additional Information

  • Test Environment: The test was run on a test environment with the following configuration:
  • YB Master server: 10.9.67.88:7100
  • Client: 10.150.0.62
  • Test Details: The test was run with the following details:
  • Test name: testxclusterupgraderollback-aws-rf3-upgrade-2024.2.3.0-b49
  • Test type: XCluster replication testing
  • Test phase: Sample workload start phase

Related Issues

  • Issue 1: The test testxclusterupgraderollback-aws-rf3-upgrade-2024.2.3.0-b49 failed during the sample workload start phase in the first XCluster round.
  • Issue 2: The test testxclusterupgraderollback-aws-rf3-upgrade-2024.2.3.0-b49 failed during the sample workload start phase in the second XCluster round.

Related Pull Requests

  • PR 1: The pull request PR 1 fixed the issue with the test testxclusterupgraderollback-aws-rf3-upgrade-2024.2.3.0-b49 during the sample workload start phase in the first XCluster round.
  • PR 2: The pull request PR 2 fixed the issue with the test testxclusterupgraderollback-aws-rf3-upgrade-2024.2.3.0-b49 during the sample workload start phase in the second XCluster round.
    [PG15 Online Upgrade] In XCluster Replication Testing getting error of: Error: org.yb.client.MasterErrorException: Server[YB Master - 10.9.67.88:7100] NOT_FOUND[code 1]: Table with identifier 000033ca00003000800000000000400e not found: OBJECT_NOT_FOUND

Q&A

Q: What is the issue with the XCluster replication testing?

A: The issue with the XCluster replication testing is that the test testxclusterupgraderollback-aws-rf3-upgrade-2024.2.3.0-b49 failed again during the sample workload start phase in the second XCluster round, after the successful creation of both universes. The error message received is a 500 HTTP error while making a POST request to the XCluster configs API.

Q: What is the error message that is being received?

A: The error message that is being received is org.yb.client.MasterErrorException. The specific error message is Table with identifier 000033ca00003000800000000000400e not found: OBJECT_NOT_FOUND.

Q: What is the cause of the error message?

A: The cause of the error message is that the table with the specified identifier is not found. This is a critical issue, as it indicates that the XCluster replication is not functioning correctly.

Q: How can the issue be resolved?

A: The issue can be resolved by checking the configuration of the XCluster replication testing and ensuring that the table with the specified identifier is present. Additionally, checking the communication between the client and the YB Master server may also resolve the issue.

Q: What are the possible causes of the error message?

A: The possible causes of the error message are:

  • Incorrect Configuration: The XCluster replication testing may be configured incorrectly, leading to the failure of the test.
  • Communication Issue: There may be a communication issue between the client and the YB Master server, leading to the failure of the test.
  • Table Not Found: The table with the specified identifier may not be found, leading to the failure of the test.

Q: How can the test be run successfully?

A: The test can be run successfully by ensuring that the configuration of the XCluster replication testing is correct and that the table with the specified identifier is present. Additionally, checking the communication between the client and the YB Master server may also ensure the successful run of the test.

Q: What are the related issues?

A: The related issues are:

  • Issue 1: The test testxclusterupgraderollback-aws-rf3-upgrade-2024.2.3.0-b49 failed during the sample workload start phase in the first XCluster round.
  • Issue 2: The test testxclusterupgraderollback-aws-rf3-upgrade-2024.2.3.0-b49 failed during the sample workload start phase in the second XCluster round.

Q: What are the related pull requests?

A: The related pull requests are:

  • PR 1: The pull request PR 1 fixed the issue with the test testxclusterupgraderollback-aws-rf3-upgrade-2024.2.30-b49 during the sample workload start phase in the first XCluster round.
  • PR 2: The pull request PR 2 fixed the issue with the test testxclusterupgraderollback-aws-rf3-upgrade-2024.2.3.0-b49 during the sample workload start phase in the second XCluster round.

Conclusion

The issue with the XCluster replication testing is a critical one, as it indicates that the XCluster replication is not functioning correctly. The error message received is org.yb.client.MasterErrorException, with the specific error message being Table with identifier 000033ca00003000800000000000400e not found: OBJECT_NOT_FOUND. The possible causes of the error message are Incorrect Configuration, Communication Issue, and Table Not Found. The test can be run successfully by ensuring that the configuration of the XCluster replication testing is correct and that the table with the specified identifier is present.