[PG15 Online Upgrade] Unable To Connect To Ycql After Upgrade To Master ( PG15 ) From 2024.2.3.0 With Error Of Leaderless Tablets And Health Check Failure.

Apr 21, 2025 by ADMIN 156 views

PG15 Online Upgrade: Unable to Connect to YCQL After Upgrade to Master (PG15) from 2024.2.3.0 with Error of Leaderless Tablets and Health Check Failure

Description

When upgrading from version 2024.2.3.0 to 2.25.2.0-b332, users may encounter a health check error that prevents them from connecting to YCQL. This issue is particularly concerning as it affects the overall performance and reliability of the system. In this article, we will delve into the details of this problem, explore possible causes, and provide a step-by-step guide to resolving the issue.

Understanding the Health Check Error

The health check error is characterized by the following message:

Error executing command timeout 20 bash -c 'set -o pipefail; /home/yugabyte/tserver/bin/cqlsh 10.9.114.79 9042 -e "SHOW HOST"': 
Connection error: ('Unable to connect to any servers', {'10.9.114.79:9042': OperationTimedOut('errors=None, last_host=None')})

This error indicates that the system is unable to connect to any servers, resulting in a timeout. The error message suggests that the issue is related to the connection to the YCQL server, which is running on port 9042.

Reproducing the Issue

To reproduce this issue, users can follow these steps:

Upgrade from version 2024.2.3.0 to 2.25.2.0-b332.
Run the health check command to verify the connection to the YCQL server.
Observe the health check error and note the error message.

Investigating the Issue

To investigate this issue, users can follow these steps:

Verify System Resources: Ensure that the system resources are within limits. In this case, the CPU usage was around 30-35%, memory usage was around 50-60%, IOPS was around 500, and disk usage was around 10%. These values are well within the recommended limits.
Check YCQL Server Configuration: Verify that the YCQL server is configured correctly. Check the YCQL server logs for any errors or warnings.
Check Network Configuration: Verify that the network configuration is correct. Check the network logs for any errors or warnings.

Possible Causes

Based on the investigation, the possible causes of this issue are:

Leaderless Tablets: The error message suggests that the issue is related to leaderless tablets. Leaderless tablets are a type of tablet that does not have a leader node. In this case, the system is unable to connect to any servers, resulting in a timeout.
Health Check Failure: The health check failure is a critical issue that prevents the system from functioning correctly. In this case, the health check failure is preventing the system from connecting to the YCQL server.

Resolving the Issue

To resolve this issue, users can follow these steps:

Downgrade to Previous Version: Downgrade to the previous version (2024.2.3.0) to verify that the issue is resolved.
Reconfigure YCQL Server: Reconfigure the YCQL server to ensure that it is running correctly.
Verify Network Configuration: that the network configuration is correct.
Run Health Check Command: Run the health check command to verify that the connection to the YCQL server is successful.

Conclusion

In conclusion, the health check error that prevents users from connecting to YCQL after upgrading to master (PG15) from 2024.2.3.0 is a critical issue that requires immediate attention. By following the steps outlined in this article, users can investigate and resolve this issue. It is essential to verify system resources, check YCQL server configuration, and check network configuration to resolve this issue.

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information

[x] I confirm this issue does not contain any sensitive information.

Additional Information

Test Environment: The test environment consists of a single node with a bank workload.
Upgrade Method: The upgrade was performed using the online upgrade method.
Rollback Method: The rollback was performed using the online rollback method.
Test Results: The test results indicate that the issue is consistently reproducible.
PG15 Online Upgrade: Unable to Connect to YCQL After Upgrade to Master (PG15) from 2024.2.3.0 with Error of Leaderless Tablets and Health Check Failure: Q&A

Q&A

Q: What is the cause of the health check error that prevents users from connecting to YCQL after upgrading to master (PG15) from 2024.2.3.0?

A: The cause of the health check error is related to leaderless tablets and health check failure. Leaderless tablets are a type of tablet that does not have a leader node, and in this case, the system is unable to connect to any servers, resulting in a timeout.

Q: What are the possible causes of this issue?

A: The possible causes of this issue are:

Leaderless Tablets: The error message suggests that the issue is related to leaderless tablets. Leaderless tablets are a type of tablet that does not have a leader node. In this case, the system is unable to connect to any servers, resulting in a timeout.
Health Check Failure: The health check failure is a critical issue that prevents the system from functioning correctly. In this case, the health check failure is preventing the system from connecting to the YCQL server.

Q: How can I verify that the issue is resolved after downgrading to the previous version (2024.2.3.0)?

A: To verify that the issue is resolved after downgrading to the previous version (2024.2.3.0), you can run the health check command to verify that the connection to the YCQL server is successful.

Q: What are the steps to resolve this issue?

A: The steps to resolve this issue are:

Downgrade to Previous Version: Downgrade to the previous version (2024.2.3.0) to verify that the issue is resolved.
Reconfigure YCQL Server: Reconfigure the YCQL server to ensure that it is running correctly.
Verify Network Configuration: Verify that the network configuration is correct.
Run Health Check Command: Run the health check command to verify that the connection to the YCQL server is successful.

Q: What are the system resources that I should verify to ensure that the issue is not related to resource constraints?

A: The system resources that you should verify to ensure that the issue is not related to resource constraints are:

CPU Usage: Verify that the CPU usage is within the recommended limits.
Memory Usage: Verify that the memory usage is within the recommended limits.
IOPS: Verify that the IOPS is within the recommended limits.
Disk Usage: Verify that the disk usage is within the recommended limits.

Q: What are the network configuration settings that I should verify to ensure that the issue is not related to network configuration?

A: The network configuration settings that you should verify to ensure that the issue is not related to network configuration are:

IP Address: Verify that the IP address is correct.
Port Number: Verify that the port number is correct.
Network Protocol: Verify that the network protocol is correct.

Q: What are the YCQL server configuration settings that I should verify to ensure that the issue not related to YCQL server configuration?

A: The YCQL server configuration settings that you should verify to ensure that the issue is not related to YCQL server configuration are:

YCQL Server Port: Verify that the YCQL server port is correct.
YCQL Server IP Address: Verify that the YCQL server IP address is correct.
YCQL Server Configuration File: Verify that the YCQL server configuration file is correct.

Conclusion

Additional Information

Test Environment: The test environment consists of a single node with a bank workload.
Upgrade Method: The upgrade was performed using the online upgrade method.
Rollback Method: The rollback was performed using the online rollback method.
Test Results: The test results indicate that the issue is consistently reproducible.