Ccl/changefeedccl: TestCDCQuerySelectSingleRow Failed

May 3, 2025 by ADMIN 54 views

**Investigating the Failure of TestCDCQuerySelectSingleRow in CockroachDB**

Introduction

CockroachDB is a distributed relational database that provides a scalable and highly available solution for storing and managing data. However, like any complex system, it is not immune to failures and errors. In this article, we will investigate the failure of the TestCDCQuerySelectSingleRow test in CockroachDB and provide a detailed analysis of the issue.

Background

The TestCDCQuerySelectSingleRow test is a part of the CockroachDB test suite, which is designed to ensure the correctness and reliability of the database. This test is specifically designed to test the CDC (Change Data Capture) feature of CockroachDB, which allows users to capture and process changes to the database in real-time.

The Failure

The failure of the TestCDCQuerySelectSingleRow test is reported in the following GitHub issue: CRDB-50336. The issue is marked as "open" and has a status of "failed".

Stacktrace

The stacktrace for the failure is as follows:

goroutine 141066 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x2d

The stacktrace indicates that the test timed out after 14 minutes and 57 seconds.

Log Preceding Fatal Error

The log preceding the fatal error is as follows:

* created by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 123508
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:462 +0x4f
* 
* goroutine 123654 [select]:
* github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:414 +0x85
* created by github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue in goroutine 123508
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:411 +0x3be
* 
* goroutine 132921 [select]:
* github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:243 +0xcb
* created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 123626
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:238 +0x58
* 
* goroutine 125949 [chan send, 2 minutes]:
* github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.TestCDCQuerySelectSingleRow.func1.1({0x83d11a0,0xc0097c75f0})
* 	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:9731 +0x26
* github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*changefeedResumer).resumeWithRetries(0xc007362528, {0x84230b8, 0xc00a975c80}, {0x8484d90, 0xc00a979258}, 0xed536a83cc28001, {0xc00e723470, {0xc00655c140, 0x3d}, 0xc00e723410, ...}, ...)
* 	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_stmt.go:1305 +0x493
* github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*changefeedResumer).Resume(0xc007362528, {0x84230b8, 0xc00a975c80}, {0x676e5c0?, 0xc00a979258})
* 	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_stmt.go:1130 +0x2f2
* github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*reportErrorResumer).Resume(0x84230b8?, {0x84230b8?, 0xc00a975c80?}, {0x676e5c0?, 0xc00a979258?})
* 	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/testfeed_test.go:362 +0x2f
* github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func2(0xc009320b60?, 0x0?, 0xc0046ef940, {0x83fc480?, 0xc00d1b56e0?}, {0x84230b8?, 0xc00a975c80?}, {0x676e5c0?, 0xc00a979258?})
* 	github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1593 +0xbb
* github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine(0xc006b3db88, {0x84230b8, 0xc00a975bc0}, {0x676e5c0, 0xc00a979258}, {0x83fc480, 0xc00d1b56e0}, 0xc009320b60, {0x6b0249f, 0x7}, ...)
* 	github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1594 +0xaa5
* github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob(0xc006b3db88, {0x84230b8, 0xc00a975bc0}, {0x83fc480, 0xc00d1b56e0}, 0xc009320b60, {0x6b0249f, 0x7}, {0xc00839d500, 0x17})
* 	github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:451 +0x55a
* github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Start.func2({0xc007789008?, 0xc005193260?})
* 	github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:834 +0xe7
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:480 +0x13a
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 125926
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:471 +0x3fe
* 
* goroutine 123510 [select]:
* github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:243 +0xcb
* created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 123508
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:238 +0x58
* 
* goroutine 123655 [select]:
* github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:491 +0x14d
* created by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 123508
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:462 +0x4f
* 
*

The log indicates that the test timed out after 14 minutes and 57 seconds, and the stacktrace suggests that the issue is related to the changefeedResumer and reportErrorResumer functions.

Analysis

Q: What is the TestCDCQuerySelectSingleRow test in CockroachDB?

A: The TestCDCQuerySelectSingleRow test is a part of the CockroachDB test suite, which is designed to ensure the correctness and reliability of the database. This test is specifically designed to test the CDC (Change Data Capture) feature of CockroachDB, which allows users to capture and process changes to the database in real-time.

Q: What is the purpose of the TestCDCQuerySelectSingleRow test?

A: The purpose of the TestCDCQuerySelectSingleRow test is to verify that the CDC feature of CockroachDB is working correctly and can capture and process changes to the database in real-time.

Q: What is the issue with the TestCDCQuerySelectSingleRow test?

A: The issue with the TestCDCQuerySelectSingleRow test is that it timed out after 14 minutes and 57 seconds, indicating that there may be an issue with the test itself or the CDC feature of CockroachDB.

Q: What is the stacktrace for the failure of the TestCDCQuerySelectSingleRow test?

A: The stacktrace for the failure of the TestCDCQuerySelectSingleRow test is as follows:

goroutine 141066 [running]:
testing.(*M).startAlarm.func1()
	GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
	GOROOT/src/time/sleep.go:177 +0x2d

Q: What is the log preceding the fatal error for the TestCDCQuerySelectSingleRow test?

A: The log preceding the fatal error for the TestCDCQuerySelectSingleRow test is as follows:

* created by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 123508
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:462 +0x4f
* 
* goroutine 123654 [select]:
* github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:414 +0x85
* created by github.com/cockroachdb/cockroach/pkg/util/admission.initWorkQueue in goroutine 123508
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:411 +0x3be
* 
* goroutine 132921 [select]:
* github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:243 +0xcb
* created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 123626
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:238 +0x58
* 
* goroutine 125949 [chan send, 2 minutes]:
* github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.TestCDCQuerySelectSingleRow.func1.1({0x83d11a0,0xc0097c75f0})
* 	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:9731 +0x26
* github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*changefeedResumer).resumeWithRetries(0xc007362528, {0x84230b8, 0xc00a975c80}, {0x8484d90, 0xc00a979258}, 0xed536a83cc28001, {0xc00e723470, {0xc00655c140, 0x3d}, 0xc00e723410, ...}, ...)
* 	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_stmt.go:1305 +0x493
* github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*changefeedResumer).Resume(0xc007362528, {0x84230b8, 0xc00a975c80}, {0x676e5c0?, 0xc00a979258})
* 	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_stmt.go:1130 +0x2f2
* github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*reportErrorResumer).Resume(0x84230b8?, {0x84230b8?, 0xc00a975c80?}, {0x676e5c0?, 0xc00a979258?})
* 	github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/testfeed_test.go:362 +0x2f
* github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func2(0xc009320b60?, 0x0?, 0xc0046ef940, {0x83fc480?, 0xc00d1b56e0?}, {0x84230b8?, 0xc00a975c80?}, {0x676e5c0?, 0xc00a979258?})
* 	github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1593 +0xbb
* github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine(0xc006b3db88, {0x84230b8, 0xc00a975bc0}, {0x676e5c0, 0xc00a979258}, {0x83fc480, 0xc00d1b56e0}, 0xc009320b60, {0x6b0249f, 0x7}, ...)
* 	github.com/cockroachdb/cockroach/pkg/jobs/registry.go:1594 +0xaa5
* github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob(0xc006b3db88, {0x84230b8, 0xc00a975bc0}, {0x83fc480, 0xc00d1b56e0}, 0xc009320b60, {0x6b0249f, 0x7}, {0xc00839d500, 0x17})
* 	github.com/cockroachdb/cockroach/pkg/jobs/adopt.go:451 +0x55a
* github.com/cockroachdb/cockroach/pkg/jobs.(*StartableJob).Start.func2({0xc007789008?, 0xc005193260?})
* 	github.com/cockroachdb/cockroach/pkg/jobs/jobs.go:834 +0xe7
* github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:480 +0x13a
* created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 125926
* 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:471 +0x3fe
* 
* goroutine 123510 [select]:
* github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:243 +0xcb
* created by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 123508
* 	github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/disk_health.go:238 +0x58
* 
* goroutine 123655 [select]:
* github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:491 +0x14d
* created by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 123508
* 	github.com/cockroachdb/cockroach/pkg/util/admission/work_queue.go:462 +0x4f
* 
*

Q: What is the next step to investigate the failure of the TestCDCQuerySelectSingleRow test?

A: The next step to investigate the failure of the TestCDCQuerySelectSingleRow test is to analyze the stacktrace and log to identify the root cause of the issue. This may involve reviewing the code and testing environment to determine if there are