Frontend Crashes After Creating A Postgres-cdc Source

by ADMIN 54 views

=====================================================

Introduction

Creating a PostgreSQL CDC source in RisingWave can sometimes lead to a frontend crash, causing the application to enter a crash loop. This issue is particularly frustrating for users who rely on RisingWave for their data processing needs. In this article, we will delve into the details of this bug, explore the error messages and logs, and provide a step-by-step guide on how to reproduce the issue.

Describe the Bug

The bug occurs when creating a PostgreSQL CDC source (pg17) in RisingWave. The frontend crashes and enters a crash loop, making it impossible to continue using the application. This issue is not specific to any particular version of RisingWave, but it has been observed in the nightly build of version 20250420.

Error Message/Log

The error message is quite cryptic, but it provides a hint about the root cause of the issue. The message indicates that the normalized SQL is not parsable, and it points to a specific line in the src/sqlparser/src/ast/mod.rs file. The stack backtrace reveals that the issue is related to the ParserError exception, which is thrown when the CDC source is created with columns and constraints.

thread 'rw-batch-local' panicked at src/sqlparser/src/ast/mod.rs:1677:45:
normalized SQL should be parsable: ParserError("CDC source cannot define columns and constraints\nLINE 1: CREATE SOURCE pg_source_2 (payload JSONB) WITH (connector = 'postgres-cdc', hostname = 'xxx', port = '5432', username = 'postgres', password = 'postgres', database.name = 'risingwave', schema.name = 'public') FORMAT PLAIN ENCODE JSON\n                                                                                                                                                                                                                                                                                                             ^")
stack backtrace:
   0:     0xaaaae9827c44 - std::backtrace_rs::backtrace::libunwind::trace::h68fa4f2ef0c30713
                               at /rustc/f9e0239a7bc813b4aceffc7f069f4797cde3175c/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9
   1:     0xaaaae9827c44 - std::backtrace_rs::backtrace::trace_unsynchronized::h27df4f6f457f4fd5
                               at /rustc/f9e0239a7bc813b4aceffc7f069f4797cde3175c/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14
   2:     0xaaaae9827c44 - std::sys::backtrace::_print_fmt::h18ce864b39d76b45
                               at /rustc/f9e0239a7bc813b4aceffc7f069f4797cde3175c/library/std/src/sys/backtrace.rs:66:9
   3:     0xaaaae9827c44 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::hff8abd39e1f90e87
                               at /rustc/f9e0239a7bc813b4aceffc7f069f7cde3175c/library/std/src/sys/backtrace.rs:39:26
   4:     0xaaaadd2a9404 - core::fmt::rt::Argument::fmt::h9058bde717e94412
                               at /rustc/f9e0239a7bc813b4aceffc7f069f4797cde3175c/library/core/src/fmt/rt.rs:177:76
   5:     0xaaaadd2a9404 - core::fmt::write::h86ac288c85fd4a0a
                               at /rustc/f9e0239a7bc813b4aceffc7f069f4797cde3175c/library/core/src/fmt/mod.rs:1449:21
   6:     0xaaaae9821d04 - std::io::Write::write_fmt::h7b482adf221f73f5
                               at 
...
 212:     0xaaaae982f1b0 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hff1fa87650c6a275
                               at /rustc/f9e0239a7bc813b4aceffc7f069f4797cde3175c/library/alloc/src/boxed.rs:1972:9
 213:     0xaaaae982f1b0 - std::sys::pal::unix::thread::Thread::new::thread_start::h3cd332fa8e79d294
                               at /rustc/f9e0239a7bc813b4aceffc7f069f4797cde3175c/library/std/src/sys/pal/unix/thread.rs:109:17
 214:     0xffffa20f595c - start_thread
                               at ./nptl/pthread_create.c:447:8
 215:     0xffffa215ba4c - thread_start
                               at ./misc/../sysdeps/unix/sysv/linux/aarch64/clone3.S:76:0
 216:                0x0 - <unknown>

To Reproduce

To reproduce this issue, follow these steps:

  1. Create a new RisingWave cluster using the rwc cluster create command.
  2. Create a PostgreSQL CDC source using the CREATE SOURCE statement, specifying the connector as postgres-cdc and the hostname, port, username, and password as required.
  3. Run the CREATE SOURCE statement to create the CDC source.

Expected Behavior

The expected behavior is that the CDC source is created successfully, and the frontend continues to run without any issues.

How Did You Deploy RisingWave?

To deploy RisingWave, we used the following command:

rwc cluster create --name 'test' --tier 'Invited' --compute 'p-1c4g' --compactor 'p-0.5c2g'  --frontend 'p-1c4g' --meta 'p-1c4g' --tag nightly-20250410

The Version of RisingWave

We are using the nightly build of RisingWave version 20250420.

Additional Context

The PostgreSQL version used in this test is:

select version();
                                             version                                             
-------------------------------------------------------------------------------------------------
 PostgreSQL .4 on aarch64-unknown-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit

In conclusion, the frontend crash after creating a PostgreSQL CDC source is a known issue in RisingWave. The error message and log provide valuable insights into the root cause of the issue, which is related to the ParserError exception. To reproduce this issue, follow the steps outlined above. We hope that this article has provided a helpful guide to understanding and resolving this issue.

Introduction

In our previous article, we explored the issue of the frontend crashing after creating a PostgreSQL CDC source in RisingWave. In this article, we will provide a Q&A section to address some of the common questions and concerns related to this issue.

Q: What is the root cause of the frontend crash?

A: The root cause of the frontend crash is related to the ParserError exception, which is thrown when the CDC source is created with columns and constraints.

Q: Why is the CDC source created with columns and constraints?

A: The CDC source is created with columns and constraints because of the way the CREATE SOURCE statement is written. The statement includes the payload JSONB clause, which is not allowed in the CDC source.

Q: How can I avoid creating the CDC source with columns and constraints?

A: To avoid creating the CDC source with columns and constraints, you can modify the CREATE SOURCE statement to exclude the payload JSONB clause. Alternatively, you can use the CREATE SOURCE statement with the FORMAT PLAIN clause, which does not allow columns and constraints.

Q: What are the consequences of creating the CDC source with columns and constraints?

A: Creating the CDC source with columns and constraints can lead to a frontend crash, which can cause the application to become unresponsive. In addition, the CDC source may not function correctly, leading to data inconsistencies and other issues.

Q: How can I troubleshoot the frontend crash?

A: To troubleshoot the frontend crash, you can check the error message and log for any clues about the root cause of the issue. You can also try to reproduce the issue by creating a CDC source with columns and constraints and see if the frontend crashes.

Q: What are the system requirements for RisingWave?

A: The system requirements for RisingWave are:

  • Operating System: Linux (Ubuntu 18.04 or later)
  • CPU: 2x Intel Xeon E5-2690 v4 or equivalent
  • Memory: 16 GB RAM
  • Storage: 1 TB SSD

Q: How can I deploy RisingWave?

A: To deploy RisingWave, you can use the rwc cluster create command, which creates a new RisingWave cluster with the specified configuration.

Q: What are the different tiers of RisingWave?

A: RisingWave has three tiers:

  • Invited: This tier is suitable for development and testing purposes.
  • Standard: This tier is suitable for production environments.
  • Enterprise: This tier is suitable for large-scale production environments.

Q: How can I upgrade RisingWave?

A: To upgrade RisingWave, you can use the rwc cluster upgrade command, which upgrades the RisingWave cluster to the specified version.

Q: What are the different formats supported by RisingWave?

A: RisingWave supports the following formats:

  • JSON
  • Avro
  • Parquet
  • CSV

Q: How can I configure the CDC source?

A: To configure the CDC source, you can use the CREATE SOURCE statement with the WITH clause, which specifies the connector, hostname, port, username, and password.

Q: What are the different connectors supported by RisingWave?

A: RisingWave supports the following connectors* Postgres-CDC

  • Kafka
  • RabbitMQ
  • Amazon S3

Q: How can I monitor the performance of RisingWave?

A: To monitor the performance of RisingWave, you can use the rwc cluster monitor command, which provides real-time metrics and logs for the RisingWave cluster.

Q: What are the different tools supported by RisingWave?

A: RisingWave supports the following tools:

  • RisingWave CLI
  • RisingWave UI
  • RisingWave API

Q: How can I integrate RisingWave with other tools?

A: To integrate RisingWave with other tools, you can use the RisingWave API, which provides a RESTful interface for interacting with the RisingWave cluster.

Q: What are the different security features supported by RisingWave?

A: RisingWave supports the following security features:

  • Authentication
  • Authorization
  • Encryption
  • Access Control

Q: How can I secure the RisingWave cluster?

A: To secure the RisingWave cluster, you can use the rwc cluster secure command, which configures the security features for the RisingWave cluster.

Q: What are the different backup and restore options supported by RisingWave?

A: RisingWave supports the following backup and restore options:

  • Full backup
  • Incremental backup
  • Restore from backup

Q: How can I backup and restore the RisingWave cluster?

A: To backup and restore the RisingWave cluster, you can use the rwc cluster backup and rwc cluster restore commands, respectively.

Q: What are the different scaling options supported by RisingWave?

A: RisingWave supports the following scaling options:

  • Horizontal scaling
  • Vertical scaling

Q: How can I scale the RisingWave cluster?

A: To scale the RisingWave cluster, you can use the rwc cluster scale command, which configures the scaling options for the RisingWave cluster.

Q: What are the different monitoring and logging options supported by RisingWave?

A: RisingWave supports the following monitoring and logging options:

  • Prometheus
  • Grafana
  • ELK Stack

Q: How can I monitor and log the RisingWave cluster?

A: To monitor and log the RisingWave cluster, you can use the rwc cluster monitor and rwc cluster log commands, respectively.

Q: What are the different troubleshooting options supported by RisingWave?

A: RisingWave supports the following troubleshooting options:

  • Debug mode
  • Log level
  • Error reporting

Q: How can I troubleshoot the RisingWave cluster?

A: To troubleshoot the RisingWave cluster, you can use the rwc cluster debug command, which configures the troubleshooting options for the RisingWave cluster.

Q: What are the different support options supported by RisingWave?

A: RisingWave supports the following support options:

  • Community support
  • Enterprise support

Q: How can I get support for RisingWave?

A: To get support for RisingWave, you can contact the RisingWave support team, which provides 24/7 support for the RisingWave cluster.