Refactor: Implement Full NDJSON Support For CLI And HTTP API

by ADMIN 61 views

Introduction

The current implementation of the repository only supports CSV and pretty-printed JSON serialization and streaming. However, with the addition of full NDJSON support, we can deliver a compact, line-oriented format that pipelines and streaming consumers require. This feature is crucial in addressing a core functionality gap. In this article, we will explore the implementation of full NDJSON support for both the Command-Line Interface (CLI) and HTTP API.

Context

The repository includes a feature specification for newline-delimited JSON (NDJSON) serialization and streaming. However, the current source code only supports CSV and pretty-printed JSON. This limitation restricts the use of the repository in scenarios where a compact, line-oriented format is required. By implementing full NDJSON support, we can ensure that the repository meets the needs of pipelines and streaming consumers.

Acceptance Criteria

To ensure that the implementation meets the requirements, we have established the following acceptance criteria:

1. Unit Tests for serializeNDJSON(data)

  • Given an array of points (e.g., [x0,y:0,x1,y:1]), serializeNDJSON returns one JSON.stringify line per point with no trailing newline.

    The unit test for serializeNDJSON(data) should verify that the function returns a string with one JSON line per point in the input array. The test should also ensure that there is no trailing newline character at the end of the string.

2. CLI Support

  • mainCLI(['--expression','x','--range','0:2','--points','3','--format','ndjson']) returns exactly three NDJSON lines without a trailing newline.

    The CLI test should verify that the mainCLI function returns a string with exactly three NDJSON lines when the --format flag is set to ndjson. The test should also ensure that there is no trailing newline character at the end of the string.

  • --format ndjson --output-file <file> writes a valid NDJSON file matching the returned string.

    The CLI test should verify that the --format ndjson --output-file <file> command writes a valid NDJSON file to the specified file. The test should also ensure that the contents of the file match the returned string.

  • Missing required flags (--expression or --range) with --format ndjson produce the same errors as other formats.

    The CLI test should verify that the mainCLI function produces the same errors as other formats when the required flags (--expression or --range) are missing and the --format flag is set to ndjson.

3. HTTP GET /ndjson Endpoint

  • GET /ndjson?expression=x&range=0:2&points=3 returns status 200, Content-Type application/x-ndjson, and a body of three JSON lines matching the series.

    The HTTP test should verify that the GET /ndjson endpoint returns a status code of 200, a Content-Type of application/x-ndjson, and a body with three JSON lines matching the series.

  • Invalid or missing query parameters yield a 400 error with descriptive error messages.

    The HTTP test should verify that the GET /ndjson endpoint returns a status code of 400 with descriptive error messages when the query parameters are invalid or missing.

4. Documentation Updates

  • USAGE.md and README.md include:

    • a. CLI example for --format ndjson showing output.

      The documentation update should include a CLI example for the --format ndjson flag, showing the output.

    • b. HTTP endpoint documentation for GET /ndjson, its query parameters, headers, and a sample curl command with expected NDJSON response.

      The documentation update should include the HTTP endpoint documentation for the GET /ndjson endpoint, including its query parameters, headers, and a sample curl command with the expected NDJSON response.

Verification

To ensure that the implementation meets the requirements, we will perform the following verification steps:

  • All existing and new tests pass via npm test.

    We will run the npm test command to verify that all existing and new tests pass.

  • Manual CLI test:

repository0-plot-code-lib --expression "x" --range "0:2" --points 3 --format ndjson


    We will run the manual CLI test to verify that the `mainCLI` function returns a string with exactly three NDJSON lines when the `--format` flag is set to `ndjson`.

*   **Manual HTTP test:**

    ```bash
curl -s "http://localhost:3000/ndjson?expression=x&range=0:2&points=3"

We will run the manual HTTP test to verify that the `GET /ndjson` endpoint returns a status code of 200, a Content-Type of `application/x-ndjson`, and a body with three JSON lines matching the series.

Q: What is NDJSON and why is it important?

A: NDJSON (Newline-Delimited JSON) is a compact, line-oriented format for serializing and streaming JSON data. It is essential for pipelines and streaming consumers, as it allows for efficient and scalable data processing. By implementing full NDJSON support, we can ensure that our repository meets the needs of these use cases.

Q: What are the benefits of implementing full NDJSON support?

A: The benefits of implementing full NDJSON support include:

  • Improved performance: NDJSON allows for efficient and scalable data processing, making it ideal for large-scale data processing and analytics.
  • Increased flexibility: NDJSON can be used with various data processing tools and frameworks, making it a versatile choice for data processing and analytics.
  • Better support for streaming consumers: NDJSON is designed for streaming consumers, making it an excellent choice for real-time data processing and analytics.

Q: How will the implementation of full NDJSON support affect the CLI?

A: The implementation of full NDJSON support will affect the CLI in the following ways:

  • New CLI flag: A new CLI flag, --format ndjson, will be added to support NDJSON output.
  • Improved error handling: Error handling will be improved to provide more descriptive error messages when required flags are missing.
  • Updated documentation: Documentation will be updated to include examples and usage instructions for the new CLI flag.

Q: How will the implementation of full NDJSON support affect the HTTP API?

A: The implementation of full NDJSON support will affect the HTTP API in the following ways:

  • New endpoint: A new endpoint, /ndjson, will be added to support NDJSON output.
  • Improved error handling: Error handling will be improved to provide more descriptive error messages when query parameters are invalid or missing.
  • Updated documentation: Documentation will be updated to include examples and usage instructions for the new endpoint.

Q: What are the acceptance criteria for the implementation of full NDJSON support?

A: The acceptance criteria for the implementation of full NDJSON support include:

  • Unit tests for serializeNDJSON(data): Unit tests will be written to verify that the serializeNDJSON function returns a string with one JSON line per point in the input array.
  • CLI support: CLI tests will be written to verify that the mainCLI function returns a string with exactly three NDJSON lines when the --format flag is set to ndjson.
  • HTTP GET /ndjson endpoint: HTTP tests will be written to verify that the GET /ndjson endpoint returns a status code of 200, a Content-Type of application/x-ndjson, and a body with three JSON lines matching the series.
  • Documentation updates: Documentation will be updated to include examples and usage instructions for the new CLI flag and endpoint.

Q: How will the implementation of full NDJSON support be verified?

A: The implementation of full NDJSON support will be verified the following steps:

  • All existing and new tests pass via npm test: The npm test command will be run to verify that all existing and new tests pass.
  • Manual CLI test: A manual CLI test will be performed to verify that the mainCLI function returns a string with exactly three NDJSON lines when the --format flag is set to ndjson.
  • Manual HTTP test: A manual HTTP test will be performed to verify that the GET /ndjson endpoint returns a status code of 200, a Content-Type of application/x-ndjson, and a body with three JSON lines matching the series.

By implementing full NDJSON support for both the CLI and HTTP API, we can ensure that our repository meets the needs of pipelines and streaming consumers. The Q&A above provides a comprehensive overview of the implementation, acceptance criteria, and verification steps.