Can't Roundtrip F-strings That Starts With Single-quote And Has Dict-keys In Double Quotes

Introduction

F-strings, also known as formatted string literals, are a powerful feature in Python that allows you to embed expressions inside string literals. However, when working with dictionary keys inside f-strings, you may encounter issues with roundtripping, especially when the f-string starts with a single quote and includes dictionary key expressions with double quotes. In this article, we will explore this issue and provide a reproducible example to demonstrate the problem.

The Issue

The issue arises when you try to roundtrip an f-string that starts with a single quote and includes dictionary key expressions with double quotes. The astor library, which is used for roundtripping, fails to correctly parse the f-string. However, if the f-string starts with a double quote and includes dictionary key expressions with single quotes, the roundtripping process is successful.

Reproducible Example

Below is a reproducible example that demonstrates the issue:

# sample.py
# test dict keys with F-strings (formatted string literals)
import sys
import astor

print(sys.version)
print(f'astor version: {astor.__version__}')

print("-=" * 20)

print(f'this string has "double quotes"')  # No error
print(f"this string has 'single quotes'")  # No error

print(f"this works! {'a' + 'b'}")  # No error
print(f'this also works! {"a" + "b"}')  # No error

a_dict = {
    'key1': 'value1',
    'key2': 'value2',
    'key3': 'value3',
}

print(a_dict["key1"])  # No error
print(f"a_dict['key2']: {a_dict['key2']}")  # No error
print(f'a_dict["key2"]: {a_dict["key2"]}')  # ERROR on astor.rtrip

print('Now try f-string debug print feature:')
print(f"{a_dict['key2']=}")  # No error
print(f'{a_dict["key2"]=}')  # ERROR on astor.rtrip

print(f'this string has "double quotes"')  # No error
print(f"this string has 'single quotes'")  # No error

Roundtrip Test

To demonstrate the issue, we can run the astor.rtrip command on the sample.py file:

C:\>python -m astor.rtrip sample.py

Trashing tmp_rtrip
Converting sample.py
    calculating dump -- bad

Files failed to round-trip to AST:
    sample.py

After running the roundtrip test, we can examine the output file tmp_rtrip\sample.py:

import sys
import astor
print(sys.version)
print(f'astor version: {astor.__version__}')
print('-=' * 20)
print(f'this string has "double quotes"')
print(f"this string has 'single quotes'")
print(f"this works! {'a' + 'b'}")
print(f"this also works! {'a' + 'b'}")
a_dict = {'key1': 'value1', 'key2': 'value2', 'key3': 'value3'}
print(a_dict['key1'])
print(f"a_dict['key2']: {a_dict['key2']}")
print(f'a_dict["key2"]: {a_dict[\'key2\']}')
print('Now try f-string debug print feature:')
print(f"a_dict['key2']={a_dict['key2']!r}")
print(f'a_dict["key2"]={a_dict[\'key2\']!r}')

As we can see, the roundtrip process failed to correctly parse the f-string that starts with a single quote and includes dictionary key expressions with double quotes.

Conclusion

Q: What is the issue with roundtripping f-strings that start with a single quote and include dictionary key expressions with double quotes?

A: The issue arises when you try to roundtrip an f-string that starts with a single quote and includes dictionary key expressions with double quotes. The astor library, which is used for roundtripping, fails to correctly parse the f-string.

Q: What is the expected behavior when roundtripping an f-string that starts with a single quote and includes dictionary key expressions with double quotes?

A: The expected behavior is that the astor library should correctly parse the f-string and roundtrip it to the original code.

Q: What is the actual behavior when roundtripping an f-string that starts with a single quote and includes dictionary key expressions with double quotes?

A: The actual behavior is that the astor library fails to correctly parse the f-string and roundtrips it to a modified version of the code.

Q: Can I work around this issue by using double quotes for the f-string and single quotes for the dictionary key expressions?

A: Yes, you can work around this issue by using double quotes for the f-string and single quotes for the dictionary key expressions. However, this is not a recommended solution as it can lead to inconsistent code and make it harder to maintain.

Q: Is this issue specific to the astor library or can it occur with other libraries as well?

A: This issue is specific to the astor library. However, it is possible that other libraries may have similar issues with roundtripping f-strings.

Q: How can I report this issue to the astor library maintainers?

A: You can report this issue to the astor library maintainers by opening a new issue on the astor library's GitHub page. Be sure to provide a clear and concise description of the issue, including any relevant code examples and error messages.

Q: What is the current status of this issue in the astor library?

A: The current status of this issue in the astor library is that it is a known problem and is being tracked in the astor library's issue tracker. However, there is no estimated timeline for when this issue will be resolved.

Q: Can I contribute to the resolution of this issue in the astor library?

A: Yes, you can contribute to the resolution of this issue in the astor library by submitting a pull request with a fix for the issue. Be sure to follow the astor library's contribution guidelines and to thoroughly test your fix before submitting it.

Q: What are some alternative libraries that I can use for roundtripping f-strings?

A: Some alternative libraries that you can use for roundtripping f-strings include astunparse and pyupgrade. However, be sure to evaluate each library's features and limitations before choosing one use.