Rename `pydra.compose.base.fields.Out.callable` To `pydra.compose.base.fields.Out.finder` And Permit Glob Or Regexes

Introduction

Pydra is a Python library designed to simplify the process of composing and executing workflows. Its compose module is a crucial component, allowing users to define and manage complex workflows with ease. However, there is a common use case where the Out.callable field is used to locate generated files within the output directory. In this article, we propose renaming Out.callable to Out.finder and permitting glob or regex patterns to be used. This change will streamline a common use case, clarify the purpose of the field, and make it easier to use.

The Current State of Out.callable

The Out.callable field in Pydra's compose module is used to specify a callable function that is executed when a workflow is run. However, in many cases, this callable function is simply used to locate a generated file within the output directory. This can be achieved using a glob pattern or a regex. The current implementation requires users to define a separate callable function, which can be cumbersome and unnecessary.

The Proposed Change: Renaming Out.callable to Out.finder

We propose renaming Out.callable to Out.finder to better reflect its purpose. This change will make it clear that the field is intended for locating files or resources, rather than executing arbitrary code. Additionally, we suggest permitting glob or regex patterns to be used in the Out.finder field. This will allow users to easily locate files using patterns, without the need to define a separate callable function.

Benefits of the Proposed Change

The proposed change will have several benefits:

• Streamlined workflow definition: By allowing glob or regex patterns to be used in the Out.finder field, users can define workflows more easily and efficiently.
• Improved clarity: The new name Out.finder clearly conveys the purpose of the field, making it easier for users to understand and use.
• Reduced boilerplate code: Users will no longer need to define separate callable functions to locate files, reducing the amount of boilerplate code required.

Auto-conversion of Strings to Callable Functions

To further simplify the use of Out.finder, we propose auto-converting strings to default glob or regex patterns. This will allow users to easily convert a string pattern to a callable function that performs the glob. For example, a user can specify a string pattern like output/*.txt in the Out.finder field, and Pydra will automatically convert it to a callable function that performs the glob.

Implementation Details

To implement the proposed change, we will need to modify the Out.callable field in Pydra's compose module. We will rename the field to Out.finder and add support for glob and regex patterns. We will also implement auto-conversion of strings to callable functions.

Example Use Cases

Here are some example use cases that demonstrate the benefits of the proposed change:

• Locating a generated file: Suppose we have a workflow that generates a file named output.txt in the output directory. We can use the Out.finder field to locate the file using a glob pattern like output/*.txt.
• Using a regex pattern: Suppose we have a workflow that generates a file with a name that matches a specific regex pattern. We can use the Out.finder field to locate the file using a regex pattern like output/.*\.txt$.

Conclusion

In conclusion, renaming Out.callable to Out.finder and permitting glob or regex patterns will streamline a common use case, clarify the purpose of the field, and make it easier to use. The proposed change will also reduce boilerplate code and improve the overall usability of Pydra's compose module. We believe that this change will benefit the Pydra community and make it easier for users to define and manage complex workflows.

Future Work

In the future, we plan to explore additional features and improvements for the Out.finder field, such as:

• Support for multiple patterns: Allowing users to specify multiple glob or regex patterns in the Out.finder field.
• Improved auto-conversion: Enhancing the auto-conversion of strings to callable functions to support more complex patterns.
• Integration with other Pydra features: Integrating the Out.finder field with other Pydra features, such as the Out.file field, to provide a more seamless user experience.
  Frequently Asked Questions: Renaming Out.callable to Out.finder ====================================================================

Q: What is the purpose of renaming Out.callable to Out.finder?

A: The purpose of renaming Out.callable to Out.finder is to better reflect its purpose and make it easier to use. The Out.finder field is intended for locating files or resources, rather than executing arbitrary code.

Q: Why is the current name Out.callable misleading?

A: The current name Out.callable is misleading because it implies that the field is intended for executing code, rather than locating files or resources. This can lead to confusion and make it harder for users to understand and use the field.

Q: What benefits will the proposed change bring?

A: The proposed change will bring several benefits, including:

• Streamlined workflow definition: By allowing glob or regex patterns to be used in the Out.finder field, users can define workflows more easily and efficiently.
• Improved clarity: The new name Out.finder clearly conveys the purpose of the field, making it easier for users to understand and use.
• Reduced boilerplate code: Users will no longer need to define separate callable functions to locate files, reducing the amount of boilerplate code required.

Q: How will the auto-conversion of strings to callable functions work?

A: The auto-conversion of strings to callable functions will work by automatically converting a string pattern to a callable function that performs the glob. For example, a user can specify a string pattern like output/*.txt in the Out.finder field, and Pydra will automatically convert it to a callable function that performs the glob.

Q: Will the proposed change affect existing workflows?

A: No, the proposed change will not affect existing workflows. The Out.finder field will be backwards compatible with existing workflows, and users will be able to continue using the field as they do now.

Q: What are the potential drawbacks of the proposed change?

A: The potential drawbacks of the proposed change are:

• Increased complexity: The proposed change may add complexity to the Out.finder field, making it harder for users to understand and use.
• Potential for errors: The auto-conversion of strings to callable functions may lead to errors if the string pattern is not correctly formatted.

Q: How will the proposed change be implemented?

A: The proposed change will be implemented by modifying the Out.callable field in Pydra's compose module. The field will be renamed to Out.finder and support for glob and regex patterns will be added. The auto-conversion of strings to callable functions will also be implemented.

Q: What is the timeline for implementing the proposed change?

A: The timeline for implementing the proposed change is not yet determined. However, we anticipate that the change will be implemented in the near future, pending further discussion and review.

Q: How can users provide feedback on the proposed change?

A: Users can provide feedback on the change by submitting comments and suggestions through the Pydra issue tracker. We encourage users to share their thoughts and opinions on the proposed change to help us make an informed decision.