Feature Request: Introduce `modelDescriptionHook` For Dynamic Tool Descriptions

by ADMIN 80 views

Problem Statement

In recent times, the integration of multiple languageModelTools in our extension has been a significant achievement. However, we've encountered a limitation in the modelDescription field within package.json, which is insufficient for providing the necessary guidance for Large Language Models (LLMs) to use the tools correctly. This limitation has led us to explore alternative solutions to ensure that our extension users receive accurate and model-specific instructions.

Current Workaround

Initially, we attempted to supplement the description via a .github/copilot-instructions.md file. This file is designed to provide additional instructions for LLMs, but managing it from within an extension is problematic. The file resides in the user's workspace, which is outside the extension's control. This makes it challenging to maintain and update the file, leading to inconsistencies and potential errors.

Proposal: Introduce modelDescriptionHook

To address the limitations of the current approach, we propose the introduction of a modelDescriptionHook. This hook would be called whenever an LLM requests a tool description, allowing extension developers to dynamically generate the tool description based on various factors. The hook would consider the following aspects:

  • LLM Model: The specific LLM model being used, such as Sonnet, GPT-4.1, or GPT-4o. As extension developers, we often don't know which LLM a user will use, making it essential to provide model-specific instructions.
  • Tool Capabilities: The capabilities of the tool, which can vary depending on the LLM model and the specific use case.
  • Contextual Factors: Contextual factors that might affect usage, such as the user's preferences, the environment, or the specific task at hand.

By introducing a modelDescriptionHook, extension developers can provide much more precise and model-specific instructions without requiring manual modification of workspace files. This approach would ensure that users receive accurate and relevant guidance, leading to a better overall experience.

Benefits

The introduction of a modelDescriptionHook would bring several benefits to extension developers and users alike:

  • Improved Accuracy: Dynamic tool descriptions would ensure that users receive accurate and model-specific instructions, reducing the likelihood of errors and inconsistencies.
  • Increased Flexibility: The hook would allow extension developers to adapt to different LLM models and use cases, providing a more flexible and responsive experience.
  • Simplified Maintenance: By managing the tool description within the extension, developers can easily update and maintain the descriptions, reducing the need for manual modifications to workspace files.

Related Discussion

This proposal is related to the discussion on GitHub, as suggested by @isidorn. The discussion can be found at https://github.com/microsoft/vscode/issues.

Conclusion

In conclusion, the introduction of a modelDescriptionHook would address the limitations of the current approach and provide a more reliable and manageable way to provide dynamic tool descriptions. By considering the specific LLM model, tool capabilities, and contextual factors, extension developers can provide accurate and model-specific instructions, leading to a better overall experience for users. We believe that this proposal would be a valuable to the extension ecosystem and look forward to considering your feedback.

Future Work

If the modelDescriptionHook proposal is accepted, we plan to:

  • Implement the Hook: Develop and implement the modelDescriptionHook within the extension, ensuring seamless integration with the existing codebase.
  • Test and Refine: Thoroughly test the hook and refine it based on user feedback and testing results.
  • Document the Hook: Provide clear documentation on how to use the modelDescriptionHook, including examples and best practices.

Frequently Asked Questions

We've received several questions regarding the feature request to introduce a modelDescriptionHook for dynamic tool descriptions. Below, we've compiled a list of frequently asked questions and answers to provide clarity on this proposal.

Q1: What is the current limitation with the modelDescription field in package.json?

A1: The modelDescription field in package.json is too limited for providing the level of guidance necessary for LLMs to use the tools correctly. It does not allow for dynamic or model-specific instructions, which can lead to inconsistencies and potential errors.

Q2: Why can't we just use a .github/copilot-instructions.md file?

A2: While a .github/copilot-instructions.md file can provide additional instructions for LLMs, managing it from within an extension is problematic. The file resides in the user's workspace, which is outside the extension's control, making it challenging to maintain and update the file.

Q3: What is the modelDescriptionHook proposal?

A3: The modelDescriptionHook proposal is a feature request to introduce a hook that would be called whenever an LLM requests a tool description. This hook would allow extension developers to dynamically generate the tool description based on various factors, including the specific LLM model, tool capabilities, and contextual factors.

Q4: How would the modelDescriptionHook benefit extension developers and users?

A4: The introduction of a modelDescriptionHook would bring several benefits, including:

  • Improved accuracy: Dynamic tool descriptions would ensure that users receive accurate and model-specific instructions.
  • Increased flexibility: The hook would allow extension developers to adapt to different LLM models and use cases.
  • Simplified maintenance: By managing the tool description within the extension, developers can easily update and maintain the descriptions.

Q5: How would the modelDescriptionHook be implemented?

A5: If the proposal is accepted, we plan to implement the modelDescriptionHook within the extension, ensuring seamless integration with the existing codebase. We would also thoroughly test the hook and refine it based on user feedback and testing results.

Q6: What is the next step for this proposal?

A6: We are currently seeking feedback and consideration from the community. If the proposal is accepted, we would begin implementing the modelDescriptionHook and provide clear documentation on how to use it.

Q7: How can I provide feedback on this proposal?

A7: You can provide feedback on this proposal by commenting on this article or reaching out to us directly. We value your input and look forward to hearing your thoughts.

Q8: What are the potential risks or challenges associated with implementing the modelDescriptionHook?

A8: While we believe that the modelDescriptionHook would bring several benefits, there are potential risks or challenges associated with implementing it. These include:

  • Complexity: Implementing the hook may add complexity to the extension codebase.
  • Maintenance: Managing the tool description within the extension may require additional maintenance efforts.
  • Compatibility: Ensuring compatibility with different LLM models and use cases may be challenging.

Q9: How would the modelDescriptionHook be documented?

A9: We plan to provide clear documentation on how to use the modelDescriptionHook, including examples and best practices. This documentation would be available in the extension's documentation and would be regularly updated to reflect any changes or updates to the hook.

Q10: What is the expected timeline for implementing the modelDescriptionHook?

A10: If the proposal is accepted, we would begin implementing the modelDescriptionHook as soon as possible. The exact timeline would depend on the complexity of the implementation and the availability of resources. We would provide regular updates on the progress and expected completion date.