Expansion of AI Features to Include Private LLM (local) and Azure Open AI

This proposal outlines the concept of integrating Private LLM and Azure Open AI alongside the existing Open AI gateway. This expansion will provide users with more options, enhanced privacy control, and greater flexibility in leveraging AI within Capacities.

Concept:

The proposed feature allows users to choose from a variety of AI models for their AI Assistant needs:

Private LLM: Capacities will support the integration of privately hosted language models, ensuring that sensitive data remains within the user's infrastructure.

Azure Open AI: Integration with Azure Open AI allows users to utilize Microsoft's powerful AI capabilities, expanding the range of options available.

Advantages:

• Enhanced Privacy Control:
• Private LLM will empower users to keep their data within their own environments, addressing privacy concerns.
• Azure Open AI, with its robust security measures, adds an extra layer of data protection.
• Flexibility and Customization:
• Private LLM cater to users with stringent security requirements or those who prefer on-premises solutions.
• Azure Open AI broadens the range of AI capabilities available, providing users with a diverse set of tools to enhance productivity.
• Reduced Dependency on Third-Party Services:
• Hosting your own models or utilizing private services reduces reliance on external platforms, contributing to data sovereignty and independence.
• Scalability and Performance:
• Users can choose models that best fit their requirements, optimizing for performance, scalability, and cost-effectiveness.

While supporting Custom LLM can be a tricky and challenging task (but, to be honest in case of https://github.com/jmorganca/ollama this task is pretty simple - just add ability to setup url like in logseq), support Azure Open AI should be the easiest feature ever - just add a toggle which will allow use endpoint from azure (with appropriate model and deployment setup)