What needs improvement with Azure AI Foundry?

Azure AI Foundry could be improved with better integration within all the other tools from Microsoft.

With regard to security, compliance, or governance features in Azure AI Foundry, this is something that we have started looking into, primarily using Microsoft Purview for our governance, data governance. There is this new module called DSPM for AI, and we are exploring it while trying to operationalize it with different policies and so forth, but we're still not where we want to be on the governance, AI governance side. It's a process and a path, and we are trying to work through that right now. Azure AI Foundry can be improved from the governance perspective, as a lot can be done. The promising part is the recent announcement on the Foundry control plane. A couple of days back, there was an announcement regarding it bringing in some of the gaps that were on the platform, so it's a really positive direction in terms of where it's going. More governance is what is lacking, but the control plane will really play a big role there.

The platform's effect on my management of privacy, performance, and compliance across different regions is quite complex because Azure AI Foundry does not make it very clear how to deploy. We set up a foundry in a region and it is hard to know how to deploy different models into other regions that Azure AI Foundry has created. We have a lot of back and forth just trying to figure that out and how to optimize our deployments. What did not work well for us regarding Azure AI Foundry includes the security piece, being able to identify how to deploy to multiple regions, reducing latency, and managing tokens per minute. Determining the right amount of tokens and gauging that before going to production, as well as load testing, would be helpful, especially because we have load tested with the real AI and incurred high costs. Finding the right balance without high costs is essential. The experience with pricing, setup cost, and licensing for Azure AI Foundry has been straightforward, as pricing involves tokens charged per token. The setup was fairly easy. I have not seen a return on investment with Azure AI Foundry because we have not fully deployed it into production; we are looking to go live soon. We have not really utilized or been able to measure that metric yet, but from what we have looked at, we have probably seen a seventy percent reduction on the translation services for some things we have done that will be implemented soon. We have not captured metrics on the document comparison piece since we have not deployed it to production.

Azure AI Foundry, alongside Agent Foundry and the classic Azure OpenAI components, offers ease of switching between them. However, I find that the online documentation can sometimes be confusing, requiring extensive searching through multiple articles to locate specific information. The tools are satisfactory, but understanding them might be challenging at times, which makes in-person events beneficial for conducting hands-on labs and enhancing my learning. I feel the documentation may benefit from clearer presentation on occasion.

To improve Azure AI Foundry, I think I'd appreciate a more consistent story so it's easier for my clients to understand where Foundry fits in the ecosystem of Microsoft tools as well as what it can and can't do. I would also like to see the observability tooling improve a great deal. I feel it's inadequate compared to some of the other tooling that's out there, with an example being LangSmith, which I would say is the market leader in that space, so catching up in that area would be a great help.

I assess the integration of Azure AI Foundry with existing cloud services as good, but it can be improved. Even though I have only been utilizing Azure AI Foundry for the past four months, I think the understanding between Copilot Studio and Azure AI Foundry is still somewhat unclear regarding which one to use when and why, and how they complement each other is a journey we are currently undertaking.

I think Microsoft has been improving these, especially with some of the announcements they had this week, but my biggest critique is some of the fragmentation of their different AI services they have, including AI Open, OpenAI, Azure OpenAI, and Azure Foundry. They feel very disjointed sometimes, so having a unified single experience for all of that, which I think is what they're trying to do here with Azure AI Foundry and some of the improvements that they've made, would be ideal.

To improve Azure AI Foundry, I think the next release should mainly include self-hosted containers for AI agents and the ability to build multi-agent orchestration.

I think Azure AI Foundry can be improved with more integration with some of the low-code aspects of Copilot Studio. I'm seeing some of that at Ignite, but I haven't seen it myself in practice yet. Being able to take something from a low-code scenario into a pro scenario that we have an experience with in Azure AI Foundry would be great.

One of the big things Azure AI Foundry could improve is continuously evolving the governance elements and how, while I know they exist, the more control we can have over different elements and observation of what different agents are doing, the better.

I would appreciate it if Microsoft could improve Azure AI Foundry by releasing new features immediately because sometimes we have to wait for weeks or months to use them. Since these technologies are changing rapidly, once we decide on the platform and choose Microsoft, everything goes with Microsoft, which makes it difficult to move things around.

Azure AI Foundry can be improved by adding educational features.

To improve Azure AI Foundry, I would integrate Copilot Studio and Azure AI Foundry because at the moment they are two different resources. What our customers ask for is one resource that brings together Copilot and Foundry, and that is what is missing now for a unified experience. For Azure AI Foundry, there is no actual clear pricing structure, which can be confusing for customers to understand, as every feature that you activate has its own price, and it is not very clear sometimes to define the pricing.

I can only assess the integration of Azure AI Foundry with existing cloud services for Azure. While I find it pretty simple, I think there is more that could be done in Foundry that is missing, as there are third parties such as AIM Security and PromptFeed that are not integrated in Foundry as of now. I would improve Azure AI Foundry by adding more functions within Foundry itself, as right now it is quite basic in what it does. I would also enhance integration with Azure policies and Azure functions where orchestration can be done.

Assessing the stability and reliability of Azure AI Foundry is hard. When we were building it and trying to see which models to even choose, it was hard to keep track. I have seen some new tools for prompt evaluations and prompt improvement coming out, but it is hit and miss. You are really writing prompts now instead of writing code, which are language prompts, but you really need tools to test and figure out what the results came out and then have a measurable way to figure out if the prompt can be improved. Right now, it was all Excel or just try hit and miss. We are not really organized there. With code, you know what the binary result is, but with prompting, it is a lot harder. You definitely need tools that can help. In Azure AI Foundry, I did not see many tools inside to test it that can really help us. I wish there were more tooling. Even in Visual Studio, something should take the prompt, run it on all the different models, and then show which one is a better result by itself. Right now, we had to do it by hand.

I would assess the integration of Azure AI Foundry with existing cloud services as fairly good. My feedback is that it is not really out-of-the-box. Integration-wise, it does integrate with many Azure services, but in order to use it in Azure AI Foundry, you need to get permissions. Once you try it out, you hit into conditions and policy issues. So, in terms of experience, it is probably not very good. However, in terms of integrating with other components, I think the possibilities are excellent. From a usability standpoint, I do not think it is that user-friendly. The first step to improve Azure AI Foundry is to get it approved and accepted by stakeholders. For a financial institution, Azure AI Foundry probably does not provide enough information for us to actually acknowledge that it is secure enough to adopt it. We cannot fully utilize it. We use it to try things out and to see what is there, but we cannot fully utilize it in production yet. Providing data on the internal workings of Azure AI Foundry would help customers like us feel more comfortable adopting it.