Practical Guide: Building and Testing AI Agents
(For CIOs and Engineering Teams)

This guide is designed to help CIOs, engineering leads, and technical teams navigate the evolving landscape of AI agents. Whether you're experimenting with low-code platforms or preparing for full-scale deployment, this document offers a step-by-step approach, from rapid prototyping using visual tools to advanced model testing, fine-tuning, and infrastructure planning. It highlights key platforms like Flowise, Amazon Q, Microsoft Copilot Studio, and hyperscaler solutions from Azure, AWS, and Google, empowering teams to build, evaluate, and scale AI agents with confidence and control.

C. Microsoft Copilot Studio

• Build internal copilots and flows via Teams or Web

• Customize triggers, topics, and dialog flows

• Integrate with Azure OpenAI or plugins

• Link to Power Automate and SharePoint

• Use connectors for business process automation

What You Can Do with Copilot Studio

• Create multi-turn agent dialogs

• Train behavior via prompt variations and fallbacks

• Add context using Power Platform data

• Configure custom actions and workflows

• Add logging and telemetry via Azure Monitor

C. Google Vertex AI Studio

Overview:

Google Vertex AI provides access to Gemini 1.5 (formerly Bard), Codey, Imagen (for vision), and PaLM 2. The platform is tightly integrated with Google Cloud's data and ML tools and includes a rich set of features for prototyping, tuning, and deploying generative models.

Key Capabilities:

• Model Access: Gemini 1.5 Flash and Pro, PaLM 2, Imagen (multimodal), Codey (code generation)

• Prompt Playground: Live prompt testing with temperature, top-k/p, safety filters, and grounding options

• Model Comparison: Available in notebooks or side-by-side prompt testing UI

• Fine-Tuning: Adapter-based tuning or instruction tuning with evaluation reports

• Contextual Grounding: Integrates with Google Search, Document AI, and Retrieval-augmented generation (RAG)

• Security & Governance: IAM, audit trails, data isolation, no model retraining from your inputs

• Tool Integrations: Easily connects with BigQuery, Cloud Storage, and GKE for deployment

Free vs Paid:

• $300 in free credits for new Google Cloud users

• Model usage billed per token; Gemini Flash is lower cost than Pro

• Daily quota limits apply under the free tier

What Is a Model Leaderboard?

A model leaderboard ranks AI models based on standard benchmarks such as:

• MMLU (Massive Multitask Language Understanding)

• GPQA (Graduate-Level Professional QA)

• TruthfulQA (Factual accuracy)

• HumanEval (Code generation)

• DROP (Reading comprehension)

Multimodal benchmarks (for models that handle text, image, audio)

These leaderboards help users evaluate models for tasks like reasoning, coding, summarisation, and multimodal understanding.

• LLM Stats: Tracks over 100 models including GPT-4o, Claude, Gemini, Grok, LLaMA, DeepSeek, Mistral, and more

• ArtificialAnalysis.ai: Offers comparisons by intelligence, latency, cost, and context window

• Azure AI Foundry Portal: Azure Foundry includes model leaderboards and benchmarking tools in its portal (currently in preview)

You can compare models by performance, cost per million tokens, context length, and modality support

PACT Framework for AI Model Evaluation

The PACT framework offers a strategic lens for assessing AI models across four enterprise-relevant dimensions:

• Performance: Evaluates model accuracy, reasoning capabilities, and benchmark scores (e.g., MMLU, HumanEval, TruthfulQA), ensuring alignment with business-critical tasks.

• Architecture: Reviews model design, including modality support (text, image, audio), context window, and openness (proprietary vs open-source), to determine scalability and integration potential.

• Cost: Assesses total cost of ownership, including inference pricing, latency, and infrastructure requirements, enabling budget-conscious decision-making.

• Task Fit: Measures how well the model aligns with specific operational use cases, such as summarisation, retrieval-augmented generation (RAG), or multimodal interaction.

This framework supports executive-level decisions by balancing technical performance with financial and strategic fit.

RICE Framework for AI Model Evaluation

• Adapted from product prioritisation, the RICE framework is increasingly used to evaluate and rank AI models based on business impact:

• Reach: Estimates the number of users, systems, or workflows that will benefit from the model.

• Impact: Quantifies the expected improvement in performance, efficiency, or strategic outcomes.

• Confidence: Reflects the reliability of performance estimates, grounded in benchmark data and prior deployments.

• Effort: Accounts for the resources required to deploy, fine-tune, and maintain the model.

The RICE score is calculated as:

RICE Score = (Reach × Impact × Confidence) / Effort

This approach enables prioritisation of models that deliver the highest value with the least complexity, supporting enterprise-wide alignment and resource optimisation. 

Azure AI Foundry

• Compare OSS models in the same interface

• LoRA-based fine-tuning supported

• Community models for quick testing

Google Vertex AI

• Access models via Model Garden

• Test prompt variations in the Playground

• Use pipelines to compare and track outputs

AWS Bedrock / SageMaker

• Use Bedrock for Claude, Mistral, Titan

• Configure prompt and settings

• Fine-tune and deploy with SageMaker workflows

How to Compare Models

• Use same prompt across Claude, GPT, Mistral, Gemma

• Tools: PromptLayer, OpenRouter.ai

• Benchmark latency, cost per call, and model accuracy

• Useful for tuning temperature, logging results, and prompt A/B testing