Add optional retrieval confidence metadata to RAG response thoughts

[Nithin00614]

Add Retrieval Confidence Metadata

Problem

Currently, there's no visibility into the confidence/quality of retrieved items in RAG responses. This makes debugging and monitoring difficult.

Solution

Add optional, non-breaking metadata to track retrieval confidence:

• Top retrieval score
• Average retrieval score
• Confidence threshold (configurable)
• Confidence level classification (high/low)

Benefits

• Non-breaking (backward compatible)
• Helps debug low-quality retrievals
• Provides metrics for monitoring
• Configurable threshold via self.confidence_threshold

Changes

• Added confidence calculation in answer() method
• Added ThoughtStep with retrieval metrics
• Safe handling for items without scores

Testing

• Tested with items that have scores
• Tested with items without scores
• Verified backward compatibility
• Checked ThoughtStep rendering

Example Output

{
  "title": "Retrieval Confidence",
  "props": {
    "top_score": 0.85,
    "avg_score": 0.72,
    "threshold": 0.7,
    "confidence_level": "high",
    "scored_items_count": 5
  }
}

[Nithin00614]

@microsoft-github-policy-service agree