Troubleshooting AI Issues

​

Inconsistent results

​

Problem: Different output each time

• Prompt is too vague or open-ended
• Missing essential context
• Relying on AI to infer rather than instruct

Add specific constraints: "Focus only on payment terms"
Include clear context: "We are the vendor with low leverage"
Define output format: "List exactly 3 issues in order of importance"

​

Problem: Quality varies wildly

• Prompt works for some document types but not others
• Context requirements vary by situation
• Model updates affecting performance

• Create document-specific prompts rather than universal ones
• Add conditional logic: “If customer is government entity, then…”
• Test prompts regularly and update as needed

​

Wrong focus

​

Problem: AI focuses on wrong issues

• No priority guidance provided
• Equal weight given to all issues
• Missing risk framework

"Ignore formatting and stylistic issues"
"Focus only on provisions creating financial risk over $10,000"
"Prioritize: 1) Liability 2) Payment 3) IP rights - ignore everything else"

​

Problem: Too much or too little detail

• No length specification
• Audience unclear
• Purpose undefined

"One sentence per issue"
"Detailed analysis with specific clause references"
"Executive summary: 3 bullet points maximum"

​

Context problems

​

Problem: AI doesn’t understand the situation

• Insufficient context provided
• Wrong context emphasized
• Contradictory information given

1. Who you are: "Small startup vendor"
2. Document type: "Enterprise MSA"
3. Leverage: "Very low - competitive RFP"
4. Constraints: "Cannot accept uncapped liability per insurance"

​

Problem: AI ignores critical context

• Context buried in long prompt
• Competing contexts confusing priority
• Key context mentioned too late

• Lead with critical context
• Repeat important constraints
• Use headers to organize context clearly

​

Output format issues

​

Problem: Wrong format for needs

• Format not specified
• Conflicting format instructions
• AI defaulting to its preferences

"Create a table with these exact columns: Issue | Risk | Action"
"Draft as email suitable for sending to opposing counsel"
"Bullet points only - no prose paragraphs"

​

Problem: Information scattered

• No structure requested
• Multiple asks in one prompt
• AI trying to be comprehensive

"Organize by: 1) Critical issues 2) Medium priority 3) Minor points"
"Start with your recommendation, then explain"
"Group all payment-related issues together"

​

Accuracy issues

​

Problem: AI misses important provisions

• Rule/prompt too narrow
• Keywords not matching document language
• Important sections in unexpected places

• Broaden search terms: “audit, inspection, review, examination”
• Check entire document: “Review all sections including exhibits”
• Be explicit: “Check for liability caps anywhere in document”

​

Problem: False positives

• Prompt too broad
• Missing negative prompting
• No threshold for materiality

"Only flag if liability exceeds $100,000"
"Ignore standard industry terms"
"Do not flag: payment terms, notice provisions, standard definitions"

​

Performance issues

​

Problem: AI takes forever

• Overly complex prompts
• Multiple large documents
• Requesting too much in one prompt

• Break into smaller requests
• Process documents separately
• Simplify prompt structure

​

Problem: AI cuts off mid-response

• Output length limits reached
• Token limits exceeded
• Complex chains causing timeouts

• Request more concise output
• Break into multiple prompts
• Ask for summary first, details separately

​

Model-specific issues

​

Problem: Prompt works in one model but not another

• Models trained differently
• Format preferences vary
• Instruction interpretation differs

• Adjust formatting for the specific model you’re using
• Test prompts on your target platform
• Use model-appropriate structure (some models prefer markdown headers, others respond better to XML tags)

​

Problem: Quality degraded after update

• Model improvements changing behavior
• Prompt exploiting previous quirks
• Overfitting to old model version

• Simplify and clarify prompts
• Remove workarounds for old issues
• Test and update prompt library

​

Quick diagnostic checklist

1. Context: Is it clear who you are and what you need?
2. Specificity: Are instructions precise or vague?
3. Format: Did you specify how you want output?
4. Scope: Is the AI looking at the right things?
5. Constraints: Are there clear boundaries?
6. Priority: Does the AI know what matters most?

​

Iterative refinement

​

Step 1: Identify the gap

​

Step 2: Make one change at a time

Attempt 1: "Review this agreement for risks."
→ Too broad. Add scope.

Attempt 2: "Review this agreement for risks related to liability and indemnification."
→ Right focus, but too much detail. Add constraints.

Attempt 3: "Review this agreement for risks related to liability and indemnification. List the top 3 issues only."
→ Getting there. Add context.

Attempt 4: "We are the vendor with low leverage. Review this agreement for risks related to liability and indemnification. List the top 3 issues only."
→ Focused, contextual, constrained.

​

Step 3: Save what works

​

Emergency fixes

1. ​

   When nothing works

"I'm the vendor. Find the three biggest risks in this agreement."

2. ​

   When pressed for time

3. ​

   When results are dangerous

​

Prevention strategies

• Test early and often Don’t wait until the big deal to discover your prompt doesn’t work.
• Document what works When you find a good prompt, save it immediately with notes on why it worked.
• Build gradually Start simple and add complexity only as needed.
• Stay updated AI models evolve. What worked last month might need adjustment today.

​

Remember