📊 Enterprise LLMOps & Evaluation
CI/CD & LLM Testing
Automating Prompt Regression Testing with GitHub Actions, Prompt Versioning, and Executing Zero-Risk Shadow & A/B Deployments
In traditional software, modifying code changes deterministic outcomes. In LLM engineering, modifying a single prompt word can fix one edge case while quietly breaking five others (Prompt Regression). Production AI teams must treat prompts like source code — versioning them in Git, running automated evaluation suites on Pull Requests via CI/CD, and deploying new models via Shadow Testing before exposing them to live users.
Key Concepts
Prompt Versioning & GitOps Workflow
Storing prompts as version-controlled text files (or via tools like PromptLayer/Braintrust) to track historical prompt iterations alongside code changes.
Automated Evaluation on Pull Requests (CI/CD)
Running evaluation suites (e.g. Promptfoo, DeepEval) inside GitHub Actions to compare baseline prompt scores against PR changes before merging.
Shadow Deployment (Zero-Risk Traffic Mirroring)
Mirroring live user traffic to both Production Model (V1) and Candidate Model (V2). Returning V1 response to the user while silently logging V2 response for latency and quality comparison.
A/B Canary Traffic Shifting
Gradually shifting 5% -> 25% -> 100% of live traffic to the candidate model while monitoring user feedback (thumbs up/down) and latency metrics.
Automated AI CI/CD & Shadow Deployment Pipeline
1. GitHub Actions Workflow for Prompt Regression Testing
Automatically run an evaluation suite (using Promptfoo) whenever a developer opens a Pull Request modifying prompt templates. The CI job fails if the new prompt degrades accuracy or raises costs beyond defined limits.
.github/workflows/prompt-eval.ymlyaml
1name: LLM Prompt Evaluation CI
2
3on:
4 pull_request:
5 paths:
6 - 'prompts/**'
7 - 'src/prompts.ts'
8
9jobs:
10 evaluate-prompts:
11 runs-on: ubuntu-latest
12 steps:
13 - name: Checkout Code
14 uses: actions/checkout@v4
15
16 - name: Setup Node.js
17 uses: actions/setup-node@v4
18 with:
19 node-version: '20'
20
21 - name: Install Promptfoo
22 run: npm install -g promptfoo
23
24 - name: Run Prompt Assertions & Evals
25 env:
26 OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
27 run: |
28 promptfoo eval --config prompts/promptfooconfig.yaml --no-write
29 # Promptfoo exits with code 1 if pass rate falls below threshold2. Non-Blocking Shadow Deployment Traffic Mirroring Proxy
A Shadow Proxy sends user requests synchronously to Model V1 (Production) while asynchronously mirroring the request to Model V2 (Candidate). The response from V1 is immediately returned to the user, ensuring zero latency impact.
lib/shadow-proxy.tstypescript
1import { openai } from '@ai-sdk/openai';
2import { generateText } from 'ai';
3
4export async function handleUserQueryWithShadow(userPrompt: string) {
5 // 1. Primary synchronous call to Production Model (V1)
6 const primaryPromise = generateText({
7 model: openai('gpt-4o-mini'), // Production model
8 prompt: userPrompt,
9 });
10
11 // 2. Non-blocking shadow call to Candidate Model (V2)
12 generateText({
13 model: openai('gpt-4o'), // Candidate model / new prompt
14 prompt: userPrompt,
15 }).then(shadowResult => {
16 // Log shadow performance & quality metrics asynchronously
17 logShadowMetrics({
18 prompt: userPrompt,
19 candidateResponse: shadowResult.text,
20 candidateLatency: shadowResult.usage,
21 });
22 }).catch(err => console.error('Shadow execution error:', err));
23
24 // Return primary response instantly without waiting for shadow
25 const primaryResult = await primaryPromise;
26 return primaryResult.text;
27}💡
Senior Architect Insight: Never trust manual spot-checking for prompt changes. A prompt that sounds better on 3 test inputs may fail on 30% of edge cases in production. Automated regression testing in CI/CD and Shadow Deployments are non-negotiable for enterprise AI engineering.