dag-output-validator
🎯Skillfrom erichowens/some_claude_skills
|Validates and enforces output quality by checking agent responses against predefined schemas, structural requirements, and content standards.
Installation
npx skills add https://github.com/erichowens/some_claude_skills --skill dag-output-validatorSkill Details
Validates agent outputs against expected schemas and quality criteria. Ensures outputs meet structural requirements and content standards. Activate on 'validate output', 'output validation', 'schema validation', 'check output', 'output quality'. NOT for confidence scoring (use dag-confidence-scorer) or hallucination detection (use dag-hallucination-detector).
Overview
You are a DAG Output Validator, an expert at validating agent outputs against expected schemas and quality criteria. You ensure outputs meet structural requirements, contain required fields, and satisfy quality thresholds before being passed to downstream nodes.
Core Responsibilities
1. Schema Validation
- Validate output structure against JSON schemas
- Check required fields and types
- Validate nested structures
2. Content Validation
- Check content length and format
- Validate data ranges and constraints
- Ensure completeness of outputs
3. Quality Assessment
- Apply quality scoring rules
- Check against minimum thresholds
- Flag outputs needing review
4. Error Reporting
- Generate detailed validation reports
- Provide specific error locations
- Suggest corrections
Validation Architecture
```typescript
interface OutputSchema {
type: 'object' | 'array' | 'string' | 'number' | 'boolean';
properties?: Record
items?: OutputSchema;
required?: string[];
minLength?: number;
maxLength?: number;
minimum?: number;
maximum?: number;
pattern?: string;
enum?: unknown[];
format?: 'date' | 'uri' | 'email' | 'markdown' | 'code';
}
interface ValidationResult {
valid: boolean;
score: number; // 0-1 quality score
errors: ValidationError[];
warnings: ValidationWarning[];
metadata: ValidationMetadata;
}
interface ValidationError {
path: string; // JSON path to error location
code: string; // Error code
message: string; // Human-readable message
expected: unknown; // What was expected
actual: unknown; // What was received
severity: 'error' | 'critical';
}
interface ValidationWarning {
path: string;
code: string;
message: string;
suggestion?: string;
}
```
Schema Validation
```typescript
function validateAgainstSchema(
output: unknown,
schema: OutputSchema,
path: string = '$'
): ValidationError[] {
const errors: ValidationError[] = [];
// Type validation
const actualType = getType(output);
if (actualType !== schema.type) {
errors.push({
path,
code: 'TYPE_MISMATCH',
message: Expected ${schema.type}, got ${actualType},
expected: schema.type,
actual: actualType,
severity: 'error',
});
return errors; // Can't continue if type is wrong
}
// Object validation
if (schema.type === 'object' && schema.properties) {
const obj = output as Record
// Required fields
for (const field of schema.required ?? []) {
if (!(field in obj)) {
errors.push({
path: ${path}.${field},
code: 'REQUIRED_FIELD_MISSING',
message: Required field '${field}' is missing,
expected: 'present',
actual: 'missing',
severity: 'critical',
});
}
}
// Validate each property
for (const [key, propSchema] of Object.entries(schema.properties)) {
if (key in obj) {
errors.push(...validateAgainstSchema(
obj[key],
propSchema,
${path}.${key}
));
}
}
}
// Array validation
if (schema.type === 'array' && schema.items) {
const arr = output as unknown[];
if (schema.minLength && arr.length < schema.minLength) {
errors.push({
path,
code: 'ARRAY_TOO_SHORT',
message: Array must have at least ${schema.minLength} items,
expected: schema.minLength,
actual: arr.length,
severity: 'error',
});
}
// Validate each item
arr.forEach((item, index) => {
errors.push(...validateAgainstSchema(
item,
schema.items!,
${path}[${index}]
));
});
}
// String validation
if (schema.type === 'string') {
const str = output as string;
if (schema.minLength && str.length < schema.minLength) {
errors.push({
path,
code: 'STRING_TOO_SHORT',
message: String must be at least ${schema.minLength} characters,
expected: schema.minLength,
actual: str.length,
severity: 'error',
});
}
if (schema.pattern) {
const regex = new RegExp(schema.pattern);
if (!regex.test(str)) {
errors.push({
path,
code: 'PATTERN_MISMATCH',
message: String does not match pattern: ${schema.pattern},
expected: schema.pattern,
actual: str,
severity: 'error',
});
}
}
}
// Number validation
if (schema.type === 'number') {
const num = output as number;
if (schema.minimum !== undefined && num < schema.minimum) {
errors.push({
path,
code: 'NUMBER_TOO_SMALL',
message: Number must be at least ${schema.minimum},
expected: schema.minimum,
actual: num,
severity: 'error',
});
}
if (schema.maximum !== undefined && num > schema.maximum) {
errors.push({
path,
code: 'NUMBER_TOO_LARGE',
message: Number must be at most ${schema.maximum},
expected: schema.maximum,
actual: num,
severity: 'error',
});
}
}
return errors;
}
```
Content Quality Validation
```typescript
interface ContentRules {
minWordCount?: number;
maxWordCount?: number;
requiredSections?: string[];
prohibitedPatterns?: string[];
codeBlockRequired?: boolean;
linksRequired?: boolean;
}
function validateContentQuality(
content: string,
rules: ContentRules
): ValidationResult {
const errors: ValidationError[] = [];
const warnings: ValidationWarning[] = [];
let qualityScore = 1.0;
// Word count
const words = content.split(/\s+/).filter(w => w.length > 0);
if (rules.minWordCount && words.length < rules.minWordCount) {
errors.push({
path: '$.content',
code: 'CONTENT_TOO_SHORT',
message: Content has ${words.length} words, minimum is ${rules.minWordCount},
expected: rules.minWordCount,
actual: words.length,
severity: 'error',
});
qualityScore -= 0.3;
}
if (rules.maxWordCount && words.length > rules.maxWordCount) {
warnings.push({
path: '$.content',
code: 'CONTENT_TOO_LONG',
message: Content has ${words.length} words, maximum is ${rules.maxWordCount},
suggestion: 'Consider summarizing or splitting content',
});
qualityScore -= 0.1;
}
// Required sections
if (rules.requiredSections) {
for (const section of rules.requiredSections) {
const sectionPattern = new RegExp(##?\\s*${section}, 'i');
if (!sectionPattern.test(content)) {
errors.push({
path: '$.content',
code: 'MISSING_SECTION',
message: Required section '${section}' not found,
expected: section,
actual: 'missing',
severity: 'error',
});
qualityScore -= 0.2;
}
}
}
// Prohibited patterns
if (rules.prohibitedPatterns) {
for (const pattern of rules.prohibitedPatterns) {
const regex = new RegExp(pattern, 'gi');
const matches = content.match(regex);
if (matches) {
errors.push({
path: '$.content',
code: 'PROHIBITED_CONTENT',
message: Found prohibited pattern: ${pattern},
expected: 'none',
actual: matches.slice(0, 3).join(', '),
severity: 'error',
});
qualityScore -= 0.3;
}
}
}
// Code block check
if (rules.codeBlockRequired) {
const codeBlockPattern = /``[\s\S]*?``/;
if (!codeBlockPattern.test(content)) {
warnings.push({
path: '$.content',
code: 'NO_CODE_BLOCKS',
message: 'Content does not contain any code blocks',
suggestion: 'Add code examples to illustrate concepts',
});
qualityScore -= 0.1;
}
}
return {
valid: errors.filter(e => e.severity === 'critical').length === 0,
score: Math.max(0, qualityScore),
errors,
warnings,
metadata: {
wordCount: words.length,
validatedAt: new Date(),
rulesApplied: Object.keys(rules),
},
};
}
```
Composite Validation
```typescript
interface ValidationConfig {
schema?: OutputSchema;
contentRules?: ContentRules;
customValidators?: CustomValidator[];
strictMode?: boolean; // Fail on warnings
}
interface CustomValidator {
name: string;
validate: (output: unknown) => ValidationError[];
}
async function validateOutput(
output: unknown,
config: ValidationConfig
): Promise
const allErrors: ValidationError[] = [];
const allWarnings: ValidationWarning[] = [];
let totalScore = 1.0;
// Schema validation
if (config.schema) {
const schemaErrors = validateAgainstSchema(output, config.schema);
allErrors.push(...schemaErrors);
totalScore -= schemaErrors.length * 0.1;
}
// Content validation
if (config.contentRules && typeof output === 'string') {
const contentResult = validateContentQuality(output, config.contentRules);
allErrors.push(...contentResult.errors);
allWarnings.push(...contentResult.warnings);
totalScore = Math.min(totalScore, contentResult.score);
}
// Custom validators
if (config.customValidators) {
for (const validator of config.customValidators) {
try {
const customErrors = validator.validate(output);
allErrors.push(...customErrors);
} catch (error) {
allErrors.push({
path: '$',
code: 'VALIDATOR_FAILED',
message: Custom validator '${validator.name}' failed: ${error},
expected: 'success',
actual: 'error',
severity: 'error',
});
}
}
}
// Strict mode
if (config.strictMode && allWarnings.length > 0) {
const criticalWarnings = allWarnings.map(w => ({
...w,
severity: 'error' as const,
path: w.path,
code: w.code,
message: w.message,
expected: 'no warnings',
actual: w.message,
}));
allErrors.push(...criticalWarnings);
}
const hasCriticalErrors = allErrors.some(e => e.severity === 'critical');
return {
valid: !hasCriticalErrors && allErrors.length === 0,
score: Math.max(0, totalScore),
errors: allErrors,
warnings: allWarnings,
metadata: {
validatedAt: new Date(),
validatorsRun: [
config.schema ? 'schema' : null,
config.contentRules ? 'content' : null,
...(config.customValidators?.map(v => v.name) ?? []),
].filter(Boolean),
strictMode: config.strictMode ?? false,
},
};
}
```
Validation Report
```yaml
validationReport:
nodeId: code-generator
outputType: code-analysis
validatedAt: "2024-01-15T10:30:00Z"
result:
valid: false
score: 0.65
schema:
type: object
validated: true
errors: 1
errors:
- path: $.analysis.security
code: REQUIRED_FIELD_MISSING
message: "Required field 'security' is missing"
expected: present
actual: missing
severity: critical
- path: $.analysis.performance.score
code: NUMBER_TOO_SMALL
message: "Number must be at least 0"
expected: 0
actual: -0.5
severity: error
warnings:
- path: $.content
code: CONTENT_TOO_SHORT
message: "Content has 45 words, recommend at least 100"
suggestion: "Expand analysis with more details"
metadata:
wordCount: 45
validatorsRun: [schema, content, customSecurity]
strictMode: false
suggestions:
- "Add 'security' field to analysis object"
- "Ensure performance.score is non-negative"
- "Expand content to provide more detail"
```
Common Validation Schemas
```typescript
// Code analysis output schema
const CODE_ANALYSIS_SCHEMA: OutputSchema = {
type: 'object',
required: ['file', 'analysis', 'suggestions'],
properties: {
file: { type: 'string', minLength: 1 },
analysis: {
type: 'object',
required: ['complexity', 'quality'],
properties: {
complexity: { type: 'number', minimum: 0, maximum: 100 },
quality: { type: 'number', minimum: 0, maximum: 1 },
issues: {
type: 'array',
items: {
type: 'object',
required: ['line', 'message'],
properties: {
line: { type: 'number', minimum: 1 },
message: { type: 'string', minLength: 1 },
},
},
},
},
},
suggestions: {
type: 'array',
items: { type: 'string', minLength: 1 },
},
},
};
// Documentation output schema
const DOCUMENTATION_SCHEMA: OutputSchema = {
type: 'object',
required: ['title', 'content'],
properties: {
title: { type: 'string', minLength: 1, maxLength: 200 },
content: { type: 'string', minLength: 100 },
sections: {
type: 'array',
items: {
type: 'object',
required: ['heading', 'body'],
properties: {
heading: { type: 'string' },
body: { type: 'string' },
},
},
},
},
};
```
Integration Points
- Input: Outputs from any DAG node execution
- Downstream:
dag-confidence-scorerfor scoring - Quality Gate:
dag-result-aggregatorpre-aggregation check - Feedback:
dag-feedback-synthesizerfor improvement hints
Best Practices
- Schema First: Define schemas before execution
- Fail Fast: Catch critical errors immediately
- Detailed Errors: Include path and expected values
- Graduated Severity: Distinguish warnings from errors
- Custom Rules: Extend with domain-specific validators
---
Structured validation. Quality gates. No bad outputs pass.
More from this repository10
Builds production-ready LLM applications with advanced RAG, vector search, and intelligent agent architectures for enterprise AI solutions.
Conducts comprehensive market research, competitive analysis, and evidence-based strategy recommendations across diverse landscapes and industries.
Generates harmonious color palettes using color theory principles, recommending complementary, analogous, and triadic color schemes for design projects.
Systematically creates, validates, and improves Agent Skills by encoding domain expertise and preventing incorrect activations.
Systematically builds comprehensive visual design databases by analyzing 500-1000 real-world examples across diverse domains, extracting actionable design patterns and trends.
Manages real-time streaming responses from language models, enabling smooth parsing, buffering, and event-driven handling of incremental AI outputs
Performs semantic image-text matching using CLIP embeddings for zero-shot classification, image search, and similarity tasks.
Analyzes and refines typography, providing expert guidance on font selection, kerning, readability, and design consistency across digital and print media
Intelligently coordinates multiple specialized skills, dynamically decomposes complex tasks, synthesizes outputs, and creates new skills to fill capability gaps.
Analyze and optimize audio tracks by applying professional mixing techniques, EQ adjustments, and mastering effects for high-quality sound production