mono/packages/kbot/tests/unit/overview.md

kbot Test Overview

This document outlines the categories of tests implemented for the kbot system, along with potential new tests that could be added.

Existing Test Categories

• Basic Operations: Simple operations to test basic functionality (addition, multiplication, division)
• Math Operations: More complex mathematical operations (factorials, square roots, equations)
• Language Operations: Tests focused on language processing (translation, grammar, summarization)
• Format Operations: Tests for formatting different data types (JSON, markdown, code, dates)
• Coding Capabilities: Tests that generate and execute code (factorial, bubble sort)
• File Operations: Tests that process files (single file, multiple files, glob patterns)
• LLM Tools Operations: Tests that use LLM tools (reading/writing files, directory operations)

Potential New Tests

Option-Specific Tests

These tests would validate specific options from zod_schema.ts:

1. Router Selection

   • File: tests/unit/router.test.ts
   • Prompt: Use the openai router to generate a haiku about AI. Return only the haiku, no explanation.
   • Expected: A valid haiku about AI
   • Options: { router: 'openai' }

2. Mode Selection

   • File: tests/unit/mode.test.ts
   • Prompt: What files are in the current directory? Use tools to help you.
   • Expected: A JSON array of files
   • Options: { mode: 'tools' }

3. Filter Applications

   • File: tests/unit/filters.test.ts
   • Prompt: Write a JavaScript function that calculates the factorial of a number. Include comments.
   • Expected: Clean code without markdown formatting
   • Options: { mode: 'completion', filters: 'code', dst: './output/factorial.js' }

4. Variable Substitution

   • File: tests/unit/variables.test.ts
   • Prompt: Tell me about ${TOPIC} in one sentence.
   • Expected: A sentence about the specified topic
   • Options: { variables: { TOPIC: 'artificial intelligence' } }

5. Include Patterns

   • File: tests/unit/include.test.ts
   • Prompt: Analyze the code in these files and list all function names as a JSON array.
   • Expected: A JSON array of function names
   • Options: { include: ['src/*.ts', 'tests/*.ts'] }

Integration Tests

1. Multi-File Processing

   • File: tests/integration/multi_file.test.ts
   • Prompt: Compare the implementations in these two files and suggest optimizations.
   • Expected: Meaningful comparison and optimization suggestions
   • Options: { include: ['src/file1.ts', 'src/file2.ts'] }

2. Tool Combination

   • File: tests/integration/tool_combo.test.ts
   • Prompt: Analyze this codebase, identify performance bottlenecks, and suggest fixes.
   • Expected: Analysis with specific file references and code suggestions
   • Options: { mode: 'tools', tools: ['read', 'write', 'analyze'] }

3. End-to-End Workflow

   • File: tests/integration/e2e.test.ts
   • Prompt: Create a simple Node.js application that fetches data from an API and saves it to a file.
   • Expected: Full application with multiple files created
   • Options: { mode: 'tools', output: './output/app/' }

Implementation Considerations

When implementing these tests:

1. Use runTest from commons.ts for consistency
2. Support multiple models using getDefaultModels()
3. Generate reports using generateTestReport
4. Set appropriate timeouts for API calls
5. Check both expected outputs and side effects (file creation, etc.)