79 lines
3.6 KiB
Markdown
79 lines
3.6 KiB
Markdown
# kbot Test Overview
|
|
|
|
This document outlines the categories of tests implemented for the kbot system, along with potential new tests that could be added.
|
|
|
|
## Existing Test Categories
|
|
|
|
- **Basic Operations**: Simple operations to test basic functionality (addition, multiplication, division)
|
|
- **Math Operations**: More complex mathematical operations (factorials, square roots, equations)
|
|
- **Language Operations**: Tests focused on language processing (translation, grammar, summarization)
|
|
- **Format Operations**: Tests for formatting different data types (JSON, markdown, code, dates)
|
|
- **Coding Capabilities**: Tests that generate and execute code (factorial, bubble sort)
|
|
- **File Operations**: Tests that process files (single file, multiple files, glob patterns)
|
|
- **LLM Tools Operations**: Tests that use LLM tools (reading/writing files, directory operations)
|
|
|
|
## Potential New Tests
|
|
|
|
### Option-Specific Tests
|
|
|
|
These tests would validate specific options from `zod_schema.ts`:
|
|
|
|
1. **Router Selection**
|
|
- **File**: `tests/unit/router.test.ts`
|
|
- **Prompt**: `Use the openai router to generate a haiku about AI. Return only the haiku, no explanation.`
|
|
- **Expected**: A valid haiku about AI
|
|
- **Options**: `{ router: 'openai' }`
|
|
|
|
2. **Mode Selection**
|
|
- **File**: `tests/unit/mode.test.ts`
|
|
- **Prompt**: `What files are in the current directory? Use tools to help you.`
|
|
- **Expected**: A JSON array of files
|
|
- **Options**: `{ mode: 'tools' }`
|
|
|
|
3. **Filter Applications**
|
|
- **File**: `tests/unit/filters.test.ts`
|
|
- **Prompt**: `Write a JavaScript function that calculates the factorial of a number. Include comments.`
|
|
- **Expected**: Clean code without markdown formatting
|
|
- **Options**: `{ mode: 'completion', filters: 'code', dst: './output/factorial.js' }`
|
|
|
|
4. **Variable Substitution**
|
|
- **File**: `tests/unit/variables.test.ts`
|
|
- **Prompt**: `Tell me about ${TOPIC} in one sentence.`
|
|
- **Expected**: A sentence about the specified topic
|
|
- **Options**: `{ variables: { TOPIC: 'artificial intelligence' } }`
|
|
|
|
5. **Include Patterns**
|
|
- **File**: `tests/unit/include.test.ts`
|
|
- **Prompt**: `Analyze the code in these files and list all function names as a JSON array.`
|
|
- **Expected**: A JSON array of function names
|
|
- **Options**: `{ include: ['src/*.ts', 'tests/*.ts'] }`
|
|
|
|
### Integration Tests
|
|
|
|
1. **Multi-File Processing**
|
|
- **File**: `tests/integration/multi_file.test.ts`
|
|
- **Prompt**: `Compare the implementations in these two files and suggest optimizations.`
|
|
- **Expected**: Meaningful comparison and optimization suggestions
|
|
- **Options**: `{ include: ['src/file1.ts', 'src/file2.ts'] }`
|
|
|
|
2. **Tool Combination**
|
|
- **File**: `tests/integration/tool_combo.test.ts`
|
|
- **Prompt**: `Analyze this codebase, identify performance bottlenecks, and suggest fixes.`
|
|
- **Expected**: Analysis with specific file references and code suggestions
|
|
- **Options**: `{ mode: 'tools', tools: ['read', 'write', 'analyze'] }`
|
|
|
|
3. **End-to-End Workflow**
|
|
- **File**: `tests/integration/e2e.test.ts`
|
|
- **Prompt**: `Create a simple Node.js application that fetches data from an API and saves it to a file.`
|
|
- **Expected**: Full application with multiple files created
|
|
- **Options**: `{ mode: 'tools', output: './output/app/' }`
|
|
|
|
## Implementation Considerations
|
|
|
|
When implementing these tests:
|
|
|
|
1. Use `runTest` from `commons.ts` for consistency
|
|
2. Support multiple models using `getDefaultModels()`
|
|
3. Generate reports using `generateTestReport`
|
|
4. Set appropriate timeouts for API calls
|
|
5. Check both expected outputs and side effects (file creation, etc.) |