44 lines
1.6 KiB
Markdown
44 lines
1.6 KiB
Markdown
# Web URL Support for kbot
|
|
|
|
The kbot tool now supports including web URLs as source material for AI processing alongside local files.
|
|
|
|
## Features
|
|
|
|
- **Web Page Support**: Process HTML content and convert it to markdown for the AI
|
|
- **JSON API Support**: Handle JSON responses from web APIs
|
|
- **Caching**: Automatically cache web content for one week to improve performance
|
|
- **Mixed Sources**: Combine local files and web URLs in a single command
|
|
|
|
## How to Use
|
|
|
|
Include web URLs in your commands using the `-i` or `--include` parameter:
|
|
|
|
```bash
|
|
# Basic usage with a single web URL
|
|
kbot "Summarize this documentation" -i https://raw.githubusercontent.com/polymech/polymech-mono/main/README.md
|
|
|
|
# Multiple sources (local file and web URL)
|
|
kbot "Compare this code with the documentation" -i src/index.ts,https://docs.npmjs.com/cli/v10/commands/npm-install
|
|
|
|
# For JSON APIs
|
|
kbot "Extract user emails" -i https://jsonplaceholder.typicode.com/users
|
|
```
|
|
|
|
## Caching
|
|
|
|
Web content is cached in the `./.cache/https` directory with a default expiration of one week. This reduces unnecessary network requests and improves performance for repeated queries on the same URLs.
|
|
|
|
## Technical Details
|
|
|
|
- HTML content is converted to markdown using the Turndown library
|
|
- JSON responses are formatted and presented as code blocks
|
|
- All web requests use axios with a custom user agent
|
|
- Cache management automatically handles expiration and refreshing of content
|
|
|
|
## Testing
|
|
|
|
A test script is included to verify web URL functionality:
|
|
|
|
```bash
|
|
node test_web_urls.js
|
|
``` |