# Web URL Support for kbot

The kbot tool now supports including web URLs as source material for AI processing alongside local files.

## Features

- **Web Page Support**: Process HTML content and convert it to markdown for the AI
- **JSON API Support**: Handle JSON responses from web APIs
- **Caching**: Automatically cache web content for one week to improve performance
- **Mixed Sources**: Combine local files and web URLs in a single command

## How to Use

Include web URLs in your commands using the `-i` or `--include` parameter:

```bash
# Basic usage with a single web URL
kbot "Summarize this documentation" -i https://raw.githubusercontent.com/polymech/polymech-mono/main/README.md

# Multiple sources (local file and web URL)
kbot "Compare this code with the documentation" -i src/index.ts,https://docs.npmjs.com/cli/v10/commands/npm-install

# For JSON APIs
kbot "Extract user emails" -i https://jsonplaceholder.typicode.com/users
```

## Caching

Web content is cached in the `./.cache/https` directory with a default expiration of one week. This reduces unnecessary network requests and improves performance for repeated queries on the same URLs.

## Technical Details

- HTML content is converted to markdown using the Turndown library
- JSON responses are formatted and presented as code blocks
- All web requests use axios with a custom user agent
- Cache management automatically handles expiration and refreshing of content

## Testing

A test script is included to verify web URL functionality:

```bash
node test_web_urls.js
```