Web URL Support for kbot

The kbot tool now supports including web URLs as source material for AI processing alongside local files.

Features

Web Page Support: Process HTML content and convert it to markdown for the AI
JSON API Support: Handle JSON responses from web APIs
Caching: Automatically cache web content for one week to improve performance
Mixed Sources: Combine local files and web URLs in a single command

How to Use

Include web URLs in your commands using the -i or --include parameter:

# Basic usage with a single web URL
kbot "Summarize this documentation" -i https://raw.githubusercontent.com/polymech/polymech-mono/main/README.md

# Multiple sources (local file and web URL)
kbot "Compare this code with the documentation" -i src/index.ts,https://docs.npmjs.com/cli/v10/commands/npm-install

# For JSON APIs
kbot "Extract user emails" -i https://jsonplaceholder.typicode.com/users

Caching

Web content is cached in the ./.cache/https directory with a default expiration of one week. This reduces unnecessary network requests and improves performance for repeated queries on the same URLs.

Technical Details

HTML content is converted to markdown using the Turndown library
JSON responses are formatted and presented as code blocks
All web requests use axios with a custom user agent
Cache management automatically handles expiration and refreshing of content

Testing

A test script is included to verify web URL functionality:

node test_web_urls.js

1.6 KiB Raw Blame History