mono/packages/kbot/docs/inpaint.md

Image Inpainting and Masking Options

This document outlines potential approaches for implementing an inpainting feature, allowing a user to brush over an area of an image to create a mask that guides the AI for object placement or editing.

Core Concept: Image Masking

The fundamental requirement for inpainting is to create a mask. This is typically a black-and-white image where the white (or black, depending on the AI model's requirements) area indicates the region to be modified by the AI. The original image and this mask are then sent to the AI model.

---

Option 1: Frontend (Client-Side) Approach (Recommended)

This approach handles the mask creation entirely in the user's browser or the Tauri webview.

How it Works

1. Display Image: The source image is loaded and displayed to the user.
2. Canvas Overlay: An HTML <canvas> element is placed directly over the image.
3. Brush Interaction: The user can "paint" on the canvas. The brush strokes are rendered as white shapes on a transparent or black background.
4. Mask Generation: When the user is done, the contents of the canvas are exported as a base64 encoded PNG image. This PNG is the mask.
5. API Call: The original image and the newly generated mask image are sent to the AI for inpainting.

Libraries & Implementation

• Custom Canvas Logic: A simple implementation can be achieved with plain JavaScript and the HTML Canvas API to handle mouse events (mousedown, mousemove, mouseup) and draw lines. This is the most lightweight option.
• Fabric.js / Konva.js: These are powerful canvas libraries that simplify drawing, shapes, and user interaction. They provide a more robust feature set if more advanced editing tools are needed in the future.
• React Components: Libraries like react-canvas-draw or react-sketch-canvas offer pre-built components that can be integrated quickly.

Pros

• Lightweight: No heavy native dependencies are needed on the user's machine. The entire experience is handled by the webview.
• Interactive & Fast: The user gets immediate visual feedback as they draw the mask.
• Cross-Platform: Works everywhere the Tauri application runs without changes.
• Simpler Backend: The backend (images.ts) only needs to receive the image and the mask, without needing to perform any image processing itself.

Cons

• Frontend Complexity: Requires implementing the drawing logic in the React application.

---

Option 2: Backend (Server-Side) Approach

This approach offloads the mask creation to the Node.js backend.

How it Works

1. Capture Coordinates: The frontend captures the user's brush strokes as a series of coordinates (e.g., [{x: 10, y: 20}, {x: 11, y: 21}]).
2. Send to Backend: These coordinates, along with the original image path, are sent to the images.ts script.
3. Process with Sharp/Jimp: A Node.js library like sharp or Jimp is used to:
   • Read the original image to get its dimensions.
   • Create a new blank (black) image of the same size.
   • Draw white lines or shapes onto the blank image using the coordinates received from the frontend.
   • Save this new image as the mask.
4. API Call: The backend then sends the original image and the generated mask to the AI.

Libraries

• sharp: Very fast and powerful, but it is a native Node.js module. This means it requires compilation during npm install and can introduce cross-platform compatibility issues (e.g., needing different binaries for Windows, macOS, Linux, and different architectures like ARM vs. x86). This adds significant complexity to the build and distribution process.
• Jimp: Pure JavaScript, so it has no native dependencies. It's much easier to install and more portable than sharp, but it is significantly slower, which could be a problem for large images or complex masks.

Pros

• Thinner Client: Keeps the image processing logic out of the frontend application.

Cons

• Native Dependencies: Using sharp introduces significant build and maintenance complexity.
• Performance/Latency: There is a delay between drawing and seeing the final mask. Sending large arrays of coordinates can also be slow.
• Less Interactive: The user doesn't get a "live" view of the mask as they are drawing it.

---

Recommendation

The Frontend (Client-Side) Approach is strongly recommended for this application.

Given the interactive nature of the task and the user's explicit concern about native dependencies, a client-side solution using the HTML Canvas is the most practical and efficient choice. It provides the best user experience, avoids the complexities of native modules, and keeps the backend logic simpler.