llm-plastic/README.md

Complete Markdown Guide — Training & Serving an Open‑Source LLM on Your Own PDFs / Chat Logs

Goal: Start with raw PDFs, Discord logs, forum dumps, or scraped HTML and end with a tuned, production‑ready large‑language‑model (LLM).
Toolchain: 100 % open‑source; every major step is linked.

---

0 . Quick bird’s‑eye view

Phase	Core Tools	What happens
Ingest & Clean	unstructured, LangChain loaders	Parse PDFs / chats → structured text & metadata
Dataset Build	🤗 datasets, dedup scripts	Combine, filter, split into training / eval
Fine‑Tuning (SFT / QLoRA)	Axolotl, Unsloth, Torchtune, PEFT	Lightweight parameter‑efficient updates
Alignment (RLHF / DPO)	TRL	Reward modelling & preference optimization
Distributed Training	DeepSpeed, Colossal‑AI	Multi‑GPU / multi‑node scaling
Evaluation	lm‑eval‑harness, Ragas	Benchmarks + domain‑specific tests
Serving	vLLM, Hugging Face TGI	Fast, OpenAI‑compatible inference APIs

---

Finetuning

1 . Set up the environment

# CUDA 12‑ready image (example)
docker run --gpus all -it --shm-size 64g nvcr.io/nvidia/pytorch:24.04-py3 bash
conda create -n llm python=3.11 && conda activate llm
pip install "torch>=2.2" "transformers>=4.40" accelerate bitsandbytes


---

# Fine‑Tuning‑Only vs. Full Alignment Pipeline for Open‑Source LLMs

| Aspect | Fine‑Tuning Only<br>(SFT / LoRA / QLoRA) | Full Pipeline<br>(SFT → DPO / RLHF) |
|--------|------------------------------------------|--------------------------------------|
| **Purpose** | Adapt model to new domain/tasks | Adapt **and** align answers with human preferences, safety rules |
| **Data needed** | 1‑100 k *single* instruction‑response pairs | Same SFT set **plus** 5‑100 k *preference* pairs (chosen + rejected) |
| **Compute** | Fits on 1 × A100‑40 GB for an 8 B model with QLoRA (−60 % VRAM, +39 % time) :contentReference[oaicite:0]{index=0} | Adds reward‑model + alignment pass ⇒ ≈ 2‑3× GPU hours (DPO cheapest) :contentReference[oaicite:1]{index=1} |
| **Training time** | Hours → Llama‑Factory LoRA = 3.7 × faster than P‑Tuning :contentReference[oaicite:2]{index=2} | Hours‑to‑days; alignment stage can add 50‑200 % wall‑time :contentReference[oaicite:3]{index=3} |
| **Cash cost** | Example SFT‑only: 72.5 h on 2 × A100 ≈ €200 | Same run + 1 DPO epoch ⇒ €312 (+56 %) :contentReference[oaicite:4]{index=4} |
| **Quality gains** | ↑ task accuracy, but may hallucinate or be off‑style | +18–30 % win‑rate on preference evals; fewer toxic / off‑policy replies :contentReference[oaicite:5]{index=5} |
| **Safety** | Relies on prompt guardrails | Alignment directly penalises unsafe outputs |
| **Complexity** | Single command; no extra data pipeline | Multi‑stage: collect feedback, train reward, run PPO/DPO, tune hyper‑params |
| **When to choose** | Narrow internal apps, low‑risk use, tight budget | Public‑facing chatbots, regulated domains, brand‑sensitive content |

---

## Slimmed‑Down “Fine‑Tuning‑Only” Recipe (QLoRA + Unsloth)

```bash
pip install "unsloth[colab-new]" datasets bitsandbytes accelerate
python -m unsloth.finetune \
  --model meta-llama/Meta-Llama-3-8B \
  --dataset ./my_corpus.parquet \
  --lora_r 16 --lora_alpha 32 --lr 2e-5 --epochs 3 \
  --output_dir llama3-domain-qlora


## Portability concerns

Form | Size (7 B model) | Typical file | Pros | Cons
LoRA / QLoRA adapter(only Δ‑weights) | 25‑200 MB | adapter_model.safetensors | Tiny, upload to HF Hub, stack several at once, hot‑swap at inference | Needs identical base checkpoint & tokenizer; some runtimes must support LoRA
Merged FP16 weights | 8‑16 GB | pytorch_model‑00001‑of‑00002.safetensors | Single self‑contained model; any engine that speaks Hugging‑Face can load it | Heavy; re‑quantise for each HW target
Quantised GGUF | 2‑8 GB | model.Q4_K_M.gguf | Runs on CPU / mobiles with llama.cpp, Ollama; LoRA can be loaded too | GPU engines (vLLM/TGI) ignore gguf

2. Runtimes & current LoRA support

Runtime / Client | Load LoRA live? | Merge required? | Note
🤗 Transformers | ✅ peft.auto | ❌ | canonical reference
vLLM ≥ 0.4 | ✅ --enable‑lora → pulls from HF Hub GitHub | ❌ | remote LoRA download at startup
Hugging Face TGI 1.3 | ✅ (--lora-adapter) | ❌ | hot‑swap w/out restart
llama.cpp / GGUF | ✅ load LoRA GGUF side‑file (or merge) GitHub | ❌ | convert PEFT LoRA → GGUF first
ONNX / TensorRT | ⚠️ must be merged first | ✅ | quantise after merge

3. Format conversions you can rely on

Conversion | Command / Tool | Portability gain
PEFT LoRA → GGUF | llama.cpp/convert_lora_to_gguf.py | lets CPU‑only clients consume your fine‑tune GitHub
Merged weights → GGUF | python llama.cpp/convert.py --outtype q4_0 GitHub | shrink & run on laptops
PyTorch → Safetensors | model.save_pretrained(..., safe_serialization=True) GitHub | faster, pickle‑free, HF‑native