agent-smith/packages/GLiNER2/tutorial/10-lora_adapters.md

Tutorial 10: LoRA Adapters - Multi-Domain Inference

Table of Contents

1. Introduction
2. Why Use LoRA Adapters?
3. Training Your First Adapter
4. Training Multiple Domain Adapters
5. Loading and Swapping Adapters
6. Real-World Use Cases
7. Best Practices
8. Troubleshooting

Introduction

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique that allows you to train specialized adapters for different domains without modifying the base model. This enables:

• Fast domain switching: Swap between domains in milliseconds
• Minimal storage: Adapters are ~2-10 MB vs ~100-500 MB for full models
• Domain specialization: Train separate adapters for legal, medical, financial, etc.
• Easy deployment: Keep one base model + multiple lightweight adapters

Why Use LoRA Adapters?

Memory Efficiency

Full Model Fine-tuning:
- Legal model:     450 MB
- Medical model:   450 MB
- Financial model: 450 MB
Total:            1.35 GB

LoRA Adapters:
- Base model:      450 MB
- Legal adapter:     5 MB
- Medical adapter:   5 MB
- Financial adapter: 5 MB
Total:             465 MB (65% less!)

Fast Training

LoRA adapters train 2-3x faster than full fine-tuning because:

• Only ~1-5% of parameters are trainable
• Smaller gradient computations
• Less GPU memory required

Easy Multi-Domain Inference

# One base model, multiple domains
model = GLiNER2.from_pretrained("fastino/gliner2-base-v1")

# Legal domain
model.load_adapter("./legal_adapter")
legal_results = model.extract_entities(legal_text, ["company", "law"])

# Medical domain (swap in <1 second)
model.load_adapter("./medical_adapter")
medical_results = model.extract_entities(medical_text, ["disease", "drug"])

Training Your First Adapter

Step 1: Prepare Domain-Specific Data

from gliner2.training.data import InputExample

# Legal domain examples
legal_examples = [
    InputExample(
        text="Apple Inc. filed a lawsuit against Samsung Electronics.",
        entities={"company": ["Apple Inc.", "Samsung Electronics"]}
    ),
    InputExample(
        text="The plaintiff Google LLC accused Microsoft Corporation of patent infringement.",
        entities={"company": ["Google LLC", "Microsoft Corporation"]}
    ),
    InputExample(
        text="Tesla Motors settled the case with the Securities and Exchange Commission.",
        entities={
            "company": ["Tesla Motors"],
            "organization": ["Securities and Exchange Commission"]
        }
    ),
    # Add 100-1000+ examples for best results
]

Step 2: Configure LoRA Training

from gliner2 import GLiNER2
from gliner2.training.trainer import GLiNER2Trainer, TrainingConfig

# LoRA configuration
config = TrainingConfig(
    output_dir="./legal_adapter",
    experiment_name="legal_domain",
    
    # Training parameters
    num_epochs=10,
    batch_size=8,
    gradient_accumulation_steps=2,
    encoder_lr=1e-5,
    task_lr=5e-4,
    
    # LoRA settings
    use_lora=True,                              # Enable LoRA
    lora_r=8,                                   # Rank (4, 8, 16, 32)
    lora_alpha=16.0,                           # Scaling factor (usually 2*r)
    lora_dropout=0.0,                          # Dropout for LoRA layers
    lora_target_modules=["encoder"],           # Apply to all encoder layers (query, key, value, dense)
    save_adapter_only=True,                    # Save only adapter (not full model)
    
    # Optimization
    eval_strategy="epoch",  # Evaluates and saves at end of each epoch
    eval_steps=500,  # Used when eval_strategy="steps"
    logging_steps=50,
    fp16=True,  # Use mixed precision if GPU available
)

Step 3: Train the Adapter

# Load base model
base_model = GLiNER2.from_pretrained("fastino/gliner2-base-v1")

# Create trainer
trainer = GLiNER2Trainer(model=base_model, config=config)

# Train adapter
trainer.train(train_data=legal_examples)

# Adapter automatically saved to ./legal_adapter/final/

Training output:

🔧 LoRA Configuration
======================================================================
Enabled            : True
Rank (r)           : 8
Alpha              : 16.0
Scaling (α/r)      : 2.0000
Dropout            : 0.0
Target modules     : query, key, value, dense
LoRA layers        : 144
----------------------------------------------------------------------
Trainable params   : 1,327,104 / 124,442,368 (1.07%)
Memory savings     : ~98.9% fewer gradients
======================================================================

***** Running Training *****
  Num examples = 1000
  Num epochs = 10
  Batch size = 8
  Effective batch size = 16
  Total optimization steps = 625
  LoRA enabled: 1,327,104 trainable / 124,442,368 total (1.07%)

Training Multiple Domain Adapters

Let's train adapters for three different domains: Legal, Medical, and Customer Support.

Complete Multi-Domain Training Script

from gliner2 import GLiNER2
from gliner2.training.trainer import GLiNER2Trainer, TrainingConfig
from gliner2.training.data import InputExample

# ============================================================================
# Define Domain Data
# ============================================================================

# Legal domain
legal_examples = [
    InputExample(
        text="Apple Inc. filed a lawsuit against Samsung Electronics.",
        entities={"company": ["Apple Inc.", "Samsung Electronics"]}
    ),
    InputExample(
        text="The plaintiff Google LLC accused Microsoft Corporation of patent infringement.",
        entities={"company": ["Google LLC", "Microsoft Corporation"]}
    ),
    # Add more examples...
]

# Medical domain
medical_examples = [
    InputExample(
        text="Patient diagnosed with Type 2 Diabetes and Hypertension.",
        entities={"disease": ["Type 2 Diabetes", "Hypertension"]}
    ),
    InputExample(
        text="Prescribed Metformin 500mg twice daily and Lisinopril 10mg once daily.",
        entities={
            "drug": ["Metformin", "Lisinopril"],
            "dosage": ["500mg", "10mg"]
        }
    ),
    # Add more examples...
]

# Customer support domain
support_examples = [
    InputExample(
        text="Customer John Smith reported issue with Order #12345.",
        entities={
            "customer": ["John Smith"],
            "order_id": ["Order #12345"]
        }
    ),
    InputExample(
        text="Refund of $99.99 processed for Order #98765 on 2024-01-15.",
        entities={
            "order_id": ["Order #98765"],
            "amount": ["$99.99"],
            "date": ["2024-01-15"]
        }
    ),
    # Add more examples...
]

# ============================================================================
# Training Function
# ============================================================================

def train_domain_adapter(
    base_model_name: str,
    examples: list,
    domain_name: str,
    output_dir: str = "./adapters"
):
    """Train a LoRA adapter for a specific domain."""
    
    adapter_path = f"{output_dir}/{domain_name}_adapter"
    
    config = TrainingConfig(
        output_dir=adapter_path,
        experiment_name=f"{domain_name}_domain",
        
        # Training
        num_epochs=10,
        batch_size=8,
        gradient_accumulation_steps=2,
        encoder_lr=1e-5,
        task_lr=5e-4,
        
        # LoRA
        use_lora=True,
        lora_r=8,
        lora_alpha=16.0,
        lora_dropout=0.0,
        lora_target_modules=["encoder"],  # All encoder layers
        save_adapter_only=True,
        
        # Logging & Checkpointing
        eval_strategy="no",  # Set to "epoch" or "steps" if you have validation set
        eval_steps=500,  # Used when eval_strategy="steps"
        logging_steps=50,
        fp16=True,
    )
    
    # Load base model
    print(f"\n{'='*60}")
    print(f"Training {domain_name.upper()} adapter")
    print(f"{'='*60}")
    
    model = GLiNER2.from_pretrained(base_model_name)
    trainer = GLiNER2Trainer(model=model, config=config)
    
    # Train
    results = trainer.train(train_data=examples)
    
    print(f"\n✅ {domain_name.capitalize()} adapter trained!")
    print(f"📁 Saved to: {adapter_path}/final/")
    print(f"⏱️  Training time: {results['total_time_seconds']:.2f}s")
    
    return f"{adapter_path}/final"

# ============================================================================
# Train All Adapters
# ============================================================================

if __name__ == "__main__":
    BASE_MODEL = "fastino/gliner2-base-v1"
    
    # Train adapters for each domain
    legal_adapter_path = train_domain_adapter(
        BASE_MODEL, legal_examples, "legal"
    )
    
    medical_adapter_path = train_domain_adapter(
        BASE_MODEL, medical_examples, "medical"
    )
    
    support_adapter_path = train_domain_adapter(
        BASE_MODEL, support_examples, "support"
    )
    
    print("\n" + "="*60)
    print("🎉 All adapters trained successfully!")
    print("="*60)
    print(f"Legal adapter:    {legal_adapter_path}")
    print(f"Medical adapter:  {medical_adapter_path}")
    print(f"Support adapter:  {support_adapter_path}")

Loading and Swapping Adapters

Basic Usage

from gliner2 import GLiNER2

# Load base model once
model = GLiNER2.from_pretrained("fastino/gliner2-base-v1")

# Load legal adapter
model.load_adapter("./adapters/legal_adapter/final")

# Use the model
result = model.extract_entities(
    "Apple Inc. sued Samsung over patent rights.",
    ["company", "legal_action"]
)
print(result)

Swapping Between Adapters

# Load base model
model = GLiNER2.from_pretrained("fastino/gliner2-base-v1")

# Legal domain
print("📋 Legal Analysis:")
model.load_adapter("./adapters/legal_adapter/final")
legal_text = "Google LLC filed a complaint against Oracle Corporation."
legal_result = model.extract_entities(legal_text, ["company", "legal_action"])
print(f"  {legal_result}")

# Swap to medical domain
print("\n🏥 Medical Analysis:")
model.load_adapter("./adapters/medical_adapter/final")
medical_text = "Patient presents with Pneumonia and was prescribed Amoxicillin."
medical_result = model.extract_entities(medical_text, ["disease", "drug"])
print(f"  {medical_result}")

# Swap to support domain
print("\n💬 Support Analysis:")
model.load_adapter("./adapters/support_adapter/final")
support_text = "Customer reported Order #12345 not delivered on time."
support_result = model.extract_entities(support_text, ["order_id", "issue"])
print(f"  {support_result}")

# Use base model without adapter
print("\n🔧 Base Model (no adapter):")
model.unload_adapter()
base_result = model.extract_entities("Some generic text", ["entity"])
print(f"  {base_result}")

Output:

📋 Legal Analysis:
  {'entities': [{'text': 'Google LLC', 'label': 'company', ...}, 
                {'text': 'Oracle Corporation', 'label': 'company', ...}]}

🏥 Medical Analysis:
  {'entities': [{'text': 'Pneumonia', 'label': 'disease', ...},
                {'text': 'Amoxicillin', 'label': 'drug', ...}]}

💬 Support Analysis:
  {'entities': [{'text': 'Order #12345', 'label': 'order_id', ...}]}

🔧 Base Model (no adapter):
  {'entities': [{'text': 'text', 'label': 'entity', ...}]}

Batch Processing with Adapter Swapping

def process_documents_by_domain(model, documents_by_domain, adapters):
    """
    Process multiple documents across different domains efficiently.
    
    Args:
        model: Base GLiNER2 model
        documents_by_domain: Dict[domain_name, List[document_text]]
        adapters: Dict[domain_name, adapter_path]
    
    Returns:
        Dict[domain_name, List[results]]
    """
    results = {}
    
    for domain, documents in documents_by_domain.items():
        print(f"Processing {domain} domain ({len(documents)} documents)...")
        
        # Load domain-specific adapter
        model.load_adapter(adapters[domain])
        
        # Process all documents for this domain
        domain_results = []
        for doc in documents:
            result = model.extract_entities(doc, get_entity_types(domain))
            domain_results.append(result)
        
        results[domain] = domain_results
    
    return results

def get_entity_types(domain):
    """Get entity types for each domain."""
    types = {
        "legal": ["company", "person", "law", "legal_action"],
        "medical": ["disease", "drug", "symptom", "procedure"],
        "support": ["customer", "order_id", "product", "issue"]
    }
    return types.get(domain, ["entity"])

# Example usage
model = GLiNER2.from_pretrained("fastino/gliner2-base-v1")

documents_by_domain = {
    "legal": [
        "Apple Inc. filed suit against Samsung.",
        "Microsoft acquired LinkedIn for $26B.",
    ],
    "medical": [
        "Patient has Type 2 Diabetes.",
        "Prescribed Metformin 500mg daily.",
    ],
    "support": [
        "Issue with Order #12345 reported.",
        "Refund processed for Order #98765.",
    ]
}

adapters = {
    "legal": "./adapters/legal_adapter/final",
    "medical": "./adapters/medical_adapter/final",
    "support": "./adapters/support_adapter/final",
}

results = process_documents_by_domain(model, documents_by_domain, adapters)

# Results organized by domain
for domain, domain_results in results.items():
    print(f"\n{domain.upper()} Results:")
    for i, result in enumerate(domain_results, 1):
        print(f"  Document {i}: {len(result['entities'])} entities found")

Real-World Use Cases

Use Case 1: Multi-Tenant SaaS Platform

class MultiTenantEntityExtractor:
    """Entity extraction service for multi-tenant platform."""
    
    def __init__(self, base_model_name: str, tenant_adapters: dict):
        """
        Args:
            base_model_name: Path to base model
            tenant_adapters: Dict mapping tenant_id to adapter_path
        """
        self.model = GLiNER2.from_pretrained(base_model_name)
        self.tenant_adapters = tenant_adapters
        self.current_tenant = None
    
    def extract_for_tenant(self, tenant_id: str, text: str, entity_types: list):
        """Extract entities for specific tenant."""
        # Load tenant-specific adapter if needed
        if self.current_tenant != tenant_id:
            adapter_path = self.tenant_adapters.get(tenant_id)
            if adapter_path:
                self.model.load_adapter(adapter_path)
            else:
                self.model.unload_adapter()  # Use base model
            self.current_tenant = tenant_id
        
        return self.model.extract_entities(text, entity_types)

# Setup
extractor = MultiTenantEntityExtractor(
    base_model_name="fastino/gliner2-base-v1",
    tenant_adapters={
        "legal_firm_123": "./adapters/legal_adapter/final",
        "hospital_456": "./adapters/medical_adapter/final",
        "ecommerce_789": "./adapters/support_adapter/final",
    }
)

# Usage
legal_result = extractor.extract_for_tenant(
    "legal_firm_123",
    "Apple sued Samsung",
    ["company"]
)

medical_result = extractor.extract_for_tenant(
    "hospital_456",
    "Patient has diabetes",
    ["disease"]
)

Use Case 2: Document Classification Pipeline

def classify_and_extract(document: str, model: GLiNER2, adapters: dict):
    """
    Classify document type and extract relevant entities.
    
    1. Classify document type using base model
    2. Load appropriate domain adapter
    3. Extract domain-specific entities
    """
    # Step 1: Classify document type
    doc_type_result = model.extract_entities(
        document,
        ["legal_document", "medical_record", "support_ticket", "financial_report"]
    )
    
    # Determine document type
    if doc_type_result['entities']:
        doc_type = doc_type_result['entities'][0]['label']
        doc_type = doc_type.replace("_document", "").replace("_record", "").replace("_ticket", "").replace("_report", "")
    else:
        doc_type = "general"
    
    # Step 2: Load appropriate adapter
    adapter_mapping = {
        "legal": adapters.get("legal"),
        "medical": adapters.get("medical"),
        "support": adapters.get("support"),
        "financial": adapters.get("financial"),
    }
    
    if doc_type in adapter_mapping and adapter_mapping[doc_type]:
        model.load_adapter(adapter_mapping[doc_type])
    
    # Step 3: Extract domain-specific entities
    entity_types = {
        "legal": ["company", "person", "law", "legal_action"],
        "medical": ["disease", "drug", "symptom", "procedure", "dosage"],
        "support": ["customer", "order_id", "product", "issue", "status"],
        "financial": ["company", "amount", "date", "stock_symbol"],
    }
    
    entities = model.extract_entities(
        document,
        entity_types.get(doc_type, ["entity"])
    )
    
    return {
        "document_type": doc_type,
        "entities": entities['entities']
    }

# Usage
model = GLiNER2.from_pretrained("fastino/gliner2-base-v1")

adapters = {
    "legal": "./adapters/legal_adapter/final",
    "medical": "./adapters/medical_adapter/final",
    "support": "./adapters/support_adapter/final",
}

document = "Patient John Smith diagnosed with Type 2 Diabetes on 2024-01-15."
result = classify_and_extract(document, model, adapters)

print(f"Document Type: {result['document_type']}")
print(f"Entities: {result['entities']}")

Use Case 3: A/B Testing Adapters

import random

class AdapterABTester:
    """A/B test different adapter versions."""
    
    def __init__(self, base_model_name: str, adapter_variants: dict):
        """
        Args:
            adapter_variants: {"v1": path1, "v2": path2, ...}
        """
        self.model = GLiNER2.from_pretrained(base_model_name)
        self.adapter_variants = adapter_variants
        self.results = {variant: [] for variant in adapter_variants}
    
    def test_sample(self, text: str, entity_types: list, true_entities: list):
        """Test a sample with all adapter variants."""
        sample_results = {}
        
        for variant, adapter_path in self.adapter_variants.items():
            # Load variant
            self.model.load_adapter(adapter_path)
            
            # Get predictions
            pred = self.model.extract_entities(text, entity_types)
            
            # Compute metrics
            f1 = self.compute_f1(pred['entities'], true_entities)
            
            sample_results[variant] = {
                "predictions": pred['entities'],
                "f1_score": f1
            }
            
            self.results[variant].append(f1)
        
        return sample_results
    
    def compute_f1(self, predicted, ground_truth):
        """Simple F1 computation (simplified for demo)."""
        pred_set = {(e['text'], e['label']) for e in predicted}
        true_set = {(e['text'], e['label']) for e in ground_truth}
        
        if not pred_set and not true_set:
            return 1.0
        if not pred_set or not true_set:
            return 0.0
        
        tp = len(pred_set & true_set)
        precision = tp / len(pred_set) if pred_set else 0
        recall = tp / len(true_set) if true_set else 0
        
        if precision + recall == 0:
            return 0.0
        return 2 * precision * recall / (precision + recall)
    
    def get_summary(self):
        """Get A/B test summary."""
        summary = {}
        for variant, scores in self.results.items():
            if scores:
                summary[variant] = {
                    "avg_f1": sum(scores) / len(scores),
                    "samples": len(scores)
                }
        return summary

# Usage
tester = AdapterABTester(
    base_model_name="fastino/gliner2-base-v1",
    adapter_variants={
        "v1_r4": "./adapters/legal_v1_r4/final",
        "v2_r8": "./adapters/legal_v2_r8/final",
        "v3_r16": "./adapters/legal_v3_r16/final",
    }
)

# Test samples
test_samples = [
    {
        "text": "Apple Inc. sued Samsung Electronics.",
        "entity_types": ["company"],
        "true_entities": [
            {"text": "Apple Inc.", "label": "company"},
            {"text": "Samsung Electronics", "label": "company"}
        ]
    },
    # More samples...
]

for sample in test_samples:
    results = tester.test_sample(
        sample["text"],
        sample["entity_types"],
        sample["true_entities"]
    )

# Get summary
summary = tester.get_summary()
for variant, metrics in summary.items():
    print(f"{variant}: Avg F1 = {metrics['avg_f1']:.3f} ({metrics['samples']} samples)")

Best Practices

1. Choosing LoRA Hyperparameters

# Small datasets (< 1K examples)
config = TrainingConfig(
    lora_r=4,           # Lower rank = fewer parameters
    lora_alpha=8.0,     # alpha = 2 * r
    num_epochs=10,
)

# Medium datasets (1K-10K examples)
config = TrainingConfig(
    lora_r=8,           # Standard rank
    lora_alpha=16.0,
    num_epochs=5,
)

# Large datasets (> 10K examples)
config = TrainingConfig(
    lora_r=16,          # Higher rank = more capacity
    lora_alpha=32.0,
    num_epochs=3,
)

2. Target Module Selection

Understanding Module Groups:

GLiNER2 supports fine-grained control over which layers receive LoRA adaptation:

# Option 1: Encoder only - all layers (query, key, value, dense)
# Use case: General domain adaptation, good starting point
# Memory: Moderate (~1-2% of model parameters)
lora_target_modules=["encoder"]

# Option 2: Encoder - attention layers only
# Use case: Very memory-constrained scenarios
# Memory: Low (~0.5-1% of model parameters)
lora_target_modules=["encoder.query", "encoder.key", "encoder.value"]

# Option 3: Encoder - FFN layers only
# Use case: Alternative to attention-only, sometimes better for certain tasks
# Memory: Low (~0.5-1% of model parameters)
lora_target_modules=["encoder.dense"]

# Option 4: Encoder + task heads
# Use case: When you want to adapt both representation and task-specific layers
# Memory: Moderate-High (~2-4% of model parameters)
lora_target_modules=["encoder", "span_rep", "classifier"]

# Option 5: All modules (DEFAULT)
# Use case: Maximum adaptation capacity, best performance
# Memory: High (~3-5% of model parameters)
lora_target_modules=["encoder", "span_rep", "classifier", "count_embed", "count_pred"]

Recommendations:

• Start with encoder only (["encoder"]) for most tasks
• Add task heads if performance is insufficient
• Use attention-only for extreme memory constraints
• Use all modules (default) when you need maximum performance

3. Adapter Organization

project/
├── base_model/
│   └── gliner2-base-v1/
├── adapters/
│   ├── legal/
│   │   ├── v1_r8/
│   │   │   └── final/
│   │   └── v2_r16/
│   │       └── final/
│   ├── medical/
│   │   └── final/
│   └── support/
│       └── final/
└── scripts/
    ├── train_adapters.py
    └── evaluate_adapters.py

4. Version Control for Adapters

# adapter_metadata.json
{
    "legal_v1": {
        "path": "./adapters/legal/v1_r8/final",
        "base_model": "fastino/gliner2-base-v1",
        "lora_r": 8,
        "lora_alpha": 16.0,
        "trained_on": "2024-01-15",
        "training_samples": 5000,
        "eval_f1": 0.87,
        "notes": "Initial legal domain adapter"
    },
    "legal_v2": {
        "path": "./adapters/legal/v2_r16/final",
        "base_model": "fastino/gliner2-base-v1",
        "lora_r": 16,
        "lora_alpha": 32.0,
        "trained_on": "2024-02-01",
        "training_samples": 10000,
        "eval_f1": 0.92,
        "notes": "Improved with more data and higher rank"
    }
}

5. Monitoring Adapter Performance

def evaluate_adapter(model, adapter_path, test_data):
    """Evaluate adapter performance on test data."""
    model.load_adapter(adapter_path)
    
    results = {
        "total": 0,
        "correct": 0,
        "precision_sum": 0,
        "recall_sum": 0,
    }
    
    for sample in test_data:
        pred = model.extract_entities(sample["text"], sample["entity_types"])
        
        # Compute metrics
        metrics = compute_metrics(pred['entities'], sample["true_entities"])
        results["total"] += 1
        results["precision_sum"] += metrics["precision"]
        results["recall_sum"] += metrics["recall"]
    
    avg_precision = results["precision_sum"] / results["total"]
    avg_recall = results["recall_sum"] / results["total"]
    f1 = 2 * avg_precision * avg_recall / (avg_precision + avg_recall)
    
    return {
        "precision": avg_precision,
        "recall": avg_recall,
        "f1": f1,
        "samples": results["total"]
    }

Troubleshooting

Issue 1: Adapter Not Affecting Predictions

Symptom: Predictions are the same with and without adapter.

Solution:

# Check if adapter is actually loaded
print(f"Has adapter: {model.has_adapter}")

# Check LoRA layers
from gliner2.training.lora import LoRALayer
lora_count = sum(1 for m in model.modules() if isinstance(m, LoRALayer))
print(f"LoRA layers: {lora_count}")

# Should be > 0 if adapter is loaded
assert lora_count > 0, "No LoRA layers found!"

Issue 2: Out of Memory During Training

Solution:

config = TrainingConfig(
    # Reduce batch size
    batch_size=4,  # Instead of 8
    gradient_accumulation_steps=4,  # Maintain effective batch size
    
    # Use smaller LoRA rank
    lora_r=4,  # Instead of 8
    
    # Enable mixed precision
    fp16=True,
    
    # Target only attention layers (fewer parameters)
    lora_target_modules=["encoder.query", "encoder.key", "encoder.value"],
)

Issue 3: Adapter File Not Found

Solution:

import os
from gliner2.training.lora import LoRAAdapterConfig

adapter_path = "./adapters/legal_adapter/final"

# Check if path exists
if not os.path.exists(adapter_path):
    print(f"Path does not exist: {adapter_path}")
    # List available checkpoints
    checkpoint_dir = "./adapters/legal_adapter"
    if os.path.exists(checkpoint_dir):
        checkpoints = os.listdir(checkpoint_dir)
        print(f"Available checkpoints: {checkpoints}")

# Check if it's a valid adapter
if LoRAAdapterConfig.is_adapter_path(adapter_path):
    print("Valid adapter path!")
    config = LoRAAdapterConfig.load(adapter_path)
    print(f"Adapter config: {config}")
else:
    print("Not a valid adapter path!")

Issue 4: Slow Adapter Switching

Problem: Switching between adapters takes too long.

Solution:

# Pre-load adapters in memory (if you have enough RAM)
adapters = {}
for domain, path in adapter_paths.items():
    # Load adapter weights into memory
    adapters[domain] = load_adapter_to_memory(path)

# Fast switching from memory (not implemented in base API,
# but possible with custom caching layer)

Summary

Key Takeaways

✅ LoRA adapters enable efficient multi-domain inference
✅ Training is 2-3x faster than full fine-tuning
✅ Storage savings of 65-95% compared to multiple full models
✅ Swapping adapters takes < 1 second
✅ Domain specialization improves accuracy on specific tasks

Quick Reference

# Training
config = TrainingConfig(
    use_lora=True,
    lora_r=8,
    lora_alpha=16.0,
    save_adapter_only=True,
)
trainer.train(train_data=examples)

# Loading
model = GLiNER2.from_pretrained("base-model")
model.load_adapter("./adapter/final")

# Swapping
model.load_adapter("./other_adapter/final")

# Unloading
model.unload_adapter()

# Checking
print(model.has_adapter)
print(model.adapter_config)

Next Steps

1. Train your first adapter with domain-specific data
2. Evaluate performance on test set
3. Experiment with hyperparameters (rank, alpha, target modules)
4. Deploy multiple adapters for different use cases
5. Monitor and iterate based on real-world performance

For more information:

• LoRA Paper: https://arxiv.org/abs/2106.09685
• Implementation: gliner2/training/lora.py
• Tests: tests/test_lora_adapters.py
• Verification Guide: LORA_VERIFICATION_TESTS.md