> ## Documentation Index
> Fetch the complete documentation index at: https://hyper.julian.sc/llms.txt
> Use this file to discover all available pages before exploring further.

# LoRA Training

> Complete guide to training LoRA adapters with HyperGen

## What is LoRA?

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning technique that trains small adapter layers instead of the entire model. This makes training:

* **Faster**: Train in minutes instead of hours
* **Cheaper**: Use 80% less VRAM
* **Portable**: LoRA files are 50-200MB vs 5-10GB for full models
* **Flexible**: Easily switch between different LoRAs on the same base model

## Basic Training

### Simple Example

Train a LoRA with default settings:

```python theme={null}
from hypergen import model, dataset

# Load model
m = model.load("stabilityai/stable-diffusion-xl-base-1.0")
m.to("cuda")

# Load dataset
ds = dataset.load("./my_images")

# Train LoRA
lora = m.train_lora(ds, steps=1000)
```

That's it! HyperGen handles everything automatically.

## Training Parameters

### Core Parameters

**dataset** `Dataset` *required*

Dataset to train on (from `dataset.load()`)

**steps** `int` (default: 1000)

Number of training steps to run. More steps = better learning, but longer training.

* **Quick test**: 100-500 steps
* **Normal training**: 1000-2000 steps
* **High quality**: 2000-5000 steps

**learning\_rate** `float | 'auto'` (default: 0.0001)

Learning rate for optimizer. Controls how quickly the model learns.

* **Default**: `1e-4` (0.0001) works well for most cases
* **Higher** (1e-3): Faster learning, risk of instability
* **Lower** (1e-5): Slower but more stable
* **Auto**: Let HyperGen choose (coming in Phase 2)

### LoRA Configuration

**rank** `int` (default: 16)

LoRA rank - controls the number of parameters in the adapter.

* **Lower** (4-8): Fewer parameters, faster training, smaller files
* **Medium** (16-32): Good balance (recommended)
* **Higher** (64-128): More capacity, larger files

Higher rank = more capacity to learn, but diminishing returns after 32-64.

**alpha** `int` (default: 32)

LoRA alpha scaling parameter. Usually set to 2x the rank.

* Affects the strength of the LoRA adapter
* Common formula: `alpha = 2 * rank`
* Higher alpha = stronger LoRA effect

### Training Configuration

**batch\_size** `int | 'auto'` (default: 1)

Number of images to process at once.

* **1**: Safest, works on 8GB VRAM
* **2-4**: Better training, needs 12-16GB VRAM
* **'auto'**: Let HyperGen optimize (coming in Phase 2)

Larger batch size can improve training quality but uses more VRAM.

**gradient\_accumulation\_steps** `int` (default: 1)

Accumulate gradients over multiple steps before updating.

Effective batch size = `batch_size * gradient_accumulation_steps`

Use this to simulate larger batches without using more VRAM:

* `batch_size=1, gradient_accumulation_steps=4` H `batch_size=4`

**save\_steps** `int | None`

Save a checkpoint every N steps.

* **None**: Only save at the end
* **500**: Save every 500 steps
* Useful for long training runs to avoid losing progress

**output\_dir** `str | None`

Directory to save checkpoints and final LoRA weights.

* **None**: Don't save to disk (weights returned in memory only)
* **"./checkpoints"**: Save to this directory

## Advanced Examples

### Custom LoRA Configuration

```python theme={null}
lora = m.train_lora(
    ds,
    steps=2000,
    learning_rate=5e-5,
    rank=32,                        # Higher capacity
    alpha=64,                       # 2x rank
    batch_size=2,                   # Process 2 images at once
    gradient_accumulation_steps=4,  # Effective batch size = 8
    save_steps=500,                 # Save every 500 steps
    output_dir="./my_lora_checkpoints"
)
```

### Memory-Constrained Training

For GPUs with limited VRAM (8-12GB):

```python theme={null}
lora = m.train_lora(
    ds,
    steps=1000,
    rank=8,              # Lower rank uses less memory
    alpha=16,
    batch_size=1,        # Single image at a time
    gradient_accumulation_steps=8  # Simulate batch_size=8
)
```

### High-Quality Training

For best results with plenty of VRAM (24GB+):

```python theme={null}
lora = m.train_lora(
    ds,
    steps=5000,
    learning_rate=5e-5,
    rank=64,             # High capacity
    alpha=128,
    batch_size=4,
    save_steps=1000,
    output_dir="./checkpoints"
)
```

### Using the LoRATrainer Directly

For more control, use the `LoRATrainer` class:

```python theme={null}
from hypergen import model, dataset
from hypergen.training import LoRATrainer

m = model.load("stabilityai/stable-diffusion-xl-base-1.0")
m.to("cuda")
ds = dataset.load("./images")

# Create trainer with custom configuration
trainer = LoRATrainer(
    m,
    rank=16,
    alpha=32,
    target_modules=["to_q", "to_k", "to_v", "to_out.0"],
    dropout=0.1,
)

# Train
lora = trainer.train(
    ds,
    steps=1000,
    learning_rate=1e-4,
)
```

## Different Model Types

HyperGen's API is the same for all models:

### SDXL Training

```python theme={null}
m = model.load("stabilityai/stable-diffusion-xl-base-1.0")
m.to("cuda")
lora = m.train_lora(ds, steps=1000)
```

### FLUX.1 Training

```python theme={null}
m = model.load("black-forest-labs/FLUX.1-dev", torch_dtype="bfloat16")
m.to("cuda")
lora = m.train_lora(ds, steps=1500, rank=32)
```

### Stable Diffusion 1.5 Training

```python theme={null}
m = model.load("runwayml/stable-diffusion-v1-5")
m.to("cuda")
lora = m.train_lora(ds, steps=800, rank=8)  # Smaller model, lower rank
```

### Stable Diffusion 3 Training

```python theme={null}
m = model.load("stabilityai/stable-diffusion-3-medium-diffusers")
m.to("cuda")
lora = m.train_lora(ds, steps=1000)
```

## Saving and Loading LoRAs

### Saving During Training

```python theme={null}
lora = m.train_lora(
    ds,
    steps=2000,
    save_steps=500,              # Save every 500 steps
    output_dir="./checkpoints"   # Save to this directory
)
```

This creates:

```
checkpoints/
  checkpoint-500/
    adapter_config.json
    adapter_model.safetensors
  checkpoint-1000/
    adapter_config.json
    adapter_model.safetensors
  checkpoint-1500/
    adapter_config.json
    adapter_model.safetensors
  checkpoint-2000/
    adapter_config.json
    adapter_model.safetensors
```

### Loading LoRAs for Inference

> [!NOTE]
>
> LoRA loading for inference is coming in Phase 2. For now, LoRAs are saved but not yet loaded for generation.

Future API (coming soon):

```python theme={null}
# Load base model
m = model.load("stabilityai/stable-diffusion-xl-base-1.0")
m.to("cuda")

# Load LoRA
m.load_lora("./checkpoints/checkpoint-2000")

# Generate with LoRA
image = m.generate("A photo in my style")
```

## Training Recommendations

### Hyperparameter Guidelines

> [tab] Style Transfer
>
> **Goal:** Learn an artistic style or aesthetic
>
> ```python theme={null}
> lora = m.train_lora(
>     ds,
>     steps=1500,
>     learning_rate=1e-4,
>     rank=16,
>     alpha=32,
> )
> ```
>
> * Dataset: 50-200 images
> * Captions: Describe content, not style

> [tab] Subject/Character
>
> **Goal:** Learn a specific person, character, or object
>
> ```python theme={null}
> lora = m.train_lora(
>     ds,
>     steps=1000,
>     learning_rate=5e-5,
>     rank=32,
>     alpha=64,
> )
> ```
>
> * Dataset: 20-100 images
> * Captions: Describe poses, angles, clothing

> [tab] Concept
>
> **Goal:** Learn a new concept or composition style
>
> ```python theme={null}
> lora = m.train_lora(
>     ds,
>     steps=2000,
>     learning_rate=1e-4,
>     rank=24,
>     alpha=48,
> )
> ```
>
> * Dataset: 30-150 images
> * Captions: Focus on composition and elements

### VRAM Requirements

> [card] 8GB VRAM
>
> **Models:** SD 1.5, SDXL (with optimizations)
>
> ```python theme={null}
> batch_size=1
> rank=8-16
> ```

> [card] 12GB VRAM
>
> **Models:** SDXL, SD3
>
> ```python theme={null}
> batch_size=1-2
> rank=16-32
> ```

> [card] 16GB VRAM
>
> **Models:** SDXL, SD3, FLUX.1
>
> ```python theme={null}
> batch_size=2-3
> rank=32-64
> ```

> [card] 24GB+ VRAM
>
> **Models:** All models including FLUX.1
>
> ```python theme={null}
> batch_size=4+
> rank=64-128
> ```

## Troubleshooting

### Out of Memory Errors

If you encounter CUDA out of memory errors:

1. **Reduce batch size:**
   ```python theme={null}
   lora = m.train_lora(ds, batch_size=1)
   ```

2. **Lower LoRA rank:**
   ```python theme={null}
   lora = m.train_lora(ds, rank=8)
   ```

3. **Use gradient accumulation:**
   ```python theme={null}
   lora = m.train_lora(ds, batch_size=1, gradient_accumulation_steps=4)
   ```

4. **Use lower precision:**
   ```python theme={null}
   m = model.load("model_id", torch_dtype="float16")  # or "bfloat16"
   ```

### Training Too Slow

If training is taking too long:

1. **Reduce steps:**
   ```python theme={null}
   lora = m.train_lora(ds, steps=500)  # Quick test
   ```

2. **Increase batch size** (if you have VRAM):
   ```python theme={null}
   lora = m.train_lora(ds, batch_size=2)
   ```

3. **Enable optimizations** (coming in Phase 2)

### Poor Quality Results

If the trained LoRA doesn't work well:

1. **Increase training steps:**
   ```python theme={null}
   lora = m.train_lora(ds, steps=2000)
   ```

2. **Increase LoRA rank:**
   ```python theme={null}
   lora = m.train_lora(ds, rank=32, alpha=64)
   ```

3. **Improve dataset:**
   * Add more images
   * Add captions
   * Increase image quality
   * Add more variety

4. **Adjust learning rate:**
   ```python theme={null}
   lora = m.train_lora(ds, learning_rate=5e-5)  # Lower
   # or
   lora = m.train_lora(ds, learning_rate=2e-4)  # Higher
   ```

## Current Limitations

> [!WARNING]
>
> HyperGen is in **pre-alpha**. The training loop is not fully implemented yet.

**What Works:**

* Model loading and configuration
* Dataset loading
* LoRA setup with PEFT
* Checkpoint saving structure

**In Development:**

* Complete training loop with loss calculation
* Noise scheduling
* Validation and metrics

**Coming in Phase 2:**

* Gradient checkpointing
* Mixed precision training
* Flash Attention
* Auto-configuration
* LoRA loading for inference

## Next Steps

> [card] [Datasets](https://hyper.julian.sc/training/datasets.md)
>
> Learn how to prepare training data

> [card] [Training Overview](https://hyper.julian.sc/training/overview.md)
>
> Understand the training architecture

> [card] [Serving](https://hyper.julian.sc/serving/quickstart.md)
>
> Deploy trained models with the API

> [card] [Examples](https://github.com/ntegrals/hypergen/tree/main/examples)
>
> View complete training examples


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.