/setup-conda-llm-finetune

You are helping the user set up a conda environment for LLM fine-tuning.

Process

1. Create base environment

   conda create -n llm-finetune python=3.11 -y
   conda activate llm-finetune
   

2. Install PyTorch with ROCm

   pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.0
   

3. Install core fine-tuning libraries

   Hugging Face ecosystem:

   pip install transformers
   pip install datasets
   pip install accelerate
   pip install evaluate
   pip install peft  # Parameter-Efficient Fine-Tuning
   pip install bitsandbytes  # Quantization (may need special build for ROCm)
   

   Training frameworks:

   pip install trl  # Transformer Reinforcement Learning
   pip install deepspeed  # Distributed training (if needed)
   

4. Install quantization and optimization tools

   pip install optimum
   pip install auto-gptq  # GPTQ quantization
   pip install autoawq   # AWQ quantization
   

5. Install evaluation and monitoring tools

   pip install wandb  # Weights & Biases for experiment tracking
   pip install tensorboard
   pip install rouge-score  # Text evaluation
   pip install sacrebleu   # Translation metrics
   

6. Install data processing tools

   pip install pandas
   pip install numpy
   pip install scipy
   pip install scikit-learn
   pip install nltk
   pip install spacy
   

7. Install specialized fine-tuning tools

   pip install axolotl     # LLM fine-tuning framework
   pip install unsloth     # Fast fine-tuning (if compatible with ROCm)
   pip install qlora       # Quantized LoRA
   

8. Install Jupyter for interactive work

   conda install -c conda-forge jupyter jupyterlab ipywidgets -y
   

9. Create example fine-tuning script

   • Offer to create ~/scripts/llm-finetune-example.py with basic LoRA setup

10. Test installation

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model

print(f"PyTorch: {torch.__version__}")
print(f"GPU available: {torch.cuda.is_available()}")
print("All libraries imported successfully!")

11. Create resource estimation script

• Offer to create script to estimate VRAM needs for different model sizes

12. Suggest popular models for fine-tuning

• Llama 3.2 (3B, 8B)
• Mistral 7B
• Qwen 2.5 (7B, 14B)
• Phi-3 (3.8B)

Output

Provide a summary showing:

• Environment name and setup status
• Installed libraries grouped by purpose
• GPU detection status
• VRAM available for training
• Suggested model sizes for available hardware
• Example command to start fine-tuning
• Links to documentation/tutorials