Hyperparameter Search

This guide explains the automation of the "trial and error" process for finding optimal model settings. Instead of manual guessing, a range of possibilities (a "search space") is defined, allowing Protean AI to intelligently test combinations to find the most effective configuration.

In the process, the best settings for a specific dataset are identified. Trials that do not yield the desired results are pruned (stopped early), saving both time and resources.

What is a "Search Space"?

A search space functions as a menu of options for the search algorithm to choose from. Rather than providing a single fixed value, limits and boundaries are established for the system to explore.

• Specific: A single static value used for every trial.
• Collection: A single option is selected from a specific list of values: [Option A, Option B, Option C].
• Step: A range is searched between a defined min and max, sampled at fixed increments (a step size).
• Log: A range is searched between min and max on a logarithmic scale, with a focus on smaller magnitudes (e.g., 0.0001) rather than larger ones.

Configurable Parameters

The following section details how to configure the search for specific settings.

Learning Rate

• Description: The speed at which the model learns. It controls how large a step the optimizer takes on each update, so it directly shapes how quickly the model adapts to your dataset. It matters because a rate that is too high can destabilize training while a rate that is too low wastes time, which is why it is usually the most impactful parameter to search.
• Best Search Strategy: Log.
  • Reasoning: The difference between 0.0001 and 0.001 is significant (10x), whereas the difference between 0.1 and 0.101 is negligible. A "log" search treats 10x magnitudes equally.
• Recommended Range: 1e-6 (very slow) to 2e-4 (standard).

Weight Decay

• Description: A stabilizer used to prevent overfitting. It works by gently penalizing large weights during training, which keeps the model from memorizing the training data too closely. It matters because the right amount improves how well the resulting model generalizes, while the wrong amount either lets the model overfit or holds it back from learning.
• Best Search Strategy: Collection or Log.
  • Reasoning: Typically, the desired value is either "None" (0.0), "Standard" (0.01), or "High" (0.1).
• Recommended Options: [0.0, 0.01, 0.1]

Warmup Step Factor

• Description: The fraction of training spent ramping the learning rate up from zero, expressed as a value between 0 and 1 rather than a step count. During this warmup phase the learning rate climbs gradually instead of starting at full speed, which eases the model into training. It matters because a short ramp can help avoid early instability before the run settles into its full learning rate.
• Best Search Strategy: Step.
  • Reasoning: Testing every fractional value is unnecessary. Sampling the range at fixed increments is sufficient.
• Recommended Range: 0.0 to 0.2 (Step: 0.05).

LoRA Rank

• Description: The "brain capacity" of the adapter, configurable across the range 4 to 512 (step 2). It sets how many trainable parameters the adapter adds, so a higher rank gives the adapter more room to capture patterns from your dataset. It matters because too low a rank can limit what the adapter learns while too high a rank increases cost and risks overfitting, which is why values are typically chosen from powers of 2.
• Best Search Strategy: Collection.
  • Reasoning: Computational efficiency is optimized for powers of 2 (8, 16, 32, 64). Testing arbitrary values like "Rank 13" is generally inefficient.
• Recommended Options: [8, 16, 32, 64]

LoRA Alpha

• Description: The "loudness" or strength of the adapter, configurable across the range 4 to 1024. It scales how strongly the adapter's learned changes are applied on top of the base model, so it effectively amplifies or dampens the adapter's influence. It matters because it is usually tuned in relation to the rank (a common rule is Alpha = 2 * Rank), keeping the adapter's effect balanced against its capacity.
• Best Search Strategy: Collection (Dependent on Rank).
  • Note: A common rule is Alpha = 2 * Rank. However, if independent search is required, a list of standard values can be used.
• Recommended Options: [16, 32, 64, 128]

LoRA Dropout

• Description: The random disabling of neurons to improve reliability, configurable across the range 0 to 1. During training it randomly drops a fraction of the adapter's neurons on each pass, which forces the adapter to learn more robust patterns rather than relying on any single path. It matters because a modest amount (commonly searched from 0% to 10%) reduces overfitting, while too much dropout can slow or weaken learning.
• Best Search Strategy: Step (linear range) or Collection.
  • Reasoning: This represents a simple percentage, commonly searched from 0% to 10%.
• Recommended Range: 0.0 to 0.1 (Step: 0.05).

Quick "Copy-Paste" Search Space

The following values serve as a reliable starting point for chat objective on instruction tuned models.

Parameter	Mode	Suggested Range/Values
Learning Rate	Log	1e-5 ... 2e-4 (log)
Weight Decay	Collection	0.0, 0.01, 0.1
Warmup Step Factor	Step	0.0 ... 0.2 (step=0.05)
Lora R	Collection	8, 16, 32, 64
Lora Alpha	Collection	16, 32, 64
Lora Dropout	Step	0.0 ... 0.1 (step=0.05)