Sampling at Negative Temperature

Summary: Inspired by the definition of temperature in statistical mechanics⁢ and the possibility for it⁤ to be below zero, we try sampling LLaMA at $T=-0.001$

The notion of temperature ​comes from⁣ statistical⁢ mechanics. ‍Consider a system that has states with ⁢energies $E_1,dots,E_n$

Wait – this is just the ​Boltzmann distribution, up to a constant!^[1]

[1]	There’s no minus sign in the ⁤exponent ‌because⁣ while higher-energy states are less likely, larger logits are more likely.

In a language model, temperature ‍is used to define how creative text generations are. As a notable example, in the⁢ zero temperature limit, ⁢the model should deterministically generate​ the most likely token. In the infinite temperature limit, all ⁤tokens are ‍equally likely and the model output will be⁤ random noise. For an interactive explanation, see here.

What would ⁢it mean to⁣ have a temperature that is below zero? (this isn’t the ‌same ​as the negative ​Fahrenheit or Celsius‍ temperatures we get​ on a cold day in​ Vermont – I mean below zero on an​ absolute scale like Kelvin).

Isn’t it‌ weird that $T=infty$