Nvidia researchers argue huge AI context windows are a dead end
What happened
Nvidia researchers argue that language models often don't need their full context windows—which are slow, expensive, and prone to errors—and propose an alternative. The team even proposes a new class of models that works with short notes instead of long context. NVIDIA is a semiconductor company based in Santa Clara.
GPT-4 launched in March 2023 with 8,192 tokens. In tests, periodic summaries and targeted keyword search usually matched full-context results at a fraction of the cost.
An Nvidia research team says that's often overkill and expensive. In many tests, periodic summaries and targeted lookups worked about as well. A language model's context window (how much text a model can consider at once) is its working memory, covering everything it holds in view when generating a reply.