WORLDTECH NEWS Global technology intelligence.Contact
AI SINGLE SOURCE

Nvidia researchers argue huge AI context windows are a dead end

Close-up of two NVIDIA RTX 2080 graphics cards with dual fans, high-performance hardware.
Illustrative photo.Photo by Nana Dua on Pexels

What happened

Nvidia researchers argue that language models often don't need their full context windows—which are slow, expensive, and prone to errors—and propose an alternative. The team even proposes a new class of models that works with short notes instead of long context. NVIDIA is a semiconductor company based in Santa Clara.

GPT-4 launched in March 2023 with 8,192 tokens. In tests, periodic summaries and targeted keyword search usually matched full-context results at a fraction of the cost.

An Nvidia research team says that's often overkill and expensive. In many tests, periodic summaries and targeted lookups worked about as well. A language model's context window (how much text a model can consider at once) is its working memory, covering everything it holds in view when generating a reply.

Key facts

  • The team even — proposes: a new class of models that works with short notes instead of long context

Sources & evidence