When Should You Start a New AI Chat?

The chat has been running for hours, and it has been good. The proposal outline is done, three options are out, one is decided. Then you ask about a small detail, and the AI suggests the exact option the two of you threw out two hours ago. The answer also takes noticeably longer than it did at the start.

A client recently asked us exactly this: How do I know when to start a new chat? And how do I carry over what matters without writing a handover document every time? It is a better question than it sounds, because the usual answer is a handover document, and that only solves half the problem.

In this article you will learn why long chats get worse, how to recognise the right moment, what happens during automatic compaction and which approach works better than a handover document.

---

Table of contents

1. Why do long AI chats get worse?
2. How can you tell a chat is too long?
3. What happens when the AI compacts a chat automatically?
4. Why are handover documents the weaker fix?
5. What works better than a handover document?
6. How does this work in Hermes Agent?
7. Frequently asked questions

---

Why do long AI chats get worse?

In short: A language model rereads the entire chat for every answer. The longer the chat, the more it has to read, the longer it takes, the more it costs and the worse it gets at finding what matters.

An AI chat has no memory in the human sense. With every new message, the complete history is sent to the model again: every question, every answer, every pasted document. The model processes this text as its context. The upper limit is called the context window and is measured in tokens, which are fragments of words.

That leads to three effects you can feel directly:

• Slower: More text in the context means more computation before the first line of the answer appears.
• More expensive: Most providers bill per processed token. A chat with 100,000 tokens of history pays for that history again with every follow-up question. Prompt caching lowers the price of repeated parts, but the amount of text stays the same.
• Less precise: This is the most important point, and it is well researched.

In 2023, researchers led by Nelson F. Liu at Stanford University showed in Lost in the Middle that language models use information at the beginning and the end of a long context best. When the key information sits in the middle, accuracy drops significantly, even for models built explicitly for long contexts.

In its 2025 report Context Rot, the database company Chroma tested 18 current models, including GPT-4.1, Claude 4, Gemini 2.5 and Qwen3. The result: performance becomes increasingly unreliable as input length grows, even on simple tasks. One test with real chat histories is especially telling. When the models received only the relevant parts of a conversation (around 300 tokens), they answered significantly better than with the full history of around 113,000 tokens on average, even though the answer was contained in both.

In practice this means a large context window is a ceiling, not a quality guarantee. A model can accept 200,000 tokens and still perform worse at 80,000 than at 8,000.

How can you tell a chat is too long?

In short: When the AI forgets decisions, repeats mistakes or gets noticeably slower, the chat is too long. It is better to switch before that, every time a stage of the work is done.

Typical warning signs:

1. The AI suggests something you already rejected.
2. Agreements from the start (tone, format, names) are no longer followed.
3. A mistake you corrected comes back.
4. Answers get more generic and refer less to concrete details from the chat.
5. Response times go up noticeably.

The problem with these signs is that they only show up once quality has already suffered. You usually notice because of a mistake, not before it. Fixed triggers for a fresh chat are more useful:

• New topic: One chat per task. Asking the proposal chat to quickly draft an email to your accountant mixes contexts.
• Stage complete: Once a decision is made or a draft is finished, that is a good cut.
• Large document import: Long PDFs or spreadsheets fill the context fast. Afterwards, start fresh with the result.

What almost no chat interface does is tell you how full the current chat is. That gap is exactly what prompted our client’s question.

What happens when the AI compacts a chat automatically?

In short: Many AI tools summarise long chats automatically before the context window is full. That keeps the chat going, but every summary loses details.

Some tools compact long chats automatically once the context reaches a certain size. Older parts of the conversation are replaced with a summary. In Hermes Agent, the open source agent we work with, compaction kicks in by default somewhere between half and three quarters of the context window, depending on the model.

That makes sense, because otherwise the chat would eventually break. But a summary is always a selection. Whatever the model considers unimportant while summarising is gone: the exact figure from the third message, the reason option B was dropped, the name of the contact person. You do not notice what is missing, because you cannot see what was cut.

Automatic compaction is a safety net, not a way of working. If you regularly run into it, you would have been better off ending the chat yourself.

Why are handover documents the weaker fix?

In short: A handover document takes time, is itself a lossy summary and fills the new chat with text right from the start. It moves the problem instead of solving it.

The usual advice for switching chats is: ask for a summary at the end, paste it into a new chat, carry on. It works, but it has four weaknesses.

1. The timing is bad. The document is written exactly when the chat is already too long. So the model summarises in the state in which it is worst at finding details.
2. It is another selection. A handover document is nothing but manual compaction. Whatever is not in it does not exist in the new chat.
3. It grows and goes stale. Handover documents get longer with every switch, because nobody wants to delete anything. After the third handover, nobody reads them in full, including you.
4. The new chat starts half full. Paste three pages of handover notes and you begin the new chat with exactly the baggage you wanted to drop.

Then there is the effort. If every switch means requesting, checking and pasting a document, you switch less often than you should. And you are back where you started.

What works better than a handover document?

In short: A short pointer to the old chat instead of a long summary. In the new chat, the AI looks up whatever it needs right now.

The core idea: the old chat is not gone. If your AI system can search earlier conversations, the new chat does not need a copy of the old one. It only needs a pointer to where to look, plus the current status in two or three sentences.

That has three advantages:

• The new chat stays lean. It starts with a few lines instead of pages.
• Nothing gets lost. The full old history stays intact and searchable. The model retrieves the part that fits the current question instead of working from a summary someone selected earlier.
• It matches the research. Chroma’s results show that models answer better with focused, relevant context than with the full history. Targeted lookup delivers exactly that focused context.

Which leaves the question of timing. The best pointer is useless if you notice too late that it is time to switch.

How does this work in Hermes Agent?

In short: With the free chat-length plugin, Hermes tells you itself when a chat is getting too long and hands you a ready-made opener for the new chat. Hermes pulls the rest from the old conversation via session search.

Our client’s question turned into a small plugin for Hermes Agent: chat-length. It does exactly the two things this article has been missing: it tells you when it is time, and it gives you the pointer.

How it works:

1. After every answer, the plugin records how many tokens the AI provider actually processed. It does not estimate, it uses the real numbers.
2. It measures against the point where Hermes would compact the chat. So the notice comes in time, before details are lost.
3. At 70 percent (configurable), a short notice appears below the normal answer, at 90 percent a more urgent version. Each level appears only once per chat.
4. The notice contains a ready-made opener: title and ID of the old chat, two or three status bullets, the open items.
5. You paste that opener into a new chat. Hermes finds the old history itself with its built-in session search (session_search).

This is what such an opener looks like:

Continue our chat "Summer party in Cologne" (session 20261015_141559_2a617f).
First retrieve the history yourself with session_search.
Status:
* Beach theme chosen
* Decoration, tech and catering budget roughly planned
Open: venue, bad-weather plan.

What you should know to put it in context:

• chat-length is free, open source under the MIT license and public on GitHub.
• It does not change the system prompt, so prompt caching stays intact. It makes no network connections and stores no chat content.
• It has been live-tested with the Hermes CLI on v0.21.6 and v0.22.2. It runs server-side, so it is built for WebUI, Desktop and Telegram too, but we have not live-tested it there yet.
• It is listed in the official Hermes plugin catalog. Nous Research reviewed the entry and accepted it on 23 October 2026. It installs with one command:

hermes plugins install chat-length --enable
hermes gateway restart

Pro tip: Add a rule to your Hermes SOUL.md telling it to check session_search first whenever you say things like “as discussed” or “in our last chat”. Then switching works even without an opener. The ready-made wording is in the plugin’s README.

We collect all our Hermes plugins and skills on the Hermes plugins page.

Frequently asked questions

How long can an AI chat be?

There is no fixed limit, it depends on the model and the task. Research shows, however, that quality drops well before the end of the context window. One chat per task and a switch after every completed stage is a good rule of thumb.

Is a bigger context window the solution?

Only partly. A bigger window moves the hard limit but does not solve the problem that models get worse at finding what is relevant in long contexts. Focused context beats a lot of context.

How do I carry important information into a new chat?

Ideally with a short opener: what it is about, what has been decided, what is still open. If your AI system can search earlier chats, add a reference to the old chat instead of summarising it.

What is the difference between compaction and a new chat?

With compaction, the AI replaces older parts of the chat with a summary, and details get lost. A new chat starts empty, while the old one stays complete and searchable.

Does this also work with ChatGPT or Claude?

The rules about timing apply to every AI chat. The chat-length plugin is only available for Hermes Agent, because it relies on its interfaces and session search.

Conclusion

Long AI chats do not get smarter. They get slower, more expensive and less precise. This is not a flaw of individual models, it is well documented. The right response is to switch in time, and for the switch a short pointer to the old chat is enough. A long handover document, by contrast, is one more lossy summary, written at the worst possible moment.

If you use Hermes Agent, chat-length handles both for you: the notice at the right time and the ready-made opener. If you are looking for an AI system that comes with this kind of thing from day one and runs on a server you own, take a look at B.R.A.I.N..

About MadeByBrain: We build AI automation for small and mid-sized companies: custom AI agents, n8n workflows and GEO strategies, live in production. From the first idea to productive automation.

Book a free strategy call