Context Management
How Context Works in Caret Canvas
The AI functionality on the canvas is very sensitive to how different nodes are connected together. Underneath the hood, each node is rolled up into a conversation that can then be sent to the LLM provider (LLM stands for large language model). The conversation context used for a given conversation is whatever the context is at the time the sparkle button is pressed. You can change and re-run conversations by disconnecting and reconnecting different nodes using the arrows.
Context Window:
This is the limit of how many "tokens" (chunks of words) a given model can process at one time. Smaller models running locally will have smaller context windows, whereas larger models such as those from OpenAI can process about 128,000 tokens. The Gemini models from Google can process millions of tokens at once. Different models provide different functionalities here.
Understanding What Context is Being Used
The conversation path that will be used is highlighted in green. Notes, text, and PDFs included along this path will be pulled in, up to the limit of the context window.
Modifying Context
You can change the context in the conversation by adding/removing/changing nodes, arrows, and additional references.
- Delete Node: Delete a node to remove its content from the conversation.
- Create Node: Create a new node with the desired content. Connect it into the conversation with arrows.
- Change Context: Modify the content of a node to update the information used in the conversation. You can change
assistantorusercontent and the next time you run the conversation it will use the newly modified content. - Add / Remove Arrows: Add or remove arrows to change the context path. The conversation will only be connected nodes. So if you remove arrows you are shortening the conversation. On the flip side, you can link up a ton of arrows to create a really long conversation.