GPT token cache

[archasek]

Hi,

I noticed that when using the standard OpenAI Chat Completions API - where I only modify the final fragment of the user prompt while keeping the full system prompt and the majority (initial part) of the user prompt identical across requests - prompt caching works effectively, with most tokens being cached.

However, when using AX in a similar pattern (only changing one input field in the signature + a bit longer signature description, typically the last one), caching is rarely utilized. Is this due to how AX constructs the prompt structure, the order of inputs, or something else? Can this be addressed?

I believe this is a significant disadvantage of AX that might be relatively easy to fix.

Context on GPT Prompt Caching:
Token caching in GPT models like 4o-mini automatically stores prefixes of repeated input prompts (starting from 1024 tokens, in 128-token blocks), reducing latency by up to 80% and input costs by 50-90% on cache hits.

What do you think?

Thanks

[dosco]

This is how the system prompt starts, changing fields is a big deal which probably causes the cache to be invalidated. From my research I came to the understanding that putting this at the top of the system prompt ensures the fields are correctly formatted and handler. Open to ideas and suggestions.

Why not use optional fields?

[ SYSTEM ]
<identity>
You will be provided with the following fields: `Input Field1`, `Input Field2`, `Input Field3`. Your task is to generate new fields: `Output Field1`, `Output Field2`.

.... description goes here ....

[archasek]

From my research I came to the understanding that putting this at the top of the system prompt ensures the fields are correctly formatted and handler.

Probably true for smaller, older models, but now I think the common approach is to have main instructions (role, context, task) at the top of the prompt.

I've tested this inversion (Instructions First -> Schema Second) and the trade-off is huge:

• Format Stability: Zero degradation (it still works perfectly for me).
• Cache Hit Rate: Massive improvement. By keeping the "Instructions" block constant at the start, we get cache hits even when switching between different signatures (as long as instructions share the same prefix).

I think defaulting to "Instructions First" is the way to go now.
Systemically, adopting a Prompt Adapter pattern (like DSPy uses) would be even better - it would solve this cache issue, handle the custom prompt styles from #469, and let users tweak ordering if they really need to support legacy models.

[dosco]

good points, i'll add a config option to change the ordering in the system prompt. as for the prompt adapter we do have something but its not exposed since we parse and validate while streaming (saves tokens etc in prod workflows) making it customizable will be a bit more complex but i agree give power users more control so i'll work on it.

[archasek]

I have created a PR just in case that could be helpful for you. Feel free to close if it's not good. #487

[dosco]

thanks it looks good, let me review and merge tomorrow

[archasek]

@dosco could you please have a look at #487? I'm waiting for it to be merged

[dosco]

sorry just been really busy

[archasek]

@dosco ?

[archasek]

@dosco merged main and resolved conflicts for the n-th time already... Please.. :) #487

[archasek]

Let me know if it's not possible to merge it, then I'll use my forked version and forget about it.

[archasek]

any chance @dosco ?
#493

[archasek]

@dosco lol it's been a month already...