Skip to content

Make cache list batchable#743

Merged
awni merged 2 commits into
mainfrom
batchable_cache_list
Jan 9, 2026
Merged

Make cache list batchable#743
awni merged 2 commits into
mainfrom
batchable_cache_list

Conversation

@awni
Copy link
Copy Markdown
Member

@awni awni commented Jan 8, 2026

  • Sets a merge property an cache types to make it easier to turn them into batch caches and check if they support batching.
  • Implements cache list merging using the above property

@awni awni requested a review from angeloskath January 8, 2026 22:32
Copy link
Copy Markdown
Member

@angeloskath angeloskath left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! I left one comment that will need fixing unless I misunderstood sth.

Comment thread mlx_lm/models/cache.py Outdated
@awni awni merged commit 7a86c12 into main Jan 9, 2026
2 checks passed
@awni awni deleted the batchable_cache_list branch January 9, 2026 18:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants