Tag: inference efficiency
-
Simon Willison’s Weblog: lm.rs: run inference on Language Models locally on the CPU with Rust
Source URL: https://simonwillison.net/2024/Oct/11/lmrs/ Source: Simon Willison’s Weblog Title: lm.rs: run inference on Language Models locally on the CPU with Rust Feedly Summary: lm.rs: run inference on Language Models locally on the CPU with Rust Impressive new LLM inference implementation in Rust by Samuel Vitorino. I tried it just now on an M2 Mac with 64GB…
-
The Cloudflare Blog: Making Workers AI faster and more efficient: Performance optimization with KV cache compression and speculative decoding
Source URL: https://blog.cloudflare.com/making-workers-ai-faster Source: The Cloudflare Blog Title: Making Workers AI faster and more efficient: Performance optimization with KV cache compression and speculative decoding Feedly Summary: With a new generation of data center accelerator hardware and using optimization techniques such as KV cache compression and speculative decoding, we’ve made large language model (LLM) inference lightning-fast…