Fastly has released the company’s first AI solution, Fastly AI Accelerator, designed to create a better experience for developers by helping improve performance and reduce costs.
“AI technologies generally and large language models specifically are aggressively reshaping the technology industry and the way millions worldwide – developers included – work every day,” said Stephen O’Grady, Principal Analyst with RedMonk.
“There’s a lot of focus on the largest models.
“However, developers and enterprises alike are turning in large numbers to medium and smaller models.
“Whether it’s to lower costs, to shorten training cycles or to run on more limited hardware profiles, they’re an increasingly important option.”
Fastly AI Accelerator is designed to reduce API calls and costs with intelligent, semantic caching.
Built on Fastly’s Edge Cloud Platform and leveraging industry leading caching technology, AI Accelerator uses a specialised API gateway to dramatically improve performance for apps using popular LLMs, beginning with ChatGPT and expanding support to include additional models.
According to the company, popular AI applications can process hundreds of thousands of API calls or questions daily.
Many of the questions users ask are likely very similar and may have been asked before.
Without semantic caching, each call to the LLM requires going back to the provider for the information, potentially increasing costs and latency.
However, Fastly AI Accelerator’s semantic caching provides a cached response for repeated queries directly from Fastly’s high performance edge platform, instead of going back to the AI provider, helping to deliver a better experience by improving performance while reducing costs.
“At Fastly, we’re always listening to developers, to understand both what they’re excited about and what their biggest pain points are,” said Anil Dash, Vice President of Developer Experience with Fastly.
“Fastly AI Accelerator gives developers exactly what they want, by making the experience of their favorite LLMs a lot faster and more efficient, so they can focus on what makes their app or site unique and what keeps their users happy.”
When using Fastly AI Accelerator, developers only need to update their app to use a new API endpoint, which typically only requires changing a single line of code.
Fastly AI Accelerator will then transparently implement semantic caching for OpenAI compatible APIs.
This approach goes beyond traditional caching as Fastly AI Accelerator is able to understand the context of the requests and queries and will send a similar response if two or more requests are alike.
