Today, many companies are working to reduce their CO2 emissions. In cloud environments, this means creating lean, sustainable architectures. AWS has included a Sustainability pillar in its Well-Architected Framework since 2021, which focuses on reducing energy consumption and improving efficiency. This goal aligns well with FinOps principles. Recently, Devoteam published an article called “Frugal AI: How to Rethink Intelligence in a Resource-Hungry Era“, which inspired me to explore the practical building aspects from a DevOps perspective.
With the emergence of ChatGPT and other large language models, many companies are now developing their own chatbot services, both for customer use and internal purposes. As a DevOps consultant specialising in AWS, I’d like to share some practical tips for building a sustainable in-house chatbot. A frugal chatbot approach prioritises both cost efficiency and environmental sustainability.
Choosing the right model
Comparing the actual energy consumption of different chatbots is challenging. Here are some key points to keep in mind:
- We know that the main energy consumption is used during the model training and inference phase. Globally, more powerful IAs are trained with a larger dataset and consume more than others. For example, Llama 3.1-8B consumes less than Llama 3.1-405B (source).
- On the other hand, newer model versions are often more efficient. For example, Llama-3.3-70B consumes less energy than Llama-3.1-70B.
- However, rebound effects and the Jevons paradox show that increasing the use of LLMs cancels out efficiency gains.
In short, it’s generally better to use the smallest versions of the latest models, and to favour Small Language Models (SLMs) over Large Language Models (LLMs) when possible. The key question is: what’s your chatbot’s purpose? For instance, a FAQ bot or a product recommendation assistant will likely work well with an SLM. However, if you need to analyse and interpret philosophical texts, you’ll need a larger model. The Hugging Face catalogue is a great resource for finding the right model for your needs.
Finally, I did not find a study that compared all LLMs. But it seems that Llama generally uses fewer resources than ChatGPT. Google Gemini and Mistral AI also claim to be less consuming than ChatGPT. These considerations are essential when designing a frugal chatbot.
Reduce cost on fine-tuning
This diagram from Mistral AI shows that most of GenAI’s water consumption and greenhouse gas emissions come from model training and fine-tuning. So what are the different steps involved, and how can we minimise energy consumption?
Data collection and cleaning
Training an LLM starts with a large, diverse, high-quality text dataset. The process includes cleaning (removing duplicates to avoid overfitting), filtering (eliminating low-quality or toxic content), and normalisation (fixing encoding, punctuation, and casing before tokenising the text).
That said, you can skip this energy-intensive training phase entirely by using pre-trained models in Amazon Bedrock. If you must train your own model, use serverless services like Glue and Lambda for data collection on S3, and EMR, SageMaker Processing Jobs, or AWS Batch for cleaning. When EC2 instances are necessary, choose spot instances with m7g types for better energy efficiency.
Also consider regional differences: Ireland, Stockholm, and Paris use more renewable energy than Frankfurt or London.
Choose the Model Architecture Transformer
This phase is crucial because it determines the complexity of your future model. As mentioned earlier, start by clarifying your chatbot’s purpose.
First, you’ll need to choose the right type of transformer architecture. Generally, use decoder-only models for conversational AI and encoder-decoder models for tasks like translation or summarisation.
Next, you’ll configure key parameters like the number of transformer layers and the maximum token sequence length (which determines how much conversation history the chatbot considers when responding). These choices directly impact both model complexity and energy consumption.
For large-scale, multi-domain chatbots, consider using Mixture of Experts (MoE). This architecture has many parameters but activates only a subset for each token, allowing a large model to consume roughly the same energy as a medium-sized one. This significantly reduces energy consumption per inference and training step. However, keep in mind that MoE doesn’t reduce GPU memory requirements.
Once again, it’s better to use pre-trained models from SageMaker JumpStart or the Hugging Face marketplace rather than training from scratch. However, if you do need to train your own model, consider using smaller decoder-only architectures, such as ALiBi or FlashAttention, which can handle long contexts more efficiently. These lightweight approaches are ideal for a frugal chatbot implementation.
Pretraining (Self-Supervised)
This is the most energy-intensive phase, where the transformer uses unsupervised learning to analyse data with minimal human intervention.
To optimise compute efficiency during training, use P4d or P5 EC2 instances with NVIDIA GPUs, enable SageMaker Managed Spot Training for non-critical workloads, and remember to choose an energy-efficient AWS region.
Fine-Tuning
This is where you adapt a general-purpose model to your specific use case by training it on company-specific documents and information. Important: never include personal or sensitive data in this process.
You can skip all the previous steps by using pre-trained models from Amazon Bedrock. To minimise fine-tuning costs, consider using Retrieval-Augmented Generation (RAG). RAG is particularly effective for keeping models current as data changes—it enables continuous updates without expensive retraining cycles while keeping your model smaller and more cost-effective. Another simple approach is to enhance model performance by dynamically adding relevant context to your prompts.
If you need to retrain an existing large foundation model (FM), use SageMaker HyperPod.
You can also build a continuous learning workflow: cache user queries with API Gateway, vectorise them with Lambda, store them in DynamoDB or OpenSearch, and feed them back to your Bedrock LLM. This approach aligns perfectly with frugal chatbot principles by maximising efficiency.
Reinforcement and Benchmarking
In this final phase, you can apply Reinforcement Learning from Human Feedback (RLHF) to refine your model and conduct testing and benchmarking.
To keep evaluations efficient, automate your evaluation pipelines using serverless workflows, such as Step Functions and Lambda, and use representative sample datasets for benchmarks rather than running exhaustive tests.
Deploy a sustainable infrastructure on AWS
For a compact but efficient chatbot architecture, here’s what I recommend:
- Amazon CloudFront – Content delivery
- S3 – Static web hosting
- AWS WAF – Traffic control and security
- Application Load Balancer (ALB) – Traffic distribution
- ECS – Hosting conversational logic
- DynamoDB – Storing conversation history
- Bedrock LLM – Generating responses
So how can we maximise the sustainability impact of this setup?
Cache strategy
First, as we’ve seen, Bedrock will account for much of the environmental impact. We can significantly reduce this through caching strategies.
Use an in-service cache—such as Valkey (replacing Redis/ElastiCache) or DynamoDB with TTL—to reuse recent responses in near-real-time. Combine this with response hashing and deterministic prompts so identical requests hit the cache. This approach is especially effective for FAQ-style chatbots with frequently repeated questions.
Bedrock now offers native prompt caching for models like Claude and Amazon Nova, which allows the service to skip reprocessing input tokens. This can dramatically reduce compute requirements for repeated prompts.
You can also implement caching at the CloudFront level with longer TTLs for templated or commonly repeated responses.
Batched work where possible
For tasks like summaries or long reports, consider implementing an SQS queue with an autoscaling worker pool. This approach avoids wasting compute time on container startup delays and ensures efficient resource utilisation.
Compute impact
To optimise for sustainability, prioritise EC2 spot instances over on-demand instances whenever possible. Even better, favour serverless Fargate over EC2 unless you have specific requirements like specialised networking or GPU access. When using Fargate, opt for Graviton3 (ARM) processors, which deliver better performance per watt and are typically more cost-effective. Finally, keep container images as small as possible to reduce resource consumption.
Autoscaling
Configure autoscaling based on business metrics like request queue length and pending LLM requests, not just CPU and memory usage. Keep minimum capacity low during off-peak hours to avoid idle resources, while ensuring the system can scale up quickly to handle traffic spikes.
Network
- Use VPC endpoints (not NAT) for S3 and DynamoDB. Compress payloads and prefer binary formats.
- WAF: Implement rate limits, bot control, and managed rules to block abusive traffic and prevent wasted LLM calls. Add per-user quotas for fair usage.
Efficient storage and retention
- DynamoDB: Use on-demand billing and enable TTL (Time to Live) to automatically clean up old conversation records.
- S3: Enable Intelligent-Tiering and set up lifecycle policies to archive or delete unused files automatically.
- Logging: Set retention periods based on actual observability and compliance needs—avoid keeping logs longer than necessary.
Bedrock
Token optimization: Minimize input token size by pre-filtering questions and compressing context where possible.
Carbon footprint monitoring: Use the AWS Customer Carbon Footprint Tool to measure carbon intensity across regions and select the least emission-intensive options when latency and compliance permit. AWS measures efficiency using Power Usage Effectiveness (PUE). As mentioned earlier, prioritize regions like Stockholm or Paris over Frankfurt or London for their lower carbon footprint (8).
Monitoring and cost tracking: Implement comprehensive monitoring that tracks request counts, LLM invocations, token usage, and cache hit rates. Consider external tools like LiteLLM proxy for spend tracking, budget management, and per-user limits, or Langfuse for monitoring, debugging, and analyzing LLM applications. Tag all resources to enable detailed Cost & Usage Reports and budget tracking.
Conclusion
Finally, remember the first principle of the 5Rs (Refuse, Reduce, Reuse, Recycle, Rot): Refuse. Only build a chatbot if you genuinely need one. If you do decide to move forward, I hope this article has provided valuable guidance on minimising your chatbot’s environmental impact. By following these best practices, you can build a truly frugal chatbot that serves your business needs while respecting both your budget and the environment.