{"id":771003,"date":"2025-10-20T14:44:00","date_gmt":"2025-10-20T12:44:00","guid":{"rendered":"https:\/\/www.devoteam.com\/expert-view\/frugal-chatbot-on-aws\/"},"modified":"2025-11-05T15:44:41","modified_gmt":"2025-11-05T14:44:41","slug":"frugal-chatbot-on-aws","status":"publish","type":"expert-view","link":"https:\/\/devoteam.info\/uk\/expert-view\/frugal-chatbot-on-aws\/","title":{"rendered":"How to Train and Host a Frugal Chatbot on AWS"},"content":{"rendered":"\n

Today, many companies are working to reduce their CO2 emissions. In cloud environments, this means creating lean, sustainable architectures. AWS has included a Sustainability pillar in its Well-Architected Framework<\/a> since 2021, which focuses on reducing energy consumption and improving efficiency. This goal aligns well with FinOps<\/a> principles. Recently, Devoteam published an article called “Frugal AI: How to Rethink Intelligence in a Resource-Hungry Era<\/a>“, which inspired me to explore the practical building aspects from a DevOps perspective.<\/p>\n\n

With the emergence of ChatGPT and other large language models, many companies are now developing their own chatbot services, both for customer use and internal purposes. As a DevOps consultant specialising in AWS, I’d like to share some practical tips for building a sustainable in-house chatbot. A frugal chatbot<\/strong> approach prioritises both cost efficiency and environmental sustainability.<\/p>\n\n

Choosing the right model<\/h2>\n\n

Comparing the actual energy consumption of different chatbots is challenging. Here are some key points to keep in mind:<\/p>\n\n

    \n
  1. We know that the main energy consumption is used during the model training and inference phase<\/a>. Globally, more powerful IAs are trained with a larger dataset and consume more than others. For example, Llama 3.1-8B consumes less than Llama 3.1-405B (source<\/a>).<\/li>\n\n\n\n
  2. On the other hand, newer model versions are often more efficient. For example, Llama-3.3-70B consumes less energy than Llama-3.1-70B.<\/li>\n\n\n\n
  3. However, rebound effects and the Jevons paradox show that increasing the use of LLMs cancels out efficiency gains.<\/li>\n<\/ol>\n\n

    In short, it’s generally better to use the smallest versions of the latest models, and to favour Small Language Models (SLMs) over Large Language Models (LLMs) when possible. The key question is: what’s your chatbot’s purpose? For instance, a FAQ bot or a product recommendation assistant will likely work well with an SLM<\/a>. However, if you need to analyse and interpret philosophical texts, you’ll need a larger model. The Hugging Face<\/a> catalogue is a great resource for finding the right model for your needs.<\/p>\n\n

    Finally, I did not find a study that compared all LLMs. But it seems that Llama generally uses fewer resources than ChatGPT. Google Gemini and Mistral AI also claim to be less consuming than ChatGPT. These considerations are essential when designing a frugal chatbot.<\/p>\n\n

    Reduce cost on fine-tuning<\/h2>\n\n

    This diagram<\/a> from Mistral AI shows that most of GenAI’s water consumption and greenhouse gas emissions come from model training and fine-tuning. So what are the different steps involved, and how can we minimise energy consumption?<\/p>\n\n

    Data collection and cleaning<\/h3>\n\n

    Training an LLM starts with a large, diverse, high-quality text dataset. The process includes cleaning (removing duplicates to avoid overfitting), filtering (eliminating low-quality or toxic content), and normalisation (fixing encoding, punctuation, and casing before tokenising the text).<\/p>\n\n

    That said, you can skip this energy-intensive training phase entirely by using pre-trained models in Amazon Bedrock. If you must train your own model, use serverless services like Glue and Lambda for data collection on S3, and EMR, SageMaker Processing Jobs, or AWS Batch for cleaning. When EC2 instances are necessary, choose spot instances with m7g types<\/a> for better energy efficiency.<\/p>\n\n

    Also consider regional differences: Ireland, Stockholm, and Paris use more renewable energy <\/a>than Frankfurt or London.<\/p>\n\n

    Choose the Model Architecture Transformer<\/h3>\n\n

    This phase is crucial because it determines the complexity of your future model. As mentioned earlier, start by clarifying your chatbot’s purpose.<\/p>\n\n

    First, you’ll need to choose the right type of transformer architecture. Generally, use decoder-only models for conversational AI and encoder-decoder models for tasks like translation or summarisation.<\/p>\n\n

    Next, you’ll configure key parameters like the number of transformer layers and the maximum token sequence length (which determines how much conversation history the chatbot considers when responding). These choices directly impact both model complexity and energy consumption.<\/p>\n\n

    For large-scale, multi-domain chatbots, consider using Mixture of Experts (MoE). This architecture has many parameters but activates only a subset for each token, allowing a large model to consume roughly the same energy as a medium-sized one. This significantly reduces energy consumption per inference and training step. However, keep in mind that MoE doesn’t reduce GPU memory requirements<\/a>.<\/p>\n\n

    Once again, it’s better to use pre-trained models from SageMaker JumpStart or the Hugging Face marketplace<\/a> rather than training from scratch. However, if you do need to train your own model, consider using smaller decoder-only architectures, such as ALiBi or FlashAttention, which can handle long contexts more efficiently. These lightweight approaches are ideal for a frugal chatbot implementation.<\/p>\n\n

    Pretraining (Self-Supervised)<\/h3>\n\n

    This is the most energy-intensive phase, where the transformer uses unsupervised learning to analyse data with minimal human intervention.<\/p>\n\n

    To optimise compute efficiency during training, use P4d or P5 EC2 instances <\/a>with NVIDIA GPUs, enable SageMaker Managed Spot Training for non-critical workloads, and remember to choose an energy-efficient AWS region.<\/p>\n\n

    Fine-Tuning<\/h3>\n\n

    This is where you adapt a general-purpose model to your specific use case by training it on company-specific documents and information. Important: never include personal or sensitive data in this process.<\/p>\n\n

    You can skip all the previous steps by using pre-trained models from Amazon Bedrock. To minimise fine-tuning costs, consider using Retrieval-Augmented Generation (RAG). RAG is particularly effective for keeping models current as data changes\u2014it enables continuous updates without expensive retraining cycles while keeping your model smaller and more cost-effective. Another simple approach is to enhance model performance by dynamically adding relevant context to your prompts.<\/p>\n\n

    If you need to retrain an existing large foundation model (FM), use SageMaker HyperPod<\/a>.<\/p>\n\n

    You can also build a continuous learning workflow: cache user queries with API Gateway, vectorise them with Lambda, store them in DynamoDB or OpenSearch, and feed them back to your Bedrock LLM. This approach aligns perfectly with frugal chatbot principles by maximising efficiency.<\/p>\n\n

    Reinforcement and Benchmarking<\/h3>\n\n

    In this final phase, you can apply Reinforcement Learning from Human Feedback (RLHF) to refine your model and conduct testing and benchmarking.<\/p>\n\n

    To keep evaluations efficient, automate your evaluation pipelines using serverless workflows, such as Step Functions and Lambda, and use representative sample datasets for benchmarks rather than running exhaustive tests.<\/p>\n\n

    Deploy a sustainable infrastructure on AWS<\/h2>\n\n

    For a compact but efficient chatbot architecture, here’s what I recommend:<\/p>\n\n

      \n
    1. Amazon CloudFront<\/strong> \u2013 Content delivery<\/li>\n\n\n\n
    2. S3<\/strong> \u2013 Static web hosting<\/li>\n\n\n\n
    3. AWS WAF<\/strong> \u2013 Traffic control and security<\/li>\n\n\n\n
    4. Application Load Balancer (ALB)<\/strong> \u2013 Traffic distribution<\/li>\n\n\n\n
    5. ECS<\/strong> \u2013 Hosting conversational logic<\/li>\n\n\n\n
    6. DynamoDB<\/strong> \u2013 Storing conversation history<\/li>\n\n\n\n
    7. Bedrock LLM<\/strong> \u2013 Generating responses<\/li>\n<\/ol>\n\n

      So how can we maximise the sustainability impact of this setup?<\/p>\n\n

      Cache strategy<\/h3>\n\n

      First, as we’ve seen, Bedrock will account for much of the environmental impact. We can significantly reduce this through caching strategies.<\/p>\n\n

      Use an in-service cache\u2014such as Valkey (replacing Redis\/ElastiCache) or DynamoDB with TTL\u2014to reuse recent responses in near-real-time. Combine this with response hashing and deterministic prompts so identical requests hit the cache. This approach is especially effective for FAQ-style chatbots with frequently repeated questions.<\/p>\n\n

      Bedrock now offers native prompt caching<\/a> for models like Claude and Amazon Nova, which allows the service to skip reprocessing input tokens. This can dramatically reduce compute requirements for repeated prompts.<\/p>\n\n

      You can also implement caching at the CloudFront level with longer TTLs for templated or commonly repeated responses.<\/p>\n\n

      Batched work where possible<\/h3>\n\n

      For tasks like summaries or long reports, consider implementing an SQS queue with an autoscaling worker pool. This approach avoids wasting compute time on container startup delays and ensures efficient resource utilisation.<\/p>\n\n

      Compute impact<\/h3>\n\n

      To optimise for sustainability, prioritise EC2 spot instances over on-demand instances whenever possible. Even better, favour serverless Fargate over EC2 unless you have specific requirements like specialised networking or GPU access. When using Fargate, opt for Graviton3 (ARM) processors, which deliver better performance per watt and are typically more cost-effective. Finally, keep container images as small as possible to reduce resource consumption.<\/p>\n\n

      Autoscaling<\/h3>\n\n

      Configure autoscaling based on business metrics like request queue length and pending LLM requests, not just CPU and memory usage. Keep minimum capacity low during off-peak hours to avoid idle resources, while ensuring the system can scale up quickly to handle traffic spikes.<\/p>\n\n

      Network<\/h3>\n\n