![\"Integrating](\"https:\/\/devoteam.info\/wp-content\/uploads\/2024\/11\/chatbot-4071274_1920.jpg\")
<\/a><\/figure>\n\n\t\t\n\t\tThe white paper discussed a possible use case of an AI ecommerce chatbot, where MuleSoft was used as the integration layer to access the relevant LLM. As I proposed that use case, I feel it incumbent upon me to build a Proof-of-Concept (PoC) to demonstrate the capability of this use case. <\/p>\n\n\n\n
From this article, \u201cIntegrating MuleSoft with OpenAI: Building an intelligent chatbot\u201d, you\u2019ll discover that the software tools provided by MuleSoft are highly capable of integrating with this technology, allowing the development of solutions that utilise AI. You\u2019ve got the backbone of the infrastructure for these solutions using the core architectural concepts of API-Led Connectivity and Application Networks, combined with Anypoint Platform for full-lifecycle API Management iPaaS. <\/p>\n\n\n\n
Architecture and Design<\/h2>\n\n\n\n
I proposed the building of an e-commerce chatbot. However, re-evaluating that original idea, I\u2019ll modify the aim slightly. For the PoC, I feel it\u2019s better to build a generic and easily adaptable chatbot that uses public or easily accessible services to provide the functionality. As this is a generic solution, it can easily be extended to use different services to create a custom-tailored chatbot or data analysis tool for any specific task.<\/p>\n\n\n\n
So, the Brief:<\/p>\n\n\n\n
Build a Smart chatbot (i.e. LLM-based) accessible over the internet from a Web Browser or a Personal Assistant hardware device (e.g. Amazon\u2019s Echo\/Alexa). The chatbot should use conversational AI technology to interact with a user and have accessibility features allowing it to use audio hardware (i.e. microphones and speakers).<\/em><\/p>\n\n\n\n That\u2019s what\u2019s required; how\u2019s it going to be achieved?<\/p>\n\n\n\n Starting with the conversational Language Model. Most of you will be aware of the two most well-known LLMs currently in the spotlight; OpenAI\u2019s ChatGPT<\/a> and Google\u2019s Bard<\/a>, and as ChatGPT has a publicly accessible API, that\u2019s the one we will use for this PoC.<\/p>\n\n\n\n For Automatic Speech Recognition (ASR), OpenAI\u2019s Whisper<\/a> Speech-to-Text (STT) service will interact with a microphone. It will capture and transcribe any spoken prompts\/requests from a user. OpenAI doesn\u2019t have a Text-to-Speech (TTS) service, so Google Translate\u2019s free TTS API will be used (sufficient for this PoC).<\/p>\n\n\n\n One final Requirement that mustn\u2019t be forgotten is the ability to interact with a Personal Assistant device to access the chatbot. With Amazon\u2019s Echo device, it is possible to create custom functionality called Alexa Skills<\/em>. A Skill<\/em> will be developed to integrate with the chatbot and make it accessible from any such device.<\/p>\n\n\n\n Before getting into the Build, the first stage is to take a deeper look into those services and technology, to understand them and, technically, how to call the respective APIs.<\/p>\n\n\n\n Two services from OpenAI: ChatGPT (Conversational AI) and Whisper (STT), will be used:<\/p>\n\n\n\n The Chat<\/em> endpoint is:<\/p>\n\n\n\n https:\/\/api.openai.com\/v1\/chat\/completions<\/a><\/p>\n\n\n\n This POST endpoint accepts JSON and contains the text prompt from the user.<\/p>\n\n\n\n As this API is well documented, there is no need for further detail apart from providing a link to the API Reference<\/a> page.<\/p>\n\n\n\n The Transcriptions<\/em> endpoint is:<\/p>\n\n\n\n https:\/\/api.openai.com\/v1\/audio\/transcriptions<\/a><\/p>\n\n\n\n This POST endpoint accepts multi-part data indicating the path to the relevant audio file.<\/p>\n\n\n\n Again, this API is well documented, so you only need a link to the API Reference<\/a> page.<\/p>\n\n\n\n However, before proceeding further, you should note: <\/p>\n\n\n\n Though note that GPT-4<\/a> (OpenAI\u2019s next iteration of the GPT series of LLMs) already has a much higher token limit. Additionally, further research<\/a> aims to raise the threshold substantially higher, effectively removing this limit for most use cases.<\/p>\n\n\n\n As stated earlier, this TTS service is sufficient for this PoC. But note there\u2019s no documentation on the API or how to use it (other than anecdotal evidence on developer-oriented sites such as StackOverflow<\/em>).<\/p>\n\n\n\n To document what is known about this API for the necessary purposes of this PoC.<\/p>\n\n\n\n https:\/\/translate.google.com\/translate_tts?ie=UTF-8&tl=en-UK&client=tw-ob&q={text}<\/a><\/p>\n\n\n\n It is a GET endpoint with all parameters passed as query parameters<\/em> in the URL. The two essential parameters are tl which specifies the type\/accent of the voice, and q, which is the text to be spoken.<\/p>\n\n\n\n Note Google does have a much more advanced TTS API – but it\u2019s not free. Hence, the reason for using this one in the PoC.<\/p>\n\n\n\n The documentation<\/a> detailing how to build an Alexa Skill and how to call APIs from those Skills is extensive. There are also many blogs and posts to follow. As this article aims not to cover building such functionality, it will not be detailed here. All that is needed is a relatively simple Skill that can call out to APIs.<\/p>\n\n\n\n The chatbot GUI is also simple; a Python Flask<\/em> app using HTML. This article isn\u2019t concerned with Python web development, so it isn\u2019t covered. However, the documentation<\/a> is extensive, and several Python libraries (e.g. Streamlit<\/a>) already exist that can be used to build chatbot interfaces if preferred.<\/p>\n\n\n\n If we re-focus on MuleSoft and apply the principles of API-Led Connectivity<\/a>, this is what a possible High-level Application Network<\/a> will look like.<\/p>\n\n\n\n If you\u2019ve not yet done so, I encourage you to read the <\/em>Whitepaper<\/em><\/a> to understand more about MuleSoft and concepts such as API-Led Connectivity.<\/em><\/p>\n\n\n\n With the Design in place, the Build can begin. Let\u2019s demo the necessary features and specific usability aspects.<\/p>\n\n\n\n First, this chatbot needs a persona relevant to its use case. As this is just a generic chatbot, it’ll take on the persona of \u2018Max, the Mule\u2019 – a helpful personal assistant.<\/p>\n\n\n\n To do this, a System Message<\/em> has to be passed to ChatGPT, which initiates the interaction and tells ChatGPT how to act.<\/p>\n\n\n\n {“role”: “system”, “content”: “You are Max the Mule, a helpful assistant.”}<\/p>\n\n\n\n There is a slight delay on initial loading as the screen makes an API call to ChatGPT to get the chatbot\u2019s initial greeting (as with all Web Sites making an API call). So while that is happening in the background, the screen is only partially rendered. It contains some static images, which have been added for aesthetics – to give the chatbot a certain \u2018look and feel\u2019. These can be customised or replaced as required. And, of course, the screen displays the obligatory Loading Wheel<\/em>.<\/p>\n\n\n\n Once the initial call to ChatGPT is complete, the screen is then fully rendered.<\/p>\n\n\n\n This initial screen contains several different objects:<\/p>\n\n\n\n The screenshot below shows that the chatbot has picked up the user\u2019s question and added it to the conversation history. The chatbot then responded to it. Again, audio controls have been given to replay the response verbally.<\/p>\n\n\n\n The Grand Prix race the chatbot is referring to here was back in 2019 – its information doesn\u2019t appear to be completely up-to-date. But having read the Whitepaper<\/a>, you\u2019ll know that LLMs are trained on existing data with a cut-off date. ChatGPT\u2019s training cut-off was towards the end of 2019, which makes sense considering the answer given.<\/p>\n\n\n\n There are ways to overcome this by giving LLMs access to additional data processing tools and more recent data. This will be discussed again later on.<\/p>\n\n\n\n A critical feature of any chatbot is its ability to keep the conversation’s context as it progresses. And as stated earlier, this is possible by re-submitting the conversation history on each interaction.<\/p>\n\n\n\n Remember, the resubmission of the complete conversation has implications, and there is a limit. However, there are ways to reduce this impact – such as dropping older interactions (probably not needed as the conversation has progressed) or only keeping track of the chatbot\u2019s responses (this will generally be sufficient to give a good overview of the conversation).<\/p>\n\n\n\n However, in this PoC, the whole conversation history is retained. For this, a Key-Value<\/em> store is the appropriate solution, and MuleSoft has one available, the Object Store<\/a>.<\/p>\n\n\n\n The final interesting aspect of the chatbot is its ability to take verbal input directly from the microphone. There are multiple ways of doing this, but for this PoC, there are simple buttons for turning the microphone on and off (bottom right of the screen).<\/p>\n\n\n\n Remember, the chatbot uses OpenAI\u2019s Whisper service to transcribe the spoken words. Notice that not only can it discern each word, but it can add correct punctuation – such as Question Marks.<\/p>\n\n\n\n One final point is that the answer to the last question is wrong, referred to as a hallucination<\/em> – the LLM has made up the answer. This is a fundamental flaw with generative AI and has to be taken seriously. However, many articles refer to this subject and ways to mitigate against it – more focused prompts, additional training, etc.<\/p>\n\n\n\n The chatbot now runs happily from a Web Browser. However, the final aspect of this PoC was accessing it from a Personal Assistant device. The GUI itself cannot run on a device. Still, as MuleSoft\u2019s approach of API-Led Connectivity has been used, all that needs to be done is build an Alexa Skill to access the APIs within the Application Network. Below is the same conversation as above but from Alexa.<\/p>\n\n\n\n The following screenshot is taken from the Alexa Skills Developer Console<\/a>.<\/p>\n\n\n\n The first Command, open mule bot, is required to open the Alexa Skill (named \u2018mule bot<\/em>\u2019). Then, each question is prefixed with the Command ask max to inform the Skill that a question is being asked.<\/p>\n\n\n\n Did we complete the brief of integrating MuleSoft with OpenAI and building an intelligent chatbot? <\/p>\n\n\n\n Well the complete chatbot can be accessed from both a Web Browser and a Personal Assistant device. So I would say the Brief has been met. But this was only a PoC. What additional enhancements could be added in the future?<\/p>\n\n\n\n The most apparent enhancement (and probably the simplest) is to allow for further customisation to the responses from ChatGPT; many default settings were chosen for this PoC. Though, several parameters can be passed in with a user\u2019s prompt, such as temperature<\/a>, which can be used to tune the creativity and randomness of the response.<\/p>\n\n\n\n Another enhancement would be to replace the Object Store with a database (such as MongoDB) depending on whether the extra features and functionality of a complete NOSQL database would be helpful.<\/p>\n\n\n\n The final improvement to mention (and would be obligatory if this PoC were to be developed further) would be to allow multiple users to access the chatbot simultaneously. Currently, only one user can use the chatbot at any time, and the current retention of the conversation history reflects this. For multiple users, each conversation must be distinguishable from all other discussions. This could be done by giving each user a unique ID to be embedded in the key of the conversation database.<\/p>\n\n\n\n So we\u2019ve finished, and Yes<\/em>, this has been a fun, technical challenge, and it demonstrates that it is possible to integrate with this technology, but is there any actual<\/em> use to it? We must first understand what is happening in this sector to answer this question.<\/p>\n\n\n\n Advances in Conversational AI and Natural Language Processing are exceptionally rapid at the moment – new progress is seemingly being documented each week (for both good and bad). For example, OpenAI\u2019s GPT-3.5, which was only released at the end of last year, has already been superseded by GPT-4 (a model that\u2019s much more sophisticated).<\/p>\n\n\n\n OpenAI has also recently introduced the ability to build Plugins<\/em><\/a> for ChatGPT, allowing access to third-party applications (such as internet search capabilities and computational functionality). Note that Plugin development is still in the early stages but will considerably increase the usefulness of LLMs and solutions built upon them. <\/p>\n\n\n\n Python-based projects, such as LlamaIndex<\/a> and LangChain<\/a>, provide similar capabilities extending the reach and capabilities of LLMs even further.<\/p>\n\n\n\n Privacy concerns and ethical questions are continually highlighted and debated, with new policies being written and reworked. Fully open-source LLM variants<\/a> are being developed, giving broader access and demanding reduced hardware requirements<\/a> for running. All of which aim to combat the exclusive and closed nature of the technology.<\/p>\n\n\n\n So, to answer whether such technology is of actual use. The answer is Yes<\/em>. It’s becoming possible to build solutions that allow LLMs to access data far beyond their initial training input, giving them access to up-to-date and non-public data.<\/p>\n\n\n\n Consider that many Organisations already hold and retain massive amounts of structured and unstructured data stored in documents, databases and private repositories – all key to their success. This technology is extremely good at tasks such as Summarisation, Extraction and Classification, which are invaluable but can also be utilised further to create Recommendations and perform Sentiment Analysis. These Organisations must start thinking about how AI-driven solutions can support them when challenged to maintain that competitive edge.<\/p>\n\n\n\n By creating new services for internal and external consumption, they can provide value and benefit to consumers at all levels. Achieved through access, analysis, and ultimately from seeing meaningful and valuable insights into their business – all with the help of MuleSoft Integration. Contact us<\/a> today to see how we can help you with integrating MuleSoft with OpenAI.<\/p>\n","protected":false},"excerpt":{"rendered":" Introduction Recently, I wrote a Whitepaper giving an overview of Conversational Artificial Intelligence (AI), the relationship to Large Language Models (LLMs), and their potential use cases. If you\u2019ve not yet read the whitepaper, I\u2019d recommend doing so; to get a grounding in the technology and to understand this new article more effectively. The white paper […]<\/p>\n","protected":false},"featured_media":112223,"template":"","categories":[752],"tags":[1148],"industry":[],"class_list":["post-160863","expert-view","type-expert-view","status-publish","has-post-thumbnail","hentry","category-business-platforms","tag-uk-import-uk"],"acf":[],"cards":"\n\tOpenAI<\/h3>\n\n\n\n
\n
\n
\n
Google<\/h3>\n\n\n\n
Alexa Skills<\/h3>\n\n\n\n
Python Flask<\/h3>\n\n\n\n
MuleSoft<\/h3>\n\n\n\n
![\"\"](\"https:\/\/devoteam.info\/wp-content\/uploads\/2024\/11\/image-9-1.png\")
<\/figure>\n\n\n\nBuilding the Chatbot<\/h2>\n\n\n\n
Initial Loading Screen<\/h3>\n\n\n\n
![\"\"](\"https:\/\/devoteam.info\/wp-content\/uploads\/2024\/11\/image-10-1.png\")
<\/figure>\n\n\n\nLoaded Chatbot<\/h3>\n\n\n\n
![\"\"](\"https:\/\/devoteam.info\/wp-content\/uploads\/2024\/11\/image-12-1.png\")
<\/figure>\n\n\n\n\n
\n
\n
Asking the First Question<\/h3>\n\n\n\n
![\"\"](\"https:\/\/devoteam.info\/wp-content\/uploads\/2024\/11\/image-13-1.png\")
<\/figure>\n\n\n\nConversation History<\/h3>\n\n\n\n
![\"\"](\"https:\/\/devoteam.info\/wp-content\/uploads\/2024\/11\/image-14-1.png\")
<\/figure>\n\n\n\nSpoken Input<\/h3>\n\n\n\n
![\"\"](\"https:\/\/devoteam.info\/wp-content\/uploads\/2024\/11\/Screenshot-2023-05-09-at-11.55.24.png\")
<\/figure>\n\n\n\nAlexa Skills<\/h3>\n\n\n\n
![\"\"](\"https:\/\/devoteam.info\/wp-content\/uploads\/2024\/11\/image-11-1.png\")
<\/figure>\n\n\n\nConclusion<\/h2>\n\n\n\n
Next Steps<\/h3>\n\n\n\n
Unstoppable AI<\/h3>\n\n\n\n
Final Thoughts<\/h3>\n\n\n\n
\n\n\t\t<\/a><\/figure>\n\n\t\t\n\t\t