--- title: "Improve LLM Chatbot Services | AI Consulting Services For Private Equity" url: https://www.pressw.ai/blog/improving-user-experience-of-llm-applications-through-streaming-and-user-engagement-tricks description: "PressW helps ambitious brands integrate AI into their teams and products. Managed AI Services and AI Transformation for custom AI engineering - one partner across the full lifecycle." --- [What's manual work costing your firm?Free AI ROI Calculator for PE FirmsAI ROI Calculator for Private Equity](https://www.pressw.ai/pe-calculator) [![PressW](https://www.pressw.ai/images/logo.svg?dpl=dpl_82sxYaBNW8k3DYpsymzCnFSQufoY)](https://www.pressw.ai/) - [Methodology](https://www.pressw.ai/methodology) - Divisions Divisions Two ways to work together [#### Managed AI Services](https://www.pressw.ai/managed-ai)[#### AI Transformation](https://www.pressw.ai/ai-transformation) [Contact →](https://www.pressw.ai/contact) Official Anthropic Partner ### Managed AI Services As an Official Anthropic Partner, we deploy, manage, and optimize Claude across your organization - from enterprise rollouts and license management to custom agent development and cost optimization. Services Claude Deployment & ConfigurationUsage Analytics & Cost OptimizationSecurity, Compliance & GovernanceClaude License ManagementCustom Claude Agent DevelopmentOngoing Training & Support [Explore Managed AI Services](https://www.pressw.ai/managed-ai) - Verticals [Verticals→](https://www.pressw.ai/verticals) Deep domain expertise #### Capital AI infrastructure & value creation for investors [Read more](https://www.pressw.ai/verticals/capital) #### Technology Ship AI features on your release cadence [Read more](https://www.pressw.ai/verticals/technology) #### Professional Services Automate the work so your people focus on clients [Read more](https://www.pressw.ai/verticals/services) #### Industrial Enterprise-grade AI for regulated environments [Read more](https://www.pressw.ai/verticals/industrial) [Book intro call →](https://www.pressw.ai/contact) ### Capital We partner with capital investors to manage AI infrastructure across their portfolios — supporting deal teams with AI-powered diligence and driving value creation at the portco level. Top Industries Private EquityVenture CapitalFinancial Services [Explore Capital](https://www.pressw.ai/verticals/capital) - Resources ### Resources We'll help take your Claude and AI knowledge to the next level. [Book intro call →](https://www.pressw.ai/contact) AI Resources #### AI ROI Calculator→ Discover just how much you can save with AI workflows. [Read more](https://www.pressw.ai/pe-calculator) #### Case Studies→ Real results from AI initiatives we've taken to production. [Read more](https://www.pressw.ai/case-studies) #### Blog→ Insights on AI strategy, engineering, and operations. [Read more](https://www.pressw.ai/blog) #### AI Glossary→ Key AI and ML terms explained for operators and decision-makers. [Read more](https://www.pressw.ai/glossary) About #### About Us→ Our vision, values, and the team behind PressW. [Read more](https://www.pressw.ai/about) #### Careers→ Join our team of senior AI engineers and operators. [Read more](https://www.pressw.ai/careers) [Explore Partnership](https://www.pressw.ai/contact) [Back to Blog](https://www.pressw.ai/blog) Industry InsightsMarch 22, 2024 # Improving User Experience of LLM Applications through Streaming and User Engagement Tricks Liam DeCoste CTO, PressW Send to Send to ![Improving User Experience of LLM Applications through Streaming and User Engagement Tricks](https://www.pressw.ai/_next/image?url=https%3A%2F%2Fcdn.sanity.io%2Fimages%2Fbmhbpjvu%2Fproduction%2F3106fe15a8a2fc125ea18f8d3b702f65d05cebd7-1200x630.jpg%3Fw%3D1400&w=3840&q=75) _5 minute read time_ ### The Problem We are in the exponential growth phase of Large-Language Models (LLMs). There are new releases of OpenAI’s GPT, Mistral, and Gemini every few weeks and they are getting smarter. Under the hood, there is a tug-of-war going on between increased intelligence and the speed at which it responds to the user. Currently, speed is losing the war to intelligence and thus the experience for users within the extremely intelligent applications being released is far from enjoyable. This experience is further ruined by plain LLMs like ChatGPT running extremely fast because they are past the cost-of-scale hurdle and are less complex operation pipelines in nature. This article aims at diving in deep into two areas that we focus on at PressW: optimizing time to first token and showing intermediate LLM steps as user experience tricks to help your application feel closer to the current golden standard of ChatGPT. ### User Experience Techniques Now that LLM applications have become mainstream, there are a few techniques that have come into play to create an engaging and pleasant user experience. #### Time-To-First Token Traditionally, applications measure latency and response time of the requests that the interface makes to the backend to run operations. This metric is something that backend systems developers are continually trying to improve in systems so that the user can have a quick experience within the application. A few examples where this metric are used are: - Time to complete a user login - Fetching data from a database - Processing a user’s uploaded avatar image LLM applications have a different metric that is extremely important to the user’s experience: **time-to-first token.** This metric takes into account the time that the system takes to send back the very first LLM token to the user. When putting these two numbers side-by-side, they can be drastically different values. As you can see in the table above, the time to first token is ~2 seconds. **This is the time the user spends waiting for feedback from the system.** Complex LLM systems will see this number increase as well as you add in agents, tools, and RAG systems (check out this article on RAG). When you optimize the time to first token as much as you can within your system, it’s time to employ other UX tricks to provide a great user experience. ##### Ways to optimize this metric **Response Streaming** Implementing response streaming is the very first step in optimizing the time to first token metric. It provides the system with a way to return partial responses. This immediately reduces the metric as we are not waiting on the LLM to fully finish it’s thought before it starts sending data back to the user’s application window. This is the main factor in why ChatGPT feels so fast even when it is returning an extremely long-winded answer. **Backend Parallelization** Modern LLM tooling for backend developers allows them to make certain requests in parallel. Many complex LLM applications rely on multiple agents to generate intermediate outputs and understandings of problems. You can then parallelize these operations and reduce the latency to the final LLM that will then start streaming responses back to the users of the application. _Non-Parallel Backend_ _Parallel Backend_ As you can see with the above example, we are able to condense the steps that the first backend system was doing into a much shorter step via parallelizing the system. This is definitely a more advanced technique that developers will need to spend time on to understand the different data dependencies and where it makes sense to run operations in parallel, but it is a powerful technique when used effectively. #### Showing Intermediate Steps The above diagrams show extremely simplified versions of LLM applications. In production applications, there are going to be even more steps that the system has to go through in order to even begin returning tokens to users. Example production pipeline These intermediate steps are always going to have to happen and leave the user in a state of not knowing what is going on. Nielsen Norman Group has a set of heuristics that you can read about [here](https://www.nngroup.com/articles/ten-usability-heuristics/) that PressW abides by. I’m going to highlight the key one for this pain point: **Visibility of system status**. The N/N Group’s guidelines for this heuristic are: **_The design should always keep users informed about what is going on, through appropriate feedback within a reasonable amount of time._** _When users know the current system status, they learn the outcome of their prior interactions and determine next steps. Predictable interactions create trust in the product as well as the brand._ LLM systems by and large can be transparent and provide insights into the inner mechanisms of what the current state are. Streaming the responses back (as described above) is not only limited to final responses! Using modern tooling, the frontend application can actually get the status of the system as a whole back. At PressW, we use this intermediate data to provide out what we call the “pizza tracker experience”. ##### Real-World Examples **Dominos Tracker** If you’ve ordered Dominos online in the past decade, you are probably familiar with their Dominos Tracker. This is a prime example of an opaque system, the process of preparing a pizza from order to pickup, that has been shattered open to provide users with a clear visual of what the current state of the system is. LLM applications can use the same concepts in order to provide users with feedback even when complex and long-running processes (30+ seconds) are taking place in the background to get the user the answers that they expect from a system. [**Perplexity.ai**](http://perplexity.ai/) Perplexity is an LLM-based search engine. It takes in standard queries that a user would type into Google such as “Who is the quarterback of the Kansas City Chiefs” and then returns text-based answers with inline source citations. It’s very similar to Bing’s AI search if you have used that. What is unique about Perplexity is their Pro Search feature. When enabled, the system becomes interactive and performs much more complex actions on behalf of a user. If I want to create a whole dinner menu that involves meatballs and red wine, I enable pro search and type in my request. **_Steps in the experience:_** - Perplexity takes in a query - Asks follow-up questions - Takes in answers - Creates new search queries - Cites sources - Generates an answer (streamed back to me) **Example Experience** Below, I have asked Perplexity “Help me create a dinner menu that involves meatballs and red wine”. The first thing that the system does is show a loading indicator to understand my question, and then asks me some follow up questions. Once I have answered the questions, it goes through and shows me, linearly, the actions that it is taking to fully answer my question. This linear loading is a technique that addresses the visibility issue into what’s going on within the system. The next step that the system goes through is to begin streaming back the response to the users and displaying the sources that it used. We have now gotten our first tokens back and the user is happy that they are receiving their answer (and a delicious sounding recipe). Perplexity is a great case study in showing the intermediate steps of the workflow that the system is going through. They provide enough contextual information for users to understand their query and provide out the best answer that it can without it seeming like there is a black box and they are waiting with no context. **Final Example: Custom GPT Creation** OpenAI has a recently released feature of their ChatGPT platform that allows users to create their own specialized versions of GPT. As part of this process, the user creates the personality, tone, and system prompt of the custom GPT through a modified chat window. When the system receives an input from the user to configure the GPT and needs to make backend changes, the user sees specialized loading indicators that provide visibility into what OpenAI is doing in the background. The above screenshot shows a sample loading message for the system where OpenAI is using a backend service to generate the profile photo of the custom GPT. This loading indicator eventually gets replaced with the text response that is streamed back to the user saying that it was a success and asking the next follow-up questions. We have an article [here](https://pressw.ai/blog/how-to-create-your-own-openai-gpt-personalized-conversational-ai-in-minutes) on creating your own custom GPT, check it out! _Note on Loading Indicators_ Loading indicators such as this can be a great way of providing users with updates into an LLM system, even if there is not streaming support. Good communication between the product team and engineering can allow for psuedo-visibility into the system. An example of this is a timer-based loading indicator that cycles between different messages to show that multiple things are happening in the background, even if the interface is not sure which step is happening at the current moment. Source: [https://dribbble.com/shots/11342077-Revolver-Loading-Indicator-Animation](https://dribbble.com/shots/11342077-Revolver-Loading-Indicator-Animation) #### Conclusion User experience in LLM applications is crucial and can be enhanced by optimizing the time to the first token and showing intermediate steps of the process. Techniques such as response streaming and backend parallelization can improve the time-to-first-token metric, while providing visibility into system status can enhance user engagement. The future of LLM applications is exciting, and by placing a strong emphasis on user experience, developers and application owners can ensure they stay ahead of the curve, delivering applications that are not only technologically advanced but also user-friendly and intuitive. In a world where user satisfaction is paramount, the importance of these techniques cannot be overstated. [![PressW](https://www.pressw.ai/images/logo-white.svg?dpl=dpl_82sxYaBNW8k3DYpsymzCnFSQufoY)](https://www.pressw.ai/) ## Making AI Work For You. [Book Intro Call](https://www.pressw.ai/contact) Divisions [Managed AI Services](https://www.pressw.ai/managed-ai)[AI Transformation](https://www.pressw.ai/ai-transformation) Verticals [Capital](https://www.pressw.ai/verticals/capital)[Technology](https://www.pressw.ai/verticals/technology)[Professional Services](https://www.pressw.ai/verticals/services)[Industrial](https://www.pressw.ai/verticals/industrial) Resources [Case Studies](https://www.pressw.ai/case-studies)[Blog](https://www.pressw.ai/blog)[Glossary](https://www.pressw.ai/glossary) Company [About](https://www.pressw.ai/about)[Careers](https://www.pressw.ai/careers)[Contact](https://www.pressw.ai/contact) Legal [Privacy Policy](https://www.pressw.ai/privacy)[Terms & Conditions](https://www.pressw.ai/terms-and-conditions)[Terms of Service](https://www.pressw.ai/terms-of-service) © 2026 PressW LLC. All rights reserved. Official Anthropic Partner --- More PressW context for agents: [https://www.pressw.ai/llms.txt](https://www.pressw.ai/llms.txt) - Contact: hello@pressw.ai