One of the key features that make conversational AI assistants feel more human-like is their ability to maintain context and remember past conversations.
By default, large language models (LLMs) are stateless, meaning each interaction is processed independently, without retaining any memory of previous exchanges. However, there are various ways to provide memory to LLMs, allowing chatbots to maintain a conversational context and deliver more coherent and personalized responses.
Passing Chat History as Context
A straightforward approach to providing memory to chatbots is to pass the chat history as context along with the current prompt. This way, the LLM can process the entire conversation thread, enabling it to understand and respond appropriately to the current query within the context of the previous exchanges.
Here’s a sample code using Langchain to illustrate this approach:
from langchain_openai import ChatOpenAI
chat = ChatOpenAI(model="gpt-4o", api_key="")
from langchain_core.messages import AIMessage, HumanMessage
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
prompt = ChatPromptTemplate.from_messages(
[
(
"system",
"You are a helpful assistant. Answer all questions to the best of your ability.",
),
MessagesPlaceholder(variable_name="messages"),
]
)
chain = prompt | chat
chat_history = [
AIMessage(content="Hi Nishi! It's nice to meet you. As a software engineer, you must work on a variety of interesting projects. How can I assist you today? Are you looking for help with a specific programming problem, career advice, or something else?"),
HumanMessage(content="What career advise would you give me ?"),
]
result = chain.invoke(
{
"messages": [
HumanMessage(
content="Hi, My name is Nishi. I am software engineer."
)
] + chat_history ,
}
)
content="Certainly, Nishi! Here are some career tips tailored for a software engineer:\n\n1. **Continuous Learning**:\n - **Stay Updated**: Technology evolves rapidly. Stay current with new programming languages, frameworks, and tools.\n - **Online Courses and Certifications**: Platforms like Coursera, Udemy, and edX offer courses that can boost your skills.\n\n2. **Improve Problem-Solving Skills**:\n - **Practice Coding**: Websites like LeetCode, HackerRank, and CodeSignal are great for practicing coding problems.\n - **Algorithm and Data Structures**: Deepen your understanding of fundamental algorithms and data structures as they are crucial for technical interviews and efficient coding.\n\n3. **Contribute to Open Source**:\n - **GitHub**: Contributing to open source projects can improve your skills, help you learn from others, and build a strong portfolio.\n\n4. **Networking**:\n - **Professional Networks**: Join LinkedIn, attend industry conferences, meetups, and webinars to connect with other professionals.\n - **Mentorship**: Seek mentors who can provide guidance and advice based on their experiences.\n\n5. **Work on Soft Skills**:\n - **Communication**: Effective communication is key in team environments.\n - **Collaboration**: Work well in teams, as software development is often a collaborative effort.\n\n6. **Build a Strong Portfolio**:\n - **Showcase Projects**: Create a portfolio showcasing your best projects, including descriptions, technologies used, and your role.\n - **Personal Website**: Consider creating a personal website to display your portfolio, resume, and blog posts.\n\n7. **Career Planning**:\n - **Set Goals**: Define short-term and long-term career goals. This could include mastering a new language, taking on leadership roles, or transitioning into a specialized field.\n - **Seek Feedback**: Regularly seek feedback from peers and supervisors to understand areas for improvement.\n\n8. **Explore Different Domains**:\n - **Specializations**: Explore various fields such as web development, mobile development, data science, AI, cybersecurity, etc., to find what excites you the most.\n\n9. **Stay Healthy and Balanced**:\n - **Work-Life Balance**: Maintain a healthy work-life balance to avoid burnout.\n - **Physical and Mental Health**: Regular exercise, hobbies, and taking breaks are important for long-term productivity.\n\n10. **Consider Advanced Education**:\n - **Graduate Degrees**: If you're interested in research or specialized knowledge, consider pursuing a master's degree or Ph.D.\n\nFeel free to ask more specific questions if you have any particular areas you want advice on!"
"usage_metadata":{'input_tokens': 106, 'output_tokens': 545, 'total_tokens': 651}}
While this method is straightforward, it has some limitations. As the chat history grows longer, the number of input tokens increases, which can cause issues when using models with low context length. Additionally, longer input sequences can lead to higher computational costs and increased latency.
Summarizing Previous Conversations
Another approach to providing conversational memory to chatbots is to summarize the previous conversations and include the summary as additional context along with the current prompt. This can help reduce the number of tokens being passed as the conversation grows longer, mitigating the limitations associated with models that have a low context length.
Here’s an example of how you can implement this:
stored_messages = [
AIMessage(content="Hi Nishi! It's nice to meet you. As a software engineer, you must work on a variety of interesting projects. How can I assist you today? Are you looking for help with a specific programming problem, career advice, or something else?"),
HumanMessage(content="What career advise would you give me ?"),
AIMessage(result.content)
]
summarization_prompt = ChatPromptTemplate.from_messages(
[
MessagesPlaceholder(variable_name="chat_history"),
(
"user",
"Distill the above chat messages into a single summary message. Include as many specific details as you can.",
),
]
)
summarization_chain = summarization_prompt | chat
summary_message = summarization_chain.invoke({"chat_history": stored_messages})
conversation_history = AIMessage(summary_message.content)
result = chain.invoke(
{
"messages": [
conversation_history,
HumanMessage(
content="What should be my focus areas to help me grow in my career?"
)
]
}
)
print(result)
content='To help grow your career as a software engineer, consider focusing on the following areas:\n\n1. **Technical Skills:**\n - **Programming Languages:** Master at least one or two programming languages deeply (e.g., Python, Java, C++).\n - **Data Structures and Algorithms:** Strengthen your understanding of these fundamentals to solve complex problems efficiently.\n - **Software Development Practices:** Learn about version control (e.g., Git), testing (unit, integration, and system testing), and continuous integration/continuous deployment (CI/CD) pipelines.\n - **System Design and Architecture:** Understand how to design scalable, reliable, and maintainable systems.\n\n2. **Specialization:**\n - Explore different fields such as web development, mobile development, machine learning, data science, cloud computing, cybersecurity, DevOps, etc., and develop expertise in one or more areas.\n\n3. **Soft Skills:**\n - **Communication:** Improve your ability to explain complex technical concepts to non-technical stakeholders and collaborate effectively with team members.\n - **Teamwork:** Develop the ability to work well within a team, including participating in code reviews and pair programming.\n - **Problem-Solving:** Hone your analytical and critical thinking skills to tackle challenging problems.\n\n4. **Networking and Professional Development:**\n - **Networking:** Attend industry conferences, meetups, webinars, and workshops to meet other professionals and stay informed about industry trends.\n - **Mentorship:** Seek out mentors who can provide guidance and advice, and consider mentoring others to reinforce your own knowledge.\n - **Certifications and Courses:** Pursue relevant certifications and take courses to stay current with new technologies and methodologies.\n\n5. **Contributions and Portfolio:**\n - **Open Source:** Contribute to open-source projects to gain experience, improve your skills, and expand your network.\n - **Portfolio:** Build and maintain a portfolio of your projects on platforms like GitHub to showcase your work to potential employers.\n\n6. **Career Planning:**\n - **Goal Setting:** Set clear, achievable career goals and create a roadmap to reach them.\n - **Continuous Learning:** Stay curious and committed to lifelong learning. Read books, follow industry blogs, listen to podcasts, and take online courses.\n\n7. **Work-Life Balance:**\n - Ensure you maintain a healthy work-life balance to avoid burnout and maintain long-term productivity and job satisfaction.\n\n8. **Adaptability:**\n - Be open to change and willing to adapt to new roles, responsibilities, and technologies as the industry evolves.\n\nBy focusing on these areas, you can build a strong foundation for growth and advancement in your software engineering career.' response_metadata={'token_usage': {'completion_tokens': 538, 'prompt_tokens': 185, 'total_tokens': 723}, 'model_name': 'gpt-4o', 'system_fingerprint': 'fp_319be4768e', 'finish_reason': 'stop', 'logprobs': None} id='run-89700a7e-935b-4e02-a325-12e763f260bd-0' usage_metadata={'input_tokens': 185, 'output_tokens': 538, 'total_tokens': 723}
Incorporating conversational summaries as context can be especially useful in scenarios where the conversation spans multiple topics or when the chat history becomes excessively long. It allows the chatbot to maintain a high-level understanding of the conversation without being bogged down by the verbatim details of every previous exchange.
Similarity search on Chat History
While the summarization technique can be effective in managing context length, there is a risk that important details from the conversation may be missed or oversimplified during the summarization process.
To address this issue, an alternative approach is to perform a similarity search on the stored chat history and retrieve the conversations most relevant to the user’s current query. This method ensures that the chatbot receives context that is directly relevant to the user’s question or topic, reducing the risk of missing crucial information. By selectively retrieving relevant sections of the chat history, we can keep the number of tokens in check while avoiding the potential information loss that can occur with summarization.
I will go through the similarity search in detail in the next blog. Till then stay tuned.
Here is the colab for this post.