New LLM draws wisdom from ancient texts

Researchers from the University of Galway, Ireland, and IIIT Delhi, India, have introduced a new framework that combines the spiritual teachings of the Bhagavad Gita with advanced artificial intelligence to create a more holistic approach to mental health support. The study, authored by Janak Kapuriya, Aman Singh, Jainendra Shukla, and Rajiv Ratn Shah, explores how the wisdom of this ancient Hindu scripture can be integrated into large language models (LLMs) to provide deeper and more meaningful emotional support than what is currently available.

The problem, as the researchers see it, is that existing mental health support systems, including many AI-powered chatbots, often provide superficial responses based on a user’s immediate emotional state. While these systems can be helpful, they may not address the underlying emotional and spiritual needs of the individual. This new research seeks to bridge that gap by creating an AI that can offer guidance rooted in the profound philosophical and psychological insights of the Bhagavad Gita.

To achieve this, the researchers developed a new dataset called GITes, which stands for Gita Integrated Therapy for Emotional Support. They started with an existing mental health dataset, ExTES, and enriched it with 10,729 spiritually guided responses. These new responses were generated by the powerful GPT-4o language model and then carefully evaluated by domain experts to ensure their accuracy and appropriateness.

The process of creating the GITes dataset was a meticulous one. The researchers, in consultation with experts from ISKCON (the International Society for Krishna Consciousness), identified relevant verses, or shlokas, from the Bhagavad Gita that offer guidance on a wide range of human emotions and life situations. These verses, along with their detailed explanations, or purports, were then used to train the AI to generate responses that are not only empathetic but also spiritually insightful. The goal was to create a system that can understand a user’s emotional state and provide guidance that is both contextually relevant and spiritually uplifting.

EmoNet signals new wave of emotionally aware AI models

One of the key challenges in this research was figuring out how to measure the “spiritual relevance” of the AI-generated responses. Traditional metrics used to evaluate language models, such as those that measure word overlap, are not well-suited for capturing the nuances of spiritual wisdom. To address this, the researchers proposed a new metric called “Spiritual Insight.”

To automate the assessment of this new metric, they developed a clever “LLM-as-Jury” framework. This involved using a panel of different large language models to evaluate the spiritual quality of the responses. This innovative approach allowed the researchers to assess the spiritual depth of the AI’s responses in a way that goes beyond simple keyword matching.

The results of the study are striking. When the researchers tested their GITes-trained models, they found significant improvements across a range of metrics. The best-performing model, Phi3-Mini 3.2B Instruct, showed a dramatic increase in both standard NLP metrics and the newly developed spiritual metrics.

Compared to its performance without the specialized training, the model’s ROUGE score (a measure of content overlap) improved by over 122%, and its METEOR score (which considers synonyms and word order) increased by over 126%. More importantly, the model’s “Spiritual Insight” score saw a nearly 16% increase, and its scores for “Sufficiency” and “Relevance” also improved significantly. These results strongly suggest that integrating spiritual guidance into AI-driven support systems can lead to more effective and satisfying user experiences.

Featured image credit