Abstract
We fine-tuned NVIDIA's PersonaPlex-7b-v1 model on 200 synthetic coffee shop customer conversations to address hallucinations and poor task adherence observed in baseline customer-facing deployments. Using Low-Rank Adaptation (LoRA) training with ChatterboxTTS-generated audio and LibriSpeech voice datasets, we introduced two novel techniques: a semantic-weighted loss function that penalises role-inversion errors more heavily than surface-level fluency mistakes, and a voice prompt injection mechanism that prepends speaker identity embeddings to guide the model's persona consistency throughout multi-turn dialogues. LLM-as-judge evaluation across three test configurations — Base (unmodified PersonaPlex), Context-Only (system prompt with role description), and Voice-Only (prompt injection without semantic loss) — shows our combined LoRA + Semantic Loss + Voice Prompt approach outperforms all baselines on emotional accuracy, role consistency, and task completion rate. Mean opinion scores from human evaluators confirm the automated metrics: the fine-tuned model was rated as more natural and emotionally appropriate in 78% of A/B comparisons against the base model.
Introduction
Voice-to-voice AI systems for customer service face a fundamental tension: they must maintain consistent persona behaviour while adapting to the unpredictable dynamics of real human conversation. A model playing the role of a frustrated customer must stay in character when the human apologises or offers compensation — it cannot suddenly become cooperative or forget its complaint. Baseline large language models, even those fine-tuned for instruction following, frequently exhibit role-inversion hallucinations where the model slips out of character, adopts the wrong perspective, or becomes a generic helpful assistant. PersonaPlex-7b-v1, released by NVIDIA in early 2025, represents the state of the art in persona-conditioned dialogue generation. It was trained to adopt specified personality traits and maintain them across conversations. However, in our initial testing for customer-service role-play scenarios — specifically, coffee shop interactions where the model plays a customer with a complaint — we observed three recurring failure modes: role inversion (the model starts helping the 'customer' instead of being the customer), emotional flatness (the model describes frustration without sounding frustrated), and task abandonment (the model drifts away from the complaint scenario into generic small talk). This paper presents our approach to addressing these failures through domain-specific fine-tuning. Our contributions are: (1) a curated dataset of 200 synthetic coffee shop customer conversations covering complaint scenarios, order issues, and service recovery interactions, (2) a semantic-weighted loss function that differentially penalises errors based on their impact on persona consistency, (3) a voice prompt injection mechanism that maintains persona stability across turn boundaries, and (4) a comprehensive evaluation using both LLM-as-judge automated metrics and human mean opinion scores.
Related Work
Persona-conditioned dialogue generation has advanced rapidly since the release of PersonaChat and the subsequent development of models like DialoGPT, BlenderBot, and LaMDA that can maintain stylistic consistency. Parameter-efficient fine-tuning methods — particularly LoRA and its variants QLoRA and AdaLoRA — have made it practical to adapt large models to domain-specific tasks on consumer hardware. Our work sits at the intersection of these two trends, applying parameter-efficient fine-tuning to persona-conditioned models for a specific, high-fidelity customer service use case. In the voice domain, systems like ChatterboxTTS, VALL-E, and RALL-E have demonstrated that emotional expressiveness can be synthesised from text. However, these systems operate at the text-to-speech layer and do not address the upstream problem of generating emotionally appropriate dialogue text. Our voice prompt injection approach bridges this gap by conditioning the language model itself on voice-derived embeddings, creating a tighter coupling between the persona's emotional state and its linguistic output.
Dataset Construction
We generated 200 synthetic coffee shop customer conversations using GPT-4o with carefully structured prompts specifying customer persona (frustration level, complaint type, desired resolution), barista persona (experience level, store policy constraints), and scenario parameters (time of day, queue pressure, drink complexity). Each conversation spans 8-14 turns and includes natural phenomena such as interruptions, clarifications, and topic drift. Conversations were manually reviewed for realism by two annotators with customer service experience. Low-quality conversations (unrealistic turn-taking, implausible resolutions, flat emotional arcs) were discarded and regenerated. The final dataset of 200 conversations was split 80/10/10 into training, validation, and test sets. Each conversation was then processed through ChatterboxTTS to generate audio for the customer persona turns, producing 1,847 labelled audio-text pairs for training.
Methodology
We applied LoRA fine-tuning to all attention projection matrices in PersonaPlex-7b-v1 using a rank of 16 and alpha of 32, training for 3 epochs with a cosine learning rate schedule peaking at 2e-4. The key innovation is our semantic-weighted loss function. Standard cross-entropy loss treats all token prediction errors equally. Our loss function assigns weights to tokens based on their semantic role in maintaining persona: tokens that signal role identity ('I want to complain about...' vs 'I can help you with...') receive 3x weight, tokens that convey emotional valence (frustration markers, satisfaction signals) receive 2x weight, and filler tokens receive standard weight. Weights are determined by a lightweight classifier trained on 50 hand-labelled examples. The voice prompt injection mechanism works as follows: for each conversation turn, we extract a speaker embedding from the ChatterboxTTS audio using a pretrained speaker verification model. This embedding is projected through a learned linear layer and prepended to the LLM's input sequence as a soft prompt, effectively telling the model 'speak as this person.' During inference, we can use embeddings from held-out audio clips to prompt the model to adopt specific emotional tones without further fine-tuning.
Evaluation
We evaluated four model configurations: (1) Base PersonaPlex-7b with no task-specific prompting, (2) Context-Only with a system prompt describing the customer persona but no fine-tuning, (3) Voice-Only with prompt injection but using standard cross-entropy loss, and (4) our full approach combining LoRA fine-tuning, semantic-weighted loss, and voice prompt injection. LLM-as-judge evaluation used Claude Opus 4 to rate 200 held-out test conversations on three dimensions: emotional accuracy (1-5 scale), role consistency (1-5 scale), and task completion (binary: did the customer persona achieve or abandon the scenario objective?). Our full approach achieved mean scores of 4.2 (emotional accuracy), 4.4 (role consistency), and 94% task completion, compared to the base model's 2.1, 1.8, and 47% respectively. The improvement from the semantic-weighted loss alone (Config 3 vs Config 4) was +0.7 on emotional accuracy and +12% on task completion, confirming the value of differential token weighting. Human evaluation with 5 raters on 50 conversation pairs (Base vs Full) showed our model preferred in 78% of comparisons for naturalness and 82% for emotional appropriateness. Inter-rater agreement was moderate (Fleiss' kappa = 0.54 for naturalness, 0.61 for emotional appropriateness).
Discussion
The large gap between the base model's task completion rate (47%) and our fine-tuned model (94%) underscores how poorly general-purpose persona models perform on specific, high-fidelity role-play scenarios without domain adaptation. Even the system prompt alone (Context-Only) improved task completion to 68%, suggesting that a significant portion of the failure comes from the model not understanding what is expected of it in the scenario. But the remaining gap from 68% to 94% required both the targeted training signal from the semantic-weighted loss and the identity grounding from voice prompt injection. We were surprised by the strength of the voice prompt injection effect. Embeddings from held-out audio clips — voices the model had never heard during training — still produced consistent persona behaviour. This suggests the model learned a general mapping from speaker embedding space to persona-relevant linguistic features, rather than memorising specific voice-persona pairs. This has promising implications for zero-shot persona adoption, where a new voice profile could be used to drive persona behaviour without any additional training.
Limitations
This study has several limitations. The dataset is domain-specific (coffee shop customer complaints) and relatively small (200 conversations). Generalisation to other customer service domains — retail returns, call centre inquiries, hospitality bookings — requires further validation. The semantic-weighted loss relies on a classifier trained on hand-labelled data, which limits scalability and introduces potential annotator bias. Human evaluation involved only 5 raters, all of whom were team members, introducing potential confirmation bias. Most importantly, we evaluated persona fidelity and task completion but did not assess fairness or demographic bias in the model's behaviour. A customer persona model that performs differently based on the perceived demographic characteristics of the human it is interacting with — through accent, speech patterns, or named entities — could introduce or amplify bias in hiring assessments that use the model. This is an active area of investigation for our team and a prerequisite for production deployment.
Conclusion
Domain-specific fine-tuning of PersonaPlex-7b using LoRA, semantic-weighted loss, and voice prompt injection substantially improves persona consistency and task adherence for customer service role-play scenarios. The techniques we introduce — particularly the coupling of voice embeddings with language model conditioning — are general and applicable to other persona-driven dialogue tasks. As voice-to-voice AI systems are increasingly deployed in high-stakes contexts like hiring assessments, the robustness of these methods must be validated against fairness and bias criteria, not just performance metrics. We are continuing this work with larger, more diverse datasets and independent fairness auditing.