The source code of this notebook can be found in the Literal AI Github Cookbooks.

This notebook shows you how to retrieve your Customer Support Conversations from Literal and evaluate user satisfaction on this conversational data.

We leverage OpenAI’s API to score the threads with a custom prompt template, and finally plot user satisfaction.

Import the Literal AI SDK

Create Threads (Optional)

If you have conversational data – Thread objects in Literal AI, skip to View Thread details.

Otherwise, run the next cell to populate a few fictional conversations.

View Thread details

The below screen capture shows how the first thread appears in Literal AI, with the user and the assistant going back and forth on the issue at hand.

You should see an equivalent flow if you have actual conversational data in your project.

Thread Details

Thread Details

Get Threads

We are now set to get the threads to score.

From the Threads table, you can filter on the threads with the to_score tag, and then click “Export”:

Threads Table

Threads Table

The Export Threads window pops up and provides the exact filters to pass to the get_threads API.
Copy and paste below to fetch your selected threads via the SDK.

Export Threads

Export Threads

With our fictional threads, we will simply filter down on the to_score tag:

Create evaluation prompt template

Score threads

Define helper to build thread messages

Score using OpenAI

To ensure we get an integer user satisfaction score, we rely on Outlines to guarantee the generated output is in the 1 to 5 range.

Visualize User Satisfaction

Output visualization

Output visualization

Change Threads tags

Change the tags from to_score to scored.

And the threads we had to score show up as scored !

Scored Threads

Scored Threads