Delvinia - Technical Innovation of the Year

Company: Delvinia
Company Description: Delvinia is a research technology and consumer data collection company that is transforming how data is collected, protected and used to underpin business decision-making at every level. It has spawned a successful portfolio of products and services including the research automation platform Methodify, the online consumer panel AskingCanadians, and CRIS, which automates qualitative research.
Nomination Category: Company / Organization Categories
Nomination Sub Category: Technical Innovation of the Year - At Organizations With Up to 1,000 Employees

Nomination Title: CRIS by Delvinia

CRIS by Delvinia is a virtual moderator that uses AI to conduct text-based interviews at scale – three times faster and at a third of the cost of traditional qualitative methods. It was acquired by Delvinia at the end of 2019, and has continued to grow and develop in that time.

CRIS collects both qualitative and quantitative data on a secure web-based messaging platform. Participants chat with CRIS at a time and place that is convenient for them and enjoy a research experience that is more interesting and engaging than traditional static open-ends, and unmoderated bulletin boards.

CRIS can ask for and show anything that can be done over chat, including free text, pictures, links to websites and videos, emojis and prompted questions (which are used to collect quantitative data). The sentiment API from Google Cloud language is integrated into CRIS’s algorithm. This means that CRIS can gauge sentiment and respond appropriately. It also means that users get the sentiment score for their overall study and by question as part of their report.

Our challenge in the last 12 months has been to improve the follow up algorithm so that CRIS continues to evolve as a true qualitative moderator. To help us with this challenge we worked with Massively Inc., who has been with us from the beginning in developing CRIS.

To bring us closer to a real moderated experience, we have worked to improve the AI so that CRIS now:

1. Knows when to follow up by evaluating completeness of a response relative to responses from preceding interviews, with no intervention required

2. Follows up more specifically, to get detail on subjects that have the most relevance. In brief, it does this by:

-Generating keywords using the statistical measure Term Frequency-Inverse Document Frequency (TFIDF).
-Then an NLP tagging technique is employed to tag Parts-Of-Speech (POS), and process and extract Noun Phrases, known as chunks. If any of the keywords are contained in the noun chunk(s), the ‘chunk’ is added to a follow-up phrase, “e.g. can you elaborate more on…?” Because the keyword is ranked by score, we are able to select the appropriate noun chunk if more than one is selected.
-If no noun chunks are extracted, CRIS proceeds to follow up using the original decision tree logic.

The result is more specific follow ups that are relevant to the study’s objectives, enabling clients to gather more insightful data at scale.

What is unique about CRIS is that it replicates a true moderated experience. While there are other research chat bots out there, CRIS behaves as a moderator would, by establishing focus, engagement and empathy. This results in a more engaging experience for participants, and richer layered data for clients.

A discussion guide, much like what a moderator would use to conduct a traditional one on one interview, is programmed into CRIS and CRIS asks the questions. CRIS then follows up when the answer seems incomplete to get more information.

Unlike a customer experience chat bot, the challenge with a research chat bot is that each project covers different subject areas and objectives. Each project begins with a discussion guide that is designed for the unique needs of the situation. The exercise is to get feedback from a target audience on a particular topic in order to understand their perspective, often with the goal to make a business decision. This means that CRIS has no previous corpus of knowledge to draw on when beginning the interviews.

In its original iteration, the trigger for CRIS to follow up was based on expected word count, which needed to be defined for each question in advance. In addition, the follow ups were generic (e.g. can I get more detail on that? What else can you tell me about that?). The upgrades completed over the past year bring a more rigorous AI-driven method to the follow up process, and better simulate a real-life moderator.

You can learn more about how CRIS and is innovating the qualitative research space here: https://www.delvinia.com/solutions/cris/

You can see more details about the recent upgrades here: https://www.delvinia.com/delvinia-advances-ai-for-virtual-moderator/

We also have several case studies available demonstrating the ways that CRIS can be used to improve the research process for a variety of clients:

https://www.delvinia.com/case-study/cris/
https://www.delvinia.com/case-study/government-of-canada/
https://www.delvinia.com/case-study/cris-latin-america/
https://www.delvinia.com/case-study/john-st/