Quantcast
return Home

Measuring chatbots, and what makes for success

Botanalytics Ilker Koksal ()

While some industry sectors are only beginning to consider the potential of chatbots, others are long past the point. Their conversations are shifting to, among other things, figuring out how to better measure engagement and retention of users within their bot environments.

Retention, in particular, is currently a particular challenge, says Ilker Köksal, co-founder and CEO of Botanalytics. This challenge is not only for “bot makers just starting out but also those with well over a million users”. 

Botanalytics is a conversational and engagement analytics platform for chatbots. “We are to bots today what web analytics were to websites a decade or so ago,” explains Ilker. The San Francisco-based company was set up only in early 2016, but is already well placed to identify, analyse and extrapolate chatbot user insights and trends across a number of sectors, in relative depth. “We launched shortly before Facebook announced its bots at the f8 conference in April 2016. After the f8 announcement, things speeded up everywhere,” he says. 

Shortly after f8, Botanalytics received funding from 500startups and now “has several enterprise [clients] with hundreds of bots” on their platform, according to Ilker. This includes areas such celebrity, education, media, finance, travel, utilities, and more, with bot examples including for Grey Group, Harper Collins Publishers, CNN, Domino’s Pizza and singer Robbie Williams’ bot.

The team is now 12-strong, focused on back-end development, product design, marketing and business development and account management. Their key focus with measurement and analyses is on enhancing human-bot communication. This is done mainly through:

● Filtering conversations for bot makers, based on engagement steps and tracking the details of conversations including rich media (video, audio, location and images);

● Measuring engagement and retention. Standards vary depending on the nature of the bot, for example with average session length for entertainment bots tending to be longer than for on-demand bots, such as for Uber. “Bot makers should think about their bots within a specific category to compare their engagement and retention rates,” he advises. 

● Identifying bottlenecks in interactions. Bots sometimes do not understand users due to varying Natural Language Processing (NLP) standards. “When people say, for example, ‘I would like to call a taxi’ or ‘could you get me a cab?’ the bot should understand these commands as a unified message.” Currently there are still limitations in this area and in order to improve their bots, bot makers need deep analysis of conversations to understand where “bottlenecks in conversations occur, identify patterns and design and iterate their bots accordingly.”

Botanalytics conversations ()

Through its multi-sector analytics and data, Botanalytics have unique insight into what currently works in the bot economy and what doesn’t, and where usage trends are pointing. In Ilker’s words, “We are creating, helping and educating the market with our insights.”

● Subscribe to our weekly FIPP World newsletter to receive more stories such as these directly to your inbox.

Having launched experimental bots, several of their clients (and prospective clients) are now at the point where they want to better know and deeper engage their audiences, with improved retention a particular objective. “This is where we come in,” explains Ilker, “to measure their bot performance in terms of conversation and user metrics, for them to be able to iterate and better design their bots on the back of robust data and automated analyses, adding to that AI learning that serve to make their bots more intelligent.”

Currently, he says, there are broadly two types of bots in the market:

● Designed with good UIs (user interfaces). These fulfil current market needs, but is limited in functionality compared to the second type.

● Smart NLPs (natural language processing), adapting to consumer expectations as they evolve. “Bots in banking are moving rapidly towards the NLP side, for example for use in their FAQ sections.”

Some of the most interesting chatbot developments for him are:

● Customer support in the banking sector, particularly around FAQs as mentioned above. An example is American Express on FB Messenger. 

● New acquisition channels for brands. For example Burberry

● Celebrity bots aimed at increasing fan engagement. For example singer Robbie Williams

● Offline to online use with messenger apps, where, for example pre-ordering a pizza, or booking a hotel or flight or ordering a cab are shifting from to chatbots (something Ilker says is poised to impact on general online habits soon). For example Domino’s Pizza.

Botanalytics examples 1 ()

According to Ilker a good chatbot has the following characteristics:

● It has extended capabilities in terms of comprehension. A good NLP score is a crucial metric you want to measure, getting it as high as possible.

● Sentiment is another, which is a vital consideration says Ilker. “When people are trying to communicate with bots, you’d want to know whether they are sad, happy, angry, etc. Bots should be able to detect sentiments, in order to ‘know’ the emotional context of conversation. People would want to manage their feelings on bots, and in order to do that bots need to be designed to understand the user sentiment.” 

● Conversational flows that are designed according to human behaviour patterns.

“All bot makers should be focused on these areas when designing their bots,” he says, adding that, like with the web, “it is about analysing your data and knowing your user, with the rest flowing from there”.

Get the FIPP Innovation World Report 2017/18 which dedicates a whole chapter to bots

• We also have a long-read on developing chatbots strategy, here

In terms of useful user metrics, he believes the following are most important at this point:

● Average conversation/user

● Average session length

● Conversation steps/user

● Retention rates

● Some correlation analyses in order to evaluate combinations of metrics

Botanalytics 3 ()

Botanalytics example 2 ()

His message for media companies is to get involved. “You have people who are only thinking about building bots now, but in reality many are already way beyond this point. They are thinking about things like how they can keep their audiences active, how they can be more useful, and so on”, allowing them to develop share of market and capture user time early on.

With the proliferation of chatbots people will start to bypass the normal web, getting their content from bots, and for publishers, this means “thinking about how their users can engage with their content without having (or wanting to) visit a website; what they get at different time intervals such as hourly, daily, weekly; what the notifications strategy is; and so on. And then start measuring the same things all others are looking at to see how effective your strategy is, to iterate and develop more value from there.”

He suggests retail, banking/finance and technology are good sector examples to consider for ideas, with brands such as Starbucks, Burberry, Bank of America, Capital One, Taco Bell, Estée Lauder, Sephora, Pizza Hut, American Express, and Whole Foods, and Transferwise some of his standouts.

More like this

Innovation 'how to' report gives the lowdown on chatbots

How to develop your chatbot strategy

John Wilpers showcases top trends included in new FIPP Innovation World Report

How chatbots are going to change the world and the media landscape

  • Millennials no more: The Week Junior and AwesomenessTV on engaging younger audiences While the industry continues its pursuit of ‘millennials’, the generation of media consumers coming up behind them is still to a large extent being overlooked. Here, Kerin O’Connor from The Week Junior, and Kelly Day from AwesomenessTV, give us their take on how publishers can better understand even younger audiences.  28th Jun 2017 MagWorld
  • Building superior brand experiences through a multi-channel approach Deepak Lamba, CEO of Worldwide Media (WWM) in India, explains how the business is embracing multi-platform media campaigns to boost the reach and impact of its brands – with exceptional results – and says WWM has its eyes set firmly on becoming India’s leading entertainment and lifestyle content creators for every available platform. 26th Jun 2017 MagWorld
  • Building a subscription-driven business from the ground up Geoff Ramsey, co-founder and chief innovation officer of eMarketer started his working life as a magician. Here he explains how eMarketer found its magic formula – how a traditional ad house was transformed into the research and data giant we know today, the tricks it deployed to survive some turbulent times, and how the business plans to continue coming up trumps in the future… 26th Jun 2017 MagWorld
  • Four industry experts share insights on innovating live events Live events are a rapidly growing revenue source for publishers, but as the field becomes ever more competitive event organisers need to be nimble, creative and innovative to remain on top of the game. Here, we combine the insights of four leading industry experts on developments within the sector. 25th Jun 2017 MagWorld
  • How Aller Media Finland unlocked the value of data-for-profit In less than three years, Aller Media Finland has become the leading company in Finland to harvest consumer data as an income asset. Sami Lahtinen, director, data and digital solutions, and Sini Kervinen, head of business operations, data and digital solutions, explain how the establishment of the Aller Refinery data unit managed to place data at the heart of their business’s growth. 19th Jun 2017 MagWorld
  • How Marie Claire is embracing a more diverse range of voices Focusing on social and political issues, giving readers current events news in context may sound more like a news magazine than a Marie Claire, but women’s magazines have been producing this kind of content since creation, according to Kate Lewis, Hearst Digital Media’s SVP/editorial director in the US. There is good reason why it is now more noticeable... 21st Jun 2017 MagWorld
  • Nine steps to rapidly transform your legacy teams No amount of tinkering at reorganisation will work. Tinkering has failed. New business models and new editorial models require new thinking, new roles, new workflows, and new organisational structures, writes INNOVATION Media Consulting senior director John Wilpers. 19th Jun 2017 Insight News
  • Reuters Digital News Report 2017 highlights lessons for publishers across multiple markets Growth in social media for news discovery is flattening out, ad-blocking on desktop has stalled, brand recall on social is low, voice-activated digital assistants are emerging as news platforms in some countries and regions. If the Reuters Institute Digital News Report 2017 makes one thing clear, it is that the digital revolution is full of contradictions and exceptions, for the simple reason that countries and regions across the globe started this revolution at different times and places and aren’t progressing at the same pace. 22nd Jun 2017 Insight News
  • Four industry experts share insights on innovating live events Live events are a rapidly growing revenue source for publishers, but as the field becomes ever more competitive event organisers need to be nimble, creative and innovative to remain on top of the game. Here, we combine the insights of four leading industry experts on developments within the sector. 25th Jun 2017 MagWorld

Video

Visit our Youtube channel

FIND OUT MORE

SUBSCRIBE

FIPP newsletters allow you to keep up with industry trends, research, training and events across the world

FIND OUT MORE

SHARE YOUR NEWS

Get global coverage of your launches, company news and innovations

FIND OUT MORE
Go to Full Site