Reinforcement Learning for Online Optimization of Banner Format and Delivery

Reinforcement Learning for Online Optimization of Banner Format and Delivery

Benoit Baccot (Sopra Group, France & University of Toulouse, France), Romulus Grigoras (University of Toulouse, France & Devatics, France) and Vincent Charvillat (University of Toulouse, France)
Copyright: © 2011 |Pages: 19
DOI: 10.4018/978-1-60960-189-8.ch002


Results, showing the power and the efficiency of the two models to solve our problems, are also given. By comparing to a “ground truth” acquired by observing user browsing session on a test site, we conclude that our models are able to determine optimal advertising policies concerning banner formats and delivery.
Chapter Preview


Today, online advertising has grown into one of the most successful advertising channels, since browsing the web has become a daily activity for a majority of users. Website owners call on the experience of marketers or online advertising agencies in order to design, produce and deploy ad campaigns (see for example (Marketing Sherpa, 2008) or (McCoy et al., 2007) if you are interested in this process).

Among the different types of online advertising (emails, games, etc.), we are particularly interested in contextual ads using rich media banners. In fact, on a commercial website, contextual ads are used to drive users and transform their navigation into a transaction. Traditionally, the banner's content has been presented in the form of text and hyperlinks. Recent studies such as (Rosenkrans, 2009) have shown the benefits of using rich media for displaying motion and exploit sensory information such as video, audio, animation etc. Rich media content is considered more attractive since it can grab users' attention easily and can also leave stronger memories (Mei et al., 2007). Rich media naturally fights the banner blindness problem, when users tend to completely ignore banners. These are some of the reasons that made rich media advertising very popular.

Unfortunately, some sites make excessive use of it, leading to the commonly called ad overload problem. The overabundance of banners or the poor targeting of ad campaigns make sometimes the user navigation on a web site difficult or unenjoyable. Moreover, users tend to learn (by reinforcement...) how to avoid clicking on banners, since they can lead to unwanted content, which is clearly contrary to the aim of the ad campaign.

Once the various banners of an ad campaign are produced, a legitimate question arises (Baccot et al., 2009): among various options, and for the same banner content, what is the optimal banner format and delivery policy?

In this chapter, we intentionally put aside the banner content issue and consider it (as other authors, like Hauser et al. in 2009) as a separate question. Thus, the problem is how to take into account three classical dimensions of a banner seen as a hypermedia document:

  • the logical and spatial layout. The logical layout includes the different elements (and their links) that can be inserted in a banner (e.g. an image, or an image with a caption, a video etc.) whereas the spatial layout gives the way these elements are presented (e.g. a caption above or below the image).

  • the level of interactivity of banners. Basic or more complex banners may be available, enabling users to click, scroll, type etc.

  • the timing of delivery: the instants when banners are added. Adding them at very specific times (e.g. when the user level of interest falls) can increase their effectiveness and therefore their click-through rate (CTR).

A variety of options for designing and delivering the banners are available. Obviously, the various possibilities do not have the same impact, therefore what should we (or should we not) do, what are the criteria that allow us to choose?

The remaining of this chapter is organized in three sections. The first one (section 2) introduces the three banner problems we will deal with: the right format for a banner, the right time to display it and the right sequence of banners. While presenting these problems, we will introduce step by step all the ingredients that enable us to work in the reinforcement learning framework. In the next section (section 3), we formalize these problems and present more precisely the reinforcement learning framework. Two stochastic models that will help to solve the previous problems are also detailed. The third section (section 4) presents results obtained by solving the banner problems using reinforcement learning. The results prove the strength and the efficiency of the models. Finally, a discussion about the benefits of reinforcement learning is conducted. The chapter ends with a conclusion and some perspectives.


1,2,3… Banners Problems

Among the various possible use cases related to banner optimization of format and delivery, we choose to address three problems. These problems are important to solve, since they handle essential aspects of the banners: the right format, the right time to insert and the right sequence of banners.

Complete Chapter List

Search this Book: