Connect with us

Hi, what are you looking for?

Business

Live Events and Big Data: Handling Over 150,000 Searches on 30m+ Tweets

BrandWatch

Whether you’re an NFL fan or not, the yearly Super Bowl hype is always unavoidable – it’s not only the largest annual sporting event in the world, but also one of the most important advertising opportunities for major global brands.

Waiting to see each brands’ Super Bowl ad has become an international pastime, much like department store Christmas ads in the UK.

Of course, brands want to track and analyze the response to those ads to understand their impact.

Using platforms like Brandwatch, they monitor the conversation in real-time to understand the response and even, in some cases, modify what they’re showing at the next ad break in response to consumer feedback online.

This analysis helps not only understand how well the ad lands with audiences, but can also help marketers understand the return on the dollars spent through analysis of, for instance, purchase intent.

Tracking live events

Every year, Brandwatch collects millions of tweets about the event, with hundreds of our clients tracking the different aspects of the conversation that are most useful to them.

We know how important it is for our clients to be able to track that conversation in real-time, whether via their Analytics Dashboards or within large-scaled command centers and social hubs using Vizia.

Vizia Dmexco 2015 BlogheaderThat’s no mean feat and not something every platform can provide. In 2015, the volumes of data we were processing peaked at 5,000 tweets per second – that’s 300,000 tweets per minute.

This year, over 30 million tweets streamed into the Brandwatch platform during the time of the game – around double volumes on a normal day. Those volumes alone can be a challenge when it comes to processing, but it’s when we come to the Queries that it gets really interesting.

Our clients were running over 150,000 searches in Brandwatch to collect specific segments of that Twitter conversation that they were interested in – many of those searches were complex strings looking for, for example, a particular brand or advert, or even tracking purchase intent as a result of those ads.

IStock 000053254530 Large 1024x682

That means our platform was carrying out a complex matching algorithm in real-time to find tweets that match search terms 750 million times per second.

That kind of processing power doesn’t come easy – it’s the result of years of expertise and hard work to build an architecture that can handle such a huge task.

Testing, adapting, growing

We’ve been building our technology and hardware for nearly 10 years, continuously improving and updating it to handle ever more complex problems. This year, expecting even larger volumes than 2015, we put in additional measures to ensure the data would flow in smoothly without any lag. We knew that that much processing in real-time would need a lot of crawlers. Historically, that’s been a challenge due to physical limitations on the number of crawlers we can connect up to our database.

IStock 000080742571 Large 1024x768 300x225

This year, we made several changes to circumvent this limitation and subsequently were able to increase the number of crawlers running for the event from 60 to 180, trebling our capacity.

Once we’d done this, we then modelled the data from previous Super Bowl events to test how much impact this change would have – with the most important thing being avoiding a lag in data coming in.

We were confident from this testing that our trebled capacity would allow us to handle volumes similar to previous years, plus a bit more.

Of course, we were unable to fully predict the actual volumes for this year in advance, so we had a dedicated team of senior developers working through the night alongside our customer support team just in case there were any issues.

IStock 000032338466 Large 1024x682

As it turned out, the 2016 volumes we brought into our platform were actually a little lower than last year (perhaps due to a lack of a badly dancing #leftshark?) but still very large at double the normal daily volume.

All our hard work paid off and our users were able to track the conversation in real-time without any disruption.

As more and more brands want to track huge events like these in real-time, this dedication to speed and performance is essential and a constant focus for us.

We’re continuously working on further improvements, and are currently exploring the ability to automatically use the Cloud when we experience massive peaks in data volume. Stay tuned.

 

This article was cross-posted with permission from www.brandwatch.com

Click to comment

Leave a Reply

Your email address will not be published. Required fields are marked *

fifteen − eight =

Most Popular

Advertisement

You May Also Like

Opinion

We’ve all heard of the 50/30/20 rule when it comes to budgeting our expenses. It is one of the most popular ways to manage...

News

A photonic quantum computer chip did nine thousand years’ worth of work in 36 microseconds. Now that’s fast. And I can’t even finish my...

Health

Since the pandemic started over a year ago, signs of burnout in employees have increased exponentially. A little over two-thirds of employees are suffering...

Entertainment

Kanye West is releasing Donda 2 exclusively on the Stem Player, a device created by Kanye in collaboration with Kano Computing. The device includes...

Copyright © 2020-2022 Bold TV. Bold TV is owned and operated by the Foundation for American Content and Entertainment, a 501(c)(3) non-profit organization.