Explore patterns, outliers, and seasonality in any topic using Python and Google Trends

Introduction

Google Trends is a trusted tool that analyzes the popularity of some of the top search queries. For the year 2020, the top trending search queries globally were “Coronavirus” and “Election Results”. The data from Google Trends can provide extremely valuable insights for many different use cases.  It can be used to do keyword research when producing new content, determining seasonal trends to help decide when to launch new products, figuring out what topics were just a short-lived-fad and are no longer trending (whipped coffee and Tiger King perhaps) and so much more. 

Google holds a lot of powerful data, but only a fragment of the data is shown in the Google Trends interface. To paint a bigger picture of a specific topic of interest, which requires comparing more than just the allowed 5 keywords,  you would need to manually grab data for potentially hundreds or thousands of individual queries. This is a very tedious and inefficient task. Here is where being able to use an API to automate and pull data for a customized set of keywords can be very powerful. It allows for keywords to be grouped together forming higher-level topics of interest, and the analysis and comparison of trends across these groupings, rather than just looking at individual terms and not getting a full and representative picture of the overall trend.

The Power of Automation

Here at Ayima, we build custom Python scripts, allowing our clients to pull data from a variety of sources at any desired level of aggregation. In order to efficiently pull the required data to draw meaningful insights, we built a script that requests Google Trends insights data for a desired set of keywords and relevant topic of interest, and then visualize these search trends.

How the script works

Inputs

The script takes in a CSV of hundreds of keywords and their groupings as an input (example below). A user can also specify filters like the timeframe (e.g. 5 years) and granularity (e.g. weekly) for which they would like the data.

Figure 1. Example input table

Pulling the data

The process of pulling the interest levels via a Google Trends API is automated by iterating through and making requests to the API for each keyword in the input table. It then populates the table with the returned interest data.

Figure 2. Example populated output data table

Processing and visualizing the data

Once the data has been pulled for every keyword, it then averages the overall interest level across all keywords in each grouping. This results in a table with an interest level for each of the overall topics of interest, which can then be visualized to compare and explore patterns in and between each topic.

Figure 3. Example processed data table

To visualize and make sense of this data, we use Matplotlib, a simple Python data visualization library, to output several time series charts. We will be going through some of these visualizations below!

Exploring A Relevant Trending Topic: COVID-19

There are many reasons why you may want to closely examine google trends data. For example in eCommerce, seasonality is very important and will help you decide how and when to market certain products. Trends can help you predict user behaviour and optimize landing pages, create new content ahead of these shifts, and prepare your content roadmap ahead of the new year. 

For the healthcare industry, COVID-19 changed everything. Not only did it affect everyone’s livelihoods, it also affected search. Were people searching about other illnesses more during Covid or less because Covid was the only topic on everyone’s mind? To answer this question, we explore different trends related to illnesses and symptoms to see how current events (in this case, COVID-19) affects interest in other related topics. The categories of keywords chosen are:

• Covid
• Allergy
• Cold
• Flu
• Headache
• Pain

Within each category, the data presented represents an aggregate of 50 of the highest search volume keywords related to the category. We don’t just want to know the trend of a specific keyword like “allergy”, but also its related keywords such as “allergy symptoms” and “seasonal allergies”. Comparing how these keywords trend seasonally as well as relative to each other provides the most insight on the variation in trend between these topics. 

The first graph below shows the trend for our categories over a 5 year period. It’s easy to see that once Covid hit the United States in March 2020, we see a huge spike in interest for Covid related keywords, no surprise. The greater interest lies in how other symptoms and illnesses were impacted by Covid. Were people searching about other illnesses more during Covid or less? 

Figure 4. US keyword trends over 5 years

This next graph explores exactly that. Here we take a look at the same date range, the same categories, and the same keywords, but removing Covid and adding 2 additional topics; depression and anxiety. What was discovered is that almost every category has a spike in interest at the same time as Covid with a couple outliers. 

Figure 5. US keyword trends over 5 years (without Covid)

The outliers being pain and anxiety. Although anxiety and pain do not have a significant spike in interest in March, these keywords have been consistently trending upwards for the past 5 years, until March 2020, when everything else seemed to gain interest. This tells us that March 2020 is probably not the best time to be launching a new pain relief product. So what would you market or create content for instead? The graph below takes a look at each of the individual categories and their interests over 5 years. It’s easy to see that headaches and allergy have hit their all time highs in interest around March 2020.

Figure 6. US keyword trends over 5 years – Individual charts

The overall trend here is that the increased interest in Covid related terms also led to an increased interest in other illness and symptom related search terms. Another common reason for increased interests a lot of the time is seasonality.  Seasonality is a popular topic in marketing. It’s no surprise that consistently every year, certain months will have higher search volumes. It definitely gives you an advantage if your business is capturing this wider audience during peak seasons. Discovering these seasons, the patterns, and general trending topics can make a big difference and help your business succeed. Please feel free to reach out to us at Ayima if you would like to explore how we can help enable you to make data-driven decisions and increase your user engagement!