Competitive Research Update: Bigger database, AI-powered algorithm, more accurate metrics
Driving better SEO strategies requires reliable access to data and insights. At SE Ranking, we prioritize two key elements from this statement: high-quality data and your trust. That’s why we’ve made significant upgrades to our Competitive Research Tool, resulting in:
- 30% more keywords in our Competitive Research databases compared to 2023; we significantly expanded datasets for Germany, Spain, Brazil, France, Italy, Australia, Canada, New Zealand, Austria, and Switzerland.
- 29% more accurate organic traffic estimations due to updated formula and a new AI-powered algorithm.
- More accurate metrics, including Domain and Page Trust, backlink and referring domain count, and keyword search volume and difficulty.
This article explores SE Ranking’s enhancements and how they impact your SEO. Let’s start with our Competitive Research keyword database update.
Expanded keyword database across major markets
Our vast keyword database is the core component of our Competitive Research Tool. Combine our huge database with powerful algorithms and you get a tool that boasts exceptional competitive intelligence in terms of accuracy.
Moreover, our broader range of keywords opens up space for more targeted optimization strategies. Create content that’s highly relevant to users and addresses their exact search intents and queries. This leads to better user engagement and satisfaction, which is why expanding our keywords database is one of our major focuses.
Our efforts here were successful!
Over the past year, we added 350M keywords to our Competitive Research databases across major markets. Overall, our datasets increased by 30% with the most rapid growth in Q1 2024. The countries we focused on during the past quarter included Germany, Spain, Brazil, France, Italy, Australia, Canada, New Zealand, Austria, and Switzerland. In total, we added 192M search queries to our tool.
The database expansion resulted in impressive growth among these countries:
- Germany: 39.7%
- Spain: 17.9%
- Brazil: 84.0%
- France: 45.7%
- Italy: 44.5%
- Australia: 84.2%
- Canada: 103.3%
- New Zealand: 129.8%
- Austria: 192.8%
- Switzerland: 100.1%
In the graph below, you can see the new total number of keywords by country. The highlighted amount shows newly added search terms:
Prior to this update, our Competitive Research tool’s keyword database was already substantially larger than other similar tools on the market. We widened the gap even further with this expansion. Our Competitive Research keyword database is now nearly two times bigger than keyword databases in Ahrefs’ Organic Search Traffic index and Semrush’s Domain Analytics report.
New formula and AI-powered algorithm for organic traffic
In SE Ranking, organic traffic refers to the number of clicks a website receives through organic search results within a set period of time. While it is an estimated amount, our new approach ensures that the data you get is more precise than it used to be.
Having more precise organic traffic information allows you to more accurately gauge the effectiveness of your SEO strategies. It enables you to identify areas for improvement, allocate budgets, and make informed decisions.
Earlier organic traffic estimation methods focused namely on analyzing keyword rankings, monthly search volume, and CTR. But we improved our method of extracting the latter two and integrated AI into our traffic forecasting procedures.
Improved search volume and CTR
Search volume is one of the most critical components for calculating possible organic traffic. By factoring in accurate search volumes for each keyword, you’ll get a more realistic traffic potential estimation.
Once we’ve tweaked our algorithm for calculating search volume and added such factors as seasonality and keyword popularity, the ungrouping process has become more precise. You can see this in the screenshots below.
All of the tested keywords have different search volume values. For comparison, Google Keyword Planner shows you all these same keywords grouped into one cluster and with an average search volume range for the cluster, and not for each individual keyword.
With grouped keywords and search volume ranges, it’s challenging to determine the exact level of demand for a particular keyword. Moreover, it affects keyword research and organic traffic estimation precision.
As for CTR calculation, previously, it only included the position of website URLs in the search results, as higher SERP rankings typically lead to increased organic traffic.
Our CTR coefficient now takes into account the following additional factors:
- SERP features: Modern SERPs are no longer limited to traditional listings. They often include featured snippets, knowledge graphs, images, videos, and more, that can get all the clicks, which would have otherwise gone to the blue links. In addition, SGE’s (Search Generative Experience) introduction has shaken things up even more, as AI-generated snippets will soon occupy most of the SERP’s space.
- Branded keywords: Websites found through keywords that are directly associated with the business’s brand or name tend to get the lion’s share of organic clicks.
- User intent: Our neural network model monitors SERPs for various keywords and carefully examines the features they activate to identify search intent. If users spot something on the SERP (title and description with the right keywords or feature that aligns with their intent) that makes them think your page might be a good fit, they’re more likely to click and check it out, giving your CTR a nice boost.
Important!
Remember that our calculation of organic traffic relies heavily on the keywords that are currently available in our databases. Having a limited number of terms for a specific topic or niche can result in incomplete or inaccurate organic traffic predictions. If this is your case and you think that the forecast isn’t good enough, you can add your topic-specific keywords through our Expand Database feature. We’ll then recalculate organic traffic, and next month you’ll get more precise data for the analyzed site and niche.
Based on the factors outlined above, SE Ranking initially computes the predicted traffic for each website’s URLs and then aggregates the data to get the expected organic traffic for the entire domain. This cumulative approach enables us to understand the estimated monthly search traffic that any website is likely to receive.
New algorithm’s AI component
SE Ranking leverages machine learning to ensure our estimation processes remain dynamic and capable of adapting to constantly changing variables.
Our machine learning methods allow us to more accurately calculate the CTR by analyzing variable parameters, which include SERP features, position fluctuations, and search intent. When these parameters change, the outcome of the calculations changes as well.
Our algorithm continuously tracks, captures, and learns from these changes to ensure that your organic traffic estimations remain up-to-date and reliable.
So, how accurate has our organic traffic data become?
After introducing the algorithm, we conducted an experiment and compared the data produced by our old formula, the stats generated by our updated calculation method, and information from the Google Search Console.
Let’s look at the results we’ve got after analyzing the customer service software niche. We’ll take the helpcrunch.com website as the object of our test.
The predicted amount of organic traffic calculated through our old formula was 3.2K.
After our algorithm started to factor in more accurate search volume and CTR, the organic traffic value changed to 2.5K.
If we compare this estimation with the data from the connected GSC, we’ll see that they completely match.
Although we (yet) can’t guarantee a 100% match in all cases due to possible deviations from GSC data, our results for helpcrunch.com demonstrated a complete alignment. Previously, our variance from GSC data was 81.8%, but thanks to the new algorithm, we managed to reduce it by 23.7% to 58.1%.
The graphs below show how the deviation changed. The vertical axis represents the percentage of deviation from the GSC data (0-30%, 30-60%, 60-90%, etc). The horizontal axis shows the analyzed websites. As you can see, the number of cases where our data differs from the GSC by more than 100% has drastically decreased. In most cases, our traffic forecast has become closer to the GSC, as shown by the first three bars with a median deviation of 58.1%, marked with a red line in the second graph.
While a deviation of 58.1% may seem significant, it depends on the market niche and website type. For some cases, the deviation could be between 0 and 30%, as you can see in the first bar of the chart.
Additionally, it’s important to consider the differences in approaches. For instance, SEMrush deviates from GSC by 68.36%, while Ahrefs has a median deviation of 49.52%. However, Ahrefs experimented with a sample size of 1,635 websites, whereas SE Ranking analyzed data from a much larger sample of 7,675 sites.
This extensive dataset allows for a more accurate calculation of average deviation values, resulting in a more comprehensive assessment of organic traffic algorithm accuracy.
More accurate metrics in Competitive Research Tool
In addition to higher precision search volume values, you can now get more accurate data on backlinks, referring domains, Domain and Page Trust, and keyword difficulty scores. This is all possible thanks to the following enhancements we made to our backlink database:
- We’ve increased the frequency of backlink status updates, verifying the status of 48% more backlinks within the last 90 days.
- We have improved how quickly we identify backlinks and referring domains for new websites. Compared to Ahrefs, we detect 9% more backlinks and 11% more referring domains for websites younger than 6 months.
- Overall, we locate 5% more total backlinks and 17% more referring domains than before.
With such powerful backlink intelligence, we can calculate Domain and Page Trust scores and keyword difficulty with higher precision. Since these metrics are linked to website authority and SERP competitiveness, you can now more accurately assess how likely your site or pages are in ranking at the top of search results.
P.S.
At SE Ranking, we believe the sky’s the limit when it comes to upgrading our tool and providing you with the best SEO insights on the market.
Stay tuned as we continue to expand our databases, integrate new technologies, and push the boundaries of SEO analysis. For instance, we’ll soon release our new SGE tracking feature to prepare you for Google’s upcoming AI-powered search experience. We’re also upgrading our AI-based Content Marketing tool so you can generate entire articles in just one click.
Exciting updates await you, so keep your eyes peeled!
