What does the Robots.txt file mean for SEO?

What exactly is the robots.txt protocol?

Robots.txt protocol allows you to guide spiders on your website so they only crawl the pages you want them to crawl. Although this might seem contrary to the goals of SEO, preventing the spiders from exploring some of your content can actually be beneficial. For example, this might apply when you are building a new website, or if you have multiple versions of a page on your site and you do not want to get penalized for duplicate content.

The origins of the protocol can be traced back to the mid-1990s when early webmasters began to have concerns about what spiders visited their websites. The idea has grown to meet the needs of modern web designers and the current version will be accepted by the spiders sent from the major search engines.

Do I need robots.txt?

Since robots.txt tells search engine spiders not to crawl certain pages, some site owners might mistakenly think that this is a good way to keep certain information private. Unfortunately, although honest spiders, such as those from Google or Bing, will respect the protocol, there are plenty of malicious spiders that will not, and thus your information can still be stolen.

It is also possible for the information to end up indexed in other ways, such as another site linking to the content. If you have personal information, you should have much stronger security, such as through a firewall.

Where should robots.txt be located?

When using robots.txt, it is a good idea to be familiar with the basic symbols. A ‘*’ tells the spider that this command applies to all web crawlers. You can replace this with the name if a certain bot if you only want to prevent certain search engines from indexing a page. A ‘/’ will indicate that the rule applies to all the pages on a particular site. This symbol can be replaced with the name of a particular directory or page.

For example, if you want to tell all web crawlers to ignore all the pages on your site, then your command will look like:

If you want to tell all web crawlers to ignore a particular directory, the command will look like:

If you want to tell all web crawlers to ignore certain pages, you will write:

If you wanted to block only a particular bot from your site, then you modify the user-agent and write:

Finally, if you want to allow only one web crawler and block the rest, you will write:

The first part of the command tells Google it can explore the whole site, since there is nothing next to the disallow section. The second part of the command speaks to the rest of the bots, telling them to stay away.

It is a good idea to keep the following guidelines in mind:

• You always have to write robots.txt in lower case
• The commands must be located in the top-level directory of the web server
• You can only have one ‘disallow’ for each URL on the site
• Subdomains that have a common root domain need different robots.txt files
• Do not use spaces to separate the parameters of the command

Using the protocol is relatively straightforward, but simple typos can easily derail the commands. After you write your protocal, make sure to use the Google testing tool to see how the search engine giant will interpret your protocol.

When used wisely, robots.txt can be a valuable resource and help you control what and when search engine spiders explore your offerings.

<< Back to the SEO Glossary main page

What was the Phantom III Update?

Google Phantom III update supposedly took place in 2015 and addressed the issue of low-quality content. It was never officially announced as an update by Google, but since the update is largely based upon the conjecture of the websites who noticed the differences in rankings and traffic, this update has been referred to as the Phantom III update.

Was Phantom III a real update?

Beginning around November 19, 2015, there were a number of reports that suggested that Google had done a search ranking update. Although the reports are unconfirmed by Google that does not necessarily mean that nothing happened. Since the update is largely based upon the conjecture of the websites who noticed the differences in rankings and traffic, this update has been referred to as the Phantom III update.

What were the past phantom updates?

In April and May 2015, there was another update that Google confirmed about two weeks after the reports of changes had been made. This update has been termed the Phantom II update, or the Quality Update. The first Phantom occurred back in 2013. Both of these updates focused heavily on penalties for poor content and unnatural link profiles. Websites that used the same anchor text, sites that continually link back and forth to each other or otherwise built a link profile that was clearly unnatural, were all hit. Sites with duplicate or poor content, designs that are not user-friendly and sites that were riddled with technical problems also saw some damage.

What did the Google Phantom update target?

Like many other updates over the past few years, Phantom III was also interested in targeting low-quality content. The key difference between Panda and the Phantom III update, according to SEO Glenn Gabe, is that the Phantom III update seemed to focus more on the user experience. Sites that had popups or modal windows that were hard to click off as you scrolled through the content were often penalized. Sites that used the sidebars and other space around the main content to promote unorganized additional information, rather than clearly relevant opportunities, also saw penalties. Finally, websites that attempted to force visitors to click through numerous pages to view an entire article, just so that each page could load new ads, also saw penalties.

How do I avoid a Google SEO Phantom III update penalty?

1. Focus on the user experience. Make sure that you plan the layout of the site to help the user to better navigate and engage with your content easily, not make things more difficult for them.
2. Do not use excessive ads on the page.
3. Monitor your backlink profile to make sure that you are building quality, organic backlinks
4. Make it easy for people to physically read your content-- so remove challenging popups and do not make people click through multiple pages to read a new paragraph of your content.
5. Do a site audit regularly to find any potential technical problems so they can be rectified right away.

The Phantom III update definitely helps to keep site owners on their toes as they are forced to focus even more on quality and the user experience. Google has made it clear that they care more about the user experience than promoting particular brands, and site owners need to follow the same path.

What is the Google Panda SEO Update?

Google Panda update was introduced in 2011 and was designed to punish thin or poor content. The filter that came in the update was aimed at stopping poor content that had managed to rank highly for certain queries despite having little to offer readers. From 2011 through 2015, Google updated the filter periodically, which resulted in new sites being penalized. Beginning in January 2016, however, Google announced that Panda was now a core part of their algorithm. Although Google has confirmed that this does not mean the filter will be updated in real time, it is used to help measure the quality of a site when determining its rank.

How does Google Panda work?

Google Panda penalties occur when websites manage to rank highly despite thin or poor content that does not serve the end user. We have seen penalties occur for:

• Duplicate content
• Pages that have a poor-content-to-ad ratio
• Pages with excessively general information
• Content that offers little information

When we analyzed the Google Panda update 4.1 that occurred in the fall of 2014, we found that the update even impacted a number of well-known sites, such as:

• Independent.co.uk, likely for a poor content-to-ad ratio
• Answers.com, likely for generic content
• CheaperThanDirt.com, likely for thin, uninformative content

On the other hand, pages like NYTimes.com and OrganicGardening.com benefited from the update because they already focused on highly informative content with a low ad ratio.

How do I recover from a Panda penalty?

If you are hit with a Google Panda update penalty, you will notice a steady decline in traffic, followed by stabilization. It is important to note that if you are hit by Panda, it will impact your entire site. So, if you notice a drop across all of your keywords, it is likely that you were hit by Panda.

Google offers this traffic report as an example of a site that has been hit by Panda.

If you notice this drop here is how to recover.

1. Review your site for duplicate pages or extra pages that offer little to the user. Look for duplicate Title Tags as a potential indicator of duplicate content.
2. Eliminate or update these pages so they provide value for the reader
3. Use robots.txt on pages that are duplicate but needed on your page
4. Make sure your site offers a positive user experience, and does not contain excessive ads
5. When Google refreshes the algorithm, you should start to see your site’s rank improve.

What was the main purpose of the Panda update?

As you move forward with your website, you want to make sure that you produce content that the Panda filter will like and will not threaten your site in the future. Here is how to produce Panda-friendly content:

• Each page should have a purpose for the end-user, not just to rank for a particular keyword. You should have a particular buyer persona and a particular stage of the buyer’s journey in mind to help you produce the level of quality that your audience, and Google, want to see
• Regularly check for duplicate content
• Keep an eye on your ad ratio to make sure that your load time and your user experience are not hurt by excessive ads
• Use Google Webmaster tools to make sure the website is functioning optimally and providing the best possible user experience.

Since 2011, the Google Panda update has worked to provide users with high-quality content while minimizing the presence of websites that do not provide the answers the user sought. Regularly monitor your traffic to watch for any potential hits of this penalty and always focus on quality and the user experience when producing content moving forward.

Understanding Google RankBrain: The Ultimate Guide to AI and SEO

What is the RankBrain Algorithm?

RankBrain is a major part of the Google algorithm that uses artificial intelligence and machine learning to learn about users and how they respond to the search results, particularly on previously unseen queries. The program watches how people respond, learns the meaning behind the query and the information that people seek.

This technology was developed to help Google process the nearly 15% of queries entered each day that the search engine has never seen before. Rankbrain was launched in October 2015 and has since then risen to be the third most important ranking signal. Now, RankBrain is involved in most queries entered into Google.

Why is RankBrain important?

Although RankBrain was developed to focus on the completely unique queries seen daily, Google has indicated that this part of the algorithm is now a critical part of organic search. According to Google senior research scientist, Greg Corrado, taking away the program would be hinder the user experience so much that it would be about the equivalent of failing to index half of the pages on Wikipedia.

How do I optimize for RankBrain?

Although Google has not explained how exactly RankBrain impacts search, it is likely that it has a role in helping the algorithm understand content to create more accurate results. To keep your content ranking well, use the following steps:

1.    Focus on creating high-value content that explores topics in depth and answers customer questions

2.    Use schema markups to make sure that it is easy for the algorithm to understand your content. Although this does not currently seem to impact ranking, it is possible it will in the future if Google introduces more AI into its algorithm.

3.    Monitor your traffic and conversion rates to watch for dips or rises that can tell you how well your efforts are succeeding.

RankBrain was an exciting development because it introduced AI into the Google algorithm. It is likely that this trend will continue in the future, as the search engine works to make their algorithm smarter to maximize the user experience. Preparing now with user-friendly content and machine-friendly markups will ensure your content is ready for any algorithm changes.

What was the Mobile Update in April 2015?

What was the updated Google algorithm for mobile?

The mobile update was launched on April 21, 2015 by Google and targeted sites that were not mobile-friendly. Unlike most other updates, marketers and site owners had a warning several weeks in advance that the update was going to take place, giving them the opportunity to make sure their site was compliant before the update went live.

Under the new mobile update, sites that are not mobile-friendly will have their rankings penalized. This ensures that mobile users receive results that will work well with their devices, improving the user experience.

Complete Your Mobile SEO Strategy

Why was the mobile update important?

The mobile update was in response to the growing prevalence of mobile in the lives of users. The year 2015 actually also saw the number of searches on mobile pass the number of searches on desktop. Customers are also increasingly using mobile for everything from searches to shopping-- mobile is now an estimated 30 percent of all U.S. e-commerce.

How do I avoid a mobile-update penalty?

The key to avoiding a mobile-update penalty is ensuring that your site complies with Google’s guidelines for mobile sites. You can test how mobile-friendly your site is by using Google’s mobile-friendly test, which will tell you if you have a compliant design.

In general, these are the points you need to cover in a mobile website:

1. Have a mobile-friendly layout. This can be achieved with either a responsive design website, which will automatically configure itself to fit the size of the device it is on, or it can be done with designated mobile website.
2. Make sure that all of the videos on your site are compatible with mobile devices.
3. If you are using a designated mobile site, make sure that you do not have any faulty redirects and that every page on your desktop site corresponds with a mobile page.
4. Make sure your pages do not appear as 404s for users on mobile
5. Do not use interstitials or overlays that cover portions of the intended content, as they can be very challenging to get rid of on mobile devices.
6. If you use a mobile version of your website, make sure that links on the pages link to other pages of the mobile site
7. Make sure the content is readable and usable for mobile users. This means not using a small font size and making sure any touch buttons are spaced far apart and easy to use with fingers.
8. Think about the purpose of your visitor on mobile and design the page for their use. If your visitors often use mobile to look up your phone number or address, make this information easy for people to find on the front page.

Creating a mobile-friendly website is critical in the age of the mobile update, but it is a valuable part of any marketing strategy. It improves your ability to engage with users on-the-go. With an increasing number of visitors using mobile devices for search and shopping, sites that are hard to use on mobile can lose you customers regardless of the update. Make sure your site is mobile-friendly today.

<< Back to the SEO Glossary main page

What is the Google Penguin Update?

What is the Google Penguin update?

Google Penguin update was designed to reduce ‘web spam’ and encourage high-quality content for users. Since its initial release in April 2012, Google Penguin has released several updates. With each Google Penguin update, the algorithm once again scans websites for potential violations, opening up the possibility that brands might be penalized.

On January 10, 2016, Google released the latest Google Penguin update, which made Penguin run in real time. This means you can recover almost immediately once you make the corrections necessary to your page.

How does Google Penguin work?

You can generally tell if your page has been hit by a Penguin penalty by looking at Google Webmaster Tools and monitoring your traffic. Since Penguin is a page-specific update, you will notice a sharp drop for specific keywords. Google offered this traffic report as an example of a page hit by Penguin.

How do I avoid a Penguin update penalty?

The Penguin penalty largely focuses on identifying poor-quality websites based upon backlinks. Google’s algorithm works on the assumption that quality sites link to other quality sites while poor sites generally link to other poor sites. In other words, when you have websites that Google already trusts linking to you, it helps to boost your rankings. On the other hand, if you are linking to spam sites, then there is a good chance that the algorithm will not trust you either.

Google particularly wanted to penalize the sites that artificially boosted their rankings by buying links from link farms or other spam sites.

The best way to avoid a Penguin update penalty is to focus on cultivating positive backlinks. You can do this by writing content that people believe will help their users. When a site links to you, they are entrusting you with their visitors, so to get high-quality sites to provide you with backlinks, you need to be producing similar quality content.

You also want to regularly review your backlink profile. Even though there is still disagreement about the ability of spam sites to lower your quality rating by linking to your site, the possibility remains. Use Google’s disavow tool to let the search engine know that you are not interested in these links to make sure you do not incur any penalties.

To review, here is how to avoid a Google Penguin update penalty:

1. Get rid of any backlinks you have purchased or otherwise illicitly gained
2. Focus on creating high-quality content and distributing it to attract the desired backlinks
3. Regularly monitor your backlink profile to quickly find undesirable links
4. Disavow any poor-quality links to protect your site’s reputation

Although Google Penguin update can look intimidating, particularly if you get hit by the penalty and see your traffic drop drastically. Taking the necessary steps to correct the problem, however, can help you quickly recover, particularly since the update is now run in real time.

What Search Engines Exist Around the World?

Following Google, three of the most important search engines globally are:

1. Naver
2. Baidu
3. Yandex

Marketers need to become familiar with these engines if they want to successfully market to customers in those world regions where local prefered search engines have a significant market share. 

Why do I need to understand world search engines?

As brands increasingly go overseas and enter new markets, it is important to realize that Google is not the only dominant search engine in the world. Although Google is available in most countries, there are local, preferred search engines in many regions. If brands want to successfully market to customers in these areas, they will need to become familiar with these world search engines. 

Image source: Technode.com

What is Naver?

Naver is the number one search engine in South Korea, far outpacing Google locally. Naver used to be powered by Yahoo’s Overture System, but it then built and launched its own keyword advertising system. It is now a leading search engine that has a reputation for innovation. Anyone who wants to expand into South Korea will need to be familiar with this search engine.

Naver was built around the Korean language and culture. US-based sites should develop a version of their website catering to this culture, rather than just translating their text. Naver also draws heavily from user-generated content in their Knowledge IN area as well as from the social community Cafe. Brands that want to succeed in Korea should focus on building their presence in these areas.

What is Baidu?

Baidu is the number one search engine in China and the fifth most popular site in the world. There have been some controversies with Baidu in the past, including discussions about intellectual property. Regardless, it is important for those expanding into the Chinese region to familiarize themselves with this search engine.

Baidu punishes harshly for duplicate content, even more so that other major search engines. It also has a strong preference for simplified Chinese characters over traditional characters or romanized words. Baidu also strongly ranks content based on quality, so working with a native Chinese speaker will be necessary for most brands to avoid word-usage errors.

It is also important to note that Baidu is not just a search engine. It also offers a range of services, including Baidu Encyclopedia, Baidu Government Information Search and Baidu Games.

What is Yandex?

Yandex is the largest search engine in Russia and the surrounding region. It offers many similar services as Google, but there are also some key differences. For example, Yandex does not index pages as fast, forcing brands to focus more on long-term success with this search engine, rather than immediate results.

On the other hand, like Google, Yandex has a strong emphasis on quality content, quality linking and keyword relevance.

Being familiar with the other world search engines can help brands that want to expand their offerings overseas. While Google has a firm hold on search in the United States, in some other countries native search engines take priority.

<< Back to the SEO Glossary main page

SEO and Site Redesign: How Do I Do It?

Why is SEO important during a site redesign?

One survey found that 94 percent of subjects reported design elements when discussing reasons why they did not trust a particular website. To build trust with your customers, you need to make sure that your design communicates quality and trust to your customers.

At the same time, our own research at BrightEdge has found that 51 percent of traffic to websites arrives through organic search. That means SEO is the main tool for getting visitors on your page so that you have the opportunity to build that important trust with them.

When you are redesigning your site, you have an excellent opportunity to add SEO elements throughout your site to ensure that your new-and-improved web property is attractive to search engines and customers alike.

Does changing website design affect SEO?

Yes, and it can be very negative for your website if a redesign isn't properly handled. SEO needs to be a part of the site redesign process from the beginning. Navigation, URLs, folders and the layout of the site should all be optimized.

• Name URLs using keywords
• Make navigation, folders and layout all intuitive and common sense
• Include meta descriptions on all web pages
• Markup your pages with schema to ensure maximum visibility to the search engines

How do I build SEO for my website?

As you add content to the site, you will need to make sure that you are still placing the needs of the user first. This means using data to research topics and keywords that will guide your content. It also means including high-quality images with keyword-centric alt text. This alt text tells the search engine what the image is and the value it has to offer users.

What do I need to consider after I finish my redesign?

SEO and site redesign are living things-- they are continually growing, maturing and changing. Although you have finished this site redesign, you need to be ready to come back to the project in the future as technology and trends change the best ways to meet the needs of customers.

To see how your current site is performing, you should always be monitoring traffic and bounce rates as well as customers’ movements around the site. Compare behavior before and after the redesign to see if your efforts have improved the customer experience.

A site redesign is a great time to really incorporate SEO elements into the web property to maximize your exposure to customers. Keep these SEO concepts in mind as you begin your project and remember to always focus your efforts on making things easier and more intuitive for the customer.

<< Back to the SEO Glossary main page

What Does it Mean to Optimize a Website?

What is website optimization?

Also called SEO optimization, website optimization is a set of activities used to make a website rank well in organic search. Website optimization includes on-page and off-page optimization and other activities.

Why is website optimization important?

It is easy to put off the important task of optimizing an existing website, thinking that it will just take too much work. While optimization requires effort, the return you will get for making your website fully compliant with Google best practices cannot be underestimated. Remember that 51 percent of your site’s traffic comes from organic search, so making sure you maximize your visibility on the SERPs can have a big impact on your revenue and growth.

How do I optimize my website?

1. Start with a site SEO audit. This will help you find any technical errors or oversights that could be hindering your rankings.
2. Organize your folders and your site map. Make sure that navigation is intuitive and that your sitemap has been uploaded to Google Webmaster Tools.
3. Outline your buyer personas and align them with their buyer’s journeys.
4. Do data-led keyword research to learn about the topics and keywords that matter most to these customers at various points in their buyer's journey.
5. Map out your content to see how well you are aligned with your buyer personas and their journeys.
6. Modify and optimize content to match the topics and keywords needed to communicate with your customers. Make sure each piece of content adds value for people searching for that topic.
7. Create more content to fill in any gaps that might be left.

How long does it take for SEO to work?

Once you have optimized your site, the next logical step will be determining if your efforts were successful. The only way to know this is through metrics. Examine measurements such as:

1. Your traffic rates
2. User behavior on site
3. Bounce rates
4. Leads
5. Conversion rates

Compare these metrics to the data that you pulled from your site before you completed the optimization process. Look at areas where you have improved and see where you might still need to do some work.

Site optimization is an ongoing process that concerns itself with making your site as user-friendly and valuable as possible. It is not about conforming to an algorithm, it is about creating a user experience that people will regard as useful and leave them wanting to return to your site time and again, and thus building a relationship that leads to conversion.

Related materials

• SEO Needs Better PR: 5 Steps to Elevate SEO in Your Organization
• Latest Google SERP Layout Changes
• BrightEdge platform for ultimate website optimization

<< Back to the SEO Glossary main page

What is the Buyer's Journey?

The past two decades have seen an explosive change in the buyer’s journey. Before most people started using the Internet, customers interacted with brands largely on the company’s terms. Consumers had to wait for businesses to reach out to them in their buyer journey with information about the latest products. There were clear touchpoints that marketers and salespeople could identify to keep the customer engaged and moving forward during the modern buyer's journey.

For the modern buyer, this has all changed. Now 67 percent of the journey takes place online, where the customer is in control. They expect to receive personalized care and they want to receive the information desired on their schedule. This new journey is largely in the hands of the customers through the first few stages. It is the job of the brand to supply the information and be ready to meet the customer where they are, whether that is online or in store.

What are the stages of the buying process?

Awareness stage: The modern buyer’s journey begins with the awareness stage, when customers first begin to realize that they have a problem. They are interested in researching possible solutions to their pain points. At this stage in the buyer’s journey, they are open to many different brands and products. These customers are researching online, visiting social media pages and perhaps clicking on paid ads.

Consideration and evaluation stage: This type of buyer is still early in their buyer journey and not too confined in their decisions. They use search, social, and paid ads to learn more about the companies and products. Their queries will likely reflect their existing research, such as asking questions about products and brands.

Purchase stage: Now the customer has narrowed down their possibilities to just a brand or two. Their searches reflect their final decision making, and often these last searches include keywords comparing one brand to another. Customers might be meeting with sales people or emailing back and forth during this part of the buyer's journey.

Post-purchase stage: In this stage, the brand focuses on turning this first-time customer into a returning one. Salespeople might continue to contact the customer to see how well the product or service is working for them and to help them troubleshoot any struggles. Customer searches might focus on answers to common questions or learning how to maximize usage of the product.

How do I use the buyer’s journey in marketing?

1. Map the buyer’s journey to your customer personas to understand the path your customers take to convert to your brand.
2. Use the information to identify the types of topics, keywords and content marketing customers want to see, and create the material needed to guide them along the buyer's journey.
3. Use the buyer journey to understand the best touchpoints to reach out to the customer and what customers need to convert.

The buyer’s journey offers valuable insight about how customers respond to brands and how companies can encourage people to move towards their purchase. Understanding how the journey influences buyer behavior can help you channel your journey maps efforts more effectively.

<< Back to the SEO Glossary main page