maspillera

M Posted 8 years ago

t 9 min read

The origins of the robots.txt protocol, or the "robots exclusion protocol," can be traced back to the mid-1990s, during the early days of web spiders traveling the internet to read websites. Some webmasters became concerned about which spiders were visiting their sites. A file containing directions on which site sections should be crawled and which shouldn't offered site owners the promise of having more control over which crawlers could visit their URLs and how much capacity they were allowed to consume. Since then the robots.txt has grown to meet the needs of modern web designers and website owners.

The current versions of the protocol will be accepted by the spiders and the major search engines to send out to gather information for their respective ranking algorithms. This is a common agreement among the different search engines thus making the commands a potentially valuable, but often overlooked, tool for brands in their SEO reports.

What is robots.txt?

Robots.txt is a series of commands that tells web robots, usually search engines, which pages to crawl and not to crawl. When a search engine lands on a site, it looks at the command for instructions. It can seem counterintuitive for a site to want to instruct a search engine not to crawl its pages, but it can also give webmasters powerful control over their crawl budget.

When writing out your protocol file, you will use simple, two-line commands. The first line says, "user-agent." This portion of the protocol dictates who the instructions apply to, and an asterisk "*", usually referred to as a wildcard, will mean that the command applies to all the web robots. Under the "user-agent" it will say, “disallow.” This tells the robots what they cannot do. If there is a "\", it means that the spiders should not crawl anything on the site. If this portion remains blank, then the spiders can crawl the entire site.

Why would I want to use robots.txt?

Understanding how Google crawls websites will help you see the value in using robots.txt protocol. Google has a crawl budget. This describes the amount of time that Google will dedicate to crawling a particular site. Google calculates this budget based upon a crawl rate limit and crawl demand. If Google sees that their crawling of a site slows down that URL, and thus hurts the user experience for any organic browsers, they will slow the rate of the crawls. This means that if you add new content to your site, Google would not see it as quickly, potentially hurting your SEO.

The second part of the budget calculation, the demand, dictates that URLs with greater popularity will receive more visits from the Google spiders. In other words, as Google stated, “you don’t want your server to be overwhelmed by Google’s crawler or to waste crawl budget crawling unimportant or similar pages on your site.” The protocol will allow you greater control over where the search engine crawlers go and when, helping you to avoid this problem. In addition to helping you direct search engine crawlers away from the less important or repetitive pages on your site, robots.txt can also serve other important purposes:

• It can help prevent the appearance of duplicate content. Sometimes your website might purposefully need more than one copy of a piece of content. For example, if you create a printable version of a piece of content, you may have two different versions. Google has a well-known duplicate content penalty. This would allow you to avoid that.
• If you are reworking parts of your website, you can use robots.txt to hide unfinished pages from being indexed before they have been prepared.
• You also likely have pages on your website that you do not want displayed to the public. For example, this might include a thank you page after someone has made a purchase or submitted a login page. These pages should not appear on a search engine, making it worthless for Google or other search engines to index them.

According to Google, here are some of the most common rulesets for the protocol:

1. It is important that while noting the various useful features of robots.txt, you do not try to use the protocol to keep sensitive information private. Think of the protocal as a request, but not a command.
2. Although "good" spiders, the ones run by reputable organizations such as Google or Bing, will listen to the protocol's orders, it remains quite possible for crawlers designed by those with more nefarious intentions to ignore the command and crawl the page regardless of the code specified. Pages can also become indexed in other ways. For example, if another site or another page on your own site links to the page, your hidden page could end up indexed.
3. When planning out your site infrastructure it's important to make a distinction between your private pages. Are these pages that need to be private and non-discoverable for solely for SEO reasons, or is keeping the content on these pages away from crawlers a security issue - e.g. exposure of sensitive customer data? The value of the robots.txt comes from the SEO strategy, not keeping confidential information private.

How do I configure robots.txt?

Setting up your protocol can be straightforward, but first let’s review what the two parts of the protocol mean:

• User-agent: refers to the crawlers that the text refers to
• Disallow: indicates what you want to block, what the crawler should not read

In addition to these two main portions, you can also use a third portion, labeled ‘allow’ if you need it. This section will be used if you have a subdirectory that falls within a blocked directory. For example, if you wanted to block the majority of a directory but had a single small subdirectory, the fastest way to set this up might be to say: user-agent: * (remember that the asterisk indicates that the protocol applies to all the spiders) disallow: /directory allow: /subdirectory1

This will tell the crawlers to look at this single subdirectory, even though it falls within the larger, blocked directory. If you want to let the crawlers look at the entire site, then you will leave the ‘disallow’ portion blank. If you want to set up your robots.txt to block specific pages, such as your login page or a thank you page, then in the "disallow" portion of the protocol you will put the portion of your URL that comes after the ‘.com’. As you think about the pages that you may want to block, consider these types of content and see if you have any on your site.

• Login pages
• Thank you pages for after someone has downloaded or purchased something
• Needed duplicate content, such as a PDF or printable version of a webpage
• New pages that you have begun to develop, but you do not want search engines indexing them yet

Although robots.txt looks simple, there are a few rules that must be followed to ensure that the code will be interpreted properly.

• • Use all lower case letters for the file name, "robots.txt"
  • The protocols must be located in the top-level directory of the web server
  • You can only have one "disallow" for each URL on the site
  • Subdomains that have a common root domain need different protocol files

After you set up the protocol, you should test your site with your Google Webmasters account. Under the menu there will be a "Crawl" option. Clicking on this will open a dropdown menu, which will include a protocol tester option. If Google says that the text is allowed, it means that your text was written correctly. Robots.txt can be a useful tool in the hands of SEOs who understand the value of controlling how and when spiders crawl their websites. Consider how it might benefit your website and take this next step in SEO. 

enewton@brightedge.com

M Posted 8 years ago

t 9 min read

Google Ads, previously known as Adwords, rebranded on July 24th, 2018. We look at the history of Google's paid search program and how it became the centerpiece of the company's business model. Paid search advertising is more than 20 years old. Before Google had decided how to monetize its technology, there was the Goto.com search engine. Goto.com established the PPC auction system with advertisers bidding for ad position on keywords and ads ranked by bid value. After the dotcom crash, Goto.com rebranded as Overture and sold itself to Yahoo for $1.63 billion. Google continued to evolve its business model from AdSense and impression CPM billing to AdWords and CPC billing. Google invested heavily in improving the fraud control and relevancy of its offering and outgrew Yahoo handily. Google has grown to over $110B in revenue with nearly 90% of Google's revenue coming from PPC and AdWords.

Following that, Google bought DoubleClick, a digital display advertising company, for $3.1B in 2007, which at the time seemed like a great deal of money, as the paid ads business was still recovering from the dotcom crash. It also made huge investments in the Chrome browser and Android operating system which allows it to stay involved in nearly every area of digital media. The major investments that Google made at the time paid off. Mobile ads now account for 67% of digital ad spending and mobile video spending will exceed traditional TV spending in 2018, according to eMarketer.

Rebranding of Google Adwords

I received this email message from Google:

Hello, You're receiving this message because you currently have an active AdWords account. Earlier this week, we announced that Google AdWords is becoming Google Ads. The new Google Ads brand represents our full range of campaign types across Search, Display, Video, App and more. These help you connect with more consumers as they're turning to Google, and our partner sites and apps, to find information and get things done. Starting July 24, 2018, you will begin to see the new Google Ads brand — including the new name and logo — reflected in the product interface, Help Center, billing documents, and more. Our URL will also be changing from adwords.google.com to ads.google.com. There is no action required on your part, and changes to the Google Ads branding will not impact your campaign performance, navigation or reporting. If you have any questions or would like to contact us, please visit the Help Center. Sincerely, The Google AdWords team (soon to be Google Ads)

According to Google, the rebranding is part of a campaign to simplify the company's paid advertising platforms. As part of that initiative, the full set of rebrands is as follows:

1. AdWords to rebrand as Google Ads
2. DoubleClick and Google Analytics 360 Suite unified as Google Marketing Platform
3. DoubleClick for Publishers and DoubleClick Ad Exchange unified as Google Ad Manager

In contrast to AdWords, Google Ads will have more automated Google Ads campaign development features that use machine learning and AI to help marketers launch campaigns with less administrative and platform knowledge.

As part of the Google Marketing Platform for larger customers, Google also announced Display & Video 360. "Display & Video 360 brings together features from DoubleClick Bid Manager, Campaign Manager, Studio and Audience Center to allow creative, agency, and media teams to collaborate and execute ad campaigns end-to-end in a single place."

Aligning Google Ads Campaigns to the Industry

This move combines the Google DoubleClick and Google Analytics 360 Suite platforms under a single brand. Similarly, DoubleClick for Publishers and Ad Exchange will be merged into Google Ad Manager. This move towards branding consolidation, as well as some actual platform consolidation, aligns Google's paid advertising programs with other advertising platforms in the space. For example, Facebook has a similar offering it calls Facebook Ads. Adobe Cloud also enables cross-platform and cross-channel campaign setup and management. So, it is a battle of the titans in advertising, each offering huge inventory, and more breadth and convenience that will make it harder for independent networks to secure ad dollars. 

koleary

M Posted 8 years ago

t 9 min read

It is no question that every business now has to consider their marketing strategies on all device types. Understanding how your customers engage with you throughout the buying journey, typically starts with research on a mobile device. For many industries, mobile traffic from organic has surpassed desktop traffic; the latest BrightEdge research has found that on average 57% of the traffic to your website now arrives from either mobile or tablet devices. Ninety-three percent of people who perform research on mobile devices will go on to make a purchase and mobile drives or influences over 40% of revenue for leading B2B organizations. In recent years, your customers are no longer just researching on their mobile devices, but are also starting to convert at higher rates on mobile. It has become critical to have a mobile-first strategy, and a mobile-friendly site. These mobile users want to see quality websites--mobile users who do make a purchase through their devices will visit the website for the brand an average of 6 times before they hit "buy." As the importance of paying attention to mobile users becomes clear, so too does the need to optimize specifically for customers on these devices. The mobile SERP has become increasingly contextual. In an effort to boost personalization, search engines use factors such as the person’s physical location when performing searches. The SERP can also differ greatly from desktop searches. Forty-seven percent of keywords in positions 1 to 20 rank differently on mobile versus desktop SERPs. This indicates that people need to create a mobile-specific optimization strategy. Only by focusing efforts on these mobile users can brands effectively engage with on-the-go users and begin to tap into this continually growing market.

Establish baseline to prepare for mobile SEO optimization

The first step of creating an effective mobile optimization strategy is establishing a baseline. The baseline describes the state of your mobile marketing efforts right now before you change anything. You can use this baseline to understand where you are today, and help you prioritize your efforts as you begin your SEO for mobile.

Step 1

On the BrightEdge platform, log in to view your Share of Voice report. Here you will be able to benchmark your current content against your natural competitors. You want to track and understand how your natural competitors on mobile differ from desktop. This will let you know the strength, or areas to improve, your brand presence when your targeted customers are on mobile.

Step 2

Next, click over to see your traffic report. To better understand how your customers find you, create a chart based on traffic by device type. You will be able to see what percentage of your traffic comes from people on mobile devices versus desktop.

Step 3

As a part of this baseline report, you also want to look for potential mistakes. Use BrightEdge ContentIQ to help you identify potential errors in your content development. You can run a site audit crawl based on a Google smartphone bot, similar to how you would run a mobile friendly test with Google. When you setup the crawl, make sure you also add your mobile domain, or if a responsive site (which is preferred) run the crawl on your entire domain.

Step 4

You also want to examine your site for common mobile errors. These might include:

• Using Flash, which is not particularly mobile friendly. Many mobile devices will not play these elements
• Issues with your viewport that make it difficult for users to see what you posted
• Content or images not appearing clearly on mobile devices. Small fonts or images can both cause issues here
• Touch elements living too closely together. People on mobile devices use their fingers to navigate your site
• Elements, such as submit buttons, placed too closely together will make it difficult for people to properly use the site

Find content opportunities to optimize SEO for mobile

Once you have established your baseline, the next task will be to find opportunities for mobile-centered content development. BrightEdge Data Cube will help you to surface great mobile keyword opportunities. You can measure traffic rates and see which words will likely provide the most return. Central to your strategy should also be "striking distance" keywords. This describes the keywords that you already rank for, where your page falls between positions 5 and about 15. Check your keyword ranking reports. You can filter for mobile devices to find your own list of keywords in these positions on mobile devices. Through fixing site errors and optimizing the content based on your mobile target keywords, you have the best shot of boosting the rankings for these pages. As you use these two strategies, do not neglect to see how the SERP looks for these particular keywords. On mobile devices, the screen can easily be dominated completely by a Quick Answer or video results. You want to select keywords that have the highest possibility of visibility above the fold. Verify the SERP of your selected keywords and tailor your content development efforts for these needs. The best way to understand your organic opportunity is to run the Intent Signal dashboard for mobile (smartphone). We have created a dashboard template for you within StoryBuilder. Target the keywords in the Optimize and Create categories. This is where there is organic content above the fold, but you need to either optimize content in striking distance or create new content for those keywords. Once you have categorized those keywords, you can also use Keyword Reporting to understand the content type, regular web listing or a universal result. If a high-value mobile keyword has a Quick Answer (a universal result), consider using our Quick Answer white paper for more information about earning this coveted rich answer position.

Optimize content for mobile

Now that you have identified your mobile optimization opportunities, you want to create your content. The material you develop should be smart. It should:

• Target exactly what customers need
• Be optimized to ensure visibility
• Be up-to-date
• Be integrated with the rest of your marketing efforts
• Be profitable

Remember the goals that your mobile users most likely have, and incorporate the answers to their needs into your content. Data Cube can help you find semantically related keywords. These related keywords can help you further optimize your content and demonstrate your value to the search engine. Google uses these keywords to better understand content and see how it relates to queries. Incorporating the appropriate ones will show depth.

Measure and improve for mobile

Once you begin to publish your mobile-centric content, you will want to keep a careful eye on your results. This will let you know if your efforts are helping you accomplish your goals. It will also help you redirect your efforts if necessary. Page Reporting will be used to track your traffic and compare it to your revenue. Here you will see how well your traffic boost rates compare to revenue. If you do not see corresponding lifts in revenue, then you can look for problems towards the end of your sales funnel and make the appropriate corrections. Page Reporting will also help you see how much time people spend on your page as well as your bounce rates. Poor numbers in these areas can be good indications that your content does not align well with user intent. You will want to reexamine other high ranking content on the SERP for these queries to see where important adjustments need to be made. BrightEdge also offers a mobile-first dashboard. You can add in any more charts that you want to include, customizing it to best meet your needs. As you look at your progress, watch your rankings. Look for improvements as well as new striking-distance keywords that you can capitalize on. The modern consumer is a mobile consumer. People use their mobile devices to engage with people around them and businesses they care about. The better prepared you are to work with these customers, the easier it will be to interest them in engaging with your business. Get started to today!

1. Establish your baseline
2. Prioritize mobile keywords
3. Optimize/Create content targted to mobile
4. Measure and track progress
5. Iterate plan based on progress