A BrightEdger

M Posted 7 years 9 months ago

t 9 min read

On the morning of Tuesday, June 12 we received the sad news that Michael Kirchhoff, our Senior Director of Content, had passed away. His contributions here at BrightEdge were many, and the energy and dedication he brought will be sorely missed. Michael’s passing is a huge loss for BrightEdge and the SEO community, but far more importantly, it is a loss for his family. A striking aspect of any testimonial about Michael is that they mention not just his skill as an SEO professional, but his genuineness, humility, positivity, and overall quality of character. He was truly one of the good guys of SEO. In memorial to Michael, and to honor his contributions to BrightEdge, the discipline of SEO, and his community, we are introducing the Michael Kirchhoff Technical SEO Award to be awarded annually at the BrightEdge Share conference.

As an SEO

Before joining BrightEdge, Michael honed his SEO skills in several places after getting his start as a digital strategist in 1996. Based out of Tulsa, he contributed his marketing skills at a variety of local brands, most notably PennWell Publishing. At PennWell, Michael worked in several roles, including digital media product management, SEO and search marketing manager, and finally as director of SEO and product support. As director of SEO, a role he held for ten years, he defined an enterprise SEO strategy for an astounding 118 websites across 10 different industries. The strategy he developed led the brand to growth in SEO of 1,187%. He also created and led a social media steering committee to provide management of a multi-site social strategy and a content taxonomy to organize and tag the content published across those sites. Michael joined the BrightEdge team in 2016. As part of our team he became a key contributor and one of the most experienced working SEOs within the organization, bringing that experience to the planning and development of our Content Optimizer customers for on and off-page SEO auditing and optimization.

In the community

Michael’s generosity and mindfulness went beyond just the professional realm. He also put a significant amount of time into charity and volunteer work in his community. Most notably he spent four years as a committee member for the Spirit of Youth’s Casino Royal Fund Raiser. Spirit of Youth raises funds to help run the Special Olympics of Oklahoma. In addition, Michael was a technology consultant for local agency members in the Tulsa Area United Way. In this capacity he provided digital strategy for member agencies to reach their digital goals. He helped with everything from SEO to paid search, website architecture, social media strategy, and more.

Michael remembered

We at BrightEdge remember Michael as a key contributor on the projects many of us worked on, but more importantly, as a person of tremendous kindness with his time and expertise. Jim Yu, CEO and cofounder of BrightEdge, highlighted Michael’s great contributions at PennWell and BrightEdge as well as his legacy as a figure in the SEO industry:

“Michael was a friend and customer when he was at PennWell. He was one of the top SEO thought leaders in our industry, spoke at Share and other conferences, and that was why we were so happy when he told us he wanted to work for BrightEdge. He led our Content Optimizer team and helped grow that business line for the years he worked on it. We are sad he is gone, but we are so glad we knew him. In his honor, we created the Michael Kirchhoff Technical SEO Award to recognize professionals of great character and technical expertise. Please nominate great people from the community for consideration.”

Lemuel Park, BrightEdge co-founder and CTO, had this to say:

“Michael was a tremendous thought leader in content marketing and search. His acumen to look ahead and see the big picture, while diving into the details, was fantastic!

Another personal touch I’d like to add is Michael’s character. He was a person who I could trust, always reliable, and open to honest discussion any time.”

Erik Newton, VP of customer marketing and head of SEO, worked with Michael on a number of content and SEO initiatives at BrightEdge in his capacity as a member of our internal SEO Team.

“Michael was a member of SEO Team and AskSEO where we worked closely here at BrightEdge. He was warm, upbeat, and cheerful and a dynamic presenter on stage, which you can see in the photos throughout this post. He was our go-to colleague for deep technical topics, and you can see a great series of his on the BrightEdge Blog.”

 Andy Betts, executive adviser and VP of global content, marketing, and development at BrightEdge, worked frequently with Michael on many different projects.

“Michael is without doubt the most genuine, caring, and knowledgeable SEO I have met in over 15 years in this industry. In an industry where the perception of people’s dedication and knowledge can be skewed by the number of posts they do or the number of followers they have, gems like Michael can be overlooked. Many people in the industry who have met Michael came away from conversations feeling they have a friend for life, someone they can trust, and someone who was always willing to help, promote, and give advice.”

Andy nominated Michael for the Say Something Nice About an SEO/SEM series earlier this year. In that interview Michael said what he wanted to be known for in the SEO space was:

"Being one of the semantics and taxonomy SEO pioneers. It’s an area I’ve been working on since about 2005 and am fascinated and impressed with how much can be accomplished. The importance of semantics and taxonomy for ultimate website performance, personalization, and audience reach in SEO is profound."

And don't forget to backlink to this page with the anchor text >really nice guy<. We are sure this will rank first and that would make Michael happy.

maspillera

M Posted 7 years 9 months ago

t 9 min read

The origins of the robots.txt protocol, or the "robots exclusion protocol," can be traced back to the mid-1990s, during the early days of web spiders traveling the internet to read websites. Some webmasters became concerned about which spiders were visiting their sites. A file containing directions on which site sections should be crawled and which shouldn't offered site owners the promise of having more control over which crawlers could visit their URLs and how much capacity they were allowed to consume. Since then the robots.txt has grown to meet the needs of modern web designers and website owners.

The current versions of the protocol will be accepted by the spiders and the major search engines to send out to gather information for their respective ranking algorithms. This is a common agreement among the different search engines thus making the commands a potentially valuable, but often overlooked, tool for brands in their SEO reports.

What is robots.txt?

Robots.txt is a series of commands that tells web robots, usually search engines, which pages to crawl and not to crawl. When a search engine lands on a site, it looks at the command for instructions. It can seem counterintuitive for a site to want to instruct a search engine not to crawl its pages, but it can also give webmasters powerful control over their crawl budget.

When writing out your protocol file, you will use simple, two-line commands. The first line says, "user-agent." This portion of the protocol dictates who the instructions apply to, and an asterisk "*", usually referred to as a wildcard, will mean that the command applies to all the web robots. Under the "user-agent" it will say, “disallow.” This tells the robots what they cannot do. If there is a "\", it means that the spiders should not crawl anything on the site. If this portion remains blank, then the spiders can crawl the entire site.

Why would I want to use robots.txt?

Understanding how Google crawls websites will help you see the value in using robots.txt protocol. Google has a crawl budget. This describes the amount of time that Google will dedicate to crawling a particular site. Google calculates this budget based upon a crawl rate limit and crawl demand. If Google sees that their crawling of a site slows down that URL, and thus hurts the user experience for any organic browsers, they will slow the rate of the crawls. This means that if you add new content to your site, Google would not see it as quickly, potentially hurting your SEO.

The second part of the budget calculation, the demand, dictates that URLs with greater popularity will receive more visits from the Google spiders. In other words, as Google stated, “you don’t want your server to be overwhelmed by Google’s crawler or to waste crawl budget crawling unimportant or similar pages on your site.” The protocol will allow you greater control over where the search engine crawlers go and when, helping you to avoid this problem. In addition to helping you direct search engine crawlers away from the less important or repetitive pages on your site, robots.txt can also serve other important purposes:

• It can help prevent the appearance of duplicate content. Sometimes your website might purposefully need more than one copy of a piece of content. For example, if you create a printable version of a piece of content, you may have two different versions. Google has a well-known duplicate content penalty. This would allow you to avoid that.
• If you are reworking parts of your website, you can use robots.txt to hide unfinished pages from being indexed before they have been prepared.
• You also likely have pages on your website that you do not want displayed to the public. For example, this might include a thank you page after someone has made a purchase or submitted a login page. These pages should not appear on a search engine, making it worthless for Google or other search engines to index them.

According to Google, here are some of the most common rulesets for the protocol:

1. It is important that while noting the various useful features of robots.txt, you do not try to use the protocol to keep sensitive information private. Think of the protocal as a request, but not a command.
2. Although "good" spiders, the ones run by reputable organizations such as Google or Bing, will listen to the protocol's orders, it remains quite possible for crawlers designed by those with more nefarious intentions to ignore the command and crawl the page regardless of the code specified. Pages can also become indexed in other ways. For example, if another site or another page on your own site links to the page, your hidden page could end up indexed.
3. When planning out your site infrastructure it's important to make a distinction between your private pages. Are these pages that need to be private and non-discoverable for solely for SEO reasons, or is keeping the content on these pages away from crawlers a security issue - e.g. exposure of sensitive customer data? The value of the robots.txt comes from the SEO strategy, not keeping confidential information private.

How do I configure robots.txt?

Setting up your protocol can be straightforward, but first let’s review what the two parts of the protocol mean:

• User-agent: refers to the crawlers that the text refers to
• Disallow: indicates what you want to block, what the crawler should not read

In addition to these two main portions, you can also use a third portion, labeled ‘allow’ if you need it. This section will be used if you have a subdirectory that falls within a blocked directory. For example, if you wanted to block the majority of a directory but had a single small subdirectory, the fastest way to set this up might be to say: user-agent: * (remember that the asterisk indicates that the protocol applies to all the spiders) disallow: /directory allow: /subdirectory1

This will tell the crawlers to look at this single subdirectory, even though it falls within the larger, blocked directory. If you want to let the crawlers look at the entire site, then you will leave the ‘disallow’ portion blank. If you want to set up your robots.txt to block specific pages, such as your login page or a thank you page, then in the "disallow" portion of the protocol you will put the portion of your URL that comes after the ‘.com’. As you think about the pages that you may want to block, consider these types of content and see if you have any on your site.

• Login pages
• Thank you pages for after someone has downloaded or purchased something
• Needed duplicate content, such as a PDF or printable version of a webpage
• New pages that you have begun to develop, but you do not want search engines indexing them yet

Although robots.txt looks simple, there are a few rules that must be followed to ensure that the code will be interpreted properly.

• • Use all lower case letters for the file name, "robots.txt"
  • The protocols must be located in the top-level directory of the web server
  • You can only have one "disallow" for each URL on the site
  • Subdomains that have a common root domain need different protocol files

After you set up the protocol, you should test your site with your Google Webmasters account. Under the menu there will be a "Crawl" option. Clicking on this will open a dropdown menu, which will include a protocol tester option. If Google says that the text is allowed, it means that your text was written correctly. Robots.txt can be a useful tool in the hands of SEOs who understand the value of controlling how and when spiders crawl their websites. Consider how it might benefit your website and take this next step in SEO.