In this third and final segment of our guide to SEO basic concepts, we’ll touch on the more advanced SEO definitions and concepts of website “crawlability,” including basic search engine directives, common client and server errors, best practices and web developer resources.

You can access Part 1 of our series, covering on-page SEO concepts, and Part 2, explaining essential linking terms and related search engine directives.

Crawlability

In simple terms, “crawlability” refers to the ease with which search engine robots (or “bots” like Googlebot) can “crawl” a website in doing their work of indexing pages to build the search engine results pages (SERPs).

There are several tools and best practices webmasters and developers can employ to optimize a website for search engine bots – in other words, maximize its crawlability – as well as minimize the usual crawling errors.

The ultimate goal behind site crawlability is to expedite the speed and accuracy of the bot’s crawling and indexing of individual pages.

SEO basic search engine directives

User-agents are a general term for software that acts on behalf of a user or a program and their corresponding Web browsers or identity (e.g., Chrome, Internet Explorer, Firefox).

Web developers and SEOs employ user-agent switchers, which change the user agent of a browser (e.g. Chrome, Internet Explorer, Firefox) when testing how a site renders when crawled by different search engine robots. Changing the user-agent of a browser is an advanced method generally reserved for a brand’s “geek squad.”

A short list of search engine robots and corresponding browsers would include:

• Googlebot for Chrome
• Bingbot for Bing
• Slurp for Yahoo!
• MSNbot for MSN/Live

There are also an extensive number of user-agents and switchers that Web developers can apply to see how different browsers work on mobile devices. Web developers have the option of using custom user-agents; this is a more common practice for larger, more complex websites.

Another set of search engine directives centers on redirects, which forward a Web page URL to a new Web page address, directing both site visitors and search engine robots to a different Web page. There are two redirects commonly used: permanent (301) and temporary (302).

• 301 indicates a permanent redirect, reflecting the HTTP (hypertext transfer protocol) status code of a Web page (HTTP status codes are further discussed below). It is the recommended method for Web page redirects, as it passes most of the PageRank status of the original page to the new page.
• 302 designates a temporary redirect. It does not pass PageRank and is generally not recommended.

Errors and Best Practices

There are several common and seemingly persistent issues that compromise the performance of websites, resulting in a poor user experience. Among the more typical problems are Web server glitches, faulty redirects, broken links, slow page speeds, duplicate content and multiple URLs. Fortunately, there are counter-measures that webmasters and developers can adopt to address these issues. Here, we define the problems most often encountered and best practices for deterring them.

Redirects

Errors are HTTP response status codes, ranging from 1xx to 5xx, indicating five classes of standardized responses to search queries. The most common are the 3xx redirection (described previously), 4xx client (website owner) error, 5xx server error and 444 no response.

You’re most likely familiar with 404 not found error message, which simply indicates the page URL requested could not be located. This is usually result of a broken or defunct link. A best practice is to develop a custom 404 page to display to the (likely frustrated) searcher, offering help or guidance in non-technical language. A second common 4xx error is 444 no response, indicating that the server has failed to return information and shut down the connection. This is often used to fend off malware.

5xx server error response codes signify the server is aware of an error, and cannot execute the user’s request. There are 18 5xx responses, ranging from 500 internal server error to 504 gateway timeout.

Canonical link element and canonical HTTP headers: In cases where Web page content may be accessed through multiple HTTP headers (URLs), has syndicated content that is published elsewhere, or is otherwise duplicated, canonicalization is recommended.

Canonicalization means defining the single, preferred Web page URL for your content, which consolidates and strengthens both link and ranking signals for greater search visibility. There are several ways to do this, such as specifying a canonical link in your HTTP header for downloadable white papers and PDFs, all of which you can find via Google’s Webmaster Help forum. Learn how our own ContentIQ can crawl your site to detect 4xx and 5xx errors and direct you towards getting them fixed.

Site speed

Site speed is a major signal in Google’s search ranking algorithm, and the search giant continues to push for a faster internet experience with its mobile-friendly initiative, which encourages webmasters to improve page load time. While rich media is a medium to embrace, it’s important to pay attention to the size of images and “bulkiness” of videos, as they may significantly slow upload time.

Sitemaps

XML Sitemaps lists a site’s Web pages in file with XML tags that details the organization of your website using “extensible markup language” (i.e., XML) schema. Submitting an XML Sitemap to the search engines is a recommended best practice to help search engine bots crawl and index your site’s pages quickly and accurately. You can learn more about XML Sitemaps from our article on the BrightEdge blog.

Resources and Tools

For SEO glossary purposes, we’ve only scratched the surface of how to optimize your website for crawlability.

There are several resources that go much farther in-depth, including BrightEdge’s ContentIQ, Google Webmaster Tools and Webmaster Central Help Forum.

We hope you’ve found our introduction to basic SEO concepts helpful!

With the recent news in our market, it felt like a good time to share our thoughts about search marketing landscape and what we are working on at BrightEdge.

Since Lem and I founded BrightEdge, it seems that every year brings more change and plenty of exciting opportunities for search professionals and digital marketers.

Like you, we see Search changing. A lot. When we talk with our customers, we hear the same common themes. Search is becoming more and more integrated into the marketing department. Organic Search is highly valued for insights about customers, the market, and competitors.

Those insights help companies grow revenue, drive product innovation, and create winning content marketing strategies. All of this change means it is more important for us to deliver the innovations that our customers need and want to be successful in their jobs.

Innovation

Our brand promise starts with innovation. In the early days of BrightEdge, our customers valued our rapid innovation to stay ahead. Our customers were on the front lines – they either delivered or they didn’t deliver – and we wanted to make sure they had insights, technology, and support to be successful.

Over the years, we’ve led the market with great technology and innovation. We’re incredibly proud of what we have accomplished and flattered that many of these features are now standard across the industry.  

 

We’re going to keep pushing more new innovations to market– and bringing them to market as fast as we can. A few recent, market-leading innovations are highlighted below:

Intent Signal

BrightEdge Intent Signal helps search marketers understand and optimize for organic potential. It identifies which topics show organic results above or below the fold. It goes a step further to tell which pages show above the fold – and are highly visible – and where to capture more of the organic opportunity.

You can read more about Intent Signal below:

BrightEdge Intent Signal: Optimize for Organic Opportunity “Above the Fold”

BrightEdge Intent Signal

HyperLocal

HyperLocal search allows customers to understand how topics and content perform for searches conducted in every corner of the globe. Marketers can track demand and content performance for over 32,000 US and 36,000 non-US city search locations or submit custom locations for both desktop and mobile.

You can read more about HyperLocal below:

BusinessInsider: BrightEdge Announces Next Wave of Organic Search Innovation with ContentIQ and HyperLocal Releases

HyperLocal Targeting: Content Performance From the Eyes of Your Customers

ContentIQ

ContentIQ is our next-generation site audit solution with advanced crawling capabilities and full integration within our SEO and content performance platform. All BrightEdge customers can add ContentIQ to their BrightEdge platform to identify the site errors that impact organic search performance.

You can read more about ContentIQ below:

ContentIQ: Detect and Fix Site Errors with Comprehensive Site Auditing

Customer Success

At BrightEdge we believe that our greatest responsibility is to ensure our customers’ success. In the past 12 months, we’ve made a big investment in supporting our customers - from the BrightEdge SEO power users who love DataCube to the digital marketers and content marketers who are just beginning to understand the power of SEO.

Search is hard and complex, and as Search moves into mainstream marketing we realize that it is more important than ever to invest in ways to support our customers – always with an eye on helping them deliver real results and ROI. We continue to work hard in this area and keep looking for innovative ways to make search easier, more accessible, and to be the main driver of marketing strategy.

You can view more customer success stories below:  

BrightEdge Case Studies   

Community

Building community is another area where we continue to make big investments and see traction in the market. With SEO going mainstream, we saw a need to foster collaboration and share insights across search, content, and digital marketing communities.

Our annual Share Conference is recognized by Forbes and Inc.com as a must-attend conference, and this year we are taking Share on the road to local communities and across the globe.

You can read more about the Share17 Global Insights tour below:  

Share17 Goes on the Road

We’ve also launched User Groups in NYC, London, SF, Chicago, and Minneapolis to make it easier for search marketers to network, make key connections, and learn from one another. And we’ll continue to ramp up community events to support the digital marketers and content marketer on SEO best practices.

We’ve been very excited about the turnout and passion around these impactful customer events.

Building the BrightEdge community has been one of the most rewarding parts of my job – above all, I value spending time with customers and hearing from them. We love our customers and the community that we support. All the passion in our industry is a sign that SEO is here to stay, SEO is more important than ever, and SEO is becoming a marketer’s (not so) secret weapon.

Before I sign off, I need to thank the 300+ employees at BrightEdge for the hard work, enthusiasm, and customer-obsessed mentality that they bring each day. This blend of passion, desire, business acumen, and market maturity that comprises the BrightEdge team makes me extremely proud, and I’m excited for what lies ahead.

BrightEdge proudly announces its next-generation site audit solution with advanced crawling capabilities and full integration within our SEO and content performance platform.

Unplanned site changes severely impact organic search performance

In fast-paced, agile environments, multiple teams across an organization are responsible for creating content. It is estimated that 4.6 billion pieces of content are produced daily. And content creation is only getting bigger: 43% of companies reported increasing their content marketing staff levels in 2016.

With so many moving parts, and some team members failing to adhere to SEO best practices, technical errors, and unplanned site changes can occur. Site errors – including duplicate content, pricing errors, missing tags, and broken links – lead to penalties and lower ranking and provide a poor user experience. And when web pages see a downturn in organic performance, the result is decreased traffic, conversions, and revenue.

To prevent this from happening, a site audit is a critical first step in assessing your website’s health and identifying the technical errors that impede organic performance.

ContentIQ provides the power and flexibility needed in today’s site audit solution

We are pleased to announce the launch of BrightEdge ContentIQ, a new and improved site audit solution. Now all BrightEdge customers can add ContentIQ to their BrightEdge platform to identify the site errors that impact organic search performance.

Using ContentIQ you can:

• Utilize best-in-class crawl technology, parsing the content elements used to build today’s web experiences
• Target site crawls to just a few sections of your website or crawl your entire site – scalable to hundreds of millions of pages
• Set custom crawl settings to match your organization’s unique digital standards

Updated crawl technology for a comprehensive assessment of site health

Web developers and content creators are now tasked with creating dynamic web experiences across multiple devices including desktop, tablet, and mobile. Traditional site audit solutions, which focus on crawling static HTML in a desktop environment, no longer provide the insights needed to fix the site errors that impede performance today.

ContentIQ technology parses the content elements used to create today’s web experiences, including progressive web applications, sophisticated JavaScript frameworks, schema, social tags, and more. Errors are displayed according to severity and by section of your site, allowing you to click in to investigate further and prioritize errors to fix.

Customize your crawl and access insights directly within the BrightEdge SEO platform

Other site audit solutions are fixed and rigid and may not single out the specific site violations that you care about. ContentIQ lets you adjust your crawl settings to comply with your organization’s digital standards, including content governance, web accessibility standards, and website architecture. You can isolate your crawl to focus on just a few pages or to complete a full audit of your entire website.

And since ContentIQ is available within the BrightEdge SEO platform, you maximize efficiency by identifying, fixing, and monitoring site errors within a single workflow. This means that you can use BrightEdge platform features in conjunction with your ContentIQ site audit data. For example, use StoryBuilder to build engaging dashboards and charts that demonstrate the impact of recent site revisions.

With ContentIQ, you gain an upper hand on the site violations that are keeping your performance down. Providing customizable settings, best-in-class crawl technology, and integration within an SEO platform, ContentIQ gives you peace of mind as you identify, target, fix, and monitor the violations that most threaten your site traffic, conversions, and revenue.

ContentIQ is available to all customers as an add-on to the BrightEdge search and content platform. Ask your Customer Success representative about ContentIQ, or schedule a demo.