Since joining Bidmark, one of the most impressive things that has struck me is the agency-placed importance of getting the basics nailed for all clients - whether they be one of the local printers or one of the global B2B businesses that we work with. All clients are guided closely from the very beginning of their project to ensure that our on-page and technical SEO service efforts are set to provide (and exceed!) the results that they desire, and ones we can be proud of.
In this article, I take a deeper look at Technical SEO and provide my key technical SEO tips for different areas of a website's SERP performance ability.
Contents
- Why Technical SEO Matters
- The Technical SEO Basics
- Advanced Technical SEO
- Securing Top SERP Positions
- Our Key Technical SEO Tips
- How to Do An Technical SEO Audit
- Technical SEO FAQs
Why Technical SEO Matters
Technical SEO is crucial for improving a website's visibility and performance on search engines. It involves optimising the technical aspects of a site to ensure search engines can crawl, index, and rank it effectively.
By understanding the technical SEO basics , you can build a strong foundation for your website's SEO strategy. Implementing some of the technical SEO basics, can lead to better performance and greater visibility in search engine results. By staying on top of technical SEO, you ensure your site is well-optimised for both users and search engines.
The Technical SEO Basics
As a digital marketing agency, part of our offering is to ensure that our clients are getting the basics of the critical SEO areas optimised, ensuring a solid foundation for any following work. Here I take a look at the primary areas and provide some technical SEO hints and tips deemed critical when providing our recommendations.
Critical Area 1: Crawlability
The first of our technical SEO tips is that search engine crawlers need to be able to access all pages that drive organic search traffic. Whether it be on-page text, links coded with an anchor tag bound to anchor text or country/language preferences for ranking internationally are all handled optimally for SEO with unique URLs, crawlers need to be able to access all areas.
Anything that doesn’t need to be crawled, should be added to the robots.txt file and blocking crawlers to ensure they are only crawling and indexing what is necessary.
Critical Area 2: HTTPS
Our second tip is that search engines, especially Google, consider having a secure hosting package on your website as a ranking signal that builds trust with users and search engines. Install an SSL certificate and redirect all HTTP URLs to HTTPS versions. Essentially this means that if your website doesn’t have HTTPS, search engines do not see your website as secure for users. You can see if a website is secure through the use of HTTPS in the address bar.
Once you have purchased an SSL Certificate, ensure a 301 redirect from the HTTP URLs to their HTTPS variations is put in place.
Critical Area 3: Page Speed
Thirdly, page speed is a key factor in ensuring that users visit and stay on your website. A website that takes too long to load, is likely to have a high bounce rate. There are many things you can do (likely with the help of a developer) to ensure your pages are high performing:
Use caching
Caching stores parts of your site so they don’t need to reload every time, making it faster for repeat visitors.
Compress images and code
Big images and bulky code slow down your site. Use tools to shrink image size and clean up code.
Minify JavaScript and CSS
This means removing unnecessary characters (like spaces and comments) from your JavaScript and CSS files to make them load quicker.
Use Accelerated Mobile Pages (AMP)
AMP makes your site super fast on mobile by using a simplified version of your content.
Improving speed is one of the core technical SEO basics every website should prioritise. Google’s PageSpeed Insights will provide an overview of your score on both mobile and desktop, and offers technical SEO tips on how your site could load more quickly, thus rank higher, convert more traffic and ultimately drive more revenue.
Google’s PageSpeed Insights will provide an overview of your score on both mobile and desktop, and offers technical SEO tips on how your site could load more quickly, thus rank higher, convert more traffic and ultimately drive more revenue.
Critical Area 4: Mobile Friendliness
Next every site should be responsive. By responsive, we mean that it should work on all screens - no matter what the size. Among our top technical SEO tips, ensuring mobile responsiveness stands out as a foundational best practice.
Google and Bing’s mobile-friendly tests will allow you to understand if there are any indexation and ranking issues with your site.
Critical Area 5: URLs
Then URLs should be short, in the lower case and contain the relevant keywords to that page. URLs should not contain special characters, and for foreign variants of your site, be translated into the local language of the site. It is also recommended that in place of a space between words you use a hyphen over other characters such as underscores.
Critical Area 6: Heading Tags
Next you need to ensure that you use the correct tags for your on-page content. This is a great opportunity to incorporate technical SEO basics into your content structure.
Ideally, there would only be one <H1> heading on your page with subsequent headings using H2 through to H6 (remembering 1 is the most important). Using heading tags effectively helps structure your content for both readers and search engines. Here’s a simplified guide:
H1: The Main Page Title
You should use only one H1 per page. This is the main heading that describes the primary topic of the page. It’s like the title of a book and sets the theme for everything that follows.
H2: Main Sections Under H1
H2 headings divide your page into major sections. These help organise your content into topics related to the main H1. Each H2 introduces a new idea or section of the page.
H3: Subsections Under H2
Use H3 headings to break down the content within each H2 section. These allow you to go into more detail and keep the page well-structured and easy to follow.
H4–H6: Deeper Subdivisions
H4, H5, and H6 can be used to divide content even further within an H3 section. However, most pages don’t need to go beyond H3. It’s best not to overcomplicate the structure unless you really need to.
Heading Tag Hints & Tips:
Using heading tags properly helps organise your content, improves user experience, and supports search engine optimisation (SEO). Below are key guidelines to follow when working with headings on your web pages.
Divide Your Content Logically
Start with a single H1 to define the main topic of the page. Use H2 tags for major sections under the H1, and H3 tags for smaller sections within each H2. Only use H4, H5, or H6 if further breakdown is necessary. This creates a clear, readable hierarchy that benefits both readers and search engines.
Avoid Skipping Levels
Headings should follow a natural, sequential order. Don’t skip from an H2 directly to an H4. Each level should build on the one before it, maintaining a structured flow throughout the page.
Include Relevant Keywords
Incorporate important keywords into your headings to signal the topic of each section. This supports SEO by helping search engines understand what your content is about, and also makes it easier for users to find what they’re looking for.
Keep Headings Concise
Make your headings short and clear. They should summarise the content of the section they introduce without being too long or complicated. Concise headings improve readability and make your page easier to scan.
Critical Area 7: Server Errors
Then you should consider server errors. An internal server error is an error on the web server that you are trying to access. An error shows when the server is misconfigured and prevented from responding to what you're asking it to do. There are a number of different server error codes, with the most common being:
400 – Bad Request
This error occurs when the server can’t understand the request due to invalid syntax. It’s often caused by a broken link, a malformed URL, or incorrect request formatting.
401 – Unauthorised
The 401 error means that the user is not authorised to access the requested resource. It usually requires login credentials that are either missing or incorrect.
403 – Forbidden
A 403 error appears when access to the resource is denied. Even if the user is logged in, they don’t have permission to view the content.
404 – Not Found
This is one of the most well-known errors. It means the server can’t find the requested page or resource. This might be due to a mistyped URL or a page that has been removed.
500 – Internal Server Error
A 500 error indicates a problem with the server itself. It’s a general error when the server encounters something it doesn’t know how to handle.
502 – Bad Gateway
This error occurs when a server acting as a gateway or proxy receives an invalid response from an upstream server. It often relates to temporary issues with another server in the chain.
503 – Service Unavailable
The 503 error means the server is temporarily unavailable, often due to maintenance or being overloaded. It usually resolves once the server is back online.
504 – Gateway Timeout
A 504 error happens when a server acting as a gateway or proxy doesn’t get a response from the upstream server in time. It can indicate network delays or server timeouts.
Here you can find the entire list of HTTP response status codes. Where possible, the use of relevant, ideally unique, anchor text should be used on all links - rather than simply showing the URL.
Critical Area 8: Links
After this, links become a key consideration and the importance of backlinks is high. The 2 types of links - internal and external both are key to SEO strategies but ultimately serve very different purposes.
In layman's terms, an internal link is one that points to another page on the same website. They help search engine robots crawl your website more easily and efficiently. Internal links are useful for three reasons:
Reason 1: Easy Navigation of a Website
Internal links help users move around your website smoothly. By linking to related content, visitors can find what they’re looking for more easily, spend more time on your site, and engage more with your content.
Reason 2: An Informational Hierarchy Is Established
Internal links help create a clear structure for your website. They show the relationship between different pages and organise your content from general to specific. This makes it easier for search engines to crawl and understand your site’s layout.
Reason 3: Link Equity Is Shared Throughout the Website
When one page on your site earns authority (or "link equity"), internal links help pass some of that value to other pages. This can improve the rankings of less visible pages and strengthen the SEO performance of your entire website.
An external link is when another website puts a backlink pointing to your website. Securing these links are believed to help tremendously when looking to rank in SERPs as search engines use links as a signal of reliability. The more quality and relevant links you have the more Google, Bing etc accept you as a credible, authoritative source and something they should be displayed more prominently on their results pages.
Critical Area 9: Indexation
Following the crawl by search engines, the next step is the indexing of the website in SERPs. A key part of this is the size of your website and how much search engines are going to have to add to their database. In order to review and reduce your indexation size, consider looking at:
Assess Performing vs Non-Performing Pages
Start by reviewing which pages are driving traffic, engagement, or conversions. If a page isn’t performing and adds no real value - such as duplicate content, outdated posts, or thin pages - it might be worth removing it or setting it to noindex. Focus on keeping only pages that serve a purpose for users or search engines.
Optimise or Consolidate Content
Look for pages that could be improved rather than removed. Can the content be updated, rewritten, or merged with a similar page? Reducing thin or redundant content helps boost the overall quality of your site and improves crawl efficiency.
Review Your XML Sitemap
Your XML sitemap should accurately reflect the pages you want indexed. Remove any URLs that point to low-value content, redirects, broken pages, or sections not meant for search engines (like admin or login pages). An up-to-date and clean sitemap ensures search engines prioritise the right content when crawling your site.
Critical Area 10: Duplicate Content
Finally, consider duplicate content as this is classed as content that appears on another website in more than one place. Duplicate content generally refers to sections of content within or across domains that either directly match.
In order to bypass duplicate content issues, there are a number of ways that Google recommend to ensure that search engines and users see the content you want them to see including:
Use 301 Redirects
A 301 redirect is a permanent redirect from one URL to another. If you have duplicate pages or content that shouldn’t be indexed, use a 301 redirect to send users and search engines to the correct page. This ensures that any link equity is passed along to the correct page and avoids splitting your content’s authority.
Use Top-Level Domains
If your website targets multiple countries or regions, it’s a good idea to use top-level domains (TLDs) that are specific to each country (like .co.uk, .fr, or .de). This can help Google understand which content is intended for which audience, reducing the chances of duplicate content across regions. Alternatively, you can use hreflang tags to specify language and regional targeting.
Minimise Boilerplate/CTA Copy
Boilerplate copy and repetitive call-to-action (CTA) elements can be a source of duplicate content. While it's common to use similar copy across multiple pages (like footers or sidebars), it’s best to minimise excessive repetition and keep it as unique as possible. Avoid duplicating long sections of content across pages to prevent it from being flagged by search engines.
Minimise Similar Content
If you have multiple pages with highly similar content (for example, product pages with only slight variations), try to merge or consolidate them. Alternatively, use canonical tags to tell search engines which version of the page should be considered the primary one. This helps ensure that Google indexes the most relevant version and avoids penalising your site for duplicate content.
Advanced Technical SEO
Critical Area 11: JavaScript SEO
JavaScript SEO focuses on ensuring that search engines can properly crawl, index, and render JavaScript content. As many modern websites rely on JavaScript for dynamic content, optimising how search engines interact with JS is essential for maintaining visibility in search results. Techniques such as server-side rendering (SSR) or static rendering (SSG) can help improve crawlability.
Critical Area 12: Schema-Markup
Schema Markup, or structured data, is code added to a website using a standard vocabulary from Schema.org to help search engines better understand and display content, such as products, reviews, or events. By enhancing how pages appear in search results - like showing star ratings or prices - it can boost visibility through rich snippets, support voice search, and improve indexing and relevance. This can lead to higher click-through rates, making it particularly valuable for e-commerce and local business sites aiming to stand out.
Critical Area 13: International SEO
International SEO optimises a website to rank across multiple regions and languages. This involves technical setups such as using hreflang tags to signal to search engines which version of your content is intended for which audience. Proper country and language targeting, along with local hosting or domain extensions (e.g., .co.uk, .fr), can also improve rankings.
Learn more about securing top listings in other countries in our detailed guide to international SEO.
Critical Area 14: Site Architecture Optimisation
Site architecture optimisation ensures that search engines can crawl and index a website’s content efficiently. This involves creating a logical structure, using internal linking to distribute link equity, and ensuring that important pages are easily accessible. Clear, concise URLs, a clean navigation system, and XML sitemaps all play a role.
Securing those top search engine ranking positions
There are, of course, many areas of Search Engine Optimisation that need to be continuously tweaked to fall in line with Google’s latest algorithm, so ensuring that the technical basics are covered, will help your website perform in the all-important SERPs. Securing top search engine rankings involves optimising your website through a mix of using our technical SEO tips and making strategic content adjustments. By focusing on aspects like improving crawlability, fixing broken links, optimising Core Web Vitals and using structured data, you can enhance search engine visibility and user experience. Employing tools like Google Search Console and Screaming Frog ensures efficient site audits and resolution of critical issues, setting the foundation for higher rankings and sustained traffic growth.
Our 6 Key Technical SEO Tips
To enhance your website's technical performance and search engine visibility, consider these proven best practices:
Tip 1: Leverage Key SEO Tools
Use platforms like Google Search Console to track indexing issues, identify crawl errors, and optimise search performance. Tools like Screaming Frog can assist with in-depth technical audits, flagging issues like broken links, duplicate content, and redirect issues and chains.
Tip 2: Prioritise Core Web Vitals
Core Web Vitals are essential metrics for assessing page speed, interactivity, and visual stability. Tools like PageSpeed Insights can help identify and resolve performance bottlenecks.
Tip 3: Fix Broken Links
Broken internal and external links disrupt user experience and SEO. Regularly audit your site with tools like Ahrefs or SEMrush to detect and correct broken links.
Tip 4: Optimise robots.txt and XML Sitemaps
Ensure your robots.txt file allows search engines to crawl vital pages while blocking irrelevant ones. An updated XML sitemap should reflect your site's current structure, helping search engines index content effectively.
Tip 5: Implement Structured Data
Use schema markup to provide search engines with detailed information about your content, improving visibility through rich snippets.
Tip 6: Regularly Audit Technical SEO
Frequent audits allow you to catch and fix emerging issues. Evaluate crawl budgets, page speed, and indexation regularly to keep your website in top shape.
Technical SEO tips: How to Do An Technical SEO Audit
Conducting a technical SEO audit is crucial for improving your site's performance and visibility. Follow this simple step-by-step guide to get started with a basic review of your website's current performance:
Step 1. Set Up Essential SEO Tools
To begin your audit, it's important to set up essential SEO tools that will provide the data you need to identify any issues and improve your site's performance. Google Search Console is a must-have for monitoring crawl errors, indexing issues, and tracking Core Web Vitals performance. Google Analytics will give you insights into user behaviour and traffic sources, helping you understand how visitors interact with your site. For a more in-depth audit, tools like Screaming Frog SEO Spider are invaluable for scanning your site and finding problems like broken links, duplicate content, and meta tag issues that could affect your on-page technical SEO.
Step 2. Check for HTTPS Implementation
One of the fundamental aspects of technical SEO is ensuring that you have a secure website. This means your site should be running on HTTPS rather than HTTP, as Google uses HTTPS as a ranking signal. If your website isn’t secure, visitors may also see a warning in their browser. To check for any issues, use tools like WhyNoPadlock to identify any mixed content problems - such as images or scripts that are still being served over HTTP - so you can ensure all content is loaded securely.
Step 3. Audit Page Speed
Page speed is a crucial component of technical SEO and is directly linked to both user experience and search engine rankings. Use tools like Google PageSpeed Insights or GTmetrix to analyse your site’s load times and identify any bottlenecks. Focus on improving your site’s speed by compressing large images, enabling browser caching, and using Content Delivery Networks (CDNs) to speed up content delivery globally. A fast-loading website is not only beneficial for rankings but also helps reduce bounce rates and improve user engagement.
Step 4. Review Mobile Friendliness
With the majority of web traffic now coming from mobile devices, it’s essential to ensure that your website is optimised for mobile users. Use Google’s Mobile-Friendly Test to check your site’s mobile optimisation and usability. If the tool detects any issues, it could mean your site isn’t providing a seamless experience for mobile users, which can harm your rankings. Mobile-first indexing is a key consideration for modern SEO, so make sure your site is easy to navigate and fully functional on mobile devices.
Step 5. Analyse Crawlability
Crawlability is a fundamental aspect of on-page technical SEO, as search engines need to be able to access and index your content. Start by checking your robots.txt file to ensure that no important pages are being blocked from being crawled by search engines. Tools like Screaming Frog SEO Spider can help generate a comprehensive crawl report, highlighting any issues with blocked or orphaned pages. Orphan pages are those that aren’t linked to any other pages, making it difficult for both users and search engines to discover them. By fixing crawlability issues, you help search engines index your content properly.
Step 6. Optimise XML Sitemaps
Your XML sitemap is a crucial element of technical SEO that helps search engines discover all the important pages on your site. Ensure your XML sitemap is up to date and includes only the most valuable content, removing any pages that shouldn’t be indexed. After reviewing and updating your sitemap, submit it to Google Search Console and Bing Webmaster Tools to make sure search engines have the most accurate information about your site’s structure.
Step 7. Fix Broken Links
Broken links can frustrate users and affect your site’s credibility. They also waste search engine crawl budget, which could be better spent crawling pages that are important for SEO. Use tools like Ahrefs or SEMrush to identify and fix both internal and external broken links. Repairing these links helps maintain a smooth user experience and ensures search engines can easily navigate and index your site.
Step 8. Evaluate On-Page Elements
One of the final steps in your technical SEO audit involves reviewing your on-page elements to make sure they follow SEO best practices. Check that your title tags and meta descriptions are optimised for relevant keywords, and ensure that each page has a clear, descriptive header tag structure. H1 tags should reflect the main topic of the page, while H2 and H3 tags can be used for subheadings and sections within the content. Additionally, make sure all images on your site have relevant alt text, as this helps with image search rankings and accessibility.
Step 9. Check for Duplicate Content
As part of technical SEO tips, it’s crucial to regularly check for duplicate content across your website. Tools like Siteliner or Copyscape can help you identify pages that contain duplicate or near-duplicate content. Once identified, you can take action to either consolidate those pages or apply appropriate redirects or canonical tags. Resolving duplicate content issues is vital, as search engines may penalise sites that present too much repetitive content, leading to a drop in rankings. Keeping your content unique and focused is one of the foundational components of technical SEO.
Step 10. Implement Structured Data
Structured data, or schema markup, is another crucial element in modern SEO. By adding schema to your website, you provide search engines with extra information about your content, which helps them understand it better. This, in turn, can improve the way your site appears in search results - such as rich snippets, featured snippets, or knowledge panels. To implement structured data, you can use tools like Schema.org for guidance or Google’s Structured Data Testing Tool to validate your markup. This helps ensure that search engines understand your content’s context and display it properly, ultimately improving your click-through rates and user engagement. Structured data is a key part of both technical SEO and on-page technical SEO, contributing to better site performance and visibility.
Technical SEO Tools & Performance Monitoring Tools
Technical SEO tools are essential for uncovering and fixing backend issues that can impact how search engines crawl, index, and rank your website. There are many tools that can be used in technical SEO, which either help with the performance of the website or with monitoring the performance of a website.
Performance Optimisation tools include:
Google Search Console
Google Search Console is a fundamental free tool from Google that lets you monitor your site’s performance in search results. It provides insights into index coverage, crawl errors, submitted sitemaps, mobile usability, and Core Web Vitals performance.
Screaming Frog SEO Spider
The Screaming Frog SEO Spider is a powerful desktop crawler that mimics how search engines navigate your site. It’s widely used to find broken links, audit redirects, analyse meta data, check for duplicate content, and much more.
Ahrefs Webmaster Tools
Ahrefs Webmaster Tools offers a free site audit and backlink analysis solution. It identifies technical SEO issues, highlights internal linking opportunities, and provides reports that are easy to digest and act upon.
Sitebulb
Sitebulb is a user-friendly desktop SEO audit tool that turns complex data into clear visualisations. It’s especially handy for viewing site architecture and diagnosing crawlability, content, and Core Web Vitals issues.
Lumar (formerly DeepCrawl)
Lumar, previously known as DeepCrawl, is a cloud-based enterprise-grade crawler. It provides ongoing technical SEO monitoring, issue prioritisation, and integrates with team workflows to support collaboration.
JetOctopus
JetOctopus is a cloud-based SEO crawler and log analyser that specialises in visualising crawl behavior, log files, and JavaScript rendering. It’s a go-to for advanced SEO professionals working on large, complex websites.
Google PageSpeed Insights
PageSpeed Insights analyses a page’s performance on both mobile and desktop devices, providing real-world (field) and simulated (lab) data. It scores Core Web Vitals and offers actionable tips for improving load speed and user experience.
Google Lighthouse
Google Lighthouse is an open-source tool built into Chrome DevTools. It audits your site for performance, accessibility, SEO, and best practices, and is useful for diagnosing issues affecting Core Web Vitals.
WebPageTest
WebPageTest is a robust testing platform for website speed and performance. It delivers in-depth data such as waterfall charts, time-to-first-byte, and filmstrip comparisons to help uncover load time bottlenecks.
GTmetrix
GTmetrix combines Lighthouse and custom performance metrics to help you track site speed over time. It also allows for performance monitoring from different locations and devices, which is great for global SEO strategies.
Chrome User Experience Report (CrUX)
The Chrome User Experience Report (CrUX) provides anonymised real-user data collected from Chrome users across the web. It’s a trusted source for measuring how real people experience your site in terms of speed, responsiveness, and visual stability.
Performance Monitoring Tools include:
SEMrush
SEMrush is a powerful all-in-one digital marketing platform widely used for SEO, PPC, content marketing, and competitor research. It offers keyword tracking, backlink analysis, site audits, traffic analytics, and on-page SEO recommendations—all from a single dashboard. SEMrush is ideal for businesses, marketers, and agencies looking to manage all aspects of their SEO strategy in one place.
Ahrefs
Ahrefs is a well-respected SEO toolset best known for its robust backlink analysis and keyword research capabilities. With features like Site Explorer, Content Explorer, and Rank Tracker, it helps users identify what’s working for competitors and discover high-potential opportunities to improve rankings. It's particularly popular among content marketers and link-building professionals.
Moz Pro
Moz Pro offers a suite of SEO tools to help you improve search rankings and visibility. It includes a keyword explorer, site audits, rank tracking, and a unique metric called Domain Authority (DA). Moz also provides on-page SEO suggestions and link analysis, making it a strong choice for marketers and website owners focused on improving organic performance.
Serpstat
Serpstat is an affordable, feature-rich SEO platform ideal for teams and agencies. It covers keyword research, site audits, backlink analysis, and rank tracking, all in one interface. With a clean layout and actionable insights, Serpstat helps you monitor SEO health and track competitors with ease.
SpyFu
SpyFu excels at uncovering your competitors’ strategies, especially in paid and organic search. It shows every keyword a competitor has bought on Google Ads, along with their top-ranking organic keywords. Ideal for competitive analysis, SpyFu is great for businesses that want to improve their paid search ROI and identify untapped SEO opportunities.
How can Bidmark help…
These top technical SEO tips, grounded in technical SEO basics, can significantly improve visibility and user experience, so using our technical SEO tips to help boost your website. If you need further help our team of SEO optimisation experts based in Leeds can help you understand how your website is currently performing and how you can further optimise your website to ensure optimum performance. Contact us if these technical SEO tips havent answered your question or to discuss how our technical onsite SEO services could help you conquer your targeted search results.
Technical SEO TIPS FAQs
Why is Technical SEO Important?
Technical SEO ensures that your website is accessible, well-structured, and optimised for both users and search engines. It lays the foundation for strong on-page SEO by addressing the technical issues that can impact rankings, like slow load times or crawlability issues. Without solid technical SEO, even great content might not perform well in search results.
How Do I Check if My Site is Mobile-Friendly?
To check if your website is mobile-friendly, use Google’s Mobile-Friendly Test. This tool evaluates how well your site performs on mobile devices and provides suggestions for improvement. Mobile-friendliness is a key component of modern SEO, as Google uses mobile-first indexing, meaning it prioritises the mobile version of your site for ranking.
What is an XML Sitemap and Why is it Important?
An XML sitemap is a file that lists all the important pages of your website, helping search engines discover and crawl them. It’s crucial for ensuring that search engines index the right content, especially if your website is large or has complex navigation. Submitting an XML sitemap to Google Search Console and other search engines makes it easier for them to find and index your content.
What is a Robots.txt File?
The robots.txt file is a text file placed in the root directory of your website that tells search engines which pages or sections of your site they are allowed to crawl and index. It's important to check your robots.txt file regularly to ensure that important pages aren't accidentally blocked from being crawled.
