본문으로 건너뛰기

SEO Tools - robots.txt, sitemap, rss

SEO Tools - robots, sitemap, rss

There are several tools in SEO that help effectively introduce the website to search engines and efficiently deliver content.

In this lesson, we will look at the most important ones: robots.txt, sitemap, and rss.


robots.txt

robots.txt is a text file within a website that tells search engines which pages to crawl or not to crawl.

Example:

Example of robots.txt
User-agent: *
Disallow: /private/
Allow: /public/

In the example above, all search engines are instructed not to access the /private/ directory and are allowed to access the /public/ directory.


Sitemap

A sitemap is an XML file that represents a list of pages on a website and their relationships.

It serves as a map showing the links within a webpage.

Example:

Example of sitemap
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://example.com/page1</loc>
<lastmod>2023-10-15</lastmod>
<changefreq>daily</changefreq>
</url>
<url>
<loc>https://example.com/page2</loc>
</url>
</urlset>

The sitemap example above lists two web pages, and page1 is also informing about the last modification date and frequency of changes.


RSS

rss is an XML file format that automatically notifies subscribers of content updates on a website. It is widely used in blogs, newsletters, and web services.

Example:

Example of rss
<?xml version="1.0"?>
<rss version="2.0">
<channel>
<title>My Website News</title>
<link>https://example.com</link>
<description>Latest updates from My Website</description>
<item>
<title>New Article</title>
<link>https://example.com/new-article</link>
<description>This is a new article.</description>
</item>
</channel>
</rss>

The example above introduces 'My Website News' with updates and the content titled 'New Article' within it.