{"id":1255,"date":"2024-02-15T11:03:11","date_gmt":"2024-02-15T16:03:11","guid":{"rendered":"https:\/\/www.blindemanwebsites.com\/today-i-learned\/?p=1255"},"modified":"2024-02-27T19:17:16","modified_gmt":"2024-02-28T00:17:16","slug":"robots-txt-and-ai-web-crawlers","status":"publish","type":"post","link":"https:\/\/www.blindemanwebsites.com\/today-i-learned\/2024\/robots-txt-and-ai-web-crawlers\/","title":{"rendered":"robots.txt and AI web crawlers"},"content":{"rendered":"\n<p>An interesting article from The Verge on robots.txt and AI web crawlers and who is still respecting robots.txt. The comment section I thought was interesting too.<\/p>\n\n\n\n<p><a href=\"https:\/\/www.theverge.com\/24067997\/robots-txt-ai-text-file-web-crawlers-spiders\">https:\/\/www.theverge.com\/24067997\/robots-txt-ai-text-file-web-crawlers-spiders<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>An interesting article from The Verge on robots.txt and AI web crawlers and who is still respecting robots.txt. The comment section I thought was interesting too. https:\/\/www.theverge.com\/24067997\/robots-txt-ai-text-file-web-crawlers-spiders<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[224,225,226,227],"class_list":["post-1255","post","type-post","status-publish","format-standard","hentry","category-new-things","tag-ai","tag-robots-txt","tag-search-engines","tag-the-verge","infinite-scroll-item"],"_links":{"self":[{"href":"https:\/\/www.blindemanwebsites.com\/today-i-learned\/wp-json\/wp\/v2\/posts\/1255","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.blindemanwebsites.com\/today-i-learned\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.blindemanwebsites.com\/today-i-learned\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.blindemanwebsites.com\/today-i-learned\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.blindemanwebsites.com\/today-i-learned\/wp-json\/wp\/v2\/comments?post=1255"}],"version-history":[{"count":2,"href":"https:\/\/www.blindemanwebsites.com\/today-i-learned\/wp-json\/wp\/v2\/posts\/1255\/revisions"}],"predecessor-version":[{"id":1259,"href":"https:\/\/www.blindemanwebsites.com\/today-i-learned\/wp-json\/wp\/v2\/posts\/1255\/revisions\/1259"}],"wp:attachment":[{"href":"https:\/\/www.blindemanwebsites.com\/today-i-learned\/wp-json\/wp\/v2\/media?parent=1255"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.blindemanwebsites.com\/today-i-learned\/wp-json\/wp\/v2\/categories?post=1255"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.blindemanwebsites.com\/today-i-learned\/wp-json\/wp\/v2\/tags?post=1255"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}