# Every public page is prerendered: content pages in full, data pages with the # night's data under their own head (a stock's tabs are self-canonical, like its # hub). All of it is public and free; the api that fills the data pages is a separate host # (api.sweetvaluelab.com) that declines indexing on its own. # # CONTENT SIGNALS (contentsignals.org — the vocabulary Cloudflare's AI Crawl # Control reads). Three independent signals, declared per user-agent group: # # search=yes index these pages and link to them # ai-input=yes quote them as the input to an AI answer. Being cited IS the # point, so this is the signal we most want honoured # ai-train=yes keeping them as training or fine-tuning corpus is welcome too. # A model that has read the numbers is one more place the site # is known from; the data is public and free either way # # A signal says how the content may be USED; it is not access control, and the # two are deliberately kept apart: this file ALLOWS every crawler below, because # a page that cannot be fetched cannot be cited either. Enforcement lives in # Cloudflare (Security -> AI Crawl Control). It blocked the training group at the # bottom at the edge from 2026-10-06; since 2026-10-18 it lets every crawler # through, and the signal below says yes to training to match. The two halves # must say the same thing: if Cloudflare starts blocking a group again, turn this # signal back to `ai-train=no` in the same change. # # One group per crawler: robots.txt gives a crawler only its MOST SPECIFIC # matching group, so each group below restates the signal and the `/game` # exclusion rather than inheriting them from `User-agent: *`. # # `/game` is a client-rendered trading game whose page is an empty app shell # (`noindex`, #517) — nothing a crawler could index. Until it earns a real page # in PUBLIC_PAGES, keep crawlers out of it entirely. # Search engines and anything not named below. User-agent: * Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /game # AI search: crawlers that build the index an assistant answers from, and that # carry a link back. Same terms as a search engine. User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: Applebot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /game # AI assistants fetching a page because a person just asked for it. The closest # thing to a reader we have; nothing here is behind a paywall or a session. User-agent: ChatGPT-User User-agent: Claude-User User-agent: Perplexity-User User-agent: DuckAssistBot Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /game # Training corpora. Allowed and welcome, like every group above; they keep a # group of their own because readiness scanners look for explicit rules per AI # crawler, and because this is the group a block would go in. `Google-Extended` and `Applebot-Extended` are control tokens rather # than crawlers -- they carry no traffic of their own and exist only to be # answered here. User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot User-agent: meta-externalagent User-agent: Bytespider Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /game Sitemap: https://sweetvaluelab.com/sitemap.xml # Not a directive — robots.txt has no keyword for this, and an unknown one is # ignored rather than honoured. It is here because this is the file an agent (or # the person auditing one) reads first, and until now /llms.txt could only be # found by guessing the filename: https://sweetvaluelab.com/llms.txt — what each # family of pages is, generated with the site. Every page's head also carries # ``.