User-agent: * Allow: /llms.txt Allow: /llms-full.txt # Specific AI Crawlers User-agent: GPTBot Allow: /llms.txt User-agent: Google-Extended Allow: /llms.txt