AI agents expose revenue gaps across major websites

Most of the world's largest websites lack policies for managing AI agents that answer customer questions, complete purchases or collect content, according to research from D3 Research.

The study assessed the 10,000 highest-ranked domains and assigned a median readiness score of 56 out of 100. Among the 5,577 sites with readable policy files, 84% received a D or F grade. Only 13 achieved an A.

D3 Research found that most sites allowed AI agents to access content without deliberately granting access or recording what the agents collected. Other sites used broad blocking rules that restricted customer-facing agents alongside training crawlers and other automated services.

Traffic shift

Automated agents accounted for 57.5% of web traffic in June 2026, according to Cloudflare Radar figures cited in the study. Humans generated the remaining 42.5%.

Agent traffic grew by 7,851% over the previous year, based on figures from HUMAN Security cited by D3 Research. More than 1,500 AI services were already accessing websites.

D3 Research separated the traffic into agents that answer questions, agents that act for customers and services that collect content. It found that websites often treated these different activities as a single category.

The study said websites generally managed automated traffic through robots.txt files. The format was created to tell automated services which parts of a site they could access.

Those files rely on names supplied by the visiting service. D3 Research found that 95.8% of 2,175 known agents provided no additional information through which a website could verify their identity.

Poor scores

The research assigned grades using four measures. Answer eligibility and buyer access each accounted for 35 points. Content exposure contributed 20 points, while identity controls represented the final 10 points.

A total of 4,198 sites received a D grade. Another 484 received an F. There were 461 sites graded C and 421 graded B.

D3 Research said 73% of sites had no written rules covering AI traffic. These websites often remained accessible to agents but collected no evidence about which services entered or what content they accessed.

Of the 100 largest websites examined, 93% received a D or F.

Nvidia recorded a score of 93 and received an A. Traveloka scored 90. Google, Apple and Microsoft each scored 56, while Amazon scored 27.

The results were based on the policies visible when the sites were scanned. Individual scores may change when a company updates its robots.txt file.

Customer access

The study found that 693 sites blocked at least one agent capable of shopping or acting for a customer. This represented around one in eight of the sites with readable policies.

Only 151 of those sites had named and deliberately blocked the relevant agents. The remaining 542 blocked customer-facing agents through older blanket rules or inconsistent vendor-specific controls.

A total of 373 sites blocked every shopping agent assessed. Another 320 allowed an agent from one provider while blocking an equivalent service from another.

Just 125 sites had policies deliberately welcoming a shopping agent. This represented 2.2% of the sample. Another 4,759 sites neither explicitly permitted nor blocked those agents.

Online marketplaces recorded the weakest purchasing access among the commercial sectors examined. Forty-three per cent blocked at least one shopping agent. The sector received an average score of 44.

News organisations also imposed extensive restrictions. Among 177 newspaper sites assessed for purchasing access, 37% turned away at least one customer-facing agent.

Policy gaps

D3 Research found that companies had written more policies for services collecting training data than for agents carrying out tasks for customers.

GPTBot was covered by policies at 1,215 sites. CCBot appeared in 1,076 policies and ClaudeBot in 1,043.

ChatGPT-User, which acts for an individual, appeared in 642 policies. Perplexity-User appeared in 305, while Claude-User appeared in 280.

Among the 1,508 sites that had established a deliberate AI policy, 59.1% blocked at least one service that answered customer questions. Almost a third blocked a shopping agent.

The company-specific approach also produced uneven treatment. D3 Research identified 750 sites that allowed certain answer engines while blocking others. It found 418 sites with rules for OpenAI's customer-facing services but no equivalent policy for Anthropic.

Content records

The study found that 4,114 sites, or 74% of the readable sample, kept no policy record covering companies that collected their content.

Only 225 sites had rules covering every content-collecting company assessed. D3 Research said such records could help organisations determine which services accessed their material and whether those services generated customer referrals.

Technology sites allowed customers to reach them in 77% of cases, while 7% recorded any content exposure. Five per cent blocked a buyer.

E-commerce sites recorded content exposure more frequently, at 25%. Their customer accessibility rate was lower at 64%, and 21% turned away a buyer.

The study measured declared policies rather than live website traffic. It excluded network-level restrictions, security controls and other enforcement mechanisms that were absent from the published robots.txt files.