ABC UK ABC UK

Site Navigation

  • ABC Data
    • Reports & Certificates
    • Buy Data Subscription
    • Reporting Standards
    • Get Audited
      • Magazines
      • Newsbrands
      • Events (ABC Audit)
      • Events (UFI Audit)
      • Digital Media
    • Media Owner Support
    • Media Buyer Support
    • Certified Suppliers
    • Data & Research Partners
    • Submit Data
  • Assurance
    • Digital Advertising
      • TAG Certifications
      • Brand Safety Certified
      • Certified Against Fraud
      • Certified Against Malware
      • Content Verification
    • Data Privacy
      • EDAA (European Interactive Digital Advertising Alliance)
      • AATP Transparency Seal
      • EDAA Trust Seal
      • MRS (Market Research Society)
    • TV Market
    • Retail Media
    • Viewability
    • COUNTER
    • Bespoke Audits
    • Spiders & Robots
    • Marketing Support
  • News
    • Latest News
    • Press Enquiries
  • About
    • About Us
    • Our Board
    • Our Team
    • Our Commitments and Accreditations
    • Our History
    • Contact Us
    • FAQs

trust in media

  • Login/Register
  • Contact us
ABC UK ABC UK
  • ABC Data
    • Reports & Certificates
    • Buy Data Subscription
    • Reporting Standards
    • Get Audited
      • Magazines
      • Newsbrands
      • Events (ABC Audit)
      • Events (UFI Audit)
      • Digital Media
    • Media Owner Support
    • Media Buyer Support
    • Certified Suppliers
    • Data & Research Partners
    • Submit Data
  • Assurance
    • Digital Advertising
      • TAG Certifications
      • Brand Safety Certified
      • Certified Against Fraud
      • Certified Against Malware
      • Content Verification
    • Data Privacy
      • EDAA (European Interactive Digital Advertising Alliance)
      • AATP Transparency Seal
      • EDAA Trust Seal
      • MRS (Market Research Society)
    • TV Market
    • Retail Media
    • Viewability
    • COUNTER
    • Bespoke Audits
    • Spiders & Robots
    • Marketing Support
  • News
    • Latest News
    • Press Enquiries
  • About
    • About Us
    • Our Board
    • Our Team
    • Our Commitments and Accreditations
    • Our History
    • Contact Us
    • FAQs
  1. Home
  2. Assurance
  3. Spiders & Robots

Ensure your data’s clean with the IAB/ABC International Spiders & Bots List

Since 2006 we’ve worked with the IAB to create this international data set. Today, we continue to feed into its maintenance, offering our Spiders & Robots service to businesses globally.

Up to 40% of web traffic is invalid*

Invalid traffic refers to any web activity that’s generated by bots, click farms, or other forms of non-human activity.

Designed to inflate website and ad campaign numbers, invalid traffic has an impact on advertising costs and can skew reported data making it difficult to accurately measure engagement and traffic sources.

*Source: CHEQ, The state of fake traffic 2023

Make sure you’re reporting valid traffic

The IAB/ABC International Spiders & Bots list helps filter known, non-human activity that can significantly inflate ad impression and site traffic counts. 

Effective use of this data is a requirement of various standards globally, and can lead to a more transparent and accurate measurement for you or your clients.

Benefits of subscribing to our Spiders & Robots Service

  • Best practice guidelines
  • Cleaner, more accurate data
  • Comply with international standards
  • Industry leading spider & robot detection and filtering
  • Technical support

About our Spiders & Robots Service

By subscribing to our Spiders and Robots Service, you’ll receive access to three data files:

1. The IAB/ABC International Spiders & Bots List

To help exclude known robotic traffic, the IAB/ABC International Spiders & Bots List contains a list of User-Agent strings previously found to have made material levels of robotic activity.

2. The IAB/ABC International list of valid browsers

Media Owners may wish to use a more rigorous robot exclusion process by adopting the dual-pass approach recommended by the IAB. This approach first checks that the User-Agent matches the include list and then excludes it if it matches the exclude list.

3. The ABC IP Address Exclusion List

ABC also publishes a list of IP addresses that have previously been found to make material levels of invalid activity. These IP addresses typically represent activity from external site monitoring tools, and from certain robots which use generic User-Agents.

Updated monthly, we’ll provide guidance on how to implement these files into your filtering process.

Annual subscriptions start from £3,474 + VAT

Enquire now

FAQ's

How is the data managed?

We work with the IAB US, the Alliance for Audited Media (AAM) and the Policy Board to deliver this service.

A Spider & Robot Policy Board oversees and approves modifications to the List which is updated monthly (on or before the 25th of each month) to reflect changes that are brought to the attention of ABC UK, AAM and the Policy Board.

AAM performs evaluation and management services associated with maintaining and publishing the industry Spider and Robot list.

Spiders & Robots Policy Board

The board is comprised of representatives from various sectors including advertising technology companies (e.g., Adobe, Criteo, Google, Yahoo), media companies (e.g., Hearst), and industry organisations (e.g., IAB Tech Lab). It oversees and approves modifications to the list. This diverse representation ensures a balanced perspective and industry-wide buy-in. To find out more about the board or if you are considering applying then please contact

What format does the data come in?

The IAB/ABC International Spiders & Bots List is made up of two text files, both are lists of User-Agents. There are two ways of implementing these lists; using the single-pass method (excluding robots only), or the dual-pass method (including valid browsers THEN excluding remaining robots) which is recommended by the IAB US.

What type of IPs are provided?

The ABC IP Address Exclusion List typically represents activity from external site monitoring tools and from certain robots which use generic User-Agents. The current IP address exclusion list is always available in 3 formats: CIDR, plain text and also in a perl-style regex format.

How do I gain access to the data?

Both the Spiders & Bots list (file approx. 40kb) and IP Exclusion list (file approx. 3kb) are available to download from this site in a text file format.

How do I submit newly discovered spiders & robots?

Email the team at ABC via

Was this page useful?

trust in media

       

ABC Data

  • Reports & Certificates
  • Reporting Standards
  • Get Audited
  • Media Owner Support
  • Media Buyers
  • Certified Suppliers

Assurance

  • Digital Advertising
  • Data Privacy
  • TV Market
  • Viewability
  • COUNTER
  • Bespoke Audits
  • Spiders & Robots
  • Marketing Support

Support

  • Contact Us
  • FAQ
  • Careers

Legal

  • Cookies
  • Privacy Policy
  • Terms & Conditions
  • Company information
  • Copyright & Database right - all rights reserved

Like some help, a quote or have a general question?

Enter your details and we’ll be in touch as soon as we can.

Alternatively visit our contact us page for more options

Contact us