Ensure your data’s clean with the IAB/ABC International Spiders & Bots List
Since 2006 we’ve worked with the IAB to create this international data set. Today, we continue to feed into its maintenance, offering our Spiders & robots service to businesses globally.
Up to 40% of web traffic is invalid*
Invalid traffic refers to any web activity that’s generated by bots, click farms, or other forms of non-human activity.
Designed to inflate website and ad campaign numbers, invalid traffic has an impact on advertising costs and can skew reported data making it difficult to accurately measure engagement and traffic sources.
*Source: CHEQ, The state of fake traffic 2023
Make sure you’re reporting valid traffic
The IAB/ABC International Spiders & Bots list helps filter known, non-human activity that can significantly inflate ad impression and site traffic counts.
Effective use of this data is a requirement of various standards globally, and can lead to a more transparent and accurate measurement for you and your clients.
Benefits of subscribing to our spiders & robots service
- Best practice guidelines
- Cleaner, more accurate data
- Comply with international standards
- Industry leading spider & robot detection and filtering:
- Technical support
About our spiders & robots service
By subscribing to our Bots and Spiders service, you’ll receive access to three data files:
1. The IAB/ABC International Spiders & Bots List
To help exclude known robotic traffic, the IAB/ABC International Spiders & Bots List contains a list of User-Agent strings previously found to have made material levels of robotic activity.
2. The IAB/ABC International list of valid browsers
Media Owners may wish to use a more rigorous robot exclusion process by adopting the dual-pass approach recommended by the IAB. This approach first checks that the User-Agent matches the include list and then excludes it if it matches the exclude list.
3. The ABC IP Address Exclusion List
ABC also publishes a list of IP addresses that have previously been found to make material levels of invalid robotic activity. These IP addresses typically represent activity from external site monitoring tools, and from certain robots which use generic User-Agents.
Updated monthly, we’ll provide guidance on how to implement these files into your filtering process.
Annual subscriptions start from £3,075 + VAT
FAQ's
How is the data managed?
We work with the IAB US, the Alliance for Audited Media (AAM) and the Policy Board to deliver this service.
A Spider & Robot Policy Board oversees and approves modifications to the List which is updated monthly (on or before the 25th of each month) to reflect changes that are brought to the attention of ABC UK, AAM and the Policy Board.
AAM performs evaluation and management services associated with maintaining and publishing the industry Spider and Robot list.
Spiders & Robots Policy Board (as of 1/11/2022)
- Richard Gibbons, ABC (UK)
- Martin Liljenback, Adobe
- Steve Guenther, Alliance for Audited Media
- Todd Martens, Alliance for Audited Media
- Antoine Comets, Criteo
- Thomas Kent, Extreme Reach
- Breen Baker, Google
- Zhong Wang, Google
- Michael Smith, Hearst
- Jaelene Price, Microsoft
- Jill Wittkopp, IAB Tech Lab
What format does the data come in?
The IAB/ABC International Spiders & Bots List is made up of two text files, both are lists of User-Agents. There are two ways of implementing these lists; using the single-pass method (excluding robots only), or the dual-pass method (including valid browsers THEN excluding remaining robots) which is recommended by the IAB US.
What type of IPs are provided?
The ABC IP Address Exclusion List typically represents activity from external site monitoring tools and from certain robots which use generic User-Agents. The current IP address exclusion list is always available in 3 formats: CIDR, plain text and also in a perl-style regex format.
How do I gain access to the data?
Both the Bots file (approx. 40kb) and IP Exclusion file (approx. 3kb) download in a text file format. Access can be direct through our website or we can set up access via FTP and or SFTP.
How do I submit newly discovered spiders & robots?
Email the team at ABC via
ABC - Audit Bureau of Circulations
207/209 High Street, Berkhamsted,
HP4 1AD, United Kingdom