TU Delft Research

We define a hosting provider as the entity responsible for IP ranges with hosting services according to WHOIS data — we do not use Autonomous Systems (AS) as a starting point. In an earlier study, we found that there are on average seven providers per AS. In this first version of the benchmark, we have only included providers that are members of NBIP, ISPConnect, or DHPA and for whom we could retrieve the WHOIS information. That amounts to 129 providers.

We then take a number of abuse feeds [2]. Per feed, we count the number of incidents observed at each provider during the period January–August 2018.

Large providers have more incidents than small ones because they have more customers and more infrastructure. That does not mean they are less secure. The type of services also makes a difference. We therefore collect several characteristics of the providers, such as how many IP addresses they advertise, how many domains they host, and how much shared hosting they offer.

The abuse and provider data are fed into a statistical model. The model looks at the number of incidents, taking into account the size of the provider and, to some extent, the type of services in the network. It goes too far to explain exactly how the model works, but it is essentially the same as an IQ test or a standardized exam. Such a model estimates how good someone is at math based on their score distribution across the different questions. In the benchmark model, each feed is essentially a test question on which the provider scores a number of points. The model then estimates which underlying capability best explains that score distribution relative to the other “students.” It also provides a margin of uncertainty around that score — some scores are fairly robust, others have a wider margin.

The benchmark is a score that expresses where the provider stands within the total group of 129 providers according to the model. We express this score as a percentile. A provider with a score of 20 is in the 20th percentile, meaning that 20% of all providers have more abuse than this provider and 80% have less, taking into account size and type of services. A score of 20 is therefore a poor score in terms of abuse prevention — it places you among the worst 20% of the market. We use the following simple labels to communicate these results: a score between 1–20 is poor, between 20–80 is average, and between 80–100 is good.

Finally, in addition to the abuse benchmark, we have also calculated a vulnerability benchmark. This follows the same steps but uses vulnerability data instead of abuse data [3]. This benchmark is also expressed using the labels poor (1–20), average (20–80), and good (80–100). It is therefore possible for a provider to perform well in terms of abuse and poorly in terms of vulnerabilities.