[Air-l] NYTimes.com Article: Ratings Agency Says It Erred in Measuring Web Site Use

[frankthomasft at netscape.net]

This article from NYTimes.com 
has been sent to you by frankthomasft at netscape.net.


this sounds interesting: web audience mesurement agencies are not interested having independant auditors as "the internet is so different"

/Frank Thomas

frankthomasft at netscape.net


Ratings Agency Says It Erred in Measuring Web Site Use

February 24, 2003
By SAUL HANSELL 




 

ComScore Media Metrix, one of the leading companies that
measure Web site audiences, has discovered flaws in the
methodology that it introduced in October, and it has
restated its measurements for the last three months of
2002. 

Web ratings services have faced questions about accuracy
since they began trying to estimate audiences by projecting
the behavior of a panel of presumably representative users.
Despite their limits, ComScore's ratings, and those of its
main competitor, Nielsen/NetRatings, are widely used by
advertisers, investors, journalists and the Web sites
themselves. 

The biggest differences in ComScore's ratings, announced
last week, come in its estimates of Web use at the
workplace, always the most difficult to measure. Big
companies in particular do not want employees to install
the software that the ratings companies use to track Web
site usage. When ComScore adjusted its formulas to account
for the underrepresentation at big companies, its audience
projections increased, in some cases sharply. 

For instance, ComScore now estimates that Primedia's sites,
mainly those run by about.com, had 56 million users in the
United States in December, rather than the 44 million
counted under its previous formula, a 25 percent increase.
EBay's audience was increased 19 percent, to 54 million.
The estimate for Terra Lycos was increased 21 percent, to
51.6 million users. That change moved Terra Lycos to the
sixth-most-visited site in December, from seventh under the
previous numbers, putting it in front of Google. 

As with any ranking, being placed at No. 1 is an invitation
to gloat. WeatherBug, a fast-growing service of AWS
Convergence Technology, has noted in news releases that it
is "the No. 1 source for weather information on the Web,
according to ComScore Media Metrix." 

ComScore's restated data, however, shows WeatherBug second
to Weather.com, the site run by the Weather channel, owned
by Landmark Communications. 

"There was some bragging rights associated with being No.
1," said Andy Jedynak, the general manager of WeatherBug.
"To the extent we have lost those, we are disappointed." 

WeatherBug's bragging, of course, did not go down so easily
at Weather- .com, which was still listed as the No. 1
weather site by Nielsen. 

"It was annoying," the chief executive of Weather.com,
Deborah Wilson, said, " I can't say that it undermined our
sales efforts, but to have obviously bad data floating
around was a nuisance." 

The restatements result from the financial collapse last
June of Jupiter Media Metrix, a company that combined Web
audience measurement with Internet prognostication. As
Jupiter was disassembled, it sold the Media Metrix business
and its panel of Web users to ComScore, an upstart with a
different technology that was created mainly to monitor
products that Web users bought online. 

However, the software program that Media Metrix had used to
track Web audiences was not sold to ComScore but to
Nielsen, ending a patent dispute between the two companies.


ComScore very quickly had to start using its own monitoring
technology. It also expanded the size of the Media Metrix
panel of users at work, introducing in October what it
called Media Metrix 2.0. 

In response to complaints by its clients, the company has
discovered some flaws in its new system. It decided to
publish restated data for last fall rather than change its
formulas from now on. 

"We could have made these changes on a forward basis,"
Peter Daboll, president of ComScore's Media Metrix
division, said. "We felt it was the right thing to do to
make the changes to provide an accurate trend." 

Marketers and adverting experts agreed that ComScore's
approach ultimately improves the quality of the data
available. 

"It's kind of embarrassing to have to admit this," said Rex
Briggs, the principal of Marketing Evolution, an online
advertising research firm. "I applaud them for trying to
make their measurements better." 

The impact of a restatement, even of this magnitude, Mr.
Briggs added, is less now than it would have been a few
years ago, when billion-dollar deals were made on the basis
of Media Metrix ratings. 

"Stock prices are no longer tied to the number of unique
visitors you have," Mr. Briggs said. "Now investors have
this little idea of being profitable." 

Furthermore, these audience measures are not used to set
Internet advertising rates, as they are in radio and
television. Since broadcasters have no way to tell how many
people are watching a given program, they must use
Nielsen's television ratings to determine how much to
charge advertisers. On the Internet, Web sites charge to
display an advertisement a certain number of times. 

Still, some industry executives worry that the restatement
will add further doubt about the online medium, which is
still trying to establish its credibility with marketers. 

"They have some explaining to do," said Geoffrey Ramsey, an
analyst with eMarketer, a research firm that uses data from
ComScore, Nielsen and other sources. 

This uncertainty and potential undercounting of daytime
audiences is especially problematic for news and
information sites, which are used mostly by people at work.


"Daytime is prime time on the Internet," said Michael
Zimbalist, executive director of the Online Publishers
Association, a trade group. "The at-work numbers of the
ratings companies are notoriously bad." 

The biggest changes that ComScore made involved how it
estimated work audiences. But there were other issues as
well. For example, after the acquisition, ComScore changed
its method for calculating how much time users spend on
each site, in some cases cutting the numbers by far more
than half. 

Some publishers that boasted that their users spent an hour
a month on their site, an unusually high amount of time,
found the new system giving them credit for only 15
minutes. ComScore discovered that it had been allocating
some of the time that users spend on some Web pages to the
sites of advertisers rather than the Web publisher. Its
revised system changes that. 

Not surprisingly ComScore and Nielsen each insist that
their methods are more statistically rigorous. ComScore
says that its sample of people at work has 30,000 people,
far more than the 7,000 claimed by Nielsen. 

Nielsen, by contrast, says its work panel is chosen by
placing phone calls to random telephone numbers, an
expensive but methodologically superior method. ComScore
uses various other methods to recruit its work panel, like
sending e-mail messages to random names on lists it buys. 

It does not help that Nielsen and ComScore are vastly
different in their estimates of most sites' audiences.
ComScore, for example, now says Yahoo had 107 million users
in the United States in December. Nielsen counted only 81
million. 

Some of that difference is because ComScore measures usage
on college campuses and Nielsen does not. But the
difference in the estimates of Yahoo's audience, 26 million
users, far exceeds the total American dormitory population.


ComScore claims that its numbers are far closer to the
internal counts generated by Web site operators. But
William Pulver, chief executive of NetRatings Inc., the
company that publishes the Nielsen/NetRatings measurements,
argues that there are good reasons that a measurement
service will count fewer people than a site's own
computers. 

"Just because numbers are higher doesn't mean they are more
accurate," he said. 

Many advertisers and publishers, however, say they are
tired of such claims and counterclaims and want both rating
services to submit to the sort of independent audits that
are regularly conducted of the services that measure
traditional media. Both ratings services have so far
refused, saying the audits are expensive and apply
standards that are not realistic for the Internet. 

"There is no consistency over time or common guidelines
applied to either of these services," Mr. Zimbalist said.
"They need to be audited just like radio and television
ratings." 

http://www.nytimes.com/2003/02/24/technology/24NET.html?ex=1047086118&ei=1&en=6f3cb6fbecae9904



HOW TO ADVERTISE
---------------------------------
For information on advertising in e-mail newsletters 
or other creative advertising opportunities with The 
New York Times on the Web, please contact
onlinesales at nytimes.com or visit our online media 
kit at http://www.nytimes.com/adinfo

For general information about NYTimes.com, write to 
help at nytimes.com.  

Copyright 2002 The New York Times Company