A small cybersecurity firm claimed this summer to have uncovered a scam by Russian Internet thieves to amass a mountain of stolen information from 420,000 Web and FTP sites. The hacker network, dubbed “CyberVor,” possessed 1.2 billion unique credentials—a user name and matching password—belonging to 500 million e-mail addresses, asserted Hold Security, LLC.

Those numbers made Internet security watchers and even some consumers sit up and take notice—people use such credentials to access banking, investment and social media accounts after all. If true, the CyberVor haul would dwarf last December’s data breach of retailer Target, in which 40 million customer credit cards were compromised. Although a New York Times story lent credibility to Hold Security’s claims, some observers question whether the cybersecurity vendor’s big reveal was more of a publicity stunt than a public service. The firm’s decision to charge potential victims a $120 fee for their Breach Notification Service did not help matters.

Panic and publicity certainly play a role in cybersecurity efforts, as companies that make antivirus and other protective software try to provide computer users with a sense of the unseen threats facing their devices and data on a daily basis. But questions arise when these companies yoke together the part of their businesses that finds and analyzes security threats with the part that sells software and services to mitigate those threats.

Even large, established firms such as Symantec Corp. have been accused of exaggerating the gravity of security threats to boost sales. A decade ago U.S. regulators cracked down on financial services firms for the questionable practice of having their equity research and investment banking divisions work together to endorse and then sell certain investments. No such oversight exists for cybersecurity companies. Although not surprising, given the relatively nascent nature of cyber threats, this conflict of interest means these companies walk a thin line between defending computers and other Internet-connected devices and profiting from people’s fear that their personal data is vulnerable at any time to online attackers.

The people behind CyberVor started off buying stolen credentials on the black market, according to Hold Security. At some point the gang changed tactics and purchased information about certain Web sites’ security weaknesses. This data had been gleaned by botnets, a group of virus-infected computers secretly controlled by a hacker. CyberVor then used these vulnerabilities to steal more credentials from the sites. It is unclear, however, how long this process took or whether those credentials were ever used to commit fraud or steal from the credentials’ owners.

Hold Security did not respond to a media inquiry from Scientific American. But a FAQ page on the company’s Web site insisted that private e-mail users would not have to pay $120 to find out if they were on the CyberVor list. Only Web site owners and Internet service providers would be charged for the Breach Notification Service, which “also provides a full year of service for other breach notifications.”

Hold Security’s actions exposed key flaws in the way certain cybersecurity companies operate but were not emblematic of the industry as a whole, some cybersecurity executives argue. “There’s always some skepticism about the latest cyber crime of the day,” acknowledges Michael Malloy, executive vice president of products and strategy at Webroot, a provider of antivirus and antispyware services as well as research. Still, cyber crime is a real threat—large antivirus software makers must defend their customers’ computers from hundreds of thousands of new threats everyday, he says.

One red flag concerning the CyberVor warning was the lack of specifics about how much of a threat the hacker network posed and whom it affected. Hold Security begged off naming targeted Web sites, citing nondisclosure agreements, says Jasper Graham, a former U.S. National Security Agency technical director and now senior vice president of cyber technologies and analytics at network security provider Darktrace. But security companies usually offer breakdowns “that'll say something like 30 percent of the data was financial…60 percent of the data was social media,” he adds. These figures help other cybersecurity researchers and their customers assess the seriousness of the attack for themselves, but Hold Security withheld even that level of detail.

It is not uncommon for cybersecurity companies to research and hype threats that could drive sales, says Dan Guido, chief executive of information security company Trail of Bits and hacker-in-residence at New York University’s Polytechnic School of Engineering. The cybersecurity market and its customers would benefit from government regulation and research to help buyers separate good and bad security products and get objective reports on emerging cyber threats, he says.

And the same companies selling security engage in outright fraud. In March 2013 Symantec agreed to pay a settlement of $11 million to end a class-action lawsuit. The Norton antivirus program developer had been accused of tricking users into buying an unnecessary “registry cleaner” for $29.95 after running free scans with Symantec tools that identified nonexistent problems on their computers. Similarly, tiny companies such as MediaFire, Alpha Red and Branch Software have been sued for using pop-up advertisements and other tactics to scare unsuspecting computer users into buying antivirus products, some of which actually damaged consumers’ computers, once installed.*

Cybersecurity firms ideally should perform their work of finding and stopping malware and cyber attacks in a way that enables measurement of their success or failure, says Stefan Savage, a computer scientist in the Systems and Networking Group at the University of California, San Diego. Establishing a method by which people could determine and assess the quality and value of a product or service, he says, would go a long way toward helping the industry shake concerns that companies are simply resorting to scare tactics.

*Editor's Note (9/18/14): MediaFire disputes the story as originally reported in The Wall Street Journal and contends that the user in question had been victimized by one of many Chinese imposter Web sites, not by MediaFire.