A security flaw found in H.323, the most widely deployed VOIP communications protocol worldwide, can cause VOIP networks to shut down, but it’s only a theoretical problem for now, experts say.

VOIP carriers on Wednesday said they were scrambling to patch the vulnerability, which was first brought to light yesterday by the National Infrastructure Security Coordination Center (NISCC), a U.K. government agency that was continuing work started by the University of Oulu Security Programming Group (OUSPG) in Finland. It was also covered in an advisory issued by CERT.

Paul Jones, chairman of the International Telecommunication Union (ITU) group for H.323 and a systems developer in the VOIP architecture team at Cisco says the problem is real, but its impact is minimal.

“There are specific implementation vulnerabilities in H.323-related protocols that are critical to the U.K. national infrastructure," says Victoria Eld, spokeswoman for the NISCC. "We can't give any more details on this notice for security reasons, but we are working with vendors to fix it.”

The security hole affects VOIP products from a host of vendors including Cisco Systems Inc. (Nasdaq: CSCO), Avaya Inc. (NYSE: AV), Nortel Networks Corp. (NYSE/Toronto: NT), and Radvision Ltd. (Nasdaq: RVSN), as well as Microsoft Corp.'s (Nasdaq: MSFT) Internet Security and Acceleration Server 2000, which is included with Small Business Server 2000 and 2003 editions.

Each of these companies has released security advisories describing which of their products are affected, and in most instances they've provided fixes. (For the Cisco notice, click here.)

The ITU's Jones says the hole is not a problem with the H.323 standard itself but a lower-level protocol that H.323 depends on called ASN.1 -- Packet Encoding Rules (PER) for encoding and decoding messages and handling improper messages on a VOIP network. The flaw will arise in networks that do not properly check for bad messages. Jones says a bad message, for example, might indicate to the decoding library that the next part of the message is longer than the actual length of the entire message. “In some implementations of ASN.1 the checks are not in place to spot these problems, so the library would try to allocate more memory than it should, potentially leading to buffer overruns and system crashes… It’s a problem of bad coding."

In addition, he says H.323 uses TCP to establish a connection with another H.323 device. In bad implementations, the connection is left open for long periods of time, exhausting memory resources. If all ports on a device are left open, that device could feasibly be taken out of commission, he says.

Despite the seriousness of the problem, Jones says the impact is mitigated by the fact that most VOIP systems are operated on private networks that are out of reach of most hackers who would attempt to exploit such vulnerabilities.

Martin Euchner, H.323 security expert at Siemens AG (NYSE: SI; Frankfurt: SIE), adds that ASN.1 is used in many other protocols. “X500 and X509 also rely on it,” he says. “Right now, this has come up because of systematic testing by a British lab. It’s a particular H.323 application run over a particular implementation of ANS.1 -- it may not occur in real situations, just theoretical combinations. All the same," he warns, "it must be guarded against.”

ITXC Corp. (Nasdaq: ITXC) the largest domestic VOIP carrier in the U.S., has a network that runs almost exclusively on Cisco H.323. The company has not run into this problem. “We are going in very quickly to check our equipment is patched and protected,” says John Landau, executive VP of product management at ITXC.

VOIP carriers iBasis Inc. (OTC: IBAS), China Unicom Ltd., and FastWeb SpA also run exclusively on H.323, but Jones says he has not heard any reports that these reported flaws have surfaced in their production networks.

— Jo Maitland, Senior Editor, Boardwatch