"@Dan_LRMobile: On #VoLTE no one can hear you scream! http://add.vc/ddT via @Light_Reading" < or rather everyone ...
2014: A VoLTE Security Nightmare?
NEW YORK – Mobile Network Security Strategies – On Voice-over-LTE, no one can hear you scream.
Upcoming 4G packet voice services were highlighted as a significant new security risk in a panel discussion Thursday in New York on threats to the LTE network, devices, and users. Speakers from Cloudmark Inc. , Juniper Networks Inc. (NYSE: JNPR), Nokia Networks , and Symantec Corp. (Nasdaq: SYMC) all agreed that VoLTE has the potential to be a threat as the services are deployed
"It's opening a Pandora's Box," stated Basheer Nasir Ahmed, senior solutions manager at NSN.
Potential attacks include caller ID spoofing and distributed denial-of-service (DDoS) attacks. "Telephony DoS [or] TDoS," suggested John Veizades, senior product manager of mobile security at Juniper Networks, "is a can of worms."
Operators are just now starting to introduce call services over the all-IP LTE network and are planning the slow move away from circuit-switched voice calls over 3G and 2G technology.
Operators in Asia, Europe, and the US are planning to start VoLTE services over the next few years. Hong Kong operator CSL became one of the first to offer VoLTE services on Thursday. (See VoLTE Hits Hong Kong.)
— Dan Jones, Mobile Editor, Light Reading

Yeah, probably so, the topic came up on a panel discussion about different LTE threats, of which there many as far I can tell.
You are simplifying things a little.
Let's say for the moment that intercarrier calls go via TDM gateways. All that means is that the voice network dould be run as a private address space and could have a separate IP connection than the Internet connection.
Now the reality of those 2 IP presences aren't even required at the phone as the Mobile IP stack requires an intervening element to get to a pure TCP/IP handoff.
seven
The resulting bit rates are of course mostly needed for data applications. But telephone calls still need to be made, and telephony requires the kind of low-jitter low-loss QoS that best-efforts IP can't deliver. Atop that, the Internet is a sewer, subject to malware and DDoS, which needs to be kept away from the PSTN. So while it's perfectly rational to use IP within the muxing stream of the voice, it makes no sense to expose it to the Internet. Anyone who designs wireline networks understands that -- it drives a lot of MPLS, Carrier Ethernet, and other isolation technologies.
Look at PacketCable for an example. Same PMD, but QoS via time slot management, and it goes into separate private IP pipes from the pubilc Internet. Works great. What's stupid is that PacketCable 2 is trying to be more like VoLTE, by using RubeIMS, though it's still kept isolated from the script kiddies.
I like it!
Diametriq @Diametriq