x
Optical/IP Networks

Internet Core Router Test

Juniper Wins!

It’s worth repeating: Juniper wins.

But it’s also worth noting: its archrival, Cisco Systems, ran a close second.

Here's the sound bite: After 10 years at the top, Cisco Systems no longer has to worry about the competition catching up. Now it has a new challenge: Playing catch-up to the performance of routers from rival Juniper Networks.

That’s the simple conclusion to be drawn from six months of arduous testing that pitted Juniper’s flagship M160 against Cisco’s brand-new 12416 in the first head-to-head comparison of 10-Gbit/s routers. It’s also the first time Cisco has agreed to let any of its gear be evaluated in an independent public test.

Actually, “test” is a pretty bland word for what would be considered cruel and unusual punishment in most states. Basically, we threw all the traffic on the Internet — and then some — at these bit-blasters in hopes of cutting through the white noise of vendor white papers. At every step of the way we were ably aided and abetted by our partners in crime: Network Test Inc. of Hoboken, N.J., a benchmarking and network design consultancy; and Spirent Communications of Calabasas, Calif., a test equipment supplier.

Here’s what we found:

Juniper’s M160 is the best of breed. It beat out Cisco’s product in three out of four overall areas: IP, MPLS, and OC192 (10 Gbit/s). Cisco and Juniper shared the honors in the fourth category: OC48 (2.5 Gbit/s) performance.

In some areas, the M160 is in a class by itself: It holds more BGP routes and more MPLS label-switched paths than any other box. It deals with network instability far better. And it exhibits much lower average latency and latency variation.

Specifically, the M160 outpaced the Cisco 12416 in seven out of the 16 individual tests offered, and tied for first with Cisco in five events (see Table 1). Where does that leave Cisco? With an impressive product that pulled ahead of Juniper in the four remaining tests.

Table 1: Results Summary
Charlotte's Networks Cisco Systems Foundry Networks Juniper Networks Winner
IP baseline tests: OC48 3* 1 4 2 Cisco
MPLS baseline tests: OC48 N/A 1 N/A 1 Cisco, Juniper
IP baseline tests: OC192 N/A 1 N/A 1 Cisco, Juniper
MPLS baseline tests: OC192 N/A 1 N/A 1 Cisco, Juniper
Longest-match lookup: OC48 N/A 1 2 2 Cisco
Longest-match lookup: OC192 N/A 1 N/A 2 Cisco
BGP table capacity N/A 2 3 1 Juniper
MPLS LSP capacity N/A 2 N/A 1 Juniper
Route flapping: OC48 N/A 1 3 1 Cisco, Juniper
Route flapping: OC192 N/A 2 N/A 1 Juniper
Convergence: OC48 N/A 2 3 1 Juniper
Convergence: OC192 N/A 2 3 1 Juniper
Filtering: OC48 N/A N/A 2 1 Juniper
Filtering: OC192 N/A N/A N/A 1 Juniper
Class of service: OC48 N/A 1 N/A 1 Cisco, Juniper
Class of service: OC192 N/A 1 N/A 2 Cisco
* Numbers represent relative ranking
N/A = Not applicable
OC48 = 2.5 Gbit/s
OC192 = 10 Gbit/s
BGP = Border gateway protocol
LSP = Label-switched path
MPLS = Multiprotocol label switching


For a yet more detailed breakdown of the test results, click here. There’s no doubt that the 12416, with its OC192c interfaces and 320-Gbit/s switching fabric, is a vast improvement over its GSR predecessor. Keep in mind that Cisco’s product is new — and thus less seasoned than Juniper’s M160, which has been shipping since spring 2000. In fact, Cisco’s new offering is just a memory upgrade and a couple of features away from being a serious threat to Juniper.

Now things get interesting.

There’s only one way for Cisco Systems Inc. (Nasdaq: CSCO) to take its second-place finish: personally. Over the past few years it’s watched Juniper Networks Inc. (Nasdaq: JNPR) walk away with market share (some would suggest that “run away” is a more apt description). Its stock, which once seemed to deny the laws of gravity, is in freefall. Cisco didn’t just want a win; it needed one. But the test results prove it’s not about to walk away from the core market.

So if Cisco and Juniper have at each other, does that mean other core router vendors stand to benefit from the bloodshed? Not in this market. True, there are a bunch of startups out there that claim they can deliver something the market leaders can’t. Unfortunately, what most can’t deliver is a core router. We issued our call for products to 11 vendors. There were only two other takers besides Cisco and Juniper: Charlotte’s Networks Ltd. and Foundry Networks Inc. (Nasdaq: FDRY).

How did they do? Let’s just say we were underwhelmed by the test results. Maybe Foundry got the message. After we’d finished testing its product, it announced that it was bailing out of the core router business. Smart move (see Foundry Retreats from the Core). This market belongs to our two top finishers for the foreseeable future.

Then again, both Foundry and Charlotte’s Networks deserve credit for having the guts to show up. Avici Systems Inc. (Nasdaq: AVCI; Frankfurt: BVC7), which is the number three core router player in terms of market share, had agreed to take part but, in the end, thought better of it — they didn't show.

Links to the individual sections:

The Core of the Problem
Building a Better Testbed
OC48 Throughput and Forwarding
OC192 Throughput
MPLS Throughput
Looking at Latency
Packet Ordering
Longest-Match Lookups
BGP Table Capacity
MPLS Tunnel Capacity
Route Flapping
Convergence Testing
Focus on Filtering
Quality of Service

Test Methodology
1 of 15
Next Page
dnewman 12/4/2012 | 7:31:33 PM
re: Internet Core Router Test
There is exactly one set of Light Reading results; they are posted here.

Since publication of the Light Reading test, Charlotte's Web Networks has commissioned multiple tests of its own. These are vendor-sponsored projects; as with any vendor-sponsored tests, these are neither verified nor endorsed by Light Reading or Network Test.

Regards,
David Newman
Network Test
Jeongho 12/4/2012 | 7:34:52 PM
re: Internet Core Router Test I am confusing of the facts that the differences between the test result in Charlotte's home and that in Lightreading's home comes.

Which one is latest ? The two are representing the big difference.
thanks.
balakrig 12/4/2012 | 7:44:18 PM
re: Internet Core Router Test Thanks a lot David.I really appreciate it.
dnewman 12/4/2012 | 7:45:21 PM
re: Internet Core Router Test "How many labels were used during the MPLS testing? "

Hi Ganesh,

Thanks for your inquiry. In the capacity tests, we used up to 10,000 LSPs. As noted in the article, the capacity of the devices under test is probably much higher when refresh reduction is used.

The number of labels in the baseline tests was a function of the number of interfaces on the testbed. In the OC48 testbed, each of three interfaces on each of four routers (3 * 4 = 12) forwarded traffic to each of three interfaces on three other routers (3 * 3 = 9). Ergo, we used 12 * 9 or 108 labels. In the OC48 testbed, there were 48 total edge interfaces and 36 possible destinations, so we used 48 * 36 = 1,728 labels.

Hope this helps.

Regards,
David Newman
Network Test
balakrig 12/4/2012 | 7:46:29 PM
re: Internet Core Router Test Hi,
We are in the process of writing an MPLS forwarding benchmark for the network processor forum.
My question is :
How many labels were used during the MPLS testing?
Thanks,
Ganesh
jacobi 12/4/2012 | 7:52:08 PM
re: Internet Core Router Test Dear Testers,
I'm interested in some reliability assesment regarding the core router test. Have you noted/counted any BER (of Cell Err Rate) for each of the DUTs, except the testing for out-of-order packets ?
That is, can you distinguish between intentional packet dropping and accidental one ?
Moreover, have you tried to stress the router (i.e. pull out redundant cards etc.) so it'll have to use it's redundant data path / schedulers ?

Another question is, have you checked whether a conjested router backpressures or just drops? is there a better way of handling conjestion, and if so, did you measure the difference between the router tested ?

regards,
Shimshon Jacobi
jerry33 12/4/2012 | 7:54:02 PM
re: Internet Core Router Test Skeptic wrote:
"And from what I've heard, the people who are
looking at it are not necessarly looking at
it as a core router."
--------------------------------------------------

Well how would you call a router that runs
OC-192 and OC-48 packets?

I would call it a core router!
Like Sisco and Juniper does!

And those people whom you are talking about, probably they suffer from visual or cognitive problem! LOL

regards
jerry33 12/4/2012 | 7:54:03 PM
re: Internet Core Router Test Skeptic wrote:
"There is a difference between walking through
a lab and working with a lab that you are
missing. The lab is full of equipment yes,
but every bit of it has an owner who is
constantly fighting for more (or to keep theirs
from being taken away) and corporate people
who are constantly either trying to reduce
capital spending or trying to divert equipment
from the lab to make a customer sale."

--------------------------------------------------
Again, you do'nt want us to belive that, do you?
We don't talk here about a small company, and we don't talk about a new product. We are talking of an interface that Juniper is producing regularly, and if Juniper knew long enough about the test like Mr. David Newman said, Juniper could produce some extra interfaces for the test and after the test is finished she could use the interfaces for selling! isn't it?
Unless Juniper had other reasons not to do so!
That is something I would expect you to be skeptic about. Unless you have other reasons not to do so!!!


ramborouter 12/4/2012 | 7:54:45 PM
re: Internet Core Router Test Read message 160.

hehehehehehehe
dnewman 12/4/2012 | 7:54:50 PM
re: Internet Core Router Test
Hi Rob,

We're currently conducting a comparison of 2-Gbit/s Fibre Channel switches for Light Reading/Byte & Switch, with results to be published next month. We've also agreed in principle to conduct further router and switch tests for LR, but we haven't yet committed to specific testing and publication dates.

Sorry, but we don't discuss tests for private clients.

Regards,
David Newman
Network Test
Skeptic_II 12/4/2012 | 7:55:11 PM
re: Internet Core Router Test David,
As president of your own company, you do spend a lot of time on posting messages on this board. <observation>

What are your next several projects down the road?

Thanks, Rob
</observation>
duffeck 12/4/2012 | 7:56:32 PM
re: Internet Core Router Test "I half expected .......all of them to attack the moderator)."

Damn David, I both expected and hoped.

duff
dishwasher 12/4/2012 | 7:56:51 PM
re: Internet Core Router Test Did any new (at least for non-insiders) information emerge? New product developments? New results? Did CW tell something about their OC-192?
dnewman 12/4/2012 | 7:56:51 PM
re: Internet Core Router Test "Did any new (at least for non-insiders) information emerge? New product developments? New results? Did CW tell something about their OC-192?"

Nope. CWNT presented information about its OC48, and only that. It has had OC192 for a while -- it had demo boards as far back as Interop Atlanta last fall -- but there was no mention of it at this event.

One of the audience questions was when vendors would have "time-to-market" and "real" versions of OC-768. Not surprisingly, no vendor committed to a date.

Regards,
David Newman
Network Test
dnewman 12/4/2012 | 7:56:52 PM
re: Internet Core Router Test "Did anyone attend the LR test presentation ?"

I attended. :-)

The vendor participants were *very* cordial with one another and with me -- perhaps too much so. I half expected the vendors to attack one another (or, more likely, for all of them to attack the moderator).

Didn't happen. They each went out of their way to say their competitors made excellent products.

Even when handed slow-pitch questions from the audience (e.g., "What are the biggest holes in your competitors' lines that you claim to fill?"), the vendors didn't take the bait.

Maybe it's just as well. We had a useful conversation about engineering and benchmarking issues rather than engaging in more entertaining but less constructive vendor-bashing.

I'd also note a couple of quirks: First, Foundry didn't show; no word on why.

Second, there were many requests from the audience on when the next test would be. That's not yet settled with LR, but we should be setting a date before long.

Regards,
David Newman
Network Test
null0 12/4/2012 | 7:57:00 PM
re: Internet Core Router Test Did anyone attend the LR test presentation ?

If so, any comments ?
dishwasher 12/4/2012 | 7:57:38 PM
re: Internet Core Router Test Does LR plan any reporting on the event?

Thanks.
dishwasher 12/4/2012 | 7:57:39 PM
re: Internet Core Router Test Long Haul Track:

Internet Core Routers: Who's Best?
David Newman, president of Network Test, presents the results of a ground-breaking test of Internet core routers conducted earlier this year. He also gives the results of subsequent work aimed at pinning down further details of performance issues relating to OC192 interfaces. This session will:

Quantify performance differences between the major router vendors
Analyze the relevance of these results to service providers
Demonstrate the complex issues faced by carriers selecting core routers
Show that the choice of the best router depends on the circumstances
Moderator: David Newman, President, Network Test

Speakers:

Robert Redford, VP, Marketing, Cisco
Nici Conrad, Juniper Networks, Inc.
Marshall Eisenberg, Foundry Networks
Gideon Kaempfer, CTO, Charlotte's Web

--------------------------------------------------------------------------------
prashantc 12/4/2012 | 7:57:52 PM
re: Internet Core Router Test Hi David,

Thanks very much for the prompt response. The
reason I asked these questions were because we
are currently trying to define a benchmark spec
within the Network Processing Forum and had to
tackle these issues.

Can you please email your contact information to
[email protected] I would like to
talk to you more about these issues.

Thanks.
-prashant
dnewman 12/4/2012 | 7:57:54 PM
re: Internet Core Router Test Hi Prashant,

"1. How were the routing table entries derived?
Were they based on any real world route
statistics. Real world routing tables such as
the one from Mae-East, Mae-West are about
30-50k. "

The current full table has around 100K entries, not 30K or 50K. (Tony Bates' weekly "CIDR report" on the NANOG mailing list tracks current table size.)

"Was this extrapolated to 200k? If yes,
how? If no, how was the routing table derived?"

At the time we put the test together, a full table was around 88K entries, and it was growing at a pace to hit around 200K entries within 18-36 months -- about the life span of many core routers. That's where the number 200K came from.


"2. How many entries of the 200k routing table
were excercised during the throughput test.
Current network test equipment can generate a
limited number of streams (with different IP
destination addresses). It is usually in the
order of several thousands --- couple of orders
of magnitude lower than 200k."

In the IP baseline tests, we hit each and every one of the 200K entries in the table. In the longest-match-lookup and flapping tests, the numbers were closer to 300K, and again we hit every prefix.

The Spirent SmartBits test system has a unique scripting facility that allowed us to define and exercise a virtually unlimited number of routes. This was a major criterion in our decision to use the SmartBits for this project.


"3. This is related to the above. How were the
traffic streams generated? Were the destination
IP addresses specifically programmed into the
network traffic generator? Or was the generator's
ability to "autoincrement" IP addresses used to
get different IP addresses?"

Both things are true. The SmartBits scripts take a seed value, a step value, and a number-of-increments value.

To avoid aggregation, we also used odd-numbered octets only in our prefixes: 1.x.x.x, 3.x.x.x, and so on. Since the shortest prefix we advertised was a /12, no aggregation was possible by the systems under test -- and hence they really had to hold 200K entries.

"4. What was the distribution of the routes hit
by the traffic stream used in the Throughput test?
Was it the same as the distribution of prefix
lengths in the routing table itself? Or was the
distribution skewed towards some common prefixes?"

The methodology describes the prefix length distribution models we used in the test. Note that we used slightly different models for the OC48 and OC192 testbeds.

Regards,
David Newman
Network Test
prashantc 12/4/2012 | 7:57:59 PM
re: Internet Core Router Test This message is for David Newman of Network Test.
I apologize if this has already been discussed in
this message board.

The Internet Core Router test claims to have used
200K routing entries (even scaling it to 2 million
routing entries). I have several questions about
this that I have listed below.

1. How were the routing table entries derived?
Were they based on any real world route
statistics. Real world routing tables such as
the one from Mae-East, Mae-West are about
30-50k. Was this extrapolated to 200k? If yes,
how? If no, how was the routing table derived?

2. How many entries of the 200k routing table
were excercised during the throughput test.
Current network test equipment can generate a
limited number of streams (with different IP
destination addresses). It is usually in the
order of several thousands --- couple of orders
of magnitude lower than 200k.

3. This is related to the above. How were the
traffic streams generated? Were the destination
IP addresses specifically programmed into the
network traffic generator? Or was the generator's
ability to "autoincrement" IP addresses used to
get different IP addresses?

4. What was the distribution of the routes hit
by the traffic stream used in the Throughput test?
Was it the same as the distribution of prefix
lengths in the routing table itself? Or was the
distribution skewed towards some common prefixes?

Thanks in advance for the answers.
-prashant
myhui 12/4/2012 | 8:00:03 PM
re: Internet Core Router Test Juniper's routers were built to handle multicast in hardware from the beginning, and they perform rather well in this regard.
dnewman 12/4/2012 | 8:00:51 PM
re: Internet Core Router Test "Just out of interest David, How much notice did you give to each of the candiates that took part in the last test ?

I'm guessing, you been a fair man, gave the same notice to each and every contender. So that none had an unfair advantage over the other. "

You guess correctly. Testing began in late November 2000. We began discussions with vendors in July 2000 and first issued tentative invitations in August 2000. So vendors knew about the project three to four months in advance.

Regards,
David Newman
Network Test
skeptic 12/4/2012 | 8:00:58 PM
re: Internet Core Router Test You don't really expect us to believe that, do you?

I take it you've never walked through one of the many, many LABs of a major vendor, Cisco could of probably provided you enough Juniper OC48 interfaces to fully populate 4 M160s.
-----------------------------

There is a difference between walking through
a lab and working with a lab that you are
missing. The lab is full of equipment yes,
but every bit of it has an owner who is
constantly fighting for more (or to keep theirs
from being taken away) and corporate people
who are constantly either trying to reduce
capital spending or trying to divert equipment
from the lab to make a customer sale.


fairplay 12/4/2012 | 8:01:01 PM
re: Internet Core Router Test Just out of interest David, How much notice did you give to each of the candiates that took part in the last test ?

I'm guessing, you been a fair man, gave the same notice to each and every contender. So that none had an unfair advantage over the other.

If this is the case then I can understand the argument for not being able to supply more cards as I suspect the notice was short.

What was the notice given to each candidate?

And, to anyone that may be thinking of doing any similar LR like tests in the future.

Please approach you prospective winners, nearly wins, also rans and cannon fodder/guinea pigs, sorry candidates, well in advance of the testing date so as to give them plenty of time to allocate and provide a full complement of interfaces to fully populate the chassis under test.

dnewman 12/4/2012 | 8:01:20 PM
re: Internet Core Router Test "You don't really expect us to believe that, do you?

I take it you've never walked through one of the many, many LABs of a major vendor"

Not only have I walked through those labs many times, I have many personal friends working as test engineers at Cisco and elsewhere. Regardless of vendor, the story they tell is the same: Resources are finite, and they have to contend for equipment.

Maybe it's just me, but this just doesn't seem like earth-shattering news.

Regards,
David Newman
Network Test
fairplay 12/4/2012 | 8:01:27 PM
re: Internet Core Router Test By the way, inventory shortages are not limited to startups by any means. Even the biggest companies have limited pools of test equipment. Test engineers at even the largest networking vendors have to contend for equipment, just like the rest of us.

-----------------------------
You don't really expect us to believe that, do you?

I take it you've never walked through one of the many, many LABs of a major vendor, Cisco could of probably provided you enough Juniper OC48 interfaces to fully populate 4 M160s.

Indeed, Cisco provided the Juniper kit for the recent test by mier.com
dnewman 12/4/2012 | 8:01:35 PM
re: Internet Core Router Test Hi skeptic,

"No, the real reason is that a test configuration
for a fully populated router is a very expensive
thing to put together in terms of both vendor
equipment and test equipment."

True, but in this instance test equipment wasn't the limitation. Spirent was very, very good about equipment supply and we had more test interfaces available than we actually used.

The main issue, as you suggest, is that routers on this scale cost real money, and there's just not that much inventory around.

CWNT was a case in point. When we were putting together the test methodology CWNT told us it could supply the full complement of interfaces for the OC48 test bed, but that at most it could supply "just one or two" OC192 interfaces -- far short of the 12 x OC192 and 48 x OC48 it would have needed for benchmarking on the OC192 testbed.

Note to all you CWNT conspiracy theorists out there: This is NOT a knock on CWNT. It's simply an indication that the company (correctly, in my view) delivered the few interfaces it had to revenue customers first and test labs second.

By the way, inventory shortages are not limited to startups by any means. Even the biggest companies have limited pools of test equipment. Test engineers at even the largest networking vendors have to contend for equipment, just like the rest of us.

Regards,
David Newman
Network Test
skeptic 12/4/2012 | 8:01:36 PM
re: Internet Core Router Test About your suggestion, from what I've heard, CWNT
already has a great router!
-------------
And from what I've heard, the people who are
looking at it are not necessarly looking at
it as a core router.

But unless you have seen or heard something
that isn't public, there isn't enough information
available on CWNT to know how good or bad the
product actually is.


skeptic 12/4/2012 | 8:01:36 PM
re: Internet Core Router Test Well well well. Why didn't you desing the test for full load with the maximum number of interfaces as each vendor declers that he can support?
You don't want us to belive that juniper or cisco aren't able to supply those numbers, are you?

I think that the real reason is that they knew there are problems
with full configuration.
-------------------------------------

No, the real reason is that a test configuration
for a fully populated router is a very expensive
thing to put together in terms of both vendor
equipment and test equipment. Especially for
the routers with large capacity.



psims 12/4/2012 | 8:01:39 PM
re: Internet Core Router Test But can they do multicast video? This is where the rubber meets the road.
dnewman 12/4/2012 | 8:02:06 PM
re: Internet Core Router Test Jerry33,

"you adopt a pathetic way of avoiding the answers by shifting the discussion"

I urge you, in the nicest possible way, to do two things:

1. Please reread this thread. I've spent considerable time responding to your unfounded allegations.

I do not understand what you mean by "shifting." You keep making up new accusations, and I keep refuting them.

2. Reveal your true name and affiliations. It's pretty obvious you work for and/or hold shares of CWNT, but it would sound a lot better coming from you.

Please note that I am making these requests in a respectful manner. I am not calling you names. I am not attacking your character. Instead I am merely asking that you reveal any incentive you may have for your ceaseless and unfounded attacks.

Sincerely,
David Newman
Network Test
jerry33 12/4/2012 | 8:02:10 PM
re: Internet Core Router Test Yes! You did it again.

Like other times when you were addressed with tough questions, you adopt a pathetic way of avoiding the answers by shifting the discussion towards personal matters that have nothing to do with the answer.

Let me tell you a secret, as much as it will surprise you, we live in a real world not in a bubble, and in the real world there are somtimes tough questions that should be answered even if we dont like the answer.

Your evasive way of unanswering tough questions
doesn't bring you much respect. I'm sure that the honourable members of this board deserve those answers, and I'm sure they can figure out why you don't want to answer the questions.

About your suggestion, from what I've heard, CWNT
already has a great router!

regards
dnewman 12/4/2012 | 8:02:46 PM
re: Internet Core Router Test "That's what we tested"

"Youtested what they wanted you to test!"

Nope. We tested what all the vendors -- including the one you presumably work for -- told us they were able to supply.

Regards,
David Newman
Network Test

ps. Your previous posts strongly suggest you're a employee and/or shareholder of CWNT. True? If not, it's really quite simple for you to prove otherwise. All you need to do is:

1. Come out from behind all the anonymous_coward usernames you've been hiding behind; and

2. Sign your real name and disclose any financial affiliations you may have. I've done this from the get-go; why are you afraid to?

If you are affiliated in some way with CWNT, may I respectfully suggest you spend more time creating a great router and less time crawling around message boards? It doesn't reflect well upon your benefactor, and it's something prospective investors won't approve of.
jerry33 12/4/2012 | 8:02:53 PM
re: Internet Core Router Test "That's what we tested"
Wrong. You tested what they wanted you to test! End of story (actualy a sad one).
dnewman 12/4/2012 | 8:02:59 PM
re: Internet Core Router Test "I think that the real reason is that they knew there are problems with full configuration."

Wrong. All vendors told us they'd be able to supply a given number of interfaces. All vendors agreed in advance to be tested with that number of interfaces. That's what we tested. End of story.

Regards,
David Newman
Network Test
JERRY11 12/4/2012 | 8:03:08 PM
re: Internet Core Router Test "We did ask each vendor how many interfaces they'd be able to supply for testing (as well as Spirent), and this was a very important consideration in our test design."


Well well well. Why didn't you desing the test for full load with the maximum number of interfaces as each vendor declers that he can support?
You don't want us to belive that juniper or cisco aren't able to supply those numbers, are you?

I think that the real reason is that they knew there are problems
with full configuration.
null0 12/4/2012 | 8:05:01 PM
re: Internet Core Router Test Wow, at least Cweb had the dignity of using exactly the same test setup and independant (Spirent) personell when they kicked Cisco's ass second time round. I wonder why Cisco, didn't invite cwnt along to this party?

I would suggest someone is afraid and so they should be.

dnewman 12/4/2012 | 8:05:58 PM
re: Internet Core Router Test Comparisons between the Cisco-sponsored test and the LR test are BOGUS. (When have you heard this before? :-)

There are at least three reasons right off the bat:

--The testbed in the Cisco-sponsored test was totally different than the LR testbed (for starters, one box on the testbed, vs. four in the LR tests)

--The competing vendor, Juniper Networks, did not validate the configuration of its DUT

--The test is sponsored by a vendor. Miraculously, the vendor paying the bill came out looking best.

I'm neither for nor against Cisco. I'm merely pointing out that you are making an improper comparison.

This will be my only post on the Cisco-sponsored test. Flame away, all you message board denizens. It's Friday, and I'm looking forward to the weekend.

Regards,
David Newman
Network Test
nmsguy 12/4/2012 | 8:06:06 PM
re: Internet Core Router Test Looks like someone at Cisco already thought about this - see
http://newsroom.cisco.com/dlls...

A Juniper M160, fully configured with 32 OC48 ports and being driven with 40-byte traffic, bi-directionally, suffered 28.7% packet loss - or by the LR methodology, hit only 67.1% line rate (the point at which it started to drop).

And no, I don't work for Cisco, and don't own any of their stock.

..john
dnewman 12/4/2012 | 8:06:09 PM
re: Internet Core Router Test "Something that confuses me about all the 'line rate' tests is why they seem to have been done with less-than-fully-configured systems."

Of course an ideal test would have an infinite number of interfaces, offering more traffic than any device could possibly handle. Unfortunately, we don't live in that ideal world.

We designed our experiment for a given number of interfaces, not a given number of specs from vendor data sheets. We had no advance knowledge of device capacity from Juniper or any other vendor.

We did ask each vendor how many interfaces they'd be able to supply for testing (as well as Spirent), and this was a very important consideration in our test design.

"would we continue to see the same results if every slot was fully populated ? Probably not - It'd certainly be interesting to see how they held up in this scenario."

With all due respect, Mr nmsguy, isn't nms all about *management* and *measurement*? We only manage what we can measure. Everything else is conjecture.

"It's also unclear from the methodology whether the traffic offered was uni- or bi-directional for the line rate tests."

Traffic was bidirectional and partially meshed. In other words, we offered packets to all edge interfaces. The destination addresses of all packets were not local to the router to which the packets were offered. Sorry this wasn't stated explicitly in the methodology; it should have been.

Regards,
David Newman
Network Test
hardcore router 12/4/2012 | 8:06:21 PM
re: Internet Core Router Test it convinces me that Juniper lost !!!
nmsguy 12/4/2012 | 8:06:24 PM
re: Internet Core Router Test Something that confuses me about all the 'line rate' tests is why they seem to have been done with less-than-fully-configured systems.

According to the published methodology, the systems were configured with 3 OC192s and 12 OC48s for these tests - which in Juniper's case leaves 2 slots empty, meaning it's only running at 75% of its design limit.

Since Juniper's switching fabric (shared memory) and packet processors are shared across all the line cards, would we continue to see the same results if every slot was fully populated ? Probably not - It'd certainly be interesting to see how they held up in this scenario.

It's also unclear from the methodology whether the traffic offered was uni- or bi-directional for the line rate tests.

If only uni-directional traffic was offered, then it could be argued the M160 was only loaded with about 35% of its rated capacity (75%/2).

..john
dnewman 12/4/2012 | 8:06:28 PM
re: Internet Core Router Test Hi Shimson,

"(1) Could you please explain why 100% of line rate of 40-byte IP packets comes up with only ~73Mcell/sec for a full 24-port OC48? "

That's an aggregate rate for 12 edge interfaces, not 24. Please see the schematic diagrams of the OC48 and OC192 testbeds in the methodology that accompanies the article.

Also, I have no idea how you worked out 30-40 percent of line rate. Even doubled, those percentages are way off. The 73 million aggregate pps rate with 40-byte IP packets represents 100 percent utilization of the medium once we factor for L2 overhead and interframe gaps.

(2) What is the effective BW utilization for the Imix?

In the best case, the bandwidth utilization was 100 percent for both Imix and 40-byte IP packets. This is noted in the first chart in the article on throughput.

Also, one terminology nit regarding your subject line "Cell Rate Revisited": The devices we tested used POS interfaces, not ATM ones. Regardless of what some routers may or may not do internally, all traffic enters and leaves the testbed as packets and not cells.

Regards,
David Newman
Network Test


jacobi 12/4/2012 | 8:06:37 PM
re: Internet Core Router Test Dear Testers,
(1) Could you please explain why 100% of line rate of 40-byte IP packets comes up with only ~73Mcell/sec for a full 24-port OC48? Isn't it quite low, giving effective relative BW utilization of not more than ~30-40%?
(2) What is the effective BW utilization for the Imix?

Thanks,
Shimshon Jacobi
dnewman 12/4/2012 | 8:08:10 PM
re: Internet Core Router Test As noted in the test methodology, we offered ALL traffic to OC-48 interfaces at the edge. This was true for both testbeds -- that with OC48 at the core, and that with OC192 in the core.

In both cases, we offered enough traffic to fully saturate (but not overload, except in the CoS tests) all core links.

Regards,
David Newman
Network Test

TriteReading 12/4/2012 | 8:08:30 PM
re: Internet Core Router Test Is this because SmartBits doesn't support OC-192? I wouldn't be surprised.
[email protected] 12/4/2012 | 8:08:53 PM
re: Internet Core Router Test Hi I am wondering what a phone companies need to do build a metrop optiocal netowkrs in a city?

I am not sure how much hassle there is and what kind of hurdles there are (e.g asking for yeild of rights from property owners that kind of stuff) anyone knows or can can give me pointers?

Thankx
TCT 12/4/2012 | 8:11:29 PM
re: Internet Core Router Test Visit http://www.fztct.com for more
jerry33 12/4/2012 | 8:13:55 PM
re: Internet Core Router Test You personally would buy cisco, I'm sure of that.
And that is becouse you are an old fasion gay.
All others are buying juniper and will start buying cwnt.
dnewman 12/4/2012 | 8:18:32 PM
re: Internet Core Router Test Thanks for your inquiry. Numbers like 270 million pps are aggregate statistics for the entire testbed. In the LR test, that meant four routers and a total of 48 OC-48c edge interfaces.

Regards,
David Newman
Network Test
msceller 12/4/2012 | 8:19:05 PM
re: Internet Core Router Test I was wondering if someone could explain to me how the Juniper M160 could forward 270Mpps if Juniper only rates the system at 4 switch fabics x 40Mpps = 160Mpps?

Also, does anyone know what the forwarding rate or switching capacity is of each of the Cisco 12410's 4 switch fabrics?
perry1961 12/4/2012 | 8:19:54 PM
re: Internet Core Router Test You ain't buying squat IDboy.Not stocks,and certainly not routers.
Your job is to sit in mom's trailer house wondering why MRVC would fire the worlds best toilet cleaner in the first place....
The rest of us have figured it out already,LOL.
larrymoeandcurly 12/4/2012 | 8:19:55 PM
re: Internet Core Router Test Let's see. I can buy a router with great customer service, support, and historical performance to back up their product. Also, the manufacturing base to to deliver timely inventory.........

Or I can buy a product from some third tier provider with unproven customer service, support and little manufacturing capabilities.....

I think I'll buy..........
duffeck 12/4/2012 | 8:19:56 PM
re: Internet Core Router Test "Excellent question. There's a lot at stake for CWNT here. For starters, they need to show prospective customers their product performs well. This is a big step in that direction."

David
Thanks for the prompt reply. I agree there IS a lot at stake for CWNT here. Of course in the real world just about everything is "relative" with few things being "absolute". So whether you like it or not prospective customers will demand to know how well CWNT's router performs relative to the competition using the best available data despite its inherent latency. That is a fact that I believe you acknowledged by saying that "This is a big step in that direction".

duff
dnewman 12/4/2012 | 8:19:58 PM
re: Internet Core Router Test "Otherwise a good and timely first glimpse article."

Agreed. I thought Peter Heywood did a good job with a sensitive subject.

"You wrote it even though you told me earlier that you would not. "

True. LR's decision to run an article on the CWNT retest came as a surprise to me.

"I will look forward to specific numerical comparisons of the latest test results with the earlier data that are bound to be made in the future by others if not yourself."

Haven't I made this clear? These comparisons are bogus. Even MRVC management says so.

"After all if no comparisons are to be made what was the point of going to all the trouble"

Excellent question. There's a lot at stake for CWNT here. For starters, they need to show prospective customers their product performs well. This is a big step in that direction.

CWNT's biggest requirement is to look like a credible player in this market. That's why they're posting their numbers, and only their numbers. Again, it's just not credible to compare their new stuff with other folks' old stuff.

Regards,
David Newman
Network Test


duffeck 12/4/2012 | 8:19:58 PM
re: Internet Core Router Test Gee while I was writing this, null0 was actually comparing the test data. Good job!

http://www.lightreading.com/bo...

And yes David based on these comparisons I agree with the conclusion that CW's "routers posted an excellent showing on the OC48 testbed".

duff
duffeck 12/4/2012 | 8:19:59 PM
re: Internet Core Router Test <<the <br="" article="" me="" on="" quotes="" this.="">The retest appears to be a vast improvement over the previous showing. Obviously I can't verify the results of this or any other vendor-sponsored test, but if we take CWNT at its word the company's routers posted an excellent showing on the OC48 testbed.
Regards,
David Newman
Network Test"

David,
This implies to me that CW told you that the Areana router posted "an excellent showing". Either that or you figured it out yourself.

I say "excellent" COMPARED TO WHAT? I am sure you are aware that excellent is NOT an absolute term, it implies a COMPARISON. So somebody, presumably CW or yourself must have already made the "apples to oranges" comparison, despite your rather high minded philosophical protestations to the contrary. Perry, JohnyBoyOh and others were just asking to see the data supporting this conclusion.

Otherwise a good and timely first glimpse article. You wrote it even though you told me earlier that you would not.

http://www.lightreading.com/bo...

I will look forward to specific numerical comparisons of the latest test results with the earlier data that are bound to be made in the future by others if not yourself.

I of course understand your point about the Cisco and Juniper routers possibly undergoing significant improvements in the intervening interval. But hopefully we are all adults here and can take those possibilities into consideration when viewing the comparisons.

After all if no comparisons are to be made what was the point of going to all the trouble to assure that:

"The test was performed at Spirent Communications state of the art testing facility located in Calabasas, Calif. Spirent used the exact equipment, and identical test methodology from Light Reading's first test of core routers by leading manufacturers Cisco, Juniper, Charlotte's and Foundry."

duff</the>
dnewman 12/4/2012 | 8:20:02 PM
re: Internet Core Router Test Toni, your comparison is utterly bogus.

Don't just take my word for it. Ask MRV's CEO Noam Lotan, who's agreed that it's not proper to make such comparisons. So even the vendor you're flogging doesn't agree with you.

Sorry if it upsets your prejudices, but CWNT didn't win. JNPR did -- last time. Next time -- well, we'll see, won't we?

Regards,
David Newman
Network Test
dnewman 12/4/2012 | 8:20:03 PM
re: Internet Core Router Test Toni, your comparison is utterly bogus.

Don't just take my word for it. Ask MRV's CEO Noam Lotan, who's agreed that it's not proper to make such comparisons. So even the vendor you're flogging doesn't agree with you.

Sorry if it upsets your prejudices, but CWNT didn't win. JNPR did -- last time. Next time -- well, we'll see, won't we?

Regards,
David Newman
Network Test
dnewman 12/4/2012 | 8:20:03 PM
re: Internet Core Router Test I have the same understanding about Juniper's current architecture, specifically about sending traffic over four paths on the OC192 card.

Whether this is better or worse with OC192c remains to be seen.

By the way, in our tests we saw a lower threshold for reordering with SHORT packets than the Imix load -- 56 percent for 40-byte packets, vs. 73 percent with Imix. Whether that will be true with OC192c (and indeed whether that card reorders at all) remains to be seen.

Regards,
David Newman
Network Test

null0 12/4/2012 | 8:20:03 PM
re: Internet Core Router Test As promised I have compared the current cwnt figures to the figures that jnpr and csco posted a couple of months ago.

Throughput Test

cwnt 40 byte IP99.8
jnpr 40 byte IP99.8
csco 40 byte IP100

cwnt imix 99.9
jnpr imix 99.9
csco imix 100

Cisco managed a 100% line rate but suffered average latency in the millisecond range. CWNT and Juniper measured latency in the low microseconds. With CWNT posting the best latency figures for IMIX. So in my opinion CWNT have the best overall throughput figures.

MPLS Throughput test
The results for throughput are the same with poor latency figures for Csco, Slightly better latency figures for jnpr on 40 byte mpls and superior latency results for Cwnt with IMIX. So again in my opinion cwnt have the best overall throughput figures for MPLS.

latency with 40-byte IP Packets (microseconds)
cwnt
min 11.15
avg 19.97
max 23.35

jnpr
min 13.2
avg 15
max 21

csco
min 17.7
avg 1934.6
max 15561 <-- ouch the cost of line rate

Latency with IMIX IP Packets (microseconds)
cwnt
min 11.4
avg 41.17
max 61.25

jnpr
min 13.2
avg 64.9
max 169.5

csco
min 15.9
avg 251.9
max 1765.6

IMIX is simulated real world traffic and so is the better benchmark, so cwnt win again.

Latency over MPLS with 40-byte IP packets
cwnt
min 10.8
avg 21.08
max 23.75

jnpr
min 13.5
avg 15
max 20.7

csco
min 16.3
avg 1603.6
max 91215.5

Latency over MPLS with Internet mix
cwnt
min 11.15
avg 41.3
max 63.05

jnpr
min 14.2
avg 69.7
max 186.2

csco
min 16
avg 642.4
max 6003.5

cwnt win again.

BGP table capacity

CWNT
Maximum routes: 1,480,000
Maximum routes (no swapping): 1,480,000
Forwarding table capacity: 700,000

JNPR
Maximum routes: 2400000 (swapping to hard disc)
Maximum routes (no swapping): 1,400,000
Forwarding table capacity: 700,000

CSCO
Maximum routes: 400000
Maximum routes (no swapping): 400000
Forwarding table capacity: 400000

I don't have an opinion on who won here as I'm sure all could have hooked up to a SUN and stored billions of prefixes if they wanted to. The important figure is max routes no swapping. Guess who came top again.


MPLS tunnel capacity

cwntjnprcsco
LSP's18375100005000

Pretty conclusive. cwnt win again.


Route flapping
Took a bit of working out this one. The current ranking is surprise, surprise cwnt 1st with jnpr and csco tied 2nd.

Convergence
Finally some one else comes first, jnpr, cwnt 2nd and bringing up the rear csco.


Filtering
Both cwnt and jnpr got a perfect score and so tied 1st. Apparently csco can't filter :-(

QOS
Here we go again cwnt first with jnpr and csco tied 2nd.

Very interesting results for a startup company that has only been around a couple of years and appear to have produced a damn fine router. Definately worth a look.

Can I have one please ?

To sumarise cwnt won.

Its worth repeating Charlottes Web WON.

Toni

kj 12/4/2012 | 8:20:04 PM
re: Internet Core Router Test What I'm saying is that my assumption is that you'll get worse reordering using full clear channel OC-192c instead of just OC-192. I believe this is the case after looking at the Juniper architecture.

I believe that the Juniper router takes the 10G stream and then divides it into four 2.5G streams and then processes each of the four 2.5G streams individually in parallel.

This will work fine if you are feeding it four OC-48c streams because the router can send one stream to each of the parallel processors. Then during egress from the router, the four streams get aggregated back into 10G. Since the four streams are unrelated, you won't have a reordering problem after the aggregation.

However, if you run full clear channel OC-192c, then when the router aggregates the four 2.5G streams back together, reordering could happen because now the four streams are related. The router can't just blindly aggregate the four streams, it must figure out the ordering of the packets first.

I would assume the worst case would be when running varying packet lengths. In this case, the time to get processed through each of the four streams could vary a lot. So, the reordering could get even worse.

- kj.

Previous message below:
-----------------------
1. We are planning to test again, and with OC-192. Don't have a date set yet, sorry.

2. The test published in March already shows that reordering occurs with Juniper's OC-192 cards. The fact that we fed them with OC-48 cards is irrelevant. Reordering occurred only when the OC-192 cards were in use. The M160 did not reorder packets on the all-OC48 testbed.

Regards,
David Newman
Network Test
dnewman 12/4/2012 | 8:20:05 PM
re: Internet Core Router Test 1. We are planning to test again, and with OC-192. Don't have a date set yet, sorry.

2. The test published in March already shows that reordering occurs with Juniper's OC-192 cards. The fact that we fed them with OC-48 cards is irrelevant. Reordering occurred only when the OC-192 cards were in use. The M160 did not reorder packets on the all-OC48 testbed.

Regards,
David Newman
Network Test


null0 12/4/2012 | 8:20:05 PM
re: Internet Core Router Test touchy touchy.............
kj 12/4/2012 | 8:20:05 PM
re: Internet Core Router Test Are you guys planning on running the next test with OC-192c? I was wondering if the Juniper packet reordering (sorry to bring it up again) gets worse at OC-192c. I would guess that it will.

My understanding is that the Juniper "fix" was to ensure that within a microflow (ie OC-48c), all its packets went through the same path through the router. However, at OC-192c, I believe that this won't be possible because the router needs to spread the load out across four processors; hence, causing the reordering problem. Thanks.

- kj.

Previous message below:
-----------------------
"Does that mean that the tests weren't done by feeding the routers with full clear channel OC-192c? It looks like testing was done using a bunch of OC-48c's. Is that correct?"

Correct. We offered traffic to OC-48c edge interfaces, up to and inlcuding enough traffic to completely saturate the OC-192 core interfaces.

Regards,
David Newman
Network Test
dnewman 12/4/2012 | 8:20:06 PM
re: Internet Core Router Test You could do this. The validity of this exercise would be, er, null.

Regards,
David Newman
Network Test
larrymoeandcurly 12/4/2012 | 8:20:07 PM
re: Internet Core Router Test Did you enjoy your encounter with one of MRVC's Yahooooos?

Don't prejudge all MRVC experts to be as "uninformed and uneducated" as our resident moron, Perry.

His "say my bicycle is better than yours or I'll tell my mommy" mentality is something we have to deal with on a daily basis.

Your viewpoint is very objective and well appreciated.
null0 12/4/2012 | 8:20:07 PM
re: Internet Core Router Test Oh Perry,Perry,Perry.

What you could do is print off Charlotte's new results then print out the original LR test results and compare them for yourself. However, as I'm quite interested in the current comparison I'll do it for you.

I'll be back in a short while with the information and my appraisee of the current situation.
delmarbill 12/4/2012 | 8:20:09 PM
re: Internet Core Router Test Thanks for your reply. I'll pass it on.
dnewman 12/4/2012 | 8:20:09 PM
re: Internet Core Router Test "You said earlier that lightreading personnel were there.Surely you trust THEM,or do you?
I can't believe this site....UNREAL!"

I said no such thing. What I said was "people involved with the Light Reading test." As you're surely aware, the LR test was quite large in scope and involved multiple outside suppliers, most of whom DON'T work for Light Reading.

The caveats apply to the CWNT retest because it was done under NDA for CWNT and only for CWNT.

Regards,
David Newman
Network Test
dnewman 12/4/2012 | 8:20:09 PM
re: Internet Core Router Test -- is this a competitive router in today's market?

Yes, it is. So are all the others in LR test.

-- what do you like about the results?
-- what are the greatest weaknesses?

More than any of the specific numbers I very much like the fact that CWNT has gone to great lengths to be "clean" in its retest. As noted, it did everything to ensure an exact retest, and it's refrained from inappropriately comparing its new numbers with its competitors' old ones.

I was a little surprised by the lack of OC192 testing, but I wouldn't read anything into this. It may have been an inventory problem.

-- for what specific market do you see this product most viable?

CWNT markets this as an Internet core router. I have no reason to doubt them.

-- do you think it will succeed in the market?
-- if yes, why. if no, why

I have no idea. Success in the marketplace depends on many factors, including but certainly not limited to technology. I'm not a finance, sales, or marketing guy and as such I'm unqualified to comment on those aspects of the company. As a technologist I can say that they APPARENTLY (that caveat again) have addressed the software problems that plagued them awhile back.

Regards,
David Newman
Network Test
perry1961 12/4/2012 | 8:20:10 PM
re: Internet Core Router Test "but if we take CWNT at its word the company's routers posted an excellent showing on the OC48 testbed."

You said earlier that lightreading personnel were there.Surely you trust THEM,or do you?
I can't believe this site....UNREAL!

cfaller 12/4/2012 | 8:20:10 PM
re: Internet Core Router Test Mr. Newman:

Pretty please?
dnewman 12/4/2012 | 8:20:10 PM
re: Internet Core Router Test The article quotes me on this.

The retest appears to be a vast improvement over the previous showing. Obviously I can't verify the results of this or any other vendor-sponsored test, but if we take CWNT at its word the company's routers posted an excellent showing on the OC48 testbed.

Regards,
David Newman
Network Test
delmarbill 12/4/2012 | 8:20:11 PM
re: Internet Core Router Test
Mr Newman, I've been asked to forward these questions to you regarding Aranea-1

"since you don't want to compare results, could you offer an opinion (personal)"

-- is this a competitive router in today's market?
-- what do you like about the results?
-- what are the greatest weaknesses?
-- for what specific market do you see this product most viable?
-- do you think it will succeed in the market?
-- if yes, why. if no, why

Thanks
perry1961 12/4/2012 | 8:20:11 PM
re: Internet Core Router Test "Would YOU consider it fair or appropriate if your competitor compared their new widget with your old one?"

Okay,the Aranea-1 numbers.Are they so-so? Butt-ugly,WHAT?
Since we have nothing to compare them to,AT LEAST tell us how they look on a scale of 1-10.
Something.......ANYTHING!!!!!

delmarbill 12/4/2012 | 8:20:12 PM
re: Internet Core Router Test I meant microseconds.

See what I mean..you can't trust anyone to get the data right <g></g>
dnewman 12/4/2012 | 8:20:12 PM
re: Internet Core Router Test "Thanks for defending your point, I think I understand it well now.

I just think it's really stupid."

Please suggest an alternative. In doing so, please put yourself in the shoes of one of the three other vendors that participated. Would YOU consider it fair or appropriate if your competitor compared their new widget with your old one?

Again, CWNT agrees with me on this one. I commend them for refraining from making comparisons that are, um, stupid.

Regards,
David Newman
Network Test




Shareholdervalue 12/4/2012 | 8:20:12 PM
re: Internet Core Router Test David,

Thanks for defending your point, I think I understand it well now.

I just think it's really stupid.

dnewman 12/4/2012 | 8:20:13 PM
re: Internet Core Router Test " if John Smith runs a 4-minute mile and John Doe runs a faster time a month later,we can't compare the results?
LUDICROUS!!!!!"

Did John Doe engage undergoing some genetic re-engineering during the month after John Smith set his record, enabling him to improve his performance? Obviously not.

But that is exactly what computer and networking makers do all the time in rewriting their software. That's why it is appropriate to compare human achievement records spaced years apart, but not appropriate to compare computer records with any time gap at all.

If I go shopping today, I can buy a better router from ANY of the participants in the LR test than I could at the time the test was conducted. Unfortunately, we humans can't achieve the same kind of rapid performance gains...

Regards,
David Newman
Network Test

delmarbill 12/4/2012 | 8:20:13 PM
re: Internet Core Router Test I see your point. Like it or not, I think people are going to make the comparisons. Somebody has already raised the issue that Araneas latency is 1000 fold longer (milliseconds vs nanoseconds)than the competition. I guess if comparisons are going to be made, I'd prefer to hear them from someone who knows what they are talking about- i.e. you.
perry1961 12/4/2012 | 8:20:14 PM
re: Internet Core Router Test So if John Smith runs a 4-minute mile and John Doe runs a faster time a month later,we can't compare the results?
LUDICROUS!!!!!

This kind of b.s. is what lightreading is famous for.Partison reporting....
dnewman 12/4/2012 | 8:20:14 PM
re: Internet Core Router Test Hi Bill,

The CWNT team used the same methodology, the same lab, the same equipment, and even some of the same people as those involved in the Light Reading project.

There was one big change, however: The CWNT device tested is several months newer than the devices evaluated in the LR report. That's a long time in the router business, enough so to make comparisons inappropriate.

Put another way: CWNT today can tell customers it has performance of X. It cannot, however, say vendor Y has worse performance based on months-old data. The info on Vendor Y may not be true any longer; in a business where software updates appear weekly (and somedays daily) it's probably not.

Regards,
David Newman
Network Test
dnewman 12/4/2012 | 8:20:15 PM
re: Internet Core Router Test "Surely Charlottes didn't smoke Juniper THAT bad....
I've got to assume by your response that they do though."

You didn't understand a word I said, did you?

CWNT didn't smoke anyone. It was a one-product test.

"Unfortunately I've got no way to compare these router tests"

Yes, and that's exactly the point. Any such comparison is apples and oranges.

"Is there anyone at lightreading that COULD answer a simple question?"

Yes, I can. The answer is this: Don't attempt bogus comparisons.

Sorry you're having trouble grasping this rather simple concept. The CWNT folks understood it right away.

Regards,
David Newman
Network Test
dnewman 12/4/2012 | 8:20:15 PM
re: Internet Core Router Test Sorry, that's a hypothetical situation.

No offense, but I think any attempt to rewrite history is off base. To understand why, let's extend the would-have, could-have line of questioning a bit further:

--How would Cisco have compared if it had been able to submit its '03 model year 12000 against the current competition?

--Would the Nasdaq index have tanked as badly last year if we use tech stocks' somewhat improved '01 record in place of the '00 numbers?

--Would the LA Dodgers have won the 2000 National League pennant if we compare their '01 record against other teams' '00 records?

...and so on. They may be nice scenarios to contemplate, but none of them are real and neither is any attempt to compare CWNT's retest numbers vs. those in the LR test.

Sorry if this comes as a disappointment.

Regards,
David Newman
Network Test


delmarbill 12/4/2012 | 8:20:15 PM
re: Internet Core Router Test Are you saying that CWNT did not perform any tests that provided results that can be directly compared to CSCO and JNPR's previous results? That's not the impression that I got when I read..

"The people involved with the Charlotte's Networks-sponsored tests have told me they did reproduce exactly what we did for Light Reading," says David Newman, president of Network Test.

Please explain.
perry1961 12/4/2012 | 8:20:15 PM
re: Internet Core Router Test Surely Charlottes didn't smoke Juniper THAT bad....
I've got to assume by your response that they do though.
Unfortunately I've got no way to compare these router tests,and you obviously aren't going to help.
Is there anyone at lightreading that COULD answer a simple question?
dnewman 12/4/2012 | 8:20:16 PM
re: Internet Core Router Test Sorry, that's a hypothetical situation.

No offense, but I think any attempt to rewrite history is off base. To understand why, let's extend the would-have, could-have line of questioning a bit further:

--How would Cisco have compared if it had been able to submit its '03 model year 12000 against the current competition?

--Would the Nasdaq index have tanked as badly last year if we use tech stocks' somewhat improved '01 record in place of the '00 numbers?

--Would the LA Dodgers have won the 2000 National League pennant if we compare their '01 record against other teams' '00 records?

...and so on. They may be nice scenarios to contemplate, but none of them are real and neither is any attempt to compare CWNT's retest numbers vs. those in the LR test.

Sorry if this comes as a disappointment.

Regards,
David Newman
Network Test


dnewman 12/4/2012 | 8:20:16 PM
re: Internet Core Router Test Martians are prefixes that are reserved or illegal, and shouldn't appear on production networks (ducking -- martian filtering is grist for too many stupid mailing list flame wars).

The first column of the spreadsheet, stream ID, is internal to the SmartBits. The second column is prefix length.

Columns C-F are the octets in the prefix.

We "rolled" (incremented) the second octet by the number of times listed in column H. The number of times we rolled the octet was a function of prefix length.

So, for example, we used:

17.0.0.0/16
17.1.0.0/16
...
17.254.0.0/16

Hope this helps.

Regards,
David Newman
Network Test
dnewman 12/4/2012 | 8:20:17 PM
re: Internet Core Router Test "Does that mean that the tests weren't done by feeding the routers with full clear channel OC-192c? It looks like testing was done using a bunch of OC-48c's. Is that correct?"

Correct. We offered traffic to OC-48c edge interfaces, up to and inlcuding enough traffic to completely saturate the OC-192 core interfaces.

Regards,
David Newman
Network Test
dnewman 12/4/2012 | 8:20:17 PM
re: Internet Core Router Test Any comparison between the Charlotte's Networks retest and the Light Reading results is totally bogus.

That's because CWNT had a few months to refine its code, and the other vendors didn't. I know that Foundry did some internal retesting before the article's publication (this is noted in the article). I'm also quite sure Cisco and Juniper have been busy improving their products since the article appeared. It's neither fair nor appropriate to compare the other vendors' older code with CWNT's newer stuff.

A benchmark is merely a snapshot of product(s) at a given moment. We should take the same approach to comparing networking devices that the Spec folks apply to servers -- compare like with like at the same time.

Again, any attempt to rank or compare new results versus old ones is improper.

By the way, the CWNT folks have indicated to me that they agree with the above. I commend CWNT for keeping clean on this -- running the same test, and posting only its own numbers instead of attempting a false comparison.

Regards,
David Newman
Network Test



perry1961 12/4/2012 | 8:20:17 PM
re: Internet Core Router Test All I'm asking is how the results WOULD HAVE compared IF Charlottes Web HAD received the same scores THEN.
I've seen the results,but have no idea if they are good,bad,or ugly.
Why is this like pulling teeth?????
perry1961 12/4/2012 | 8:20:19 PM
re: Internet Core Router Test Sorry about the dropped "d".Hopefully Mr.Newman will figure the question out anyway.
How bout that router punk?
larrymoeandcurly 12/4/2012 | 8:20:20 PM
re: Internet Core Router Test Please rephrase your ebonics enlaced question. It is difficult to get a response from such a poorly written post. Do you need the response written in an elementary school level grammatical format?
perry1961 12/4/2012 | 8:20:21 PM
re: Internet Core Router Test Please tell us how Charlottes web would have compare to Juniper and Cisco had these results been achieved during the first test.
I realize the retest wasn't indepently verified,and Juniper and Cisco may have also made improvements in the meantime,but I'd REALLY like to know.
BTW,thanks for keeping this thread alive...
netskeptic 12/4/2012 | 8:20:26 PM
re: Internet Core Router Test > I think that you care about how latency changes
> between 0% bronze traffic and 150% bronze
> traffic. The change in the latency is the
> jitter, which is a very different measurement.

I want gold latency(min/avg/max) and loss at 150% of bronze traffic, latency and loss without overload are already published, so I can figure out the rest myself, thank you.

Thanks,

Netskeptic



kj 12/4/2012 | 8:20:26 PM
re: Internet Core Router Test Hi David,

I took a look at the test setup shown in the diagrams in the link. Does that mean that the tests weren't done by feeding the routers with full clear channel OC-192c? It looks like testing was done using a bunch of OC-48c's. Is that correct?

- kj.
fairplay 12/4/2012 | 8:20:26 PM
re: Internet Core Router Test There is a trade off in everything.

Please correct me if I'm wrong but in simple terms the M160 is basically an m40 on augmentive surgery (a tit job). i.e it has the data path of an m40*4.

And so, without corrective surgery of course the DUT/core router will reorder packets (4 internal paths from ingress to egress in a shared memory architecture = misordering of packets).

Now the trade off.

Juniper have touted from start to finish, line rate for all packets on all interfaces. That was essentially true on the m40 (single data path).

The M160 touted the first 10G interface unfortnately it potentially only had 1/4 of the performance had they implemented a source/destination hash algorithm to ensure in order packets per flow ?

They chose throughput over goodput. If my assertions are correct then they made a poor choice of through put over goodput.

Tell me about it.

tony1athome 12/4/2012 | 8:20:27 PM
re: Internet Core Router Test >> p.s. Are we still sure that we want to test latency?

>I had a very simple question: how delays (and
>loss) in gold traffic are affected by pumping
>system with 150% of bronze traffic ?

I think that you care about how latency changes between 0% bronze traffic and 150% bronze traffic. The change in the latency is the jitter, which is a very different measurement.

>Are preliminary results actually so bad (say
>hockey stick) that nobody dares to
>test/publish/discuss this metric ?

Actually, I suspect that we just need to understand what questions we really want answered. The questions that I'm interested in are: How much jitter is introduced by lower priority traffic? How much jitter is there in the system at 0% load?

Regards,
Tony

p.s. Is this horse dead yet?
null0 12/4/2012 | 8:20:27 PM
re: Internet Core Router Test Hi again

Apologies

The format was not good so from the last line I'm interested in columns n,254,0,1,0,2,0,1 and what is a martian, a network from Mars ?

731 24 191 n 254 0 256 1 0 2 0 1
^ ^ ^ ^ ^ ^ ^ ^
Sorry for the mess.

Toni
null0 12/4/2012 | 8:20:27 PM
re: Internet Core Router Test Hi David

I have downloaded your excel spreadsheet that contains the IP prefixes used during the LR test.
Is there a detailed explanation available for each column of the spreadsheet that you used for creating the advertised prefixes.

I have given a snapshot of the file below. ^ below a column indicates that I don't understand the data.
S# Len IP Prefixes Martian
63 16 125 n 0 0 256 4 0 0 0 2
69 17 138 n 0 0 256 0 0 256 0 0
70 17 139 n 0 0 256 0 0 256 0 0
80 18 149 n 0 0 256 0 0 128 0 0
81 18 149 n 128 0 256 0 0 128 0 0
150 19 171 n 192 0 256 1 0 64 0 1
731 24 191 n 254 0 256 1 0 2 0 1
^ ^ ^ ^ ^ ^ ^ ^
Please explain each column as most do not have a heading and some values never change. Also what is martian ?

I hope the formating is ok after I hit the post button.

Thanks

Toni
tony1athome 12/4/2012 | 8:20:29 PM
re: Internet Core Router Test If you truly want to characterize the latency induced by a device (a highly questionable goal), then you need to test that device under zero load. That's to say, you should test with one and only one packet.

Why? Well, what you don't want to measure is queueing delay. Suppose that packets A and B both want to exit on interface I. If A and B arrive at the UUT simultaneously, and are placed sequentially on the interface's output queue, then one of them will not be the first packet in the queue. Thus, it experiences queueing delay.

In fact, even if A and B don't arrive simultaneously, the latter can still experience queueing delay if it arrives at the output interface before the previous one has been completely transmitted. Insuring this is somewhat challenging.

Even insuring that packets for various different output interfaces don't induce queueing delay for one another is challenging. Suppose that a device has some internal resource that can be congested, such as a crossbar switch. Depending on the resource design, some traffic may cause other traffic to congest internally to the machine. Designing a test that avoids this internal queueing delay implies that the test is highly customized to each and every architecture.

Thus, testing with more than one packet likely means that you're seeing something other than the true delay introduced by the system.

Regards,
Tony

p.s. Are we still sure that we want to test latency?
netskeptic 12/4/2012 | 8:20:29 PM
re: Internet Core Router Test > p.s. Are we still sure that we want to test latency?

I had a very simple question: how delays (and loss) in gold traffic are affected by pumping system with 150% of bronze traffic ?

Say we have 4 - port (A, B, C, D) and we pump

A -> D: 75% - bronze
B -> D: X% - gold
C -> D: 75% - bronze

What are delays and loss for gold traffic with
X 30, 60, 90 ?

A -> D: 50% - bronze, X% - gold
B -> D: 50% - bronze, X% - gold
C -> D: 50% - bronze, X% - gold

What are delay and loss for gold traffic with X
10, 20, 30 ?

I suppose that this is one of them most important questions one would like to ask before committing to something like for-profit-VoIP. Are preliminary results actually so bad (say hockey stick) that nobody dares to test/publish/discuss this metric ?


Thanks,

Netskeptic

tink 12/4/2012 | 8:20:30 PM
re: Internet Core Router Test I also didn't suggest only measuring delay at the point right near the throughput. I suggested testing at a series of fixed percentages of wire rate up to and including the throughput rate. You imply that the new standard will include this and further tests at rates greater than the throughput.

Tink
tink 12/4/2012 | 8:20:30 PM
re: Internet Core Router Test I didn't ask which measurement is best (that would be like asking which router is best ... best at what?). I asked which number best reflects the "delay introduced by this router"?

Tink
dnewman 12/4/2012 | 8:20:31 PM
re: Internet Core Router Test Ignore the RFC at your peril. It is the only agreed-upon methodology for latency measurement.

You might want to check the IETF bmwg mailing list from mid-1999 (I think), where I raised the question of how to measure the "hockey stick" type of curve that device latency exhibits as the offered load approaches the throughput rate.

Please also see earlier comments on this list about the question you're asking. As I've noted before, I'm currently coauthoring a draft that defines terminology for a delay measurement. When we do the methodology, we fully intend to do a step test -- not just the point right before throughput, as you suggest, but a range of values before and after the congestion point.

You ask which measurement is best. I can't answer that. Only one of them can be rightfully called latency, of course. Calling any other measurement "best" is a function of device design, network conditions, and of course the marketing goals of the vendor in question.

Regards,
David Newman
Network Test
tink 12/4/2012 | 8:20:59 PM
re: Internet Core Router Test David,

I am asking for an opinion. I like to analyze problems and information, not simply to through my hands in the air and say, "oh well, that's what the RFC says." Please give your opinion: which number best reflects the "delay introduced by this router"?

By the way, a router can be designed to forward without loss at 100% wire rate, then configured to drop packets such that the throughput of the router is less (say 99.8, 99.9, 92.2, or 90.0% of wire rate). This is an excellent way of controlling the RFC-defined "latency" while accepting a lower throughput rate.

This is one of the compelling reasons to ignore the RFC and instead measure the "delay" at a series of fixed percentages of wire rate up to and including the throughput rate. Such tests make comparisons easier and provide data that has more relevance in the real world where routers should not be allowed to reach their throughput level without someone taking intervening measures.

Thanks,
Tink
dnewman 12/4/2012 | 8:21:01 PM
re: Internet Core Router Test a) why is the delay so much greater when traffic is offered at the throughput rate?

I have no idea. Perhaps you should fix your hypothetical router.

b) which number should I think of as the "delay introduced by the router"?

RFC 2544 says at the throughput level.

This horse isn't just dead. The glue they made from the dead horse has already dried.

Regards,
David Newman
Network Test
tink 12/4/2012 | 8:21:03 PM
re: Internet Core Router Test David,

Let me pose the following questions. Suppose I measure delay through a DUT with traffic sent (and forwarded) at throughput and at .99*throughput. The interfaces are POS. The delay measured at throughput is 100,000 microseconds and the delay measured at throughput is 100 microseconds.

a) why is the delay so much greater when traffic is offered at the throughput rate?

b) which number should I think of as the "delay introduced by the router"?

Thanks,
Tink
tink 12/4/2012 | 8:21:03 PM
re: Internet Core Router Test oops:

the delay at throughput is 100,000 microseconds
the delay at 0.99*throughput is 100 microseconds
dnewman 12/4/2012 | 8:21:04 PM
re: Internet Core Router Test "I understand that the Smartbits tracks every frame, but does it report to you which ones it performed byte-stuffing (or un-stuffing) on?"

Yes.

Regards,
David Newman
Network Test
tink 12/4/2012 | 8:21:04 PM
re: Internet Core Router Test I understand that the Smartbits tracks every frame, but does it report to you which ones it performed byte-stuffing (or un-stuffing) on?

Thanks,
Tink
dnewman 12/4/2012 | 8:21:13 PM
re: Internet Core Router Test 1) Yes. If necessary, the Smartbits can track every single frame sent and received.

2) See answer 1, especially the term "received."

Regards,
David Newman
Network Test
tink 12/4/2012 | 8:21:36 PM
re: Internet Core Router Test David,

Two follow-up questions:
1) Can the Smartbits report the occurance of frames for which byte-stuffing was necessary?

2) The DUT might receive a packet and change the encap or decrement the TTL in such a way that byte-stuffing is necessary. For example, the new CRC after decrementing a TTL might contain a 7E. This can happen, yes?

Thanks,
Tink
dnewman 12/4/2012 | 8:21:39 PM
re: Internet Core Router Test Hi Tink,

The Smartbits traffic generators do not insert 0x7E bytes in packet payloads. These bytes may "naturally" occur in other places, like IP sequence numbers or the FCS fields. I do not think this had a significant impact, since the traffic on the wire still goes no faster than line rate, not line rate plus 0x7D.

As for 2544, it very much does apply. My understanding of byte- (not bit-) stuffing is that it occurs at the sender and receiver -- which in this case were the Smartbits interfaces, not the devices under test.

Regards,
David Newman
Network Test
tink 12/4/2012 | 8:21:44 PM
re: Internet Core Router Test David,

If a POS "L2 frame" is sent containing a 7D or 7E (eg, in the PPP CRC) then it must be bit-stuffed. In such a situation, if traffic is input at the throughput rate, then it must be output at a rate higher than the throughput rate (because there are more bytes to send). The "delay" goes up because the router cannot keep up with this higher output rate and queues build up.

I have two questions:

a) did you encounter this issue in the Light Reading tests?

b) RCF 2544 doesn't deal with this, or with POS at all. Does the specification that "latency" is only for input traffic at the throughput rate really apply in the case of POS?

Thanks,
Tink
dnewman 12/4/2012 | 8:22:14 PM
re: Internet Core Router Test "It seems like delay under overload WITH QoS configurations on a router would be a very relevant measurement. That is to say, don't we want the gold traffic to get there faster?"

It would be nice if, under congestion, gold gets there at all. We can't measure delay on packets we don't receive :-)

How fast that happens is currently the subject of a lot of hype. Some folks are claiming they can apply TDM-like mechanisms to IP or Ethernet to set strict upper bounds on delay. We'll see. That's why we're working on a delay metric in the first place.

Regards,
David Newman
Network Test
tink 12/4/2012 | 8:22:17 PM
re: Internet Core Router Test It seems like delay under overload WITH QoS configurations on a router would be a very relevant measurement. That is to say, don't we want the gold traffic to get there faster?

Tink
dnewman 12/4/2012 | 8:22:19 PM
re: Internet Core Router Test Yes, and I've been meaning to address this.

Of course you are correct that throughput is a rate, not a percentage. This was a rather lame usage error on my part, for which I apologize profusely. If it's any consolation I've had my ears boxed by the chairman of the IETF's benchmarking working group because of it. I won't be expressing throughput as a percentage ever again!

Now, on to your question about packet rates.

On the OC48 testbed, at line rate we offered 2,200,554,576 40-byte IP packets to 12 OC48 interfaces over a 30-second test duration.

Dividing by 30 seconds and by 12 interfaces works out to 6,112,651 packets per interface per second. (Well, it's actually 0.6 packets/second larger than that, but test instruments calculate interframe gap by rounding down to the nearest whole packet.)

The total length of our 40-byte IP packet on the wire was 48 bytes -- 4 more bytes each for the header and CRC. Adding an MPLS tag means the packet grows to 52 bytes total. So, on the wire the IP packet is 92 percent the size of the packet with an MPLS label.

Now let's see if the MPLS packet count is 92 percent of the IP total. Again on the OC48 testbed and this time with 40-byte IP packets with MPLS labels, at line rate we offered 2,034,475,344 packets over the 30-second test duration. Again dividing by 30 and by 12 and rounding down, we get 5,651,320 MPLS packets per second per interface.

That number is 92 percent of the pure IP packet count. Ergo, in percentage utilization terms the packet counts are consistent.

I'm also very pleased to say that LR will be publishing the raw data from the core router test, including previously unpublished measurements of jitter and latency distribution. Look for this to appear in the next two to three weeks.

Regards,
David Newman
Network Test
johnsonw10 12/4/2012 | 8:22:30 PM
re: Internet Core Router Test In the IP and MPLS 40-byte througput tests, the results were shown as percentages. What are the 100% throughput in MPPS for both cases? The MPLS MPPS number should be smaller than IP numbers given MPLS overhead, so it is really not valid to compare the throupguts based on percentages. I would like to know the formula used to calculate the MMPS for the two cases. Thanks.
dnewman 12/4/2012 | 8:22:53 PM
re: Internet Core Router Test "3 rumours

1. CW would of come out top had they tested with Dave Newman."

Sorry, Network Test does NO vendor testing. Perhaps you're confusing us with some other "independent" lab.

And, given a moniker like "fairplay," perhaps you should do your homework before pointing the dirty end of the stick our way.

Regards,
David Newman
Network Test

fairplay 12/4/2012 | 8:23:14 PM
re: Internet Core Router Test 3 rumours

1. CW would of come out top had they tested with Dave Newman.

2. 3 Big players are looking to aquire....

3. If they came out top isn't $3B a small price to pay ?

Answers on a notice board soon..........

FP.
deadbeef 12/4/2012 | 8:23:32 PM
re: Internet Core Router Test Agreed.
dnewman 12/4/2012 | 8:24:01 PM
re: Internet Core Router Test Tony's much better qualified than I (or anyone else :-) to comment on the CPUs in the Cisco and Juniper boxes we tested.

We generally don't get into component comparisons, preferring instead to stick with what's externally observable. There are pros and cons to "black box," "white box," and "gray box" forms of testing, but for router benchmarking black box seems to be the norm.

Regards,
David Newman
Network Test
dnewman 12/4/2012 | 8:24:01 PM
re: Internet Core Router Test "I am looking through QoS testing results and I am wondering what was overload impact on latency."

Not applicable.

We didn't use latency as a metric in this test because RFC 2544 specifies latency measurements MUST be taken only at the throughput (zero-loss) level, and there was loss in this test.

It is valid to measure delay under loss conditions, but it's not valid to call that measurement latency. Besides delay introduced by the device's forwarding engine (which is what latency is about), queue depth also comes into play when measuring delay -- so much so that any measurement can dwarf the latency number.

Regards,
David Newman
Network Test



duffeck 12/4/2012 | 8:24:07 PM
re: Internet Core Router Test http://biz.yahoo.com/bw/010521...

"ANDOVER, Mass.--(BUSINESS WIRE)--May 21, 2001--Charlotte's Networks, a leading developer of carrier-class routers for the emerging IP infrastructure, today said it has completed the core routing test that was conducted by the Tolly Group, using Spirent Communications SmartBits systems.

The test was performed at Spirent Communications state of the art testing facility located in Calabasas, Calif. Spirent used the exact equipment, and identical test methodology from Light Reading's first test of core routers by leading manufacturers Cisco, Juniper, Charlotte's and Foundry. The Tolly Group, long recognized as a leading independent networking test benchmarking firm, ensured the test was identical in every manner to the original Light Reading test."

duff

tony1athome 12/4/2012 | 8:24:13 PM
re: Internet Core Router Test Some of that capability is tested in the convergence testing that you see.

The last time I looked, Cisco was using a MIPS processor, while Juniper was using a 250Mhz x86.

This will undoubtedly spark the "why not a bigger CPU" discussion. ;-)

There are issues. Clock rate is not everything. In fact, these systems are seldom CPU bound. It's much more common for them to be memory bound. Improving the performance of the front side bus is much more relevant than more cycles. And then there's the issue of physical form factor. CPUs that have to sit perpendicular to their motherboard (ala Slot One) are a non-starter. Laptop processors are much more applicable.

Tony
avi 12/4/2012 | 8:24:14 PM
re: Internet Core Router Test Hi

I was wondering if you could help with this question
What are the microprocessors (if any) that the 4 tested routers are using in order to run the routing algorithms and protocols? (And how what kind of CPU power need for these tasks)?
Did you test that ability in some way?


Thanks
avi 12/4/2012 | 8:24:15 PM
re: Internet Core Router Test Hi

I was wondering if you could help with this question
What are the microprocessors (if any) that the 4 tested routers are using in order to run the routing algorithms and protocols? (And how what kind of CPU power need for these tasks)?
Did you test that ability in some way?

Thanks

avi 12/4/2012 | 8:24:15 PM
re: Internet Core Router Test Hi

I was wondering if you could help with this question
What are the microprocessors (if any) that the 4 tested routers are using in order to run the routing algorithms and protocols? (And how what kind of CPU power need for these tasks)?
Did you test that ability in some way?

Thanks

netskeptic 12/4/2012 | 8:24:34 PM
re: Internet Core Router Test Hi,

I am looking through QoS testing results and I am wondering what was overload impact on latency.

Thanks,

Netskeptic
JohnyBoyOh 12/4/2012 | 8:25:19 PM
re: Internet Core Router Test From a press release Friday March 16:
http://biz.yahoo.com/bw/010316...
"...
The test will be conducted by Spirent Communications, a worldwide leader in performance analysis solutions. The test will be performed at Spirent's lab using the exact equipment and identical test methodology used in Light Reading's first test of core routers. The Tolly Group, long recognized as a leading independent networking test benchmarking firm, will ensure that the test will be identical in every manner to the Light Reading test. Charlotte's and Light Reading have agreed to publish the results.
..."


Cheers,
John
dnewman 12/4/2012 | 8:25:38 PM
re: Internet Core Router Test www.caida.org/tools is a good starting point, especialy the links under "taxonomy."

CAIDA is a way cool resource, and deserves the strong support of well-funded operators and vendors -- if there are any left...

Regards,
David Newman
Network Test
wzlobicki 12/4/2012 | 8:27:10 PM
re: Internet Core Router Test Andrew,

I am looking for similar software and in the middle of compiling a list. If you like, please email me at [email protected] and I will let you know when I have a page up.
dnewman 12/4/2012 | 8:27:31 PM
re: Internet Core Router Test "It will commission" are the key words here.

Vendor-sponsored tests by definition aren't indepedent. This particular one *should* be clean, since it simply replicates the Light Reading tests. However, we're not involved in the retest and have no way of verifying that.

As far as I know, Light Reading does not and will not be publishing results from this or any other vendor-sponsored test.

Regards,
David Newman
Network Test
veitcha 12/4/2012 | 8:27:40 PM
re: Internet Core Router Test

This may be an off the beat question, but I was wondering if anyone knew of a scaled down version of their traffic modeler was avaible. Here, and at every university we could really use the ability to model traffic after the students leave. Does anyone know of anything along thoose lines that is avaible?

Andrew Veitch
NOC-CIS
Brown University
duffeck 12/4/2012 | 8:27:58 PM
re: Internet Core Router Test You said that;

"CharlotteG«÷s Networks has indicated that it will commission a private test this spring that reproduces all the events in the Light Reading project"

http://www.lightreading.com/do...

Would you please update us on the status of the retest of CharlotteG«÷s Aranea router with the newer rev. 2.1 software and your schedule for publishing the results in Lightreading.


dnewman 12/4/2012 | 8:29:11 PM
re: Internet Core Router Test I would have expected the OC-192 to be 4x the OC-48.

The numbers on the OC192 testbed ARE four times those on the OC48 testbed -- in the aggregate. Because we needed direct comparisons between the OC48 and OC192 products, we present all numbers on a per-port basis. Measured per port, OC48 line rate is the same regardless of the speed of the core.



With regard to packet order - Did you test the Cisco box with a single flow greater than 2.5G?

The class-of-service tests were the only ones involving overloads. Results from these tests do indicate packets out of sequence, but these numbers are not meaningful since the reordering is an artifact of the buffering that must occur, and not any misbehavior by line cards.

Regards,
David Newman
Network Test
vlsi 12/4/2012 | 8:29:49 PM
re: Internet Core Router Test Why are the number of packets the same for both the OC-48 and OC-192 line cards ( about 6M / sec)? I would have expected the OC-192 to be 4x the OC-48.

With regard to packet order - Did you test the Cisco box with a single flow greater than 2.5G? If so, did it then continue to maintain packet order?

Thanks
JoePerches 12/4/2012 | 8:31:06 PM
re: Internet Core Router Test Could you please define "looking for reordering"? Is reordering defined using IP sequence number only? Is it keeping a + and - histogram of sequence delta received? something like a count of each sequence number: -more -3 -2 -1 0 +1 +2 +3 +more (0==duplicate sequence received, +1 is in sequence? Is it knowing that the stream sequence number is monotonically incrementing only?
dnewman 12/4/2012 | 8:32:18 PM
re: Internet Core Router Test yjim0140,

Thanks for your inquiry. Testing of multiservice switches is definitely on the drawing board. I'd welcome your suggestions (and those of fellow posters) as to what issues that test should address.

MPLS is a big, big area. We could do a whole test (hell, several tests) in that area alone -- one on traffic engineering, one on VPNs, and on and on.

Anyway, I'd welcome suggestions as to what issues a multiservice switch methodology should address.

Regards,
David Newman
Network Test
yjim0140 12/4/2012 | 8:32:23 PM
re: Internet Core Router Test Do you have any test plan for multi-service switch? I think that the previous Internet Core Router Test was just focused on pure IP router. But there are also many switch routers supporting ATM,IP,MPLS, etc.

Thanks in advance.
PBC 12/4/2012 | 8:33:11 PM
re: Internet Core Router Test THX dnewman,

PBC
dnewman 12/4/2012 | 8:33:13 PM
re: Internet Core Router Test Hi PBC,

There was a thread on the NANOG mailing list about 7-10 days ago that dealt in part with control-plane requirements. The thread dealt with the merits of PC-based routers running *nix vs. purpose-built boxen.

Craig Partridge, BBN's chief scientist (and incidentally co-author of the IEEE/ACM packet reordering paper), made the point that efficient route lookups depend on other factors besides the CPU. There's also bus and memory speed, optimization of the routing code, and line card driver optimization to take into account.

The thread is good reading for anyone interested in router architecture. Partridge's message, with references to earlier and later messages, is here:

http://www.merit.edu/mail.arch...

Regards,
David Newman
Network Test
PBC 12/4/2012 | 8:33:15 PM
re: Internet Core Router Test Change route to routes and comute to compute...

THX

PBC
PBC 12/4/2012 | 8:33:15 PM
re: Internet Core Router Test
Trying to figure out, if the next generation of gigabit switch routers or terabit switch routers require the control plane to actually run Symmetric Multi Processors to handle the increase in the amount of route, the comute intensive protocol stacks and the need to have all the routing tables calculated and updated and rates never seen before.

The reason I ask is that Motorola SPS has come out with SMP boards that run 2 Power PC 7450 (G4), running at 500-600 and 700 Mhz, and this is getting a lot of attention in the Networking space....but if I look at Avici's technical specs, they mention that their Control Plane processor is a single, 200 Mhz, risc (so that means either PowerPC or Mips, probably not Xscale). And we know Avici can deal with up to 500,000 routes!

Thoughts?

PBC
sstaplet 12/4/2012 | 8:33:33 PM
re: Internet Core Router Test Does anyone know where the actual requirements are for DWDM BER testing. Or are there any listed outside the GR 253.
dnewman 12/4/2012 | 8:34:45 PM
re: Internet Core Router Test Sorry, forgot to mention: In addition to hosting, the client I mentioned also provides transit service for numerous downstream ISPs. This is the context in which I'm referring to the big routers in its data centers as core routers.

Regards,
David Newman
Network Test
dnewman 12/4/2012 | 8:34:45 PM
re: Internet Core Router Test Thanks for the clarification, Tony. Good points all. We'll be sure to emphasize jitter measurements in future tests.

There was no confusion with the ISPs -- they did explicitly say latency (delay itself) was interesting, as distinct from jitter (delay variation).

Though I agree with everything you say here I can think of one case where delay might be important, and that's in hosting centers comprising multiple core boxes, or a mix of core and border routers.

For a private client, I recently was involved in just such a network design (involving equipment from two companies in which you're long ;-). In that scenario, the client (a Web hosting provider) was very interested in delay between the customer cage and the demarc with the various ISPs.

Again, though, I generally agree that in the wide area jitter is a far more important concern than delay.

Thanks again for some excellent suggestions.

Regards,
David Newman
Network Test


dnewman 12/4/2012 | 8:34:45 PM
re: Internet Core Router Test Hi Femtokid,

There are schematic diagrams of the testbeds in the methodology document here:

http://www.lightreading.com/do...

We used the Smartbits 6000B equipped with Terametrics 3505 POS card to generate IP and MPLS traffic. On the wire there is no difference between IP or MPLS packets from a Smartbits and those from a host.

Regards,
David Newman
Network Test
femtokid 12/4/2012 | 8:34:48 PM
re: Internet Core Router Test Are you guys using NW Analysers or actual Solutions to generate your OC-48 and 192. Maybe a System diagram would help?
It is a monday so i may have missed this on your site.

cheers
Femtokid
tony1athome 12/4/2012 | 8:34:50 PM
re: Internet Core Router Test Hi David,

Regarding latency: in the interesting parts of the Internet where core boxes are deployed, the end hosts are typically thousands of miles apart physically and thus are milliseconds apart in the speed of light delay alone.

In any situation where the end hosts were only meters apart, it's very likely that they would be part of the same enterprise and it's significantly more likely that the user would prefer an enterprise based product.

One should note that even in the enterprise, for well behaved sliding window protocols on reasonably large transfers, the latency becomes less important as it does not affect throughput. Thus, the only real world case where latency truly matters is for stop and wait protocols such as NFS (over UDP) in the enterprise.

The requests for latency histograms are very useful to the ISPs. That's one way of characterizing jitter. Recall that jitter is simply the amount of fluctuation in latency. ISPs would like to be able to offer a low-jitter service for time sensitive applications such as VoIP. Low jitter would yield a latency histogram that was essentially a single peak. Perhaps there has been some confusion about the difference between jitter and latency in your earlier discussions with ISPs.

Regards,
Tony Li
dnewman 12/4/2012 | 8:35:38 PM
re: Internet Core Router Test They were /24 entries over /16s, stuff like that. Sorry for the vague answer, but I don't have the actual prefixes handy and wanted to answer your question as quickly as possible.

The distribution was symmetrical across all interfaces.

Regards,
David Newman
Network Test
tink 12/4/2012 | 8:35:39 PM
re: Internet Core Router Test David,

I've been looking at your spreadsheets and I see how you used the prefixes for BGP and traffic. But, as far as I can tell, the spreadsheets don't show what the 50000 additional shorter prefix routes were for the longest match test. Can you tell me more about what these routes looked like? How were they distributed?

Thanks,
Tink
dnewman 12/4/2012 | 8:35:44 PM
re: Internet Core Router Test Hi Tony,

Thanks very much for your input. These are great suggestions and we will add as many of these as we can in future tests.

Quick followup question: Do you see no utility in latency measurements, even on a per-box basis? You'd mentioned earlier that speed-of-light considerations dwarf these numbers, and that's true for long spans. In this case the boxes were 2 meters apart. Still not relevant?

There also have been many requests for latency histograms. Do you think these too aren't useful, or is it that jitter is simply so much more important?

Thanks again.

Regards,
David Newman
Network Test



dnewman 12/4/2012 | 8:35:45 PM
re: Internet Core Router Test "But what means 100% of line rate at oc48 or 0c192?"

There is a formula for this, but I cannot render it in ASCII.

Suffice to say it's 2.4 Gbit/s for OC48 and 9.6 Gbit/s for OC192. Note that these are Sonet payload rates. The actual carrier rate is higher.

With 40-byte IP frames at line rate, we offered 2,200,554,576 packets to 12 ingress ports for a duration of ~30 seconds.

Note that if you divide that number by 12 and then by 30 you're not going to get an integer. That doesn't mean we offered half a packet!

Rather, the actual duration is rounded down to less than 30 seconds (the difference is nanoseconds or microseconds) to accommodate the maximum number of packets that can be transmitted within 30 seconds.

On the OC192 testbed at line rate, we offered 8,802,219,456 packets to 48 ingress interfaces. This too doesn't work out to an integer when divided by 48 and by 30, for the same reason. (Yes, that is 8 billion -- a good reason why OC192 gear and the systems that manage it MUST support 64-bit counters).

All traffic generators I'm familiar with use this method to calculate test duration.

Regards,
David Newman
Network Test
dnewman 12/4/2012 | 8:35:45 PM
re: Internet Core Router Test "Can you point me in the direction of control-plane vs. data-plane definitions/discussions?"

See message 235 on this list, where I provided a list of links. I don't have any further links.

Don't get too hung up over control- or data-plane labels. They're both important, and there will be situations where you'll need to measure what happens in both areas.

Regards,
David Newman
Network Test
riolan 12/4/2012 | 8:35:54 PM
re: Internet Core Router Test Thanks for the reply. But what means 100% of line rate at oc48 or 0c192 if we cannot know the IGP.
One way you can clarify may be it to add the number of IP packet effectively forworded at line rate--
at OC48 if you have 1 byte IGP you should find something around 6,12 Million P/s
Packet = 40 bytes IP + 8 bytes PPP
IGP = 1 bytes
cb
tony1athome 12/4/2012 | 8:35:57 PM
re: Internet Core Router Test David,

I would suggest the following:

-- Do not bother testing service level enforcement. While there has been a great deal of hype about service levels, there is little deployment (if any) of this feature. I would consider it another fancy marketing feature that no one really wanted. Let it rest in peace.

-- I would suggest that instead of studying latency, you should test jitter instead. As we've discussed before, latency is less relevant. However jitter, the change in latency, is very relevant.

-- I would suggest that you test buffering capacity. Providers want BW*RTT buffering to protect against congestion events.

-- I would suggest that you stop testing the RIB capacity of the system and instead focus on FIB capacity.

-- As part of your failover testing, you might want to look into testing APS or MPLS fast failover features.

-- As part of your MPLS testing, you should test out traffic engineering as well as statically (or semi-statically) configured LSPs.

Good luck.

Tony Li

p.s. Disclaimer: long Cisco and Juniper
tink 12/4/2012 | 8:35:58 PM
re: Internet Core Router Test
My understanding of control-plane vs. data-plane is vague so I can't really go any further on how they relate to THIS test.

But I do think that in spirit this test fits the definition of a "packet filtering" test (it was even called a packet filtering test in the test document). And it does appear that only one router did "packet filtering" in order to meet the goals of the test.

Thanks for the pointers to documents and resources. Still learning here. Can you point me in the direction of control-plane vs. data-plane definitions/discussions?

Thanks,
Tink
dnewman 12/4/2012 | 8:36:03 PM
re: Internet Core Router Test "Yes, there's nothing in the packet that indicates prefix length, so the router has to use information from packet AND other information to make a decision of whether to forward or drop"

My point exactly. This is why I don't think definitions that focus exclusively on control- or data-plane functions are terribly useful in describing THIS test.

In calling this packet filtering you *may* be moving the goalposts a bit, in that you're now suggesting packet filtering may have data-plane and control-plane components, but route filtering is control plane only. Please correct me if I'm misinterpreting your comments.

Regards,
David Newman
Network Test




tink 12/4/2012 | 8:36:04 PM
re: Internet Core Router Test David,

Yes, there's nothing in the packet that indicates prefix length, so the router has to use information from packet AND other information to make a decision of whether to forward or drop. That other information includes the "filter" (a set of rules), which may in turn use information from the routing table.

The point is, the router forwards or drops the PACKET based on information in the PACKET plus a set of imposed rules. This I would definitely call "packet filtering."

Thus, based your test documents and assertions, I would say that Juniper did "packet filtering" and Foundry did "route filtering."

I'm not sure of the control-plane vs. data-plane terminology, but I think it is correct to say that Foundry altered the control-plane in such a way that the data-plane had no idea where to send the packets and thus dropped them. Juniper used the control-plane to send a set of rules to the data-plane for deciding if and where to forward the packets.

Juniper's feat impresses me more. To me, "route filtering" doesn't jive with the "spirit" or defined goal of the test. So it looks like four vendors showed up, but only one could fairly compete in the test.

I'm also convinced that Cisco uses the the term "route filtering" to mean only the filtering of route advertisements and that their new stuff CAN do this. But they CAN'T do packet filtering, or in their words, use "ACLs" to filter traffic.

Tink

dnewman 12/4/2012 | 8:36:05 PM
re: Internet Core Router Test It sounds like you're defining route filtering exclusively in terms of control-plane functions, and packet filtering exclusively in terms of data-plane functions. Do I understand your distinction correctly?

Let's say we want to block packets destined for 1.2.0.0/24. In the LR test, Juniper accomplished this with a packet filter (your definition).

You said a packet filter looks at some quality of the PACKET. But here's the catch -- there is nothing in the packet that indicates prefix length.

So, using your definitions, did Juniper use a route filter or a packet filter?

Curious,
David Newman
Network Test
dnewman 12/4/2012 | 8:36:08 PM
re: Internet Core Router Test "But Foundry did this by using "one form" of "route filtering"? "

I believe Foundry blocked BGP updates, using the RFC1812-style definition. I am not 100 percent certain of this, though. We examined packets on the wire, not router table contents.

"Can you point me to some sources of definitions for "other" forms of "route filtering"?"

--RFC 2544, which addresses filtering in a generic sense without reference to whether it's prefixes or packets that are filtered

--http://www.ietf.org/internet-d.... Although this draft focuses on E-BGP convergence measurement, it does have a data-plane component. The discussion of filtering refers to RFC 2827 (guarding against distributed denial-of-service attacks), which advocates use of what you're defining as packet filters (the ID here calls them route filters).

Whether DDoS filtering is relevant in the context of core routers is a whole other topic. Personally, I recommend to clients that every router filter for martians/bogons/bad stuff, but I recognize that not everyone feels the same way.

--various L2 forms of CAC (connection admission control)

--and, of course, the discussion you and I have had today on this message board.

Regards,
David Newman
Network Test

tink 12/4/2012 | 8:36:08 PM
re: Internet Core Router Test David,

I'm convinced. The definition of "route filtering" is much more restrictive than the way you are using it. It applies ONLY to the filtering of routing updates so that certain routes are omitted from the routing table (and also not propogated). I can not find any instance of the term "route filter" meaning anything other than this.

I'm also convinced that the definition of "packet filter" is broader than your usage. "Packet filter" implies that the router denies or forwards the PACKET based on some quality of the PACKET (and perhaps based on additional information). The point is that anytime a router has to look inside a packet to make the decision, it is doing "packet filtering."

Other types of filtering ("prefix," whatever you wish to call them) exist, but they are NOT route filters unless they apply to routing updates and they ARE packet filters if they decide what to do with individual packets based on the contents of those packets.

Thanks,
Tink




tink 12/4/2012 | 8:36:11 PM
re: Internet Core Router Test David,

I've now gone over the detailed methodology document and find your statement:

"This test was to determine the point at which packet filtering rules degrade forwarding rate, latency, or jitter."

But Foundry did this by using "one form" of "route filtering"?

I thought I was catching on, but I'm lost again.

Thanks,
Tink

P.S. Can you point me to some sources of definitions for "other" forms of "route filtering"?
dnewman 12/4/2012 | 8:36:13 PM
re: Internet Core Router Test "So we're clear that "route filtering" means, "to [accept,deny] specific BGP updates, with the accompanying effect of [forwarding,blocking] traffic sent to addresses within the networks described in those updates," as you state."

That's one form of route filtering. There are others. Our methodology asked only that vendors allow or deny specific prefixes in a manner that was externally observable.

We externally observed whether a deny filter worked by offering packets and verifying that the system under test did not forward them.

The definition you're using looks only at a control-plane function. It's a perfectly valid definition. However, the stated objective of our test was a performance measurement, not a functional validation.

"And when the Cisco engineers said, "that feature was not supported in the line cards Cisco supplied for this project" they were asserting that these line cards do not do route filtering?"

We asked them to filter prefixes, and they said their equipment could not.

Regards,
David Newman
Network Test
tink 12/4/2012 | 8:36:15 PM
re: Internet Core Router Test David,

So we're clear that "route filtering" means, "to [accept,deny] specific BGP updates, with the accompanying effect of [forwarding,blocking] traffic sent to addresses within the networks described in those updates," as you state.

This conforms with RFC 1812 and the standard use of the term "route filtering."

And when the Cisco engineers said, "that feature was not supported in the line cards Cisco supplied for this project" they were asserting that these line cards do not do route filtering?

Thanks,
Tink
dnewman 12/4/2012 | 8:36:20 PM
re: Internet Core Router Test Our test is fully compatible with the section of RFC 1812 you cite. Vendors were free to [accept,deny] specific BGP updates, with the accompanying effect of [forwarding,blocking] traffic sent to addresses within the networks described in those updates. As stated earlier, I believe this is what Foundry did.

I don't know whether the Cisco 12416 can or can't do this. The Cisco engineers that configured the device told us that feature was not supported in the line cards Cisco supplied for this project.

Regards,
David Newman
Network Test
dnewman 12/4/2012 | 8:36:22 PM
re: Internet Core Router Test "There are differing opinions about packet filtering on core routers. But I doubt anyone would consider it as a feature in anywhere near the same importance as route filtering in BGP"

Finally, something on which we agree.

As I've stated, we explicitly asked vendors to filter specific routes, not specific packets.

I've pointed this out repeatedly and you seem intent on ignoring it. Why?

I've also refuted every other charge you've made about the test, so you keep making up new charges. Why are you on this fishing expedition?

Regards,
David Newman
Network Test
dnewman 12/4/2012 | 8:36:22 PM
re: Internet Core Router Test "Some of them (like me) find it incredible that you ignored route filtering (a crtical feature) in the testing"

Ah, but we didn't. We specifically asked vendors to filter routes. Prefixes -- not packets.

How much plainer can I make this?

Regards,
David Newman
Network Test
tink 12/4/2012 | 8:36:23 PM
re: Internet Core Router Test David,

Okay, so it seems that what you asked vendors to do was to prevent your end device from receiving any packets that could conceivably have been headed for a set of specific networks (or "prefixes"), no matter how the vendor achieved that goal. Yes?

But the terminology is still confused. I reread the filtering section of your article and in no place do you use the term "route filter," nor do you use "prefix filter." Only the guy who commented on the tests, Bill St. Arnaud, uses the term "route filter."

Later in your posts, however, you do use the term "route filter." For example, you say, "That package includes the methodology, which explicity asks vendors to do route filtering."

Loathe though I am to do so, I have to agree with Skeptic here: the term "route filter" applies to the filtering of routing updates. It does not mean filtering traffic based on prefix, no matter how this is achieved.

I was able to find the following in RFC 1812 (Requirements for IP Version 4 Routers):

"7.5.2 Basic Route Filtering

Filtering of routing information allows control of paths used by a router to forward packets it receives. A router should be selective in which sources of routing information it listens to and what routes it believes. Therefore, a router MUST provide the ability to specify:

o On which logical interfaces routing information will be accepted and which routes will be accepted from each logical interface.

o Whether all routes or only a default route is advertised on a logical interface.

Some routing protocols do not recognize logical interfaces as a source of routing information. In such cases the router MUST provide the ability to specify

o from which other routers routing information will be accepted.

For example, assume a router connecting one or more leaf networks to the main portion or backbone of a larger network. Since each of the leaf networks has only one path in and out, the router can simply send a default route to them. It advertises the leaf networks to the main network."

So Mr. St. Arnaud thinks that the router used in your tests can't do this? And you say, "The system we tested does not support route filtering, packet filtering, or any other type of filtering."

But doing this isn't exactly what you asked vendors to do (though it may be one possible way to do what you asked?)

Given that "basic route filtering" is defined by RFC 1812 and is a MUST for any IPv4 router, can it really be true that the Cisco product (as tested) can't do this?

Thanks,
Tink

skeptic 12/4/2012 | 8:36:26 PM
re: Internet Core Router Test Whether vendors did this with filters that screen on prefixes (the network part of an IP address) or for packets (specific destination addresses) is an implementation issue.

Yes, I'm very well aware there's a BIG difference between prefix and packet filtering. That's not the point. We told vendors we needed some prefixes filtered, and it was up to them to do so however best they could.
-----------------------------------------

Look,

The filtering that matters most in a core router is the filtering that goes on in BGP. Most people consider that to be route filtering. If you don't, I suggest you think about new terminology because you are confusing at least some people.

There are differing opinions about packet filtering on core routers. But I doubt anyone would consider it as a feature in anywhere near the same importance as route filtering in BGP.

skeptic 12/4/2012 | 8:36:26 PM
re: Internet Core Router Test I first saw the Canarie reaction about 48 hours before the article was published. I believed then and I believe now that the representative got it just about right: The system we tested does not support route filtering, packet filtering, or any other type of filtering.
----------------------------

David,

If you say route filtering to a service provider or most people who have worked with BGP, few if any of them are going to think you mean applying filters to transit traffic.

Some of them (like me) find it incredible that you ignored route filtering (a crtical feature) in the testing in favor of packet filtering test (desirable to some, but not critical in a core router).
dnewman 12/4/2012 | 8:36:29 PM
re: Internet Core Router Test "But they have to be "in" something. The "network part" of the address is either in the packet or it is in the route advertisement"

There are many ways to filter. One is straight packet filtering. Another is by filtering updates as they're advertised. Another is by doing a route lookup and making an allow/deny decision based on the prefix. There may well be other ways. Packet filtering is not the only way to do this.

"So, as I understand it, then, you are saying that for these tests you gave them the choice to

(a)remove prefixes by filtering routing updates and omitting certain routes from the routing table

or

(b)filter and ignore packets destined for routes or "prefixes" that WERE in the routing table"

This was not the choice. We examined packets forwarded (or not). The metrics stated in the methodology are packet loss, latency, and jitter.

Again, we focused solely on the externally observable. We did not inspect the contents of the devices' routing tables in this test (except in troubleshooting, for example to determine if a peer had dropped).

"Also, if this is correct, can you tell me the correct terminology for (a) and (b)? "

Since it's not correct, no, I can't :) Kidding.

I think your definition A is simply determining whether advertisements are propagated, and B is straight packet filtering.

There are many RFCs that discuss the use of route filtering for various purposes -- filtering martian addresses, filtering DDoS attacks, etc. The one that specifically addresses testing of filters in routing is RFC 2544.

Regards,
David Newman
Network Test
tink 12/4/2012 | 8:36:30 PM
re: Internet Core Router Test David,

I'm getting there, but I'm still a bit confused. Let me ask for some more clarification.

You say, "Whether vendors did this with filters that screen on prefixes (the network part of an IP address) or for packets (specific destination addresses) is an implementation issue."

So prefixes are "the network part of an IP address." But they have to be "in" something. The "network part" of the address is either in the packet or it is in the route advertisement (and is added to the routing table).

So, as I understand it, then, you are saying that for these tests you gave them the choice to

(a)remove prefixes by filtering routing updates and omitting certain routes from the routing table

or

(b)filter and ignore packets destined for routes or "prefixes" that WERE in the routing table

So, in your tests, Foundry did (a), Juniper did (b), and Cisco did neither?

Also, if this is correct, can you tell me the correct terminology for (a) and (b)? And is there an RFC that is relevant for this subject?

Learning every day...

Thanks,
Tink



dnewman 12/4/2012 | 8:36:32 PM
re: Internet Core Router Test "But I do not think this is how David is using the terms. He uses "route filtering" to imply filtering packets "

No, that's not correct.

Just to be clear on this point: The methodology unambiguously instructs vendors to filter on prefixes. We further reinforced this by giving vendors lists of prefixes, not addresses, on which to filter.

Whether vendors did this with filters that screen on prefixes (the network part of an IP address) or for packets (specific destination addresses) is an implementation issue.

Yes, I'm very well aware there's a BIG difference between prefix and packet filtering. That's not the point. We told vendors we needed some prefixes filtered, and it was up to them to do so however best they could.

We did NOT inspect vendors' filters for implementation correctness or tight code -- nor should we have done so. Those weren't stated metrics or objectives. The general issue here is that in the context of device/system benchmarking what counts is what is externally observable.

In this test I believe Foundry filtered on prefix, Juniper filtered on address (or more specifically on ranges of addresses), and Cisco was unable to filter at all with the specific gear they supplied.

Just to reiterate the point I made with skeptic, this does *not* mean the 1241X can't do prefix or packet filtering; most GSRs in production nets today do both kinds of filtering. What we observed in *this* project was that filtering was not supported with *this* specific set of line cards and software that Cisco supplied.

As noted in the article, Cisco says it will support filtering in future versions of the equipment we tested but the company did not name a release date.

Regards,
David Newman
Network Test
tink 12/4/2012 | 8:36:40 PM
re: Internet Core Router Test Skeptic and David,

I think you guys are arguing over two different uses of the same phrase. To some people, including me, "route filtering" means filtering through routing updates for certain network advertisements. And "packet filtering" means filtering packets based on some portion of the packet (including the destination address).

But I do not think this is how David is using the terms. He uses "route filtering" to imply filtering packets based on the destination route/prefix to which they are headed.

I think that "packet filtering" is better terminology since, to me, it implies that one must look inside the packet. This is clearly necessary in order to determine the intended destination.

Is there an RFC that addresses this?

Thanks,
Tink
dnewman 12/4/2012 | 8:36:42 PM
re: Internet Core Router Test Light Reading asked representatives of Canarie and two other service providers to read the entire test package and offer their impressions. That package includes the methodology, which explicity asks vendors to do route filtering.

There was no effort to confuse anyone into believing anything, so you're off base there.

I first saw the Canarie reaction about 48 hours before the article was published. I believed then and I believe now that the representative got it just about right: The system we tested does not support route filtering, packet filtering, or any other type of filtering.

This isn't a statement of opinion. It's a statement of fact, and it's something Cisco's engineers told us themselves.

You ask "Do you really think that the equipment cisco gave you did not support route filtering?"

Well, yes, I do. Cisco's engineers told me so. I have no reason to believe they misled me.

Note that the article makes clear that filtering is not supported in the line cards we tested. Cisco product manager Kevin Macaluso was pretty clear in making clear this was a future feature of the cards, and not related to the router chassis.

The does NOT say whether a 1241X chassis equipped with other line cards would support filtering. That's a hypothetical, and out of scope for this project.

Regards,
David Newman
Network Test

Regards,

gp2 12/4/2012 | 8:36:48 PM
re: Internet Core Router Test Trying to take a Service Provider's perspective
(to maximize the value of the test suite), what
about:

Throughput:
- delve into the effect of <100% throughput on
SP network performance.
- purhaps cover max forwarding rate in addition
to throughput.

ACL:
- continue covering this as the line between
edge and core gets greyed.

As with convergence and route flapping (which
indicated network performance):
- delve into mpls lsp limits (hitting/exceeding
limits) and how they affect network
performance.
- delve into BGP table limits (hitting/exceeding
limits) and how they affect network
performance.

Scalability:
- test maximum scalability of a single platform.
A larger platform can be a real value IF it
works.

Fault Toloerance:
- Card Pulls
- Fail Overs (Fabric/Route Processor)

Ease of installation/configuration/management:
- this is inherently subjective -> but you could
develop some objective test points.

-gp

skeptic 12/4/2012 | 8:36:51 PM
re: Internet Core Router Test Sorry if you're confusing me with an employee of Canary. I'm not. Sorry if you're unhappy that he doesn't like Cisco, but that's not my opinion. Personally, I think Cisco is a great company with some fine products and many, many talented people.
---------------------------

Mr. Newman,

This is not a matter of "liking" or not liking cisco. It is a matter of the accuracy of statements made and light reading leaving up material that it knows is not true about a vendor.


Light reading confused a person into believing that Cisco does not support route filtering and did not provide route filtering software to your test.

The person in question made his comments after being given that impression by someone. That someone has to be either you or light reading.

Don't you feel any ethical responsibility to make sure that if someone reviewing your results got such a wrong impression, that his comments should either be corrected or removed.

This is not a question of "liking" cisco or "not liking" cisco. This is a question of technical honesty on your part and on the part of light reading in printing material you have long now known is incorrect on this website.

A gentleman was given something by light reading to review, he came to a false conclusion based on materials presented to him by either light reading or you.

And beyond just giving him the wrong impression, nobody involved either caught the mistake or has corrected the mistake since.

Do you really think that the equipment cisco gave you did not support route filtering? Do you think its responsible for light reading to publish comments by someone who was obviously confused by the material presented to him into making a statement that nobody in their right mind would consider accurate?

If you think the statements are accurate, step up and defend them. If you think they are incorrect, why has light reading refused to correct the material or take down the material?

Do you think its ethical for a journalist to give an interview subject a wrong impression of an issue in order to draw out a negative comment which the journalist knows to be wrong on its facts. And then further for the journalist to publish that material, not doing a proper review or fact check and afterward refuse to correct it.
And finally, to blame the interview subject after the fact for those comments.


dnewman 12/4/2012 | 8:37:01 PM
re: Internet Core Router Test You mean besides banning anonymous posts from LR? ;-)

It's an excellent question. Here are some of the issues we're considering. This is just a starting point -- PLEASE feel free to add/subtract from this list.

--resiliency (OIR, failover, etc.)

--IGP (OSPF,IS-IS) testing

--include TCP as well as IP

--more BGP options, like next-hop, confederations, MEDs, etc.

--route redistribution

--MPLS. This is a big enough area to merit a test of its own (or several)

--service-level enforcement

These are just starting points. I welcome any and all suggestions for future tests.

One request: For clarity's sake, please indicate whether you're requesting something for a future test of routers, or for some other class of equipment. Thanks.

Regards,
David Newman
Network Test
gp2 12/4/2012 | 8:37:01 PM
re: Internet Core Router Test Network Test,
Given all of the stimulating debate, what changes
or additions(if any) are your considering for
the test suite.

its been very interesting!
-gp
dnewman 12/4/2012 | 8:37:01 PM
re: Internet Core Router Test
I'm not disowning anything. LR commissioned my company to do a test, which I did. I stand by that test.

The statements made by telco reps are clearly attributed to them. (For the record, LR asked them to review the *test results,* not the methodology. There were many more contributors to the actual methodology, including network architects at UUnet, Genuity, Sprint, and other service providers; all vendors that took part in the test and some that didn't; RFCs 2544 and 2889; input from participants in the IETF's benchmarking working group; and input from Spirent's engineers.)

I didn't write the telco-exec-response-to-test-result articles, nor did I interview those folks.

Sorry if you're confusing me with an employee of Canary. I'm not. Sorry if you're unhappy that he doesn't like Cisco, but that's not my opinion. Personally, I think Cisco is a great company with some fine products and many, many talented people.

Regards,
David Newman
Network Test
skeptic 12/4/2012 | 8:37:05 PM
re: Internet Core Router Test You're now attacking me for what someone else said. Please comment on the ethics of that.
------------------
Ok.

Light reading has stated that experts have reviewed your methodology. Given that they are reviewing your work, as an ethical person you are assumed to have reviewed those comments as they were solicited by an organization you were affiliated with.

Now given that the statements are on their face incorrect, as an ethical person you should have prevented light reading from putting them out. As an ethical person when (long ago) these comments were pointed out to you as incorrect, you should have taken them down or had a correction published. You did neither. This is material that light reading solicited and published. Not random comments by someone.

And if you are selectivly disowning material, please explain which material published by light reading in regards to you test that you will stand up for and which you will not.

I've made my case. Will you now make yours as to how you justify this ethically and in other ways?


censorMe 12/4/2012 | 8:37:08 PM
re: Internet Core Router Test This is so very interesting. I have no comment on the testing aspect of this thread, please bear with me. I had a post pulled and my LR account closed yesterday because I made a critical post about LR. David Newman attacks anyone who doubts his "expertise" and lobs grenades as answers to tough questions?? WOW....me thinks light readings future may not be so bright! Could the LR bubble be ready to POP? You can only piss off so many people for so long...that includes potential advertisers to...Has anyone found a good source / website for info on the Optical community?
Formerly, Light_on_dude
dnewman 12/4/2012 | 8:37:09 PM
re: Internet Core Router Test You're now attacking me for what someone else said. Please comment on the ethics of that.

While you're at it, please read the methodology, which explicitly instructs vendors to configure their devices to filter route prefixes, not packets as you've repeatedly and falsely charged.

Don't you have any real work to do?

Regards,
David Newman
Network Test
skeptic 12/4/2012 | 8:37:10 PM
re: Internet Core Router Test It's unfortunate you've chosen this route from the beginning, because we could have had an interesting discussion instead of ending up hating each other. Your loss.

----------------------

Mr. Newman,

I suggest you review comments made before the last couple of days on this very board. There is no hate on my side. I have issues with your methodology and how you presented results, but that doesn't translate to hatred on my part.

You are doing useful work, if you do it right.


skeptic 12/4/2012 | 8:37:10 PM
re: Internet Core Router Test I stand by my article and all statements within. I also sign my name to it,
-----------------------------

Are you really standing by this:

G«£The fact that Cisco doesnG«÷t have their route filter software ready is very significant,G«• he [Bill St. Arnaud] writes in a memo to Light Reading. G«£This isnG«÷t a trivial task.

This is from an expert that reviewed your methodology. Explain this and explain why these comments are still up on light reading after all this time.

As a journalist, do you think its proper to not review material related directly to your work product? After all, this person was reviewing your test methodology correct?

As a jorurnalist, do you think its proper to keep incorrect material up on the site after it has been pointed out to you that it is wrong and misleading with no correction.

Please comment on the ethics (in a financial responsibility sense) of getting an expert to make a highly negative comment that is obviously incorrect and then not correcting it even when it is pointed out to you.



skeptic 12/4/2012 | 8:37:11 PM
re: Internet Core Router Test Actually, skeptic, I *invite* criticism -- responsible criticism. You've leveled a bunch of baseless charges at me that seem to stem from some allegiance to Cisco, yet you're too cowardly to say who you are and what your motivation may be.
----------------------
Actually David, you invite criticism up to a point. If you don't like the question, you ignore it. If you get a question about "how" you did something, you answer. If you get a question as to "why", you don't. If you get a question critical of your methods, you blow it off.

I've endlessly put up enumerated lists of questions for you, but you don't answer them. You dismiss the questions as biased without ever explaining why you consider the particular questions to be biased.

And given your excuse for not responding to them now, lets see some consistancy. But somehow I would guess that your no-reply-without-a-real-name policy isn't going to extend beyond me.




ramborouter 12/4/2012 | 8:37:11 PM
re: Internet Core Router Test SKEPTIC WINS: hands down.

Unfortunately, most of the name calling is made by Mr Newman. There is no requirement to reveal anybody's name to join the discussion. As a professional journalist you should label the accusations aside and answer the questions. In other words, answer the criticism and ignore the name calling.

Moreover, it seems that when Mr Newman is cornered he resorts to the "Do you have any Cisco investments?" question. For God's sake answer the questions that Skeptic put forward plus many others. Questions about some of the tests having no value such as BGP capacity, MPLS capacity, etc... If you don't want to answer, why don't you close the discussion and move on to something else.


Frankly, if your work was a Masters thesis you would have been grilled on your methodology, analysis and conclusions.

I always thought that the "Light" in Light Reading means bright or light as in a beam of light. Apparently, this forum showed that Light Reading stems from Light Writing as in Loose or Not Heavy.
dnewman 12/4/2012 | 8:37:12 PM
re: Internet Core Router Test Sorry this wasn't made clear in the article. It's stated in the methodology but not until a footnote at the very end:

In all cases, packet lengths refer to the IP packet, from the first byte of the IP header to the last byte of the payload before the CRC.

For IP traffic, the link-layer encapsulation adds 8 bytes -- 4 bytes each for header and CRC. For MPLS traffic, the label adds another 4 bytes.

We do *not* count IPG in packet length because 1) no specification says you should and 2) it's variable -- IPG is actually what determines offered load. The minimum IPG is 1 flag byte at line rate; for any lesser oload the IPG is progressively larger.

Regards,
David Newman
Network Test
riolan 12/4/2012 | 8:37:13 PM
re: Internet Core Router Test To testers:
-you say 40 bytes IP packets
-Is the following correct
-add 4 bytes PPP header
-add 4 bytes CRC trailer
-add 1 byte interframe gap
-total 49 bytes by packet
or do you add more interframe gap bytes ???
Thamks
networking_legend 12/4/2012 | 8:37:17 PM
re: Internet Core Router Test dnewman: "If skeptic can't be bothered to stand behind his/her own charges, I'm not going to be bothered to consider such claims as anything but so much hot air from yet another anonymous_coward"

Suppose Skeptic is a Cisco employee as you seem to be implying. Does that make his/her questions any less important? Why are you having so much difficulty answering his questions? Frankly I'm interested in hearing what you have to say.


gladysnight 12/4/2012 | 8:37:17 PM
re: Internet Core Router Test Dr Newman,

I have some sympathy with your view, but I must also say that there may be very good reasons for skeptics reluctance to explicitly idenitfy himself and his employment circumstances. I am sure you don't need me to spell these out to you.

One thought that had occurred to me is that he might be a Juniper employee. I myself am employed by a vendor with whom I frequently have differences of opinion, and I don't identify myself for this reason. I know from experience that some managers take any criticism as treason, although I've yet to figure out why.

I believe there will be many others like me who are interested in the substance of skeptics questions, and your responses to them, minus the friction you've been generating amongst yourselves. While I don't agree with the strength of skeptics accusations of bias, I can see how he came by them, and I don't think that refusing to answer his other (perhaps more valid) questions can help your position.

Thanks

p.s. I wish. I can't hold a tune in a bucket.
mu-law 12/4/2012 | 8:37:20 PM
re: Internet Core Router Test "Cisco has a cool feature that allows IP routing-based path selection based on *source* address."

NPR? don't believe there's a fastpath for this though. Anybody out there know different?
gladysnight 12/4/2012 | 8:37:22 PM
re: Internet Core Router Test Dr Newman,

It doesn't matter who he is, or who he works for.

What matters is that the questions he asks are answered in an honest and straightofrward manner.

With all due respect, this you have resolutely failed to do. (imho) Even tho you claim to be independent and unbiased. You are the one who is causing the most doubt as to the veracity of this claim, by YOUR behaviour.

It seems (given your own stated view on re-ordering, for example) that his questions have merit, irrespective of his employers identity, or his personal motives for aksing them.

What is so difficult for you in answering them?
dnewman 12/4/2012 | 8:37:23 PM
re: Internet Core Router Test IP destination address only for the LR test.

Here's one for all you CSCO fans out there: Cisco has a cool feature that allows IP routing-based path selection based on *source* address. This is very useful for web hosting providers, where Customers A and B have traffic coming from the same box/cage (or going through the same distribution switch) and A pays for premimum service and B doesn't. With this source-based feature, it's possible to put A's traffic onto a fatter pipe -- no MPLS required.

AFAIK Juniper does not support this feature.

Regards,
David Newman
Network Test
riolan 12/4/2012 | 8:37:23 PM
re: Internet Core Router Test Thanks David, Just one more , do you do that only on the IP destination or (and also) on the IP source address? cb
dnewman 12/4/2012 | 8:37:24 PM
re: Internet Core Router Test Your various "questions" reflect a very strong Cisco bias, in that they're not very cleverly constructed to make Juniper look bad and Cisco look good.

Since you're now taking to accusing me of un-Americanism, let me remind of the chapter of American history from whence that term came: the McCarthy era. In that era reputations were ruined by anonymous and baseless accusations.

Show yourself, and quit embarrassing your employer.

Regards,
David Newman
Network Test

dnewman 12/4/2012 | 8:37:27 PM
re: Internet Core Router Test As stated in the methodology, we asked vendors to filter on prefixes.

We gave them the list of prefixes we used and asked that they write filters that allowed/denied access for alternating blocks of prefixes. The block sizes were groups of 12 blocks for the OC48 testbed and groups of 48 blocks for the OC192 testbed.

Blocks refer to the entries in our prefix lists (which are available in the URLs cited in the methodology).

For example, in the OC192 prefix list the first of prefixes is listed as 1.n.0.0/12, with 16 prefixes achieved by rolling the second octet. The next line is 32 prefixes of 3.n.0.0/13, the next line is 64 prefixes of 5.n.0.0/14, etc. Each line here is a "block."

The allow and deny cases simply said take this action on this group of prefixes.

Regards,
David Newman
Network Test

riolan 12/4/2012 | 8:37:28 PM
re: Internet Core Router Test The article is pretty opaque on the shape of the rules that were used. Is there some where a description of this set of rules (ranges , wild cards .......)
riolan 12/4/2012 | 8:37:28 PM
re: Internet Core Router Test Back to the 800K claim. Does someone have tried to download 800K on the M160 ?
skeptic 12/4/2012 | 8:37:30 PM
re: Internet Core Router Test Your claim that Juniper is hiding a bug is similarly ill-founded. Our experience was completely the opposite: Rather than trying to hide something, the vendor clearly and unambiguously explained to us why reordering occurred and why its architects made the design decision they did. This was not a bug; it was an engineering tradeoff Juniper made.
---------------

Sir,

Your statements are bizzare. Juniper has in no way explained archiecturally to few (if anyones) satisfaction what the packet re-ordering bug was.
If you have more information which are you not presenting as to what the nature of the bug is, I suggest you share it with everyone. So far as I know, you have not done that as of this time.

This post more than anything before, shows out in the open your bias toward Juniper and your willingless to JUSTIFY bugs as "design tradeoffs".

Once again, UNBIASED testing involves reporting results, NOT TRYING TO JUSTIFY THE BUGS OF A PARTICULAR VENDOR IN THE TEST REPORT. as you tried to do.

Now you can whine all you want to about bias on my part, but I'm not prasing Cisco am I? I dont see pointing out the truth about Juniper's faults is a harmful thing. You really seem to have a problem with that though. Why is telling the truth about juniper's bugs such a problem for you?


And the funny thing is, every time I put out a long message asking light reading and whoever about the biases and outright mis-statements in the test results & other material, you guys run and hide under the bed. The best I get is this sort of stuff challenging me with nothing more than hot air.

So we will try with a few questions one more time. You can find even more on this thread that went unanswered in the past:

1. What made you decide to focus test resources on packet filtering on a core router when its far down the list of features important to customers of router vendors. How do you explain your decision in terms of Juniper also having pushed packet filtering as a major feature for a long while before.

2. Who specifically misled one of your experts into thinking that Cisco did not have route filtering (as opposed to packet filtering) features on their router available for test.

3. Why did you go such lengths in the test report to justify Juniper's packet reordering bug? especially in light of your claim that you think its a bad thing. Why not just consider packet reordering a failure if it happens at all (applying the same standard you did in other tests).

4. Why didn't you call out the obvious bug in juniper's forwarding implementation which allows BGP to learn more primary routes than it can use in forwarding. Why did you continue the test to give juniper a high non-functional "number" that meant nothing and then call out that non-functional (2+ million) number out as "the most impressive single data point of the entire project" when in practice that number was the worst sort of useless testing noise that will push vendors in the wrong direction to produce products for useless benchmarks rather than for customers.

Now lets have some answers or at least some discussion this time.













skeptic 12/4/2012 | 8:37:33 PM
re: Internet Core Router Test As I've said before, the tests were not designed with any router in mind.
-----------------------------------

Then what was your motivation for testing such a relitively unimportant feature on a core router and one that only favored juniper.

Working against you is that Juniper's hype machine was behind this particular feature in a big way. And also working against you is that very few providers are going to consider transit filtering to be as important as you made it out to be in the testing. And finally working against you is that this test was positive only for juniper and nobody else.

And it doesn't help that Light Reading got one of its provider experts confused into making a strong anti-cisco comment based on a misunderstanding of what you were testing.

skeptic 12/4/2012 | 8:37:34 PM
re: Internet Core Router Test Is there any test equipment capable of generating streams of traffic with unique IP src/dest, port, protocol type (ie, discrete flows), and look for reordering on a per-flow basis? That would certainly put the argument to rest.
=============================================

The arguement will only be put to rest when Juniper explains the nature of the bug in its forwarding path that causes reordering in enough detail to understand it.

If the testers were not (unfortunatly) biased in favor of juniper, there would be no controversy and Juniper would have (probably) had to explain why they reorder.

Instead of that happening, they got an "unbiased" test to make nonsense arguements in favor of reordering and now (apparently) have set their pet testers off on the path to prove that a major bug is somehow harmless.





skeptic 12/4/2012 | 8:37:34 PM
re: Internet Core Router Test The article notes that Juniper claims a FIB capacity around 600,000 to 800,000 entries for the M160.
------------------------
However the article does not note as far as I recall that the Juniper will begin to operate incorrectly when the memory for the FIB on the line card is exhausted.







skeptic 12/4/2012 | 8:37:35 PM
re: Internet Core Router Test The only way we'd be able to say for sure that reordering has or does not have an impact on TCP behavior is to run the tests with TCP. We didn't do that this time, but we sure plan to in the future.

------------------------------------------------
Reordering is bad. You can run tests in isolation to try and pretend otherwise, but nearly everyone learned years ago that any reordering showing up in a forwarding device is a bad thing.

You should be failing people who reorder packets, NOT trying to justify errors in particular vendors (i.e. Juniper) packet forwarding solutions.

And beyond that, I'm still left wondering why you have been so hands-off with regard to getting an explaination from Juniper as to the specifics of what is causing the reordering. They certainly know by now the specifics of the problem in greater detail than any amount of "TCP testing" is ever going to show.

And yet Juniper gets a free ride while it will not explain the nature of the problem to either its customers or external testers.


dnewman 12/4/2012 | 8:37:36 PM
re: Internet Core Router Test Thanks for your inquiry. Not sure I follow your argument, but I do see some incorrect assumptions:

1. There is no direct correlation between the number of routes and number of packets in or out of order. In the Juniper architecture, the line cards don't know/care about what prefixes they're dealing with; it's all just packets from the perspective of the line card.

2. We didn't say reordering was on a per-connection basis, nor did we say reordering is proportional to packet spacing. Unfortunately, we don't know the answer to either question. The numbers presented here are aggregates of all flows on all interfaces over the entire test duration.

The only way we'd be able to say for sure that reordering has or does not have an impact on TCP behavior is to run the tests with TCP. We didn't do that this time, but we sure plan to in the future.

Regards,
David Newman
Network Test
blee008 12/4/2012 | 8:37:47 PM
re: Internet Core Router Test David,

I understand why 220,000 routes were tested in the 40-byte IP packet test for the FUTURE core routers. However, the results show a lower packet reordering percentage since each traffic microflow (SA/DA pairs) has a lower pkt/sec rate when distributed across 220,000 routes (or 220000*47/48 on each OC48 ingress port to be more precise). As stated in your article, the current Internet has about 96,000 routes. Since packet reordering is on a per-connection basis and propotional to how close the packets are to each other, 96,000 routes should be a realistic test to depict the packet reordering issue on today's Internet (assuming we drive the network to 100% which no one does). If we would test it with 300,000 routes, we might not have seen this "packet reordering" issue at all on OC192 interface.

Comments appreciated. Thanks.

dnewman 12/4/2012 | 8:37:48 PM
re: Internet Core Router Test Thanks for your interesting post. You're exactly right about readv events converging faster than flap events; we also saw this behavior in the Lone Router test back in '99.

I frankly don't know why this would be. One possible explanation -- and this is purely speculative -- is that it may less stressful to reroute onto paths in cases where a path already exists. In other words, in the flap case a device suddenly has no path and has to do a lookup; in contrast, the readv case merely asks the device to change from one established path to another.

Cisco did not use any specialized equipment for this test. Except where noted, all vendors used the exact same hardware and software for all tests.

You asked for the number of updates per session. For the flap test it's 0.25 times the total number of prefixes (~202K for the OC48 testbed and ~220K for the OC192 testbed) divided by the number of edge interfaces (12 for the OC48 testbed and 48 for the OC192 testbed). For the convergence test delete the 0.25 multiplier.

There may or may not be a correlation between update rate and number of updates. As your two examples show, it's a weak correlation at best: Any attempt to describe a linear relationship between the two is probably futile.

Note also that the flap and convergence test results shouldn't be directly compared. They're different: Convergence involves updating the full table, while flap involves only 25 percent of the table. We run flap at 100 percent of line rate and convergence at 90 percent. The flap/readv intervals are 30 seconds in the flap test and 60 seconds in the convergence test. And so on. They're different tests, and the results should be considered separately.

Regards,
David Newman
Network Test
stud_405 12/4/2012 | 8:37:53 PM
re: Internet Core Router Test Can someone (David?) explain the strange performance behavior of forwarding table updates in the OC48 tests?

Juniper:
1.In the flapping tests it took about 20 sec to update 50000 routes. In the Convergence Testing it took about 30 sec to update 200000 routes!!!
2.In the flapping tests they had better convergence in the re-advertise section. Why?

In Cisco
1.In the flapping tests it took about 30 sec to update less than 50000 routes. In the Convergence Testing it took about 60 sec to update 200000 routes.
2.In the Convergence Testing their performance was step function. It looks like special line card for the test with preconfigured forwarding instractions and start/stop triger.

I think that there may be correlation between the update rate and the number of updates per BGP session. What was the number of BGP updates per session in each test?
zaf 12/4/2012 | 8:38:19 PM
re: Internet Core Router Test Just wondering on test gear used. I recall Ixia had claimed being selected by Charlotte's web networks as their platform for testing the Arena product, does it show now that the wrong toolwas used in development? does it show that a QA low level tool was used for the initial design verification test? did anyone asked about or comented of the type and capabilities of the test tools which show what was required in the first place....
dnewman 12/4/2012 | 8:38:35 PM
re: Internet Core Router Test The article notes that Juniper claims a FIB capacity around 600,000 to 800,000 entries for the M160.

I don't know the capacity of Cisco's forwarding table. The BGP table capacity of 390,000 is useful upper bound; even if the forwarding table has a larger capacity, this is the largest number of BGP routes it can hold.

Regards,
David Newman
Network Test
ud17 12/4/2012 | 8:38:41 PM
re: Internet Core Router Test The article talks about BGP table capacity in the control plane but not the hardware forwarding table capacity.
How much is the forwarding table capacity for Cisco's OC192 and Juniper's M160? and how much memory have they put on board for the forwarding table ?

thanks
PG
dnewman 12/4/2012 | 8:39:05 PM
re: Internet Core Router Test
Rambo, Irish,

Please.

Let's keep the discussion focused on the technical merits (or otherwise) of these products, and NOT on calling one another names.

Can't we all just get along?

Thanks.

Regards,
David Newman
Network Test
ramborouter 12/4/2012 | 8:39:08 PM
re: Internet Core Router Test IrishHeart,

I'm sure we are all touched by your moderation and wise words. But are you the same guy who posted message 122?? Did you get a brain upgrade (from Microsoft)??


irish_heart 12/4/2012 | 8:39:13 PM
re: Internet Core Router Test Dear Light Reading,

Honestly, and with great respect to all your message posters - I find it disconcerting that so many people post messages here that are designed to challenge the findings or methods involved in your test design. I asked myself, can all of these people be angry Cisco employees, and I suspect that they are not all Cisco employees.
So, I guess I can assume that there are a great many strong Cisco GSR product advocates in our marketplace. Everyone is entitled to their opinion, and it makes for good discussion. But, I am just blown away that after two weeks since you reported the test results, you still receive so much challenging email on the subject.

The truth is, that both Cisco and Juniper make a very good product. Take your pick. Each make distinct architectural claims and I guess it is a matter of who you believe. One thing is clear. LR ran a set of tests, and for this particular set of tests, Juniper won hands down.

Let's all stop for a moment here and swallow that bit of news. For some, no doubt, it will taste good going down, and for others, it will taste bitter. Now, if we all agree that the test was close, let's consider a couple of other reasons that in my opinion explain why Juniper should have won.

First, they have been shipping production OC192c for a year now, and their competition, Cisco, is yet to ship production OC192c, even as you read this message. I'm not bashing anyone, I'm jsut speaking facts.

Second, Juniper has delivered product on time that perform reliably and deliver line rate with solid functionality.

This two last facts cannot be tested by LR, as they are not testworthy in a lab environment. But, if you know ANYTHING at all about the market, we know that these two statements are true.

So, if you haven't already considered doing so, shouldn't you be taking a closer look at the M160 instead of challenging the LR folks here in the message board.

The day's of, 'you never got fired for buying Cisco', are over.
dnewman 12/4/2012 | 8:40:19 PM
re: Internet Core Router Test Sorry, but *we* didn't use any filters -- the vendors crafted them all. Unfortunately we didn't think to ask vendors to save their configs; it's something we normally do in private tests, and it's something we will definitely do for the next public test.

You might try following up with Juniper or Foundry regarding these settings.

Regards,
David Newman
Network Test
avi 12/4/2012 | 8:40:24 PM
re: Internet Core Router Test Hi

Could you please give an example for an
accept and deny filters you used in the test?

Tanks

dnewman 12/4/2012 | 8:40:26 PM
re: Internet Core Router Test Simple:

--best IP is an amalgam of all IP (non-MPLS) test results

--best MPLS is an amalgam of all MPLS (non-IP) test results

--best OC48 is an amalgam of all OC48 test results

--best OC192 is an amalgam of all OC192 test results

Regards,
David Newman
Network Test



dnewman 12/4/2012 | 8:40:27 PM
re: Internet Core Router Test As I've said before, the tests were not designed with any router in mind.

We had no idea going into this that Juniper's syntax would require 150K lines, Foundry's a few dozen. We also had no idea that Cisco's line cards don't support filtering.

We just did the methodology, and the vendors brought what they brought.

Regards,
David Newman
Network Test
mczhou 12/4/2012 | 8:40:32 PM
re: Internet Core Router Test Hi, everyone
I want to earn some test reports of two specific router, but I don't know how to find them because I'm a new comer.
So who can tell me that? I will really appreciate your help.
yuning 12/4/2012 | 8:40:49 PM
re: Internet Core Router Test Yes, that's just the point. In the test article, the tester define 900 rules to filter the 100,000
entries. But what's in the real world ? I believe
in the oc48/192 interface - it means backbone connection, we seldom define ACL more than a dozen
layers. We just don't want to turn our transit router into a firewall - filtering in the aggregation layer is a basic practice to maintain scalibility.

So it seems this test item was purposefully
"designed" for someone.

yuning 12/4/2012 | 8:40:49 PM
re: Internet Core Router Test We need to clarify the router architecture first. I don't' know much
about M160, but GSR is a fully distributed system, switching fabric
for inter-line card traffic exchange, and line card for packet process.

Yes, the bottleneck lies in the packet rate and bit rate. For Juniper
it has a so-called "Internet Processor II" ASIC, which seems integrating
the indexing tech for packet lookup - so Juniper can handle large ACL
with great ease. While for GSR, they get L3FE ( Layer3 Forwarding
Engine) in the line card, and eng 0 to 4.

So I don't agree with you that "forwarding, moving packet between memory"
is bottleneck for M160, in fact moving packet in memory you need
just change the pointer, no actual memory copy.

My conclusion is: GSR get a better switching fabric than M160, while
M160 get a better L3FE than GSR, and M160 can handle more PPS with
great ease (in given lineate, more small packet). But GSR is clever
than M160 in that it achieve a balance between switching & L3F in
the Imix traffic context, while more PPS for 40 bytes packets is just
a useless luxury for M160 in real world Internet. Then it's very clear
who is the real world winner. Cisco just verify an old Chinese saying:
"Good steel should be used in the edge of knife".
avi 12/4/2012 | 8:40:53 PM
re: Internet Core Router Test Hi

I have been looking at the former discussions and nobody (as far as I saw) mentioned that the major problem with the Cisco 12416 was the lack of ACL support which resulted in a failure at about 40 tests (OC48 and OC192 filtering).
I would like however to ask why is the tested filter is so large (150000 line codes), what is a "typical" filter size (rang)? and what did this filter do (in general) ?

Thanks

avi 12/4/2012 | 8:40:53 PM
re: Internet Core Router Test Hi

I have been looking at the former descations and no body (as far as I saw) mantioned that the major problem with the cisco 12416 was the leck of ACL support which resulted in a failure at about 40 tests (OC48 and OC192 filtering).
I would like however to ask why is the tested filter is so large (150000 line codes), what is a "typical" filetr size (rang)? and what did this filter do (in general) ?

Thanks

ronhui 12/4/2012 | 8:40:53 PM
re: Internet Core Router Test Could anyone tell me what kind of RAM is used in Cisco and Juniper router, SRAM or DRAM?

I wonder whether the memory speed can keep up with the transmission technology when OC-768 introduced.
z2000 12/4/2012 | 8:40:59 PM
re: Internet Core Router Test
Can someone tell me how to get the G«£at a glance resultsG«•, I am not talking about weight for different tests... Just curious which results (or detailed results) contribute to best IP, to best MPLS and so on...

Thanks,

dnewman 12/4/2012 | 8:41:17 PM
re: Internet Core Router Test You don't happen to have an interest in Cisco, do you? :)

The article didn't "fail" to mention any retesting of the M160 on the OC192 test -- because there wasn't any.

As for optimizations, turning your argument around I could conclude that the M160 is better optimized for IP traffic because it did so much better at handling short packets than did the 12416. Ergo, I could say that the 12416 is most efficient when driven no faster than 52 mph.

Now I don't think that inference is any more valid than the one you're making, but I think you get my point.

Regards,
David Newman
Network Test
ramborouter 12/4/2012 | 8:41:20 PM
re: Internet Core Router Test Juniper is like a car manufacturer whose cars are most efficient when driven at 5mph.

C'mon, in every design there are trade-offs: smart designers optimize for the most common case. It seems that Cisco optimized for a better traffic mix.

What's funny is that the article made sure to mention that after tinkering with the config, Juniper engineers where able to bring the throughput to 100% from 99.8% on OC48: big deal!!!
On OC192, the article fails to mention whether the same tinkering was applied to achieve the whopping 89 or 90%. Juniper cannot even explain how they did better handling 40 byte packets than Imix.
This brings us back to the original thesis that they optimized for the wrong traffic.

Robert
ronhui 12/4/2012 | 8:41:21 PM
re: Internet Core Router Test Cisco and Juniper use two different kind of switching architecture. Cisco use input-buffered architecture and Juniper use a distributed output buffered architecture. For Juniper, packets are cut into cells. A long packet is stored in different buffers. When outputting a long packet, the cells has to be read in a particular sequence. That may explain why the throughput is lower when using Imix traffic.
hulin 12/4/2012 | 8:41:24 PM
re: Internet Core Router Test I think there are two kind of bottelneck for swiching capablity. One is packet rate sensitive bottleneck, the other is bit rate sensitive bottleneck.
Some processing in switch is based on packet, like policy control,swiching control. The length of the packet will not effect the speed of the processing. The short packet means high packet rate, so short packet will be more critical for such bottleneck.
Some other processing has very close relation with the bit rate, like forwording,swiching, moving packet between memory.Packet rate is not very important for such process, the most important thing is the total bit number it need to deal with every second. So bit rate will be very critical for such processing.
From the result of OC192 testing, I think the bottleneck of Cisco is mainly the first kind of bottleneck, Juniper is the second.
It means Cisco's bottleneck is on interface card, Juniper's is on the share memory.
The reason for thoughput of Imix traffic is some lower than 64 bytes traffic, I think it is because two kind of traffic has different overhead, it means different payload bit rate. Payload bit rate is the theshold.
Above is my personal opinion. Welcome discussion.
dnewman 12/4/2012 | 8:41:28 PM
re: Internet Core Router Test Both the Smartbits and Adtech generator/analyzers can alter any byte of any packet. The Smartbits can autoincrement any field, which is how we were able to generate thousands/millions of unique addresses without a lot of manual labor.

That's not enough to ultimately settle the question of impact of reordering on TCP traffic. For that real TCP sessions are needed. We're working on that, and hope to present information specific to TCP in the not-too-distant future.

Regards,
David Newman
Network Test


nmsguy 12/4/2012 | 8:41:30 PM
re: Internet Core Router Test David, can you provide any insight into how future tests will be able to make any conclusions re. the impact of reordering on TCP sessions, as you hope?

Is there any test equipment capable of generating streams of traffic with unique IP src/dest, port, protocol type (ie, discrete flows), and look for reordering on a per-flow basis? That would certainly put the argument to rest.

Also, did your tests with both 40-byte traffic and IMix generate packets with random / variant header information (other than dest address) in each packet, or were the contents of the IP headers identical for each stream of traffic?

It would be interesting to see any samples of the data streams actually submitted as traffic for these tests - is that possible?

..john
spike 12/4/2012 | 8:41:37 PM
re: Internet Core Router Test Hi Gentelmen
Can some one tell anything about the other two companies, well the one that is still around... it seams that they have good potential to join the league
tink 12/4/2012 | 8:41:44 PM
re: Internet Core Router Test David,

You say, "I set up no ranking scheme, as you suggest." My point is that any set of tests equally weighted _is_ a scheme if used to conclude a "winner."

I agree that throughput is a needed measurement. I am not suggesting that forwarding rate tests are a substitute for throughput tests. Rather, they should be used to augment such tests and give a more complete picture.

Thanks,
Tink

dnewman 12/4/2012 | 8:41:49 PM
re: Internet Core Router Test adventures with cut and paste :)

Thanks for citing this, because it clearly makes my point. In the quote you cite I note the M160 achieved more "best" results than any other box -- flapping, convergence, baseline latency, latency variation, and yes, BGP and MPLS capacity.

Note that this statement applies to all the results in total; we certainly didn't give the M160 the thumbs-up solely because it built twice as many transit LSPs as Cisco's 12416.

Regards,
David Newman
Network Test
ramborouter 12/4/2012 | 8:41:49 PM
re: Internet Core Router Test Dave, the paragraph I referred to in my previous message is this:
"In some areas, the M160 is in a class by itself: It holds more BGP routes and more MPLSlabel-switched paths than any other box. It deals with network instability far better. And it
exhibits much lower average latency and latency variation."

I forgot that the less than and greater than signs are off limits.

Cheers

Robert
ramborouter 12/4/2012 | 8:41:51 PM
re: Internet Core Router Test Forget the sensationalism of the article title and the first couple of pages.
Forget some of the marginal tests.
Think about the new OC192 from Cisco and how well it did against Juniper's. Think of how it may start to widen the shrinking market share gap.
And think of what that might do to Juniper whose life depends on shrinking that gap.

The one thing to take from this experiment is that the two products are very well matched.

The next couple of quarters will tell the story.

Take care of those packets %-)

Robert
ramborouter 12/4/2012 | 8:41:52 PM
re: Internet Core Router Test Hi Dave,

#2. So you can do 2.4 million routes on a Sun box? #Running what routing daemon? Don't tell me gated, #because it would emit loud retching noises a #looooong time before you got to 2.4 million.
I'm by no means suggesting that in the next round I'll enter the competition with my SparcRouter.
1- As others have pointed out if you want to attach a tape drive to keep your routes you will no doubt win this test.
2- The 700000 that M160 achieves before using the disk is quite admirable. But as your document mentions it is a matter of memory upgrade.
For all intents and purposes, as mentioned before, this test can not be used to differentiate the two routers.

#I don't recall stating that the M160's MPLS #capabilities were in a class of their own. #Indeed, the text suggests that both Juniper and #Cisco have work to do in scaling up their code #(as do we -- with refresh reduction support in #our test gear, we almost certainly would have #achieved higher numbers with both Cisco and #Juniper).
Dave, in Part 1 there is the following paragraph:
<<in a="" and="" any="" areas,="" average="" better.="" bgp="" box.="" by="" class="" deals="" exhibits="" far="" holds="" in="" instability="" is="" it="" itself:="" label-switched="" latency="" latency<br="" lower="" m160="" more="" mpls="" much="" network="" other="" paths="" routes="" some="" than="" the="" with="">variation.>>
From my writing skills' limited knowledge, the idea expressed by the first sentence of a paragraph is supported or illustrated by the following sentences. Thus, it seems that M160 is in a class of its own because it holds more BGP routes and more MPLS paths...

Anyway, enough grammar.

#I appreciate your taking the time to write, and #apologize for the rather cranky tone of my note.
Dave, no problem, my tone was not desirable either.

Thanks,

Robert</in>
dnewman 12/4/2012 | 8:41:54 PM
re: Internet Core Router Test Tink, I don't know if your point concerns weighting schemes, or the definition of throughput, or your unhappiness that we didn't measure device forwarding rates in the way you suggest. I'll try to cover each here.

As you probably know it's very unusual that writers pick the headlines for their articles. That is the case here. In reporting "winners" in each category, that was a simple matter of examining the best result in each category.

I set up no ranking scheme, as you suggest. Once again, every event, all nine of them (or all 16 for those vendors that completed both the OC48 and OC192 tests), weighed the same.

I have no clue why you would conclude a test measurement of 52 percent oload constitutes headline hunting. It's a data point, one among many.

Finally, I disagree that forwarding rate is suitable substitute for throughput. Here's why: It lets vendors off too easy.

Look at the numbers in this test: Two vendors' routers out of four were unable to forward packets without loss, yet both were able to forward oloads in the high 90s. Packet loss is important; some apps suffer quite a bit because of it. Forwarding rate alone doesn't tell that story.

Regards,
David Newman
Network Test

dnewman 12/4/2012 | 8:41:56 PM
re: Internet Core Router Test
Hi Robert,

Arrgh! The scorecard issue again.

You say Juniper "won" 2 less tests if we discount filtering. OK, but then you say you're sure some tests are more important than others. I gather, then, that filtering isn't important on your network.

And I'm perfectly fine with that. I am allergic to ranking schemes that impose some arbitrary view on readers. That's why we *didn't* give any one event higher priority over any other. Feel free to do this if you like. I'm not in the business of deciding your priorities for you.

2. So you can do 2.4 million routes on a Sun box? Running what routing daemon? Don't tell me gated, because it would emit loud retching noises a looooong time before you got to 2.4 million.

While we're at it, can your Sun box forward 700k (Juniper) or 390k (Cisco) of those prefixes?

The 2.4 million number has been criticized a lot -- without justification, in my not so humble opinion. The article clearly states that it is an academic excercise because of system limits, and the chart clearly notes the FIB capacity of 700K and the point where swapping begins (at 1.4M).

3. I don't recall stating that the M160's MPLS capabilities were in a class of their own. Indeed, the text suggests that both Juniper and Cisco have work to do in scaling up their code (as do we -- with refresh reduction support in our test gear, we almost certainly would have achieved higher numbers with both Cisco and Juniper).

Rather, my statement referred to the M160's superior showing in more tests than any other box tested. You yourself acknowledged this in your comments on Table 1.

4. If a box goes at 52 percent of line rate for more than half of all the packets it handles, I would think that's something a prospective buyer might want to know. But then I might be accused of doing what I'm arguing against, which is making value judgments about what matters most to readers.

Suffice to say it is standard practice to benchmark devices with minimum-size packets. If these numbers aren't relevant for your needs, please ignore them.

5. Latency. Agreed. Hear! Hear! (though it pains me to see as estimable a source as Tony Li say it's not an important benchmark)

6. Again, we didn't weight any one result any more heaviliy than any other. If your network design requires that packets are never ever reordered under any circumstances, feel free to pay more attention to those results.

Also, for the record, I should note that packet sequencing was not one of the metrics stated in the test methodology we gave to vendors before testing began. The Smartbits happens to record this information in the course of measuring packet loss and latency, and Light Reading asked me to analyze the reordering numbers along with the other metrics noted in the methodology.

I appreciate your taking the time to write, and apologize for the rather cranky tone of my note. You've raised a number of points in a reasonable way -- far more reasonable, in fact, than the way some others on this board have made some of these charges.

Again, thanks for the feedback.

Regards,
David Newman
Network Test





tink 12/4/2012 | 8:41:56 PM
re: Internet Core Router Test So if no two providers agree on the relative weighting of tests (or need to), why not just report the test results and leave out the "winner" crap?

You _did_ set up a ranking "scheme" that imposes "some arbitary view on readers" by choosing a particular set of tests, giving them equal weight, summing it all up for a "winner" and leading your article off with the declaration of this winner.

The situation is intractable unless one has the wherewithall to resist making headline-oriented summary statements. No one scheme can satisfy all the possible uses, so no one scheme should be used to declare a "winner." Can we just freakin' dispense with the notion of winner?!

And you say, "If a box goes at 52 percent of line rate for more than half of all the packets it handles, I would think that's something a prospective buyer might want to know." What does "goes at" mean? And "for more than half of all the packets it handles"?

The 12416 has throughput of 52% for 40-byte packets, but when handling internet mix (more than half of which is 40-byte packets) it "goes at" wire rate. Everyone seems to want to have their cake and eat it too. Oooh, the Cisco box drops 40-byte packets at 52% of line rate, and oooh, aren't 40-byte packets really important in the internet? But, wait, the Cisco box does just fine on imix. More headline hunting if you ask me.

The point is that (despite its very careful definition in the RFC), throughput does not provide the whole picture, especially for 40-byte packets. A prospective buyer wants the most complete information possible. And for completeness, forwarding rates at a variety of offered loads are useful.

Thanks,
Tink
dnewman 12/4/2012 | 8:41:58 PM
re: Internet Core Router Test 40-byte IP packets, OC192

See Figure 3

Regards,
David Newman
Network Test
spike 12/4/2012 | 8:42:00 PM
re: Internet Core Router Test Hi David,
Thanks for the information, and thanks for performing the test and sharing the results with us.
ramborouter 12/4/2012 | 8:42:01 PM
re: Internet Core Router Test This is definitely a better summary of the results. I strongly disagree with the by the numbers sections where

0 - Number of vendors that walked out of the lab claiming line-rate performance.

According to the results Cisco's gear achieved line rate where it mattered most (OC192 for Imix).

Maybe dnewman can explain where cisco failed to achieve line rate.
ramborouter 12/4/2012 | 8:42:02 PM
re: Internet Core Router Test I have few comments about the testing and the
conclusions. Most of these have been raised already but here is my take on them.

1- By looking at the results' summary (i.e. table-1) we see that:
cisco won 4 tests
juniper won 7 tests (5 excluding filtering)
tied for 5 tests
If wins and ties count for one point then Cisco
scores 9 and Juniper scores 12 (or 10 without filtering). However this would be a very naive way to look at these tests. I am sure that some of the tests are more important then others.

2- For example the BGP capacity test is insignificant because of the swapping and the fact that today's networks would be adequately served by both routers. Nevertheless David Newman considers that "the M160 is in a class of its own". As somebody has already put it "my Sun box would have won that test".

3- The MPLS capacity tests is also insignificant. To justify this I'll use dnewman's arguments when he says "service providers already are looking to build networks comprising thousands or millions of tunnels" (in part 11 of 15). Cisco can handle 5000 tunnels while Juniper can do 10000. This means both products are unacceptable if the criteria is ~100 thousand tunnels. Again, dnewman here claims that Juniper is in a "class of its own"; the truth is that in this category both routers are in the losers class.

4- The internet mix results should have more weight in evaluating the performance, since the 40 byte packets are unrealistic. Imagine telling a service provider that RouterX can do line rate of 40 byte packets but can only do 50% when the traffic is real internet. This is significant because it means Cisco's router handles line rate
in both OC48 and most importantly OC192 ( see part 5 of 15).

5- The latency has already been beaten to death, so I'll pass.

6- Even though the packect mis-reordering effect is not clear yet, I am sure it is more important than some of the tests conducted, namely MPLS and BGP capacities.

David, congrats on your work. Getting those companies to commit for such an experiment is an achievement in its own merit. This is definitely a good start to when the real Terabit routers arrive.


Keep up the good work

Robert
dnewman 12/4/2012 | 8:42:05 PM
re: Internet Core Router Test
Very good question The numbers are somewhat different depending on whether you count packets or bytes.

We counted packets.

Regards,
David Newman
Network Test
dnewman 12/4/2012 | 8:42:06 PM
re: Internet Core Router Test
Thanks for your inquiry. To respond:

1. The OC48 forwarding figures are aggregates across all 12 edge interfaces. Sorry we didn't make this any clearer.

2. Dunno the capacity of the forwarding table for Cisco's 12416. At least as far as exterior routing goes, the hard limit here is 390,000, the maximum number of BGP updates the device can hold.

Regards,
David Newman
Network Test
dnewman 12/4/2012 | 8:42:06 PM
re: Internet Core Router Test
I'd like to clear the air on the point of testing and payment.

Network Test does not accept commissions from vendors. Our primary sources of revenue are end-users (both service providers and large enterprises) and publications like Light Reading.

We've maintained a no-vendor-testing policy to avoid getting into the "rent-a-quote" game.

Regards,
David Newman
Network Test
tfrench 12/4/2012 | 8:42:06 PM
re: Internet Core Router Test Apologies for the obviousness of this question,.. but when you are talking about the prevalence of different packet sizes, are the percentages based on number of packets or throughput in bytes ?

regards

Tim
dnewman 12/4/2012 | 8:42:07 PM
re: Internet Core Router Test With any test involving line-rate traffic we take steps to avoid any other contention for the wire.

Please note that in the methodology we asked vendors to set keepalive timers way high to avoid contention when we offered line-rate traffic.

In Cisco's case the management timers were set in the billions of seconds, so I don't think there was contention there.

Regards,
David Newman
Network Test
dnewman 12/4/2012 | 8:42:07 PM
re: Internet Core Router Test Hi Tony,

Thanks for your suggestion. Coming from you I take it very seriously.

A couple of observations:

--latency IS one of the metrics that multiple Tier-1 providers urged us to use. In researching this methodology we talked with UUnet, Genuity, and Sprint, and others...all of them said latency would be a useful measurement of device performance

--There is no question that single-hop latency is not meaningful in the context of end-to-end measurements. Of course speed-of-light propagation dwarfs single-box latency over large distances.

This is essentially the old services-vs.-devices measurement question (or, in IETF-speak, ippm-vs.-bmwg). The services folk would say (and indeed have said) I can't characterize a beach by describing individual grains of sand. Fair point, but by the same token I can't describe a beach without saying what sand is.

I don't hold a brief for either service- or device-oriented measurement. I do both for private clients. Neither is right or wrong; they're just different.

Thanks again for your post.

Regards,
David Newman
Network Test
dnewman 12/4/2012 | 8:42:08 PM
re: Internet Core Router Test Hi Spike,

There's a link to the paper on page 8 of the LR test article, the section on packet reordering.

Regards,
David Newman
Network Test
spike 12/4/2012 | 8:42:11 PM
re: Internet Core Router Test Hello,
Can some one help us (those who did not read the paper...) and tell us where we can obtain a copy of it?
Thanks
whatafunnyworld 12/4/2012 | 8:42:12 PM
re: Internet Core Router Test >Please read the subject line, it says "I wrote >the paper.." it does not say "I performed the
>testing for LR", all of my comments on the subject relate to tests performed across the
>internet NOT the testing done for the LR article.

jon: sorry misunderstood, I just followed this thread. sorry again... :o

Tony
spike 12/4/2012 | 8:42:12 PM
re: Internet Core Router Test A. If they have paid (which I do not say nor imply for it) it is part of good marketing strategic. Thumbs up for both of them G«™
B. What the heck is the correlation between shares and products quality? These two companies are having the best engineers and managers to develop top quality products, and the knowledge how to do a minor thing with them G«Ű sell them J
C. Thee who wants to be in the same playground with the "big guysG«• shouldn't he play by the rules on that particular playground...

Instead of criticizing try to learn from them G«™
briand 12/4/2012 | 8:42:18 PM
re: Internet Core Router Test (I don't work for Cisco, never have; I am familiar with their equipment, however. That familiarity is the basis of this post.)

I have some questions, in order to further understand the fundamental nature of the equipment being tested, which was partially illustrated by the results. This understanding is necessary for anyone who might actually spend tens of millions of dollars on this equipment.

(The opinions of bystanders without wallets is not very relevant.)

What *exactly* were the configurations on the interfaces?
Specifically:
- was CDP enabled (on Ciscos)
- was layer-2 keepalive enabled (HDLC/PPP)
- what other protocols were running on the interface
- was BGP run *over* the interface used for traffic? (This can be avoided by setting next-hop, and using a second link for BGP. BGP, while not as "chatty" as other dynamic protocols, is state-ful, and has keepalives.)

While this probably is only relevant to the parties for which traffic was run at line-rate (ie, Cisco), understanding the exact configs will help understand other relevant factors, eg queue depth versus maximum loss-free load versus latency.

Clearly, offering line-rate traffic to a router, which will then send it over a link with any keepalives for any protocol, will run the serious risk of non-zero queue depth being established, and maintained (since packets arrive at line-rate.)

Clarifications of this would be greatly appreciated.

Brian Dickson
jcrb 12/4/2012 | 8:42:21 PM
re: Internet Core Router Test
>because your team choose Juniper is
>winner,I guess this (mis-ordering) is not big >deal,but I didn't see any message about
>mis-ordering is nothing in your report.
>So what should we choose ? what's your opinion ?

Please read the subject line, it says "I wrote the paper.." it does not say "I performed the testing for LR", all of my comments on the subject relate to tests performed across the internet NOT the testing done for the LR article.

All the testing of the Cisco and Juniper boxes
was done by David Newman.

nuff said

jon
ksy 12/4/2012 | 8:42:21 PM
re: Internet Core Router Test Two Questions:
1) In the figure 2 of the article, OC48 Forwarding rate value seems off. I do not understand how can you have a OC48 (~2.5G) input and over ~73 millions 40 byte packet (73MX40byteX8bit= ~23.5G) output.
For Internet Mix traffic with average packet size at ~461.7 byte. This same unusual outcome happened again (~2.5G in but over 28G out). Did I miss anything? Please advise.

2) In the article, it states that Juniper forwarding table is about 700K entries, does anyone know how large is a Cisco's forwarding table? In many cases (as stated in other previous messages), forwarding table size is equally, if not more, important than routing table size.
whatafunnyworld 12/4/2012 | 8:42:26 PM
re: Internet Core Router Test Dears:
My friend just told me (who he is also CSCO and JNPR customer) LR just did a big favor to JNPR. I didn't belive this. After I read this test result. I believe it. who win ? JNPR win, who lost? CSCO and LR lost, and LR lost his credit.
why? the key is in OC-192, this is the war place what I see. Yes, I agree David Newman using throughput which under good condition how many packets the line card can forwarding.this may based on ATM concept, however, packet discard, packet broaden or packet size changed or packet mis-ordering all are not normal situration. If you use ADtech send CBR traffic, cell lose and mis-ordeing clee all are bad situration. But LR just choose packet discard, ingroe packet mis-order. What I see is CSCO only got 52% throughput and JNPR got 56% throughput in OC-192. Unless JNPR can pass packets without packet mis-ordering under 92% throughput.
What's your opinion ? I am welcome CSCO and JNPR guys and David Newman's opinion. refresh my dead brain.

p.s. 56% is based on part 8, packet ordering part.
p.s.2 will they have difference impact to network for packet discard and packet mis-ordering ? can some one explain for me ?

thanks.

Tony Ngn
tony1athome 12/4/2012 | 8:42:27 PM
re: Internet Core Router Test David,

May I respectfully suggest that you reconsider your testing of router latency. While latency is perhaps an interesting measure of a system, in a WAN system where speed of light propagation delay typically dwarfs switch latency, this is less interesting. In practice, this has never come up as a serious technical issue in this market segment.

I would suggest that you direct your energies towards tests as suggested by a variety of Tier 1 ISPs.

Thanks,
Tony Li
lyricji 12/4/2012 | 8:42:28 PM
re: Internet Core Router Test There's different ways for Cisco and Juniper to handle the BGP routes when the memory is runned out, Swap space is used by Juniper to save the upleaking bgp routes, but releasing the swap space is very unbearably slow after backing up, well Cisco clear the memory by rebooting the box.

The ways is so different, which should not be told which one is better to handle the bigger BGP routes table...

dnewman 12/4/2012 | 8:42:30 PM
re: Internet Core Router Test I'd like to jump in here. I conducted and wrote the Light Reading test. Jon cowrote the seminal paper on which I based much of the discussion on reordering. Jon's paper used live Internet links; it was *not* a bakeoff between different vendors' routers.

The Light Reading test data is inconclusive with regard to reordering. We showed that it does exist under some conditions with Juniper's OC192 card, and that while it's nowhere near as prevalent a problem as some have claimed neither does it have zero potential impact.

We explicitly did not show that the existence of reordering would or would not have an impact on TCP sessions. We hope future tests will be conclusive on that point.

Regards,
David Newman
Network Test
dnewman 12/4/2012 | 8:42:30 PM
re: Internet Core Router Test
I don't believe the plen (packet length) stats directly correlated packet length with upper-layer protocol. However, the CAIDA folks have a LOT of data on both packet length and protocol mix -- you may find your answer there.

Regards,
David Newman
Network Test
whatafunnyworld 12/4/2012 | 8:42:30 PM
re: Internet Core Router Test jon:
the funny thing is the war between Juniper and Cisco. while few months ago, Juniper's high level guy told media (sorry,I forgot where I saw, just one Cisoc engineer send to me) that they did'nt have packet mis-order problem. Now, you show they had. Did Juniper lying or cheat to customer and media ?
Sorry, I am not technical guy, and from you point of view (sorry, althought you have detail analysis this in your report) will this mis-redering impact our network? because your team choose Juniper is winner,I guess this (mis-ordering) is not big deal,but I didn't see any message about mis-ordering is nothing in your report. So what should we choose ? what's your opinion ? thank you very much.

cheers
sroy 12/4/2012 | 8:42:34 PM
re: Internet Core Router Test Simply amazing, is it possible to tell if the 52-byte packets are TCP or UDP?
sgamble 12/4/2012 | 8:42:34 PM
re: Internet Core Router Test Is it possible to get copies of your test scripts for your Smartbits/Adtech?

S/
dnewman 12/4/2012 | 8:42:34 PM
re: Internet Core Router Test No, it really was fifty-two (52) bytes. We were as surprised as you are, but in the traffic samples we examined (from Merit -- see the methodology for the URL), 52 bytes was the fourth most prevalent packet length.

Note that 576 bytes was the third most common length.

Regards,
David Newman
Network Test
sroy 12/4/2012 | 8:42:35 PM
re: Internet Core Router Test I understand where some of the length distributions used in the test come from, the only confusing one is the 52 byte packet.

40 byte packets (the minimum packet size for TCP) which carry TCP acknowledgements but no payload,
1500 byte packets (the maximum ethernet payload size) from TCP implementations that use path MTU discovery,
But shouldn't one get 576 byte and 552 byte packets from TCP implementations that don't use path MTU discovery, and not 576 and 52?
Is this just a typo?

Snipped from discussion on internet traffic mix.

"We also used live Internet samples from Merit to develop a traffic pattern we called Internet mix
(Imix). This was based on the IP (Internet Protocol) packet size and percentages sampled over a
two-week period: 40 bytes (56 percent of all traffic); 1,500 bytes (23 percent); 576 bytes (17 percent);
and 52 bytes (5 percent G«Ų with totals adding to more than 100 because of rounding)"
jcrb 12/4/2012 | 8:42:39 PM
re: Internet Core Router Test
>Clearly, you don't even have a remote clue of >what you are talking about.

I don't have any idea how to even begin to respond to this comment.... so I'll respond to your other comments since for those I have some ideas..

>If you wrote some paper about this topic,

Im pretty sure that I did write a paper about this topic...

>I hope you included a small section on how >ashamed you feel working for Cisco.

If I did work for Cisco I don't see what I would have to be ashamed about, but I don't now, nor have I ever worked for Cisco. In fact when I was at BBN doing the resarch for this paper I was working on some consulting contracts for two companies that wanted to compete with Cisco in this space.

>From the testing we've done here in Denver,

I am afraid that you have me at a disadvantage, you know who I am and what testing I performed, would you care to tell me who is 'we' and what testing have you performed?

>it's clear that your paper is full of lies
>and mis-information about this particular issue.

How about you start listing all the 'lies' and 'mis-information' that are in our paper, either that or kindly retract your comment.

>Your analysis and statements about it are
>largely farce.

Please feel free to explain what you mean by this. While I try to make my commentary 'entertaining' I also try very hard to make it as technically accurate as possible.

>You are one of the reasons that Cisco is losing
>market share in this market-place.

Well actualy, I hope that I *am* one of the reason thats Cisco loses market share, but I am interested in their share of the cable/edge router market not the core router market.

regards

jon

optical illusion 12/4/2012 | 8:42:40 PM
re: Internet Core Router Test red herring page 58 march 20 edition

"Digital Canals"

see www.dutchwater.com

dnewman 12/4/2012 | 8:42:40 PM
re: Internet Core Router Test
Cisco has not taken to bashing these results, nor have it "bashed" the LR team. Nor has any other vendor. In general, the feedback we've received from vendors before, during, and after the test (including after publication) has been overwhelmingly positive.

On this point, I know of at least one case where Cisco corporate has asked an employee to cool it with the name calling. I join Cisco in their call to keep the discussion on testing and not on calling one another names.

Play nice,

David Newman
Network Test

optical illusion 12/4/2012 | 8:42:41 PM
re: Internet Core Router Test Be nice if LR outed those drooling morons at RED HERRING for running the APRIL's FOOL's piece on unlimited bandwidth thru the water pipes.

<

>

idiots.

tink 12/4/2012 | 8:42:41 PM
re: Internet Core Router Test Irish-heart,

Jon C.R. Bennett did not write a packet-reordering paper about Juniper and he didn't write it for Cisco. His paper analyzes test streams sent across a live network link. The tests measure the amount of reordering already present in the net and the paper explains some of the consequences of such reordering. The 1999 IEEE paper was co-authored by Craig Partridge and Nicholas Schectman of BBN.

Jon doesn't work for Cisco, but for RiverDelta Networks. Here's what they have to say about him:

Jon C. R. Bennett
Jon is Chief Engineer of RiverDelta Networks. Prior to joining RiverDelta he was a member of the
Architecture Board of Xylan Corporation, a manufacturer of high-speed network switches. Previously, he was a member of the Internetworking Research Department of BBN--one of the top-ranked networking research groups in the world. He came to BBN from FORE Systems, where he designed the per-flow queuing technology used in all of FORE's ATM switches. Jon has a BS in Applied Mathematics/Computer Science with Research Honors from Carnegie Mellon University and is an ABD Ph.D. candidate at Harvard University. He holds 9 patents in the networking area and is a member of the Editorial Board of the IEEE Magazine Network.

Of course, I'm sure you know more than these three researchers, but I do suggest that before you seek to discredit their research you take the time to READ it. Naturally, this will involve mastering reading comprehension first, but given the substantial genius you have shown on these message boards, I have confidence that you can do it.

Tink



irish_heart 12/4/2012 | 8:42:43 PM
re: Internet Core Router Test that Cisco, rather than looking at the test results here with an open mind, have taken to bashing the LR team that put this test together.
They've accused the LR team of not having any experience with Juniper. They've posted lies about Juniper's product. Have you no shame?

LR - you did a great job and a service to the industry running these tests. Cisco fans: give these guys a break. Take your medicine like an adult and stop having a temper tantrum because you didn't win.
irish_heart 12/4/2012 | 8:42:44 PM
re: Internet Core Router Test Clearly, you don't even have a remote clue of what you are talking about. If you wrote some paper about this topic, I hope you included a small section on how ashamed you feel working for
Cisco. From the testing we've done here in Denver, it's clear that your paper is full of lies and mis-information about this particular issue. Your analysis and statements about it are largely farce.

You are one of the reasons that Cisco is losing market share in this market-place.
flanker 12/4/2012 | 8:42:44 PM
re: Internet Core Router Test
Packet loss is the killer for voice applicaitons. Latency is irritating but packet loss of greater than 1.5% is noticeable. At 3% you lose syllables, at 5% you lose words, at 9% you cannot have a duplex conversation, at 13% to 17% simplex voice is is unintelligible.



BottyGuy 12/4/2012 | 8:42:44 PM
re: Internet Core Router Test XoIP wrote:
>Could some one explain why Service Providers
>require 100ms buffering.
>

Large buffer sizes are used to minimize the loss of packets. Buffers provide some elasticity during the *hopefully short* periods of congestion. The 100ms requirement comes from the expected round trip time in IP networks (which tends to really be anywhere from 0ms to 200ms). Routers in the network implementing these large buffers will implement some form of RED (Random Early Discard) or Weighted RED. These will drop packets with increasing likelyhood as the buffers fill up.

Nicely conforming TCP implemementions on clients and servers will detect the packet losses or the increase in round trip delay and reduce their packet window sizes. The reduced window sizes will provide some congestion relief to the network. Also since we used Random Early Discard to drop the packets at the router we have a greater likelyhood of dropping packets from TCP connections that are the greatest contributors to the congestion problem, so the lower bandwidth connections are not as affected.

If a Service Provider feels their average round trip time is on the order of 100ms these size buffers will provide enough buffering to absorb congestion events until the TCP bandwidth hogs back down. This increases total network throughput since most TCP connections will not lose packets. Without the "round trip" buffers and RED implementations lots of TCP connections could lose packets at the same time and the network would experience waves of packet retransmissions and congestion.

Of course not all TCP implementions are nicely conforming, although most of the common ones are. and UDP connections could care less.

On most routers the average buffer depth will be very low until the load reaches 100%. So during most traffic conditions the latency is low. You will only see latency increase at 100% load. I personally think that latency should probably be tested separately at 95% and 100% load to see how a device actually responds.

For example if one device actually passes all its traffic at 100% with rather large latency and another device only passes all its traffic at 99% with low latency you are not comparing apples to apples in terms of latency.


dnewman 12/4/2012 | 8:42:46 PM
re: Internet Core Router Test Some applications are very sensitive to delay.

For example, humans can perceive audio quality degrading on voice calls when latency exceeds about 70 ms. That's why calls bounced over satellite links, where delay can be in the 200-600 ms range, sound lousy.

Video is even more delay-sensitive; the human eye's persistence of vision lasts for about 10 ms; any greater delay may be perceived as choppiness in the image.

Regards,
David Newman
Network Test
xoip 12/4/2012 | 8:42:47 PM
re: Internet Core Router Test Could some one explain why Service Providers require 100ms buffering.

XoIP
dnewman 12/4/2012 | 8:42:48 PM
re: Internet Core Router Test "I mean burstiness in the destinations of the packets. If many packets bound for the same output port happen to arrive at the same time queues can build up temporarily."

To the extent this occurred it was behavior introduced strictly by the systems under test and not by any other factor.

Juniper and Charlotte's Networks both accused us of having some asymmetry in our traffic patterns, leading to transient overload on the kind of micro scale you describe.

In fact the offered load was always COMPLETELY symmetrical. For several months before any vendor came in, we worked very hard on this aspect of the scripts specifically to avoid the problem of asymmetry.

Ultimately both Juniper and Charlotte's Networks backed off their accusations (seeing the Adtech traces showing equally spaced packet departure times helped a lot here). We gave them packets in a symmetrical pattern; any transient queue buildup that existed was caused solely by their routers.

Regards,
David Newman
Network Test
rma 12/4/2012 | 8:42:51 PM
re: Internet Core Router Test `I don't believe buffer depth matters AT ALL from the perspective of what is externally observable. There are few networks that behave thusly: "Ooh! I have to send some data through brand foo routers! I better send just X bytes at a time to avoid hitting a limit with foo's buffers and causing higher latency!"'

Yes. One would hope in actual network operation mechanisms such as RED would be employed to keep average queue depth and latency down at the expense of a bit of throughput.

`With regard to burstiness of the offered load: It was negligible. Packet departure time variation on the Smartbits OC-48 POS cards was in the low nanoseconds, verified with the Adtech AX-4000 and with a logic analyzer.'

I mean burstiness in the destinations of the packets. If many packets bound for the same output port happen to arrive at the same time queues can build up temporarily. This depends on the pattern of destinations sent to each router port and how they might phase align. An extreme example would be to send a large burst of test packets first to destination A, then B, then C, etc. Over the entire test the load is balanced but on a micro scale large queues would build up then drain showing a large average latency.

rma


dnewman 12/4/2012 | 8:43:01 PM
re: Internet Core Router Test Thanks for an interesting and substantive post. Lots of good stuff here.

Fully agree about the possibility (nay, probability) that router queues filled partially during test runs.

I think you got it right about this leading to some corner cases. For example, Juniper's throughput fluctuated between 89 percent and 90 percent with the Imix load on the OC192 testbed.

I don't believe buffer depth matters AT ALL from the perspective of what is externally observable. There are few networks that behave thusly: "Ooh! I have to send some data through brand foo routers! I better send just X bytes at a time to avoid hitting a limit with foo's buffers and causing higher latency!"

That said, of course I agree with your comment, and those of others, that there's a "hockey stick" pattern where latency climbs sharply just before drops occur. The shape of that hockey stick differs with each box (and indeed with each offered load).

Step tests that track latency vs. load are an excellent idea. I appreciate the suggestion for future tests.

With regard to burstiness of the offered load: It was negligible. Packet departure time variation on the Smartbits OC-48 POS cards was in the low nanoseconds, verified with the Adtech AX-4000 and with a logic analyzer.

Finally, goodput cannot be derived from any of the results here. Our data plane traffic consisted of IP packets, not TCP. Goodput is a useful measurement but out of scope for this project. Just as these numbers don't say anything about what goodput might be, goodput is of no use in determining device throughput, a key objective of this test.

Regards,
David Newman
Network Test



rma 12/4/2012 | 8:43:08 PM
re: Internet Core Router Test The latency tests may only be measuring queue depth for serveral reasons.

The offered load attempted to match the routers' "throughput rate", the maximum rate at which there are no drops. However it could have been just slightly above the routers' real throughput rate for part or all of the tests. The tests could still exhibit no drops if:

(offered load - real throughput) * test time < buffer size

In other word the queues may have been filling up slowly but the test ended before they reached their maximum size and dropped packets.

Burstiness in the traffic pattern can also cause temporary queue build up and skew latency. It's not clear that the traffic pattern itself didn't cause congestion for some part of the test. Queueing may handle that without drops, but at the
cost of latency.

Service Providers typically want at least 100ms of
buffering so even a very long test run could experience this behavior. Its no surprise that some of the results were in this range when the routers were tested on the edge of congestion. The exact throughput rate may not even be a single point in such complex environments.

Another factor that is not discussed is what kind of class of service queueing was enabled in the tests, perhaps by default. Number of queues and the queueing discipline can affect latency. Measuring latency and jitter across multiple classes would give misleading results. A more
interesting test would have been to set up a class of traffic needing low latency and measure those results.

Because the different tests used different offered loads its not possible to make meaningful comparisons - even for the same router. A graph of latency vs offered load would have been a huge improvement. Reporting latency as narrowly defined by the RFCs does not give any clear picture of these routers performance. (Likewise for goodput vs offered load).

rma
mumbogumbo 12/4/2012 | 8:43:13 PM
re: Internet Core Router Test what could juniper protest? the headlines says "JUNIPER WINS!"

have you ever heard of a winner contesting their own victory.

/mumbo g
rpm 12/4/2012 | 8:43:17 PM
re: Internet Core Router Test As several other posters have mentioned, some idea of what the distribution of latencies is would be very helpful. This is the reason that people are asking for std deviation, median, and quartiles in addition to mean and min/max. (min/max was interesting back when Harvard's NDTL measured the latency of only 10 packets at very low loads).

I think that the most interesting latency metrics would be mean, min/max, std deviation, and the 95 percentile latency (i.e, at a particular load, latency is less than this 95% of the time).

This would give us an idea of how big the tail of the distribution is. BTW, You shouldn't worry too much about overwheming anyone with data. For a product of this type--more is better.

The other thing about latency, as noted by others, is that measuring it only at the throughput rate is not good and not good enough, in spite of what the RFC might say. At the throughput rate without congestion, a buffer could fill completely (max forwarding rate very slightly less than 100%) or not at all (perfect 100% forwarding rate).

Decoupling the latency measurement from the throughput is necessary because the time to clear the buffer can easily swamp the latency number--where latency is defined to be the packet processing time of the device, not the total time spent in the device.

This could be achieved by measuring latency at offered loads of 50%, 75%, 90% and the throughput rate.

irish_heart 12/4/2012 | 8:43:22 PM
re: Internet Core Router Test Thanks for running these tests. Regardless of the controversy, which there has already been way too much attention given to it. I think that this is really useful that you folks have made the effort to do a bake-off. It was a lot of work, and given who the audience is for this test, I guess you must have expected a lot os strong opinion. A lot of people have invested a lot of time and energy into learning Cisco Systems products over the years. Emotionally, I think it poses a challenge to hear that maybe there is a new kid on the block (Juniper) that just might have a better idea. If I were a Cisco afficianado, I'd be thinking about adding some Junos experience to my resume' right about now.

I happen to agree with LR's conclusions, based on personal experience with both products.

Self-acclaim is no acclaim. I am really glad that Juniper had the restraint to stay out of this controversy.
tink 12/4/2012 | 8:43:28 PM
re: Internet Core Router Test Jperser,

I would prefer to see average, standard deviation, and median for latency. My main argument is that standard deviation and median give more information than min/max.

Time is always the limiting factor isn't it? As for where my votes would go:

1) reporting frame loss rate for several offered loads - This is interesting information. I have run tests on routers that had a throughput of 50%, but could only get 40% of traffic through when offered line rate.(?!)

2) reporting average and st. dev for latency

3) I like the longest match test, but can you report more data (again avg and st. dev. for latency, did one or both of these numbers shift significantly)?

4) basic congestion management ... how does overloading the router change the forwarding rate and latency, what tools can be used to minimize the damage?

5) I think the QoS tests could use some fleshing out, but I'm not expert enough to suggest how. All I know is, I doubt I'm ever gonna be tweaking knobs to get specific ratios of traffic. Instead, I'll be thinking about how to deliver services that have guarantees. If my gold and silver customers are fighting over bandwidth, gold is gonna win. Every time.

Thanks,
Tink

jperser 12/4/2012 | 8:43:30 PM
re: Internet Core Router Test Tink,

The Frame loss rate can be calculated for any of the test runs that David did. The Smartbits recorded the numbers in a huge spreadsheet. The frame loss rate from a single port to any other single port is recorded in the spreadsheet. The author condensed all 1728 stream statistics into a simple metric.

I have two questions:

1. At what point does the reader get overwhelmed with numbers? If minimum, maximum, average, mean, standard deviation, meridian, percentile latency are reported, can the average reader understand the difference?

2. Given limited time and resources, what would you like to see in a core router test? Realize it took a long week just to run each of the switches through the 9 tests.

Jerry Perser
tink 12/4/2012 | 8:43:32 PM
re: Internet Core Router Test David,

In the interest of seeing a meaningful RFC come out of your work, I'm going to beat a dead horse into the ground here. I'm sure you are tired of reading my mail!

Please consider picking up an elementary statistics book (easy to find in used bookstores near college campuses) and reading the first two or three chapters. In the tests you ran, minimum and maximum do not convey much meaningful information. If you are making multiple measurements there are tried and true means for analyzing and reporting the data. These include: plotting the distribution, calculating the mean, the median, and even the mode. Others have stated an interest in seeing the "quartiles." This would be good, too.

Thanks,
Tink
dnewman 12/4/2012 | 8:43:33 PM
re: Internet Core Router Test
No RFC yet. We're working on it. :)

At the time RFC 2544 was written, there was no concept of min and max latency because most tools didn't measure that way.

The test described in 2544 is a blip test -- it measures the latency of ONE packet halfway into the test run. Since this is one measurement, min and max are obviously the same number. This method is fine as far as it goes, but it says NOTHING about variation in latency over the test duration.

Modern tools like the Smartbits now measure latency of EVERY packet received. With multiple measurements, it's now possible to say what min and max are.

I'll respond to your question about test duration in a minute -- am looking up something on that point now.

Regards,
David Newman
Network Test
tink 12/4/2012 | 8:43:34 PM
re: Internet Core Router Test David,

Your graph in the latency section says, "lower numbers and narrower ranges are better." Where ranges, I presume corresponds to the range between min and max.

I cannot find anything in the RFC (2544) about min or max or ranges thereof. As you know, I argue that min and max are meaningless (and that st. dev. and median are meaningful). Is there an RFC or draft that discusses min and max and why "narrower ranges" might be considered "better"?

Thanks,
Tink
tink 12/4/2012 | 8:43:34 PM
re: Internet Core Router Test David,

RFC 2544 defines a method for measuring "frame loss rate" at several offered rates of traffic. Perhaps this is a test that could be included in the next round of tests?

Thanks,
Tink
cowboy_phil 12/4/2012 | 8:43:36 PM
re: Internet Core Router Test Not only do I agree with you but either I suddenly cannot read results or there was another test not shown here that allowed Juniper to be the winnder. On the MPLS test - Cisco scored higher yet came in second - ??????

OC48 - Cisco did better but they tied ????

who in the heck declared Juniper the winner and what test were they looking at
dnewman 12/4/2012 | 8:43:42 PM
re: Internet Core Router Test We examined packet reordering on all devices. There was no reordering on Cisco's OC192 cards in any test.

Regards,
David Newman
Network Test
seize_today 12/4/2012 | 8:43:53 PM
re: Internet Core Router Test Did you test packet order on the Cisco OC192?
Reviewer 12/4/2012 | 8:43:53 PM
re: Internet Core Router Test The test and article scores a 7/10.
Too bad it does not score a 10 /10 .

The difference is that there is a lot of journalism.. but also a little bit too much sensationalism.

It is more interesting to know "Who is best for what" then "Who is best overall". That's journalism vs sensationalism

Since you give all the result, good techies can figure it out. Not mere mortal.

I would - as a lot of your passionate readers - would love to have a synthetical article on Who won what... and therefore Who is best for what.
Kendo 12/4/2012 | 8:44:09 PM
re: Internet Core Router Test Glad to have seen this test results.
Good work for all 3 companies(LR, NetwrokTest, Spirent) people.
Kendo 12/4/2012 | 8:44:09 PM
re: Internet Core Router Test > In measuring latency of IP packets, we follow
> long-established precedent of measuring at, and > only at, the throughput level.

Just a thought.
Sicne this is a two horse race.
Would be be more meaningful to demonstrate the latency @ throughput of, ex., 98%? Which both router can achieve and results into latency.

As the result stated. JNPR give up the .x% throughput with better, more consistant latency number through ingress FIFO queue tuning.

At the same time, it is kind of silly to compare a 12 months old chip than a 3 months old. The process is just not the same.
jcrb 12/4/2012 | 8:44:20 PM
re: Internet Core Router Test wanted to make a few comments...

From what I can see, most of Junipers spin on our paper is incorrect.

a) we said 90% of TCP *session* would see reordering, not 90% of *packets* (although for some sessions in our tests it was close to 90% of packets)

b) packets dont get reordered by being 'swaped' i.e. given packets
1 2 3 4 5 6 7 8
you dont usualy see
1 3 2 4 5 6 7 8
what you see is
1 2 6 7 8 3 4 5
or you see
1 3 5 7 2 4 6 8
what happens is that some form of load balanceing is going on and one path is faster or the other path stalls for a moment and all the packets on one path jump ahead of the packets on the other path.

As for the likelyhood of any given out of order packets belonging to the same TCP flow that depends on how far out of order a packet can get. If for example one path can stall for 20 milliseconds or is just a little bit slower and has a queue of packets to be processed that is 20 milliseconds long then any TCP flow sending more than 500 packets per seconds would be guaranteed to see reordering. The size of the link is not an issue nor is the number of flows sharing the link.

One way to apreciate this is that we did the testing for our paper by sending single flows of packets through the biggest exchange point we could find at the time to get our results. We were not measureing a device in the lab and looking at the ordering of all packets, just the ordering of our lonely stream of packets sent out into the big bad internet.

Oh and the bit about the next router down the line being likely to put them back in order....... the guy with the comment about smashing the pot twice and hoping to get an unbroken pot the second time had that one covered perfectly.

jon
pauljharper 12/4/2012 | 8:44:21 PM
re: Internet Core Router Test Thanks for the Test, even though it seems to generate more heat than light
it is never the less very useful to have hard numbers to judge things on..

I would find it still more useful, as others have mentioned
to have the average ,10, 50th(median) 90th and 95th percentile bands
for the delay distribution. One problem here is that to determine
these bands involves sorting all the delay times, which can take a
long time if you have millions or billions of packets. A solution to determine
these numbers is just to sample say every Nth delay so that you have a sample of about 50,000
delay times, sort them, then calculate the percentile numbers.

Another thing is that packet loss comes from buffer overflow and as such is a probabilistic thing.
(Theory of large deviations)
'No packet loss' just means a very low probability of packet loss (so low that none was observed
during the time window of your test).
In reality a packet loss of 1 in 10^6 or lower should not cause any noticable problems on a network,
and it would be more useful to have a packet loss ratio than a simple on/off (no loss/loss) situation.

Yet another point is that most real networks are not run at 100% utilization (most core OC192 routes seem to
be running at 20-50% utilization, and OC48 routes are running slightly higher).
I know that RFC 2544 specifies throughput as load where there is no packet loss, but this is presumably just
for the duration of the test (sixty seconds recommended)
i.e. probabiliy of packet loss is lower than 1/(numpackets in 60 seconds)

It would be useful to have average, std dev, and 95th percentile delay curves vs percentage load.
The above mentioned packet loss ratio could also be plotted as log10(packet loss ratio) vs percentage load.

Paul Harper
xerxes98 12/4/2012 | 8:44:23 PM
re: Internet Core Router Test Maybe if the editorial about the results started off something like the following, there may have been less controversy:

"In the 34 head-to-head tests where Cisco and Juniper competed directly, Cisco came out on top. Cisco was able to produce line rate in 7 out of 8 IP / MPLS tests using OC-48 and OC-192, whereas Juniper was able to produce line rate in 0 out of those same 8 tests. However, Juniper was able to win all tests in which it was either the only vendor competing or when it was competing solely against Foundry."

Any comments?
JKM 12/4/2012 | 8:44:24 PM
re: Internet Core Router Test In answer to your question "is there anybody out there" - try Marconi - they are combining 10 years of ATM, with 6 years of IP and 3 years MPLS (w/TE) development onto existing and new platforms.
dynge 12/4/2012 | 8:44:40 PM
re: Internet Core Router Test There is no way that this test can clearly mark Juniper as a better product. If you take out the filtering section of the test, which Cisco did not offer in this model of their router, then the contest would have been neck and neck. I think it is premature to declare Junipers Core Router "in a class by itself"
rpm 12/4/2012 | 8:44:41 PM
re: Internet Core Router Test This just points out how subjective the question of who "won" the test can be. With mixed results, it all depends on which tests one focuses on. Having the bests results on more tests doesn't necessarily equate to "winning".

Also, just because LR contracted with Spirent and Network Test for the tests doesn't mean they have the right to declare the winner, other than in their own opinion. Rather, the reader of the test descriptions and the resulting charts must decide independently who "won". Cisco is saying that in its reading of the results, it "won". Surely Cisco is biased, but what do you expect?

Bottom line: Nothing to retract....
haiyingdeng 12/4/2012 | 8:44:41 PM
re: Internet Core Router Test You do not have to abandon your ATM knowledges.

Please refer to RFC3055 "MPLS using LDP and ATM VC Switching".

You can keep your ATM backbone, add an ATM-LSR (lable switching router) to the top of each ATM switch, then you convert this ATM switch to a IP/MPLS router. Basically you can add layer 3 function to your ATM switch, layer 3 IP packet is tagged with layer 2 ATM cell by MPLS and forwarded over SVC and cell rate. This is a combination of IP, MPLS and ATM technology. The baseline is to use ATM VPI/VCI as MPLS lable, using ATM switching to do MPLS switching.

Both Newbridge and Cisco have this technology and mature product in the market, it protects your investment on ATM switches and add layer 3 function to ATM core by using MPLS technology.

And your can merge your ATM cloud with IP router cloud easily.

You can get training from both Newbridge and Cisco on ATM-LSR product.
dnewman 12/4/2012 | 8:44:42 PM
re: Internet Core Router Test
Sorry, the reasoning doesn't work in this case.

RFC 2544 specifies that latency is measured only at the throughput level. There's a good reason for this: Any observation at higher offered loads (i.e., with loss) includes latency PLUS queuing delay. Any number achieved in such circumstances is not an atomic measurement of device latency.

In our OC48 tests, we did measure latency the Charlotte's Networks and Foundry devices in the presence of packet loss. The issue for both vendors' boxes is that throughput was 0, and thus we could not make an RFC 2544-compliant latency measurement.

I'll be the first to say that "latency" is not an accurate label for these measurements; "delay" is a better term.

While there is value to delay measurements at various offered loads above and below the throughput level, those measurements were out of scope for this project. We represented to vendors that min/max/avg latency were the metrics we would use.

Regards,
David Newman
Network Test






D&K 12/4/2012 | 8:44:42 PM
re: Internet Core Router Test I have watched this message board debate the merits of the test criteria and while we are looking at the nuts and bolts of it, I am curious about a few fundamental factors.

After having looked at what Nortlel (and of course others) brought us with MPLS, I am faced with the same decisions I have faced in the past with things like DLSW for SNA, then ATM in general, and that is the learning curve of supporting it and troubling shooting it moving forward. MPLS looks to have the same problems with processing overhead that we have always had. Not to mention the same old problems with IP and packet size unpredictability.

Cisco has had a lot of years, talent and money to have found ways around these problems, still we wait. Juniper has done much in a short time with the aid of a Tony Li and others and still the problems remain.

I do not believe fundamentally that the existing architectures for packet forwarding scale to the huge numbers needed for peering one another in a hetergenuous ISP world. Basically, I am not encouraged by the results of this test of Big Nasty Routers (BNR's). Since only 2 vendors actually made a showing, considering the list of potentials, does this mean we must wait a while longer for the IP data flow to be whipped and beaten into a useful transport for all things?

I feel I have abandoned ATM at too early a date, only to say that IP will solve all my transport needs.

Perhaps IPNG? but still I have the inherent upredicatability of the packet size. Also, the out of order dilemna, a problem for switch vendors since we relegated layer 2 protocols to the dustbin.

Is it possible in any of the products tested to build flows around packet sizes rather than any other criteria and optimize que depths around that predictability? Just reaching, I know.

For me latency and jitter with no packet loss are key. I know of 1 box that handles these well, but it is never spoken of here.

Also, I would be interested in knowing when the 7770 becomes real in this lineup because of the success Newbridge had with the 36170 of the past. Of course I will believe the MF when it is sitting in a network actually doing all the fly things they say it will do.

Bottom Line: I am concerned that that arguments I see posted here are no different than they were five years ago when basic ethernet switching lifted off, with the IP slant.

I learned a valuable lesson then about packet loss or of lack thereof. In using the Extreme Test for overloading a fabric I found a few vendors that never dropped a packet but varied greatly in time to completion of the flow. I found a vendor that dropped packets but was always able to complete the flow in a set time. Thus retransmission seemed to me the lesser of two evils. After five years I am still arguing the merits of well timed delivery with loss and erratic timed delivery with no loss as design points in an IP network of any size.

So, how can a Cisco, Juniper, Sycamore, Torrent, Googleplex, Alcatel, Nortel, or other giggawad router simplify my IP problems? I'm at a loss.

Somebody tell me they have a BNR that does PFM, please.
haiyingdeng 12/4/2012 | 8:44:43 PM
re: Internet Core Router Test I do not know why they use the 40bytes frame size. David Newman said their test refer to RFC2544. In RFC2544 I found these about frame size. This RFC require to test at least five frame size from minimum to maximux, they only tested two size. the second is variable size, no max size. May they reveal more problem for both routers.

9. Frame sizes

All of the described tests SHOULD be performed at a number of frame
sizes. Specifically, the sizes SHOULD include the maximum and minimum
legitimate sizes for the protocol under test on the media under test
and enough sizes in between to be able to get a full characterization
of the DUT performance. Except where noted, at least five frame
sizes SHOULD be tested for each test condition.

Theoretically the minimum size UDP Echo request frame would consist
of an IP header (minimum length 20 octets), a UDP header (8 octets)
and whatever MAC level header is required by the media in use. The
theoretical maximum frame size is determined by the size of the
length field in the IP header. In almost all cases the actual
maximum and minimum sizes are determined by the limitations of the
media.




Bradner & McQuaid Informational [Page 5]

RFC 2544 Benchmarking Methodology March 1999


In theory it would be ideal to distribute the frame sizes in a way
that would evenly distribute the theoretical frame rates. These
recommendations incorporate this theory but specify frame sizes which
are easy to understand and remember. In addition, many of the same
frame sizes are specified on each of the media types to allow for
easy performance comparisons.

Note: The inclusion of an unrealistically small frame size on some of
the media types (i.e. with little or no space for data) is to help
characterize the per-frame processing overhead of the DUT.

9.1 Frame sizes to be used on Ethernet

64, 128, 256, 512, 1024, 1280, 1518

These sizes include the maximum and minimum frame sizes permitted by
the Ethernet standard and a selection of sizes between these extremes
with a finer granularity for the smaller frame sizes and higher frame
rates.

9.2 Frame sizes to be used on 4Mb and 16Mb token ring

54, 64, 128, 256, 1024, 1518, 2048, 4472

The frame size recommendations for token ring assume that there is no
RIF field in the frames of routed protocols. A RIF field would be
present in any direct source route bridge performance test. The
minimum size frame for UDP on token ring is 54 octets. The maximum
size of 4472 octets is recommended for 16Mb token ring instead of the
theoretical size of 17.9Kb because of the size limitations imposed by
many token ring interfaces. The reminder of the sizes are selected
to permit direct comparisons with other types of media. An IP (i.e.
not UDP) frame may be used in addition if a higher data rate is
desired, in which case the minimum frame size is 46 octets.

9.3 Frame sizes to be used on FDDI

54, 64, 128, 256, 1024, 1518, 2048, 4472

The minimum size frame for UDP on FDDI is 53 octets, the minimum size
of 54 is recommended to allow direct comparison to token ring
performance. The maximum size of 4472 is recommended instead of the
theoretical maximum size of 4500 octets to permit the same type of
comparison. An IP (i.e. not UDP) frame may be used in addition if a
higher data rate is desired, in which case the minimum frame size is
45 octets.





Bradner & McQuaid Informational [Page 6]

RFC 2544 Benchmarking Methodology March 1999


9.4 Frame sizes in the presence of disparate MTUs

When the interconnect DUT supports connecting links with disparate
MTUs, the frame sizes for the link with the *larger* MTU SHOULD be
used, up to the limit of the protocol being tested. If the
interconnect DUT does not support the fragmenting of frames in the
presence of MTU mismatch, the forwarding rate for that frame size
shall be reported as zero.

For example, the test of IP forwarding with a bridge or router that
joins FDDI and Ethernet should use the frame sizes of FDDI when going
from the FDDI to the Ethernet link. If the bridge does not support IP
fragmentation, the forwarding rate for those frames too large for
Ethernet should be reported as zero.

haiyingdeng 12/4/2012 | 8:44:44 PM
re: Internet Core Router Test
That is intresting!
Interesting. I assume you've read the test article. In which case you know who won, don't you?

It doesn't really matter how you slice the test results (and we sliced them in three different ways), Juniper won more tests.
--------------------------
I suggest you read the detailed test result,http://www.lightreading.com/do..., you know why cisco did not win. That is because Cisco did not attend the 38 filtering tests which are all won by Juniper. Without these 38 wins, Juniper wins only 15, which is less than Cisco's 18 wins. If you do a manual recount, you will see, Network Test repeatly counted the "Filtering ballot" 38 times in favor of Juniper. That is why Juniper wins.


As for the release, we're talking to Cisco about it. Apart from the obvious, it contains several factual inaccuracies (check for yourself).

For instance, Cisco says it is the "Only Vendor to Demonstrate 100%
Line Rate IP and MPLS Performance For 2.5Gbps OC-48 and 10Gbps OC-192
Throughput." In fact, the 12416's OC192 throughput in one of the two IP tests (40-byte
packets) was 52 percent, not 100 percent.
------------------------
40 bytes packets and Internet Mixed traffic, which traffic is releastic and reliable to ISP, of course Internet MIXed traffic. Anyway, Cisco router is layer 3 Internet Core router, not a ATM switch forwarding 53 bytes fixed length cell.
If you redesign the test case, test throughput at 40, 100.200.300......1500...MAX MTU for POS, who knows who wins. If there are same number of tests on packet size versus FILTERING.

So, Cisco issues a press release that says it won the test, but the *independent laboratory* that ran the test says it didn't (again, you have read the test results, right?).
-------------------------
I suggest you recount your results, you will see why?

Are you really surprised by this?
--------------------
Yes, I am suprised by your test results.

Steve




tink 12/4/2012 | 8:44:45 PM
re: Internet Core Router Test I think we all see that the Cisco PR machine chose their words carefully to give the most positive spin to the test results that they possibly could and thereby get some positive attendion.

It all seems like a silly game to many of us, but is it so very different from what the headline writers of LR do? Give an article some oomph with headlines like "Juniper Wins Monster Router Test"? Would "Juniper and Cisco Make Big Routers That Deliver Packets at Roughly Equal Rates and Have Differing Service Strengths" bring readers to your site?

Public relations spin is not a new invention, and certainly not a Cisco innovation. And all companies do it. Here's a quote from an old Juniper press release: "the M160 router is the first product to offer wire-rate OC-192c/STM-64 support." Your tests indicate that the M160 does not deliver wire rate. Was Juniper lying or spinning?

Thanks,
Tink
dnewman 12/4/2012 | 8:44:47 PM
re: Internet Core Router Test
Let's call a timeout on further comments on Cisco's press release pending Cisco's response.

Unfortunately the example I cited last night was only the tip of the iceberg; it turns out there are several other problems as well.

We have asked Cisco to correct the inaccurate and misleading statements in the release. We are awaiting their response.

In the meantime I don't think it's constructive to debate the merits (or lack thereof) of this problematic document.

Regards,
David Newman
Network Test
rasheed 12/4/2012 | 8:44:47 PM
re: Internet Core Router Test Steve Saunders, David Newman:

Please look at pages 4, 5 , 6. Do you see 100% associated with a Juniper result anywhere. The answer is no.

Cisco's results have 100% on Top of 7 out of 8 graph bars. So I think that as long as they qualify their claim and state which tests they are talking about, they have every right to say that they are the only ones who hit 100%. The fact that they have another result which is 52% does not take away the fact that they hit 100% and no one else did.

Pages 4 and 5 deal with OC48 and OC192 throughput, page 6 deals with MPLS.

Al-Rasheed Al-Ahmad (the voice of guidance)
rasheed 12/4/2012 | 8:44:49 PM
re: Internet Core Router Test On your internet, does the 50% of traffic which is made of packets less than 64 bytes come in all in one burst? And why are we talking about <64 bytes, I thaught that you magic number is 40 bytes.

The fact is if you design anything you design for the Imix not what LR wants you to design for.

What you do with your hands concerns you only.

Al-Rasheed Al-Ahmad (The one with a sence of humour unlike tekweeny)
Steve Saunders 12/4/2012 | 8:44:49 PM
re: Internet Core Router Test Interesting. I assume you've read the test article. In which case you know who won, don't you?

It doesn't really matter how you slice the test results (and we sliced them in three different ways), Juniper won more tests.

As for the release, we're talking to Cisco about it. Apart from the obvious, it contains several factual inaccuracies (check for yourself).

For instance, Cisco says it is the "Only Vendor to Demonstrate 100%
Line Rate IP and MPLS Performance For 2.5Gbps OC-48 and 10Gbps OC-192
Throughput." In fact, the 12416's OC192 throughput in one of the two IP tests (40-byte
packets) was 52 percent, not 100 percent.

So, Cisco issues a press release that says it won the test, but the *independent laboratory* that ran the test says it didn't (again, you have read the test results, right?).

Are you really surprised by this?

Steve



skeptic 12/4/2012 | 8:44:54 PM
re: Internet Core Router Test "As you certainly know, dropped packets or out-of-sequence packets will cause retransmissions (more the former than the latter)."

Re-transmissions yes, but this is also a good thing because at allows TCP to adjust its transmission rate. Multiple packet losses per stream is a bad thing. This is usually caused by buffer overflows, tail drop and the like.

---------------------------------------
False congestion signals due to ordering problems is not a good thing in any way. TCP has congestion control mechanisms, but its important that they are only triggered when there is true congestion and not packet-processing problems in a router.

Creating situations where the TCP congestion mechanism falsely detects congestion helps nothing. It creates problems for the customer (unexplained performance drops) and the provider (untraceable customer problems).




tekweeny 12/4/2012 | 8:44:56 PM
re: Internet Core Router Test >The thing to point out is that Juniper was
>behind cisco when the mix was introduced and in
>front when min size was used (pages 5 & 6). It
>begs the question, why? It must be an
>architectural feature. It would be nice if a
>Juniper expert can tell us why.

Well your obviously such a braniac so I am sure you know that on average 50% of the Internet traffic is less than 64 Bytes. Any vendor that can handle this size of packets effiecently has shown foresight into their design.

I forgot, you DL lots of graphic files so this does not apply to you. One hand on the mouse button and the other on your...

Ever wonder what might be happening as that quick 30 seconds goes by before you finish?!?
soothaandi 12/4/2012 | 8:44:59 PM
re: Internet Core Router Test I am eagerly waiting to see if you guys will end up adding this PR from Cisco about how the handily WON the same trials you guys had Network Test do for lightreading. Here's the link and the PR from Yahoo! And you guys said Juniper won! Who won according to Network Test? David Newman seems to be playing both sides, how "independent" is that?

http://biz.yahoo.com/bw/010313...

Tuesday March 13, 8:38 pm Eastern Time

Press Release

Cisco Reconfirms Leadership In Test of High-End Routers Only Vendor
to Demonstrate 100% Line Rate IP and MPLS Performance For 2.5Gbps
OC-48 and 10Gbps OC-192 Throughput

SAN JOSE, Calif.--(BUSINESS WIRE)--March 13, 2001--Cisco Systems, Inc. (Nasdaq:CSCO - news), the worldwide leader in networking for the Internet,
today said that the Cisco 12416 Internet router has outperformed all other networking vendors in nine out of twelve tests in the industry's first high-end routing
bake-off conducted by Network Test. The test against various industry competitors reviewed critical issues found in IP core network applications and
implementations while simulating seven times the Internet traffic service providers experience today.

Cisco earned first place finishes in the critical tests of:

IP forwarding for OC-48
IP forwarding for OC-192
MPLS forwarding for OC-48
MPLS forwarding for OC-192
Longest match lookups for OC-48
Longest match lookups for OC-192
IP Quality of service for OC-48
IP Quality of service for OC-192
IP convergence time for OC-48

Cisco won every category where the new 1-port OC-192 and 4-port OC-48 line cards and 10-Gbps switch fabric were tested. The tests also confirmed that Cisco
was the only vendor whose OC-192 systems did not cause packet misordering problems.

``In rigid performance tests Cisco proved it could process a torrent of data at line rate on both our OC48 and OC192 testbeds,'' said David Newman, president of
Network Test, an independent benchmarking and network design consultancy. ``In fact, the Cisco 12416 Internet Router turned in the highest single data rate
achieved in the entire test: more than 271 million packets per second.''

Newman continued, ``We were thrilled to have Cisco participate and validate our test design. Cisco's new 12416 Internet router made a very impressive debut in
our tests, showing it has the right stuff to compete in what looks to be a two-horse race.''

``It is gratifying to have the impartial judges at Network Test validate Cisco's performance leadership. Cisco was the only vendor to deliver line rate performance for
all of the IP and MPLS forwarding tests for both OC-48 and OC-192,'' said Rob Redford, senior director of marketing, IP POP systems business unit at Cisco.
``When you combine industry-leading performance with our other unique advantages, including guaranteed priority packet delivery, the highest capacity, and full line
rate performance even in a fully loaded system, our customers are well positioned to provide high bandwidth as well as profitable premium IP services such as VoIP
and video.''

Cisco offers its customers the industry's most comprehensive investment protection. Unlike competitors who offer no upgrade path, each Cisco 12400 Internet
router supports all current Cisco 12000 Internet router line cards, and OC-48 enabled Cisco 12016 Internet routers can be field upgraded to a Cisco 12416
Internet router that supports 10-Gbps per slot and a capacity of 375 million packets per second. Cisco also continues to deliver technology innovations such as Very
Short Reach Optics (VSR), offering OC-192 intra-POP connectivity at half of the cost of competitive offerings. Dynamic Packet Transport (DPT) is another
cost-effective tool that enables metro, regional, and intra-POP rings with high reliability.

The flagship Cisco 12000 series Internet routers are based on a unique distributed architecture that delivers the IP+Optical networking foundation and service
building blocks to accelerate the evolution of the Internet. The latest member of the family, the 12416 Internet router, was launched on January 31, 2001
(http://newsroom.cisco.com/dlls.... The Cisco 12400 family delivers the highest levels of scalability and performance available today and are the
only systems capable of guaranteeing high priority packet delivery, which is absolutely necessary to meet customers' increasing need for delivery of premium IP
services. This unique combination of features and capabilities make the Cisco 12400 family the premier platform for building true 10Gbps OC-192/STM-64
IP+Optical infrastructures.

The testing methodology and results of the Network Test can be found on the World Wide Web at http://www.networktest.com/tes....

About Cisco Systems

Cisco Systems, Inc. (Nasdaq:CSCO - news) is the worldwide leader in networking for the Internet. News and information are available at www.cisco.com.

Note to Editors: Cisco, Cisco Systems, and the Cisco Systems logo are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the U.S. and certain
other countries. All other trademarks mentioned in this document are the property of their respective owners.

Contact:

Cisco Systems
Martina Moscone, 408/853-6849
[email protected]
Dr. Dre 12/4/2012 | 8:45:00 PM
re: Internet Core Router Test Good Example Ashwath,
but I think the mathematics needs to be refined.

For a string of 100 packets, there
are C(100,2) = 4950 possible pairs.

Suppose one random pair of packets has been swapped.

Once the stream has passed through the second interface, the the original problem can only be fixed if the same pair is re-ordered. The probability of this occuring is of course, 1/4950. However, the probability of the problem getting worse (that is further re-ordering) is 1-(1/4950), or 4949/4950.

So, yes, the cumulative re-ordering does double, just like you said.

Thats all.

--Dr. Dre

haiyingdeng 12/4/2012 | 8:45:01 PM
re: Internet Core Router Test Manual recount was denied. Judge "Networktest" says.

gladysnight 12/4/2012 | 8:45:02 PM
re: Internet Core Router Test "Sure, I'd take that bet -- it would be one of the few investments with a positive return today. :( "

Here I'd just like to indicate that this is precisely the sort of thinking that has causes the most recent stock bubble.

Anything that pays a positive return "today" is by definition a bet and NOT an investment.

The term "day-trader" is itself misleading in that it could more accurately be rendered "stock-gambler".

lol
jsailor 12/4/2012 | 8:45:03 PM
re: Internet Core Router Test
"As you certainly know, dropped packets or out-of-sequence packets will cause retransmissions (more the former than the latter)."

Re-transmissions yes, but this is also a good thing because at allows TCP to adjust its transmission rate. Multiple packet losses per stream is a bad thing. This is usually caused by buffer overflows, tail drop and the like. As someone else mentioned (W)RED usually handles this gracefully in core routers. It would be interesting to know what the buffers looked like when the routers started discarding packets. Also someone else mentioned measuring goodput, which is really what's important.

ALso, what's important for QoS tests isn't so much how well each router holds to percentages allocated for traffic classes, but how well the applications within each class perform. That is if a high priority, low latency class is defined, does the router handle that. Also what is the goodput for the rate limited classes. Are applications capable of using the entire bandwidth defined for the class. My own tests have indicated otherwise as have many other tests for lower-end layer 3/4 switches.


drag 12/4/2012 | 8:45:03 PM
re: Internet Core Router Test I noticed that in the latency tests the latency result was not correlated with the linerate being achieved.

Were there any latency tests done with the 12416 at the same linerate the M160 was achieving on OC48 and OC192?

I think the closer you are to linerate (ie. the ingress rate = egress line rate) the more buffers will be filled by slight clocking errors, etc. These buffers have practically no way of recovering because you are hammering them at linerate.

Is my reasoning correct here??

-d
oc48 12/4/2012 | 8:45:04 PM
re: Internet Core Router Test Does LightReading have any plans of performing tests for the MAN switches/Routers?
ash3 12/4/2012 | 8:45:04 PM
re: Internet Core Router Test I had a comment to make about the cumulative / non-cumulative nature of packet re-ordering.

When a small quantity of packets has been re-ordered, an additional router will cumulate to the quantity re-ordered, I believe.

Look at it this way -

Let us take 100 packets in order to the first router. Let us say that it re-orders 1 pair randomly; the result will have 1 pair of packets swapped.

When the second router sees this stream, the probability that is re-orders exactly this pair is 1 in 99 (there are 99 "pairs"). In 2 other cases it will retain the same amount of mis-ordering (with increased distance of the out-of-order packet => reduced probability that it can be re-ordered back),

...BUT in 96 of 99 cases, it increases the re-ordering.

So the cumulative re-ordering will be nearly double.

Of course, Juniper will be right if the packets are largely re-ordered already, but in that case the network would be total disaster, so the point is moot. Either they should state that their routers do re-order slightly, and the effect is cumulative, or that the routers re-order to such an extent that the result is no longer cumulative.

Ashwath
drag 12/4/2012 | 8:45:04 PM
re: Internet Core Router Test take a vase smash it on the sidewalk, pick up the pieces and smash them on the sidewalk again -> there is a chance that they will fall together in the exact shape of the vase you originally started with.

alternatively:

crash your car, instead of taking it to the shop go ahead and crash it again, you never know the second time actually fix all the damage done the first time.

Yes people, it is this simple...

haiyingdeng 12/4/2012 | 8:45:07 PM
re: Internet Core Router Test Please read what they are talking at the end of this page:
http://www.lightreading.com/te...

Reading what they speak, I am not sure what level these so called CTO, network architector are at? Can Lightreading find somebody from MCI Worldcom, UUNET, Sprint, Earthlink or some professors from USC comment on these.

Bill St. Arnaud:
Another plus point is that JunOS is the same OS across all Juniper products, St. Arnaud says. G«£Virtually every Cisco product has a different OS.
The MPLS [multiprotocol label switching] throughput results confirmed our suspicions that MPLS does not buy you much except a big management headache. True, the throughput is higher, but not significantly higher than IP forwarding.
St. Arnaud cautions against reading too much into raw throughput figures, noting that core routers are rarely loaded to more than 25 to 50 percent of their theoretical capacity. If they were, traffic would G«£slosh aboutG«• causing problems at edge devices, he notes. G«£Also many ISPs implement, RED [random early discard] as standard practice so packet loss is intentional and not a big deal.

skeptic 12/4/2012 | 8:45:13 PM
re: Internet Core Router Test To conclude, this highlights why min-size packet processing is indeed a common-case test and why min-size performance is important.
------------------------------------------------

I agree with you that min-size testing is important, however its also important to look at the entire low end of the curve. There are vendors optimizing their systems to perform line rate with 40-byte packets. And in some equipment they are creating a step-function downward in performance a small increment larger than 40 bytes.

If you need an example why this matters, think about certain MPLS traffic (VPN's in particular) that can push up the small packet size in all sorts of ways. There are people thinking about situations with four MPLS labels prepended to these small packets.

The (silly) obsession with 40-byte packets should end. They are important, but they should not be dictating architecture of routers (as they are in certain cases at present).



MHA 12/4/2012 | 8:45:17 PM
re: Internet Core Router Test I don't see any test been done on OIR and how the
routers behaved. For core routers this is a very
important feature. Wonder why they did not make this as part of the test.

Also, I did not see any performance results with a fully loaded router since most deployment will do that.
rasheed 12/4/2012 | 8:45:27 PM
re: Internet Core Router Test Hey! Al-Rasheed made a mistake.

I want to thank you for the details, I enjoyed reading it.

I was wrong to say that 40 byte packets are not used. But the mix is a lot more important than min size packets.

You are right to say that in your example most packets from client to server are min size, but to a router it does not change the mix. There may be as many servers on one side of this router as on the other. So, the mix is the mix, it is a mix. Now, when downloading a page full of graphics, min size packets may be 1% of the whole transfer (I am guessing here). So a good-use test is the mix with proportionate amount of min size.

The thing to point out is that Juniper was behind cisco when the mix was introduced and in front when min size was used (pages 5 & 6). It begs the question, why? It must be an architectural feature. It would be nice if a Juniper expert can tell us why.


Al-Rasheed Al-Ahmad ( the pointer of the way)
briand 12/4/2012 | 8:45:28 PM
re: Internet Core Router Test I'd be interested in the availability of those raw results, if they still exist. Especially for the step tests, for Cisco and Juniper.

As to presentation methods, you might want to look into Quicktime-VR, which puts 3d-model into a 2-d window, where you can "grab" the model and spin it in two dimensions to get different views of a modeled 3d object. Very cool, and an available plug-in viewer for most browsers, for free. Don't know about authoring tools, however...

Are there any further results you can give, even in this forum, for any of the tests? Specifically, the latency distribution (median, if nothing else, would help a lot).

Brian
BigMikeJ 12/4/2012 | 8:45:30 PM
re: Internet Core Router Test
Actually, let me clarify the "one ack to multiple data" thing:

When a TCP session starts up, all the packtes are min-size (40-byte): SYN, SYN-ACK, ACK.

Consider that most internet traffic is web access. You click your mouse and you open a connection to a web page:

Client Web Server
SYN ------>
<------ SYN+ACK
ACK ------>

This three-way handshake forms the connection. All threee packets have been min-size (40 byte).

Now your client specifies the page to be downloaded:

Client Web Server
DATA(url)---->
<----- ACK
<----- DATA (page data)
ACK ----->
<----- DATA
-- mix of ACKs and DATA --

And finally a close of connection

Client Web Server
<----- DATA+FIN
FIN+ACK ----->
<----- ACK

Things to note:

1. Besides the page URL, all the packtes from the client to the server were min-size TCP control indications.
2. Until the TCP window opens, every data packet from the server will require an ACK.
3. Smaller pages may only contain a couple of data packets - so most of the server packets end up being min size too.
4. Smaller web pages, with only a few DATA packets mean that the TCP window may never open very far.

To conclude, this highlights why min-size packet processing is indeed a common-case test and why min-size performance is important.
BigMikeJ 12/4/2012 | 8:45:32 PM
re: Internet Core Router Test
A 40 byte packet is perfectly valid when using Packet Over Sonet (PPP encapsulated, no Layer 2).

TCP Ack packets are a good portion of internet traffic. Remember you get one ack packet for each handful of TCP data packets.

IP header: 20 bytes
TCP ack: 20 bytes

Correctly doing the math: priceless.
rasheed 12/4/2012 | 8:45:32 PM
re: Internet Core Router Test Hey Russ; you are killing me here.

Who the heck is going to retransmit a 40 byte packet, it does not even have a layer 4 (hence a TCP) header.

Yeah you sound knowledgable, but think clearly, 40 byte packets are only used to make somebody like Juniper look good.

your 52% is at 40 byte packets, nobody in interested in using this for real much less to retransmit them.

Al-Rasheed Al-Ahmad (The uncoverer of all that does not know)
rasheed 12/4/2012 | 8:45:33 PM
re: Internet Core Router Test When you decide who the winner is before the contest is begun, you grab anything you can to justify your action.

This is like my wife, she first decides that I am guilty, as to the what for, that is always TBD.

Al-Rasheed Al-Ahmad (The one with the guiding answers.)
rasheed 12/4/2012 | 8:45:35 PM
re: Internet Core Router Test At least it shows that Juniper did not bother to write a command parser but picked up a downloaded freeware and modified it (maybe).

So what are they selling, snake-oil in a new can.

Al-Rasheed Al-Ahamad (The wise guide and clarifier of all that is obscured)
rasheed 12/4/2012 | 8:45:36 PM
re: Internet Core Router Test hey russ, ihave a cure to your ignorance.
1- 40 byte packets do not hold any data, so if the isp waisted them it would not be a bad thing.

2- at larger size packets (imix) the ISP will take of full line rate. The only documentef waist here is juniper's 90% (89%) of full oc192 line rate. Do the math on this.
skeptic 12/4/2012 | 8:45:39 PM
re: Internet Core Router Test What is the point to have the BGP table size to evaluate the router performance or capacity.

If I have a SUN workstation with 4GB HD and 300MHz CPU, 100Base-T NIC, running (what's the "BGP server" name at MAE or NAP?), use this computer based router, I can have at least twice the size of BGP table than any backbone routers.
------------------------------------------------

This gets to what you mean by "BGP table".
Light reading and their testers are concerned
with how many unique routes can be shoved into
the forwarding software. The memory issues in putting routes into forwarding are very different than they are for storing non-unique BGP routes.

Most of the vendors cound store many more routes than the light reading numbers of they were putting in non-unique BGP routers.

But the test was seemingly constructed to show that Juniper can abuse virtual memory and calculate incorrect forwarding tables to reach
an utterly meaningless benchmark of 2.4 million prefixes.

Remember, the Juniper may be paging like crazy, it may not be adding all the routes it knows about to the FIB, performance and stability may be out the window. But it can store more useless BGP routes on disk than any of the vendors.










haiyingdeng 12/4/2012 | 8:45:41 PM
re: Internet Core Router Test What is the point to have the BGP table size to evaluate the router performance or capacity.

If I have a SUN workstation with 4GB HD and 300MHz CPU, 100Base-T NIC, running (what's the "BGP server" name at MAE or NAP?), use this computer based router, I can have at least twice the size of BGP table than any backbone routers.

If you do not count speed.
f451 12/4/2012 | 8:45:53 PM
re: Internet Core Router Test It is too bad that none of the Routers tested are True Terabit Routers. None of the routers tested can pass a Terabit Plus of traffic in a single chassis. What a waste of a test!!! Or at least a poor name choice!
tink 12/4/2012 | 8:45:56 PM
re: Internet Core Router Test Russ,

You make a very good point when you state, "As you certainly know, dropped packets or out-of-sequence packets will cause retransmissions (more the former than the latter)."

However, the retransmission rate will still not be of the order of 48% of all packets, so your original calculation still fails to represent the "useful" bandwidth a carrier might have on a 12416. Without knowing how many packets are dropped at 52% and each rate above, such a calculation is impossible. The article does tell us that "both Cisco and Juniper can forward 40-byte packets at rates well above 99 percent, albeit with some packet loss." This is helpful in estimating the number of retransmissions.

Cheers,
Tink
drag 12/4/2012 | 8:45:58 PM
re: Internet Core Router Test and you mine.

i bet you can't leave it at this... i wouldn't. Go on.
bubba 12/4/2012 | 8:45:59 PM
re: Internet Core Router Test Drag,

Again, you have proven my point!

-b
dnewman 12/4/2012 | 8:45:59 PM
re: Internet Core Router Test Hi Brian,

Thanks very much for your feedback. These are all excellent suggestions and we'll try to address as many of these as we can the next time around.

By the way, some of the information you're looking for actually does exist, deep down in the raw data. The Smartbits kept track of every packet inside the 792 flows (for the OC48 testbed) or 1,728 flows (for the OC192 testbed), and gives us min, mean, max, and jitter on each flow. Also, we ran dozens of dozens of step tests with the CSCO, FDRY, and JNPR equipment, so potentially we could have done the kind of "hockey stick" curve you're looking for. Unfortunately, time and space constraints kept us from developing the data as fully as we could.

There's also the perennial presentation issue of how to represent 3-D forms (or N-dimensional, for some of the data we'd like to present) in a 2-D medium like the Web. These are presentation issues, and we'll work to give a more complete picture in future tests.

For the record, the number of peers was 12 in the OC48 testbed and 48 on the OC192 testbed -- one peering session per edge interface.

Thanks again for your excellent suggestions.

Regards,
David Newman
Network Test
russ4br 12/4/2012 | 8:46:00 PM
re: Internet Core Router Test tink wrote:

"You are confusing forwarding-rate with throughput- rate. The throughput rate is the rate at which any packet is dropped, the "no-drop rate," if you will.

As stated in the article, "This means the 12416 will forward traffic at up to 52 percent of line rate without loss. It does not mean the device will drop 48 percent of a load offered at wire speed.""

Tink,

As you certainly know, dropped packets or out-of-sequence packets will cause retransmissions (more the former than the latter). And CSCO has made it very clear that retransmissions are BAD for us - doesn-¶t matter the reason.

Thus my simple reasoning is that the GSR 12416, in keeping with the CSCO motto of a "retransmission free network", should avoid the conditions that cause retransmission - in this case working above 52% load on their OC-192 interfaces (40-byte packets). Going above this level will put them in the "retransmission land", heavens forbid!

- russ
briand 12/4/2012 | 8:46:01 PM
re: Internet Core Router Test While interesting, the results of this article are not much use to anyone giving serious consideration to the vendors in question.

The reasons are complex, and have to do with the "6 blind men versus an elephant" problem. (Ask me offline if you need that explained.)

The testing done on different equipment with different (between vendors) packet rates or percentages of linerate, with no separate testing at differing rates *per vendor*, means there is no ability to correlate performance in real-world enviroments. The "imix" is useful, but not sufficient.

Someone interested in fully *investigating* a given box, and especially for side-by-side(-by-side^n) comparisons, would, for each category of testing, do the following kinds of comparisons:
- Determine percentile values (eg median, or 50%ile, plus 10%ile bands, plus 95%ile)
- Even in a situation where one vendor has a box which only operates at max X%, evaluate all boxes which are able to operate at given X, at X. (E.g., in addition to full linerate, Cisco box should have been tested at Juniper max of 99.8%, and those results reported as well.)
- All configuration details, where vendors munged their interfaces, absolutely *must* be presented. Where these are adjusted, performance at default values, and at sensible range bands, should be done (yes this takes a lot of time and effort, but illustrates the "knees" of performance curves. This will create a 3-d manifold, of performance values versus latency versus parameter, such as "percent linerate lossless, vs latency, vs queue depth configured".)

Especially on tests where max>>avg>>min, the rangebands are critical. In these instances, it is expected that avg>>median, and median is the really important factor. Average can be too severely skewed by a few bad samples. (See for yourself - take 1000 samples, whose values are 999 x 10 microseconds, 1 x 10 milliseconds. Median is 10 microseconds, mean is 5 milliseconds - a discrepency of 500 *times* (ie 50,000 percent).)

Ditto for other characteristics - the BGP table size I would want to vary both total number of IP routes used, and number of peers - I want to know if the problem lies in BGP table size, or in BGP syncronization. Again, full configs for the BGP portion of the router config *must* be supplied to have any meaning. Also worth testing would be BGP route reflection (a *must* for core routers), and BGP confederations (almost a must).

I would be very interested in seeing these test repeated with the criteria above. Any testing done by potential customers is generally under NDA, and while I can get those results myself by doing the same testing, I can't share them the way you can.

Comments?
drag 12/4/2012 | 8:46:01 PM
re: Internet Core Router Test "You sound like a sore looser!"

i think the word is loser - look in the mirror it's written just below your hairline, then look at your IQ it's one of the numbers on your analog watch. shouldn't you be writing marketing fud anyway??

thanks for the wide-open opportunity, remember you started it ;^)

bubba 12/4/2012 | 8:46:04 PM
re: Internet Core Router Test Drag,

You sound like a sore looser!

-b
guru 12/4/2012 | 8:46:04 PM
re: Internet Core Router Test i say all of this with great respect... i have followed the work of yourself and Mr. Mandeville and learned a lot from it. Data Comm was the best magazine, PERIOD! I miss it...

regards,
todd craw
Maple Networks System Test
guru 12/4/2012 | 8:46:05 PM
re: Internet Core Router Test Dave,

then it is incorrect as i have easily stuffed 600k BGP routes into a Netiron with only 256 MB and not even the 512MB they support now. This was over 8 months ago. They had no hard-coded limit on the BGP table but did have one on the IP route table.

You also contradict yourself in saying that FDRY only could handle 256k routes. You are saying that you did this to the BGP4 tables of the competitors but now you are admitting that you fed 600k routes through BGP4 to all of them. Well how did CSCO and FDRY handle those 600k if they maxed out at 256k and 400k for BGP4?

I know that this is incorrect as i have worked with all routers. GSR with 256MB and FDRY with 256MB will handle more BGP4 routes than that.

You also mention the JNPR FIB. The FIB is a data-structure of interfaces, MAC addrs, IP next-hops, etc. that you make forwarding decisions on. This is the L2 and IP route table not the BGP4 table. Forwarding decisions are NOT made based on the BGP4 table (also known as the RIB,created by adj-RIB in and adj-RIB out for each peer). BGP4 table is used to populate IP route table and/or FIB.

please clarify...
drag 12/4/2012 | 8:46:05 PM
re: Internet Core Router Test Great tests, very informative (although by reading the usual LR fanfare headlines and text you would have thought Juniper won by a rather large margin, reality is it was much closer to a tie. Maybe some other tests might have swayed it Cisco's way??). One things for sure, your numbers are being disseminated into untold amounts of marketing FUD by both sides as we speak.

Anyway for what it's worth, I would like to see some tests on redundancy, and some tests on the operation aspects of running a network. Like adding & removing linecards, control plane redundancy, power, fabric (if you have one), forwarding engines etc. I don't know how much Internet instability is caused by box or link failures but I'm sure redundancy is important to people.

I'm sure you guys are going to do an article highlighting the importance of mis-ordering packets as well. I note that you did mention it briefly but I think if you took this into account for all the tests you did maybe some providers may outright fail Juniper on this alone. Making all the tests irrelivant. It would be interesting to see how many of the sources you quote saying packet ordering is not a problem are stockholders. As you can probably tell along with Path MTU discovery, packet ordering is something I view with disdain.

The 2.4 million routes thing is unfortunate, because as we all know if it were actually a useful number, anyone can make any hard disk based system hold much more than this (maybe I could even do it on my Linux box?? got a 30G harddisk why not??) - start the thing swapping and boom!! Congrats on making the max BGP table size useless as a metric.

Overall great work and I look forward to the next (more service/operations focussed) tests once Juniper and Cisco have had time to ponder their weaknesses and release some new products.

d


tink 12/4/2012 | 8:46:06 PM
re: Internet Core Router Test David,

You asked, "Why is median a better indicator of latency variation than mean?"

I'm sorry, I must not have been clear. I meant that the median would be useful in addition to the mean. The mean is obviously critical and the two piece of information together help in understanding the variation among latencies. In particular, whether a lot of traffic is getting bogged down, or only a small fraction. This is information that the min and max simply do not tell us.

Thanks,
Tink
dnewman 12/4/2012 | 8:46:06 PM
re: Internet Core Router Test Hi Guru,

Correct about confusion between BGP and IP routing tables in that comment about flapping -- but not correct about the BGP capacity test.

That latter event really did look at BGP tables, period. It did not evaluate IP routing table contents.

Regards,
David Newman
Network Test
guru 12/4/2012 | 8:46:06 PM
re: Internet Core Router Test as i read i think that you are confusing the IP route table with the BGP4 table. FDRY allows more than 256,000 BGP4 routes but only 256k IP routes. I have personally put over 600k routes into FDRY BGP tables using real internet routes of which 80k were placed into the IP routing table.

Thus i believe you actually put 256k, 400k and 700k - 2.4M IP routes into the FDRY, CSCO and JNPR respectively which is not the same as putting 1M - 4M BGP4 routes into them.

Ip route tables only need to handle 100k today and maybe 200k - 300k will be fine for a few years with CIDR, etc.

BGP4 route table is different story. They need to be able to handle 600k - 2M right now. You did not test this properly from what i can read - that is you overloaded and stressed the IP route table to unrealistic levels and you did not stress the BGP4 route table capacity nearly enough - 600k is 6 internet peers today.




please respond...
dnewman 12/4/2012 | 8:46:06 PM
re: Internet Core Router Test Hi Guru,

You are correct -- the text should read "routing table" and not "BGP table."

AFAIK the BGP table does contain every route learned from every peer, as you suggest. The routing table is the collection of entries from tables (from BGP and from IGP(s) as well) the device uses to make path selection decisions.

Thanks for pointing this out.

Regards,
David Newman
Network Test


guru 12/4/2012 | 8:46:07 PM
re: Internet Core Router Test "(What Black Arts allowed us to squeeze 600,000 entries into boxes with less BGP capacity than that? We didnG«÷t. BGP tables, at least the ones we tested, count only the number of active entries.)"

i do not believe this is accurate. in order to failover from a primary BGP4 path to a secondary or tertiary path those paths must be loaded into the BGP4 table. A BGP router does not request tables from peers when a flap occurs. They download all routes into a BGP4 table from each external peer then find the best routes and load those into a separate IP route table. When a flap occurs the primary BGP4 routes are removed from both the BGP4 and IP route tables. A computation is done to find the best route from the remaining BGP4 entries if existing and it is added to the IP route table. If you do a sho ip bgp or whatever on any internet router you will see multiple BGP4 routes to a single destination if multiple paths exist and one is marked best.

Therefore a BGP4 table by default contains all routes learned. As a matter of fact it would be difficult to have any internet router with more than 80,000 BGP4 routes if this were the case and all of the ones i have worked on you have to add 80,000 for each peer.

Please clarify?
dnewman 12/4/2012 | 8:46:07 PM
re: Internet Core Router Test
Why is median a better indicator of latency variation than mean?

Not disagreeing with you, just curious as to your reasoning.

Regards,
David Newman
Network Test
tink 12/4/2012 | 8:46:08 PM
re: Internet Core Router Test The most useful information would be a plot of the DISTRIBUTION of latencies. Failing that, the standard deviation (quantifying the "width" of the distribution) would be useful. Medians would also help the reader to understand how many packets were slow compared to how many were fast. Min and max are practically meaningless numbers.

Tink
dnewman 12/4/2012 | 8:46:09 PM
re: Internet Core Router Test In measuring latency of IP packets, we follow long-established precedent of measuring at, and only at, the throughput level.

As has already been discussed here, it is inaccurate to label as "latency" any measurement made when packet loss exists (with or without an overload). This is not really latency but instead a measurement of queue depth.

The only other time I've heard requests for latency measurements below the throughput level is from makers of not-very-efficient telecom gear. Their customers tend to overprovision their networks to compensate for the packet loss of these boxes. In fact, nearly any device will give you pretty good latency when it's not taxed.

In the IP world, the spec is very clear: Measure latency at the throughput level and only at that level.

I'm currently involved in an IETF effort to define a new metric for delay. This metric, developed under the auspices of the IETF's benchmarking working group, is much more flexible than latency, as it allows for measurements under congested as well as uncongested conditions.

Regards,
David Newman
Network Test
tink 12/4/2012 | 8:46:10 PM
re: Internet Core Router Test Russ,

Regarding your comment:
"2. As for the GSR 12416, you really mean
that it can sustain the 52% 40-byte packet
forwarding in all ports in a
fully-loaded-system (15 ports)?
that's impressive ... ;^)
How much bandwidth will the "real ISP" using "fully-loaded" GSRs be throwing away? Let's
do the math: 15*0.48*10G= 72 Gigabits/s!!! (per box)!!"

You are confusing forwarding-rate with throughput- rate. The throughput rate is the rate at which any packet is dropped, the "no-drop rate," if you will.

As stated in the article, "This means the 12416 will forward traffic at up to 52 percent of line rate without loss. It does not mean the device will drop 48 percent of a load offered at wire speed." The article further clarifies with, "In fact, both Cisco and Juniper can forward 40-byte packets at rates well above 99 percent,
albeit with some packet loss."

Reading is fundamental,
Tink
ChrisChase 12/4/2012 | 8:46:11 PM
re: Internet Core Router Test The reordering observations would be a problem with MPLS when carrying traffic other than IP. There is a movement to use MPLS for layer 2 VPNs, i.e., FR/ATM/Ethernet/CE PVC-like services. When using MPLS for this application, reordering within a connection is not acceptable.

I did not find the latency tests very useful since they tested at the full load (access = uplink) which is a boundary condition between congestion and under-loaded. I would not want to make architecture decisions based on that domain. I would chose both a completely underloaded scenario (e.g., 90% of max capacity) to test switch latency and a overloaded scenario to test latency as a function of configured buffer size.
dnewman 12/4/2012 | 8:46:15 PM
re: Internet Core Router Test
Hello David (if I may),

You are correct in noting that with OC192 interfaces, Juniper's sequence threshold (to coin a term) is lower than its drop threshold.

You seem to suggest that the sequence theshold *should be* reported as the forwarding rate. It's not, and it should not be. RFCs 2285 and 2689 are quite clear on this.

Whether a potential customer should regard the sequence threshold as the top end I leave for the readers of this article.

By packet interval I presume you mean the interpacket gap. This was a function of the offered load. The IPG formula for Sonet is not something I can render in ASCII. I can say that it was 8 bits (one flag byte) at line rate, and more than that at all lower rates.

Thanks for your kind comments regarding the test. It is my hope that this test and others will encourage all vendors to boost performance of their devices.

Regards,
David Newman
Network Test
skeptic 12/4/2012 | 8:46:17 PM
re: Internet Core Router Test I'm glad that you understand the issues. My
problem isn't in your overall report (which details most of the issues), but in Light Reading's presentation of the overall results.

Either you (or Light reading) has called out
the 2.4 million BGP prefix number as (to quote)

"the most impressive single data point of the entire project".

I understand that someone wanted to throw out a
"big number" to impress people, but please
understand that in throwing out that big
and meaningless number, you are giving people
the (wrong) impression about what a Juniper
can do.

Juniper can do 700k routes in the FIB. Thats
great. But thats very different than creating
the impression that juniper can do 2.4 million
which ignorant people who don't read the entire
report are going to get.





Gibson2001 12/4/2012 | 8:46:17 PM
re: Internet Core Router Test Mr. Newman,

I am sorry about +99% credit for Juniper OC-192 which should be +90%, but this number still higher than 73%. For the application, packet out-of-order is worse than packet drop no matter "throughput" or "forwarding rate".

What is packet interval for your "1 packet burst" flow? I believe it will be more convinced if you could provide the smartbits scripts and router config files for your test scenario.

Anyway, you did very good job in pushing all the verndors to make their boxes better.

I need to get of this because it is quite late, see you in your "Terabits router test".

Thanks,

David
skeptic 12/4/2012 | 8:46:18 PM
re: Internet Core Router Test
I really have to seriously ask you if you
have ever been on a Juniper or configured a
juniper for routing.

A flip answer like this to a seriously raised
question hurts the credibility of your lab
and this whole test.

Did you actually ever get beyond the initial
login prompt on the Juniper?

Anyone who has actually used a Juniper would know that the routing configuration CLI looks nothing
like UNIX even though (as I said) UNIX is running
on the platform.



--------------------------------

"Your description of Juniper's configuration
and management interfaces as being a strong
resemblance to "unix" is wrong."

"Its well-known that JunOS is based on FreeBSD
unix."

Well, which is it? :-)

dnewman 12/4/2012 | 8:46:19 PM
re: Internet Core Router Test
skeptic,

I share your concerns. The 2.4 million number is interesting only as an academic excercise in how hard we could pound the box's control plane.

The number has little, if any, relevance for production networks.

This is why the article clearly notes the lower points of 1.4 million (where swapping begins) and ~700,000 (the FIB capacity) as more relevant points for operators of production networks.

Even with the lowest of these numbers, the M160 still has nearly twice the capacity of its nearest competitor.

Regards,
David Newman
Network Test
dnewman 12/4/2012 | 8:46:19 PM
re: Internet Core Router Test
Of course any router OS is highly optimized and does things that any general-purpose OS does not.

I actually can do some stupid Unix tricks in JunOS. In particular I loved being able to filter on stdout using grep. And if I drop down out of the shell, I *can* do basic stuff like ls.

My point was that JunOS is much, much closer to Unix than most other router OSs I've used. I should note here that the OS of Charlotte's Netorks Aranea-1 is also very BSD-like. The other router OSs I've used generally have a more proprietary look and feel.

Regards,
David Newman
Network Test
Belzebutt 12/4/2012 | 8:46:20 PM
re: Internet Core Router Test "Well, which is it? :-)"


Does any of the following look like Unix to you:

[email protected]> show isis route
IS-IS routing table Current version: L1: 376 L2: 406
Prefix L Version Metric Type Interface Via
10.0.19.0/30 1 376 20 int so-6/2/3.0 M160
so-6/2/2.0 M20a
10.6.2.0/24 1 376 20 int so-6/2/2.0 M20a
10.6.3.0/24 1 376 20 int so-6/2/2.0 M20a
192.168.3.1/32 2 406 10 int so-6/2/2.0 M20a
192.168.6.1/32 1 376 10 int so-6/2/3.0 M160


Or:

interfaces {
fe-0/0/1 {
description "La La La";
unit 0 {
family inet {
address 1.1.1.1/24;
}
}
}
ge-0/2/0 {
unit 0 {
family inet {
no-redirects;
address 2.2.2.2/24;
}
family mpls;
}
}
at-6/0/0 {
description "Sha na na";
encapsulation atm-pvc;
atm-options {
vpi 0 maximum-vcs 500;
ilmi;
}
unit 0 {
encapsulation atm-snap;
point-to-point;
vci 0.60;
inverse-arp;
family inet {
address 10.10.10.10/24;
}
}
}
}
routing-options {
static {
route 10.0.0.0/8 {
next-hop 4.4.4.4/24;
retain;
no-readvertise;
}
}
}


Command to configure an IP address:

set interface fe-0/0/0 unit 0 family inet address 2.2.2.2/24

Show commands on Juniper are similar to Cisco.

Now let's see a Unix command:

ls -l
ifconfig lan0
...


Sorry, but any resemblance to Unix is purely coincidental :)

Not everything that has brackets around it looks like Unix. Sure you can get into the BSD on a Juniper box, but you're not supposed to and there's no need to.
skeptic 12/4/2012 | 8:46:20 PM
re: Internet Core Router Test
Perhaps you can explain why you make such a
big event out of the juniper 2.4 million number
when:

- The Juniper is swapping pages to disk
at that level. If the juniper ever begins to
swap, its performance and routing stability are
going to drop rapidly to very poor levels.

I just don't understand why you didn't call
attention to the fact that even though the juniper can abuse its virtual memory to get a big 2.4M benchmark in the test, the router is reduced
to a feeble state overall at 25-50% of that load.


- Given that the Juniper can only hold 700k
routes in forwarding, all of the higher numbers
you are presenting are largely meaningless.
To a reasonable operator, if the Juniper can't
hold more than 700k on the line card, that is
the limit that matters.

What use is it for the control plain to hold
2.4 million routes if only 700k of them can
actually be used. Thats caching and most operators know that route table caching was (and is ) a bad idea.
dnewman 12/4/2012 | 8:46:21 PM
re: Internet Core Router Test
"Your description of Juniper's configuration
and management interfaces as being a strong
resemblance to "unix" is wrong."

"Its well-known that JunOS is based on FreeBSD
unix."

Well, which is it? :-)

Regards,
David Newman
Network Test

russ4br 12/4/2012 | 8:46:22 PM
re: Internet Core Router Test "As far as I know, foundry and Cisco present decent performance with maximum number high speed ports. GSR will not suprise you when you put more high speed ports into the system because of the distributed architecture."

1. Foundry underperformed and dropped packets w/ very low loads - using only the limited set of interfaces required for the test. Why add more pain?

2. As for the GSR 12416, you really mean that it can sustain the 52% 40-byte packet forwarding in all ports in a fully-loaded-system (15 ports)? that's impressive ... ;^)

How much bandwidth will the "real ISP" using "fully-loaded" GSRs be throwing away? Let's do the math: 15*0.48*10G= 72 Gigabits/s!!! (per box)!!

Some ISPs don't even have that much bandwidth in their whole network - never mind to waste it like this ...

- russ

dnewman 12/4/2012 | 8:46:22 PM
re: Internet Core Router Test
Mr. Zhou,

We may be talking at cross-purposes here. In places you suggest we're saying things in the article that we did not say.

For example, the M160 did *not* achieve throughput of > 99 percent in the OC192 tests, where we recorded reordering at load levels of 73 percent or more. There is a big difference between throughput (RFC 1242) and forwarding rate (RFC 2285), and you may be confusing the two.

Also, the Smartbits does track packet sequencing. This is not a point of dispute, and I'm not sure why you're trying to make it one. We used a burst size of 1 for all tests, by the way.

Very good of you to note the potential for the Smartflow API's sequence counter rolling over its 32-bit space. We thought about this issue quite a bit.

This problem actually came up on the OC192 test bed, since packet counts for each test run were something like 8 billion (more than the 4 billion a 32-bit counter allows).

The Smartbits hardware actually uses 64-bit counters, and we changed the custom scripts to do the same. For sequence numbers we relied on 32-bit counters, but we tracked each router separately and these numbers never came anywhere close to breaking the 32-bit barrier.

Regarding filtering, we *did* ask the boxes to filter on TOS -- that was the criterion the routers used to make filtering decisions.

As for filtering at the edge and not the core, that point is discussed in the article and in the carrier feedback from Mr. St. Arnaud of Canarie.

Regards,
David Newman
Network Test
skeptic 12/4/2012 | 8:46:24 PM
re: Internet Core Router Test
Sirs,

Once again you display an amazing lack of knowledge about the Juniper product.

Your description of Juniper's configuration
and management interfaces as being a strong
resemblance to "unix" is wrong.

The routing configuration (BGP, OSPF, etc) is
based on juniper-unique configuration language that has nothing to do with UNIX (or anything
else except perhaps gated).

Its well-known that JunOS is based on FreeBSD
unix. But the configuration of a juniper system
as a router is done in a Juniper-unique command
line interface that looks nothing like unix.


This of course raises the question of who in
your organization even bothered to log on to
any of the routers. As often as not, LR seems
to be recycling the marketing spin of big
companies and showing its own ignorance of
IP routers.




Gibson2001 12/4/2012 | 8:46:26 PM
re: Internet Core Router Test Mr. Newman,

1.It is your test, you could base any RFCs or do not quote them.

2.Juniper out-of-order packets when traffic hits 73%, how come smartbits show over 99% throughput for Juniper OC-192 interface? at least 73% should be the top for Juniper box.

3."18 bytes" signature for smartbits is not all used for packet sequence, just part of it. Smartbits has "Data Integrity" field which I bet you could not config because of the short payload. So even if Smartbits could track the packet order, the counter could circle back and your continues flow will like multiple burst flows, which may less than 73% of 10G. I will verify that in my "small" lab.

4.The test results should be weighted, not in "0" and "1" form.

5.Filters for OC-192 and OC-48? how about traffic policing and traffic shaping based on IP TOS? that is more realistic. Usually core routers will not do the filers, that is edge routers' battle field.

Regards,

David

dnewman 12/4/2012 | 8:46:28 PM
re: Internet Core Router Test
Mr. Zhou,

Thanks for your comments. To respond:

1. It's not my definition of throughput. This is from RFC 1242, the terminology document for router benchmarking. This document is nearly 10 years old, and for all that time it has served as the standard terminology for this industry.

2. The Smartbits does record misordering correctly. The accuracy of Smartbits counts was not an issue here.

3. Mr. Zhou states that "Newman's 40 bytes payload test can not allowed TCP header." This is incorrect. In fact, a TCP packet can be as short as 40 bytes -- 20 bytes apiece for the IP and TCP headers, per RFCs 791 and 793. (As stated in the methodology document, all references to packet length exclude the link-layer header and CRC.)

In this project we used IP packets for the data-plane traffic. These packets did not contain TCP headers. Nonetheless, the Smartbits was still able to track packet order because of an 18-byte signature field it places in the payload of each packet.

3. We most certainly did *not* conclude that reordering was not a problem. In fact we said the opposite -- that we disagree with Juniper's contention that it's a nonissue under all circumstances.

As stated in the article, the only bulletproof means of avoiding reordering problems is not to do it in the first place.

Gibson2001 12/4/2012 | 8:46:29 PM
re: Internet Core Router Test Alex,

It seems that you are one of the GSR user and you does not like it.

Product flaws for data network vendors are normal,
have you ever found JUNOS hanging there and you need to manully power off the box? Try it! It is fun.

Gibson2001 12/4/2012 | 8:46:29 PM
re: Internet Core Router Test Hi sschubert,

I do not agree with David Newman about his definition about "throughput".

Juniper OC-192 interface will out-of-order the packet when the traffic hits 73% guage, this behavior may not be recorded correctly by smartbits using smartwindow software because David Newman's 40 bytes payload test can not allowed TCP header included nor the packet stamp in the packet body.

If packet out-of-order is not big problem according to David Newman, 0.1% packet drop for OC-192 is fine too, since packet out-of-order and packet drop have the same result - retransmission.

For a fair game, anybody drop the packet or out-of-order the packet should score ZERO, and there is the clear winner in OC-192 baseline.

Regards,

David Zhou
Ken2000 12/4/2012 | 8:46:29 PM
re: Internet Core Router Test The testing should be about the capability of the routers, not the capability of ISP. What you said about "not many ISPs have 115 Gbit/s of capacity on their networks" reminds me "5 computers is enough for the world".
optinuts 12/4/2012 | 8:46:30 PM
re: Internet Core Router Test I would like to compliment LR and mr newman for a fair, unbiased report, for thorough and open presentation of the results.
alex 12/4/2012 | 8:46:30 PM
re: Internet Core Router Test Yes, and GSRs do not crash all the time, do not have stange SFC issues and CEF actually works... Not.
dnewman 12/4/2012 | 8:46:33 PM
re: Internet Core Router Test
Hi sschubert,

In fact we present all the numbers you're looking for here, though perhaps not all in one place.

For Cisco and Juniper, all the latency numbers are at the throughput level, as required by RFC 2544. For example, for Cisco's 12416 on the OC192 test bed, we measured latency for 40-byte IP packets with an offered load of 52 percent, because that's the highest load the device sustained without loss.

You can get the offered load for all the other Cisco and Juniper latency measurements by referring back to the throughput measurements.

This applies only to the Cisco and Juniper numbers. As discussed in knn's earlier post and my reply, the numbers for Charlotte's Networks and Foundry reflect offered loads at line rate.

Regards,
David Newman
sschubert 12/4/2012 | 8:46:33 PM
re: Internet Core Router Test Wouldn't it be more useful to show the percentage of traffic vs. it's latency verses min/ave/max?

If I look at the figures when there's a significantly higher max verses the average that tells me that there was significantly more traffic closer to the minimum. That in itself could change the winner in my mind.
Gibson2001 12/4/2012 | 8:46:34 PM
re: Internet Core Router Test I did read the article very well as always, such as like your test when Datacommunication published "The lone router" 1st Sep. 1999

I am kind of agree with about only few ISP have that much of traffic, but I also like to remind you that those ISPs are the major buyers of the routers which you tested as the "Internet Core Router".

dnewman 12/4/2012 | 8:46:35 PM
re: Internet Core Router Test
Please read the article before making ridiculous accusations. Each vendor supplied four chassis, not one as you suggest. This is clearly spelled out in the main text and the methodology sidebar.

And yes, I'd be happy to bet that not many ISPs have 115 Gbit/s of capacity on their networks--maybe only 10 to 25 out of the ~7,000 ISPs in North America alone.

Sure, I'd take that bet -- it would be one of the few investments with a positive return today. :(

dn
Gibson2001 12/4/2012 | 8:46:36 PM
re: Internet Core Router Test First of all, there is no OC-192c, only OC-192 VC-4c concatenated interface.

12 "OC-192c" for Juniper M160? That must be a 12-slot M160 box, afaik M160 has 8 slots.

Not many ISP have that kind of cap.? want a bet?
Gibson2001 12/4/2012 | 8:46:36 PM
re: Internet Core Router Test How about the performance in the fully loaded configuration box?

As far as I know, foundry and Cisco present decent performance with maximum number high speed ports, GSR will not suprise you when you put more high speed ports into the system because of the distributed architecture.

3 ports OC-192 VC-4c/c per box could not represent real ISP PoP design, why not lightreading be more realistic?
dnewman 12/4/2012 | 8:46:36 PM
re: Internet Core Router Test
er, we had 12 OC192c and 48 OC48c interfaces on the test bed. Please see the schematics in the test methodology box.

Fully loaded? Not many ISPs have that kind of capacity in their entire networks.

Regards,
David Newman
dnewman 12/4/2012 | 8:46:39 PM
re: Internet Core Router Test
Hello, I am the author of this article. :)

knn makes an excellent point regarding how latency should be measured. RFC 2544 is quite clear in stating that latency must be measured only at the throughput level.

However (you knew that was coming...), I'm afraid knn goes from there to make an incorrect assumption based on misuse of the term throughput. In particular, there are a couple of points in knn's post I'd like to respond to:

1. "All the results at OC192 related to
latency, are bogus (since the throughput was less
than 100% )"

Wrong. The fact that throughput is less than 100 percent in no way negates the legitimacy of the latency measurements.

knn may be assuming here, incorrectly, that in the OC192 tests we *only* offered traffic at line rate and used the latency measurements from those tests. As stated in the article, this was not the case.

In fact, in all the OC192 tests the latency measurements were made at the throughput level -- the highest offered load with no loss, as defined in RFCs 1242 and as specified in RFC 2544.

2. In the OC48 tests, latency was again measured at the throughput level for Cisco and Juniper.

Charlotte's Networks and Foundry were problematic, in that throughput for both vendors' boxen was 0; there was no offered load level, even 1 percent, at which these devices didn't drop at least some packets.

As the text discusses, we obtained the forwarding rate (not throughput) and latency numbers for Charlotte's Networks and Foundry by offering traffic at line rate. The article concedes that the latency measurements for the two boxes (which are *very* different, by the way) are largely a function of queue depth and routing code optimization. I fully agree with knn that queuing theory comes into play here.

Regards,
David Newman
russ4br 12/4/2012 | 8:46:39 PM
re: Internet Core Router Test Pls fix it! ;^)
knn 12/4/2012 | 8:46:40 PM
re: Internet Core Router Test
Folks,

counting average latencies in the presence of
extreme packet losses (like with 40 byte
load at 100% ) is a rather naive view of the world.

If I drop all the packets and forward only one,
I can show you average/maximum latency of 10us ..
All the results at OC192 related to
latency, are bogus (since the throughput was less
than 100% ) ...

In order to do latency/delay tests, you must
configure a traffic pattern with no losses
in the router. Then you can count delays.
If the router is over-subscribed (because
it can not forward so fast), delay calculations
will go to infinity (or be bounded by buffer
size). You don't need a test to figure that
out ...

Basic lessons in queueing theory :)
mumbogumbo 12/4/2012 | 8:46:40 PM
re: Internet Core Router Test "Its very strange that both wining comoanies are the richest ones..."

Why is that so strange, since this implies that both companies would have more capitol for R&D.

/mg
Steve Saunders 12/4/2012 | 8:46:41 PM
re: Internet Core Router Test Re: "how much did juniper & cisco pay you???"

What an incredibly disappointing question to lead off this testing discussion.

After spending 6 months on this project we were hoping for something more intelligent.

The answer is "nothing." (That's what "independent test" means).

We paid Network Test to do the test. Spirent supplied the test equipment, for free. The vendors supplied their routers for free.

Steve
fk 12/4/2012 | 8:46:42 PM
re: Internet Core Router Test What a dumbass question, and one which says a lot more about the questioner than the target. If someone was being paid off, it would have to be the independent lab that did the testing. And what would they have to gain? A little money. What would they have to lose? Their livelihoods and good name. Doesn't seem like the sort of risk that's worth taking. On its face, your accusation is ludicrous.

To demonstrate even more convincingly just how clueless you are, you then ask how the routers can perform so well given that their share prices have dropped in recent months, as if performance of a box tracks the stock market. Are you for real? I'd like to suggest in the strongest possible terms that you stop typing, because you are showing yourself to be stupider by the word.
bibiyahoo 12/4/2012 | 8:46:43 PM
re: Internet Core Router Test Its very strange that both wining comoanies are the richest ones...

I wonder how come their router is so good when their shares went down big time in the last months...

1 + 1 = ???
Peter Heywood 12/4/2012 | 8:46:43 PM
re: Internet Core Router Test We gave the test results to some network architects at service providers under NDA so they could give us their feedback.

In general, our results tallied with their own experiences when testing this equipment in their own labs. However, they interpreted the results in different ways.

Scroll down and see "Carrier Reactions" on the testing site home page to get the details.

We'd be interested in getting more feedback on:

- how these tests compare with other evaluations, particularly ones from carriers.

- what these tests demonstrate, in terms of which router is best for what circumstance.

[email protected]



mrfiber 12/4/2012 | 9:22:00 PM
re: Internet Core Router Test I'm talking single and multi-mode. I'm trying to gather enough data to get a better feel of exactly what is happening out there. We are considering having a contest with several of the main types on connectors and rating the results as to loss, spoils, ease of use and finished cost including labor and material. Your comments will certainly be appreciated.

Best regards,

william Graham

[email protected]

http://www.fiberoptictraining....
http://www.fiberoneinstallatio...
yikes_stripes 12/4/2012 | 9:25:44 PM
re: Internet Core Router Test Are we talking SM or MM?
mrfiber 12/4/2012 | 9:25:47 PM
re: Internet Core Router Test I am looking for figures and field test results on different types of fiber optic connectors. I read an article yesterday suggesting that connectors with a pre-installed fiber could expect a loss of .2dB and hand polished connectors a loss of .5dB. My experience is opposite to this with hand polished an easy .2db or less. I would like to get comments from others.
This is probably in the wrong category but I don't see how to post in a new category.

Best regards,

[email protected]
metro_ether_man 12/4/2012 | 9:46:09 PM
re: Internet Core Router Test Possibly it may be the "fastest".
For L2 it is the undisputed leader.
But are there functions in the TCL
library for OSPF ? Hmm, In addition there
is required functionality that does
not exist today. While the integration
of Anvil brings some promises it also
brings a long, long list of issues
concerning protocol conformance, in addition
to limited interface support. You spend alot
of time developing the test platform instead of testing the protocols.

QARobot and Router Tester offer a tightly
integrated platform for development, thier
suites are illustrative (most all) and not exhaustive.

The important thing about routing protocol conformance compliance testing is the required
source and sink modules (Agilent does this for you). Otherwise its a lot of extra work to set
up the correlation from scratch.

Guess it all depends on what your working on, Adtech is great for ATM stuff, but lacks
in protocol decode capability, but they do have
the test suites available also.

Watch the flows/Routing tables concerning the slowpath/fastpath!
sheikhaa786 12/4/2012 | 9:47:07 PM
re: Internet Core Router Test Hi Anil,

Let me know what specific Testing information you need for BGP, I may be able to assist u in that.
Let me know what features you want to test in BGP and what Testing Equipments you have.

Once I hear from you, I will try to give you some useful information and help (if needed).

You can Test BGP (almost All features) with the help of either Agilent Router Testers, or Ixia, or Adtech (that's what I am using in my set up)plus some Cisco or Juniper Routers if u want to test Inter-Op also.


Thanks,
--Saeed
veemee 12/4/2012 | 9:47:53 PM
re: Internet Core Router Test Hello Duane/tdelrosa,

I was wondering if you were using the drafts that the IETF was working on for OSPF convergence, in the BMWG working group.

Regards,
Veemee

=============================================
Hello,

It's not freeware (yet there are some encode/decode stacks are available as freeware), but I could certainly arrange a demostration/evaluation of the Agilent portfolio for you.

* Integrated traffic & routing simulator
* OSPF scalability testing (for IPv4 and IPv6)
* OSPF interoperability & conformance testing
* Convergence testing

[email protected]

regards

Duane Sword
Worldwide Business Development Manager
Agilent Technologies
Duane Sword 12/4/2012 | 9:50:08 PM
re: Internet Core Router Test Hello,

It's not freeware (yet there are some encode/decode stacks are available as freeware), but I could certainly arrange a demostration/evaluation of the Agilent portfolio for you.

* Integrated traffic & routing simulator
* OSPF scalability testing (for IPv4 and IPv6)
* OSPF interoperability & conformance testing
* Convergence testing

[email protected]

regards

Duane Sword
Worldwide Business Development Manager
Agilent Technologies
tdelarosa 12/4/2012 | 9:53:14 PM
re: Internet Core Router Test What version of software were you using? Ixia OSPF performance has improved dramatically since the 3.50 release. We believe we have the fastest OSPF convergence and overall scalability on our Ethernet modules with a local CPU. I would suggest you contact your local Ixia sales representative or me for more information.
TriteReading 12/4/2012 | 9:53:48 PM
re: Internet Core Router Test We used to use IXIA for this but found the box pretty weak on the routing protocols. Recommend you look at Router Tester from Agilent ... very good on the OSPF front.
bgopi 12/4/2012 | 9:57:52 PM
re: Internet Core Router Test Hi,

Can anyone tell me, where i can get the testing tool for testing OSPF. Would be happy, if it is freeware/ Demo product.

Thanks in advance
emtech_rob 12/4/2012 | 9:59:30 PM
re: Internet Core Router Test What mappings are you looking for? If you want to test OC-48 AND OC-12, OC-3, OC-1 simultaneously, you may want to check out gnubi (http://www.gnubi.com). Price and performance is superior to Acterna, Agilent, Spirent, et al.

Cheers.

Rob
Duane Sword 12/4/2012 | 10:04:11 PM
re: Internet Core Router Test Do you require traffic at IP/layer-3 over the channelized OC48 interface, or are you looking for more test focus at layer-1/2 in regards to SONET framing, payload scrambling, or just general PRBS patterns and error monitoring/injection at layer2?

There are a number of simulation solutions for non-concatenated (channelized) at OC48 thru to STS-1, including Acterna, Agilent, Anritsu, and Spirent to name a few.

regards

Duane Sword
Business Development Manager
Agilent Technologies
Duane Sword 12/4/2012 | 10:04:12 PM
re: Internet Core Router Test Checkout the test plan from Agilent which offers some test scenarios for:
- routing functional test
- routing conformance test
- routing stability and scalability
- the linkages between forwarding performance and routing topologies and dynamic effects of route flapping, convergence, etc

http://advanced.comms.agilent....

You can find on the LightReading.com site the BakeOff test that was conducted a while back which encompassed routing test scenarios, and there are other test guidelines from 3rd party independant test labs such as Network Test, BTexact, Greenwich Technology Partners to name a few that you could approach.

If you would like additional information on the Agilent BGP-4 test scenarios specifically, and in a lot more detail, I would be happy to offer this to you ([email protected]).

I know this is not a site for products/sales, I can offer you detailed whitepapers, test plans and training material without any product/marketing material.

regards

Duane Sword
Business Development Manager
Agilent Technologies
anil1977 12/4/2012 | 10:10:45 PM
re: Internet Core Router Test Dear Newman,

Can you provide me the steps needed to test BGP and how it is been tested.Sorry if this has been asked in the message board. Please let me know the basic idea of testing BGP and what devices are needed to test BGP and how it is been done?

thanks in advance

Anil
balakrig 12/4/2012 | 10:11:41 PM
re: Internet Core Router Test Hi,
For the core router testing involving Cisco,Juniper,etc, MPLS tests were performed.
I had a few questions reagrding this:
1) How many labels were pushed on the ingress routers?
2) How many labels were popped on the egress routers?
3) What was the label stack operation on the transit?
4) If multiple label stack operations were not exercised, is there a particular reason for this?
5) How were the FECs classified: IP-5 tuple, prefix or host address?
Thanks,
Ganesh
phone_doc 12/4/2012 | 10:11:59 PM
re: Internet Core Router Test I am looking for information regarding Oc48 non-concatenated test equipment. I need to be able to run Oc12 oc3 and oc1 traffic simultaniously. Any thoughts or help would be greatly apprecited..
skeptic 12/4/2012 | 10:23:25 PM
re: Internet Core Router Test 1) How big will the LFIB table be ?
Will this have equivalent 10,000+entries.
---------------
It all depends on how you are using MPLS.
If you are using LDP, the LFIB table is going
to be on the order of the number of OSPF or
ISIS routes. I dont think it will have a
direct relationship with the number of BGP
routes.

If you use RSVP-TE, your in effect creating a
virutal interface (a tunnel) between the starting
and ending points for MPLS. And so the number
of labels in the LFIB is going to be the number
of virutal interfaces/tunnels you want in your
network. That number doesn't necessarly
correspond to the number of routes in the system.


2) How will the routing table be mapped on to the FECs?
---------------

Thats too long an explaination. I would go get
a book on MPLS.

3) Alternatively , can I say that the no of FEC's created will be equivalent to the number of neighbours(directly connected routers).

No.


In general, there is no absolute rule for
calculating the number of labels that a system
will need to support (other than the number
increases every year). It really depends on
the customer and the application.
mohan_g 12/4/2012 | 10:23:27 PM
re: Internet Core Router Test MPLS Gurus,

I am a novice to MPLS technology.I need some clarifications on the following queries

A typical BGP Routing table will have 10,000+ entries. The problem with IP Forwading is that it needs to do find a best match in this big table.

MPLS overcomes this by using labels.
1) How big will the LFIB table be ?
Will this have equivalent 10,000+entries.

2) How will the routing table be mapped on to the FECs?

3) Alternatively , can I say that the no of FEC's created will be equivalent to the number of neighbours(directly connected routers).

Any help on this will be highly appreciated.

TIA
MG

Ps : I am not sure this is the right alias.
Duane Sword 12/4/2012 | 10:38:11 PM
re: Internet Core Router Test Hello Nancy,

DVMRP is yet another acronym in our world of broadband comms, Distance Vector Multicast Routing Protocol. DVMRP is a multicast routing protocol largely based on RIP, this is defined in RFC1075.

On the Agilent Technologies website you will find some whitepapers (not sales literature) on multicast routing and test plans.

Further I can recommend a basic text book "IP Routing Fundamentals" authored by Mark A. Sportack.

regards

Duane Sword
Agilent Technologies
raid 12/4/2012 | 10:48:10 PM
re: Internet Core Router Test David,

SCSI control frames are 68bytes, right?
Why does the test use 60Byte frames ? Please
explain how this number was chosen.

Thanks,

-raid
mullinst 12/4/2012 | 10:49:37 PM
re: Internet Core Router Test David -- Thanks much for the help. I'll check out the files.

-- Tim Mullins

dnewman 12/4/2012 | 10:49:59 PM
re: Internet Core Router Test Hi Tim,

Please forgive the high latency of my response. I've (finally) dug up the prefixes we advertised in the longest-match lookup tests. Versions for the OC-48 and OC-192 test beds are now available here:

ftp://public.networktest.com/L...

ftp://public.networktest.com/L...

There are a couple of explanatory notes in the spreadsheets indicating how we increment addresses, and which interfaces received which prefixes.

To see which prefixes we added in the longest-match test, you can compare these larger tables with the base-case tables that were already referenced in section 1 of the test methodology:

ftp://public.networktest.com/L...

ftp://public.networktest.com/L...

I've also updated the methodology to add links to the longest-match tables.

Hope this helps.

Regards,
David Newman
Network Test
Ncronk 12/4/2012 | 10:50:10 PM
re: Internet Core Router Test How does DVMRP relate to this field of expertise? I am not an engineer - still learning. Please don't flame me!!!
Thx, Nanci
bola12 12/4/2012 | 10:51:05 PM
re: Internet Core Router Test Hello?
mullinst 12/4/2012 | 10:51:17 PM
re: Internet Core Router Test David -- Thanks for your reply on this topic. I understand the algorithm you describe, but am still interested in the specific 50k prefixes used. Is it possible to obtain a file of the particular additional 50k values used in the test? Thanks for pointing me to the right source of information on this.

-- Tim Mullins

dnewman 12/4/2012 | 10:51:22 PM
re: Internet Core Router Test ps. The 127/whatever example in my previous post is obviously fact-free. Yes, of course it's the localhost block, and no, we didn't use it in the LR tests. We also didn't use the RFC 1918 blocks plus a few other ARIN-reserved blocks.

Regards,
David Newman
Network Test
dnewman 12/4/2012 | 10:51:28 PM
re: Internet Core Router Test Hi Tim,

Thanks for your inquiry. The extra 50K prefixes were simply more-specific variations of entries in our 200K-plus table.

For example, if the main table had an entry for:

127.32.0.0/16

then we might add an entry for:

127.32.0.0/24

Hope this helps.

Regards,
David Newman
Network Test
mullinst 12/4/2012 | 10:51:44 PM
re: Internet Core Router Test Would like to understand more detail from the Longest Match Scenario in the Core Router Test. I've found information regarding the 201k IP prefixes for the baseline test. However, can't seem to find more definition on the specific 50k additional prefixes used for the LM Test. Can someone point me to that background? Thanks.

Tim Mullins

fr7 12/4/2012 | 10:57:49 PM
re: Internet Core Router Test It would be nice to see the results on latency and jitter figures on each Class of Service... In reality an enterprise for example would like to have multiple CoS each for different applications, Gold for VoIP, Silver for SAP and Bronze for Web.

It would be interesting to see how well each vendor comes up in implementing the Differv EF and AF classes.
skeptic 12/4/2012 | 10:59:19 PM
re: Internet Core Router Test I know this isn't a multicast thread, but does anyone know if Juniper really supports dvmrp. I think Cisco refuses to and Juniper says they do, but I wonder if there are any real dvmrp networks out there. I couldn't see that tested here. Maybe there should be a multicast test given the growing use.
------------------

If there was a multicast test, it should be on
PIM sparse-mode rather than DVMRP or anything
else.

I think cisco supports some sort of DVMRP
"interoprability mode". Juniper I think support
more, but I've never used multicast on a
juniper system. But (its an old story) just
because a vendor claims to have something is
no reason to trust that they really have it or
that it works (especially in the case of multicast).
country_boy 12/4/2012 | 10:59:19 PM
re: Internet Core Router Test don't know if Juniper supports it. There are real DVMRP networks out there. Go look at starburst.com, MBONE is mostly DVMRP. Check out Unisphere, they have a pretty good mcast implementation for the edge. And support DVMRP on that edge router.
jit 12/4/2012 | 10:59:20 PM
re: Internet Core Router Test I know this isn't a multicast thread, but does anyone know if Juniper really supports dvmrp. I think Cisco refuses to and Juniper says they do, but I wonder if there are any real dvmrp networks out there. I couldn't see that tested here. Maybe there should be a multicast test given the growing use.
metrodude 12/4/2012 | 11:03:20 PM
re: Internet Core Router Test Hello David,

A would like to suggest/request from Network Test
and LR for any future testing scenario's that
you please revisit the methodologies, tools if required, Etc..

Have you looked at what a Spirent "Smartbits" unit
actually puts on the wire with a protocol analyzer ? It does some things that may effect
the validity of what your trying to accomplish.

What is your criteria for selecting test equipment?

How about very detailed representation in the tests and criteria that is defined by standards to acomplish these.

Spreadsheet format. With nothing to guess at, or missing from the equation.

You will find that not all test equipment is created equal.


Chughster 12/4/2012 | 11:22:58 PM
re: Internet Core Router Test Can some one explain the possible causes of the following trend:

With NAT enabled throughput for larger sized packets is lower than the throughput for smaller sized packets. That is, Packet streams with 64 byte size packets have a higher throughput compared to packets with 1500 byte size packets. The media is Fast ethernet. I am not sure if this is related to NAT in any way or if it is just an inherent switching performance any general router. Anyone with packet-switching knowledge please comment. The router is of VXR(NPE400). The MTU size is standard 1500 bytes for the interfaces.

What could be a logical explanation of this trend?

Janus Chu
Mathew Orman 12/5/2012 | 12:19:04 AM
re: Internet Core Router Test http://www.ultra-faster-than-l...
Mathew Orman 12/5/2012 | 12:19:04 AM
re: Internet Core Router Test http://www.ultra-faster-than-l...
bgopi 12/5/2012 | 12:57:39 AM
re: Internet Core Router Test Hi,

Is there any place where i can get the typical core router' routing table dump and the ACL rules dump of the same?

Can someone help me in this regard.

Please mail me at [email protected]
Thanks
Gopi
Tony Li 12/5/2012 | 1:21:24 AM
re: Internet Core Router Test In fact, both matter, but the packets/sec is harder to achieve. Consider that for each packet, a router has to perform an IP lookup, make a switching decision and get the packet to the correct output interface with the correct encapsulation.

The amount of work is the same regardless of the size of the packet. This leads some manufacturers to skimp on the processing power for packet processing, so they will underperform when tested with small packets. With large packets, the packet processing rate is lower, so it's simply a matter of bandwidth. Most players get that right, or come arbitrarily close.

Tony
rsunkara 12/5/2012 | 1:21:24 AM
re: Internet Core Router Test Hi,

I am interested in knowing why the router performance is measured in packets/sec and not bits/sec??

Thanks
gereizt 12/5/2012 | 3:17:16 PM
re: Internet Core Router Test Goto any looking glass and show the ip routing table. As for ACL rules, forget about it. You are asking someone to show you the addresses they allow and block and what, how and why. You aren't going to get it from a major provider and shouldn't get it from anyone with any kind of security sense.

HOME
Sign In
SEARCH
CLOSE
MORE
CLOSE