x
Optical/IP

Cisco Twists Test Results

Desperation seems to be setting in at Cisco Systems Inc. (Nasdaq: CSCO), judging by a press release it issued last night, claiming that Cisco, rather than Juniper Networks Inc. (Nasdaq: JNPR), had come out on top in Light Reading's test of Internet core routers (see Cisco Claims Router Test Success).

The press release not only misrepresents the overall conclusions of the test but also makes inaccurate statements about the results. In addition, it fails to mention that Light Reading commissioned the tests, which were carried out by Network Test Inc., and that the full results are published on this site (see Internet Core Router Test).

"I'm very disappointed in this misrepresentation," says David Newman, president of Network Test. "Cisco's 12416 put up some very, very good numbers in this test, so the company had no need to spin it the way they did."

At press time, Cisco had not responded to Light Reading's requests that it should issue a second press release correcting these errors and omissions. So, for the record, here’s a replay of parts of Cisco’s press release, annotated with our comments:

Cisco’s Headline: Cisco Reconfirms Leadership In Test of High-End Routers

Truth: Juniper did better overall.

Cisco’s Headline: Only Vendor to Demonstrate 100% Line Rate IP and MPLS Performance For 2.5Gbps OC-48 and 10Gbps OC-192 Throughput Truth: Cisco did not achieve line-rate throughput in MPLS and IP OC192. In fact, in one of these tests throughput was just 52 percent.

Cisco:The Cisco 12416 Internet router has outperformed all other networking vendors in nine out of twelve tests.

Truth: Cisco outperformed Juniper's M160 in only four of the 16 tests.

Cisco:Cisco won every category where the new 1-port OC-192 and 4-port OC-48 line cards and 10-Gbps switch fabric were tested.

Truth: Juniper tied or beat Cisco outright in multiple OC192 tests

Cisco:"Cisco was the only vendor to deliver line rate performance for all of the IP and MPLS forwarding tests for both OC-48 and OC-192," said Robert Redford, senior director of marketing, IP POP systems business unit at Cisco.

Truth: Robert Redford may have said this -- Paul Newman may have said it, too -- but it just ain't true. Cisco did not deliver line-rate performance in all tests. In particular, the 40-byte IP throughput over OC192 was just 52 percent, as already noted.

Larry Lang, Cisco’s vice president of service provider marketing says that Cisco was “pretty careful” about checking its facts in the release. Cisco will issue a correction if needs be, he adds.

The press release “doesn’t misrepresent the test. It differs from your editorial conclusions, and that’s okay,” says Lang. He adds that Cisco doesn’t dispute the test methodology. “It’s a fine test. Our difference is in the conclusions.”

Lang bobbed and weaved when asked why Cisco hadn’t acknowledged that the test had been commissioned by Light Reading and hadn’t referenced the full results on Light Reading's site. “I guess I was confused,” he says. “There’s a difference over whose test it is and who did the test.” -- Peter Heywood, international editor, Light Reading http://www.lightreading.com
net_exprt 12/4/2012 | 8:42:49 PM
re: Cisco Twists Test Results By the way, LightSwitcher, is that what you do at work all day long? Are you proud of your ability to arrange words together to come up with the following gem:
"JUST TELL THE TRUTH and GET TO WORK ON A BETTER PRODUCT"...

Tell us more about yourself, "lightSwitcher"...I hear there is a tremendous market out there for lightswitchers...make sure your resume is polished up and ready to go, oh lightswitcher...
net_exprt 12/4/2012 | 8:42:49 PM
re: Cisco Twists Test Results raja, raja, raja...

There are those that read the "monster win" article and understand it for what it is...an article based on sensationalism and poor journalism...and then there are folks like you, who are either naive enough to be mesmorized by the headline, or are Juniper employees... As I have stated before, there are alot of factors associated with such testings.
As far as "rave reviews" as you put it so elloquently, all I saw where idiotic comments coming out of you by taking sentences that were out of context and responding only to those.
Rajah 12/4/2012 | 8:43:06 PM
re: Cisco Twists Test Results
Sparky:
First, let me start by agreeing with the others on this board that you are clueless. Nothing personal but you come off as being arrogant and obstinate.

Rajah:
Other than net-exprt who has been getting his own rave reviews, can you tell me who the "others" are?

Sparky:
You are the one not sticking to the subject of this particular thread. I should know I started it. You are in your own world.

Rajah:
Uh....Are you in another planet? What's the subject name assigned to this Article Talk forum? The last I checked, it was "Cisco twists Test Results". That is the subject. Like I said, did you ever write an exam in school?

Sparky:
It is not just my opinion, if you have been reading these threads you would know that many other have been saying the same thing.

Rajah:
And many others say exactly the opposite.

Sparky:
You deny that OIR, capacity, redundancy and QoS latency tests would not have been vital to a comprehensive test of core routers?

Rajah:
You still don't get it do you? The issue is not of the type or nature of tests that would make for a "more" meaningful or comprehensive test. Its WHETHER and WHY did Cisco twist the results of the tests that were performed, regardless of the types of tests. Regarding your more "comprehensive" tests, are you implying that the tests performed were not at all comprehensive? If you seem to think so, it then brings up the question of how and why Cisco entered the tests in the first place. And please, don't make yourself look like more of an idiot with your theories of Cisco having absolutely no say in the test plans. I see you still have not replied to the other thread showing your "basis" for your argument. Looks like there never was one, was there?







stilespj 12/4/2012 | 8:43:11 PM
re: Cisco Twists Test Results I guess Cisco defends their spin by saying "it all depends what the meaning of is is"

Sheesh, Cisco, trying to spin your close second place finish into an outright win is extending the elastic properties of the truth beyond their breaking points!

Paul
Belzebutt 12/4/2012 | 8:43:15 PM
re: Cisco Twists Test Results They added the words "The test was sponsored by Light Reading". But they still won't link directly to the test results.
skeptic 12/4/2012 | 8:43:15 PM
re: Cisco Twists Test Results I guess Cisco's pride was hurt.
JUST TELL THE TRUTH and GET TO WORK ON A BETTER PRODUCT!
---------------------------------

"Better product"

1. Put virtual memory in the system so you can generate unrealistic benchmarks. After all, performance and stability doesn't matter. All that matters is a big number.

2. Break your routing implementation so it continues to incorrectly accept BGP routes even after the forwarding system has filled up with routes. Customers just don't care about this sort of thing and the testers will not catch on to the implications of it.

3. Next time you have to do a line card, forget unimportant "geek" features like preserving packet ordering. What matters is getting the product to customers fast. And when the customers report the bug, first pretend it isn't happening. Then have your marketing people work their friends outside the company to convince everyone that the bug isn't important.

4. Become really friendly with the testers. Have your CEO give an interview to light reading. Light reading will do anything for an important interview. Just ask Caspian networks.

perry1961 12/4/2012 | 8:43:16 PM
re: Cisco Twists Test Results Start-up Charlotte's Networks Joins Cisco and Juniper as Light Reading's Top Three Leading Core Router Providers
Charlotte's to Commission Spirent Communications and The Tolly Group to Authenticate Ranking in Light Reading's First Test of Core Routers
Andover, Mass.--(BUSINESS WIRE)--March 16, 2001-- Charlotte's Networks, a leading developer of carrier-class routers for the emerging IP infrastructure, today said Light Reading and Network Test have validated Charlotte's Networks as the third contender in the core router market.

Charlotte's was pleased to participate in the first test of core routers by Light Reading, the premier information resource for the optical networking industry. The test was performed by David Newman, President of Network Test, the respected pioneer of network testing.

Charlotte's, the new kid on the block, started development of its core router 2 1/2 years after Juniper and 15 years after Cisco, and is quickly growing-up -- out of the toddler years. Charlotte's was the youngest out of the four participants of the test and the only start-up with the confidence to participate one-on-one with Cisco and Juniper.

Eli Stein, vice president of engineering for Charlotte's, commented: ``Cisco and Juniper are the established market leaders. We are pleased with Light Reading's validation of Charlotte's as the third contender. We feel confident in Charlotte's position in this potential $15 billion market, especially since our box has double the speed and port density compared to any other vendor.''

Stein continued, ``Prior customer commitments prevented us from completing all aspects of the Light Reading test within the deadlines for publication. However, we feel so confident in the Aranea's ability to perform, Charlotte's is engaging the Tolly Group and Spirent Communications to reproduce Light Reading's first test of core routers. We look forward to the final results, which will establish Charlotte's as the only viable alternative to Cisco and Juniper.''

The test will be conducted by Spirent Communications, a worldwide leader in performance analysis solutions. The test will be performed at Spirent's lab using the exact equipment and identical test methodology used in Light Reading's first test of core routers. The Tolly Group, long recognized as a leading independent networking test benchmarking firm, will ensure that the test will be identical in every manner to the Light Reading test. Charlotte's and Light Reading have agreed to publish the results.

About Charlotte's Networks

Charlotte's Networks is a leading developer of carrier-class routing solutions for the emerging IP infrastructure. Charlotte's Networks' innovative solutions provide the high performance, scalability, Quality of Service and reliability required by carriers, large Internet Service Providers (ISPs) and Competitive Local Exchange Carriers (CLECs). Charlotte's Networks has headquarters in Andover, with R&D facilities in Israel. For more information, contact Charlotte's Networks toll-free at 866/738-4675, or visit the company online at http://www.cwnt.com/.

Charlotte's Networks is a technology investment partner of MRV Communications Inc. (Nasdaq:MRVC - news). For more information, visit http://www.mrv.com/.

melao 12/4/2012 | 8:43:17 PM
re: Cisco Twists Test Results http://newsroom.cisco.com/dlls...

this is the site.
it has wuotes of david newman. Well, something is very odd in this story.
Sparky 12/4/2012 | 8:43:19 PM
re: Cisco Twists Test Results First, let me start by agreeing with the others on this board that you are clueless. Nothing personal but you come off as being arrogant and obstinate.

Rajah:
So?? What does that have to do with twisting the results? Did you ever write an exam in school? One of the basic principles is to answer the question and stick to the subject.

Sparky:
You are the one not sticking to the subject of this particular thread. I should know I started it. You are in your own world.

Rajah:
That's just your opinion. Can you name the "many people" other than Cisco proponents who thought the tests weren't meaningful?

Sparky:
It is not just my opinion, if you have been reading these threads you would know that many other have been saying the same thing. You deny that OIR, capacity, redundancy and QoS latency tests would not have been vital to a comprehensive test of core routers?

Sparky
Rajah 12/4/2012 | 8:43:22 PM
re: Cisco Twists Test Results Sparky:

First, LR is just providing drama as demonstrated by rediculous editorial and comments from the author.

Rajah:

So?? What does that have to do with twisting the results? Did you ever write an exam in school? One of the basic principles is to answer the question and stick to the subject.

Sparky:

Second, the tests were not comprehensive, yet, came to a conclusion which is suspect. Additionally, some of the tests were not meaningful in many people's opinion.

Rajah:
That's just your opinion. Can you name the "many people" other than Cisco proponents who thought the tests weren't meaningful?

Rajah 12/4/2012 | 8:43:22 PM
re: Cisco Twists Test Results Sparky:
Not true, it does have basis and I stand by my statement.

Rajah:
Alright. What is the basis? Let's see it.

Sparky:
Maybe they did or were not given the chance. Who are you to say they didn't propose tests or were given the chance?

Rajah:
Are you trying to imply a company like Cisco gave in to tests of their own equipment by a third party and were not "given a chance to propose tests"??????? Ha ha ha ha ha ha ha
Are you in this industry or are you a Cisco stock holder who is feeling the pinch of the market????

Sparky:
Again, they didn't so much twist the results as point out their strengths.

Rajah:
That's still very debatable. If the LR article is correct, they definitely fudged the results.



dnewman 12/4/2012 | 8:43:24 PM
re: Cisco Twists Test Results Hi Sparky,

Any Cisco or Juniper latency measurement was obtained at the throughput rate, as specified in RFC 2544.

As noted earlier, our measured delay (not latency) at line rate on the Charlotte's Networks and Foundry routers. It's unfortunate that we're calling it latency here, since those measurements are a function of both device latency and buffer depth.

Hope this helps.

Regards,
David Newman
Network Test


dnewman 12/4/2012 | 8:43:25 PM
re: Cisco Twists Test Results Scott Bradner, the coauthor of RFC 2544, would disagree with you. He's on record as supporting a 30-second test duration.

Besides, if a device has high jitter early in a run I want to know about it. As mentioned earlier, the test described in RFC 2544 doesn't cover that case only because test tools at the time didn't support that capability.

Regards,
David Newman
Network Test
Sparky 12/4/2012 | 8:43:25 PM
re: Cisco Twists Test Results Dave, thanks for the reply.

Can you tell me...
Was the latency tested on both routers at the throughput rate?

Sparky
tink 12/4/2012 | 8:43:25 PM
re: Cisco Twists Test Results David,

I think it's best to establish a steady state (30s seems good enough) and _then_ measure latency, even if you can measure the latency of every packet. The first packets (hard to say how many) are certainly anomolous since they are greeted by an idle router just waiting to push them through!


Thanks,
Tink

tink 12/4/2012 | 8:43:25 PM
re: Cisco Twists Test Results David,

I guess I'm on shaky ground trying to extrapolate too much. Thanks for your input.

Tink
dnewman 12/4/2012 | 8:43:26 PM
re: Cisco Twists Test Results
Correct, with one qualifier.

There is no such animal as the max throughput rate, or min, or anything else. There is only throughput.

Regards,
David Newman
Network Test
loejoe 12/4/2012 | 8:43:26 PM
re: Cisco Twists Test Results 2) No Hot Swap - hrm, heard that's an issue with JNPR. Thought that could be important, somehow.

Why in the world would I want to hot swap a blade on a router carrying live traffic? No, I'll just let my downstream ISPs and their customers suffer with no service until my next maintenance window no matter how far away that is.
loejoe 12/4/2012 | 8:43:27 PM
re: Cisco Twists Test Results Perhaps Cisco is reacting to sensational journalism and unprofessionalism of the writer who wrote the eye-catching "Juniper Wins Monster Router Test" headline? Or perhaps, the statement that the M160 is "currently superior?" The results looked rather close and Cisco openly admits ACLs are not supported on the new OC-48 and OC-192 blades.
loejoe 12/4/2012 | 8:43:27 PM
re: Cisco Twists Test Results >>>I'm predicting now we will see blatant omissions of all things positive regarding cisco and intense focus on anything negative.

Isn't that just the same old reporting typically of Lightreading?
loejoe 12/4/2012 | 8:43:27 PM
re: Cisco Twists Test Results And thanks to everyone on the board for letting us know that:

5. Juniper delivers out of sequence packets

6. Juniper doesn't support OIR

7. Juniper doesn't support QOS
Sparky 12/4/2012 | 8:43:27 PM
re: Cisco Twists Test Results Correct me if I'm wrong.

To get an accurate measurement of latency the packets you feed in must not get trapped behind other packets waiting in a buffer? That is why you perform the latency test at the max. throughput rate. The object is to perform the test at a constant stream of packets being switch without being buffered.
loejoe 12/4/2012 | 8:43:27 PM
re: Cisco Twists Test Results > Finally, somebody nailed Cisco for making false and inaccurate claims. End users of networking gear have endured their BS for years.

Kind of like Juniper claiming line rate performance, capability to do OIR, and delivering packets in the proper sequence? BS from Juniper is still BS.
LightSwitcher 12/4/2012 | 8:43:28 PM
re: Cisco Twists Test Results
I guess Cisco's pride was hurt.
JUST TELL THE TRUTH and GET TO WORK ON A BETTER PRODUCT!
decimail 12/4/2012 | 8:43:29 PM
re: Cisco Twists Test Results Sparky do you work for Cisco? The test results appear very clear and Cisco was given an opportunity to say something prior.

dnewman 12/4/2012 | 8:43:29 PM
re: Cisco Twists Test Results
Sorry if I wasn't clear on this. Let's go back to the general statement that forwarding rate and loss are not related in any linear way.

It is not possible to derive what packet loss will be at a given load based on loss statistics from a different load.

Every device has a different curve in terms of the point where loss begins and where it ramps up in a "hockey stick." There's no general formula for predicting this.

The ONLY way to ascertain what forwarding rate, or latency, or loss, will be for a given offered load is to measure at that load level.

Regards,
David Newman
Network Test
Sparky 12/4/2012 | 8:43:30 PM
re: Cisco Twists Test Results Rajah:
Cisco had knowledge of the tests being performed. Your argument is baseless.

Sparky:
Not true, it does have basis and I stand by my statement.

Rajah:
Why didn't Cisco propose such tests prior to entering the tests?

Sparky:
Maybe they did or were not given the chance. Who are you to say they didn't propose tests or were given the chance?

Rajah:
Not after this. The basic issue is why did they twist the results? If they didn't agree with them, why didn't they just say so?

Sparky:
Yes, even after this generally speaking Cisco is a winner in the networking industry, which is the scope in which the statement was made. Reaction would be bolstered by news against Cisco rather than for Cisco. Again, they didn't so much twist the results as point out their strengths. Lastly, I think they did say so.

Sparky
dnewman 12/4/2012 | 8:43:31 PM
re: Cisco Twists Test Results
If I understand your question (and I'm not sure I do) I *think* you're asking whether it would be better to compare Cisco and Juniper at 99 percent offered load.

The answer is no, it would not be better to do that. We told vendors ahead of time that we would measure throughput and latency. We did *not* tell vendors we'd measure 99 percent forwarding rate.

RFC 1242 has very specific definitions of throughput and latency, and we followed those definitions to the letter in all measurements of Cisco and Juniper equipment.

Yes, I know we made an exception with Charlotte's Networks and Foundry (measuring RFC 2285 forwarding rate at maximum offered load (FRMOL) rather than throughput). We had to do that to get dealy measurements. There was no point at which those two vendors' devices did not drop packets.

If I have misunderstood your question, please rephrase it.

Regards,
David Newman
Network Test
chaodntamab 12/4/2012 | 8:43:31 PM
re: Cisco Twists Test Results Mr. Newman

Correct me if I am wrong. Csco is better off to do 99% so we can compare apple to apple. Is this correct?

Sicerely
Chaonima
dnewman 12/4/2012 | 8:43:31 PM
re: Cisco Twists Test Results
The test duration in all our baselines was 30 seconds. As noted in my previous message (#107 on the test message board), we measured latency of every packet in the test duration.

The question of test duration has been discussed extensively within the IETF benchmarking working group. There is consensus a 30-second duration is fine for baseline throughput and latency measurements.

The need for a 60-second duration in 2544 arose because test tools at the time measured latency on one packet in the middle of a stream. It was thought that a longer duration of 60 seconds was necessary for the device to "settle down" enough for a representative measurement to be taken.

Since tools like the Smartbits now measure the latency of *every* packet, there's longer any no need for guesswork about what a representative measurement might be.

Some of the discussion within bmwg has mentioned that the SHOULD in 2544 be changed to a MUST. I concur, but I suspect most folks today would go for a mandatory 30 seconds rather than 60.


Regards,
David Newman
Network Test



chowchow 12/4/2012 | 8:43:34 PM
re: Cisco Twists Test Results Hello ChaoChao,

I think you're confusing public tests with private vendor tests.

First, Newman doesn't do vendor testing as stated in http://www.networktest.com/tha....

Second, this router test is NOT a private test. The test results should be discussed with each vendor during and after the test. If Newman violated the rules, Cisco will be the first one to complain. But I don't see Cisco officially protests any unfair advantage towards Juniper or disqualify the test results.

I am a long-term Cisco supporter. I have been impressed by their performance at previous Newman's tests and am even more encouraged by their results in this test.
tink 12/4/2012 | 8:43:35 PM
re: Cisco Twists Test Results David,

RFC 2544 suggests that traffic should be running for 60 seconds before any delay times are measured. Specifically it states, "An identifying tag SHOULD be included in one frame after 60 seconds ..."

Did you let the traffic run for 60 seconds before making the first latency measurement?

Thanks,
Tink
tink 12/4/2012 | 8:43:35 PM
re: Cisco Twists Test Results David,

You state, "It's NOT safe to assume that given X loss at Y load you will get 0.75 X loss at 0.75 load."

But this is not what I assumed. I estimated that given X loss at Y load, you will get _X_ loss at .75Y load. In fact, what I am truly estimating

given X loss at Y load
estimate loss X'<=X for all Y'<=Y

So when you say that "the loss will probably NOT be 1 percent," you mean that the loss will probably be less than 1%?

Thanks,
Tink
Sparky 12/4/2012 | 8:43:35 PM
re: Cisco Twists Test Results Rajah,

I wouldn't mention going to school since you couldn't fingure out the point of the message, which was...

First, LR is just providing drama as demonstrated by rediculous editorial and comments from the author.

Second, the tests were not comprehensive, yet, came to a conclusion which is suspect. Additionally, some of the tests were not meaningful in many people's opinion.

Why wouldn't Cisco participate, they have a great product that did very well and people should see that through the editorial BS of LR. That is all that Cisco was trying to accomplish, pointing out the particular strenghts of the GSR demonstrated during the test.

Sparky
dnewman 12/4/2012 | 8:43:36 PM
re: Cisco Twists Test Results
You're making an assumption that has a 1 in 100 probability of being correct. :)

Sorry if I didn't make myself clear before, but the relationship between forwarding rate and packet loss is not linear.

It's NOT safe to assume that given X loss at Y load you will get 0.75 X loss at 0.75 load.

We did not conduct step tests at every possible load level so I don't know the answer to your specific 75 percent case, but in any event the loss will probably NOT be 1 percent.

Regards,
David Newman
Network Test
dnewman 12/4/2012 | 8:43:37 PM
re: Cisco Twists Test Results
Latency is measured at the throughput level. This is required per RFC 2544, and it's also stated in the text of the article.

The only exceptions to this are the OC48 results for Charlotte's Networks and Foundry, where throughput was 0. In those cases, we measured latency (or delay, to be more accurate) at line rate.

Regards,
David Newman
Network Test
dnewman 12/4/2012 | 8:43:38 PM
re: Cisco Twists Test Results
We offered the same load to each vendor in every test.

With regard to the results, here's what I said in the text:

"On the downside, the M160Gs average per-port forwarding rate of roughly 5.4 million pps is significantly off the 6 million pps pace it achieved in the OC48 tests."

The M160's average forwarding rate is also lower than 12416's in the OC192 test -- but only at first. As noted in the text, the 12416 exhibited more variation in forwarding rate than the M160 on both the flapped and stable paths. Also, note that by the end of the test run, the M160's forwarding rate is actually higher than the 12416's.

It's interesting to note that in all of the flap and convergence tests that results get worse over time. This is true for every router tested.

Regards,
David Newman
Network Test
tink 12/4/2012 | 8:43:38 PM
re: Cisco Twists Test Results David,

We're on the same page with regard to througput and forwarding.

But...
If a router can has a throughput of 52% and can forward 99% when offered 100%, how much can it forward when offered, say 75%? I'm estimating that number to be 74%. Is this not correct?

Thanks,
Tink
chaochao 12/4/2012 | 8:43:39 PM
re: Cisco Twists Test Results Mr. Newman

Seems like csco's max latency number is big. However, in your article, you did not explain under what load did this latency occured? 100% or 99%? What is Juniper's latency number at 100%? Or did they do 100%? Are we comparing csco's 100% number to Juniper's 99% number? If so, why?

Best Regards,
Chaonima
dnewman 12/4/2012 | 8:43:39 PM
re: Cisco Twists Test Results
Hi Tink,

The relationship between forwarding rate and packet loss is not necessarily linear.

Suppose I offer packets at 100 percent and observe 3 percent loss. This does not automatically mean the device under test will achieve throughput of 97 percent. It might, and it might not. In my experience devices are all over the map on this one.

That's what happened in the OC192 tests. Given an offered load of 100 percent utilization, both the Cisco and Juniper routers forwarded more than 99 percent of the 40-byte IP packets we transmitted. But NEITHER box achieved throughput anywhere close to 99 percent in this event.

It can't be said often enough: throughput and forwarding rate are different metrics.

Hope this helps.

Regards,
David Newman
Network Test
dnewman 12/4/2012 | 8:43:40 PM
re: Cisco Twists Test Results from the text:

This means the 12416 will forward traffic at up to 52 percent of line rate without loss. It does not mean the device will drop 48 percent of a load offered at wire speed

Regards,
David Newman
Network Test
chaochao 12/4/2012 | 8:43:40 PM
re: Cisco Twists Test Results Mr. Newman

From the OC192 route flapping graph. The max. Juniper is 5400000. The max for csco is 6000000. What gives? Different load to each box?

Thanks.

Chaonima
tink 12/4/2012 | 8:43:40 PM
re: Cisco Twists Test Results Chaochao,

The article states: "In fact, both Cisco and Juniper can forward 40-byte packets at rates well above 99 percent, albeit with some packet loss."

This means that both routers, when presented with a load of (probably) 100% line rate, drop less than 1% of the packets. The Cisco router drops some percentage (presumably somewhere around 1%) for all rates greater than 52%. For example, at an input rate of 90% of line rate, about 89% of the packets get through. Juniper's router drops nothing up to 92.2%. So if it gets 95% of line rate, 94% of the packets will get through.

And all of this is for 40-byte packets...

David, do you concur?

Thanks,
Tink
chaochao 12/4/2012 | 8:43:41 PM
re: Cisco Twists Test Results Mr. Newman

Did csco drop 48% of the packets on OC-192 test? Your number did not show. Please explain.

Sincerely
Chaonima
lightsource 12/4/2012 | 8:43:42 PM
re: Cisco Twists Test Results where does cisco - juniper stand when it comes to scalability of their routers ??
noptera01 12/4/2012 | 8:43:43 PM
re: Cisco Twists Test Results Just to keep things on course...

http://www.siliconinvestor.com...

http://quicken.excite.com/inve...

http://money.go.com/News?newsR...

http://cbs.marketwatch.com/too...

http://mktnews.nasdaq.com/news...\www\nasdaq\news\pr\2001\03\14\PR204451AT407.html&site=NASDAQ&sitesubtype=&usymbol=CSCO

http://ragingbull.lycos.com/mb...

jcrb 12/4/2012 | 8:43:44 PM
re: Cisco Twists Test Results >If you introduce scenarios that won't happen
>(like the network processor being stalled for
>10ms) then you can indeed fabricate all sorts
>of errors.

Gee, I just went and looked at the latency tests section of the report and in the MPLS over OC-192
test the M160 has an average latency of 90 micro-seconds, and a worst case latency of 40 milli-seconds..... so I appologize of the inaccuracy of my example, I should have said

"when one processor stalls for 40ms it can reorder packets...badly"

given that the latency tests were done with no loss, 40ms of latency is either an outright pause or shows that one of the packet processors can fall very far behind the other processors. In either case 40ms of latency variation is easily enough to reorder TCP packets in the real world

regards

jon
csco-grunt 12/4/2012 | 8:43:44 PM
re: Cisco Twists Test Results I wonder why more people didn't pick
on the benchmark being 40-byte packets.
Most csco equipment is designed for
benchmarking to 64-byte packets.
Without knowing specifics of the designs
involved I'll say that if you design to a
40-byte benchmark you're going to have to
sacrifice some feature or cost that the
customer won't like. No free lunch etc.

I also thought the most sensible post was
the suggestion that all of the test results
be posted and let the amateur analysts out
there post their own conclusions.

But the really most entertaining and telling
story was the one about David Newman telling
people that a test cisco won couldn't possibly
be correct :-) hee hee !
skeptic 12/4/2012 | 8:43:45 PM
re: Cisco Twists Test Results It seems this is an IMAGINARY issue for customers with REAL life network using M160.

If the issue affects the customers, they (Juniper and customers) are not gonna sit around. Customers are pretty well educated. They can tell what is IMAGINARY and what is REAL.

--------------------------

Internally Juniper knows it is a problem. The only people who don't think so are light reading and juniper's marketing department.

For competitive reasons, many vendors have compromised their fowarding hardware to meet schedules. In the rush to get OC-192 gear out, Juniper made a mistake. But its not a mistake they can't (probably) easily or quickly fix given their architecture. Hence the marketing department is sent out to convince the ignorant that its not a problem.

This sort of thing goes on all the time. There are vendors who are compromising MPLS performance in the future in order to do good numbers on 40-byte benchmarks today.

skeptic 12/4/2012 | 8:43:45 PM
re: Cisco Twists Test Results At OC-192 rates I don't think you're going to see packets from a single stream reordered. Ever. In the history of the world.

-----------------------------------------------

Every time link speeds have increased (going back a long time), someone has made this arguement. But it never holds up. The relative speed and capacity of OC-192 to other factors in the overall network is not going to stay constant. (just as it didn't at much lower speeds).

Reordering is just a bad idea. And equipment vendors should be discouraged from producing hardware that trades off benchmark performance for serious flaws like reordering.
xiaomingjia 12/4/2012 | 8:43:46 PM
re: Cisco Twists Test Results Sorry for being ignorant, I just have a few questions:

I can't help noticing that Cisco has won many editor's choice awards in David Newman's numerous articles. If David Newman is such a Cisco hater, why do they keep showing up for his tests? As far as I know, for any public test, the test methodology is made public prior to the tests, and vendors can pre-stage before the actual testing. No one is *forced* to participate.

I also noticed on some old DataComm pictures that the test site in Paris is on the top floor. How is it possible for him to *walk down the stairs* to the test area?

Curiously,

Xiaoming Jia
jcrb 12/4/2012 | 8:43:46 PM
re: Cisco Twists Test Results
>At OC-192 rates I don't think you're going to see
>packets from a single stream reordered. Ever.
>In the history of the world.

I can't imagine why you would say that, but if you want to put some money behind that statement I would be happy to take the other side of that bet.

>If you introduce scenarios that won't happen
>(like the network processor being stalled for
>10ms) then you can indeed fabricate all sorts
> of errors.

I introduced that just for the purpose of explaining the observed reordering, I didn't claim it did happen, but again I wonder why you say it dosen't? Either you work at Juniper and know such low level of detail to make this statement, or you are just making assertions without any facts to support them.

Ask youself this, do you *know* it doesn't happen? or do you just think/wish/hope it doesn't happen?

Since it is a fact that the M160 reorders packets, the only way that you can *know* it does not effect real TCP traffic in the internet is if you have performed tests in the real internet. Anything else is marketing and not science.

(Does anyone know where I can find an M160 in a live network to run some tests against????)

>The bottom line is that TCP streams are sourced
>from machines with either 100Mb or GigE
>interfaces. These are your web servers. These
>packets are spread out in time according to how
>fast TCP is transmitting them on the available
>media, throttled by the myriad of TCP controls.

As we point out in our paper, in the presence of packet reordering the inherently bursty nature of TCP transmission is *reenforced* by the reordering, so those "myriad of TCP controls" of which you speak help to make the problem worse not better.

However your next comment makes my point more clearly.

>To get to the OC-192 core, the packets have to go
>through site routers, campus routers, customer
>edge routers, provider edge routers and finally
>the OC-192 core router.

uuumm, How about the packets come out of the server, into the customers GigE switch, over the the PoP router and off into the core. If I am a sight with any serious bandwidth requirements I am going to want a direct highspeed connetion to my service provider. As an example of this I just did a traceroute to a site I would like to get good TCP performance to 'www.ophoto.com' here is the tail end of the trace (edited to fit)

15 so-2-2-2-155M.br1.WDC2.gblx.ne... [208.51.7
16 pos2-0-155M.cr1.WDC2.gblx.net [208.178.17
17 pos0-0-2488M.cr1.SNV3.gblx.net [64.211.14
18 pos0-0-2488M.hr1.SNV3.gblx.net [208.178.2
19 64.209.191.5
20 64.209.168.6
21 bor-clust01.ofoto.com [64.209.168.84]

The last 2-3 hops have the same IP prefix, so we can assume hop 20 is a local router at the server site and that hop 19 is probably the customer edge router. Since hop 18 has an OC48 link its a safe bet that hop 19 has the other side of the OC48 link (or is at least GigE, since you dont go putting 100M ports on an OC48 router) so what we have here is a server which is 1 or at most 2 hops (of either GigE or OC48) from the core routers.
Then there are co-located servers, and cache servers all of which then to be right there in the PoP with the routers. So there does not seem to be much to that arguement.

Saying that 'because its 0C-192 (10G)' it will not happen because its so fast ignores the fact that the servers all now have have GigE (1G) connections. As for the packets only spreading apart and not ever clumping together, I have to disagree, based on both studies of actual network traffic and the construction and behavior of modern routers. But this message is already long enough so I will leave that for some other time.

>Talk is cheap. Don't believe me and don't
>believe any of the other arguments here

Talk is only cheap when you have nothing of value to say.

regards

jon
puttputt 12/4/2012 | 8:43:46 PM
re: Cisco Twists Test Results Having to read through posts like this one is a waste of my time. This site is an invaluable resource for information and after reading a article like this one, I like to read through the posts to see what the opinions are of the other people in my field. Professionals. People who have valid opinions and information I can use to make me a more informed technician and smarter consumer.

People like you sound like children squacking over the last popcicle.

You have too much time on your hands and your wasting mine.
net_exprt 12/4/2012 | 8:43:48 PM
re: Cisco Twists Test Results read into this statement:


The press release not only misrepresents the overall conclusions of the test but also makes inaccurate statements about the results. In addition, it fails to mention that Light Reading commissioned the tests, which were carried out by Network Test Inc., and that the full results are published on this site (see Internet Core Router Test ).

The KEY point that LightReading is making is that cisco "fails to mention that LR commissioned the tests".....

Q. Is LR worried more about the outcome of the testings, OR, that a press release from Cisco that is seen throughout the world does not mention LR at all.....hmmmm....
Perhaps cisco can resend that press release and THANK LightReading for conducting these testings....could that mend relationships?

fiber_r_us 12/4/2012 | 8:43:48 PM
re: Cisco Twists Test Results Packet re-ordering is not an imaginary issue. The packet re-ordering was reported to Juniper by us large customers who were experiencing the problem in the early field deployments of the product. Our customers were calling us and asking why thier applications were failing. Only then was the re-ordering problem replicated in the labs. This happened over a year ago.

Once a large scale deployment of a product has occured, how do you fix it (short of tearing it all out)? We hardly sat around upon discovering the problem. We complained loudly about the re-ordering problem! However, in order for the problem to be fixed, a new box architecture for Juniper would be required and all of the installed systems would have to be swapped-out.

This problem was not sufficient to justify stopping the deployment, but it certainly created a network that didn't behave as intended with no good way of correcting it.

While all of the messages analyzing when and if re-ordering will occur are interesting intellectual fun (and are they same thought processes we went through about a year ago) the fact is that the platform does re-order packets in real-world situations. Not every application experiences it, but enough do that it causes problems.

As I stated before, and some people are coming to the same conclusion, the test results show a few certain things:

1) Juniper and Cisco are really the only useable boxes for core Internet routing at this point (i.e. it is currently a two-horse race).

2) Neither box overall completely dominates the other in the tests that were run (there is no "winner").

3) The market is better off that there are two worthy competitors. Hopefully more players will emerge.

4) In specific instances (ie. if you want packet filtering, or you want more than 8xOC192) one platform may be better than the other. This does not mean the one of them becomes the overall winner!

I suspect that most major networks will utilize both platforms (just like we did in ours).

hungerbug 12/4/2012 | 8:43:49 PM
re: Cisco Twists Test Results It seems this is an IMAGINARY issue for customers with REAL life network using M160.

If the issue affects the customers, they (Juniper and customers) are not gonna sit around. Customers are pretty well educated. They can tell what is IMAGINARY and what is REAL.

Admit I do own stocks in both and do not have experience with M160. Just aware that some of the largest customers using M160 worldwide.
jim_baldwin 12/4/2012 | 8:43:49 PM
re: Cisco Twists Test Results I'm sure most people reading this site appreciate the summary of the article in title. The fact is, the test conclusions were that Juniper won. The title was enough to make me read the article through, which apparently you haven't.

There have been so many questions on these boards about who funds LR and reporting bias. These questions come from people who work somewhere that comes up short in public opinion, as reported and discussed here.

Get your Cisco PR people off their keesters and come up with a compelling press release that demonstrates your true superiority, don't waste your time being defensive.

And this: "this is how it works in all USA", demonizing the US business community doesn't make any point here.
seekthetruth 12/4/2012 | 8:43:50 PM
re: Cisco Twists Test Results See, Light Reading, a sensational title gets you more hits, happy now ?
You really should post ALL your test results and let the users/readers decide, not spin them in favor of the Juniper guys who pay you. You have crossed the line from investigative journalism into tabloid "news". Your wit has degenerated into bias. Sad.
noptera01 12/4/2012 | 8:43:51 PM
re: Cisco Twists Test Results LR continues on a roll...

http://www.siliconinvestor.com...

http://quicken.excite.com/inve...

http://money.go.com/News?newsR...

http://cbs.marketwatch.com/too...

http://mktnews.nasdaq.com/news...\www\nasdaq\news\pr\2001\03\14\PR204451AT407.html&site=NASDAQ&sitesubtype=&usymbol=CSCO

http://ragingbull.lycos.com/mb...


james-carville 12/4/2012 | 8:43:51 PM
re: Cisco Twists Test Results This is reminiscent of the Clinton scandals.

Rule 1 - discredit the messenger - who cares about facts. Destroy those presenting the facts and the facts go away

Rule 2 - create your own facts. Get all your minions on the street spreading them. Soon no one knows what to believe. With your enemy destroyed by unfounded accusations, people believe you.
sntwk 12/4/2012 | 8:43:52 PM
re: Cisco Twists Test Results I am kind of curious whether the editorial tone with regards to the results will serve LR in any way. Will they get cooperation from Cisco and other vendors in the future?

Obviously the tests are of high standard. But at the same time the performance/features required by different users of Juniper or Cisco boxes may be far below what both can support. For example 400,000 routes? Does it really matter if you are going to support with 1GB memory a Billion routes when in fact the internet as of today doesn't have more than 100K routes?. Why would you put so much memory, increase power dissipation , super charge the box for nothing.

Also carriers who want to be Internet service providers many need port densities rather than just line rate on just 8 OC-192 ports.

It is clear that at 52% line rate for OC-192 interface, cisco is not really OC-192. It is just two OC-48. They have to accept the result. Also is 40 byet packets the appropriate load test. Are there other packet sizes more typical of internet usage. Random mix etc. If cisco is unhappy about the result they can come forward with information about the load tests they think are appropriate and reasonable for the real use scenario.

At the same time the results could have been published without providing editorial bias to avoid controversy.

Obviosly a test of this nature is bound to make unhappy those vendors who are not going to come on top in one test or the other.

So going forward will there be a formal consortium of some sort which provides testing service/framework. Vendors bring their boxes prepared for the testing and accept the results. If on the other hand the testing is more like that of consumer test reports or buyers' guide then it should also include the price and the recommended rating agains metrics such as price/performance etc. Why not? Don't we see that when we look at consumer report for handy-cam test report?

I am sure when you consider 26 M$ spent in this test the economics are going to be difficult to be worked out. How is LR going to recoup that money? If an independent consortium is to do the testing on an annual basis who is going to fund that effort? May be the Fed!

Why influence a service provider that they got to have Juniper/Cisco boxes just because they passed some tests and may be they have to pay 250K for these boxes and yet service providers might be fine with a cheaper box from another vendor? It is the hyped/image creating product information that is causing/forcing service providers to buy costly (over-priced, high gross margin) equipment and resulting in number of DLECs/CLECs folding/busting. This test result interpretation is definitely in that direction.


drag 12/4/2012 | 8:43:53 PM
re: Cisco Twists Test Results Big Mike,

I read your post with great interest and I have to admit it seems to make sense if you believe the OC192s are fully loaded and the traffic is many many small TCP streams that aren't really sensitive to reordering anyway, which for todays Internet is probably true. However, for networks designed for broadband content delivery (such as video and other attractive higher bandwidth stuff) the would not be the case. Remeber that holy grail will we ever get there Internet thing?

The point I would make is these networks will never deliver revenue or the promise of broadband services because some of higher bandwidth applications sensitive to reordering (which are usually the ones customers are willing to pay quite a lot for) simply don't work. Customer says IP can't deliver and looks back to good ole TDM. I personally don't like this.

Applications have always been constrained by technology - usually performance. It's a shame that the M160 delivers decent performance but leg-ropes some applications by misordering packets. I guess providers who feel the pain on these issues know where to look first...

d.

PS. sorry for the lack of controversy or use of !!!s or #$%[email protected]% in this post.
kingman 12/4/2012 | 8:43:54 PM
re: Cisco Twists Test Results I think the point is that why on one hand CSCO wants to tell people that their product was tested and compared w/ so many other products, on the other hand CSCO changes the result from that test. You see the conflict here? it's like saying "That test can be trusted, since I was in" AND at the same time saying that "but the result can't be trusted, so I make up mine."
As your ad. point, I think a "CSCO win" will generate far more attention than the current 'JNPR win", because this is not the first OC192 core router test, this is actually the 2nd. one. In the first test, there was NO competitor at all, the M160 was the winner. So it's not a news at all, if the 2nd. test repeats the old result.
bibiyahoo 12/4/2012 | 8:43:54 PM
re: Cisco Twists Test Results there is no other reason to why LR came out with "JUNIPER WINS" title.

there is no other reason why they published article accusing CISCO.

there is no other reason why their article's language is so in favor of juniper.

I call to someone investigating this issue.

what are really the connections between LR and JNPR?
how much JNPR payed LR? who owns and finance this site anyway?
how many JNPR stocks have LR people & their relatives?
will the LR people get nice jos in JNPR in the near future?
how much lunches & dinners were held on behalf of JUNIPER prior to this test publishings...

etc.
etc.
etc.

I guess juniper must work this way if she wants to beat cisco.

tought I dont doubt cisco is corrupt too.

this is how it works in all U.S.A !!!

play it nice guy , but throw nuclear weapon if it serves your goals.
haiyingdeng 12/4/2012 | 8:43:54 PM
re: Cisco Twists Test Results Who knows the background behind the whole thing, LR can get so many expensive routers from Cisco for test, they definitly could not buy it or borrow it. It got to be offerred by Cisco guys.

Then how did Cisco and LR communicated during this thing, maybe Cisco just asked LR to test line speed, instead LR tested BGP and Filtering. then let the David Newman guy to decide which packet size to measure throughput.
haiyingdeng 12/4/2012 | 8:43:54 PM
re: Cisco Twists Test Results Net_exprt:
This is the best testimony I saw during the whole debate.


Author: net_exprt Number: 75
Subject: david Newman's Obsession with cisco Date: 3/15/2001 1:29:28 AM


I remember David Newman from the days of the data communication testings....I was in paris for 2 weeks, and all I saw from him was unprofessional behavior and an anti-cisco attitude. I remember he once shouted at one of the Netcom guys claiming that they were pro-cisco...The cisco numbers were good, but he just kept saying that "it's impossible that the numbers are valid, test again". He would walk down the stairs into the testing area, make some comments against our testings, and go back up to his room.
Rajah 12/4/2012 | 8:43:55 PM
re: Cisco Twists Test Results net-exprt:

do you normally rebutt an argument by using phrases such as "Ha ha ha ha ha ha ha ha ha ha"? I truly thought that I was dealing with an adult. It's now fairly clear that you lack basic communication skills necessary to hold a converssation.

Rajah:

The word "Ha" or a multiple of them are used in th English language to denote laughter. Did you not study English? Most adults laugh at a number of things, including stupidity. "Ha" is used in conversation. Have you never laughed during a conversation?

net_exprt:
You displayed your ignorance by claiming that I am comparing a bakeoff to a Network test.

Rajah:

While we are at it, comparing "Magazine tests" to the Network Test tests is equally hilarious. Or did you mean something else by "Magazine tests"??

net_exprt:
that I'll just ignore you, .....

Rajah:
Yes, I agree, it's about time you run away little man....





chaochao 12/4/2012 | 8:43:55 PM
re: Cisco Twists Test Results Like I said before. You thought Tolly was bad. David Newman is equally bad!

Show us all the numbers! What are you afraid of?!
Tell us what other test you have not done! OIR is not important to you? Mr. Newman? Reduandency is not important to you? Mr. Newman? Maybe you can try this yourself, take an IP2 out on M160 and see what will happen. Hot swap a FPC and see what will happen.

You are bias and not trustworthy period.
net_exprt 12/4/2012 | 8:43:56 PM
re: Cisco Twists Test Results On the contrary....Saunders is a genius. He wants publicty. A controversial article creates press releases. The perfect proof that LR only cares about creating controversy is the following quote:


The press release not only misrepresents the overall conclusions of the test but also makes inaccurate statements about the results. In addition, it fails to mention that Light Reading commissioned the tests, which were carried out by Network Test Inc., and that the full results are published on this site (see Internet Core Router Test ).

They are angry that cisco did not mention them in their press release.

chuckjackson 12/4/2012 | 8:43:56 PM
re: Cisco Twists Test Results Rajah, read post #2.
net_exprt 12/4/2012 | 8:43:57 PM
re: Cisco Twists Test Results do you normally rebutt an argument by using phrases such as "Ha ha ha ha ha ha ha ha ha ha"? I truly thought that I was dealing with an adult. It's now fairly clear that you lack basic communication skills necessary to hold a converssation. You displayed your ignorance by claiming that I am comparing a bakeoff to a Network test. Do you know what a bakeoff is? I hope you don't think a bakeoff is something you do in a kitchen...Anyways, you've displayed so much ignorance in these last few posts, that I'll just ignore you, .....Another words, you don't need my help to make a fool out of yourself..you are doing a smashing job, as it is
Rajah 12/4/2012 | 8:43:57 PM
re: Cisco Twists Test Results net_exprt:

a simple mind like yours might not care if LR generates revenue,or uses controversial headlines.

Rajah:

It's not that I don't care. It's that that is not the issue that I wish to address or discuss presently, nor should it be brought up as an argument for the issue at hand of Cisco fudging the results. This issue of LR's revenue etc. has been discussed for a long, long time now. It's nothing new. What is new is this current Cisco issue. Light minds such as yourself who seem to see a valid argument for Cisco by pointing out LR's revenue, controversy, etc., should check to see if light passes between you ears. The issue at hand is simple, LR and Network test published test results and Cisco fudged them. Or did they? Or do they have a valid argument or reason? That's the issue at hand.


chaochao 12/4/2012 | 8:43:57 PM
re: Cisco Twists Test Results Mr Newman

Post "ALL" the numbers you have and let the readers decide!

Sincerely
chaonima
rmartin 12/4/2012 | 8:43:57 PM
re: Cisco Twists Test Results It's funny to hear LR whining about others twisting the truth. LR has twisted the truth many times in the past. It's time to start the Top 10 Twists of Truth from LR. Feel free to add your contribution.

Number 10: This one's quite ironic. It involves cisco. Light Reading posted an article claiming that Monterey was dead and that a switch was being built to replace it. The article got so much attention it was mentioned on Briefing.com and was circulated on the trading floors. Actually, the switch was being developed for another product and LR's claims were clearly off base. Shame on you Light Reading. I didn't know you could "twist" the truth like this. You seem "desperate" for stories. I didn't know you would "stoop so low".
ishot 12/4/2012 | 8:43:58 PM
re: Cisco Twists Test Results With LR's rebuttal today, their independence [and credibilty] just went out the door. Now it's obvious the test was rigged in junipers favor in the first place, and Stephen Saunders ought to be ashamed.
Rajah 12/4/2012 | 8:43:58 PM
re: Cisco Twists Test Results net-exprt:

aaaahhhh...I think I'm making progress with you, or am I?

Rajah:
uhhhh....No. Frankly, I think you have screw loose.

net-exprt:

If you've ever been involved with a bakeoff or magazine testings, then you wouldn't ask such a question. There are many factors that influence a testplan. A vendor might not necessarily be able to influence the creation of the testplan. Once again, and I think I've been clear with you, you have to look at what the testing site's MOTIVES are...somethings are very simple.

Rajah:

I've gotta tell you. That's the most ridiculous argument I've heard or read in a long while. You are comparing bakeoffs to the Networktest tests??? Ha ha ha ha ha ha ha ha ha ha. I now see why everything seems so simple to you. Are you telling me that an 800 pound Gorilla like Cisco would participate in such tests of their own equipment as if they were bakeoffs? Ha ha ha ha ha ha

I now see why it all seems so simple to you. Simple minds for simple things as they say.



net_exprt 12/4/2012 | 8:43:58 PM
re: Cisco Twists Test Results a simple mind like yours might not care if LR generates revenue,or uses controversial headlines, but then again, that's what seperates those who understand the motives behind those who publish those results, and others like yourself, who are so easily fed information. That's why I suggest that LightReading changes their name to LightMinds.
net_exprt 12/4/2012 | 8:43:59 PM
re: Cisco Twists Test Results aaaahhhh...I think I'm making progress with you, or am I?

"Why didn't Cisco propose tests that would emphasize their strenghts and JNPR's weaknesses before entering into these tests? "

If you've ever been involved with a bakeoff or magazine testings, then you wouldn't ask such a question. There are many factors that influence a testplan. A vendor might not necessarily be able to influence the creation of the testplan. Once again, and I think I've been clear with you, you have to look at what the testing site's MOTIVES are...somethings are very simple
Rajah 12/4/2012 | 8:43:59 PM
re: Cisco Twists Test Results net_exprt:

Rajah...there you go again...making a fool out of yourself. Do you understand how advertisment rates are determined. Let me educate you, little boy...

Rajah:

Oh please!!! Enough with that s---, you're like a broken record. Let's get something straight, I don't care if LR wants to make money out of this or not. Nor do I care if it wants to generate controversy, increase hits, advertisments, or recycle bins. That's not the issue, little man. The fact is that LR published the test results which are also affirmed my NetworkTest. Cisco fudged the results in their press release. That's what I care about, not if LR or your grandma is going to make money out of this or what their ulterior agendas or motivations are.

Understanding being????

net_exprt 12/4/2012 | 8:43:59 PM
re: Cisco Twists Test Results Rajah...there you go again...making a fool out of yourself. Do you understand how advertisment rates are determined. Let me educate you, little boy... This from LR:

"General Advertising Opportunities

Light Reading offers numerous advertising options, including banner advertisements, e-mail newsletter sponsorships, white paper postings, and sponsored content areas. Your marketing message can be targeted specifically to the reader base for Light Reading (general content), Light Trading (finance), or Light Work (recruitment). These content areas and newsletters reach more than half a million unique visitors per month."

So, banners are placed on websites for readers to click on. As you click banners, the original site, in this case, LR, receives revenue. It's about how many Unique readers you can pull into your website. That's the bottom line...So, Rajah, somethings are very simple...get it?
Rajah 12/4/2012 | 8:44:00 PM
re: Cisco Twists Test Results
net-exprt:

Sensationalism is part of their game.....the title "juniper....

Rajah:

The same BS over and over again. Get a life. You may very well be right, but that's not the issue.

net-exprt:

so, if I'm LR, I make sure we create a testplan that hightlights the strengths of Juniper and not shed light on Juniper's weakness (OIR, High Availibility, etc....)

Rajah:
Why didn't Cisco propose tests that would emphasize their strenghts and JNPR's weaknesses before entering into these tests?

Rajah 12/4/2012 | 8:44:00 PM
re: Cisco Twists Test Results net_exprt:

But, Dr....think of all the free press they received by claiming that the smaller guy beat the 800 pound gorilla...it brings .........

Rajah:

Either you are just being paranoid, or a you are Cisco employee:-). Assuming LR did not name a winner and yet posted those results. Even if it seems head-to-head, some if not most people will take JNPR to have outperformed Cisco. That all aside, the basic issue is why did Cisco fudge the results themselves? Whether LR wants to make money and controversy is not the issue, so stop harping on it.
net_exprt 12/4/2012 | 8:44:00 PM
re: Cisco Twists Test Results But, Dr....think of all the free press they received by claiming that the smaller guy beat the 800 pound gorilla...it brings viewers to the site...it makes lightreading famous....and that's what its all about....LR couldn't care less who won...their objective is to make money, and they have put their name on alot of press releases...it's about advertisment dollars...people flood into this site to read the "controversial" aritcles...Redback has to pay more dollars
Rajah 12/4/2012 | 8:44:01 PM
re: Cisco Twists Test Results Sparky:

It is safe to say that those who wished to see the M160 succeed engineered this test. The proof is in the test itself and the editorial comments by the author.

Rajah:
Cisco had knowledge of the tests being performed. Your argument is baseless.

Sparky:
With so much invested in equipment and time why not perform a truly comprehensive test, on core routing features such as OIR, fully loaded box, QoS latency, capacity and redundancy.

Rajah:
Why didn't Cisco propose such tests prior to entering the tests?

Sparky:
In my humble opinion, Cisco has always been a winner.

Rajah:
Not after this. The basic issue is why did they twist the results? If they didn't agree with them, why didn't they just say so?

Rajah:
net_exprt 12/4/2012 | 8:44:01 PM
re: Cisco Twists Test Results I remember David Newman from the days of the data communication testings....I was in paris for 2 weeks, and all I saw from him was unprofessional behavior and an anti-cisco attitude. I remember he once shouted at one of the Netcom guys claiming that they were pro-cisco...The cisco numbers were good, but he just kept saying that "it's impossible that the numbers are valid, test again". He would walk down the stairs into the testing area, make some comments against our testings, and go back up to his room
Rajah 12/4/2012 | 8:44:01 PM
re: Cisco Twists Test Results
Your solution does not have any foundation. Who are you to come up or even state that there is a problem that is to be solved regarding the determination of a winner? The fact is that Cisco knew very well that LR was commisioning the tests. (You don't actually believe Lang's BS do you? If Lang is serious then there are some serious holes in the Cisco organization.) Since LR commisioned the tests, LR has every right to interpret the results and determine a winner(in their opinion) whichever way they want to. Even if Cisco outperformed JNPR in every test and LR still named JNPR the winner, that's their buisness because it was they who commisioned the tests. You and I and the whole world may disagree, and LR's winner may not match the results by any stretch of the imagination, but that doesn't matter. In any form of testing, the tested parties submit to the testers for evaluation. The testers determine a pass/fail/winner/loser etc. But the testers' judgement is final because it is they who performed the tests. That said, LR listed the results aswell, so its not like they don't show the backing for their decision. Regarding polls, Cisco employees, friends and family would flood the poll. So would JNPR. You will never get an unbiased poll.

Rajah.
net_exprt 12/4/2012 | 8:44:02 PM
re: Cisco Twists Test Results Sensationalism is part of their game.....the title "juniper wins MONSTER router test" implies that Juniper won every, every, single item on the test plan.....Ofcourse, even the hardcore juniper fans cannot claim that with a straight face. So ask yourself the following questions:

Q. How does LR generate revenue?
A. Advertisment
Q. How does LR charge its sponsers?
A. Number of hits, (unique IP sessions)
Q. How does LR increase the hits
A. Controversial, sensational, David vs. Goliath
articles
and now the 64 million dollar question
Q. which one creates more controversy and sensationalism:

Juniper wins Monster router test!!!!! OR
Cisco wins Monster router test!!!!

aha.....

so, if I'm LR, I make sure we create a testplan that hightlights the strengths of Juniper and not shed light on Juniper's weakness (OIR, High Availibility, etc....)

oh, customers...always remember, every body, including LR, has their own motives. It all comes down to Money...Money...and more money...LightReading has turned its audience into Lightminds
Rajah 12/4/2012 | 8:44:02 PM
re: Cisco Twists Test Results
Sparky says:
---
"These two win the prize for pre-schoolers with the best job."

Translation...
"I didn't think Cisco would refute my BS editorial."

"I didn't think that Cisco would point out their strengths and how meaningless some of the tests were."
---
Why did Cisco enter the tests in the first place? Did they not know that the tests were commisioned by LR? Or was Lang really serious when he said the following:
GǣI guess I was confused,Gǥ he says. GǣThereGs a difference over whose test it is and who did the test.Gǥ

Sounds like Kindergarden babble to me. Did you two go to school together Sparky?

Rajah.



net_exprt 12/4/2012 | 8:44:03 PM
re: Cisco Twists Test Results actually, they haven't, but I did get you to open this message. This is a clear example of how LR increases the number of readers.... send those sensational headlines "Juniper wins monster router test", and we'll get readers, and most importantly, higher number of HITS. We can then turn around and charge our sponsers more advertisment money...
LightReading should change their name to LightMinds. LR and newman...shame on you for stooping this low just to gain readership.
BigMikeJ 12/4/2012 | 8:44:03 PM
re: Cisco Twists Test Results
At OC-192 rates I don't think you're going to see packets from a single stream reordered. Ever. In the history of the world.

If you introduce scenarios that won't happen (like the network processor being stalled for 10ms) then you can indeed fabricate all sorts of errors.

The bottom line is that TCP streams are sourced from machines with either 100Mb or GigE interfaces. These are your web servers. These packets are spread out in time according to how fast TCP is transmitting them on the available media, throttled by the myriad of TCP controls.

To get to the OC-192 core, the packets have to go through site routers, campus routers, customer edge routers, provider edge routers and finally the OC-192 core router.

At each transition the outgoing traffic stream is going to be a queued mix of packets from all over the place, potentially displacing the packets in time even further.

Unless the latency through the box is in the 10s of milliseconds, I just can't see how in the real world this is ever going to happen.

So, my stance on this is:

1. Reordering packets within a flow is a bad thing.
2. At OC-192 core speeds, packets within a flow are going to be so displaced in time that this is not going to happen.

The clumping you mention in a backlogged FIFO is also not going to happen. The TCP segments are separated by many many packets, and so transmission rate doesn't matter.


I think this highlights the difference between a lab test of unrealistic expectations and a product designed for the real world - something we (in the networking industry as a whole) are always battling against. How much development effort and money do we want to spend to pass a dumb test that means nothing in the real world but currently makes competitor X look good. Sigh.


Talk is cheap. Don't believe me and don't believe any of the other arguments here. Don't believe pictures of packets on a going out of point A and arriving at a FIFO at point B, bunching up, hailing a cab, and taking a trip to Pittsburgh. It's all mental masturbation, football fan mania and marketting spin. It's politics. Keep talking because the only thing they remember is the guy who had the last word.

I digress.

Someone suggested doing a TCP test to see what happens. I agree. Do the test - but put at least one aggregation router before the OC-192 core, and source the TCP packets from a real host - not a SmartBits running full steam. Again, only a realistic test is meaningful when talking about stream performance.
RadioGooGoo 12/4/2012 | 8:44:04 PM
re: Cisco Twists Test Results It's like the pot calling the kettle black...sheesh! Lightreading correctly chose its name; its articles should be taken "lightly".

Lightreading is guilty (many times over) of what it's accusing Cisco of doing (just go to the Corvis board and ask them). Now, LR is crying. How lame.
Dr. Dre 12/4/2012 | 8:44:04 PM
re: Cisco Twists Test Results Wow this is scary! Based on the number of press releases posted around the web, Lightreading.com is not backing down from the 800 pound gorilla (Cisco).

I think I should say a few words in hopes of preventing an all out war!

I think there are some lessons to be learned from all of this:

1) It was probably better for Lightreading.com to present the results without declaring a winner. Here's why:

The outcome of these tests are very important to investors, customers and employees and will be looked upon with great interest. To exacerbate the situation, we are dealing with the reputation of multi-billion dollar corporations who stand to gain/lose alot from any negative/positive news.

I read the tests carefully, and for the most part they check out o.k. with me (packet re-ordering issues aside). However, like many others have iterated (and re- iterated over and over again) I think the results do not show a clear winner.

Therefore, the most wise thing to do is this:
Present the individual tests, and comment on the results. Do not declare an overall winner on your own. If you would like to declare a winner, take a poll amongst the readers, and ask them to vote. Then, we can arrive at a consensus, which will reflect the opinions of the general public, and hopefully wont be subject to so much scrutiny.

It seems that there is just too much at stake here.

--Dr. Dre
alexis_phd 12/4/2012 | 8:44:05 PM
re: Cisco Twists Test Results Having used both vendors products, I would have liked to see line-rate filtering examined as well as software stability. I suppose Cisco would claim that an unfair test since they would lose again. Misinformation continues from the marketing machine -- the reason we are replacing Cisco with Juniper is because Juniper performs as Cisco continues to talk.
MHA 12/4/2012 | 8:44:05 PM
re: Cisco Twists Test Results
Only TIME will tell who really won.

No one will base his/her buying decision solely
on this disputed article PERIOD.
noptera01 12/4/2012 | 8:44:06 PM
re: Cisco Twists Test Results And the stories mount up...

http://cbs.marketwatch.com/too...

http://mktnews.nasdaq.com/news...\www\nasdaq\news\pr\2001\03\14\PR204451AT407.html&site=NASDAQ&sitesubtype=&usymbol=CSCO

http://ragingbull.lycos.com/mb...





jcrb 12/4/2012 | 8:44:07 PM
re: Cisco Twists Test Results >say I have a TCP stream 1,2,3,4,5,6,......
>
>Before entering the core, any flow must traverse
>other smaller routers.Given some form of round
> robin scheduling is used there ( in the smaller
>routers),

round-robin??? I think you are giving most routers too much credit here.. :-)
but I will humor you since it doesn't effect the answer

>will the TCP segments still be clustered
>( back to back )won't it look something like

> 6 5 * * 4 * * * 3 * * 2 * * 1

the TCP packets don't have to be back to back, I was just drawing them like that to save space. Even if the packets do spread out they can also clump up when they run into a backlogged FIFO queue in a router, when the packets finally get transmitted they will go out spaced at the output link rate even if they came in spaced out.

>now, the stream 6 5 * * 4 * * * 3 * * 2 * * 1
>is fed into a core router.

> if this is the case, if a "*" and the packet of
> my flow is rearranged, it make no difference to
> me. it makes a diff only if packets of my flow
> are interchanged. is this scenario anywhere
> close to reality?

When I said that you get the stream 1 2 3 4 5 6 7 8 in and you get
1 6 7 8 2 3 4 5 out
I was trying to make the example fit, what you would really be getting is

1 (100-1000's of other packets) 2 (100-K's of others packets) 3 GǪetc
and at the output you would be getting
1 (100-K's of OP) 6 (100-K's of OP) 7 GǪ.etc

this is why I said its best to think of reordering as how far in time, not how far in packets, for example suppose that there is something that causes one of Junipers packet processors to stop processing packets for 10 ms (im not saying there is, I just need an example) then when it started processing packets again their would be a queue of packets that came in during those 10 ms waiting to be processed, so if packets 2 3 4 & 5 were stuck in that queue, and packets 6 7 & 8 had be sent to a different processor, then as long as packets 6 7 & 8 arrived less than 10 ms behind packet 2, then they would leave ahead of packets 2 3 4 & 5 REGARDLESS of how many '*' packets there were between them. This is different than the kind of reordering you might see from a network processor with multiple engines that would tend to reorder only packets that were basically back-to-back.


> does packet reordering happen under all
> conditions i.e. when routers were fully
> loaded or otherwise

if the router exhibits reordering for the type of reasons we discuss in the paper, it will tend to happen more under load (see our paper) it doesn't have to be fully loaded for these effects to be seen.

I suppose its worth commenting that we did predict that people would continue to build routers that re-ordered packets, despite the obvious trouble it causesGǪGǪ. Sometimes you find yourself wishing you were not in a position to say "We told you so"GǪ..

regards

jon
soothaandi 12/4/2012 | 8:44:07 PM
re: Cisco Twists Test Results "( more hits == more revenue ) burst the .com bubble.

no reader here buys lasers or routers or whatever from LR.

more readers==more revenue doesn't make any sense.

try something else next time."

Actually LR is not selling lasers or routers anyway. So your point, although well stated, is not applicable to LR.

Like the TV news guys' "5 products in your kitchen that could kill your children, more at 11", they are selling advertising and more eyeballs = higher "ratings" and it does mean more revenue for LR. It does make sense.

Whether LR is really doing that of course is a different issue.
ttsmith 12/4/2012 | 8:44:07 PM
re: Cisco Twists Test Results Send one video stream like this and all you will
see is garbage and i heard that is what you see on the M160. Doesn't make any sense how can Juniper win like this, unless they had their own
people doing and conducting the tests.

chaochao 12/4/2012 | 8:44:09 PM
re: Cisco Twists Test Results I thought Tolly was bad. Mr. Newman is about the same. They just sings for different vendors!

Wait, Mr. Newman needs a pay check too.
chaochao 12/4/2012 | 8:44:10 PM
re: Cisco Twists Test Results Smartbits. Why not Agilent? Another David Newman's master piece.

optical_IP 12/4/2012 | 8:44:10 PM
re: Cisco Twists Test Results ( more hits == more revenue ) burst the .com bubble.

no reader here buys lasers or routers or whatever from LR.

more readers==more revenue doesn't make any sense.

try something else next time.

optical_IP
chaochao 12/4/2012 | 8:44:11 PM
re: Cisco Twists Test Results David Newman used to work for Datacom before he went solo/Datacom went under. Now Datacom guys started LR. hrmmm....

Chaodnlrnima
net_exprt 12/4/2012 | 8:44:12 PM
re: Cisco Twists Test Results could it be possible that naming Juniper as the winner brings in more readers? It all comes down to the number of readers on this site...more readers means higher advertisment dollars...LR, shame on you.
optical_IP 12/4/2012 | 8:44:13 PM
re: Cisco Twists Test Results Diagram from the last post

~~~~~~~~~~~~|!!!!!!|
* * * * * *-|!!!!!!|-6 5 * * 4 * * * 3 * * 2 * * 1
6 5 4 3 2 1-|router|-* * ......
* * * * * *-|!!!!!!|-* * ......
~~~~~~~~~~~~|!!!!!!|


I am expremely sorry. I did it again .

regards

optical_IP
optical_IP 12/4/2012 | 8:44:13 PM
re: Cisco Twists Test Results say I have a TCP stream 1,2,3,4,5,6,......

Before entering the core, any flow must traverse other smaller routers. Given some form of round robin scheduling is used there ( in the smaller routers), will the TCP segments still be clustered ( back to back )

won't it look something like

6 5 * * 4 * * * 3 * * 2 * * 1

by the time it exits the smaller routers ?
(where * represents IP packets from other flows)

|!!!!!!|
* * * * * *-|!!!!!!|-6 5 * * 4 * * * 3 * * 2 * * 1
6 5 4 3 2 1-|router|-* * ......
* * * * * *-|!!!!!!|-* * ......
|!!!!!!|


now, the stream 6 5 * * 4 * * * 3 * * 2 * * 1 is fed into a core router.

if this is the case, if a "*" and the packet of my flow is rearranged, it make no difference to me.
it makes a diff only if packets of my flow are interchanged.

is this scenario anywhere close to reality?

if this is true, running a TCP connection directly across a core router will not be a realistic scenario.

does packet reordering happen under all conditions i.e. when routers were fully loaded or otherwise ?

sorry about the last posting, the text editor messed up the diagram.

regards

optical_IP
jperser 12/4/2012 | 8:44:13 PM
re: Cisco Twists Test Results In any contest, the contestant can not declare themselves the winners. In any sporting event, auto races, game show, boxing match, or the Olympics, you have people whose job it is to declare the winner. These people are called judges and referees. In this contest, the judges were Steve Saunders and David Newman. These are the only people who can rightfully declare a winner.

Declaring a winner is not easy. Look at the numbers. Both Cisco and Juniper performed well. In the 70 items listed on the "Results in detail", Cisco or Juniper score number 1 for each item. So how do you pick a winner? The editors had to made a decision. Whether you agree with then or not, "the decision of the judges are final." In Cisco's press release, Rob Redford of Cisco stated "It is gratifying to have the impartial judges at Network TestGǪ"

Cisco *can not* claim they "won every category". What they can claim is that they out-performed Juniper on 7 to the 8 IP throughput tests. They can claim they went toe-to-toe with their competitors. They can claim they demonstrated 100% line rate.

Jerry Perser

pablo 12/4/2012 | 8:44:14 PM
re: Cisco Twists Test Results > ... I'm disappointed that Cisco's now tried a
> dose of marketing BS ...

Poor marketing from a second-rate marketeer, indeed. I do think it shows some people in Cisco are losing their heads. They went for a knee-jerk, silly reaction. The more elegant spin would be to gracefully accept the second rank in this particular series of tests, which in their opinion of course do not represent the ultimate in reflecting true operational requirements with carriers. Sort of say that the selected tests favored the Juniper platform by their definition, but the fact the GSR held its own so well speaks for the architectures versatility, robustness and universality yadda-dadda...

It seems the GSR marketing team has had some poor additions or replacements as of late. They used to be sharper than this press release shows.
optical_IP 12/4/2012 | 8:44:14 PM
re: Cisco Twists Test Results say I have a TCP stream 1,2,3,4,5,6,......

Before entering the core, any flow must traverse other smaller routers. Given some form of round robin scheduling is used there ( in the smaller routers), will the TCP segments still be clustered ( back to back )

won't it look something like

6 5 * * 4 * * * 3 * * 2 * * 1

by the time it exits the smaller routers ?
(where * represents IP packets from other flows)

|------|
* * * * * *-| |-6 5 * * 4 * * * 3 * * 2 * * 1
6 5 4 3 2 1-| R |-* * ......
* * * * * *-| |-* * ......
|------|


now, the stream 6 5 * * 4 * * * 3 * * 2 * * 1 is fed into a core router.

if this is the case, if a "*" and the packet of my flow is rearranged, it make no difference to me.
it makes a diff only if packets of my flow are interchanged.

is this scenario anywhere close to reality?

if this is true, running a TCP connection directly across a core router will not be a realistic scenario.

does packet reordering happen under all conditions i.e. when routers were fully loaded or otherwise ?

regards

optical_IP
net_exprt 12/4/2012 | 8:44:15 PM
re: Cisco Twists Test Results actually, they haven't, but I did get you to open this message. This is a clear example of how LR increases the number of readers.... send those sensational headlines "Juniper wins monster router test", and we'll get readers, and most importantly, higher number of HITS. We can then turn around and charge our sponsers more advertisment money...
LightReading should change their name to LightMinds. LR and newman...shame on you for stooping this low just to gain readership.
skeptic 12/4/2012 | 8:44:15 PM
re: Cisco Twists Test Results 3) Stress BGP/MPLS/routing table size, because you know it's got more RAM and a disk drive.

-------------------------------

And in the case of BGP, avoid realistic large BGP table tests in favor of a test that tries to put every route in the forwarding table. Ignore the important test, focus on an insane test that favors one vendor. And highlight a massive numerical result by that vendor where the routes are not even being used correctly.



4) Stress filtering as a crucial "winning point" when Cisco entered the contest not able to do it. Whoopee - JNPR beat Foundry in OC-48...

-------------------------------
Further, confuse a service provider into thinking that you are testing route filtering (rather than packet filtering) to get him to make a really negative comment about cisco which he probably didn't mean.


D&K 12/4/2012 | 8:44:16 PM
re: Cisco Twists Test Results I have looked at all of it and I don't see bias. I do see a whole lot of people crying foul over nothing.

Read a little more carefully, fellers. Cisco pulled the same crap they have been getting away with for years. It is surprising that they agreed to test at all considering the multitude of tests they have refused to participate in over the last five years.

I suppose if LR had chosen Cisco as the "winner" here that we wouldn't be watching a group of seemingly adult life forms slinging stones at a fairly decent source of information.

Sometimes I wonder about folks that whine like this; probably the same group that has to get a look over the wall on 101 during rush hour when a car is lying upside down over there, while the rest of us try to get by.

As for facts, packet order has always been an issue, nobody has done very well on that one to this point, even Cisco.

Hot swap. let's see, blown up a few Cisco boxes doing that.

Route table size - need sideboards for that one, can't seem to catch up.

ACL's; Cisco has had that for quite some time and still sells the legacy routers in large numbers because folks WANT this feature.

Cisco's article is not sound. Face it.

If you want Cisco so $%$##@ bad, go by it. I see lots of folks every day and week that have been buying the Cisco tag line and pouring money into those pockets. The scary thing is, over the last three months they're finally tired of the lies. Juniper has market share because people wanted an alternative to a company that is more interested in lining the pockets of the shareholder than solving real world problems. Like most of the vendors these days, that doesn't lead to any real progress.

Take a close look at how long Cisco has been working on the 12k line and compare it to those that have not had so long to do it.

Kudos to Juniper to have this much. I say "Right on" LR, especially facing a crowd that knows more about everything than I ever will.

Crawl back in you holes fellers, lessin your willing to build a better box, you all know so much about it.

Never seen so much bullshit in my life, including my own.

Please disregard this and go back to your whining.
grinch 12/4/2012 | 8:44:17 PM
re: Cisco Twists Test Results Yeah, the test was clearly biased. I know David Newman thinks it isn't because Cisco engineers were there, but it is.

Let's see:
1) Packet reordering is an issue, and LR didn't test a real-life packet stream to see what that defect would do to, say, a video feed.

2) No Hot Swap - hrm, heard that's an issue with JNPR. Thought that could be important, somehow.

3) Stress BGP/MPLS/routing table size, because you know it's got more RAM and a disk drive.

4) Stress filtering as a crucial "winning point" when Cisco entered the contest not able to do it. Whoopee - JNPR beat Foundry in OC-48...

Face it, LR - you like the David vs. Goliath battles, and now you've picked your fight. Cisco's article is sound, and if you read your own report you contradict yourself over and over.
Peter Heywood 12/4/2012 | 8:44:18 PM
re: Cisco Twists Test Results Good night, light_on_dude, whoever you are.

Seeing as you sort-of-ask ask, this is what I sort-of-think.

Light Reading, Network Test and Spirent poured an enormous amount of money, time and effort into this project. The vendors involved also dedicated a lot of resources to it.

We didn't do it for a cheap thrill. Everybody involved in the test program, including Cisco, went into it thinking that the results would be useful and would help focus discussions on realities rather than marketing BS.

I think we achieved this goal.

I'm disappointed that Cisco's now tried a dose of marketing BS to mask the fact that it came a close second to Juniper when you look at the overall results.

I also think that we owe it to Juniper to set the record straight.
jcrb 12/4/2012 | 8:44:18 PM
re: Cisco Twists Test Results
I co-wrote the packet reordering paper cited in the initial test, as well as by both Cisco and Juniper and wanted to make a few comments about it. I posted this under the original discussion but wanted to add some comments so I am reposting it here.

From what I can see, most of Junipers spin on our paper is incorrect.

a) we said 90% of TCP *session* would see reordering, not 90% of*packets* (although for some sessions in our tests it was close to 90% of packets)

b) packets dont get reordered by being 'swaped' i.e. given packets
1 2 3 4 5 6 7 8
you dont usualy see
1 3 2 4 5 6 7 8
what you see is
1 2 6 7 8 3 4 5
or you see
1 3 5 7 2 4 6 8
what happens is that some form of load balanceing is going on and one path is faster or the other path stalls for a moment and all the packets on one path jump ahead of the packets on the other path.

As for the likelyhood of any given out of order packets belonging to the same TCP flow that depends on how far out of order a packet can get. If for example one path can stall for 20 miliseconds (its better to think of reodering as a time delay than a packet exchange, since thats usualy how it was caused in the first place) or is just a little bit slower and has a queue of packets to be processed that is 20 milliseconds long then any TCP flow sending more than 500 packets per seconds would be guaranteed to see reordering. The size of the link is not an issue nor is the number of flows sharing the link.

One way to apreciate this is that we did the testing for our paper by sending single flows of packets through the biggest exchange point we could find at the time to get our results. We were not measureing a device in the lab and looking at the ordering of all packets, just the ordering of our lonely stream of packets sent out into the big bad internet.

A simple way to settle the issue is if LR or anyone else with an M160 to test can just run a TCP connection across the router while jamming it with test traffic and see what happens. (to the LR editors, I might still have my test code around somewhere if you wanted to use it.....)

Oh and the bit about the next router down the line being likely to put them back in order....... the guy with the comment about smashing the pot twice and hoping to get an unbroken pot the second time had that one covered perfectly.

jon
Fred Snarff 12/4/2012 | 8:44:19 PM
re: Cisco Twists Test Results we'll see. I'm unbiased on this one - I could see that at least they had a FEW good things to say about them in the past, even if it seemed the majority was negative. I'm predicting now we will see blatant omissions of all things positive regarding cisco and intense focus on anything negative.
tink 12/4/2012 | 8:44:19 PM
re: Cisco Twists Test Results Despite the fact that we're always hungry for more tests and more data, we seem to be losing track of the fact that the person who has to make a buying decision is the real winner in this case.

We now know:

1) that Juniper's M160 converges fast

2) that Juniper's M160 does not forward at "line rate"

3) that Foundry and Charlotte's Web are not yet in the running

4) that Cisco's current offering doesn't filter

We still don't know everything, but we know more this week than we did last week. Forward...

Tink

light_on_dude 12/4/2012 | 8:44:20 PM
re: Cisco Twists Test Results Peter, All I can say is now you know how it feels to be a small start up with great ideas and products..and what it's like to try to go up against Cisco, Nortel et all...You think you make an inroad...and BANG...they slam yer arse against the wall. Maybe you can remember how that feels BEFORE you go back to your knee jerk, gossipy style. Get your chin off your desk Mr. Heywood...you're drooling! You really thought this "Independent test" was going to nail cisco didn't you? You think they are that frail? WRONG...Everyone be warned...they are very, very nervous over there...and they sure as hell won't go down easy. Optical start ups unite!! Give cisco a fright!!
Light on DUDES!!!
psumner 12/4/2012 | 8:44:20 PM
re: Cisco Twists Test Results Finally, somebody nailed Cisco for making false and inaccurate claims. End users of networking gear have endured their BS for years. Good job!
xerxes98 12/4/2012 | 8:44:21 PM
re: Cisco Twists Test Results Maybe if the editorial about the results started off something like the following, there may have been less controversy:

"In the 34 head-to-head tests where Cisco and Juniper competed directly, Cisco came out on top. Cisco was able to produce line rate in 7 out of 8 IP / MPLS tests using OC-48 and OC-192, whereas Juniper was able to produce line rate in 0 out of those same 8 tests. However, Juniper was able to win all tests in which it was either the only vendor competing or when it was competing solely against Foundry."

Any comments?
go_csco 12/4/2012 | 8:44:21 PM
re: Cisco Twists Test Results "...clearly, now we can look for all anti-cisco "reporting". before, there wasn't a clear motive. now the motive is there and obvious. it will be interesting to see how LR covers CSCO going forward..."

Can't get much worse, or can it?.......

go_csco!!

net_exprt 12/4/2012 | 8:44:22 PM
re: Cisco Twists Test Results "Desperation seems to be setting in at Cisco Systems Inc."

This comment is exteemly childish. Is this LightReading vs. Cisco systems? Or are you an independent site conducting meaningful testings for the benefit of the customers. If you are biased, at least be a little more subtle :-)

Fred Snarff 12/4/2012 | 8:44:22 PM
re: Cisco Twists Test Results clearly, now we can look for all anti-cisco "reporting". before, there wasn't a clear motive. now the motive is there and obvious. it will be interesting to see how LR covers CSCO going forward.

Fred
decimail 12/4/2012 | 8:44:22 PM
re: Cisco Twists Test Results Both Cisco & Juniper are very good Engineering firms. The test(s) appear fair. Why didn't anybody say anything prior to conducting the test? Lightreading doesn't really post B.S but industry information. I sorry to hear that Cisco has to resort to this...especially when it's still number one.
MHA 12/4/2012 | 8:44:24 PM
re: Cisco Twists Test Results Juniper badly fails in OIR and fully loaded
box, so one wonders why this was not part of
test.

How a FULLY loaded router performs is CRITICAL
and IMPORTANT to know.

Any comments from LR and folks who did this test ?
objective 12/4/2012 | 8:44:24 PM
re: Cisco Twists Test Results
Fred Snarff 12/4/2012 | 8:44:25 PM
re: Cisco Twists Test Results word up to photonic and light_on_dude.

I must say seeing this little pissing match between LR an Cisco is QUITE amusing. peotic justice, irony, and some good old fashioned payback all rolled into one - you gotta love it.

Fred


>>After accusing all us readers of being cry babies, it is really funny to see Light Reading editors turn into one themselves.

>>I love it.

>>Hey Peter, how does it feel to have YOUR words twisted? sounds like Cisco did to Lightreading, what Lightreading usually does to vendors...hows it feel to be twisted?? LOL!! (please don't say i work for cisco...I still hate them as much as ever) It's just damn funny to see this turnabout!!!
Sparky 12/4/2012 | 8:44:25 PM
re: Cisco Twists Test Results These two win the prize for pre-schoolers with the best job.

"I'm shocked," said Stephen Saunders, co-founder and US Editor of LightReading. "I didn't think Cisco would stoop this low."

"I'm very disappointed in this misrepresentation," said David Newman, president of Network Test -- also quoted in Cisco's release. "Cisco's 12416 put up some very, very good numbers in this test, so the company had no need to spin it the way they did."

Translation...
"I didn't think Cisco would refute my BS editorial."

"I didn't think that Cisco would point out their strengths and how meaningless some of the tests were."

Sparky
decimail 12/4/2012 | 8:44:25 PM
re: Cisco Twists Test Results This test was commissioned by "Networks test" whom conducted a very fair test. Both boxes are excellent however Juniper is probably more suitable for the core. I expect Juniper to release it's M320 this year which may increase it's market share to 50% - 55%. This doesn't hurt Cisco in anyway due to it's product portfolio. For example Juniper doesn't sell transmission equipment. Plus I think our business community is better served with two very good vendors. Do we really want a monolopy ?
skeptic 12/4/2012 | 8:44:26 PM
re: Cisco Twists Test Results The test results are what they are. No test, in and of itself, favors any particular vendor.

Let's stick with facts, shall we?

---------------------------------------------

Was that lengthy discussion of the merits of misordering packets in your material a "test result"?

Were the constant justifications for juniper's problems in the material "test results".

Was confusing one of your provider experts to the extent that he didn't think cisco provided route filtering a "test result"?

And how about your route-learning test where you juniper as having the "ability" to hold 2.4 million routes even though it wasn't doing much more than uselessly storing those routes on disk.

Why did you highlight that 2.4 million number with such strong language when it was effectivly
meaningless because of the junipers totally erronious and degraded state long before that point.


soothaandi 12/4/2012 | 8:44:28 PM
re: Cisco Twists Test Results I say we take it to the Supreme Court.

James Baker for Cisco, David Boies for Juniper. Sandra Day O'Connor casts the tie breaking vote.

Debating (CSCO CSCO) the tests (CSCO CSCO) will always be interminabubble, so hand it to (CSCO CSCO) the Supreme Court (CSCO CSCO) and the (CSCO CSCO) results will be (CSCO CSCO) Subliminabubble. What's good for Dubya is good for the rest of us. And that fact is totally unshakeabubble.

skeptic 12/4/2012 | 8:44:29 PM
re: Cisco Twists Test Results
Any credibility you have as an independent testing source is rapidly going away.

Going into attack-mode with a major vendor is not only making you look biased and stupid, its making it clear to lots of other vendors that you can't be trusted, are not impartial, and have some kind of agenda in all this.


Light reading plays favorites. Its breaks its own rules to hype certain people (see for example breaking your own stated rules to hype Caspain networks). And its attached at the hip to the marketing people in certain companies.

And with regard to this testing, you have shown a totally unprofessional attitude in selectivly refusing to respond to critques of your methods.

For todays example, its been pointed out to you today that you have totally false and misleading information on your site by an "expert" saying that cisco did not do route filtering. And
yet still now there is no correction,
no change in the material and no response
on the subject.

There are lots of people watching. Most of them don't work for juniper or cisco, but their perception of you is being shaped by your poor handling of these issues.
lightsource 12/4/2012 | 8:44:30 PM
re: Cisco Twists Test Results there can't be democracy when it comes to performance issues. why the hell is voting going on ?
it is so silly to vote to find out which eqpt is better performing. it should be based on test esults and objective facts.
based on LR's reputation so far, I bet cisco's press release is factually true than most crap on this site (less said the better)
Sparky 12/4/2012 | 8:44:30 PM
re: Cisco Twists Test Results My vote is:

Results are Inconclusive

This is due to the fact that an incomplete set of tests were performed, 40-byte streams are not realistic and the BGP table capacity test is rediculous (IMHO).

Sparky
xerxes98 12/4/2012 | 8:44:31 PM
re: Cisco Twists Test Results Hmmmmm .... I count 34 events that Cisco participated in and they won 18 of them ... in those same 34 events Juniper won 17 (there was ne tie) ... Cisco DID win all of the events listed in their press release ... it's obvious that the 16 events you are thinking of and the 12 events they are thinking of are different events ...

And you're right ... you shouldn't write biased editorial's to shore up you Juniper stock portfolio ...
srini88 12/4/2012 | 8:44:31 PM
re: Cisco Twists Test Results I am in favor of that...to start it off...my vote option 3-> inconclusive.
Reason: Each one has adv/disadv and either solution from jnpr/csco will work depending on the customer's requirements. Instead of sensationalizing the news and making it sound like a stupid fashion contest...LR should have written that in these areas with these business requirements juniper is better and csco is better in other areas. They should have brought out the strong points in both the boxes but mentioned that some of the points from a business standpoint don;t carry much weight. Card removal and similar issues like QOS should have been addressed.That would have prevented all the marketing guys from both companies from "twisting" the truth.

GonePostal 12/4/2012 | 8:44:32 PM
re: Cisco Twists Test Results This testing tit for tat stuff is boring and irrelevent. The real question is whether Larry Lang still cuts his hair with a Flowbie machine.
light_on_dude 12/4/2012 | 8:44:32 PM
re: Cisco Twists Test Results OK...Now were getting way off track....let's keep this as sophmoric as possible.....Peter..how does it feel to have one of your "gems" ripped to shreds by cisco?? HaHaHa...
David Newman...I'm sure you're very good at what you do, what you don't understand is Lightreading is NOT a facts and figures website...this place is for insult, inuendo, assasination and mockery. Please stop tying to get us out of the gutter...we like it here!!! LOL....mr. Heywood...pass the mud please...lol !!!
xerxes98 12/4/2012 | 8:44:33 PM
re: Cisco Twists Test Results It's interesting to note that minus the ACL tests, Cisco actually scored 18 first-places to juniper's 17 ... Juniper got 36 of its first-place results from ACL tests alone, an area that LR knew Cisco didn't support in the first place (nor did the other two vendors for the most part) ... so what we have here is a win on ACL by default ... is that really so impressive ?

I will be interested to see if LR would repeat this test once Cisco starts ACL support ...
probably not because then they would have to actually admit that Cisco has a better box ...

I still have not seen a reply as to why there were no OIR or fully-loaded box tests done ...
dnewman 12/4/2012 | 8:44:33 PM
re: Cisco Twists Test Results
"It is safe to say that those who wished to see the M160 succeed engineered this test. "

Careful: Cisco's engineers were among those who made constructive suggestions about, and contributions to, our test methodology. In fact, during the test Cisco's engineering staff praised the methodology for its rigor and thoroughness.

I am very sorry to see this war of words escalate. Cisco's 12416 achieved fine results in this test. It is regrettable that Cisco chose to misrepresent the results rather than letting the 12416's excellent numbers speak for themselves.

Regards,
David Newman
Network Test
Dr. Dre 12/4/2012 | 8:44:33 PM
re: Cisco Twists Test Results I am offering a potential solution to this apparent deadlock.

Would Lightreading be willing to take a poll amongst the readers to see who they believe won the tests based on the presented results.
That is pick one of:

1) Cisco Won
or
2) Juniper Won
or
3)Results are Inconclusive

The poll might give us a more clear idea what the general consensus is.

Yes some votes will be biased, but hopefully the majority wont.

In my own opinion, the tests seem reasonable. But I am not so sure a true winner can really be determined.

Any comments....

--Dr. Dre
tink 12/4/2012 | 8:44:34 PM
re: Cisco Twists Test Results Sparky,

You state, "It is safe to say that those who wished to see the M160 succeed engineered this test."

If this were the case, why would they engineer a test where they failed to deliver line-rate, a claim they've been making for almost a year now? Your assertion/argument fails.

Tink
Sparky 12/4/2012 | 8:44:34 PM
re: Cisco Twists Test Results Tink,

Why do you think my assertion is incorrect? Think about it. JNPR surely thought they would be able to perform at line rate or at a minimum beat CSCO. They thought wrong. What is important to understand is why were important features to core routing not tested?

Sparky
Sparky 12/4/2012 | 8:44:35 PM
re: Cisco Twists Test Results "Still, it is an error to argue in front of your data. You find yourself insensibly twisting them around to fit your theories." Sherlock Holmes in The Adventure of Wisteria Lodge.

These articles are ridiculous and the authors and testing suspect with at best mediocre results. Evaluations should be based on quantifiable and qualitative results vs. my box is better than yours, Ha-ha-ha!

To say that the M160 is BoB based on out-performing the GSR 12416 in a few tests, keeping in mind that the composition of tests was incomplete, is myopic to say the least.

It is safe to say that those who wished to see the M160 succeed engineered this test. The proof is in the test itself and the editorial comments by the author. With so much invested in equipment and time why not perform a truly comprehensive test, on core routing features such as OIR, fully loaded box, QoS latency, capacity and redundancy. In my humble opinion, Cisco has always been a winner. There is no news or ratings in reading about Cisco Systems winning again. LR creates contention to get people to read their hype.
flanker 12/4/2012 | 8:44:35 PM
re: Cisco Twists Test Results There would only be irony if Cisco had actually won the test. Did they? It's ok to vent, though. You guys got the living crap kicked out of you on the board's yesteday.

HEH HEH

By the way, I have nothing against CSCO. Maybe they did win, but all I see on the boards are sour grapes.
optical illusion 12/4/2012 | 8:44:35 PM
re: Cisco Twists Test Results Apparently CSCO withdrew the release. Can anyone find it on the site?

http://newsroom.cisco.com/dlls...
http://newsroom.cisco.com/dlls...



csco-grunt 12/4/2012 | 8:44:36 PM
re: Cisco Twists Test Results Got to endorse the ha-ha-ha here, it's called
"hoist by your own petard."

Stop complaining, and tell the intern who writes
your editorials for you to take a class in logic.

liteguy 12/4/2012 | 8:44:36 PM
re: Cisco Twists Test Results Does the Light reading staff love juniper or what? They should stop writing articles on them. To make it easier, they should move all juniper praise / cisco bashing to a tab on the side of the home page, this way we dont have to see it, and all the juniper employees can read it and have fun with it. By the way, I dont work for cisco, and I dont own stock in either.
optiray 12/4/2012 | 8:44:36 PM
re: Cisco Twists Test Results

Wow! This is cool. Poor LR, how does it
feel to be on the receiving end? After all,
unlike you, Cisco has got its facts correct!

Ha Ha Ha Ha !!!

-optiray
tpomerhn 12/4/2012 | 8:44:37 PM
re: Cisco Twists Test Results Yeah... I especially like the "Paul Newmann" comment. Sounds like LR's whining quite a bit, now. :)

I agree with Chuck - Cisco's press release is accurate when you use Imix, and maybe that should've been spelled out. I thought the Imix was more of a applicable test, too.

I, for one, would've liked to see tests on hot-swapping and failover - I'd like to see how these monsters responded to a few "pulls on the ol' OC-192 card :)"
optical illusion 12/4/2012 | 8:44:37 PM
re: Cisco Twists Test Results A NOTE TO ALL VENDORS:

A vendor could lose a customer pretty quick if they read this site and get the impression they might be dealing with used car salesmen.

Not to mention that Juniper will play up the disputed press release for all it's worth.
dnewman 12/4/2012 | 8:44:38 PM
re: Cisco Twists Test Results Please read the disclosure statement on the first page of the test before making unfounded allegations:

http://www.lightreading.com/do...

The test results are what they are. No test, in and of itself, favors any particular vendor.

Let's stick with facts, shall we?

Regards,
David Newman
Network Test
dnewman 12/4/2012 | 8:44:38 PM
re: Cisco Twists Test Results Sorry, but even considering Imix alone Cisco's release still misrepresents the facts.

For example, Cisco did not beat all other comers in nine out of 12 events. There were 16 events in which Cisco participated, and of these 16 Cisco won only four events outright.

By the way, I think considering the test without 40-byte, or without filtering, or without whatever, is an excellent idea -- if that meets your networking requirements.

Note here that I say networking requirements; spinning revisionist history to shore up your stock portfolio or reconfirm your prejudices doesn't change the test result.

Regards,
David Newman
Network Test
iptwister 12/4/2012 | 8:44:38 PM
re: Cisco Twists Test Results Benjamin Disraeli is quoted as saying "... there are lies, damned lies, and statistics".

The tests executed seem to give an advantage to Juniper.

I wonder if the Light Reading editorial staff hols stock in JNPR.

-iptwister
(who happens to be shorting JNPR)


chuckjackson 12/4/2012 | 8:44:39 PM
re: Cisco Twists Test Results I think the key to the conflicting conclusions between LR and Cisco is the 40-byte versus internet mix. If you consider only internet mix, almost all of Cisco's claims are correct. In general, I would think that the internet mix carries more weight. It is a more realistic test case. How often do service providers see 100% 40-byte traffic? The 40-byte traffic is more of a corner case. It's more important for ego rather than a realistic fair comparison. Perhaps, Cisco should have clarified their conclusions by adding in a statement that their conclusions were based on the internet mix traffic. In the same token, LR should have clarified their conclusions by adding a statement that their conclusions were based on the 100% 40-byte only traffic.

IMO, cisco's press release is accurate and justifiable.
light_on_dude 12/4/2012 | 8:44:39 PM
re: Cisco Twists Test Results Hey Peter, how does it feel to have YOUR words twisted? sounds like Cisco did to Lightreading, what Lightreading usually does to vendors...hows it feel to be twisted?? LOL!! (please don't say i work for cisco...I still hate them as much as ever) It's just damn funny to see this turnabout!!!
photonic 12/4/2012 | 8:44:39 PM
re: Cisco Twists Test Results
After accusing all us readers of being cry babies, it is really funny to see Light Reading editors turn into one themselves.

I love it.

HOME
Sign In
SEARCH
CLOSE
MORE
CLOSE