FastTCP Commercialized Into An FTP Appliance 156
prostoalex writes "FastTCP technology, developed by researchers at CalTech, is being commercialized. A company called FastSoft has introduced a hardware appliance that delivers 15x-20x faster FTP transmissions than those delivered via regular TCP. Says eWeek: 'The algorithm implemented in the Aria appliance senses congestion by continuously measuring the round-trip time for the TCP acknowledgment and then monitoring how that measurement changes from moment to moment.'"
hmm (Score:3, Interesting)
and if so why bother using the old FTP protocol instead of just making a new and more modern protocol?
Re:hmm (Score:4, Informative)
Re: (Score:3, Interesting)
I'm more excited by the new class of devices that are acting as block-caches for generalized TCP/IP traffic. These things are really neat-o for large distributed organziations, because they really help with duplicate traffic of which there is typically LOTS in every large organization. Idea is very simple:
Devices sit at each WAN end point, transparently inserted
Re:hmm (Score:4, Informative)
And I mean that: next time you implement one of these so-called miracle devices, run a TCP dump from both ends. If the TCP syn cookie is different, DO NOT INSTALL IT, AND RETURN THE DEVICE IMMEDIATELY.
Don't say someone didn't warn you.
I've spent the last four to six months debugging peoples' networks where it has invariably come down to these WAN-accelerators getting in the way and mangling network traffic.
*VERY* poorly implemented, to a one!
Re: (Score:2)
C//
Re: (Score:2)
Re:hmm (Score:5, Informative)
Re: (Score:3)
Al Gore didn't invent the Transmission Control, he invented the Internet.
Hence, data centers today mostly use Internet Protocol routers.
All Hail Al!
Re:hmm (Score:5, Funny)
Al Gore never claimed to invent the Internet. That's just a Republican spin on relatively accurate statements that Gore Made. What Al Gore invented is Algorithms. That's why they are called that!
Re: (Score:2)
I think they were originally spelled Al-Gore-Rhythms, methods for precise timing of Internet traffic.
Re: (Score:2)
FTP was mentioned in the article as an example. Any TCP-based protocol can use the box. All the box does is change the congestion control on packets passing through it.
No Way (Score:4, Insightful)
Regular TCP can't be more than an order of magnitude away from the Shannon Limit, can it?
Re:No Way (Score:5, Interesting)
The bandwidth between point A and B may be rated at a high throughput, but TCP protocols such as FTP will never achieve that speed if the RTT is long. Increasing the bandwidth won't help !! So a slowdown of 20-30x is not uncommon on WAN links with high latency e.g. transcontinental, via satellite.
I've looked at technologies like Digital Fountain (and it's Java implementation, FileCatalyst) which use UDP and some clever mathematics to overcome latency, however it's not clear from TFA what FastTCP is doing underneath..
Re:No Way (Score:5, Interesting)
I would imagine in the typical high-latency scenario, where regular TCP is mis-interpreting long RTT as link congestion, and backing off the rate, FastTCP is able to actually keep pushing the rate up, meanwhile keeping an eye on the RTT. I mean, the RTT shouldn't increase in line with the rate, unless the link actually *is* congested. So just increase the rate until the RTT increases, at which point you are genuinely maxxing out the link. I think that must be how it is working..
Re: (Score:3, Informative)
Re: (Score:2, Informative)
Uhm, let me guess, your knowledge of TCP is based on Trumpet Winsock for Windows 3.11 ?
Modern tcp-stacks most certainly scale the window and certainly don't "mis-interpret" high latency as congestion. (they do however interpret high packet-loss as congestion, which is a reasonable guesstimate most of the time, but *DOES* break down on links that, for example, have a constant packet-loss of a few percent (regardless of traffic-levels)
Re: (Score:2)
Re:No Way (Score:5, Interesting)
I love how fastsoft likes to compare themselves to Reno. 4.3BSD "Reno" was released in 1990, and the classic Reno implementation is LONG obsolete (and does indeed suck on a wide variety of connections).
I can see how it would be quite easy to achieve 10-20 times the throughput of Reno on a high-loss or high-latency connection, in fact a stock untuned Linux stack will do so in many situations. (For example, a few months ago I was doing TCP throughput tests dealing with some faulty hardware that liked to drop bursts of packets due to a shitty network driver. A machine running VxWorks 5.4, which is pretty much vanilla Reno, could only send 160 kilobytes/second over a 100Base-T LAN to that machine due to the packet loss making it throttle back. An untuned laptop with Linux 2.6.20 managed 1.7 megabytes/second over the same connection to the same destination.)
High latency connections were a major problem for TCP prior to RFC 1323 were a problem, but TCP stack authors have had 15 years to implement RFC 1323.
FastSoft's product may have been big news in the early 1990s, but if a company has to resort to making performance comparisons against the "Reno" TCP implementation, they're a snake oil salesman because Reno is such an obsolete and shitty TCP congestion control implementation.
Re:No Way (Score:5, Informative)
Well, let's see. They won the 2005 supercomputing bandwidth challenge with their system. They also have numerous publications in peer-reviewed journals, invited presentations at conferences, etc. Sure doesn't sound like snake oil.
Re: (Score:2)
Re: (Score:3, Interesting)
It doesn't, in general. There are edge-cases.
For example, most tcp-implementations use slow-start, which mean they will, regardless of latency, start with a small window, and then if that goes trough, gradually increase the window-size until no improvement is experienced anymore.
This makes a huge difference for example if your application transfers many small files, each over its own tcp-connection, which FTP will do but which I'm aware of no other commonly used file-transfer application doing. It's no
Re: (Score:3, Insightful)
Re: (Score:3, Informative)
Re: (Score:2)
In general, what is the actual limiting factor when I'm trying to get throughput between two endpoints on opposite sides of the world over the public Internet? That is, why do I want to select a download mirror close to where I live? Is it this RTT/congestion issue that comes into play, or is it that intercontinental bandwidth is less accessible (or is that totally inaccurate), or is there some other factor?
Re:No Way (Score:4, Funny)
If you've read TFA you'll know this revolutionary technology not only increases the speed by a factor of 15 to 20 times, but also insures "overall client happiness". Amazing !
Re: (Score:2)
If you've read TFA you'll know this revolutionary technology not only increases the speed by a factor of 15 to 20 times, but also insures "overall client happiness". Amazing !
Re: (Score:3, Interesting)
FastTCP is just a fancy name for TCP Vegas? (Score:5, Informative)
Re: (Score:3, Informative)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Reno is so obsolete that you can't
Re: (Score:2)
"TCP Vegas" is better, because it clearly indicates that this is a TCP implementation and can talk to other TCPs, rather than some protocol vaguely similar to, but different from, TCP.
Re:FastTCP is just a fancy name for TCP Vegas? (Score:5, Funny)
Powered by handwavium (Score:3, Informative)
Re:Powered by handwavium (Score:5, Informative)
http://netlab.caltech.edu/FAST/ [caltech.edu]
Several highlights include:
- Caltech held the world record for data transfer for awhile
- Won the bandwidth challenge at SC05
It's one of the best ways to tune a single TCP stream. Finally, the list of about 50 TCP-related publications should indicate this isn't handwavium:
http://netlab.caltech.edu/FAST/fastpub.html [caltech.edu]
Traditional TCP streams (such as what you get with FTP) top out around 10-20 Mbps. If you want to see a single stream go a couple hundred Mbps, you need TCP tweaks like FAST (however, FAST is one of many competing TCP "fixes").
Re: (Score:2)
I have recently observed 50-60 MBYTESps on a Gigabit LAN, between a vanilla Linux FTP server and a Windows client. And that was about the hard disk read limit on the server. Didn't look like a "traditional TCP stream" limit at all. It was a 300 MB. file, filled with random bytes. If I remember correctly, I didn't even enable jumbo frames, because one of the cards couldn't do it.
Re: (Score:2)
Does it speed up ftp or TCP/IP (Score:2)
If the latter can it speed up other protocols?
Hype (Score:4, Informative)
Other possibility is some sort of header compression.
Anyway, to use this safely you'd need to be *sure* you know your link charasteristics. The reason TCP has the slow-start mechanisms in the first place is to make sure you don't overflow the link - that's why it's known as flow control
Re: (Score:3, Informative)
ftp://ftp.rfc-editor.org/in-notes/rfc3649.txt [rfc-editor.org]
ftp://ftp.rfc-editor.org/in-notes/rfc3742.txt [rfc-editor.org]
I guess this device works as some sort of wrapper so that legacy TCP implementations don't get slowdowns, but doesn't strike as anything revolutionary to me - the RFCs are from year 2003.
Nonsense (Score:2)
Re:Nonsense (Score:4, Interesting)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2, Insightful)
Re: (Score:2)
1Mbps???? (Score:2)
Where do you live, the US?
Re:Nonsense (Score:5, Insightful)
I transfer about 20 TB / day at work, and that wouldn't be possible with a "typical FTP connection".
If you read the papers coming out of Caltech, you'd see they were optimizing for 10 Gbps lines, not residential lines. 15-20x faster is a very fair estimate; look at Caltech's presentations at SC05 or SC07.
Re: (Score:2, Insightful)
To transfer 20 TB/day, you need something like 1.8 Gbps sustained, not my measly 155 MBps, but that's only (only!) an order of magnitude better. TCP has shown itself quite comfortable scaling up from 300 baud modems to GigE links (6+ orders) so what's one more among friends? This is not to say TCP can't be improved: I've always thought using dropped packets to mea
Re:Nonsense (Score:4, Interesting)
Truthfully, you have to tweak the system pretty hard to get decent performance over a single stream (for us, 155 Mbps isn't sufficient - I work on a LHC project), especially from Nebraska to Switzerland (CERN). FAST TCP helps out a whole lot. GridFTP is the other piece of the equation - it is basically FTP with multiple data streams.
We tend to lean on hundreds of streams a whole lot more than tweaking TCP settings, and the Caltech guys give us heck for that. They're right, however - if you're getting 100s of KBps per stream to some European site, it just takes a ridiculous number of streams to get up to 100 MBps. Right now, the storage systems are behind the network, so we haven't even been able to start playing with FAST TCP yet.
http://cmsdoc.cern.ch/cms/aprom/phedex/prod/Activ
Re: (Score:2)
http://cmsdoc.cern.ch/cms/aprom/phedex/prod/Activi ty::RatePlots?graph=quantity&entity=dest&src_filte r=&dest_filter=Nebraska&no_mss=true&period=l14d&up to=&.submit=Update [cmsdoc.cern.ch]
In fact, we often talk with the Caltech folk about deploying FAST TCP; the problem is that both ends need to deploy the kernel patches. Truthfully, the limiting factor becomes the disk systems, not the network. When we start to push closer to 10 Gbps instead of 4-6 Gbps, we'll need to make smarter decision
Re: (Score:2)
To gain that much speed... (Score:2)
The only way I could see this as being possible is if there is so much latency that it basically makes the TCP protocol think every packet is lost, and resends them... 20 times. If you are seriously on a network t
Re: (Score:2)
Re: (Score:2)
You're thinking way too small. FAST TCP was designed with 10 Gbps links in mind - i.e., Internet2 type applications. FAST TCP streams are able to achieve several hundred Mbps. FTP streams over TCP Reno usually max out on something relatively pathetic, like 10-20 Mbps.
Caltech's SC07 presentation showed commodity servers which could transfer 2 Gbps end-to-end using their FDT tool (Java based, actually). The servers had 4 HDDs, dual Gigabit ethernet conncetions, and ran a Linux 2.6 kernel with the FAST
Re: (Score:3, Informative)
"The Aria 2000, which is due in July, supports 1G-bps links. Existing Aria appliances support 10M-bps links, 50M-bps links and 200M-bps links."
10gbps my ass. The one they haven't released only does a tenth of that. And the smallest of their products barely handles my home cable line.
For what it's worth, my initial thought was that they must be targetted truly massive lines and that it would be a lot harder to truly use those. Too bad it wasn't true.
Re: (Score:2)
Re: (Score:2)
"Beta testers at The Post Group, a film post-production facility in Hollywood, Calif., found that the Aria delivered 15 to 20 times faster transmissions "and better overall client happiness," said CIO Darin Harris."
For single connection use of big pipes (Score:2)
This is for single-connection use of wide-bandwidth channels with long latency. If you're synchronizing two servers across a considerable distance and have more than 1Gb/s or so available, it might be useful. For anything less, don't bother.
For local connections, you don't have many packets in flight, so you don't need this. For slower connections, you don't have the bandwidth to get that useful many packets in flight, so it doesn't help there either. It's not going to help your web browsing.
HOW much speedup? (Score:2, Insightful)
As the saying goes, this requires some very extraordinary evidence. Or there are a lot of missing qualifiers like "over a specific worst-case line that TCP doesn't come close to theoretical maximum performance on".
Re: (Score:2)
Yes, this is what FAST TCP [caltech.edu] is designed for.
Re: (Score:2)
An FTP session running over a 100Mbit LAN
Oh, you didn't RTFA.
No wonder you're confused.
The *first* sentance of TFA tells you everything you need to know:
This application is for WANs
http://en.wikipedia.org/wiki/Wide_area_network [wikipedia.org]
"The largest and most well-known example of a WAN is the Internet."
Now don't you wish you had skimmed TFA?
I expect better from a 3-digit UID
Re: (Score:2)
An FTP session running over a 100Mbit LAN should see about 10MB/sec real data transfer, maxing out the line and accounting for overhead They're claiming that their gadgets could move a file between each other at 150 megabytes per second over
FastTCP == 4 LoCs per hour (Score:5, Funny)
See http://www.fastsoft.com/research.html [fastsoft.com]
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
So where's the SlowTCP? (Score:2)
Re: (Score:2, Funny)
Re: (Score:3, Informative)
If you're looking for QoS in a home envir
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
There is Packeteer [packeteer.com], but most people can't afford them.
Re: (Score:2)
impossible to know if real from site (Score:4, Insightful)
The primary problem with WIFI networks is that they naturally have a lot more packet loss than normal links. On other links, a lot of packet loss tends to indicate packet congestion, so TCP likes to decrease throughput to try to solve it. Under WIFI, that's of course unnecessary and won't solve the underlying problem.
The article is missing some important technical details and there's a little too much marketing speak, but it does clearly sound like an improved TCP implementation, and probably some kind of traffic shaping hardware on one end (so that they don't have to change the networking stack on linux and windows, patch all their machines, etc).
There were a couple of other posters that suggested that such a thing wouldn't work. One guy even suggested that it would require different routers end to end! This is of course nonsense.
1. TCP != IP. Routers don't have to know anything about TCP to work (although they generally do for NAT, ACL, and traffic shaping purposes).
2. TCP implementations have been changed a number of times in the past. Changing the implementation is not the same as changing the protocol. Nothing else on the network cares what TCP implementation you are using as long as you speak the same protocol.
Re: (Score:2)
Re: (Score:2)
With regard to firewalls, would many of them see a packet containing an unrecognized transport protocol (in general, not this case which may use the same protocol number as normal TCP) and drop it? Is it possible to run your own protocol over IP without also controlling the filters at the endpoints? Or will the avera
Re: (Score:2)
I can't comment on the "average home router", but I have tested on mine and it'll happily route protocols other than those that are natively supported, but it ends up adding an entry to the state table for that entire protocol, so any incoming packets with the same protocol number end up getting sent to the system that sent out the first packet. You can see this entry in its state table with the port number fields set to zero. This does of course block any other hosts on my LAN from using that protocol unti
Why Not UDP (Score:2, Funny)
Those who fail to understand TCP.. (Score:3, Informative)
Re: (Score:2)
Re: (Score:2)
If you just want to transfer your data as fast as possible, change a couple of parameters in your TCP implementation so it ramps up faster and drops to maybe 95% instead of all the way to 50% when it gets packet loss. That'll work about as well as completely doing
Re: (Score:2)
One interesting possibility would be to do TCP over UDP. This would allow you to use the latest TCP tweaks without the need to modify the OS.
TCP is underestimated... (Score:3, Informative)
When something with as much hig
Re: (Score:2)
I disagree
Redundant features of TCP and UDP.
all UDP provides is checksums and application multiplexing. If you really wanted you could tweak the version of TCP you were adapting to run over UDP to remove those features but even if you don't they are very low overhead.
It's not as bad as TCP over IP over PPP over SSH which is over TCP (multiple reliable protocols on top of each other),
Yes multiple reliable protocols over each other is general
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Yes, if that is indeed what it is (didn't RTFA).
It's not FTP that is stupid; it's the things that force you to use tunnels. NAT, firewalls ... in a real Internet, anyone can open TCP con
Congestion Control (Score:5, Informative)
If you use Linux, have (CU)BIC loaded, correctly setup your NIC, and tune your TCP settings (rx/tx mem, queuelen, and such) then there is be no way for FastSoft to claim a 15-20x speedup improvement. I've done full 10 gigabit transmissions with a 150ms RTT using that kind of setup. FastSoft's device doesn't even support 10 gigabit, and their 1 gigabit device still isn't released.
This article is nothing other than a Slashadvertisment.
Papers (Score:2)
XMODEM (Score:5, Funny)
Hi from FastSoft (Score:2, Interesting)
and would like to share a few things.
As several people have already pointed out that, like most TCP
variants, FastTCP is end-to-end and does not require router support,
nor does it require any hardware or software installation at the receiving
computer. It accelerates all TCP-based applications. It eliminates inefficiencies
of current TCP implementations in the presence of packet loss and long latency.
It thus provides the most benefit i
Sort of like Zmodem then? (Score:2)
Hmm.. Sounds like I want my Zmodem back
Hype, not innovation (Score:2)
Re: (Score:2, Interesting)
But that's just my thought.
Re: (Score:2)
The more things change... (Score:3, Informative)
"Many very stupid companies have tried to come up with overly clever ways to speed up TCP/IP. TCP, by its nature, is a stateful and bidirectional protocol that requires all data packets to be acknowledged; this makes the data flow reliable, by providing a mechanism for dropped packets to be retransmitted; but this also makes for a more strictly regimented flow structure involving more packets transmitted over the wire than in simpler, non-reliable protocols l
What it does? (Score:2)
(i.e. The system needs to keep a buffer of all transmitted data until it is acknowledged)
I guess it behaves as a tcp proxy, the RTT between the sending server and the applicance (on the same LAN) is small, so whatever buffer size is allocated by the server is enough.
Sam
Re: (Score:3, Insightful)