1 EQL Driver: Serial IP Load Balancing HOWTO 2 Simon "Guru Aleph-Null" Janes, simon@ncm.com 3 v1.1, February 27, 1995 4 5 This is the manual for the EQL device driver. EQL is a software device 6 that lets you load-balance IP serial links (SLIP or uncompressed PPP) 7 to increase your bandwidth. It will not reduce your latency (i.e. ping 8 times) except in the case where you already have lots of traffic on 9 your link, in which it will help them out. This driver has been tested 10 with the 1.1.75 kernel, and is known to have patched cleanly with 11 1.1.86. Some testing with 1.1.92 has been done with the v1.1 patch 12 which was only created to patch cleanly in the very latest kernel 13 source trees. (Yes, it worked fine.) 14 15 1. Introduction 16 17 Which is worse? A huge fee for a 56K leased line or two phone lines? 18 It's probably the former. If you find yourself craving more bandwidth, 19 and have a ISP that is flexible, it is now possible to bind modems 20 together to work as one point-to-point link to increase your 21 bandwidth. All without having to have a special black box on either 22 side. 23 24 25 The eql driver has only been tested with the Livingston PortMaster-2e 26 terminal server. I do not know if other terminal servers support load- 27 balancing, but I do know that the PortMaster does it, and does it 28 almost as well as the eql driver seems to do it (-- Unfortunately, in 29 my testing so far, the Livingston PortMaster 2e's load-balancing is a 30 good 1 to 2 KB/s slower than the test machine working with a 28.8 Kbps 31 and 14.4 Kbps connection. However, I am not sure that it really is 32 the PortMaster, or if it's Linux's TCP drivers. I'm told that Linux's 33 TCP implementation is pretty fast though.--) 34 35 36 I suggest to ISPs out there that it would probably be fair to charge 37 a load-balancing client 75% of the cost of the second line and 50% of 38 the cost of the third line etc... 39 40 41 Hey, we can all dream you know... 42 43 44 2. Kernel Configuration 45 46 Here I describe the general steps of getting a kernel up and working 47 with the eql driver. From patching, building, to installing. 48 49 50 2.1. Patching The Kernel 51 52 If you do not have or cannot get a copy of the kernel with the eql 53 driver folded into it, get your copy of the driver from 54 ftp://slaughter.ncm.com/pub/Linux/LOAD_BALANCING/eql-1.1.tar.gz. 55 Unpack this archive someplace obvious like /usr/local/src/. It will 56 create the following files: 57 58 59 60 ______________________________________________________________________ 61 -rw-r--r-- guru/ncm 198 Jan 19 18:53 1995 eql-1.1/NO-WARRANTY 62 -rw-r--r-- guru/ncm 30620 Feb 27 21:40 1995 eql-1.1/eql-1.1.patch 63 -rwxr-xr-x guru/ncm 16111 Jan 12 22:29 1995 eql-1.1/eql_enslave 64 -rw-r--r-- guru/ncm 2195 Jan 10 21:48 1995 eql-1.1/eql_enslave.c 65 ______________________________________________________________________ 66 67 Unpack a recent kernel (something after 1.1.92) someplace convenient 68 like say /usr/src/linux-1.1.92.eql. Use symbolic links to point 69 /usr/src/linux to this development directory. 70 71 72 Apply the patch by running the commands: 73 74 75 ______________________________________________________________________ 76 cd /usr/src 77 patch </usr/local/src/eql-1.1/eql-1.1.patch 78 ______________________________________________________________________ 79 80 81 82 83 84 2.2. Building The Kernel 85 86 After patching the kernel, run make config and configure the kernel 87 for your hardware. 88 89 90 After configuration, make and install according to your habit. 91 92 93 3. Network Configuration 94 95 So far, I have only used the eql device with the DSLIP SLIP connection 96 manager by Matt Dillon (-- "The man who sold his soul to code so much 97 so quickly."--) . How you configure it for other "connection" 98 managers is up to you. Most other connection managers that I've seen 99 don't do a very good job when it comes to handling more than one 100 connection. 101 102 103 3.1. /etc/rc.d/rc.inet1 104 105 In rc.inet1, ifconfig the eql device to the IP address you usually use 106 for your machine, and the MTU you prefer for your SLIP lines. One 107 could argue that MTU should be roughly half the usual size for two 108 modems, one-third for three, one-fourth for four, etc... But going 109 too far below 296 is probably overkill. Here is an example ifconfig 110 command that sets up the eql device: 111 112 113 114 ______________________________________________________________________ 115 ifconfig eql 198.67.33.239 mtu 1006 116 ______________________________________________________________________ 117 118 119 120 121 122 Once the eql device is up and running, add a static default route to 123 it in the routing table using the cool new route syntax that makes 124 life so much easier: 125 126 127 128 ______________________________________________________________________ 129 route add default eql 130 ______________________________________________________________________ 131 132 133 3.2. Enslaving Devices By Hand 134 135 Enslaving devices by hand requires two utility programs: eql_enslave 136 and eql_emancipate (-- eql_emancipate hasn't been written because when 137 an enslaved device "dies", it is automatically taken out of the queue. 138 I haven't found a good reason to write it yet... other than for 139 completeness, but that isn't a good motivator is it?--) 140 141 142 The syntax for enslaving a device is "eql_enslave <master-name> 143 <slave-name> <estimated-bps>". Here are some example enslavings: 144 145 146 147 ______________________________________________________________________ 148 eql_enslave eql sl0 28800 149 eql_enslave eql ppp0 14400 150 eql_enslave eql sl1 57600 151 ______________________________________________________________________ 152 153 154 155 156 157 When you want to free a device from its life of slavery, you can 158 either down the device with ifconfig (eql will automatically bury the 159 dead slave and remove it from its queue) or use eql_emancipate to free 160 it. (-- Or just ifconfig it down, and the eql driver will take it out 161 for you.--) 162 163 164 165 ______________________________________________________________________ 166 eql_emancipate eql sl0 167 eql_emancipate eql ppp0 168 eql_emancipate eql sl1 169 ______________________________________________________________________ 170 171 172 173 174 175 3.3. DSLIP Configuration for the eql Device 176 177 The general idea is to bring up and keep up as many SLIP connections 178 as you need, automatically. 179 180 181 3.3.1. /etc/slip/runslip.conf 182 183 Here is an example runslip.conf: 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 ______________________________________________________________________ 200 name sl-line-1 201 enabled 202 baud 38400 203 mtu 576 204 ducmd -e /etc/slip/dialout/cua2-288.xp -t 9 205 command eql_enslave eql $interface 28800 206 address 198.67.33.239 207 line /dev/cua2 208 209 name sl-line-2 210 enabled 211 baud 38400 212 mtu 576 213 ducmd -e /etc/slip/dialout/cua3-288.xp -t 9 214 command eql_enslave eql $interface 28800 215 address 198.67.33.239 216 line /dev/cua3 217 ______________________________________________________________________ 218 219 220 221 222 223 3.4. Using PPP and the eql Device 224 225 I have not yet done any load-balancing testing for PPP devices, mainly 226 because I don't have a PPP-connection manager like SLIP has with 227 DSLIP. I did find a good tip from LinuxNET:Billy for PPP performance: 228 make sure you have asyncmap set to something so that control 229 characters are not escaped. 230 231 232 I tried to fix up a PPP script/system for redialing lost PPP 233 connections for use with the eql driver the weekend of Feb 25-26 '95 234 (Hereafter known as the 8-hour PPP Hate Festival). Perhaps later this 235 year. 236 237 238 4. About the Slave Scheduler Algorithm 239 240 The slave scheduler probably could be replaced with a dozen other 241 things and push traffic much faster. The formula in the current set 242 up of the driver was tuned to handle slaves with wildly different 243 bits-per-second "priorities". 244 245 246 All testing I have done was with two 28.8 V.FC modems, one connecting 247 at 28800 bps or slower, and the other connecting at 14400 bps all the 248 time. 249 250 251 One version of the scheduler was able to push 5.3 K/s through the 252 28800 and 14400 connections, but when the priorities on the links were 253 very wide apart (57600 vs. 14400) the "faster" modem received all 254 traffic and the "slower" modem starved. 255 256 257 5. Testers' Reports 258 259 Some people have experimented with the eql device with newer 260 kernels (than 1.1.75). I have since updated the driver to patch 261 cleanly in newer kernels because of the removal of the old "slave- 262 balancing" driver config option. 263 264 265 o icee from LinuxNET patched 1.1.86 without any rejects and was able 266 to boot the kernel and enslave a couple of ISDN PPP links. 267 268 5.1. Randolph Bentson's Test Report 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 From bentson@grieg.seaslug.org Wed Feb 8 19:08:09 1995 332 Date: Tue, 7 Feb 95 22:57 PST 333 From: Randolph Bentson <bentson@grieg.seaslug.org> 334 To: guru@ncm.com 335 Subject: EQL driver tests 336 337 338 I have been checking out your eql driver. (Nice work, that!) 339 Although you may already done this performance testing, here 340 are some data I've discovered. 341 342 Randolph Bentson 343 bentson@grieg.seaslug.org 344 345 --------------------------------------------------------- 346 347 348 A pseudo-device driver, EQL, written by Simon Janes, can be used 349 to bundle multiple SLIP connections into what appears to be a 350 single connection. This allows one to improve dial-up network 351 connectivity gradually, without having to buy expensive DSU/CSU 352 hardware and services. 353 354 I have done some testing of this software, with two goals in 355 mind: first, to ensure it actually works as described and 356 second, as a method of exercising my device driver. 357 358 The following performance measurements were derived from a set 359 of SLIP connections run between two Linux systems (1.1.84) using 360 a 486DX2/66 with a Cyclom-8Ys and a 486SLC/40 with a Cyclom-16Y. 361 (Ports 0,1,2,3 were used. A later configuration will distribute 362 port selection across the different Cirrus chips on the boards.) 363 Once a link was established, I timed a binary ftp transfer of 364 289284 bytes of data. If there were no overhead (packet headers, 365 inter-character and inter-packet delays, etc.) the transfers 366 would take the following times: 367 368 bits/sec seconds 369 345600 8.3 370 234600 12.3 371 172800 16.7 372 153600 18.8 373 76800 37.6 374 57600 50.2 375 38400 75.3 376 28800 100.4 377 19200 150.6 378 9600 301.3 379 380 A single line running at the lower speeds and with large packets 381 comes to within 2% of this. Performance is limited for the higher 382 speeds (as predicted by the Cirrus databook) to an aggregate of 383 about 160 kbits/sec. The next round of testing will distribute 384 the load across two or more Cirrus chips. 385 386 The good news is that one gets nearly the full advantage of the 387 second, third, and fourth line's bandwidth. (The bad news is 388 that the connection establishment seemed fragile for the higher 389 speeds. Once established, the connection seemed robust enough.) 390 391 #lines speed mtu seconds theory actual %of 392 kbit/sec duration speed speed max 393 3 115200 900 _ 345600 394 3 115200 400 18.1 345600 159825 46 395 2 115200 900 _ 230400 396 2 115200 600 18.1 230400 159825 69 397 2 115200 400 19.3 230400 149888 65 398 4 57600 900 _ 234600 399 4 57600 600 _ 234600 400 4 57600 400 _ 234600 401 3 57600 600 20.9 172800 138413 80 402 3 57600 900 21.2 172800 136455 78 403 3 115200 600 21.7 345600 133311 38 404 3 57600 400 22.5 172800 128571 74 405 4 38400 900 25.2 153600 114795 74 406 4 38400 600 26.4 153600 109577 71 407 4 38400 400 27.3 153600 105965 68 408 2 57600 900 29.1 115200 99410.3 86 409 1 115200 900 30.7 115200 94229.3 81 410 2 57600 600 30.2 115200 95789.4 83 411 3 38400 900 30.3 115200 95473.3 82 412 3 38400 600 31.2 115200 92719.2 80 413 1 115200 600 31.3 115200 92423 80 414 2 57600 400 32.3 115200 89561.6 77 415 1 115200 400 32.8 115200 88196.3 76 416 3 38400 400 33.5 115200 86353.4 74 417 2 38400 900 43.7 76800 66197.7 86 418 2 38400 600 44 76800 65746.4 85 419 2 38400 400 47.2 76800 61289 79 420 4 19200 900 50.8 76800 56945.7 74 421 4 19200 400 53.2 76800 54376.7 70 422 4 19200 600 53.7 76800 53870.4 70 423 1 57600 900 54.6 57600 52982.4 91 424 1 57600 600 56.2 57600 51474 89 425 3 19200 900 60.5 57600 47815.5 83 426 1 57600 400 60.2 57600 48053.8 83 427 3 19200 600 62 57600 46658.7 81 428 3 19200 400 64.7 57600 44711.6 77 429 1 38400 900 79.4 38400 36433.8 94 430 1 38400 600 82.4 38400 35107.3 91 431 2 19200 900 84.4 38400 34275.4 89 432 1 38400 400 86.8 38400 33327.6 86 433 2 19200 600 87.6 38400 33023.3 85 434 2 19200 400 91.2 38400 31719.7 82 435 4 9600 900 94.7 38400 30547.4 79 436 4 9600 400 106 38400 27290.9 71 437 4 9600 600 110 38400 26298.5 68 438 3 9600 900 118 28800 24515.6 85 439 3 9600 600 120 28800 24107 83 440 3 9600 400 131 28800 22082.7 76 441 1 19200 900 155 19200 18663.5 97 442 1 19200 600 161 19200 17968 93 443 1 19200 400 170 19200 17016.7 88 444 2 9600 600 176 19200 16436.6 85 445 2 9600 900 180 19200 16071.3 83 446 2 9600 400 181 19200 15982.5 83 447 1 9600 900 305 9600 9484.72 98 448 1 9600 600 314 9600 9212.87 95 449 1 9600 400 332 9600 8713.37 90 450 451 452 453 454 455 5.2. Anthony Healy's Report 456 457 458 459 460 461 462 463 Date: Mon, 13 Feb 1995 16:17:29 +1100 (EST) 464 From: Antony Healey <ahealey@st.nepean.uws.edu.au> 465 To: Simon Janes <guru@ncm.com> 466 Subject: Re: Load Balancing 467 468 Hi Simon, 469 I've installed your patch and it works great. I have trialed 470 it over twin SL/IP lines, just over null modems, but I was 471 able to data at over 48Kb/s [ISDN link -Simon]. I managed a 472 transfer of up to 7.5 Kbyte/s on one go, but averaged around 473 6.4 Kbyte/s, which I think is pretty cool. :) 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529