A month ago, I pub­lished an ar­ti­cle on the com­pared per­for­mance of stun­nel, ng­inx and stud as TLS ter­mi­na­tors. The con­clu­sion was to use stud on a 64-bit sys­tem, with ses­sion caching and AES. stun­nel was un­able to scale prop­erly and ng­inx ex­hib­ited im­por­tant la­tency is­sues. I got con­struc­tive com­ments on many as­pects. There­fore, here is the sec­ond round. The pro­tag­o­nists are the same but both the con­di­tions and the con­clu­sions have changed a bit.

Benchmarks#

The hard­ware plat­form re­mains un­changed:

• HP DL 380 G7
• 6 GiB RAM
• 2 × L5630 @ 2.13GHz (8 cores), hy­per­thread­ing disabled
• 2 × In­tel 82576 NIC
• Spirent Avalanche 2900 appliance

We fo­cus on the fol­low­ing en­vi­ronment:

• Ubuntu Lu­cid, fea­tur­ing a patched ver­sion of OpenSSL 0.9.8k with AES-NI builtin and en­abled by default
• 2.6.39 ker­nel (250 HZ)
• 64-bit
• 1 core for HAProxy, 2 cores for net­work cards,¹ 4 cores for TLS, 1 core for the system
• sys­tem limit of 100,000 file de­scrip­tors per process

On the TLS side, the main com­po­nents are:

• 2048-bit cer­tificates
• AES128-SHA ci­pher suite
• 8 HTTP/1.0 re­quests per client
• HTTP re­sponse pay­load of 1024 bytes

If you are in­ter­ested in dif­fer­ent con­di­tions (32-bit, num­ber of cores, 1024-bit cer­tifi­cates, pro­por­tion of ses­sion reuses), have a look at the pre­vi­ous ar­ti­cle whose re­sults on the mat­ter are still valid. I only con­sider 2048-bit cer­tifi­cates be­cause most cer­tifi­cates is­sued to­day are 2048-bit. In SP800-57, NIST ad­vises that 1024-bit RSA keys will no longer be vi­able af­ter 2010 and ad­vises mov­ing to 2048-bit RSA keys:

If in­for­ma­tion is ini­tially signed in 2009 and needs to re­main se­cure for a max­i­mum of ten years (i.e., from 2009 to 2019), a 1024-bit RSA key would not pro­vide suf­fi­cient pro­tec­tion be­tween 2011 and 2019 and, there­fore, it is not rec­om­mended that 1024-bit RSA be used in this case.

And the pro­tag­o­nists are:

• stud, @e207dbc5a3, with ses­sion cache en­abled for 20,000 ses­sions, lis­ten back­log of 1000 con­nections;
• stun­nel 4.44, con­fig­ured with --disable-libwrap --with-threads=pthread (con­fig­u­ra­tion); and
• ng­inx 1.1.4, con­fig­ured with --with-http_ssl_module (con­fig­u­ration).

HAProxy (con­fig­u­ra­tion) is used for stun­nel and stud to add load-balancing.

For each test, we want to max­i­mize the num­ber of HTTP trans­ac­tions per sec­ond (TPS) while min­i­miz­ing la­tency and un­suc­cess­ful trans­ac­tions. The max­i­mum num­ber of al­lowed failed trans­ac­tions is set at 1 trans­ac­tion out of 1000. The max­i­mum al­lowed av­er­age la­tency is 100 ms.²

Control case#

Be­low is the first com­par­i­son be­tween stud, ng­inx and stun­nel on plain Ubuntu Lu­cid. Like in pre­vi­ous bench­marks, stun­nel is not able to scale, with a max­i­mum of 400 TPS. How­ever, early la­tency is­sues of ng­inx have been solved, thanks to the change in con­fig­u­ra­tion. stud and ng­inx get sim­i­lar per­for­mance but, as you can see on the bot­tom plot (whose scale is log­a­rith­mic) stud starts with an ini­tial la­tency of 40 ms.

This is pretty an­noy­ing. I have tried some mi­cro op­ti­miza­tion to solve this is­sue but with­out luck. If la­tency mat­ters for you, stick with stun­nel or ng­inx (or find where the prob­lem lies in stud).

OpenSSL 1.0.0e#

When I pub­lished the first round of bench­marks, Michał Tro­j­nara was du­bi­ous about the bad re­sults of stun­nel. He told me on Twitter:

I solved it! You com­piled stun­nel against a 4 years old OpenSSL with MT-un­safe SSL_accept. Up­grade OpenSSL and re­build stun­nel. This bug was fixed in OpenSSL 1.0.0b. I im­ple­mented a (slow) work­around, so stun­nel doesn’t crash with ear­lier ver­sions of OpenSSL.

There­fore, I back­ported Ubuntu Oneiric OpenSSL 1.0.0e to Ubuntu Lu­cid³ and, as you can see your­self on the plot be­low, there is a huge per­for­mance boost for stun­nel: about 400%! stud fea­tures a slight im­prove­ment too (about 8%) but still starts with a la­tency of 40 ms while ng­inx does not improve.

Intel Accelerator Engine#

Most Linux dis­tri­bu­tions are still us­ing OpenSSL 0.9.8. For ex­am­ple, De­bian Squeeze ships with OpenSSL 0.9.8o, Ubuntu Lu­cid with 0.9.8k and RHEL 5.7 with 0.9.8e. There­fore, they are not el­i­gi­ble for re­cent per­for­mance im­prove­ments. To solve this, In­tel re­leased In­tel Ac­cel­er­a­tor En­gine, which also brings, among other things, AES-NI support:

This is es­sen­tially a col­lec­tion of as­sem­bler mod­ules from de­vel­op­ment OpenSSL branch, packed to­gether as a stand­alone en­gine. “Stand­alone” means that it can be com­piled out­side OpenSSL source tree with­out patch­ing the lat­ter. Idea is to pro­vide a way to uti­lize new code in al­ready re­leased OpenSSL ver­sions con­strained by sup­port and re­lease poli­cies lim­it­ing their evolve­ment to gen­uine bug fixes.

Alas, in my tests, there was no im­prove­ment com­pared to OpenSSL 1.0.0e (al­ready patched with AES-NI support).

stun­nel comes with out-of-the-box sup­port for ad­di­tional en­gines. Just add some­thing like this to your stunnel.conf:

engine=dynamic
engineCtrl=SO_PATH:/WOO/intel-accel-1.4/libintel-accel.so
engineCtrl=ID:intel-accel
engineCtrl=LIST_ADD:1
engineCtrl=LOAD
engineCtrl=INIT

For stud, you should grab sup­port for en­gine se­lec­tion from GitHub and for ng­inx, spec­ify the en­gine with ssl_engine.

Intel Performance Primitives#

In his great ar­ti­cle “Ac­cel­er­ated SSL,” Si­mon Kuhn shows huge per­for­mance im­prove­ments by us­ing In­tel’s IPP li­brary, a li­brary aimed at lever­ag­ing power of multi-core pro­ces­sors and in­clud­ing some cryp­to­graphic prim­i­tives and a patch to make use of them in OpenSSL. Un­for­tu­nately, this li­brary is not free, not even as in “free beer.” The patch pro­vided by In­tel only ap­plies on OpenSSL 0.9.8j. Si­mon up­dated it for OpenSSL 1.0.0d. It also ap­plies with a slight mod­i­fi­ca­tion to OpenSSL 1.0.0e. Look at Si­mon’s ar­ti­cle for in­struc­tions on how to com­pile OpenSSL with IPP.

Si­mon mea­sured 220% im­prove­ment on RSA 1024-bit pri­vate key op­er­a­tions by us­ing this li­brary. Un­for­tu­nately, we do not get such a per­for­mance burst with stud which ex­hibits a 15% im­prove­ment (but re­mem­ber that pri­vate key op­er­a­tions only hap­pen once ev­ery 8 trans­ac­tions thanks to ses­sion re­use). Worse, with stun­nel, per­for­mance is re­duced by one third and ng­inx just seg­faults. I think such a li­brary is too risky to use in pro­duction.

Hyperthreading#

Dur­ing the pre­vi­ous tests, only half the ca­pac­ity of each core was used. For stud, adding more work­ers per core only slightly im­proves this sit­u­a­tion (less than 10% with 4 work­ers per core). There­fore, I thought us­ing hy­per­thread­ing could rec­tify the sit­u­a­tion here. In this case, stud, stun­nel and ng­inx are al­lo­cated 8 “vir­tual” cores in­stead of 4. stud only got a small im­prove­ment of 5%, ng­inx a mod­er­ate im­prove­ment of 11% while stun­nel got a sig­nif­i­cant im­prove­ment of 26%.

Conclusion#

Here is a quick re­cap of the max­i­mum TPS reached on each test (while keep­ing av­er­age re­sponse time be­low 100 ms). All tests were run on 4 phys­i­cal cores with 2048-bit cer­tifi­cates, AES128-SHA ci­pher suite. Each client makes 8 HTTP re­quests and re­use the first TLS session.

I also would like to stress that be­low 1500 TPS, stun­nel adds an over­head of a few mil­lisec­onds while stud adds 40 mil­lisec­onds. With OpenSSL 1.0.0e and hy­per­thread­ing en­abled, stun­nel has both good per­for­mance and low la­tency on low loads to mod­er­ate loads. You may just want to limit the num­ber of al­lowed con­nec­tions to avoid col­laps­ing on high loads. While in the pre­vi­ous round, stud was the cham­pion and stun­nel lagged be­hind, the sit­u­a­tion is now re­versed: stun­nel is on top!

Up­date (2011-12)

As said ear­lier, the la­tency is­sue with stud has been solved. Per­for­mance re­mains mostly the same. Here is what the lat­est plot looks like af­ter the fix:

1. https://vincent.bernat.ch/en/blog/2011-ssl-benchmark
2. https://github.com/bumptech/stud
3. https://www.stunnel.org/
4. https://docs.nginx.com
5. https://github.com/varnish/hitch
6. https://www.haproxy.org
7. https://web.archive.org/web/2011/http://h10010.www1.hp.com/wwpc/us/en/sm/WF05a/15351-15351-3328412-241644-241475-4091412.html
8. https://www.intel.com/content/www/us/en/products/sku/47927/intel-xeon-processor-l5630-12m-cache-2-13-ghz-5-86-gts-intel-qpi/specifications.html
9. https://web.archive.org/web/2011/https://www.spirent.com/products/avalanche.aspx
10. https://web.archive.org/web/2011/http://releases.ubuntu.com/lucid/
11. https://www.haproxy.org/
12. https://csrc.nist.gov/pubs/sp/800/57/pt1/r2/final
13. https://github.com/vincentbernat/stud/tree/e207dbc5a3d5c2753a4d83794780edf6edb2e3fb
14. https://gist.github.com/vincentbernat/1180393/a61d6a22c4f0af2b227a5afa0695400cdbb0be81
15. https://gist.github.com/vincentbernat/1145093/80f787ad334c92dcbec933c6b43856575134b1e4
16. http://mailman.nginx.org/pipermail/nginx/2011-September/029243.html
17. https://gist.github.com/vincentbernat/1163077/1a840a1a080e9cae5ffb91cc5c5717e620d7a0d9
18. https://github.com/bumptech/stud/pull/37
19. https://github.com/bumptech/stud/commit/7ad212a3f8
20. https://en.wikipedia.org/wiki/Nagle’s_algorithm
21. https://www.linkedin.com/in/mtrojnar
22. https://twitter.com/mtrojnar/status/110912432519659520
23. https://twitter.com/mtrojnar/status/110914736979316737
24. https://web.archive.org/web/2011/https://packages.ubuntu.com/oneiric/openssl
25. https://gist.github.com/vincentbernat/1272151/b1a61124d1568eb795fa82b24b875889cbd0005c
26. https://groups.google.com/forum/#!msg/mailing.openssl.dev/Z8PwfK53C2E/pdkktMcnpAEJ
27. https://github.com/bumptech/stud/pull/36
28. https://web.archive.org/web/2011/http://zombe.es/post/5183420528/accelerated-ssl
29. https://www.intel.com/content/www/us/en/developer/tools/oneapi/ipp.html