Here is the short ver­sion: to get bet­ter per­for­mance on your TLS ter­mi­na­tor, use stud on a 64-bit sys­tem with patch from Émeric Brun for TLS ses­sion re­use with some AES ci­pher suite (128 or 256, does not re­ally mat­ter), with­out DHE, on as many cores as needed, a key size of 1024 bits un­less more is needed.

Introduction#

One year ago, Adam Lan­g­ley, from Google, stated TLS was not com­pu­ta­tion­ally ex­pen­sive any more:

In Jan­u­ary this year (2010), Gmail switched to us­ing HTTPS for ev­ery­thing by de­fault. Pre­vi­ously it had been in­tro­duced as an op­tion, but now all of our users use HTTPS to se­cure their email be­tween their browsers and Google, all the time. In or­der to do this we had to de­ploy no ad­di­tional ma­chines and no spe­cial hard­ware. On our pro­duc­tion fron­tend ma­chines, SSL/TLS ac­counts for less than 1% of the CPU load, less than 10 KB of mem­ory per con­nec­tion and less than 2% of net­work over­head. Many peo­ple be­lieve that SSL takes a lot of CPU time and we hope the above num­bers (pub­lic for the first time) will help to dis­pel that.

If you stop read­ing now you only need to re­mem­ber one thing: SSL/TLS is not com­pu­ta­tion­ally ex­pen­sive any more.

This is a very in­for­ma­tive post con­tain­ing tips on how to en­hance TLS per­for­mance by re­duc­ing la­tency. How­ever, un­like Gmail, you may still be wor­ried by TLS raw per­for­mances. Maybe each of your fron­tends is able to serve 2000 re­quests per sec­ond and CPU over­head for TLS is there­fore sig­nif­i­cant. Maybe you want to ter­mi­nate TLS in your load bal­ancer—even if you know this is not the best way to scale.

Tuning TLS#

There are a lot of knobs you can use to get bet­ter per­for­mances from TLS: choos­ing the best im­ple­men­ta­tion, us­ing more CPU cores, switch­ing to 64-bit sys­tem, choos­ing the right ci­pher suite and the ap­pro­pri­ate key size and en­abling a ses­sion cache.

We will con­sider three TLS ter­mi­na­tors. They all use OpenSSL un­der the hood. stun­nel is the old­est one and uses a threaded model. stud is a re­cent at­tempt to write a sim­ple TLS ter­mi­na­tor which is ef­fi­cient and scal­able. It uses the one-process-per-core model. ng­inx is a web server and it can be used as re­verse proxy and there­fore act as TLS ter­mi­na­tor. It is known to be one of the most ef­fi­cient web servers, hence the choice here. It also fea­tures built-in ba­sic load bal­anc­ing. Since stud and stun­nel¹ do not have this fea­ture, we use them with HAProxy, a high per­for­mance load-bal­ancer that usu­ally de­fers the TLS part to stun­nel (but stud can act as a drop-in re­place­ment here).

These im­ple­men­ta­tions were al­ready tested by Matt Stan­cliff. He first con­cluded ng­inx sucked at TLS then that ng­inx did not suck at TLS (in the first case, ng­inx was the only one to se­lect a DHE ci­pher suite). The de­tails were very scarce. I hope to bring here more details.

This brings the im­por­tance of the se­lected ci­pher suite. The client and the server must agree on the ci­pher suite to use for sym­met­ric en­cryp­tion. The server will se­lect the strong­est one it sup­ports from the list pro­posed by the client. If you en­able some ex­pen­sive ci­pher suite on your server, it is likely to be selected.

The last im­por­tant knob is TLS ses­sion re­use. It mat­ters for both per­for­mance and la­tency. Dur­ing the first hand­shake, the server will send a ses­sion ID. The client can use this ses­sion ID to re­quest an ab­bre­vi­ated hand­shake. This hand­shake is shorter (la­tency im­prove­ment by one round-trip) and al­lows us to re­use the pre­vi­ously ne­go­ti­ated mas­ter se­cret (per­for­mance im­prove­ment by skip­ping the costli­est part of the hand­shake). See the ar­ti­cle from Adam Lan­g­ley for ad­di­tional in­for­ma­tion (with help­ful drawings).

The fol­low­ing ta­ble shows what ci­pher suite is se­lected by some ma­jor web sites. I have also in­di­cated whether they sup­port ses­sion re­use. See this post from Matt Palmer to know how to check this.

RFC 5077 de­fines an­other mech­a­nism for ses­sion re­sump­tion that does not re­quire any server-side state. You can find ad­di­tional in­for­ma­tion about this op­tional mech­a­nism in my ar­ti­cle about en­abling ses­sion reuse.

Benchmarks#

All these bench­marks have been run on a HP DL 380 G7, with two Xeon L5630 (run­ning at 2.13GHz for a to­tal of 8 cores), with­out hy­per­thread­ing, us­ing a 2.6.39 ker­nel (HZ is set to 250) and two In­tel 82576 NIC. Linux con­ntrack sub­sys­tem has been dis­abled and file lim­its have been raised over 100,000.

A Spirent Avalanche 2900 ap­pli­ance is used to run the benchmarks.

We are in­ter­ested in raw TLS per­for­mances in terms of hand­shakes per sec­ond. Be­cause we want to ob­serve the ef­fects of ses­sion re­use, the sce­nario run by most tests lets clients do four suc­ces­sive re­quests, reusing the TLS ses­sion for three of them. There is no HTTP keepalive. There is no com­pres­sion. The size of the HTTP an­swer from the server is 1024 bytes. These choices are made be­cause our pri­mary tar­get is the num­ber of TLS hand­shakes per second.

It should also be noted that HAProxy alone is able to han­dle 22,000 TPS on one core. Dur­ing the tests, it was never the bot­tleneck.

Implementation#

We run a first bench to com­pare ng­inx, stud and stun­nel on one core. This bench runs on a 32-bit sys­tem, with AES128-SHA1 ci­pher suite and a 1024-bit key size. Here is the re­sult for stud (ver­sion 0.1, with Émeric Brun’s patch for TLS ses­sion reuse):

The most im­por­tant plot is the top one. The blue line is the at­tempted num­ber of trans­ac­tions per sec­ond (TPS) while the green one is the suc­cess­ful num­ber of TPS. When the two lines start to di­verge, this means we reach some kind of max­i­mum: the num­ber of un­suc­cess­ful TPS also starts to rise (the red line).

There are sev­eral no­tice­able points: the max­i­mum TPS (788 TPS), the max­i­mum TPS with an av­er­age re­sponse time less than 100 ms (766 TPS), less than 500 ms (776 TPS) and the max­i­mum TPS with less than 0.01% of packet loss (783 TPS).

Let’s have a look at ng­inx (ver­sion 1.0.5):

It is able to get the same per­for­mance as stud (763 TPS). How­ever, over 512 TPS, you get more than 100 ms of re­sponse time. Over 556 TPS, you even get more than 500 ms! I get this be­hav­ior in ev­ery bench with ng­inx (with or with­out proxy_buffering, with or with­out tcp_nopush, with or with­out tcp_nodelay) and I am un­able to ex­plain it. Maybe there is some­thing wrong in the con­fig­u­ra­tion. Af­ter hit­ting this limit, ng­inx starts to process con­nec­tions by bursts. There­fore, the num­ber of suc­cess­ful TPS is plot­ted us­ing a dot­ted line while the mov­ing av­er­age over 20 sec­onds is plot­ted with a solid line.

On the next plot, you can see the com­pared per­for­mance of the three im­ple­men­ta­tions. On this kind of plot, the num­ber of TPS that we keep is the max­i­mum num­ber of TPS where loss is less than 0.1% and av­er­age re­sponse time is less than 100 ms. stud achieves a per­for­mance of 766 TPS while ng­inx and stun­nel are just above 500 TPS.

Number of cores#

With mul­ti­ple cores, we can as­sign sev­eral of them to do TLS hard work. To get bet­ter per­for­mance, we pin each process to a ded­i­cated CPU. Here is the dis­tri­bu­tion used dur­ing the tests:

Re­mem­ber, we have two CPUs, 4 cores on each CPU. Cores on the same CPU share the same L2 cache (on this model). There­fore, the ar­range­ment in­side a CPU is not re­ally im­por­tant. When pos­si­ble, we try to keep things to­gether on the same CPU. The TLS pro­cesses al­ways get ex­clu­sive use of a core since they will al­ways be the most busy. The repar­ti­tion is done with cpuset(7) for user­land pro­cesses and by set­ting smp_affinity for net­work card in­terrupts.

We keep one core for the sys­tem. It is quite im­por­tant to be able to con­nect and mon­i­tor the sys­tem cor­rectly even when it is loaded. With so many cores, we can af­ford to re­serve one core for this usage.

Be­ware! There is a trap when pin­ning pro­cesses or IRQ to a core. You need to check /proc/cpuinfo to dis­cover the map­ping be­tween ker­nel pro­ces­sors and phys­i­cal cores. Some­times, sec­ond ker­nel pro­ces­sor can be the first core of the sec­ond phys­i­cal pro­ces­sor (in­stead of the sec­ond core of the first phys­i­cal pro­cessor).

As you can see in the plot above, only stud is able to scale prop­erly. stun­nel is not able to take ad­van­tage of the cores and its per­for­mance is worse than with a sin­gle core. I think this may be due to its threaded model and the fact that the user­land for this 32-bit sys­tem is a bit old. ng­inx is able to achieve the same TPS as stud but is dis­qual­i­fied be­cause of the in­creased la­tency. We will not use stun­nel in the re­main­ing tests.

32-bit vs 64-bit#

The eas­i­est way to achieve a per­for­mance boost is to switch to a 64-bit sys­tem. TPS are dou­bled. On our 64-bit sys­tem, thanks to the use of a ver­sion of OpenSSL with the ap­pro­pri­ate sup­port, AES-NI, a CPU ex­ten­sion to im­prove the speed of AES encryption/decryption, is en­abled. Re­main­ing tests are done with a 64-bit system.

Ciphers and key sizes#

In our tests, the in­flu­ence of ci­phers is min­i­mal. There is al­most no dif­fer­ence be­tween AES256 and AES128. Us­ing RC4 adds some la­tency. The use of AES-NI may have helped to avoid this la­tency for AES.

On the other hand, us­ing a 2048-bit key size has a huge per­for­mance hit. TPS are di­vided by 5.

Session cache#

The last thing to check is the in­flu­ence of TLS ses­sion re­use. Since we are on a lo­cal net­work, we see only one of the pos­i­tive ef­fects of ses­sion re­use: bet­ter TPS be­cause of re­duced CPU us­age. If there was a la­tency penalty, we should also have seen bet­ter TPS thanks to the re­moved round-trip.

This plot also ex­plains why stud per­for­mance falls af­ter the max­i­mum: be­cause of the failed trans­ac­tions, the ses­sion cache is not as ef­fi­cient. This phe­nom­e­non does not ex­ist when ses­sion re­use is not enabled.

Conclusion#

Here is a sum­mary of the max­i­mum TPS reached dur­ing the bench­mark (with an av­er­age re­sponse time be­low 100ms). Our con­trol use case is the fol­low­ing: 64-bit, 6 cores, AES128, SHA1, 1024-bit key, 4 re­quests in the same TLS session.

There­fore, stud in our con­trol sce­nario is able to sus­tain 1500 TPS per core. It seems to be the best cur­rent op­tion for a TLS ter­mi­na­tion. It is not avail­able in De­bian yet, but I in­tend to package it.

Here is how stud is invoked:

# ulimit -n 100000
# stud  -n 2 -f 172.31.200.15,443 -b 127.0.0.1,80 -c ALL \
>   -B 1000 -C 20000 --write-proxy \
>   =(cat server1024.crt server1024.key dhe1024)

You may also want to look at HAProxy con­fig­u­ra­tion, ng­inx con­fig­u­ra­tion and stun­nel con­fig­uration.

1. https://github.com/bumptech/stud
2. https://github.com/bumptech/stud/pull/10
3. https://vincent.bernat.ch/en/blog/2011-ssl-benchmark-round2
4. https://github.com/varnish/hitch
5. https://www.haproxy.org
6. https://www.imperialviolet.org/2010/06/25/overclocking-ssl.html
7. https://www.hezmatt.org/~mpalmer/blog/2011/06/28/ssl-session-caching-in-nginx.html
8. https://www.stunnel.org/
9. https://docs.nginx.com
10. https://www.haproxy.org/
11. https://web.archive.org/web/2011/http://matt.io/
12. https://web.archive.org/web/2011/http://matt.io/entry/uq
13. https://web.archive.org/web/2011/http://matt.io/entry/ur
14. https://www.rfc-editor.org/info/rfc5077
15. https://vincent.bernat.ch/en/blog/2011-ssl-session-reuse-rfc5077
16. https://www.intel.com/content/www/us/en/products/sku/47927/intel-xeon-processor-l5630-12m-cache-2-13-ghz-5-86-gts-intel-qpi/specifications.html
17. https://web.archive.org/web/2011/https://www.spirent.com/products/avalanche.aspx
18. https://gist.github.com/vincentbernat/1145093/232d7f8c02c9fbbb709c6b60a5b31f12f3cea7a6
19. https://manpages.debian.org/cpuset.7.html
20. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=636072
21. https://gist.github.com/vincentbernat/1163077/1a840a1a080e9cae5ffb91cc5c5717e620d7a0d9
22. https://gist.github.com/vincentbernat/1180393/a61d6a22c4f0af2b227a5afa0695400cdbb0be81