Coping with the TCP TIME-WAIT state on busy Linux servers

Vincent Bernat

TL;DR

Do not en­able net.ipv4.tcp_tw_recycle—it doesn’t even ex­ist any­more since Linux 4.12. Most of the time, TIME-WAIT sock­ets are harm­less. Oth­er­wise, jump to the sum­mary for the rec­om­mended so­lutions.

The Linux ker­nel doc­u­men­ta­tion is not very help­ful about what net.ipv4.tcp_tw_recycle and net.ipv4.tcp_tw_reuse do. This lack of doc­u­men­ta­tion opens the path to nu­mer­ous tun­ing guides ad­vis­ing to set both these set­tings to 1 to re­duce the num­ber of en­tries in the TIME-WAIT state. How­ever, as stated by the tcp(7) man­ual page, the net.ipv4.tcp_tw_recycle op­tion is quite prob­lem­atic for pub­lic-fac­ing servers as it won’t han­dle con­nec­tions from two dif­fer­ent com­put­ers be­hind the same NAT de­vice, which is a prob­lem hard to de­tect and wait­ing to bite you:

En­able fast re­cy­cling of TIME-WAIT sock­ets. En­abling this op­tion is not rec­om­mended since this causes prob­lems when work­ing with NAT (Net­work Ad­dress Trans­lation).

I will pro­vide here a more de­tailed ex­pla­na­tion of how to prop­erly han­dle the TIME-WAIT state. Also, keep in mind we are look­ing at the TCP stack of Linux. This is com­pletely un­re­lated to Net­fil­ter con­nec­tion track­ing which may be tweaked in other ways.1

About the TIME-WAIT state#

Let’s rewind a bit and have a close look at this TIME-WAIT state. What is it? See the TCP state di­a­gram below:

TCP state diagram
TCP state diagram

Only the end clos­ing the con­nec­tion first will reach the TIME-WAIT state. The other end will fol­low a path which usu­ally per­mits it to quickly get rid of the con­nection.

You can have a look at the cur­rent state of con­nec­tions with ss -tan:

$ ss -tan | head -5
LISTEN     0  511             *:80              *:*
SYN-RECV   0  0     192.0.2.145:80    203.0.113.5:35449
SYN-RECV   0  0     192.0.2.145:80   203.0.113.27:53599
ESTAB      0  0     192.0.2.145:80   203.0.113.27:33605
TIME-WAIT  0  0     192.0.2.145:80   203.0.113.47:50685

Purpose#

There are two pur­poses for the TIME-WAIT state:

  • The most known one is to pre­vent de­layed seg­ments from one con­nec­tion be­ing ac­cepted by a later con­nec­tion re­ly­ing on the same quadru­plet (source ad­dress, source port, des­ti­na­tion ad­dress, des­ti­na­tion port). The se­quence num­ber also needs to be in a cer­tain range to be ac­cepted. This nar­rows a bit the prob­lem but it still ex­ists, es­pe­cially on fast con­nec­tions with large re­ceive win­dows. RFC 1337 ex­plains in de­tail what hap­pens when the TIME-WAIT state is de­fi­cient.2 Here is an ex­am­ple of what could be avoided if the TIME-WAIT state wasn’t shortened:
Duplicate segments accepted in another connection
Due to a shortened TIME-WAIT state, a delayed TCP segment has been accepted in an unrelated connection.
  • The other pur­pose is to en­sure the re­mote end has closed the con­nec­tion. When the last ACK is lost, the re­mote end stays in the LAST-ACK state.3 With­out the TIME-WAIT state, a con­nec­tion could be re­opened while the re­mote end still thinks the pre­vi­ous con­nec­tion is valid. When it re­ceives a SYN seg­ment (and the se­quence num­ber matches), it will an­swer with a RST as it is not ex­pect­ing such a seg­ment. The new con­nec­tion will be aborted with an error:
Last ACK lost
If the remote end stays in LAST-ACK state because the last ACK was lost, opening a new connection with the same quadruplet will not work.

RFC 793 re­quires the TIME-WAIT state to last twice the time of the MSL. On Linux, this du­ra­tion is not tun­able and is de­fined in include/net/tcp.h as one minute:

#define TCP_TIMEWAIT_LEN (60*HZ) /* how long to wait to destroy TIME-WAIT
                                  * state, about 60 seconds     */

There have been propo­si­tions to turn this into a tun­able value but it has been re­fused on the ground the TIME-WAIT state is a good thing.

Problems#

Now, let’s see why this state can be an­noy­ing on a server han­dling a lot of con­nec­tions. There are three as­pects of the problem:

  • the slot used in the con­nec­tion ta­ble pre­vent­ing new con­nec­tions of the same kind;
  • the mem­ory oc­cu­pied by the socket struc­ture in the kernel; and
  • the ad­di­tional CPU usage.

The re­sult of ss -tan state time-wait | wc -l is not a prob­lem per se!

Connection table slot#

A con­nec­tion in the TIME-WAIT state is kept for one minute in the con­nec­tion ta­ble. This means an­other con­nec­tion with the same quadru­plet (source ad­dress, source port, des­ti­na­tion ad­dress, des­ti­na­tion port) can­not exist.

For a web server, the des­ti­na­tion ad­dress and the des­ti­na­tion port are likely to be con­stant. If your web server is be­hind an L7 load-bal­ancer, the source ad­dress will also be con­stant. On Linux, the client port is by de­fault al­lo­cated in a port range of about 30,000 ports (this can be changed by tun­ing net.ipv4.ip_local_port_range). This means that only 30,000 con­nec­tions can be es­tab­lished be­tween the web server and the load-bal­ancer ev­ery minute, so about 500 con­nec­tions per second.

Up­date (2026-10)

This limit is less strict when the TIME-WAIT sock­ets are on the server side. Linux ac­cepts a new SYN seg­ment for a quadru­plet in the TIME-WAIT state if its se­quence num­ber is above the last one re­ceived for the pre­vi­ous con­nec­tion, as al­lowed by RFC 1122, or if its time­stamp is more re­cent, as de­scribed in RFC 6191.

If the TIME-WAIT sock­ets are on the client side, such a sit­u­a­tion is easy to de­tect. The call to connect() will re­turn EADDRNOTAVAIL and the ap­pli­ca­tion will log some er­ror mes­sage about that. On the server side, this is more com­plex as there is no log and no counter to rely on. In doubt, you should just try to come up with some­thing sen­si­ble to list the num­ber of used quadruplets:

$ ss -tan 'sport = :80' | awk '{print $(NF)" "$(NF-1)}' | \
>     sed 's/:[^ ]*//g' | sort | uniq -c
    696 10.24.2.30 10.33.1.64
   1881 10.24.2.30 10.33.1.65
   5314 10.24.2.30 10.33.1.66
   5293 10.24.2.30 10.33.1.67
   3387 10.24.2.30 10.33.1.68
   2663 10.24.2.30 10.33.1.69
   1129 10.24.2.30 10.33.1.70
  10536 10.24.2.30 10.33.1.73

The so­lu­tion is more quadru­plets.4 This can be done in sev­eral ways (in or­der of dif­fi­culty to set up):

  • use more client ports by set­ting net.ipv4.ip_local_port_range to a wider range;
  • use more server ports by ask­ing the web server to lis­ten to sev­eral ad­di­tional ports (81, 82, 83, …);
  • use more client IP by con­fig­ur­ing ad­di­tional IP on the load bal­ancer and use them in a round-robin fashion;5 or
  • use more server IP by con­fig­ur­ing ad­di­tional IP on the web server.6

The last so­lu­tion is to tweak net.ipv4.tcp_tw_reuse and net.ipv4.tcp_tw_recycle. Do not do that yet, we will cover these set­tings later.

Memory#

With many con­nec­tions to han­dle, leav­ing a socket open for one ad­di­tional minute may cost your server some mem­ory. For ex­am­ple, if you want to han­dle about 10,000 new con­nec­tions per sec­ond, you will have about 600,000 sock­ets in the TIME-WAIT state. How much mem­ory does it rep­re­sent? Not that much!

First, from the ap­pli­ca­tion point of view, a TIME-WAIT socket does not con­sume any mem­ory: the socket has been closed. In the ker­nel, a TIME-WAIT socket is present in three struc­tures (for three dif­fer­ent purposes):

  1. A hash ta­ble of con­nec­tions, named the “TCP es­tab­lished hash ta­ble” (de­spite con­tain­ing con­nec­tions in other states) is used to lo­cate an ex­ist­ing con­nec­tion, for ex­am­ple when re­ceiv­ing a new segment.

    Each bucket of this hash ta­ble con­tains both a list of con­nec­tions in the TIME-WAIT state and a list of reg­u­lar ac­tive con­nec­tions. The size of the hash ta­ble de­pends on the sys­tem mem­ory and is printed at boot:

    $ dmesg | grep "TCP established hash table"
[    0.169348] TCP established hash table entries: 65536 (order: 8, 1048576 bytes)

    It is pos­si­ble to over­ride it by spec­i­fy­ing the num­ber of en­tries on the ker­nel com­mand line with the thash_entries pa­rameter.

    Each el­e­ment of the list of con­nec­tions in the TIME-WAIT state is a struct tcp_timewait_sock, while the type for other states is struct tcp_sock:7

    struct tcp_timewait_sock {
    struct inet_timewait_sock tw_sk;
    u32    tw_rcv_nxt;
    u32    tw_snd_nxt;
    u32    tw_rcv_wnd;
    u32    tw_ts_offset;
    u32    tw_ts_recent;
    long   tw_ts_recent_stamp;
};

struct inet_timewait_sock {
    struct sock_common  __tw_common;

    int                     tw_timeout;
    volatile unsigned char  tw_substate;
    unsigned char           tw_rcv_wscale;
    __be16 tw_sport;
    unsigned int tw_ipv6only     : 1,
                 tw_transparent  : 1,
                 tw_pad          : 6,
                 tw_tos          : 8,
                 tw_ipv6_offset  : 16;
    unsigned long            tw_ttd;
    struct inet_bind_bucket *tw_tb;
    struct hlist_node        tw_death_node;
};

  2. A set of lists of con­nec­tions, called the “death row,” is used to ex­pire the con­nec­tions in the TIME-WAIT state. They are or­dered by how much time is left be­fore ex­piration.

    It uses the same mem­ory space as for the en­tries in the hash ta­ble of con­nec­tions. This is the struct hlist_node tw_death_node mem­ber of struct inet_timewait_sock.8

  3. A hash ta­ble of bound ports, hold­ing the lo­cally bound ports and the as­so­ci­ated pa­ram­e­ters, is used to de­ter­mine if it is safe to lis­ten to a given port or to find a free port in the case of a dy­namic bind. The size of this hash ta­ble is the same as the size of the hash ta­ble of con­nections:

    $ dmesg | grep "TCP bind hash table"
[    0.169962] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)

    Each el­e­ment is a struct inet_bind_bucket. There is one el­e­ment for each lo­cally bound port. A TIME-WAIT con­nec­tion to a web server is lo­cally bound to the port 80 and shares the same en­try as its sib­ling TIME-WAIT con­nec­tions. On the other hand, a con­nec­tion to a re­mote ser­vice is lo­cally bound to some ran­dom port and does not share its entry.

We are only con­cerned by the space oc­cu­pied by struct tcp_timewait_sock and struct inet_bind_bucket. There is one struct tcp_timewait_sock for each con­nec­tion in the TIME-WAIT state, in­bound or out­bound. There is one ded­i­cated struct inet_bind_bucket for each out­bound con­nec­tion and none for an in­bound con­nection.

A struct tcp_timewait_sock is only 168 bytes while a struct inet_bind_bucket is 48 bytes:

$ sudo apt-get install linux-image-$(uname -r)-dbg
[…]
$ gdb /usr/lib/debug/boot/vmlinux-$(uname -r)
(gdb) print sizeof(struct tcp_timewait_sock)
 $1 = 168
(gdb) print sizeof(struct tcp_sock)
 $2 = 1776
(gdb) print sizeof(struct inet_bind_bucket)
 $3 = 48

If you have about 40,000 in­bound con­nec­tions in the TIME-WAIT state, it should eat less than 10 MiB of mem­ory. If you have about 40,000 out­bound con­nec­tions in the TIME-WAIT state, you need to ac­count for 2.5 MiB of ad­di­tional mem­ory. Let’s check that by look­ing at the out­put of slabtop. Here is the re­sult on a server with about 50,000 con­nec­tions in the TIME-WAIT state, 45,000 of which are out­bound con­nections:

$ sudo slabtop -o | grep -E '(^  OBJS|tw_sock_TCP|tcp_bind_bucket)'
  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
 50955  49725  97%    0.25K   3397       15     13588K tw_sock_TCP
 44840  36556  81%    0.06K    760       59      3040K tcp_bind_bucket

There is noth­ing to change here: the mem­ory used by TIME-WAIT con­nec­tions is re­ally small. If your server needs to han­dle thou­sands of new con­nec­tions per sec­ond, you need far more mem­ory to be able to ef­fi­ciently push data to clients. The over­head of TIME-WAIT con­nec­tions is neg­ligible.

CPU#

On the CPU side, search­ing for a free lo­cal port can be a bit ex­pen­sive. The work is done by the inet_csk_get_port() func­tion which uses a lock and it­er­ates on lo­cally bound ports un­til a free port is found. A large num­ber of en­tries in this hash ta­ble is usu­ally not a prob­lem if you have a lot of out­bound con­nec­tions in the TIME-WAIT state (like ephemeral con­nec­tions to a mem­cached server): the con­nec­tions usu­ally share the same pro­file, the func­tion will quickly find a free port as it it­er­ates on them se­quentially.

Other solutions#

If you still think you have a prob­lem with TIME-WAIT con­nec­tions af­ter read­ing the pre­vi­ous sec­tion, there are three ad­di­tional so­lu­tions to solve them:

  • dis­able socket lingering;
  • net.ipv4.tcp_tw_reuse; and
  • net.ipv4.tcp_tw_recycle.

Socket lingering#

When close() is called, any re­main­ing data in the ker­nel buf­fers will be sent in the back­ground and the socket will even­tu­ally tran­si­tion to the TIME-WAIT state. The ap­pli­ca­tion can con­tinue to work im­me­di­ately and as­sume that all data will even­tu­ally be safely de­livered.

How­ever, an ap­pli­ca­tion can choose to dis­able this be­hav­ior, known as socket lin­ger­ing. There are two flavors:

  1. In the first one, any re­main­ing data will be dis­carded and in­stead of clos­ing the con­nec­tion with the nor­mal four-packet con­nec­tion ter­mi­na­tion se­quence, the con­nec­tion will be closed with a RST (and there­fore, the peer will de­tect an er­ror) and will be im­me­di­ately de­stroyed. No TIME-WAIT state in this case.

  2. With the sec­ond fla­vor, if any data is re­main­ing in the socket send buf­fer, the process will sleep when call­ing close() un­til ei­ther all the data is sent and ac­knowl­edged by the peer or the con­fig­ured linger timer ex­pires. It is pos­si­ble for a process to not sleep by set­ting the socket as non-block­ing. In this case, the same process hap­pens in the back­ground. It per­mits the re­main­ing data to be sent dur­ing a con­fig­ured time­out but if the data is suc­cess­fully sent, the nor­mal close se­quence is run and you get a TIME-WAIT state. And in the other case, you will get the con­nec­tion close with a RST and the re­main­ing data is discarded.

In both cases, dis­abling socket lin­ger­ing is not a one-size-fits-all so­lu­tion. It may be used by some ap­pli­ca­tions like HAProxy or Ng­inx when it is safe to use from the up­per pro­to­col point of view. There are good rea­sons to not dis­able it un­con­di­tionally.

net.ipv4.tcp_tw_reuse#

The TIME-WAIT state pre­vents de­layed seg­ments from be­ing ac­cepted in an un­re­lated con­nec­tion. How­ever, in cer­tain con­di­tions, it is pos­si­ble to as­sume a new con­nec­tion’s seg­ment can­not be mis­in­ter­preted with an old con­nec­tion’s segment.

RFC 1323 presents a set of TCP ex­ten­sions to im­prove per­for­mance over high-band­width paths. Among other things, it de­fines a new TCP op­tion car­ry­ing two four-byte time­stamp fields. The first one is the cur­rent value of the time­stamp clock of the TCP send­ing the op­tion while the sec­ond one is the most re­cent time­stamp re­ceived from the re­mote host.

By en­abling net.ipv4.tcp_tw_reuse, Linux will re­use an ex­ist­ing con­nec­tion in the TIME-WAIT state for a new out­go­ing con­nec­tion if the new time­stamp is strictly big­ger than the most re­cent time­stamp recorded for the pre­vi­ous con­nec­tion: an out­go­ing con­nec­tion in the TIME-WAIT state can be reused af­ter just one second.

How is it safe? The first pur­pose of the TIME-WAIT state was to avoid du­pli­cate seg­ments be­ing ac­cepted in an un­re­lated con­nec­tion. Thanks to the use of time­stamps, such du­pli­cate seg­ments will come with an out­dated time­stamp and there­fore be discarded.

The sec­ond pur­pose was to en­sure the re­mote end is not in the LAST-ACK state be­cause of the loss of the last ACK. The re­mote end will re­trans­mit the FIN seg­ment until:

  1. it gives up (and tears down the con­nec­tion), or
  2. it re­ceives the ACK it is wait­ing for (and tears down the con­nec­tion), or
  3. it re­ceives a RST (and tears down the con­nection).

If the FIN seg­ments are re­ceived on time, the lo­cal end socket will still be in the TIME-WAIT state and the ex­pected ACK seg­ments will be sent.

Once a new con­nec­tion re­places the TIME-WAIT en­try, the SYN seg­ment of the new con­nec­tion is ig­nored (thanks to the time­stamps) and won’t be an­swered by a RST but only by a re­trans­mis­sion of the FIN seg­ment. The FIN seg­ment will then be an­swered with a RST (be­cause the lo­cal con­nec­tion is in the SYN-SENT state) which will al­low the tran­si­tion out of the LAST-ACK state. The ini­tial SYN seg­ment will even­tu­ally be re­sent (af­ter one sec­ond) be­cause there was no an­swer and the con­nec­tion will be es­tab­lished with­out ap­par­ent er­ror, ex­cept a slight delay:

Last ACK lost and timewait reuse
If the remote end stays in LAST-ACK state because the last ACK was lost, the remote connection will be reset when the local end transition to the SYN-SENT state.

It should be noted that when a con­nec­tion is reused, the TWRecycled counter is in­creased (de­spite its name).

net.ipv4.tcp_tw_recycle#

This mech­a­nism also re­lies on the time­stamp op­tion but af­fects both in­com­ing and out­go­ing con­nec­tions. This is handy when the server usu­ally closes the con­nec­tion first.9

The TIME-WAIT state is sched­uled to ex­pire sooner: it will be re­moved af­ter the re­trans­mis­sion time­out (RTO) in­ter­val which is com­puted from the RTT and its vari­ance. You can spot the ap­pro­pri­ate val­ues for a liv­ing con­nec­tion with the ss command:

$ ss --info  sport = :2112 dport = :4057
State      Recv-Q Send-Q    Local Address:Port        Peer Address:Port
ESTAB      0      1831936   10.47.0.113:2112          10.65.1.42:4057
         cubic wscale:7,7 rto:564 rtt:352.5/4 ato:40 cwnd:386 ssthresh:200 send 4.5Mbps rcv_space:5792

To keep the same guar­an­tees the TIME-WAIT state was pro­vid­ing, while re­duc­ing the ex­pi­ra­tion timer, when a con­nec­tion en­ters the TIME-WAIT state, the lat­est time­stamp is re­mem­bered in a ded­i­cated struc­ture con­tain­ing var­i­ous met­rics for pre­vi­ous known des­ti­na­tions. Then, Linux will drop any seg­ment from the re­mote host whose time­stamp is not strictly big­ger than the lat­est recorded time­stamp, un­less the TIME-WAIT state would have expired:

if (tmp_opt.saw_tstamp &&
    tcp_death_row.sysctl_tw_recycle &&
    (dst = inet_csk_route_req(sk, &fl4, req, want_cookie)) != NULL &&
    fl4.daddr == saddr &&
    (peer = rt_get_peer((struct rtable *)dst, fl4.daddr)) != NULL) {
        inet_peer_refcheck(peer);
        if ((u32)get_seconds() - peer->tcp_ts_stamp < TCP_PAWS_MSL &&
            (s32)(peer->tcp_ts - req->ts_recent) >
                                        TCP_PAWS_WINDOW) {
                NET_INC_STATS_BH(sock_net(sk), LINUX_MIB_PAWSPASSIVEREJECTED);
                goto drop_and_release;
        }
}

When the re­mote host is a NAT de­vice, the con­di­tion on time­stamps will for­bid all the hosts ex­cept one be­hind the NAT de­vice to con­nect dur­ing one minute be­cause they do not share the same time­stamp clock. In doubt, it is far bet­ter to dis­able this op­tion since it leads to dif­fi­cult to de­tect and dif­fi­cult to di­ag­nose problems.

Up­date (2017-09)

Start­ing from Linux 4.10 (com­mit 95a22caee396), Linux will ran­dom­ize time­stamp off­sets for each con­nec­tion, mak­ing this op­tion com­pletely bro­ken, with or with­out NAT. It has been com­pletely re­moved from Linux 4.12.

The LAST-ACK state is han­dled in the same way as for net.ipv4.tcp_tw_reuse.

Summary#

The uni­ver­sal so­lu­tion is to in­crease the num­ber of pos­si­ble quadru­plets by us­ing, for ex­am­ple, more server ports. This will al­low you to not ex­haust the pos­si­ble con­nec­tions with TIME-WAIT entries.

On the server side, never en­able net.ipv4.tcp_tw_recycle due to its side ef­fects. En­abling net.ipv4.tcp_tw_reuse is use­less for in­com­ing con­nections.

On the client side, en­abling net.ipv4.tcp_tw_reuse is an­other al­most-safe so­lu­tion. En­abling net.ipv4.tcp_tw_recycle in ad­di­tion to net.ipv4.tcp_tw_reuse is mostly useless.

More­over, when de­sign­ing pro­to­cols, do not let clients close first. Clients will not have to deal with the TIME-WAIT state push­ing the re­spon­si­bil­ity to servers which are bet­ter suited to han­dle this.

And a fi­nal quote by W. Richard Stevens, in Unix Net­work Pro­gramming:

The TIME_WAIT state is our friend and is there to help us (i.e., to let old du­pli­cate seg­ments ex­pire in the net­work). In­stead of try­ing to avoid the state, we should un­der­stand it.

Up­date (2026-10)

This post was briefly dis­cussed on Hacker News.