The Case of the 50ms request<<init>>

Hello! This is a mystery where your goal is to solve a debugging problem! You'll collect clues, interpret evidence, and ultimately solve the Case of the 50ms Request!

In the sidebar, you'll notice a "What you know" list. As you collect more clues, the list will update with what you've learned.

Click "Start" to get started.

<div class="nav"> [[Start->Landing]] </div>You have a program that uploads very tiny files to a server. You notice in your monitoring that these file uploads are a little slow -- maybe about 50ms on average.

<div class="nav"> [[Reproduce the problem -> Reproduce]] </div> The uploading code is in Javascript, so you write a tiny Javascript program called <code>slow.js</code> to reproduce the problem so you can easily investigate it. The program prints out the time elapsed for the request in milliseconds.

You have a choice: you can SSH to the client machine (which is making the requests), or the server machine (which is serving the requests).

<div class="nav"> [[Look at the slow.js code -> Code]] [[SSH to the client -> Client]] [[SSH to the server -> Server]] [[Make a guess at why this is happening->diagnosis]] </div> Here's a little Javascript program that reproduces the problem. It's called <code>slow.js</code>. After the request finishes, it prints out the time elapsed in milliseconds.

<pre> const http = require('http') const fs = require('fs').promises var start; async function uploadFile(filename) { const req = http.request({ hostname: 'mysterybox.local', port: 8000, method: 'POST'}, res => { console.log("Time elapsed:", new Date() - start, "milliseconds"); }) const stats = await fs.stat(filename) start = new Date(); req.setHeader('Content-Type', 'text/plain'); req.setHeader('Content-Length', stats.size); req.flushHeaders(); const data = await fs.readFile(filename, 'utf8'); await new Promise(resolve => {req.write(data, resolve)}); } uploadFile("test.txt") </pre>

<div class="nav go-back"> [[Go back -> Reproduce]] </div> You SSH to the client machine that's making the slow requests to do some more investigation. You copy over the <code>slow.js</code> script you wrote so that you can reproduce it easily.

<div class="nav"> [[Run slow.js -> Run slow.js]] [[Use curl -> curl]] [[Use ping -> ping]] [[Use strace -> strace]] [[Use tcpdump -> tcpdump]] [[Use termshark -> termshark]] [[Go back -> Reproduce]] </div> You write a quick for loop to run the <code>slow.js</code> script 20 times.

<pre> $ for i in `seq 1 20`; do node slow.js; done Time elapsed: 51 milliseconds Time elapsed: 51 milliseconds Time elapsed: 50 milliseconds Time elapsed: 51 milliseconds Time elapsed: 50 milliseconds Time elapsed: 50 milliseconds Time elapsed: 54 milliseconds Time elapsed: 53 milliseconds Time elapsed: 53 milliseconds Time elapsed: 49 milliseconds Time elapsed: 52 milliseconds Time elapsed: 54 milliseconds Time elapsed: 53 milliseconds Time elapsed: 50 milliseconds Time elapsed: 50 milliseconds Time elapsed: 49 milliseconds Time elapsed: 51 milliseconds Time elapsed: 50 milliseconds Time elapsed: 49 milliseconds Time elapsed: 50 milliseconds </pre>

<<newinfo "The HTTP request takes about 50ms on average.">>

<div class="nav"> [[Go back -> Client]] </div> You decide to use curl to make the same POST request that <code>slow.js</code> is making, to see if you get a different result.

<<ask "What command line arguments do you need to pass to curl to make a POST request?" "$curl_cmd" "curl 2">><<you-said "The curl command you want to run:" $curl_cmd>>

This game only knows one option, so you make a POST request with:

<pre> $ time curl -d 'buy some milk' http://mysterybox:8000 a real 0m0.018s user 0m0.007s sys 0m0.010s </pre>

What do you think this means? Did using curl give us a different result?

<div class="yesno"> [[Yes->curl 3]] [[No->curl 3]] </div> The curl process took 18ms to run, which is faster than 50ms! The total time to make the request was less than that, because there's some overhead to starting the curl process. <<newinfo "curl is much faster: it takes less than 18ms to make the request">>

<div class="nav"> [[Go back -> Client]] </div>You ping the server to see how far away it is.

<pre> $ ping mysterybox PING mysterybox (1.2.3.4) 56(84) bytes of data. 64 bytes from mysterybox (1.2.3.4): icmp_seq=1 ttl=62 time=0.956 ms 64 bytes from mysterybox (1.2.3.4): icmp_seq=2 ttl=62 time=0.610 ms 64 bytes from mysterybox (1.2.3.4): icmp_seq=3 ttl=62 time=0.540 ms 64 bytes from mysterybox (1.2.3.4): icmp_seq=4 ttl=62 time=0.634 ms 64 bytes from mysterybox (1.2.3.4): icmp_seq=5 ttl=62 time=0.536 ms 64 bytes from mysterybox (1.2.3.4): icmp_seq=6 ttl=62 time=0.426 ms 64 bytes from mysterybox (1.2.3.4): icmp_seq=7 ttl=62 time=0.583 ms </pre>

What do you think? Is the request slow because the server is too far away?

<div class="yesno"> [[Yes->ping 2]] [[No->ping 2]] </div> On average the ping time is only 0.5ms and the requests we're seeing are 40ms, so that's probably not the issue. The client and the server machines are both in the same datacenter.

<<newinfo "The server is only 0.5ms away">> <div class="nav"> [[Go back -> Client]] </div> You know that the <code>-t</code> flag will show you the timestamp for every system call, so you <code>strace -t</code> the slow program.

Your goal here is to identify the system requests where it's making the HTTP request and getting the response. It's a POST request

<pre> $ strace -tt node slow.js 13:43:25.428436 execve("/usr/bin/node", ["node", "slow.js"], 0x7ffd97b262a0 /* 24 vars */) = 0 13:43:25.428859 brk(NULL) = 0x52d4000 13:43:25.428919 arch_prctl(0x3001 /* ARCH_??? */, 0x7ffebd9f3d50) = -1 EINVAL (Invalid argument) 13:43:25.429039 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) 13:43:25.429138 openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 13:43:25.429247 fstat(3, {st_mode=S_IFREG|0644, st_size=27066, ...}) = 0 13:43:25.429293 mmap(NULL, 27066, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f1c832c1000 13:43:25.429333 close(3) = 0 13:43:25.429373 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3 13:43:25.429417 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 \22\0\0\0\0\0\0"..., 832) = 832 13:43:25.429470 fstat(3, {st_mode=S_IFREG|0644, st_size=18816, ...}) = 0 13:43:25.429527 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1c832bf000 13:43:25.429590 mmap(NULL, 20752, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f1c832b9000 13:43:25.429647 mmap(0x7f1c832ba000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x7f1c832ba000 13:43:25.429709 mmap(0x7f1c832bc000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f1c832bc000 13:43:25.429748 mmap(0x7f1c832bd000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f1c832bd000 13:43:25.429836 close(3) = 0 13:43:25.429887 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = 3 13:43:25.429945 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240\341\t\0\0\0\0\0"..., 832) = 832 13:43:25.429984 fstat(3, {st_mode=S_IFREG|0644, st_size=1952928, ...}) = 0 13:43:25.430034 mmap(NULL, 1968128, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f1c830d8000 13:43:25.430092 mprotect(0x7f1c8316e000, 1286144, PROT_NONE) = 0 13:43:25.430152 mmap(0x7f1c8316e000, 983040, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x96000) = 0x7f1c8316e000 13:43:25.430209 mmap(0x7f1c8325e000, 299008, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x186000) = 0x7f1c8325e000 13:43:25.430247 mmap(0x7f1c832a8000, 57344, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1cf000) = 0x7f1c832a8000 13:43:25.430294 mmap(0x7f1c832b6000, 10240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f1c832b6000 13:43:25.430345 close(3) = 0 13:43:25.430392 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libm.so.6", O_RDONLY|O_CLOEXEC) = 3 13:43:25.430441 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300\363\0\0\0\0\0\0"..., 832) = 832 13:43:25.430480 fstat(3, {st_mode=S_IFREG|0644, st_size=1369352, ...}) = 0 13:43:25.430518 mmap(NULL, 1368336, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f1c82f89000 13:43:25.430555 mmap(0x7f1c82f98000, 684032, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xf000) = 0x7f1c82f98000 13:43:25.430605 mmap(0x7f1c8303f000, 618496, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xb6000) = 0x7f1c8303f000 13:43:25.430686 mmap(0x7f1c830d6000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x14c000) = 0x7f1c830d6000 13:43:25.430781 close(3) = 0 13:43:25.430854 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libgcc_s.so.1", O_RDONLY|O_CLOEXEC) = 3 13:43:25.430915 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\3405\0\0\0\0\0\0"..., 832) = 832 13:43:25.430960 fstat(3, {st_mode=S_IFREG|0644, st_size=104984, ...}) = 0 13:43:25.431017 mmap(NULL, 107592, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f1c82f6e000 13:43:25.431061 mmap(0x7f1c82f71000, 73728, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f1c82f71000 13:43:25.431103 mmap(0x7f1c82f83000, 16384, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15000) = 0x7f1c82f83000 13:43:25.431139 mmap(0x7f1c82f87000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x18000) = 0x7f1c82f87000 13:43:25.431193 close(3) = 0 13:43:25.431245 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3 13:43:25.431299 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220\201\0\0\0\0\0\0"..., 832) = 832 13:43:25.431427 pread64(3, "\4\0\0\0\24\0\0\0\3\0\0\0GNU\0\345Ga\367\265T\320\374\301V)Yf]\223\337"..., 68, 824) = 68 13:43:25.431477 fstat(3, {st_mode=S_IFREG|0755, st_size=157224, ...}) = 0 13:43:25.431516 pread64(3, "\4\0\0\0\24\0\0\0\3\0\0\0GNU\0\345Ga\367\265T\320\374\301V)Yf]\223\337"..., 68, 824) = 68 13:43:25.431553 mmap(NULL, 140408, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f1c82f4b000 13:43:25.431591 mmap(0x7f1c82f52000, 69632, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x7000) = 0x7f1c82f52000 13:43:25.431634 mmap(0x7f1c82f63000, 20480, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x18000) = 0x7f1c82f63000 13:43:25.431671 mmap(0x7f1c82f68000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c000) = 0x7f1c82f68000 13:43:25.431717 mmap(0x7f1c82f6a000, 13432, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f1c82f6a000 13:43:25.431768 close(3) = 0 13:43:25.431809 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3 13:43:25.431857 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360q\2\0\0\0\0\0"..., 832) = 832 13:43:25.431895 pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784 13:43:25.431930 pread64(3, "\4\0\0\0\20\0\0\0\5\0\0\0GNU\0\2\0\0\300\4\0\0\0\3\0\0\0\0\0\0\0", 32, 848) = 32 13:43:25.431965 pread64(3, "\4\0\0\0\24\0\0\0\3\0\0\0GNU\0\t\233\222%\274\260\320\31\331\326\10\204\276X>\263"..., 68, 880) = 68 13:43:25.432001 fstat(3, {st_mode=S_IFREG|0755, st_size=2029224, ...}) = 0 13:43:25.432038 pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784 13:43:25.432074 pread64(3, "\4\0\0\0\20\0\0\0\5\0\0\0GNU\0\2\0\0\300\4\0\0\0\3\0\0\0\0\0\0\0", 32, 848) = 32 13:43:25.432109 pread64(3, "\4\0\0\0\24\0\0\0\3\0\0\0GNU\0\t\233\222%\274\260\320\31\331\326\10\204\276X>\263"..., 68, 880) = 68 13:43:25.432167 mmap(NULL, 2036952, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f1c82d59000 13:43:25.432239 mprotect(0x7f1c82d7e000, 1847296, PROT_NONE) = 0 13:43:25.432315 mmap(0x7f1c82d7e000, 1540096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x25000) = 0x7f1c82d7e000 13:43:25.432512 mmap(0x7f1c82ef6000, 303104, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19d000) = 0x7f1c82ef6000 13:43:25.432578 mmap(0x7f1c82f41000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1e7000) = 0x7f1c82f41000 13:43:25.432628 mmap(0x7f1c82f47000, 13528, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f1c82f47000 13:43:25.432716 close(3) = 0 13:43:25.432796 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1c82d57000 13:43:25.432856 mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1c82d54000 13:43:25.432900 arch_prctl(ARCH_SET_FS, 0x7f1c82d54780) = 0 13:43:25.433123 mprotect(0x7f1c82f41000, 12288, PROT_READ) = 0 13:43:25.433190 mprotect(0x7f1c82f68000, 4096, PROT_READ) = 0 13:43:25.433244 mprotect(0x7f1c82f87000, 4096, PROT_READ) = 0 13:43:25.433320 mprotect(0x7f1c830d6000, 4096, PROT_READ) = 0 13:43:25.434235 mprotect(0x7f1c832a8000, 45056, PROT_READ) = 0 13:43:25.434307 mprotect(0x7f1c832bd000, 4096, PROT_READ) = 0 13:43:25.434825 mprotect(0x4586000, 12288, PROT_READ) = 0 13:43:25.434887 mprotect(0x7f1c832f5000, 4096, PROT_READ) = 0 13:43:25.434935 munmap(0x7f1c832c1000, 27066) = 0 13:43:25.435017 set_tid_address(0x7f1c82d54a50) = 106154 13:43:25.435073 set_robust_list(0x7f1c82d54a60, 24) = 0 13:43:25.435123 rt_sigaction(SIGRTMIN, {sa_handler=0x7f1c82f52bf0, sa_mask=[], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.435198 rt_sigaction(SIGRT_1, {sa_handler=0x7f1c82f52c90, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.435270 rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0 13:43:25.435339 prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0 13:43:25.435579 brk(NULL) = 0x52d4000 13:43:25.435629 brk(0x52f5000) = 0x52f5000 13:43:25.435772 futex(0x7f1c832b66bc, FUTEX_WAKE_PRIVATE, 2147483647) = 0 13:43:25.435884 futex(0x7f1c832b66c8, FUTEX_WAKE_PRIVATE, 2147483647) = 0 13:43:25.436358 brk(0x5316000) = 0x5316000 13:43:25.436589 futex(0x45a83d0, FUTEX_WAKE_PRIVATE, 2147483647) = 0 13:43:25.436800 getuid() = 0 13:43:25.436849 geteuid() = 0 13:43:25.436905 getgid() = 0 13:43:25.436950 getegid() = 0 13:43:25.436998 rt_sigprocmask(SIG_SETMASK, [USR1], NULL, 8) = 0 13:43:25.437065 fstat(0, {st_mode=S_IFCHR|0600, st_rdev=makedev(0x88, 0), ...}) = 0 13:43:25.437111 fstat(1, {st_mode=S_IFCHR|0600, st_rdev=makedev(0x88, 0), ...}) = 0 13:43:25.437147 fstat(2, {st_mode=S_IFCHR|0600, st_rdev=makedev(0x88, 0), ...}) = 0 13:43:25.437182 rt_sigaction(SIGHUP, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.437220 rt_sigaction(SIGINT, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.437255 rt_sigaction(SIGQUIT, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.437289 rt_sigaction(SIGILL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.437323 rt_sigaction(SIGTRAP, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.437357 rt_sigaction(SIGABRT, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.437391 rt_sigaction(SIGBUS, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.437424 rt_sigaction(SIGFPE, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.437458 rt_sigaction(SIGUSR1, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.437492 rt_sigaction(SIGSEGV, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.437525 rt_sigaction(SIGUSR2, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.437559 rt_sigaction(SIGPIPE, {sa_handler=SIG_IGN, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.437593 rt_sigaction(SIGALRM, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.437627 rt_sigaction(SIGTERM, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.437661 rt_sigaction(SIGSTKFLT, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.437695 rt_sigaction(SIGCHLD, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.437729 rt_sigaction(SIGCONT, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.437762 rt_sigaction(SIGTSTP, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.437803 rt_sigaction(SIGTTIN, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.437872 rt_sigaction(SIGTTOU, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.437912 rt_sigaction(SIGURG, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.437947 rt_sigaction(SIGXCPU, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.437981 rt_sigaction(SIGXFSZ, {sa_handler=SIG_IGN, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.438026 rt_sigaction(SIGVTALRM, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.438062 rt_sigaction(SIGPROF, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.438095 rt_sigaction(SIGWINCH, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.438129 rt_sigaction(SIGIO, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.438163 rt_sigaction(SIGPWR, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.438196 rt_sigaction(SIGSYS, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.438235 fcntl(0, F_GETFL) = 0x2 (flags O_RDWR) 13:43:25.438271 ioctl(0, TCGETS, {B38400 opost isig icanon echo ...}) = 0 13:43:25.438316 ioctl(0, TCGETS, {B38400 opost isig icanon echo ...}) = 0 13:43:25.438352 fcntl(1, F_GETFL) = 0x2 (flags O_RDWR) 13:43:25.438383 ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0 13:43:25.438418 ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0 13:43:25.438453 fcntl(2, F_GETFL) = 0x2 (flags O_RDWR) 13:43:25.438484 ioctl(2, TCGETS, {B38400 opost isig icanon echo ...}) = 0 13:43:25.438518 ioctl(2, TCGETS, {B38400 opost isig icanon echo ...}) = 0 13:43:25.438553 rt_sigaction(SIGINT, {sa_handler=0xa4dfb0, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_RESETHAND, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.438590 rt_sigaction(SIGTERM, {sa_handler=0xa4dfb0, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_RESETHAND, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.438625 rt_sigaction(SIGSEGV, {sa_handler=0xa4dfd0, sa_mask=[], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.438732 prlimit64(0, RLIMIT_NOFILE, NULL, {rlim_cur=1024, rlim_max=1024*1024}) = 0 13:43:25.438803 prlimit64(0, RLIMIT_NOFILE, {rlim_cur=1024*1024, rlim_max=1024*1024}, NULL) = 0 13:43:25.438988 ioctl(0, FIOCLEX) = 0 13:43:25.439067 ioctl(1, FIOCLEX) = 0 13:43:25.439132 ioctl(2, FIOCLEX) = 0 13:43:25.439167 ioctl(3, FIOCLEX) = -1 EBADF (Bad file descriptor) 13:43:25.439205 ioctl(4, FIOCLEX) = -1 EBADF (Bad file descriptor) 13:43:25.439237 ioctl(5, FIOCLEX) = -1 EBADF (Bad file descriptor) 13:43:25.439268 ioctl(6, FIOCLEX) = -1 EBADF (Bad file descriptor) 13:43:25.439299 ioctl(7, FIOCLEX) = -1 EBADF (Bad file descriptor) 13:43:25.439330 ioctl(8, FIOCLEX) = -1 EBADF (Bad file descriptor) 13:43:25.439412 ioctl(9, FIOCLEX) = -1 EBADF (Bad file descriptor) 13:43:25.439454 ioctl(10, FIOCLEX) = -1 EBADF (Bad file descriptor) 13:43:25.439485 ioctl(11, FIOCLEX) = -1 EBADF (Bad file descriptor) 13:43:25.439516 ioctl(12, FIOCLEX) = -1 EBADF (Bad file descriptor) 13:43:25.439547 ioctl(13, FIOCLEX) = -1 EBADF (Bad file descriptor) 13:43:25.439578 ioctl(14, FIOCLEX) = -1 EBADF (Bad file descriptor) 13:43:25.439608 ioctl(15, FIOCLEX) = -1 EBADF (Bad file descriptor) 13:43:25.439639 ioctl(16, FIOCLEX) = -1 EBADF (Bad file descriptor) 13:43:25.439673 getuid() = 0 13:43:25.439703 geteuid() = 0 13:43:25.439732 getgid() = 0 13:43:25.439769 getegid() = 0 13:43:25.439803 getuid() = 0 13:43:25.439835 geteuid() = 0 13:43:25.439864 getgid() = 0 13:43:25.439893 getegid() = 0 13:43:25.439922 getuid() = 0 13:43:25.439960 geteuid() = 0 13:43:25.440018 getgid() = 0 13:43:25.440073 getegid() = 0 13:43:25.440112 getuid() = 0 13:43:25.440143 geteuid() = 0 13:43:25.440172 getgid() = 0 13:43:25.440200 getegid() = 0 13:43:25.440230 getuid() = 0 13:43:25.440259 geteuid() = 0 13:43:25.440288 getgid() = 0 13:43:25.440328 getegid() = 0 13:43:25.440415 getuid() = 0 13:43:25.440460 geteuid() = 0 13:43:25.440500 getgid() = 0 13:43:25.440530 getegid() = 0 13:43:25.440600 futex(0x45af060, FUTEX_WAKE_PRIVATE, 2147483647) = 0 13:43:25.440874 getuid() = 0 13:43:25.440917 geteuid() = 0 13:43:25.440947 getgid() = 0 13:43:25.440976 getegid() = 0 13:43:25.441039 epoll_create1(EPOLL_CLOEXEC) = 3 13:43:25.441084 pipe2([4, 5], O_CLOEXEC) = 0 13:43:25.441133 write(5, "*", 1) = 1 13:43:25.441173 futex(0x45a83c8, FUTEX_WAKE_PRIVATE, 2147483647) = 0 13:43:25.441205 pipe2([6, 7], O_NONBLOCK|O_CLOEXEC) = 0 13:43:25.441252 eventfd2(0, EFD_CLOEXEC|EFD_NONBLOCK) = 8 13:43:25.441299 prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0 13:43:25.441345 mmap(NULL, 8392704, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f1c82553000 13:43:25.441388 mprotect(0x7f1c82554000, 8388608, PROT_READ|PROT_WRITE) = 0 13:43:25.441445 clone(child_stack=0x7f1c82d52f70, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tid=[106155], tls=0x7f1c82d53700, child_tidptr=0x7f1c82d539d0) = 106155 13:43:25.441533 futex(0x53144b0, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY) = 0 13:43:25.441629 prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0 13:43:25.441670 mmap(NULL, 8392704, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f1c81d52000 13:43:25.441717 mprotect(0x7f1c81d53000, 8388608, PROT_READ|PROT_WRITE) = 0 13:43:25.441817 clone(child_stack=0x7f1c82551f70, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tid=[106156], tls=0x7f1c82552700, child_tidptr=0x7f1c825529d0) = 106156 13:43:25.441920 prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0 13:43:25.441985 mmap(NULL, 8392704, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f1c81551000 13:43:25.442055 mprotect(0x7f1c81552000, 8388608, PROT_READ|PROT_WRITE) = 0 13:43:25.442119 clone(child_stack=0x7f1c81d50f70, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tid=[106157], tls=0x7f1c81d51700, child_tidptr=0x7f1c81d519d0) = 106157 13:43:25.442188 prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0 13:43:25.442227 mmap(NULL, 8392704, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f1c80d50000 13:43:25.442263 mprotect(0x7f1c80d51000, 8388608, PROT_READ|PROT_WRITE) = 0 13:43:25.442307 clone(child_stack=0x7f1c8154ff70, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tid=[106158], tls=0x7f1c81550700, child_tidptr=0x7f1c815509d0) = 106158 13:43:25.442373 prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0 13:43:25.442411 mmap(NULL, 8392704, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f1c8054f000 13:43:25.442445 mprotect(0x7f1c80550000, 8388608, PROT_READ|PROT_WRITE) = 0 13:43:25.442491 clone(child_stack=0x7f1c80d4ef70, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tid=[106159], tls=0x7f1c80d4f700, child_tidptr=0x7f1c80d4f9d0) = 106159 13:43:25.442551 futex(0x7ffebd9f3a58, FUTEX_WAIT_PRIVATE, 0, NULL) = 0 13:43:25.442616 futex(0x7ffebd9f3a00, FUTEX_WAIT_PRIVATE, 2, NULL) = 0 13:43:25.442722 futex(0x7ffebd9f3a00, FUTEX_WAKE_PRIVATE, 1) = 0 13:43:25.442757 futex(0x7ffebd9f3a5c, FUTEX_WAIT_PRIVATE, 0, NULL) = 0 13:43:25.442799 futex(0x7ffebd9f3a00, FUTEX_WAIT_PRIVATE, 2, NULL) = 0 13:43:25.442834 futex(0x7ffebd9f3a00, FUTEX_WAKE_PRIVATE, 1) = 0 13:43:25.442863 futex(0x7ffebd9f3a58, FUTEX_WAIT_PRIVATE, 0, NULL) = 0 13:43:25.442913 futex(0x7ffebd9f3a00, FUTEX_WAIT_PRIVATE, 2, NULL) = 0 13:43:25.442948 futex(0x7ffebd9f3a00, FUTEX_WAKE_PRIVATE, 1) = 0 13:43:25.442978 futex(0x7ffebd9f3a5c, FUTEX_WAIT_PRIVATE, 0, NULL) = 0 13:43:25.443016 futex(0x7ffebd9f3a00, FUTEX_WAIT_PRIVATE, 2, NULL) = 0 13:43:25.443098 futex(0x7ffebd9f3a00, FUTEX_WAKE_PRIVATE, 1) = 0 13:43:25.443293 brk(0x5337000) = 0x5337000 13:43:25.443593 epoll_create1(EPOLL_CLOEXEC) = 13 13:43:25.443645 pipe2([14, 15], O_NONBLOCK|O_CLOEXEC) = 0 13:43:25.443693 eventfd2(0, EFD_CLOEXEC|EFD_NONBLOCK) = 16 13:43:25.443827 brk(0x5358000) = 0x5358000 13:43:25.443947 openat(AT_FDCWD, "/sys/fs/cgroup/memory/memory.limit_in_bytes", O_RDONLY|O_CLOEXEC) = 17 13:43:25.444011 read(17, "9223372036854771712\n", 31) = 20 13:43:25.444056 close(17) = 0 13:43:25.444102 openat(AT_FDCWD, "/proc/meminfo", O_RDONLY|O_CLOEXEC) = 17 13:43:25.444162 read(17, "MemTotal: 1004892 kB\nMemF"..., 4095) = 1447 13:43:25.444216 close(17) = 0 13:43:25.444593 mmap(NULL, 1269760, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1c80419000 13:43:25.452826 brk(0x5379000) = 0x5379000 13:43:25.453035 brk(0x53a1000) = 0x53a1000 13:43:25.453210 getpid() = 106154 13:43:25.453322 futex(0x45ab448, FUTEX_WAKE_PRIVATE, 2147483647) = 0 13:43:25.453372 futex(0x45ab308, FUTEX_WAKE_PRIVATE, 2147483647) = 0 13:43:25.453406 futex(0x45ab2fc, FUTEX_WAKE_PRIVATE, 2147483647) = 0 13:43:25.453436 futex(0x45ab2f4, FUTEX_WAKE_PRIVATE, 2147483647) = 0 13:43:25.453467 getpid() = 106154 13:43:25.453521 getrandom("\x0f\x19\xe8\x2c\xc1\x47\xfd\x9b\xcc\xb5\x44\xfa\xe4\x71\xf9\x97\xe2\xfa\x9a\x5f\x99\x31\x95\xc3\xee\x94\xd1\xbd\xdc\xd4\xef\x54", 32, 0) = 32 13:43:25.453572 getpid() = 106154 13:43:25.453609 futex(0x45ab418, FUTEX_WAKE_PRIVATE, 2147483647) = 0 13:43:25.453647 getpid() = 106154 13:43:25.453680 getpid() = 106154 13:43:25.453717 getpid() = 106154 13:43:25.453747 getpid() = 106154 13:43:25.453791 mmap(0x143ab233b000, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x143ab233b000 13:43:25.453928 brk(0x53c2000) = 0x53c2000 13:43:25.454030 mmap(0x122891b40000, 520192, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x122891b40000 13:43:25.454070 munmap(0x122891b80000, 258048) = 0 13:43:25.454108 mprotect(0x122891b40000, 262144, PROT_READ|PROT_WRITE) = 0 13:43:25.454492 getpid() = 106154 13:43:25.454530 getpid() = 106154 13:43:25.454613 munmap(0x122891b68000, 98304) = 0 13:43:25.454657 mprotect(0x122891b40000, 163840, PROT_READ) = 0 13:43:25.454740 mmap(0x128c1d200000, 520192, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x128c1d200000 13:43:25.454799 munmap(0x128c1d240000, 258048) = 0 13:43:25.454875 mprotect(0x128c1d200000, 262144, PROT_READ|PROT_WRITE) = 0 13:43:25.454965 mmap(0xdb6ece00000, 520192, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0xdb6ece00000 13:43:25.455024 munmap(0xdb6ece40000, 258048) = 0 13:43:25.455063 mprotect(0xdb6ece00000, 262144, PROT_READ|PROT_WRITE) = 0 13:43:25.455113 mmap(0x31c8a35c0000, 520192, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x31c8a35c0000 13:43:25.455150 munmap(0x31c8a3600000, 258048) = 0 13:43:25.455183 mprotect(0x31c8a35c0000, 262144, PROT_READ|PROT_WRITE) = 0 13:43:25.455253 mmap(0x2047fb9c0000, 520192, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x2047fb9c0000 13:43:25.455333 munmap(0x2047fba00000, 258048) = 0 13:43:25.455410 mprotect(0x2047fb9c0000, 262144, PROT_READ|PROT_WRITE) = 0 13:43:25.455499 getpid() = 106154 13:43:25.455536 getpid() = 106154 13:43:25.455629 mmap(0x73f40340000, 520192, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x73f40340000 13:43:25.455672 munmap(0x73f40380000, 258048) = 0 13:43:25.455707 mprotect(0x73f40340000, 262144, PROT_READ|PROT_WRITE) = 0 13:43:25.455837 mmap(0x1b89e9140000, 520192, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x1b89e9140000 13:43:25.455875 munmap(0x1b89e9180000, 258048) = 0 13:43:25.455909 mprotect(0x1b89e9140000, 262144, PROT_READ|PROT_WRITE) = 0 13:43:25.456048 mmap(0x281e95b80000, 520192, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x281e95b80000 13:43:25.456086 munmap(0x281e95bc0000, 258048) = 0 13:43:25.456120 mprotect(0x281e95b80000, 262144, PROT_READ|PROT_WRITE) = 0 13:43:25.456245 mmap(0x1ee9fc840000, 520192, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x1ee9fc840000 13:43:25.456500 munmap(0x1ee9fc880000, 258048) = 0 13:43:25.456584 mprotect(0x1ee9fc840000, 262144, PROT_READ|PROT_WRITE) = 0 13:43:25.456762 mmap(0x3be87a680000, 520192, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x3be87a680000 13:43:25.456822 munmap(0x3be87a6c0000, 258048) = 0 13:43:25.456876 mprotect(0x3be87a680000, 262144, PROT_READ|PROT_WRITE) = 0 13:43:25.457174 mmap(0x73e0700000, 520192, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x73e0700000 13:43:25.457246 munmap(0x73e0740000, 258048) = 0 13:43:25.457310 mprotect(0x73e0700000, 262144, PROT_READ|PROT_WRITE) = 0 13:43:25.457466 mprotect(0x143ab2340000, 262144, PROT_NONE) = 0 13:43:25.457537 madvise(0x143ab2340000, 262144, MADV_DONTNEED) = 0 13:43:25.457581 mprotect(0x143ab2340000, 8192, PROT_READ|PROT_WRITE) = 0 13:43:25.457643 mprotect(0x143ab2342000, 4096, PROT_NONE) = 0 13:43:25.457678 madvise(0x143ab2342000, 4096, MADV_DONTNEED) = 0 13:43:25.457709 mprotect(0x143ab2343000, 245760, PROT_READ|PROT_WRITE) = 0 13:43:25.457743 mprotect(0x143ab237f000, 4096, PROT_NONE) = 0 13:43:25.457773 madvise(0x143ab237f000, 4096, MADV_DONTNEED) = 0 13:43:25.457892 mmap(0x2d9c12500000, 520192, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x2d9c12500000 13:43:25.457933 munmap(0x2d9c12540000, 258048) = 0 13:43:25.457968 mprotect(0x2d9c12500000, 262144, PROT_READ|PROT_WRITE) = 0 13:43:25.459937 brk(0x53e3000) = 0x53e3000 13:43:25.460946 brk(0x5405000) = 0x5405000 13:43:25.463313 munmap(0x73e0713000, 184320) = 0 13:43:25.463471 mprotect(0x143ab2358000, 4096, PROT_NONE) = 0 13:43:25.463552 madvise(0x143ab2358000, 4096, MADV_DONTNEED) = 0 13:43:25.463609 mprotect(0x143ab2359000, 159744, PROT_NONE) = 0 13:43:25.463662 madvise(0x143ab2359000, 159744, MADV_DONTNEED) = 0 13:43:25.463715 munmap(0x2d9c12502000, 253952) = 0 13:43:25.463775 mprotect(0x143ab2343000, 86016, PROT_READ|PROT_EXEC) = 0 13:43:25.463939 munmap(0x7f1c80419000, 1269760) = 0 13:43:25.464228 readlink("/proc/self/exe", "/usr/bin/node", 8191) = 13 13:43:25.464389 getuid() = 0 13:43:25.464447 geteuid() = 0 13:43:25.464495 getgid() = 0 13:43:25.464542 getegid() = 0 13:43:25.464616 futex(0x459c348, FUTEX_WAKE_PRIVATE, 2147483647) = 0 13:43:25.464701 mmap(0x17589f780000, 520192, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x17589f780000 13:43:25.464780 munmap(0x17589f7c0000, 258048) = 0 13:43:25.464852 mprotect(0x17589f780000, 262144, PROT_READ|PROT_WRITE) = 0 13:43:25.464979 brk(0x5462000) = 0x5462000 13:43:25.467093 mmap(0x2df2b0dc0000, 520192, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x2df2b0dc0000 13:43:25.467229 munmap(0x2df2b0e00000, 258048) = 0 13:43:25.467293 mprotect(0x2df2b0dc0000, 262144, PROT_READ|PROT_WRITE) = 0 13:43:25.467510 mmap(0x39ced1100000, 520192, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x39ced1100000 13:43:25.467580 munmap(0x39ced1140000, 258048) = 0 13:43:25.467645 mprotect(0x39ced1100000, 262144, PROT_READ|PROT_WRITE) = 0 13:43:25.467947 mmap(0x22642200000, 520192, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x22642200000 13:43:25.468003 munmap(0x22642240000, 258048) = 0 13:43:25.468041 mprotect(0x22642200000, 262144, PROT_READ|PROT_WRITE) = 0 13:43:25.472392 getpid() = 106154 13:43:25.472495 futex(0x45a8488, FUTEX_WAKE_PRIVATE, 2147483647) = 0 13:43:25.472585 rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1], [USR1], 8) = 0 13:43:25.472673 mmap(NULL, 36864, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f1c80546000 13:43:25.472742 mprotect(0x7f1c80547000, 32768, PROT_READ|PROT_WRITE) = 0 13:43:25.472852 clone(child_stack=0x7f1c8054df70, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tid=[106160], tls=0x7f1c8054e700, child_tidptr=0x7f1c8054e9d0) = 106160 13:43:25.472972 rt_sigprocmask(SIG_SETMASK, [USR1], NULL, 8) = 0 13:43:25.473046 rt_sigaction(SIGUSR1, {sa_handler=0xb8b6a0, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.473123 rt_sigprocmask(SIG_UNBLOCK, [USR1], NULL, 8) = 0 13:43:25.476209 getpid() = 106154 13:43:25.476722 getcwd("/root/code/40ms", 4096) = 16 13:43:25.481023 mmap(0x3c877ed80000, 520192, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x3c877ed80000 13:43:25.481120 munmap(0x3c877edc0000, 258048) = 0 13:43:25.481165 mprotect(0x3c877ed80000, 262144, PROT_READ|PROT_WRITE) = 0 13:43:25.486878 getuid() = 0 13:43:25.486975 geteuid() = 0 13:43:25.487050 getgid() = 0 13:43:25.487085 getegid() = 0 13:43:25.487132 getuid() = 0 13:43:25.487164 geteuid() = 0 13:43:25.487193 getgid() = 0 13:43:25.487222 getegid() = 0 13:43:25.488557 statx(AT_FDCWD, "/root/code/40ms/slow.js", AT_STATX_SYNC_AS_STAT, STATX_ALL, {stx_mask=STATX_ALL, stx_attributes=0, stx_mode=S_IFREG|0644, stx_size=618, ...}) = 0 13:43:25.489956 statx(AT_FDCWD, "/root", AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW, STATX_ALL, {stx_mask=STATX_ALL, stx_attributes=0, stx_mode=S_IFDIR|0700, stx_size=4096, ...}) = 0 13:43:25.490180 statx(AT_FDCWD, "/root/code", AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW, STATX_ALL, {stx_mask=STATX_ALL, stx_attributes=0, stx_mode=S_IFDIR|0755, stx_size=4096, ...}) = 0 13:43:25.490272 statx(AT_FDCWD, "/root/code/40ms", AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW, STATX_ALL, {stx_mask=STATX_ALL, stx_attributes=0, stx_mode=S_IFDIR|0755, stx_size=4096, ...}) = 0 13:43:25.490348 statx(AT_FDCWD, "/root/code/40ms/slow.js", AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW, STATX_ALL, {stx_mask=STATX_ALL, stx_attributes=0, stx_mode=S_IFREG|0644, stx_size=618, ...}) = 0 13:43:25.490863 openat(AT_FDCWD, "/root/code/40ms/package.json", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 13:43:25.491030 openat(AT_FDCWD, "/root/code/package.json", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 13:43:25.491135 openat(AT_FDCWD, "/root/package.json", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 13:43:25.491212 openat(AT_FDCWD, "/package.json", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 13:43:25.493921 openat(AT_FDCWD, "/root/code/40ms/slow.js", O_RDONLY|O_CLOEXEC) = 17 13:43:25.494079 statx(17, "", AT_STATX_SYNC_AS_STAT|AT_EMPTY_PATH, STATX_ALL, {stx_mask=STATX_ALL, stx_attributes=0, stx_mode=S_IFREG|0644, stx_size=618, ...}) = 0 13:43:25.494887 read(17, "const http = require('http')\ncon"..., 618) = 618 13:43:25.495095 close(17) = 0 13:43:25.496612 mprotect(0x143ab2380000, 262144, PROT_NONE) = 0 13:43:25.496673 madvise(0x143ab2380000, 262144, MADV_DONTNEED) = 0 13:43:25.496826 mprotect(0x143ab2380000, 8192, PROT_READ|PROT_WRITE) = 0 13:43:25.496871 mprotect(0x143ab2382000, 4096, PROT_NONE) = 0 13:43:25.496903 madvise(0x143ab2382000, 4096, MADV_DONTNEED) = 0 13:43:25.496934 mprotect(0x143ab2383000, 245760, PROT_READ|PROT_WRITE) = 0 13:43:25.496968 mprotect(0x143ab23bf000, 4096, PROT_NONE) = 0 13:43:25.496999 madvise(0x143ab23bf000, 4096, MADV_DONTNEED) = 0 13:43:25.497058 mprotect(0x143ab2383000, 245760, PROT_READ|PROT_WRITE) = 0 13:43:25.497099 mprotect(0x143ab2383000, 245760, PROT_READ|PROT_EXEC) = 0 13:43:25.498423 mmap(0x14b869000000, 520192, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x14b869000000 13:43:25.498499 munmap(0x14b869040000, 258048) = 0 13:43:25.498586 mprotect(0x14b869000000, 262144, PROT_READ|PROT_WRITE) = 0 13:43:25.504639 brk(0x5484000) = 0x5484000 13:43:25.505711 mprotect(0x143ab2383000, 245760, PROT_READ|PROT_WRITE) = 0 13:43:25.505820 mprotect(0x143ab2383000, 245760, PROT_READ|PROT_EXEC) = 0 13:43:25.506952 brk(0x54a8000) = 0x54a8000 13:43:25.507656 brk(0x54d0000) = 0x54d0000 13:43:25.510067 openat(AT_FDCWD, "/dev/null", O_RDONLY|O_CLOEXEC) = 17 13:43:25.510742 mprotect(0x143ab2383000, 245760, PROT_READ|PROT_WRITE) = 0 13:43:25.510824 mprotect(0x143ab2383000, 245760, PROT_READ|PROT_EXEC) = 0 13:43:25.511509 mprotect(0x143ab2383000, 245760, PROT_READ|PROT_WRITE) = 0 13:43:25.511618 mprotect(0x143ab2383000, 245760, PROT_READ|PROT_EXEC) = 0 13:43:25.512488 openat(AT_FDCWD, "/etc/resolv.conf", O_RDONLY) = 18 13:43:25.512581 fstat(18, {st_mode=S_IFREG|0644, st_size=708, ...}) = 0 13:43:25.512631 read(18, "# This file is managed by man:sy"..., 4096) = 708 13:43:25.512697 read(18, "", 4096) = 0 13:43:25.512748 close(18) = 0 13:43:25.512821 openat(AT_FDCWD, "/etc/nsswitch.conf", O_RDONLY) = 18 13:43:25.512877 fstat(18, {st_mode=S_IFREG|0644, st_size=510, ...}) = 0 13:43:25.512918 read(18, "# /etc/nsswitch.conf\n#\n# Example"..., 4096) = 510 13:43:25.512963 read(18, "", 4096) = 0 13:43:25.512997 close(18) = 0 13:43:25.513031 uname({sysname="Linux", nodename="mysterybox-client", ...}) = 0 13:43:25.513070 openat(AT_FDCWD, "/dev/urandom", O_RDONLY) = 18 13:43:25.513131 fstat(18, {st_mode=S_IFCHR|0666, st_rdev=makedev(0x1, 0x9), ...}) = 0 13:43:25.513173 ioctl(18, TCGETS, 0x7ffebd9f1e60) = -1 EINVAL (Invalid argument) 13:43:25.513216 read(18, "z\250\0033$\224\366\240#\t\264O\332\376\t\303V<\202\204\376\262!%F\204\232\1V\356\1\260"..., 4096) = 4096 13:43:25.513323 close(18) = 0 13:43:25.514525 prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0 13:43:25.514626 mmap(NULL, 8392704, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f1c7b7ff000 13:43:25.514725 mprotect(0x7f1c7b800000, 8388608, PROT_READ|PROT_WRITE) = 0 13:43:25.514815 clone(child_stack=0x7f1c7bffef70, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tid=[106161], tls=0x7f1c7bfff700, child_tidptr=0x7f1c7bfff9d0) = 106161 13:43:25.514915 prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0 13:43:25.514956 mmap(NULL, 8392704, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f1c7affe000 13:43:25.514994 mprotect(0x7f1c7afff000, 8388608, PROT_READ|PROT_WRITE) = 0 13:43:25.515059 clone(child_stack=0x7f1c7b7fdf70, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tid=[106162], tls=0x7f1c7b7fe700, child_tidptr=0x7f1c7b7fe9d0) = 106162 13:43:25.515197 prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0 13:43:25.515268 mmap(NULL, 8392704, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f1c7a7fd000 13:43:25.515328 mprotect(0x7f1c7a7fe000, 8388608, PROT_READ|PROT_WRITE) = 0 13:43:25.515496 clone(child_stack=0x7f1c7affcf70, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tid=[106163], tls=0x7f1c7affd700, child_tidptr=0x7f1c7affd9d0) = 106163 13:43:25.515625 prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0 13:43:25.515694 mmap(NULL, 8392704, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f1c79ffc000 13:43:25.515752 mprotect(0x7f1c79ffd000, 8388608, PROT_READ|PROT_WRITE) = 0 13:43:25.515808 clone(child_stack=0x7f1c7a7fbf70, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tid=[106164], tls=0x7f1c7a7fc700, child_tidptr=0x7f1c7a7fc9d0) = 106164 13:43:25.515879 futex(0x7ffebd9f1e40, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY) = 0 13:43:25.515940 futex(0x7ffebd9f1e40, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY) = 0 13:43:25.515981 futex(0x7ffebd9f1e40, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY) = 0 13:43:25.516019 futex(0x7ffebd9f1e40, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY) = 0 13:43:25.516058 futex(0x45a8010, FUTEX_WAKE_PRIVATE, 2147483647) = 0 13:43:25.516089 futex(0x45a8008, FUTEX_WAKE_PRIVATE, 1) = 1 13:43:25.516958 futex(0x45a8008, FUTEX_WAKE_PRIVATE, 1) = 1 13:43:25.520021 epoll_ctl(13, EPOLL_CTL_ADD, 14, {EPOLLIN, {u32=14, u64=14}}) = 0 13:43:25.520100 epoll_ctl(13, EPOLL_CTL_ADD, 16, {EPOLLIN, {u32=16, u64=16}}) = 0 13:43:25.520141 epoll_wait(13, [{EPOLLIN, {u32=16, u64=16}}], 1024, 0) = 1 13:43:25.520181 read(16, "\1\0\0\0\0\0\0\0", 1024) = 8 13:43:25.520585 socket(AF_INET, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 18 13:43:25.520649 connect(18, {sa_family=AF_INET, sin_port=htons(8000), sin_addr=inet_addr("1.2.3.4")}, 16) = -1 EINPROGRESS (Operation now in progress) 13:43:25.521263 mprotect(0x143ab2383000, 245760, PROT_READ|PROT_WRITE) = 0 13:43:25.521336 mprotect(0x143ab2383000, 245760, PROT_READ|PROT_EXEC) = 0 13:43:25.522913 futex(0x45a8008, FUTEX_WAKE_PRIVATE, 1) = 1 13:43:25.522979 futex(0x45a7fa0, FUTEX_WAKE_PRIVATE, 1) = 1 13:43:25.523063 epoll_ctl(13, EPOLL_CTL_ADD, 18, {EPOLLOUT, {u32=18, u64=18}}) = 0 13:43:25.523117 epoll_wait(13, [{EPOLLIN, {u32=16, u64=16}}, {EPOLLOUT, {u32=18, u64=18}}], 1024, 0) = 2 13:43:25.523158 read(16, "\1\0\0\0\0\0\0\0", 1024) = 8 13:43:25.523642 futex(0x45a800c, FUTEX_WAKE_PRIVATE, 1) = 1 13:43:25.523701 futex(0x45a7fa0, FUTEX_WAKE_PRIVATE, 1) = 1 13:43:25.523766 getsockopt(18, SOL_SOCKET, SO_ERROR, [0], [4]) = 0 13:43:25.524529 write(18, "POST / HTTP/1.1\r\nHost: mysterybo"..., 107) = 107 13:43:25.524845 mmap(0x3ea6a6e40000, 520192, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x3ea6a6e40000 13:43:25.524906 munmap(0x3ea6a6e80000, 258048) = 0 13:43:25.524948 mprotect(0x3ea6a6e40000, 262144, PROT_READ|PROT_WRITE) = 0 13:43:25.525314 epoll_ctl(13, EPOLL_CTL_ADD, 18, {EPOLLIN, {u32=18, u64=18}}) = -1 EEXIST (File exists) 13:43:25.525400 epoll_ctl(13, EPOLL_CTL_MOD, 18, {EPOLLIN, {u32=18, u64=18}}) = 0 13:43:25.525457 epoll_wait(13, [{EPOLLIN, {u32=16, u64=16}}], 1024, 0) = 1 13:43:25.525510 read(16, "\1\0\0\0\0\0\0\0", 1024) = 8 13:43:25.525730 futex(0x45a800c, FUTEX_WAKE_PRIVATE, 1) = 1 13:43:25.525801 futex(0x45a7fa0, FUTEX_WAKE_PRIVATE, 1) = 1 13:43:25.525884 epoll_wait(13, [{EPOLLIN, {u32=16, u64=16}}], 1024, 0) = 1 13:43:25.525931 read(16, "\1\0\0\0\0\0\0\0", 1024) = 8 13:43:25.526175 futex(0x45a800c, FUTEX_WAKE_PRIVATE, 1) = 1 13:43:25.526272 futex(0x45a7fa0, FUTEX_WAKE_PRIVATE, 1) = 1 13:43:25.526398 epoll_wait(13, [{EPOLLIN, {u32=16, u64=16}}], 1024, 0) = 1 13:43:25.526473 read(16, "\1\0\0\0\0\0\0\0", 1024) = 8 13:43:25.527839 write(18, "buy some milk\n", 14) = 14 13:43:25.528095 epoll_ctl(13, EPOLL_CTL_MOD, 18, {EPOLLIN, {u32=18, u64=18}}) = 0 13:43:25.528154 epoll_wait(13, [], 1024, 0) = 0 13:43:25.528190 epoll_wait(13, [{EPOLLIN|EPOLLERR|EPOLLHUP, {u32=18, u64=18}}], 1024, -1) = 1 13:43:25.568621 read(18, "HTTP/1.1 200 OK\r\nContent-Length:"..., 65536) = 41 13:43:25.572025 ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0 13:43:25.573094 ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0 13:43:25.573197 fcntl(1, F_GETFL) = 0x2 (flags O_RDWR) 13:43:25.573260 ioctl(1, TIOCGPTN, 0x7ffebd9ebd60) = -1 ENOTTY (Inappropriate ioctl for device) 13:43:25.573333 ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0 13:43:25.573398 fstat(1, {st_mode=S_IFCHR|0600, st_rdev=makedev(0x88, 0), ...}) = 0 13:43:25.573469 readlink("/proc/self/fd/1", "/dev/pts/0", 255) = 10 13:43:25.573586 stat("/dev/pts/0", {st_mode=S_IFCHR|0600, st_rdev=makedev(0x88, 0), ...}) = 0 13:43:25.573667 openat(AT_FDCWD, "/dev/pts/0", O_RDWR|O_NOCTTY|O_CLOEXEC) = 19 13:43:25.573760 dup3(19, 1, O_CLOEXEC) = 1 13:43:25.573844 ioctl(19, FIONBIO, [1]) = 0 13:43:25.574385 ioctl(19, FIONBIO, [0]) = 0 13:43:25.574486 ioctl(19, TIOCGWINSZ, {ws_row=36, ws_col=79, ws_xpixel=0, ws_ypixel=0}) = 0 13:43:25.574765 rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1], [], 8) = 0 13:43:25.574913 read(4, "*", 1) = 1 13:43:25.574990 rt_sigaction(SIGWINCH, {sa_handler=0x148a710, sa_mask=~[RTMIN RT_1], sa_flags=SA_RESTORER|SA_RESTART, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.575170 write(5, "*", 1) = 1 13:43:25.575249 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 13:43:25.577648 write(19, "Time elapsed: \33[33m50\33[39m milli"..., 40) = 40 13:43:25.581054 shutdown(18, SHUT_WR) = -1 ENOTCONN (Transport endpoint is not connected) 13:43:25.582292 ioctl(2, TCGETS, {B38400 opost isig icanon echo ...}) = 0 13:43:25.582444 ioctl(2, TCGETS, {B38400 opost isig icanon echo ...}) = 0 13:43:25.582526 fcntl(2, F_GETFL) = 0x2 (flags O_RDWR) 13:43:25.582587 ioctl(2, TIOCGPTN, 0x7ffebd9ef6c0) = -1 ENOTTY (Inappropriate ioctl for device) 13:43:25.582656 ioctl(2, TCGETS, {B38400 opost isig icanon echo ...}) = 0 13:43:25.582754 fstat(2, {st_mode=S_IFCHR|0600, st_rdev=makedev(0x88, 0), ...}) = 0 13:43:25.582827 readlink("/proc/self/fd/2", "/dev/pts/0", 255) = 10 13:43:25.582939 stat("/dev/pts/0", {st_mode=S_IFCHR|0600, st_rdev=makedev(0x88, 0), ...}) = 0 13:43:25.583020 openat(AT_FDCWD, "/dev/pts/0", O_RDWR|O_NOCTTY|O_CLOEXEC) = 20 13:43:25.583122 dup3(20, 2, O_CLOEXEC) = 2 13:43:25.583182 ioctl(20, FIONBIO, [1]) = 0 13:43:25.583477 ioctl(20, FIONBIO, [0]) = 0 13:43:25.583577 ioctl(20, TIOCGWINSZ, {ws_row=36, ws_col=79, ws_xpixel=0, ws_ypixel=0}) = 0 13:43:25.583736 epoll_ctl(13, EPOLL_CTL_DEL, 18, 0x7ffebd9efb14) = 0 13:43:25.583815 close(18) = 0 13:43:25.585254 fstat(0, {st_mode=S_IFCHR|0600, st_rdev=makedev(0x88, 0), ...}) = 0 13:43:25.585363 fcntl(0, F_GETFL) = 0x2 (flags O_RDWR) 13:43:25.585423 rt_sigprocmask(SIG_BLOCK, [TTOU], NULL, 8) = 0 13:43:25.585488 ioctl(0, TCGETS, {B38400 opost isig icanon echo ...}) = 0 13:43:25.585554 ioctl(0, SNDCTL_TMR_START or TCSETS, {B38400 opost isig icanon echo ...}) = 0 13:43:25.585618 ioctl(0, TCGETS, {B38400 opost isig icanon echo ...}) = 0 13:43:25.585681 rt_sigprocmask(SIG_UNBLOCK, [TTOU], NULL, 8) = 0 13:43:25.585741 fstat(1, {st_mode=S_IFCHR|0600, st_rdev=makedev(0x88, 0), ...}) = 0 13:43:25.585803 fcntl(1, F_GETFL) = 0x8002 (flags O_RDWR|O_LARGEFILE) 13:43:25.585857 rt_sigprocmask(SIG_BLOCK, [TTOU], NULL, 8) = 0 13:43:25.585916 ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0 13:43:25.585979 ioctl(1, SNDCTL_TMR_START or TCSETS, {B38400 opost isig icanon echo ...}) = 0 13:43:25.586039 ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0 13:43:25.586100 rt_sigprocmask(SIG_UNBLOCK, [TTOU], NULL, 8) = 0 13:43:25.586160 fstat(2, {st_mode=S_IFCHR|0600, st_rdev=makedev(0x88, 0), ...}) = 0 13:43:25.586221 fcntl(2, F_GETFL) = 0x8002 (flags O_RDWR|O_LARGEFILE) 13:43:25.586277 rt_sigprocmask(SIG_BLOCK, [TTOU], NULL, 8) = 0 13:43:25.586342 ioctl(2, TCGETS, {B38400 opost isig icanon echo ...}) = 0 13:43:25.586404 ioctl(2, SNDCTL_TMR_START or TCSETS, {B38400 opost isig icanon echo ...}) = 0 13:43:25.586465 ioctl(2, TCGETS, {B38400 opost isig icanon echo ...}) = 0 13:43:25.586525 rt_sigprocmask(SIG_UNBLOCK, [TTOU], NULL, 8) = 0 13:43:25.586584 rt_sigaction(SIGHUP, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.586652 rt_sigaction(SIGINT, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.586745 rt_sigaction(SIGQUIT, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.586810 rt_sigaction(SIGILL, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.586868 rt_sigaction(SIGTRAP, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.586928 rt_sigaction(SIGABRT, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.587013 rt_sigaction(SIGBUS, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.587082 rt_sigaction(SIGFPE, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.587141 rt_sigaction(SIGUSR1, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.587201 rt_sigaction(SIGSEGV, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.587260 rt_sigaction(SIGUSR2, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.587317 rt_sigaction(SIGPIPE, {sa_handler=SIG_IGN, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.587441 rt_sigaction(SIGALRM, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.587510 rt_sigaction(SIGTERM, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.587567 rt_sigaction(SIGSTKFLT, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.587628 rt_sigaction(SIGCHLD, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.587687 rt_sigaction(SIGCONT, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.587746 rt_sigaction(SIGTSTP, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.587803 rt_sigaction(SIGTTIN, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.587861 rt_sigaction(SIGTTOU, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.587920 rt_sigaction(SIGURG, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.587983 rt_sigaction(SIGXCPU, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.588044 rt_sigaction(SIGXFSZ, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.588105 rt_sigaction(SIGVTALRM, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.588162 rt_sigaction(SIGWINCH, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.588221 rt_sigaction(SIGIO, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.588280 rt_sigaction(SIGPWR, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.588338 rt_sigaction(SIGSYS, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.588429 epoll_ctl(13, EPOLL_CTL_DEL, 19, 0x7ffebd9f3754) = -1 ENOENT (No such file or directory) 13:43:25.588502 close(19) = 0 13:43:25.588559 rt_sigprocmask(SIG_SETMASK, ~[RTMIN RT_1], [], 8) = 0 13:43:25.588630 read(4, "*", 1) = 1 13:43:25.588699 rt_sigaction(SIGWINCH, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f1c82f603c0}, NULL, 8) = 0 13:43:25.588763 write(5, "*", 1) = 1 13:43:25.588823 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 13:43:25.588885 epoll_ctl(13, EPOLL_CTL_DEL, 20, 0x7ffebd9f3754) = -1 ENOENT (No such file or directory) 13:43:25.588943 close(20) = 0 13:43:25.589004 epoll_wait(13, [], 1024, 0) = 0 13:43:25.589157 epoll_wait(13, [], 1024, 0) = 0 13:43:25.589432 mprotect(0x2047fb9c0000, 262144, PROT_NONE) = 0 13:43:25.589526 madvise(0x2047fb9c0000, 262144, MADV_DONTNEED) = 0 13:43:25.589618 mprotect(0x31c8a35c0000, 262144, PROT_NONE) = 0 13:43:25.589685 madvise(0x31c8a35c0000, 262144, MADV_DONTNEED) = 0 13:43:25.589777 mprotect(0xdb6ece00000, 262144, PROT_NONE) = 0 13:43:25.589862 madvise(0xdb6ece00000, 262144, MADV_DONTNEED) = 0 13:43:25.589964 mprotect(0x128c1d200000, 262144, PROT_NONE) = 0 13:43:25.590031 madvise(0x128c1d200000, 262144, MADV_DONTNEED) = 0 13:43:25.590138 munmap(0x73f40340000, 262144) = 0 13:43:25.590312 munmap(0x1b89e9140000, 262144) = 0 13:43:25.590426 munmap(0x281e95b80000, 262144) = 0 13:43:25.590543 munmap(0x1ee9fc840000, 262144) = 0 13:43:25.590684 munmap(0x3be87a680000, 262144) = 0 13:43:25.590810 munmap(0x73e0700000, 77824) = 0 13:43:25.590907 munmap(0x2df2b0dc0000, 262144) = 0 13:43:25.591029 munmap(0x39ced1100000, 262144) = 0 13:43:25.591148 munmap(0x22642200000, 262144) = 0 13:43:25.591267 munmap(0x3c877ed80000, 262144) = 0 13:43:25.591440 munmap(0x14b869000000, 262144) = 0 13:43:25.591566 munmap(0x3ea6a6e40000, 262144) = 0 13:43:25.591715 mprotect(0x143ab2340000, 262144, PROT_NONE) = 0 13:43:25.591816 madvise(0x143ab2340000, 262144, MADV_DONTNEED) = 0 13:43:25.591934 mprotect(0x143ab2380000, 262144, PROT_NONE) = 0 13:43:25.592007 madvise(0x143ab2380000, 262144, MADV_DONTNEED) = 0 13:43:25.592108 munmap(0x2d9c12500000, 8192) = 0 13:43:25.592191 munmap(0x17589f780000, 262144) = 0 13:43:25.592310 munmap(0x122891b40000, 163840) = 0 13:43:25.592418 munmap(0x128c1d200000, 262144) = 0 13:43:25.592495 munmap(0xdb6ece00000, 262144) = 0 13:43:25.592569 munmap(0x31c8a35c0000, 262144) = 0 13:43:25.592639 munmap(0x2047fb9c0000, 262144) = 0 13:43:25.592718 munmap(0x143ab233b000, 134217728) = 0 13:43:25.593003 futex(0x5314190, FUTEX_WAKE_PRIVATE, 2147483647) = 4 13:43:25.593117 futex(0x5314140, FUTEX_WAKE_PRIVATE, 1) = 1 13:43:25.593415 write(12, "\1\0\0\0\0\0\0\0", 8) = 8 13:43:25.595572 futex(0x7f1c82d539d0, FUTEX_WAIT, 106155, NULL) = -1 EAGAIN (Resource temporarily unavailable) 13:43:25.595824 munmap(0x7f1c82553000, 8392704) = 0 13:43:25.595933 epoll_ctl(3, EPOLL_CTL_ADD, 6, {EPOLLIN, {u32=6, u64=6}}) = 0 13:43:25.596014 epoll_ctl(3, EPOLL_CTL_ADD, 8, {EPOLLIN, {u32=8, u64=8}}) = 0 13:43:25.596084 epoll_wait(3, [], 1024, 0) = 0 13:43:25.596143 close(6) = 0 13:43:25.596203 close(7) = 0 13:43:25.596265 close(8) = 0 13:43:25.596326 close(3) = 0 13:43:25.596408 futex(0x45ab1c4, FUTEX_WAKE_PRIVATE, 2147483647) = 0 13:43:25.596503 fstat(0, {st_mode=S_IFCHR|0600, st_rdev=makedev(0x88, 0), ...}) = 0 13:43:25.596583 fcntl(0, F_GETFL) = 0x2 (flags O_RDWR) 13:43:25.596641 rt_sigprocmask(SIG_BLOCK, [TTOU], NULL, 8) = 0 13:43:25.596703 ioctl(0, TCGETS, {B38400 opost isig icanon echo ...}) = 0 13:43:25.596781 ioctl(0, SNDCTL_TMR_START or TCSETS, {B38400 opost isig icanon echo ...}) = 0 13:43:25.596849 ioctl(0, TCGETS, {B38400 opost isig icanon echo ...}) = 0 13:43:25.596915 rt_sigprocmask(SIG_UNBLOCK, [TTOU], NULL, 8) = 0 13:43:25.596978 fstat(1, {st_mode=S_IFCHR|0600, st_rdev=makedev(0x88, 0), ...}) = 0 13:43:25.597043 fcntl(1, F_GETFL) = 0x8002 (flags O_RDWR|O_LARGEFILE) 13:43:25.597099 rt_sigprocmask(SIG_BLOCK, [TTOU], NULL, 8) = 0 13:43:25.597158 ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0 13:43:25.597220 ioctl(1, SNDCTL_TMR_START or TCSETS, {B38400 opost isig icanon echo ...}) = 0 13:43:25.597282 ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0 13:43:25.597343 rt_sigprocmask(SIG_UNBLOCK, [TTOU], NULL, 8) = 0 13:43:25.597403 fstat(2, {st_mode=S_IFCHR|0600, st_rdev=makedev(0x88, 0), ...}) = 0 13:43:25.597469 fcntl(2, F_GETFL) = 0x8002 (flags O_RDWR|O_LARGEFILE) 13:43:25.597524 rt_sigprocmask(SIG_BLOCK, [TTOU], NULL, 8) = 0 13:43:25.597585 ioctl(2, TCGETS, {B38400 opost isig icanon echo ...}) = 0 13:43:25.597649 ioctl(2, SNDCTL_TMR_START or TCSETS, {B38400 opost isig icanon echo ...}) = 0 13:43:25.597709 ioctl(2, TCGETS, {B38400 opost isig icanon echo ...}) = 0 13:43:25.597770 rt_sigprocmask(SIG_UNBLOCK, [TTOU], NULL, 8) = 0 13:43:25.598277 close(4) = 0 13:43:25.598354 close(5) = 0 13:43:25.598420 futex(0x45a800c, FUTEX_WAKE_PRIVATE, 1) = 1 13:43:25.598488 futex(0x45a7fa0, FUTEX_WAKE_PRIVATE, 1) = 1 13:43:25.598758 munmap(0x7f1c81d52000, 8392704) = 0 13:43:25.598876 munmap(0x7f1c81551000, 8392704) = 0 13:43:25.598961 munmap(0x7f1c80d50000, 8392704) = 0 13:43:25.599038 munmap(0x7f1c8054f000, 8392704) = 0 13:43:25.599222 exit_group(0) = ? 13:43:25.601138 +++ exited with 0 +++ </pre>

<<ask-area "Which lines from the above nonsense are making the HTTP request and receiving the HTTP response? It's a POST request, the response is a 200 OK, and the POST request body is 'buy some milk'." "$strace" "strace interpretation">> <<if !$strace eq "">> The lines you highlighted were: <pre><<print $strace>></pre> <</if>>

Here are the lines that make the HTTP request and show the HTTP response: <pre> 13:43:25.524529 write(18, "POST / HTTP/1.1\r\nHost: mysterybo"..., 107) = 107 13:43:25.527839 write(18, "buy some milk\n", 14) = 14 13:43:25.568621 read(18, "HTTP/1.1 200 OK\r\nContent-Length:"..., 65536) = 41 </pre>

<<ask "How long did it take to receive the HTTP response after the request was sent?" "$time_to_response" "strace interpretation 2">><<you-said "You said it took this long to get a HTTP response:" $time_to_response>>

Let's do some calculations using this output:

<pre> 13:43:25.524529 write(18, "POST / HTTP/1.1\r\nHost: mysterybo"..., 107) = 107 13:43:25.527839 write(18, "buy some milk\n", 14) = 14 13:43:25.568621 read(18, "HTTP/1.1 200 OK\r\nContent-Length:"..., 65536) = 41 </pre>

The difference between <code>13:43:25.568621</code> (line 3) and <code>13:43:25.527839</code> (line 2) is 41 milliseconds (568 - 527 = 41).

<<newinfo "strace shows it took 41ms to receive a response">>

<<ask "Why do you think this 41ms delay is happening?" "$why_delay" "strace 3">><<you-said "Your reason for the delay:" $why_delay>>

From just the strace information, there's no real way to tell why this delay is happening. The server could just be doing a big calculation that takes 40ms! Or maybe the server had to make a database query!

To know for sure what's happening, we'll need to get more information.

<div class="nav go-back"> [[Go back -> Client]] </div>You decide to use tcpdump to look at the TCP packets on port 8000.

<<ask "What tcpdump command will show you packets on port 8000?" "$tcpdump_command" "tcpdump 2">><<you-said "The tcpdump command you said to run:" $tcpdump_command>>

You decide to run this command:

<pre> $ sudo tcpdump -i any port 8000 14:56:11.240271 IP mysterybox-client.48126 > mysterybox.8000: Flags [S], seq 2428508727, win 64240, options [mss 1460,sackOK,TS val 1858170944 ecr 0,nop,wscale 7], length 0 14:56:11.242504 IP mysterybox.8000 > mysterybox-client.48126: Flags [S.], seq 3452350398, ack 2428508728, win 65160, options [mss 1460,sackOK,TS val 1885475954 ecr 1858170944,nop,wscale 7], length 0 14:56:11.242539 IP mysterybox-client.48126 > mysterybox.8000: Flags [.], ack 1, win 502, options [nop,nop,TS val 1858170946 ecr 1885475954], length 0 14:56:11.250852 IP mysterybox-client.48126 > mysterybox.8000: Flags [P.], seq 1:108, ack 1, win 502, options [nop,nop,TS val 1858170955 ecr 1885475954], length 107 14:56:11.294309 IP mysterybox.8000 > mysterybox-client.48126: Flags [.], ack 108, win 509, options [nop,nop,TS val 1885476006 ecr 1858170955], length 0 14:56:11.294388 IP mysterybox-client.48126 > mysterybox.8000: Flags [P.], seq 108:122, ack 1, win 502, options [nop,nop,TS val 1858170998 ecr 1885476006], length 14 14:56:11.294738 IP mysterybox.8000 > mysterybox-client.48126: Flags [.], ack 122, win 509, options [nop,nop,TS val 1885476007 ecr 1858170998], length 0 14:56:11.295013 IP mysterybox.8000 > mysterybox-client.48126: Flags [P.], seq 1:42, ack 122, win 509, options [nop,nop,TS val 1885476007 ecr 1858170998], length 41 14:56:11.295025 IP mysterybox-client.48126 > mysterybox.8000: Flags [.], ack 42, win 502, options [nop,nop,TS val 1858170999 ecr 1885476007], length 0 14:56:11.295050 IP mysterybox.8000 > mysterybox-client.48126: Flags [R.], seq 42, ack 122, win 509, options [nop,nop,TS val 1885476007 ecr 1858170998], length 0 14:56:11.295241 IP mysterybox.8000 > mysterybox-client.48126: Flags [R], seq 3452350440, win 0, length 0 </pre>

There's not a lot of detail here about exactly which data is being sent back and forth. Hmm.

<div class="nav"> [[Look at the packets in more detail -> tcpdump 3]] </div>The <code>-A</code> flag to tcpdump shows you the content of each packet.

<pre> $ sudo tcpdump -A -i any port 8000 16:13:09.074136 IP 5.6.7.8.48568 > mysterybox.8000: Flags [S], seq 2007520623, win 64240, options [mss 1460,sackOK,TS val 2381188778 ecr 0,nop,wscale 7], length 0 E..<Qc@.>[email protected]................... ............ 16:13:09.074189 IP mysterybox.8000 > 5.6.7.8.48568: Flags [S.], seq 607465505, ack 2007520624, win 65160, options [mss 1460,sackOK,TS val 2408493786 ecr 2381188778,nop,wscale 7], length 0 E..<..@[email protected].@..$50!w.Up............... ............ 16:13:09.074476 IP 5.6.7.8.48568 > mysterybox.8000: Flags [.], ack 1, win 502, options [nop,nop,TS val 2381188779 ecr 2408493786], length 0 E..4Qd@.>[email protected]$50"........... ........ 16:13:09.079810 IP 5.6.7.8.48568 > mysterybox.8000: Flags [P.], seq 1:108, ack 1, win 502, options [nop,nop,TS val 2381188784 ecr 2408493786], length 107 E...Qe@.>[email protected]$50"........... ........POST / HTTP/1.1 Host: mysterybox:8000 Content-Type: text/plain Content-Length: 14 Connection: close 16:13:09.122094 IP mysterybox.8000 > 5.6.7.8.48568: Flags [.], ack 108, win 509, options [nop,nop,TS val 2408493834 ecr 2381188784], length 0 E..4..@[email protected].@..$50"w.U............ ... .... 16:13:09.122608 IP 5.6.7.8.48568 > mysterybox.8000: Flags [P.], seq 108:122, ack 1, win 502, options [nop,nop,TS val 2381188827 ecr 2408493834], length 14 E..BQf@.>[email protected].$50".....w..... ....... buy some milk 16:13:09.122623 IP mysterybox.8000 > 5.6.7.8.48568: Flags [.], ack 122, win 509, options [nop,nop,TS val 2408493835 ecr 2381188827], length 0 E..4..@[email protected].@..$50"w.U............ ........ 16:13:09.122896 IP mysterybox.8000 > 5.6.7.8.48568: Flags [P.], seq 1:42, ack 122, win 509, options [nop,nop,TS val 2408493835 ecr 2381188827], length 41 E..]..@[email protected].@..$50"w.U............ ........HTTP/1.1 200 OK Content-Length: 1 a 16:13:09.122931 IP mysterybox.8000 > 5.6.7.8.48568: Flags [R.], seq 42, ack 122, win 509, options [nop,nop,TS val 2408493835 ecr 2381188827], length 0 E..4..@[email protected].@..$50Kw.U............ ........ 16:13:09.123177 IP 5.6.7.8.48568 > mysterybox.8000: Flags [.], ack 42, win 502, options [nop,nop,TS val 2381188827 ecr 2408493835], length 0 E..4Qg@.>[email protected].$50K........... ........ 16:13:09.123197 IP mysterybox.8000 > 5.6.7.8.48568: Flags [R], seq 607465547, win 0, length 0 E..(..@[email protected].@..$50K....P....... </pre>

<<ask "There's a delay of about 44ms here, from 16:13:09.079810 to 16:13:09.122094. What is it waiting for?" "$why_delay" "termshark interpretation">> You run : <pre> $ sudo termshark -f 'port 8000' </pre> to capture all traffic on port 8000. You then run <code>node slow.js</code> in a separate terminal, and here's what you see:

<pre style="font-size: 0.8em"> No. - Time - Source - Destination - Protocol - Length - Info - ▲ 1 0.000000000 5.6.7.8 1.2.3.4 TCP 74 55710 → 8000 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 2 0.001808336 1.2.3.4 5.6.7.8 TCP 74 8000 → 55710 [SYN, ACK] Seq=0 Ack=1 Win=65160 Len=0 MSS=1460 3 0.001852765 5.6.7.8 1.2.3.4 TCP 66 55710 → 8000 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=18565210 4 0.015515742 5.6.7.8 1.2.3.4 TCP 173 POST / HTTP/1.1 [TCP segment of a reassembled PDU] 5 0.058616727 1.2.3.4 5.6.7.8 TCP 66 8000 → 55710 [ACK] Seq=1 Ack=108 Win=65152 Len=0 TSval=188382 6 0.058652284 5.6.7.8 1.2.3.4 HTTP 80 POST / HTTP/1.1 (text/plain) █ 7 0.059055154 1.2.3.4 5.6.7.8 TCP 66 8000 → 55710 [ACK] Seq=1 Ack=122 Win=65152 Len=0 TSval=188382 8 0.059094804 1.2.3.4 5.6.7.8 HTTP 107 HTTP/1.1 200 OK Continuation 9 0.059104365 5.6.7.8 1.2.3.4 TCP 66 55710 → 8000 [ACK] Seq=122 Ack=42 Win=64256 Len=0 TSval=18565 </pre>

<<ask "There's a delay of about 43ms here. What is it waiting for?" "$why_delay" "termshark interpretation">><<you-said "You said it was waiting for:" $why_delay>>

Here's the overall timeline of what's happening.

<ol>

<li> There's a TCP handshake. It takes about 2ms to finish. </li> <li> 14ms later, the client sends part of a POST request (the headers) </li> <li> 43ms later, the server sends an ACK packet </li> <li> After the ACK, the client immediately sends the body of the POST request </li> <li> 1.4ms later, the server responds with a HTTP/1.1 200 OK response </li> </ol>

So the 43ms wait is for an ACK packet for the server.

<<newinfo "The client spends 41ms waiting for the server to send an ACK packet">>

What does this mean? Is this happening because the Python server code is slow to process the HTTP request?

<div class="yesno"> [[Yes->termshark 3]] [[No->termshark 3]] </div> This is not happening because the Python code is slow to process the HTTP request. After the POST request gets sent, the server replies almost immediately (within 2ms). The problem is that the client is unable to actually finish sending the POST request because it's waiting for the server to send an ACK packet.

<div class="nav"> [[Make a guess at why this is happening->diagnosis]] [[Go back and keep investigating->Client]] </div><<ask "Why do you think the client is waiting 41ms for the server to send an ACK?" "$nagle" "nagle">>

(warning: the next page will tell you what's happening, click "go back" if you want to keep investigating!)

<div class="nav go-back"> [[Go back -> Reproduce]] </div><<you-said "You said the reason the server is slow to send the ACK is: " $nagle>>

This is going to be a pretty big leap of logic if you haven't seen it before.

The Linux kernel's TCP implementation has a feature called <a href="https://en.wikipedia.org/wiki/TCP_delayed_acknowledgment">delayed acknowledgement</a> or "delayed ACKs". Basically the idea behind delayed ACKs is to not ACK every packet as an optimization. So instead of this:

<html> <?xml version="1.0" encoding="UTF-8" standalone="no"?> <svg xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:cc="http://creativecommons.org/ns#" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:svg="http://www.w3.org/2000/svg" xmlns="http://www.w3.org/2000/svg" xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd" xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" version="1.1" id="svg1274" width="244.89557" height="118.48486"> <metadata id="metadata1280"> <rdf:RDF> <cc:Work rdf:about=""> <dc:format>image/svg+xml</dc:format> <dc:type rdf:resource="http://purl.org/dc/dcmitype/StillImage" /> </cc:Work> </rdf:RDF> </metadata> <defs id="defs1278" /> <sodipodi:namedview pagecolor="#ffffff" bordercolor="#666666" borderopacity="1" objecttolerance="10" gridtolerance="10" guidetolerance="10" inkscape:pageopacity="0" inkscape:pageshadow="2" inkscape:window-width="640" inkscape:window-height="480" id="namedview1276" /> <inkscape:clipboard style="font-variation-settings:normal;opacity:1;vector-effect:none;fill:#000000;fill-opacity:1;stroke-width:1.00000017;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:1.5;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;stop-color:#000000;stop-opacity:1" min="202.57625,63.573074" max="447.47182,182.05793" /> <g id="g1282" transform="matrix(1.33333,0,0,-1.33333,-202.57625,1007.4269)"> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g8" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 219.643,92.381 c -0.811,0.82 0.107,4.957 0.175,5.84" style="fill:none;stroke:#000000;stroke-width:2px" id="path6" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g12" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 225.862,91.626 c 0.429,2.435 0.101,5.073 0.326,7.545" style="fill:none;stroke:#000000;stroke-width:2px" id="path10" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g16" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 216.936,105.927 c -0.493,7.683 12.813,7.772 12.813,0.286" style="fill:none;stroke:#000000;stroke-width:2px" id="path14" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g20" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 210.526,90.4 c -0.094,-0.883 -0.323,1.751 -0.361,2.638 -0.088,2.02 -0.119,4.033 -0.079,6.053 0.083,4.274 0.244,8.551 -0.08,12.819 -0.129,1.707 -0.995,4.607 -0.118,6.251 0.814,1.527 4.049,0.822 5.341,0.752 4.981,-0.269 10.005,-0.554 14.994,-0.554 1.092,0 4.44,0.992 5.302,0.237 2.045,-1.789 0.623,-9.792 0.791,-12.027 0.389,-5.148 0.382,-10.314 0.633,-15.469 0.056,-1.15 0.737,-4.167 -0.593,-4.906 -1.674,-0.93 -5.226,0.588 -6.963,0.712 -4.241,0.303 -8.569,0.43 -12.819,0.593 -3.356,0.129 -6.843,-1.291 -6.488,2.968" style="fill:none;stroke:#000000;stroke-width:2px" id="path18" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g24" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 208.355,128.777 c 0.154,0.142 -1.926,0.104 -1.949,0.106 -1.062,0.076 -1.964,1.041 -2.413,1.938 -1.329,2.657 0.554,4.493 3.244,4.194" style="fill:none;stroke:#000000;stroke-width:2px" id="path22" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g28" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 212.879,124.566 c 0.149,-0.334 -0.445,0.629 -0.459,0.993 -0.031,0.812 -0.015,1.642 0,2.453 0.048,2.638 0.108,5.46 0.751,8.032" style="fill:none;stroke:#000000;stroke-width:2px" id="path26" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g32" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 217.507,130.662 c -0.568,0.228 0.41,3.919 0.491,4.353" style="fill:none;stroke:#000000;stroke-width:2px" id="path30" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g36" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 217.783,126.892 c 0.013,0.001 -0.37,0.411 -0.457,0.527" style="fill:none;stroke:#000000;stroke-width:2px" id="path34" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g40" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 222.247,132.734 c -0.085,0.094 1.778,0.105 1.923,0.105 1.043,0 2.662,0.186 3.482,-0.633 0.98,-0.98 -1.064,-2.121 -1.701,-2.492 -0.586,-0.342 -1.273,-0.752 -1.979,-0.752 -2.317,0 -2.378,3.733 -1.82,5.222 0.998,2.662 3.831,3.16 6.331,2.968" style="fill:none;stroke:#000000;stroke-width:2px" id="path38" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g44" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 232.592,130.287 c 0.073,0.788 -0.206,1.712 -0.232,2.513 -0.017,0.523 0.005,0.778 0.039,1.305 0.358,5.54 0.015,-3.075 0.554,-4.154 2.317,-4.634 4.706,4.763 4.867,6.211" style="fill:none;stroke:#000000;stroke-width:2px" id="path42" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g48" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 240.67,123.841 c -0.048,-0.465 -0.002,0.935 -0.002,1.402 0,0.78 0.01,1.554 0.04,2.334 0.078,2.066 -0.24,11.03 3.956,7.596" style="fill:none;stroke:#000000;stroke-width:2px" id="path46" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g52" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 237.496,127.991 c 0.502,0.584 6.851,-1.094 7.682,-1.205" style="fill:none;stroke:#000000;stroke-width:2px" id="path50" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,57.9612,790.806)" id="g104" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 331.887,94.294 c 0.862,6.066 0.107,13.803 -0.48,19.924 -0.123,1.285 -1.076,5.933 0.335,6.847 3.656,2.365 11.736,0.069 15.763,-0.031 2.271,-0.056 4.675,-0.13 6.938,0.122 0.683,0.076 1.726,0.799 2.343,0.304 1.335,-1.067 0.651,-4.274 0.548,-5.66 -0.383,-5.168 0.29,-10.24 0.548,-15.398 0.08,-1.595 0.886,-4.818 -0.883,-5.842 -1.648,-0.955 -4.195,-0.417 -5.994,-0.457 -3.27,-0.072 -6.89,-0.22 -10.073,-0.943 -2.034,-0.462 -3.526,-0.655 -5.599,-0.396 -0.12,0.015 -3.333,0.312 -2.495,0.731" style="fill:none;stroke:#000000;stroke-width:2px" id="path102" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,57.9612,790.806)" id="g108" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 339.844,103.044 c -0.122,0.011 -0.259,0.132 -0.373,0.188" style="fill:none;stroke:#000000;stroke-width:2px" id="path106" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,57.9612,790.806)" id="g112" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 349.397,102.514 c -0.094,-0.031 -0.435,0.399 -0.492,0.475" style="fill:none;stroke:#000000;stroke-width:2px" id="path110" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,57.9612,790.806)" id="g116" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 338.926,111.093 c -0.324,-0.047 0.913,1.409 0.971,1.451 0.815,0.585 1.746,1.116 2.709,1.4 2.382,0.703 10.088,0.837 8.612,-3.591" style="fill:none;stroke:#000000;stroke-width:2px" id="path114" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,51.4657,791.231)" id="g120" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 333.869,130.554 c 0.022,-0.009 -3.664,-0.819 -3.527,1.648 0.125,2.247 8.099,3.342 2.009,4.717 -0.59,0.133 -1.162,0.233 -1.765,0.183" style="fill:none;stroke:#000000;stroke-width:2px" id="path118" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,51.4657,791.231)" id="g124" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 338.467,133.092 c 1.082,2.944 7.559,-3.379 1.857,-3.142 -4.511,0.188 -3.189,7.823 1.247,8.034" style="fill:none;stroke:#000000;stroke-width:2px" id="path122" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,51.4657,791.231)" id="g128" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 346.058,132.464 c -0.573,0.563 0.075,3.325 0.199,3.968 0.017,0.084 0.217,1.584 0.213,1.582 -0.606,-0.202 -0.76,-2.28 -0.821,-2.769 -0.367,-2.936 0.459,-4.621 3.53,-4.26" style="fill:none;stroke:#000000;stroke-width:2px" id="path126" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,51.4657,791.231)" id="g132" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 352.923,132.027 c -0.027,0.815 2.369,6.032 2.95,5.683 0.604,-0.362 0.872,-5.597 1.157,-6.451" style="fill:none;stroke:#000000;stroke-width:2px" id="path130" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,51.4657,791.231)" id="g136" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 361.915,133.7 c 0.272,-0.382 0.891,0.371 1.359,0.361 0.948,-0.021 1.979,-0.188 2.768,-0.755 1.739,-1.248 -0.176,-3.059 -1.617,-3.379 -0.864,-0.192 -1.559,0.496 -2.085,1.078 -2.393,2.652 -1.396,5.773 2.354,6.075 1.452,0.117 2.627,-0.501 3.829,-1.222" style="fill:none;stroke:#000000;stroke-width:2px" id="path134" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,51.4657,791.231)" id="g140" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 370.112,131.216 c -0.054,-0.016 0.328,1.525 0.352,1.605 0.173,0.594 1.549,4.202 1.006,4.745 -0.006,0.006 -0.12,-0.58 -0.126,-0.63 -0.051,-0.429 -0.121,-0.862 -0.161,-1.294 -0.126,-1.343 -0.072,-2.742 0.485,-3.99 1.12,-2.509 4.155,-1.163 6.183,-0.88" style="fill:none;stroke:#000000;stroke-width:2px" id="path138" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g184" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 268.75,75.37 c -0.161,1.083 -2.177,-0.892 -3.23,-0.591 -2.092,0.598 -3.427,2.786 -3.889,4.8 -0.985,4.294 3.451,7.155 5.957,2.979 2.393,-3.988 -0.351,-9.58 -1.437,-13.561" style="fill:none;stroke:#000000;stroke-width:2px" id="path182" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g188" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 276.709,76.51 c 0.363,-0.75 -2.018,-1.99 -2.604,-1.836 -2.814,0.741 -2.028,6.558 -0.911,8.234 2.31,3.465 4.731,-1.764 4.731,-3.854 0,-0.148 -0.158,-2.221 -0.631,-2.103 -0.206,0.052 0.703,1.965 0.736,2.033 0.913,1.87 2.068,4.284 4.415,4.415" style="fill:none;stroke:#000000;stroke-width:2px" id="path186" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g192" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 284.7,67.713 c 0.136,-0.494 -0.433,0.987 -0.363,1.495 0.186,1.354 0.511,2.696 0.806,4.029 0.703,3.183 1.589,6.442 1.822,9.706" style="fill:none;stroke:#000000;stroke-width:2px" id="path190" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g196" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 281.306,75.424 c 1.767,-0.473 4.272,-1.791 6.71,-2.082" style="fill:none;stroke:#000000;stroke-width:2px" id="path194" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g200" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 298.926,74.423 c 0.282,0.123 -0.862,0.26 -1.099,0.181 -0.972,-0.324 -1.915,-0.642 -2.873,-0.035 -1.826,1.156 -2.412,4.236 -1.717,6.167 0.862,2.393 3.293,1.513 4.625,0.105 0.572,-0.605 0.974,-1.327 1.262,-2.103 0.011,-0.029 0.358,-1.775 0.14,-1.611 -1.876,1.407 2.098,5.803 3.013,6.447" style="fill:none;stroke:#000000;stroke-width:2px" id="path198" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g204" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 256.049,83.498 c 0.459,-0.563 0.032,1.456 0.116,2.178 0.157,1.359 0.435,2.774 1.086,3.995 1.517,2.844 6.2,3.359 9.075,3.539 2.949,0.184 6.027,0.214 8.971,-0.035 2.676,-0.227 5.278,-0.91 7.919,-1.367 7.33,-1.268 24.283,-1.678 23.231,-12.719 -0.303,-3.184 -0.45,-7.756 -2.908,-10.127 -3.085,-2.974 -8.286,-2.868 -12.264,-3.258 -9.397,-0.922 -23.62,-2.627 -32.587,1.857 -2.646,1.322 -5.97,4.247 -6.483,7.323 -0.17,1.024 0.922,3.168 0.561,3.889 -0.673,1.346 -4.841,3.296 -6.062,4.661 -1.542,1.723 -2.371,3.649 -3.013,5.851 -0.323,1.107 -0.806,3.364 -0.806,3.364 0,0 0.412,-1.33 0.806,-1.892 0.922,-1.318 2.107,-2.566 3.293,-3.644 3.478,-3.161 6.324,-3.953 10.863,-4.52" style="fill:none;stroke:#000000;stroke-width:2px" id="path202" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g208" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 320.314,103.928 c -0.637,0.769 2.365,-3.702 2.847,-5.112 0.818,-2.389 1.063,-4.882 1.191,-7.393 0.009,-0.173 0.089,-2.698 0.421,-2.698 0.442,0 1.185,3.273 1.296,3.609 1,2.998 1.983,6.007 3.084,8.97 0.264,0.712 1.079,2.776 0.385,3.469" style="fill:none;stroke:#000000;stroke-width:2px" id="path206" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g212" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 323.762,100.059 c 1.122,-0.889 3.127,-1.308 4.585,-1.768" style="fill:none;stroke:#000000;stroke-width:2px" id="path210" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g216" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 342.668,94.131 c 0.187,-0.023 -2.068,-1.902 -2.898,-1.902 -2.011,0 -3.416,2.026 -3.749,3.819 -1.007,5.413 2.311,10.173 7.674,6.342" style="fill:none;stroke:#000000;stroke-width:2px" id="path214" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g220" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 347.15,91.016 c 0.132,-0.468 -0.144,0.974 -0.092,1.458 0.094,0.864 0.185,1.729 0.281,2.593 0.317,2.855 -0.165,6.737 0.911,9.426" style="fill:none;stroke:#000000;stroke-width:2px" id="path218" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g224" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 347.793,99.947 c 0.082,-3.505 5.686,-5.894 8.376,-7.508" style="fill:none;stroke:#000000;stroke-width:2px" id="path222" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g228" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 349.854,97.721 c 1.577,2.919 5.472,5.537 8.207,7.543" style="fill:none;stroke:#000000;stroke-width:2px" id="path226" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g232" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 365.066,100.036 c 0.184,0.705 -0.16,2.312 -0.065,3.149 0.168,1.49 0.731,3.354 0.215,4.827 -1.177,3.365 -7.681,3.696 -10.458,4.238 -9.72,1.897 -19.542,1.327 -29.294,0.157 -5.145,-0.618 -12.263,0.077 -13.636,-6.495 -0.531,-2.541 0.072,-5.331 0.51,-7.848 0.86,-4.943 2.904,-10.237 7.162,-13.185 2.839,-1.966 6.111,-2.154 9.437,-2.453 5.887,-0.53 11.713,-0.646 17.6,-0.078 4.537,0.437 9.393,0.246 13.813,1.451 1.643,0.448 3.888,1.321 4.709,2.963 0.759,1.518 0.373,3.687 0.373,5.337" style="fill:none;stroke:#000000;stroke-width:2px" id="path230" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g236" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 304.106,172.639 c -0.134,0.37 0.556,-1.072 0.619,-1.193 0.551,-1.053 1.132,-2.089 1.609,-3.179 1.076,-2.46 1.442,-5.025 2.06,-7.613 0.106,-0.444 0.395,-2.433 1.119,-2.433 0.869,0 1.751,2.631 2.001,3.218 1.31,3.073 2.355,6.497 2.355,9.87" style="fill:none;stroke:#000000;stroke-width:2px" id="path234" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g240" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 307.224,167.042 c 0.408,0.224 3.955,-1.245 4.486,-1.384" style="fill:none;stroke:#000000;stroke-width:2px" id="path238" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g244" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 324.126,160.45 c -1.004,-2.092 -3.269,-0.205 -4.038,1.107 -3.056,5.219 -0.741,9.752 5.337,8.143" style="fill:none;stroke:#000000;stroke-width:2px" id="path242" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g248" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 330.766,158.017 c -0.87,-0.083 -0.423,2.35 -0.396,2.657 0.282,3.252 -0.052,6.681 0.549,9.889" style="fill:none;stroke:#000000;stroke-width:2px" id="path246" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g252" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 330.925,166.17 c 0.374,-2.691 4.854,-5.143 7.097,-6.006" style="fill:none;stroke:#000000;stroke-width:2px" id="path250" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g256" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 332.293,164.456 c 1.142,1.629 3.039,2.482 4.571,3.733 1.364,1.113 2.511,2.637 4.121,3.394" style="fill:none;stroke:#000000;stroke-width:2px" id="path254" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g260" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 347.903,169.286 c -0.114,2.413 0.341,5.653 -1.071,7.772 -1.246,1.868 -3.513,2.519 -5.612,2.923 -4.474,0.86 -8.946,0.583 -13.48,0.824 -8.58,0.457 -16.384,0.726 -24.231,-3.198 -2.136,-1.068 -4.987,-2.376 -6.338,-4.434 -1.907,-2.906 -2.016,-7.977 -1.727,-11.341 0.604,-7.004 6.303,-10.486 12.793,-11.165 6.167,-0.644 12.349,-0.667 18.542,-0.667 5.391,0 11.079,-0.059 16.344,1.295 3.511,0.903 6.181,4.024 6.181,7.711 0,0.545 0.166,2.424 -0.236,2.826" style="fill:none;stroke:#000000;stroke-width:2px" id="path258" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g264" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 262.521,129.189 c 0.691,1.355 -0.293,3.753 0.609,5.106 0.125,0.187 0.02,-0.451 0.02,-0.676 v -1.391 c 0,-1.34 0.219,-3.676 1.944,-3.929 2.472,-0.362 2.967,3.702 2.967,5.361 0,0.198 0,-0.395 0,-0.593 0,-0.423 -0.011,-0.847 0,-1.269 0.035,-1.353 0.389,-3.324 2.005,-3.622 2.825,-0.52 2.403,4.317 2.619,5.832" style="fill:none;stroke:#000000;stroke-width:2px" id="path262" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g268" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 279.493,128.88 c -3.03,-1.367 -5.011,3.596 -2.777,5.271 2.745,2.058 4.522,-2.178 2.906,-4.256" style="fill:none;stroke:#000000;stroke-width:2px" id="path266" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g272" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 283.573,130.067 c -0.294,0.924 0.949,3.85 0.428,4.371 -0.171,0.17 -0.668,-2.04 -0.675,-2.149 -0.117,-1.835 0.392,-3.495 2.087,-4.379" style="fill:none;stroke:#000000;stroke-width:2px" id="path270" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g276" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 290.127,130.214 c 1.138,0.595 4.678,0.6 4.862,-1.24 0.138,-1.38 -2.794,-1.739 -3.622,-1.452 -4.064,1.406 -2.92,7.663 1.228,7.898" style="fill:none;stroke:#000000;stroke-width:2px" id="path274" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g280" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 309.932,129.174 c -0.77,0.535 -1.929,-0.354 -2.727,0.598 -1.058,1.261 -1.271,6.043 1.105,6.098 1.456,0.034 2.518,-1.625 2.926,-2.824 0.913,-2.684 -1.1,-6.744 -1.801,-9.371" style="fill:none;stroke:#000000;stroke-width:2px" id="path278" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g284" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 318.615,130.342 c 0.569,-0.347 -0.722,-1.561 -1.036,-1.634 -2.118,-0.488 -3.225,2.798 -2.701,4.338 0.798,2.347 3.044,0.016 3.52,-1.186 0.063,-0.161 0.29,-1.332 0.47,-1.392 0.137,-0.045 0.602,1.262 0.635,1.33 0.519,1.085 1.124,1.871 2.23,2.374" style="fill:none;stroke:#000000;stroke-width:2px" id="path282" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g288" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 325.084,124.33 c -0.464,0.332 0.206,2.154 0.291,2.516 0.489,2.07 0.738,4.141 0.86,6.262" style="fill:none;stroke:#000000;stroke-width:2px" id="path286" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g292" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 323.473,129.53 c 1.688,-0.253 3.249,-0.989 5.033,-1.108" style="fill:none;stroke:#000000;stroke-width:2px" id="path290" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g296" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 336.76,128.214 c -2.908,-2.217 -6.238,2.293 -3.896,4.485 2.023,1.891 3.927,-2.537 4.911,-3.274 0.526,-0.395 0.282,1.287 0.512,1.903 0.415,1.114 1.097,1.745 2.148,2.271" style="fill:none;stroke:#000000;stroke-width:2px" id="path294" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g300" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 263.057,122.783 c -0.621,0.356 -1.982,0.12 -2.71,0.319 -1.824,0.497 -3.712,1.741 -4.911,3.192 -3.778,4.574 -2.958,15.261 3.581,16.799 3.55,0.836 7.37,0.553 10.988,0.696 6.604,0.261 13.205,0.606 19.807,0.9 15.118,0.675 31.997,3.859 46.92,0.041 7.096,-1.815 14.6,-13.699 8.328,-19.971 -1.407,-1.407 -3.484,-1.976 -5.32,-2.578 -4.406,-1.445 -9.136,-2.112 -13.69,-2.906 -6.867,-1.197 -13.69,-1.907 -20.666,-1.964 -8.103,-0.066 -15.84,1.006 -23.818,2.169 -1.956,0.285 -3.915,0.684 -5.791,1.31 -0.603,0.201 -1.733,0.941 -2.373,0.941" style="fill:none;stroke:#000000;stroke-width:2px" id="path298" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g304" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 244.488,109.716 c 5.491,-1.456 11.727,3.862 14.877,7.841 1.052,1.329 2.5,2.451 3.089,4.072 0.119,0.326 0.348,1.575 0.839,1.575" style="fill:none;stroke:#000000;stroke-width:2px" id="path302" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g308" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 243.87,108.853 c 0.108,-0.316 1.562,-0.685 1.806,-0.77 2.242,-0.787 4.621,-1.132 6.998,-1.064 6.611,0.189 9.846,3.221 14.282,7.509 2.084,2.014 4.209,3.794 5.525,6.425" style="fill:none;stroke:#000000;stroke-width:2px" id="path306" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g312" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 365.487,99.46 c 1.303,-0.875 5.259,0.746 6.538,1.096 7.698,2.107 13.098,6.208 19.169,11.223 1.811,1.495 3.691,3.393 5.082,5.297 0.456,0.625 0.802,1.329 1.099,2.041 0.144,0.346 0.432,1.415 0.432,1.04" style="fill:none;stroke:#000000;stroke-width:2px" id="path310" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g316" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 366.812,92.617 c 0.227,0.354 0.81,-0.258 1.23,-0.282 0.858,-0.051 1.732,-0.037 2.59,0 2.382,0.102 4.545,0.664 6.789,1.432 8.878,3.041 15.282,9.296 19.13,17.816 0.831,1.84 3.037,5.992 2.021,8.025" style="fill:none;stroke:#000000;stroke-width:2px" id="path314" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g320" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 349.945,162.257 c -0.458,-0.115 0.872,-0.365 1.301,-0.563 0.793,-0.365 1.602,-0.699 2.414,-1.02 2.442,-0.967 4.872,-1.947 7.279,-3.002 8.208,-3.598 17.46,-8.194 23.957,-14.5 1.938,-1.88 3.896,-3.614 5.533,-5.768 0.137,-0.18 1.846,-2.068 1.197,-2.198" style="fill:none;stroke:#000000;stroke-width:2px" id="path318" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g324" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 349.164,168.429 c 0.551,0.368 2.387,-0.33 2.965,-0.436 3.281,-0.605 6.293,-2.074 9.379,-3.277 9.679,-3.773 17.328,-10.321 24.153,-17.933 2.324,-2.592 6.259,-7.012 6.259,-10.831" style="fill:none;stroke:#000000;stroke-width:2px" id="path322" /> </g> </g> </svg> </html>

the server will ACK multiple packets at once, like this:

<html> <?xml version="1.0" encoding="UTF-8" standalone="no"?> <svg class="margin: auto;" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:cc="http://creativecommons.org/ns#" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:svg="http://www.w3.org/2000/svg" xmlns="http://www.w3.org/2000/svg" xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd" xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" version="1.1" id="svg1981" width="256.83865" height="113.20025"> <metadata id="metadata1987"> <rdf:RDF> <cc:Work rdf:about=""> <dc:format>image/svg+xml</dc:format> <dc:type rdf:resource="http://purl.org/dc/dcmitype/StillImage" /> </cc:Work> </rdf:RDF> </metadata> <defs id="defs1985" /> <sodipodi:namedview pagecolor="#ffffff" bordercolor="#666666" borderopacity="1" objecttolerance="10" gridtolerance="10" guidetolerance="10" inkscape:pageopacity="0" inkscape:pageshadow="2" inkscape:window-width="640" inkscape:window-height="480" id="namedview1983" /> <inkscape:clipboard style="font-variation-settings:normal;opacity:1;vector-effect:none;fill:none;fill-opacity:1;stroke:#000000;stroke-width:1.999995px;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:1.5;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;stop-color:#000000;stop-opacity:1" min="190.98464,264.53692" max="447.82329,377.73718" /> <g id="g1989" transform="matrix(1.33333,0,0,-1.33333,-190.98464,806.46308)"> <path d="m 156.03896,583.55107 c -0.60825,-0.61501 0.0803,-3.71776 0.13125,-4.38002" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path54" /> <path d="m 160.70322,584.11732 c 0.32175,-1.82626 0.0757,-3.80476 0.2445,-5.65877" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path58" /> <path d="m 154.0087,573.39154 c -0.36975,-5.76227 9.60978,-5.82902 9.60978,-0.2145" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path62" /> <path d="m 149.20119,585.03682 c -0.0705,0.66225 -0.24225,-1.31325 -0.27075,-1.97851 -0.066,-1.515 -0.0892,-3.02475 -0.0592,-4.53976 0.0622,-3.20551 0.183,-6.41327 -0.06,-9.61427 -0.0968,-1.28026 -0.74625,-3.45526 -0.0885,-4.68827 0.6105,-1.14525 3.03676,-0.6165 4.00576,-0.564 3.73576,0.20175 7.50377,0.4155 11.24553,0.4155 0.819,0 3.33001,-0.744 3.97651,-0.17775 1.53376,1.34176 0.46725,7.34402 0.59325,9.02028 0.29175,3.86101 0.2865,7.73552 0.47475,11.60178 0.042,0.8625 0.55276,3.12526 -0.44475,3.67951 -1.2555,0.6975 -3.91951,-0.441 -5.22226,-0.534 -3.18076,-0.22725 -6.42677,-0.3225 -9.61428,-0.44475 -2.517,-0.0968 -5.13226,0.96825 -4.86601,-2.22601" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path66" /> <path d="m 147.57294,556.25399 c 0.1155,-0.1065 -1.44451,-0.078 -1.46176,-0.0795 -0.7965,-0.057 -1.473,-0.78075 -1.80975,-1.4535 -0.99675,-1.99276 0.4155,-3.36976 2.433,-3.14551" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path70" /> <path d="m 150.96595,559.41225 c 0.11175,0.2505 -0.33376,-0.47175 -0.34426,-0.74475 -0.0232,-0.609 -0.0112,-1.2315 0,-1.83976 0.036,-1.9785 0.081,-4.09501 0.56326,-6.02401" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path74" /> <path d="m 154.43696,554.84024 c -0.42601,-0.171 0.3075,-2.93926 0.36825,-3.26476" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path78" /> <path d="m 154.64396,557.66775 c 0.01,-7.5e-4 -0.27751,-0.30825 -0.34276,-0.39525" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path82" /> <path d="m 157.99196,553.28623 c -0.0637,-0.0705 1.33351,-0.0788 1.44226,-0.0788 0.78225,0 1.9965,-0.1395 2.61151,0.47476 0.735,0.735 -0.79801,1.59075 -1.27576,1.869 -0.4395,0.2565 -0.95475,0.564 -1.48425,0.564 -1.73776,0 -1.78351,-2.79976 -1.36501,-3.91651 0.74851,-1.9965 2.87326,-2.37 4.74827,-2.226" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path86" /> <path d="m 165.75074,555.12149 c 0.0547,-0.591 -0.15451,-1.284 -0.17401,-1.88476 -0.0128,-0.39225 0.004,-0.5835 0.0292,-0.97875 0.26851,-4.15501 0.0113,2.30626 0.41551,3.11551 1.73775,3.47551 3.52951,-3.57226 3.65026,-4.65826" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path90" /> <path d="m 171.80925,559.956 c -0.036,0.34875 -0.001,-0.70125 -0.001,-1.0515 0,-0.585 0.007,-1.1655 0.03,-1.75051 0.0585,-1.5495 -0.18,-8.27252 2.96701,-5.69701" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path94" /> <path d="m 169.42874,556.84349 c 0.37651,-0.438 5.13827,0.82051 5.76152,0.90376" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path98" /> <path d="m 307.14071,579.20731 c 0.64651,-4.54951 0.0802,-10.35228 -0.36,-14.94304 -0.0922,-0.96375 -0.807,-4.44976 0.25125,-5.13526 2.74201,-1.77376 8.80203,-0.0517 11.82229,0.0232 1.70325,0.042 3.50625,0.0975 5.20351,-0.0915 0.51225,-0.057 1.2945,-0.59925 1.75725,-0.228 1.00126,0.80025 0.48825,3.20551 0.411,4.24501 -0.28725,3.87601 0.21751,7.68002 0.41101,11.54853 0.06,1.19625 0.6645,3.61351 -0.66226,4.38151 -1.236,0.71625 -3.14625,0.31275 -4.49551,0.34275 -2.4525,0.054 -5.16751,0.165 -7.55477,0.70725 -1.5255,0.3465 -2.64451,0.49125 -4.19926,0.297 -0.09,-0.0113 -2.49976,-0.234 -1.87125,-0.54825" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path142" /> <path d="m 313.10848,572.64479 c -0.0915,-0.008 -0.19425,-0.099 -0.27975,-0.141" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path146" /> <path d="m 320.27325,573.04229 c -0.0705,0.0233 -0.32625,-0.29925 -0.369,-0.35625" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path150" /> <path d="m 312.41998,566.60803 c -0.243,0.0353 0.68475,-1.05676 0.72825,-1.08826 0.61125,-0.43875 1.3095,-0.837 2.03176,-1.05 1.7865,-0.52725 7.56602,-0.62775 6.45901,2.69326" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path154" /> <path d="m 302.13172,552.43724 c 0.0165,0.007 -2.74801,0.61425 -2.64526,-1.236 0.0937,-1.68526 6.07427,-2.50651 1.50675,-3.53776 -0.4425,-0.0998 -0.8715,-0.17475 -1.32375,-0.13725" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path158" /> <path d="m 305.58023,550.53373 c 0.8115,-2.208 5.66926,2.53426 1.39275,2.35651 -3.38326,-0.141 -2.39176,-5.86727 0.93525,-6.02552" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path162" /> <path d="m 311.27349,551.00474 c -0.42975,-0.42226 0.0563,-2.49376 0.14925,-2.97601 0.0128,-0.063 0.16275,-1.18801 0.15975,-1.18651 -0.4545,0.1515 -0.57,1.71001 -0.61575,2.07676 -0.27525,2.20201 0.34425,3.46576 2.64751,3.19501" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path166" /> <path d="m 316.42226,551.33249 c -0.0202,-0.61126 1.77675,-4.52402 2.2125,-4.26227 0.453,0.27151 0.654,4.19777 0.86775,4.83827" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path170" /> <path d="m 323.16627,550.07773 c 0.204,0.2865 0.66826,-0.27825 1.01926,-0.27075 0.711,0.0158 1.48425,0.141 2.076,0.56625 1.30426,0.93601 -0.132,2.29426 -1.21275,2.53426 -0.648,0.144 -1.16925,-0.372 -1.56376,-0.8085 -1.79475,-1.98901 -1.047,-4.32976 1.76551,-4.55626 1.089,-0.0878 1.97025,0.37575 2.87176,0.9165" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path174" /> <path d="m 329.31404,551.94074 c -0.0405,0.012 0.246,-1.14376 0.264,-1.20376 0.12975,-0.4455 1.16175,-3.1515 0.7545,-3.55876 -0.005,-0.005 -0.09,0.43501 -0.0945,0.47251 -0.0383,0.32175 -0.0908,0.6465 -0.12075,0.9705 -0.0945,1.00725 -0.054,2.0565 0.36375,2.99251 0.84001,1.88175 3.11626,0.87225 4.63727,0.66" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path178" /> <path d="m 195.85402,590.90693 c 0.8355,0.93301 -2.286,0.52051 -2.55375,0.3765 -2.04601,-1.10175 -1.88551,-5.75926 0.975,-5.43226 1.73025,0.198 2.3205,2.06851 2.60025,3.52876 0.63601,3.32326 -0.3345,6.53102 -1.4625,9.63378" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path326" /> <path d="m 203.39604,591.62694 c -2.70675,1.734 -5.20276,-2.64901 -2.667,-4.31327 1.43325,-0.9405 2.51775,0.62851 2.739,1.88026 0.0172,0.0975 0.066,1.85625 -0.1395,1.788 -0.83775,-0.27975 1.58101,-3.64051 1.99651,-3.90001" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path330" /> <path d="m 207.9538,597.45445 c -0.1785,0.11925 0.35401,-3.37501 0.38926,-3.54826 0.48975,-2.424 1.0215,-4.92076 1.0215,-7.40477" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path334" /> <path d="m 206.21755,592.34619 c -0.0172,0.008 1.28325,0.6765 1.359,0.70125 0.85126,0.2775 1.71451,0.44325 2.60026,0.55725" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path338" /> <path d="m 217.71283,592.73244 c 0.213,0.1425 -0.4965,0.1275 -0.73425,0.222 -0.63,0.25275 -1.338,0.3195 -1.92676,-0.0922 -1.119,-0.783 -1.4925,-2.62876 -0.975,-3.85351 1.14075,-2.70151 4.74151,0.849 3.62101,2.52976 -0.339,0.50925 0.0907,-1.23601 0.30225,-1.81051 0.41025,-1.11675 1.10925,-1.92225 2.06551,-2.59951" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path342" /> <path d="m 186.998,586.57117 c -0.0825,0.48075 -0.1335,-0.98475 -0.0352,-1.4625 0.23625,-1.1475 0.58875,-2.41276 1.3695,-3.31951 1.86526,-2.16601 5.67002,-1.88326 8.24102,-1.99651 4.97477,-0.21975 9.69603,0.2265 14.62504,0.71926 4.10926,0.411 11.67603,0.567 13.55704,5.26951 0.729,1.8225 0.783,4.12876 0.6495,6.05927 -1.02225,14.82379 -20.6483,12.55728 -30.85133,11.39778 -3.55876,-0.40425 -5.89352,-2.30101 -7.26527,-5.59427 -0.12825,-0.30675 -1.4685,-4.73551 -0.30225,-4.73551" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path346" /> <path d="m 189.51351,560.6766 c 0.0308,0.0172 0.73575,-3.83401 0.79275,-4.30351 0.069,-0.56925 0.15,-2.2875 0.15,-1.7145 0,1.5915 -0.651,5.84851 1.41375,6.40651 1.80526,0.48826 2.72476,-2.04525 2.89276,-3.36375 0.01,-0.075 0.1425,-1.60726 -0.021,-1.60726 -0.19875,0 0.147,2.08576 0.171,2.20726 0.3015,1.5075 1.3425,4.82701 3.51376,4.19926 2.83876,-0.81975 0.96525,-4.60801 2.14275,-6.25652" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path350" /> <path d="m 207.4738,561.72961 c -0.47025,1.257 -2.1855,0.8115 -2.769,-0.0427 -0.82726,-1.21201 -1.14301,-3.28876 -0.34276,-4.58552 1.17301,-1.902 3.59701,-0.0495 4.00652,1.47826 0.3375,1.2585 -0.68851,2.3145 -1.26376,3.32101" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path354" /> <path d="m 210.98231,561.08535 c 0.1185,-0.7035 0.048,-1.45125 0.10725,-2.163 0.0232,-0.2715 0.45975,-2.55826 0.10725,-2.69926 -0.5295,-0.21225 -0.777,1.79251 -0.79275,1.99276 -0.13275,1.68225 0.40125,3.64276 2.16451,4.19926" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path358" /> <path d="m 215.75158,560.3691 c 1.90275,-3.27751 7.13401,3.47176 2.23725,3.07426 -3.94651,-0.3195 -3.54451,-6.89852 0.64275,-7.11302" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path362" /> <path d="m 234.22187,561.86761 c -1.9515,1.92525 -4.87576,-1.68751 -3.97651,-4.14451 0.95101,-2.60176 3.65851,-1.43851 4.22102,0.64275 0.891,3.29776 -1.07176,6.88127 -1.07176,10.26302" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path366" /> <path d="m 240.48964,561.59611 c -1.13625,3.26475 -4.85101,-1.52551 -2.98126,-3.57301 1.47526,-1.61551 3.09751,0.996 3.34276,2.271 0.0368,0.1905 0.10125,1.41151 0.27825,1.50001 0.0172,0.008 0.4755,-1.32301 0.49275,-1.37176 0.32626,-0.8895 0.66976,-1.68825 1.47901,-2.22825" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path370" /> <path d="m 246.00291,568.45412 c -0.53476,0.18075 0.13275,-3.192 0.162,-3.33901 0.4155,-2.109 0.65025,-4.24276 0.79275,-6.38551" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path374" /> <path d="m 243.60515,563.46736 c 1.362,1.239 3.30526,1.42125 5.06701,1.56225" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path378" /> <path d="m 256.23893,564.11536 c -1.0245,-0.30225 -1.9665,1.17901 -3.11025,0.378 -0.85051,-0.59475 -2.31451,-4.65826 -0.55726,-4.73476 1.20226,-0.0525 2.39401,1.209 3.02101,2.09926 0.32775,0.46575 0.11625,1.5645 0.6855,1.5645 0.573,0 -0.002,-1.1565 0.129,-1.7145 0.258,-1.10776 1.03726,-1.97626 1.99276,-2.54926" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path382" /> <path d="m 192.03426,567.09287 c -0.3195,-0.25425 -1.60425,-0.0165 -1.9635,-0.0277 -1.55851,-0.051 -3.21826,-0.3885 -4.26376,-1.65 -4.50302,-5.43452 -2.44351,-15.5198 5.01376,-16.34855 8.76227,-0.9735 18.18755,0.35925 26.99707,0.55725 9.52728,0.21375 19.23905,-0.6765 28.71158,0.42825 6.51301,0.75976 14.61904,2.60401 16.00504,10.32753 1.2795,7.12727 -4.42126,9.10503 -10.28478,9.55653 -10.50153,0.80775 -20.2913,-0.273 -30.72533,-1.22176 -5.49526,-0.4995 -10.98153,-0.63825 -16.41229,-1.41375 -0.3975,-0.057 -4.62827,0.0248 -4.62827,-0.3645" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path386" /> <path d="m 174.68297,573.06739 c 0.6555,1.04625 4.16326,0.576 5.10301,0.5325 5.53801,-0.258 10.27278,-1.96876 12.72678,-6.87827" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path390" /> <path d="m 174.26146,574.14064 c -0.23175,-0.3375 0.78751,0.2265 1.17451,0.35925 0.65625,0.22425 1.323,0.42225 1.99275,0.6 2.25226,0.597 4.62302,0.85875 6.94202,1.02825 5.92952,0.43351 8.52602,-1.73625 12.81304,-5.12101 1.57575,-1.24425 3.94276,-3.07276 3.94276,-5.29201" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path394" /> <path d="m 173.18896,578.4464 c 2.23051,3.27901 6.68627,6.33827 10.68903,7.60202 0.69675,0.2205 2.79751,1.248 3.45001,0.92175" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path398" /> <path d="m 173.56321,578.64965 c 0.09,-0.0255 0.85951,1.42425 0.93001,1.57126 0.84525,1.761 1.4445,3.6225 2.35725,5.35651 1.75276,3.33001 6.67877,6.70502 10.41303,7.32752" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path402" /> <path d="m 234.15137,528.02452 c 1.22851,0.99825 1.88176,5.85076 2.13751,7.69652 0.078,0.56325 0.036,2.50725 0.64275,2.871 0.4095,0.246 1.32601,-2.84475 1.39276,-3.06375 0.27975,-0.91276 0.49575,-1.85026 0.6855,-2.78551 0.2805,-1.38076 0.72825,-2.97076 0.72825,-4.39201" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path406" /> <path d="m 236.29713,532.34678 c -0.0195,-0.021 0.66675,-0.0187 0.72,-0.0113 0.5985,0.0855 1.19176,0.24525 1.77826,0.38625" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path410" /> <path d="m 246.94041,537.23154 c -1.18125,-0.28875 -1.92301,0.489 -2.87401,-0.71775 -2.64151,-3.35251 -1.12125,-8.64002 3.66451,-6.53477" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path414" /> <path d="m 250.92217,538.03329 c 0.10575,0.1725 -0.38475,-0.267 -0.38475,-0.4695 0,-0.62625 0.0998,-1.26225 0.15,-1.8855 0.1545,-1.92826 0.0435,-4.23451 0.57825,-6.10652" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path418" /> <path d="m 251.14642,532.12403 c 0.64425,2.427 2.77351,3.62326 4.94026,4.86151" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path422" /> <path d="m 252.25192,534.32303 c -0.17325,-0.576 0.849,-1.15575 1.19926,-1.4085 1.61325,-1.16325 3.2985,-2.43301 5.05651,-3.36376" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path426" /> <path d="m 264.23995,530.47477 c 2.16451,-6.20101 -3.123,-7.70852 -7.85327,-8.63777 -6.83626,-1.34325 -14.94604,-2.06626 -21.3833,1.221 -2.70376,1.38076 -5.31002,3.16801 -5.61377,6.38552 -0.32325,3.42451 -0.3615,7.97627 1.45726,11.03403 3.39225,5.70676 13.83828,5.61451 19.4333,4.37101 5.20126,-1.15575 12.98478,-1.8945 12.98478,-9.19127" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path430" /> <path d="m 264.74996,536.39754 c 7.25027,3.92926 13.78803,9.52052 19.64105,15.26479 2.39025,2.34601 5.47351,4.94776 7.00652,8.01377" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path434" /> <path d="m 264.84896,530.56027 c -0.2985,-0.3285 0.74625,0.47925 1.11525,0.726 0.86625,0.57751 1.70175,1.19626 2.55001,1.79926 2.3355,1.662 4.50676,3.43801 6.70651,5.27101 7.15727,5.96477 12.44404,12.84754 16.49855,20.95506" style="clip-rule:evenodd;fill:none;fill-rule:evenodd;stroke:#000000;stroke-width:1.5px;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5" id="path438" /> </g> </svg> </html>

By itself, this is fine. It sends less ACKs! That means less packets! Great!

But delayed ACKs can combine poorly with another TCP optimization called <a href="https://en.wikipedia.org/wiki/Nagle%27s_algorithm">Nagle's algorithm</a>. The idea behind Nagle's algorithm is -- if you haven't received an ACK, maybe that's because the connection is congested! So if a packet hasn't been ACKed and a connection is using Nagle's algorithm, it'll wait to send more data. Like this:

<html> <?xml version="1.0" encoding="UTF-8" standalone="no"?> <svg xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:cc="http://creativecommons.org/ns#" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:svg="http://www.w3.org/2000/svg" xmlns="http://www.w3.org/2000/svg" xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd" xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape" version="1.1" id="svg1231" width="263.4487" height="130.13521"> <metadata id="metadata1237"> <rdf:RDF> <cc:Work rdf:about=""> <dc:format>image/svg+xml</dc:format> <dc:type rdf:resource="http://purl.org/dc/dcmitype/StillImage" /> </cc:Work> </rdf:RDF> </metadata> <defs id="defs1235" /> <sodipodi:namedview pagecolor="#ffffff" bordercolor="#666666" borderopacity="1" objecttolerance="10" gridtolerance="10" guidetolerance="10" inkscape:pageopacity="0" inkscape:pageshadow="2" inkscape:window-width="640" inkscape:window-height="480" id="namedview1233" /> <inkscape:clipboard style="font-variation-settings:normal;opacity:1;vector-effect:none;fill:#000000;fill-opacity:1;stroke-width:1.00000017;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:1.5;stroke-dasharray:none;stroke-dashoffset:0;stroke-opacity:1;stop-color:#000000;stop-opacity:1" min="421.91708,392.50327" max="685.36578,522.63847" /> <g id="g1239" transform="matrix(1.33333,0,0,-1.33333,-421.91708,678.49673)"> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g916" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 430.242,430.366 c -0.546,-1.239 0.011,8.996 0.1,10.444 0.348,5.633 -0.429,14.366 2.19,19.604 1.05,2.1 4.529,1.528 6.413,1.408 5.922,-0.378 11.582,-1.705 17.571,-1.46 1.226,0.05 5.859,1.77 6.726,0.73 2.655,-3.186 1.345,-11.5 1.251,-15.172 -0.137,-5.361 -0.591,-10.702 -0.782,-16.059 -0.049,-1.377 0.418,-4.498 -1.251,-5.214 -2.305,-0.987 -5.537,-0.767 -7.978,-0.886 -2.109,-0.103 -4.2,10e-4 -6.308,0.104 -3.979,0.194 -22.088,-1.834 -20.334,6.935" style="fill:none;stroke:#000000;stroke-width:2px" id="path914" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g920" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 441.493,432.693 c 0.685,1.709 0.564,4.547 0.871,6.391" style="fill:none;stroke:#000000;stroke-width:2px" id="path918" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g924" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 448.376,432.374 c 0.213,0.096 0.795,5.639 0.855,6.079" style="fill:none;stroke:#000000;stroke-width:2px" id="path922" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g928" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 438.621,445.343 c 0.48,0.054 0.301,0.929 0.559,1.338 0.73,1.156 1.639,2.207 2.72,3.052 4.167,3.258 11.465,1.793 10.848,-4.379" style="fill:none;stroke:#000000;stroke-width:2px" id="path926" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g932" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 428.315,474.407 c -1.658,0.245 -3.358,0.233 -4.495,1.732 -2.829,3.73 1.242,7.478 5.109,6.469" style="fill:none;stroke:#000000;stroke-width:2px" id="path930" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g936" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 432.376,468.662 c -0.383,-0.25 -0.049,0.918 0.003,1.373 0.122,1.078 0.319,2.149 0.497,3.218 0.516,3.093 0.584,6.339 1.261,9.388" style="fill:none;stroke:#000000;stroke-width:2px" id="path934" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g940" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 437.381,476.991 c -1.259,0.239 0.615,2.492 0.803,3.759" style="fill:none;stroke:#000000;stroke-width:2px" id="path938" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g944" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 437.221,472.299 c -0.094,0.012 -0.177,0.055 -0.264,0.092" style="fill:none;stroke:#000000;stroke-width:2px" id="path942" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g948" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 442.973,477.254 c 1.61,3.44 4.56,-4.737 1.116,-3.138 -4.222,1.96 -0.638,8.833 3.218,8.658" style="fill:none;stroke:#000000;stroke-width:2px" id="path946" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g952" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 451.749,476.357 c -0.138,0.64 -0.249,1.288 -0.328,1.939 -0.091,0.751 -0.13,1.471 0,2.222 0.07,0.407 0.35,1.566 0.431,1.161 0.28,-1.399 0.624,-11.959 4.578,-6.369 0.346,0.489 0.643,1.055 0.829,1.625 0.153,0.468 0.33,1.438 0.399,1.925 0.149,1.062 0.142,2.118 0.199,3.184" style="fill:none;stroke:#000000;stroke-width:2px" id="path950" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g956" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 462.534,467.137 c 0.1,0.018 0.357,2.684 0.365,2.732 0.623,3.585 1.232,7.152 1.526,10.782" style="fill:none;stroke:#000000;stroke-width:2px" id="path954" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g960" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 459.422,474.488 c 1.709,-1.353 5.878,-1.746 7.922,-2.33" style="fill:none;stroke:#000000;stroke-width:2px" id="path958" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g964" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 501.15,412.164 c -0.161,-0.141 -1.387,-1.349 -0.935,-1.349" style="fill:none;stroke:#000000;stroke-width:2px" id="path962" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g968" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 501.466,411.245 c -0.338,-1.271 -3.366,-0.306 -3.869,0.057 -2.62,1.892 -4.434,9.791 0.56,9.716 6.545,-0.098 4.828,-8.615 3.72,-12.352 -0.359,-1.209 -0.741,-2.41 -1.12,-3.612 -0.339,-1.076 -0.782,-2.174 -0.885,-3.305" style="fill:none;stroke:#000000;stroke-width:2px" id="path966" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g972" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 513.693,412.994 c -0.364,-1.678 -2.402,-1.688 -3.581,-0.825 -1.767,1.294 -2.966,5.236 -1.391,7.061 1.693,1.962 4.055,-1.289 4.497,-2.709 0.261,-0.841 0.354,-1.691 0.415,-2.564 0.023,-0.332 -0.058,-1.291 0.09,-0.993 1.093,2.184 0.214,5.58 2.511,7.187" style="fill:none;stroke:#000000;stroke-width:2px" id="path970" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g976" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 519.544,405.404 c -0.421,-0.61 0.618,4.302 0.699,4.869 0.415,2.906 0.7,5.897 0.433,8.831" style="fill:none;stroke:#000000;stroke-width:2px" id="path974" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g980" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 516.875,411.43 c 0.223,-1.222 4.29,-1.492 5.137,-1.627" style="fill:none;stroke:#000000;stroke-width:2px" id="path978" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g984" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 533.122,411.331 c 0.275,0.063 -0.499,-0.273 -0.708,-0.462 -0.57,-0.519 -1.307,-0.871 -2.094,-0.632 -1.931,0.586 -3.421,3.53 -2.926,5.436 0.366,1.41 1.907,1.564 3.052,1.065 1.426,-0.621 2.77,-2.482 2.944,-4.045 0.028,-0.258 -0.137,0.502 -0.181,0.758 -0.145,0.843 -0.23,1.672 -0.145,2.529 0.193,1.922 1.08,3.199 2.619,4.298" style="fill:none;stroke:#000000;stroke-width:2px" id="path982" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g988" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 488.164,420.269 c -0.556,1.13 -0.152,2.723 0.078,3.873 0.808,4.04 2.512,6.53 6.7,7.386 3.69,0.755 7.533,0.26 11.269,0.361 7.612,0.206 15.213,0.037 22.826,0.037 3.851,0 9.939,1.1 13.418,-1.03 1.837,-1.125 2.491,-3.696 3.052,-5.616 1.549,-5.301 2.743,-11.049 1.553,-16.542 -0.98,-4.522 -3.413,-6.123 -6.916,-8.686 -9.006,-6.59 -21.563,-6.484 -32.236,-6.484 -4.612,0 -10.577,-0.561 -14.645,2.186 -2.767,1.867 -4.833,6.035 -5.292,9.318 -0.226,1.625 0.447,4.059 -0.433,5.526 -0.754,1.257 -4.208,1.795 -5.472,2.348 -3.466,1.516 -6.569,3.5 -9.228,6.212 -1.906,1.944 -3.06,4.097 -3.864,6.682 -0.041,0.132 -0.61,2.293 -0.596,2.293 0.35,0 0.93,-1.04 1.101,-1.264 0.825,-1.076 1.792,-2.08 2.854,-2.925 3.345,-2.664 9.523,-4.365 13.598,-2.619" style="fill:none;stroke:#000000;stroke-width:2px" id="path986" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g992" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 502.693,452.525 c -0.076,-0.335 0.229,2.396 0.263,2.718 0.173,1.611 0.296,3.224 0.336,4.843 0.012,0.447 0.02,0.632 0,1.055 -0.009,0.218 0.062,0.803 -0.092,0.648 -0.325,-0.325 -0.109,-1.924 -0.104,-2.328 0.021,-2.105 0.807,-4.947 3.313,-2.781 0.329,0.284 0.607,0.611 0.846,0.973 0.361,0.548 0.602,1.149 0.753,1.785 0.217,0.912 0.231,1.818 0.151,2.745" style="fill:none;stroke:#000000;stroke-width:2px" id="path990" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g996" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 511.135,457.736 c 0.062,0.648 0.029,1.305 0.082,1.956 0.023,0.281 0.064,0.735 0.093,1.02 0.035,0.344 0.089,0.686 0.116,1.031 0.01,0.135 0.084,0.501 -0.012,0.406 -0.233,-0.233 -0.185,-1.046 -0.185,-1.333 0,-1.158 -0.021,-3.799 1.089,-4.53 1.589,-1.046 2.877,2.641 3.047,3.603 0.018,0.104 0.244,1.588 0.012,1.588 -0.157,0 0.062,-1.633 0.081,-1.75 0.184,-1.172 0.632,-3.25 2.131,-3.348 2.242,-0.146 2.462,3.418 2.63,4.935" style="fill:none;stroke:#000000;stroke-width:2px" id="path994" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g1000" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 522.541,456.318 c 0.036,0.005 0.098,1.256 0.1,1.277 0.101,1.056 0.235,2.108 0.336,3.163 0.005,0.051 0.147,1.05 -0.012,1.089 -0.075,0.019 -0.007,-1.87 0,-1.969 0.075,-1.056 0.149,-3.513 1.379,-3.974 1.515,-0.568 2.104,3.051 2.201,3.881 0.009,0.074 -0.04,1.744 -0.127,1.657 -0.319,-0.319 0.385,-2.017 0.475,-2.329 0.565,-1.955 2.439,-5.512 4.6,-2.653 0.275,0.365 0.483,0.756 0.648,1.182 0.322,0.826 0.444,1.707 0.533,2.583" style="fill:none;stroke:#000000;stroke-width:2px" id="path998" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g1004" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 539.629,461.324 c 0.857,0.476 -0.178,2.373 -0.467,2.806" style="fill:none;stroke:#000000;stroke-width:2px" id="path1002" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g1008" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 547.795,451.053 c -0.005,-0.209 0.589,2.567 0.636,3.09 0.165,1.838 0.238,3.693 0.174,5.538" style="fill:none;stroke:#000000;stroke-width:2px" id="path1006" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g1012" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 548.76,455.388 c 0.067,-0.725 1.232,-0.715 1.722,-0.631 1.708,0.293 3.393,1.559 4.067,3.174 1.253,3.008 -5.259,4.24 -6.697,2.213" style="fill:none;stroke:#000000;stroke-width:2px" id="path1010" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g1016" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 558.234,457.633 c 0.837,0.713 3.963,0.114 4.459,-0.849 0.437,-0.847 -1.304,-2.479 -2.05,-2.514 -2.047,-0.094 -3.187,3.517 -2.48,5.029 0.662,1.414 2.398,1.506 3.743,1.506" style="fill:none;stroke:#000000;stroke-width:2px" id="path1014" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g1020" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 568.738,449.562 c -0.165,3.575 1.139,7.142 1.139,10.733" style="fill:none;stroke:#000000;stroke-width:2px" id="path1018" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g1024" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 565.967,455.167 c 1.642,-0.413 3.252,-0.774 4.941,-1.013" style="fill:none;stroke:#000000;stroke-width:2px" id="path1022" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g1028" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 578.082,448.688 c -0.5,0.749 -0.127,2.393 -0.084,3.196 0.152,2.774 0.427,5.596 0.348,8.376" style="fill:none;stroke:#000000;stroke-width:2px" id="path1026" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g1032" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 574.759,455.324 c 0.838,-0.87 2.787,-0.78 3.865,-0.822" style="fill:none;stroke:#000000;stroke-width:2px" id="path1030" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g1036" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 583.561,456.842 c 0.16,0.335 0.948,-0.011 1.204,-0.092 0.591,-0.189 1.319,-0.499 1.529,-1.147 0.297,-0.917 -0.595,-2.1 -1.541,-2.028 -1.802,0.137 -4.817,3.08 -3.731,5.028 0.624,1.12 2.234,1.512 3.395,1.657" style="fill:none;stroke:#000000;stroke-width:2px" id="path1034" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g1040" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 589.785,456.658 c -0.422,1.06 -0.282,1.955 -0.282,3.034 0,0.176 -0.132,-0.327 -0.174,-0.498 -0.07,-0.292 -0.081,-0.604 -0.092,-0.903 -0.039,-0.999 0.192,-2.194 0.903,-2.943 0.953,-1.003 2.531,-0.639 3.743,-0.498" style="fill:none;stroke:#000000;stroke-width:2px" id="path1038" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g1044" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 606.367,453.951 c -0.092,0.059 -0.417,-0.038 -0.463,0.039 -0.07,0.117 0.38,1.085 0.409,1.177 0.375,1.221 0.715,4.336 2.053,4.948 1.199,0.548 1.538,-2.416 2.486,-2.534 1.074,-0.135 1.079,2.983 2.522,2.486 0.595,-0.205 0.804,-1.035 0.889,-1.574 0.263,-1.676 -0.017,-3.441 0.396,-5.092" style="fill:none;stroke:#000000;stroke-width:2px" id="path1042" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g1048" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 621.645,454.829 c -0.385,-0.278 -0.634,-0.847 -1.161,-0.863 -1.961,-0.059 -3.796,4.196 -2.618,5.608 1.054,1.265 3.069,-0.574 3.662,-1.429 0.295,-0.425 0.512,-0.899 0.697,-1.381 0.011,-0.028 0.357,-1.043 0.372,-1.033 0.171,0.114 0.08,1.355 0.108,1.55 0.141,0.961 0.526,1.569 1.345,2.077" style="fill:none;stroke:#000000;stroke-width:2px" id="path1046" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g1052" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 628.02,455.612 c 0.042,-0.07 -0.033,0.456 -0.031,0.528 0.005,0.153 0.054,0.898 0.06,0.996 0.068,0.988 0.111,2.064 0.457,3.003" style="fill:none;stroke:#000000;stroke-width:2px" id="path1050" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g1056" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 628.611,451.977 c -0.06,0.009 -0.12,0.022 -0.18,0.034" style="fill:none;stroke:#000000;stroke-width:2px" id="path1054" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g1060" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 634.471,447.312 c -0.205,0.026 0.137,2.595 0.148,2.715 0.279,3.122 0.72,6.608 0.372,9.739" style="fill:none;stroke:#000000;stroke-width:2px" id="path1058" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g1064" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 631.875,454.486 c 1.638,-0.649 3.883,-0.379 5.59,-0.412" style="fill:none;stroke:#000000;stroke-width:2px" id="path1062" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g1080" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 526.732,471.443 c -1.482,-1.581 -4.098,-1.206 -4.271,1.206 -0.245,3.397 2.05,6.733 1.326,10.114" style="fill:none;stroke:#000000;stroke-width:2px" id="path1078" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g1084" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 520.444,478.502 c 1.392,-0.92 2.841,-1.332 4.499,-1.664" style="fill:none;stroke:#000000;stroke-width:2px" id="path1082" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g1088" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 531.922,477.113 c -0.071,0.33 -1.804,-0.148 -2.028,-0.12 -1.085,0.136 -1.664,0.946 -1.918,1.961 -0.253,1.011 -0.312,2.427 0.395,3.287 1.279,1.557 3.504,-0.389 3.907,-1.749 0.356,-1.197 -0.35,-2.421 -0.747,-3.513" style="fill:none;stroke:#000000;stroke-width:2px" id="path1086" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g1092" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 535.62,480.553 c 0.032,0.59 -10e-4,1.182 0.03,1.772 0.018,0.346 0.371,1.275 0.127,1.03 -1.488,-1.488 -0.163,-5.889 1.015,-7.025" style="fill:none;stroke:#000000;stroke-width:2px" id="path1090" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,-4.5000114,803.252)" id="g1100" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 559.573,476.374 c 0.01,0.017 -1.675,-0.908 -2.806,-0.283 -1.536,0.848 -2.251,3.533 -1.467,5.078 1.306,2.574 5.573,-1.746 4.626,-3.64 -0.075,-0.151 -0.037,0.339 -0.028,0.508 0.024,0.443 0.03,0.582 0.099,1.044 0.178,1.193 0.43,2.373 1.326,3.23" style="fill:none;stroke:#000000;stroke-width:2px" id="path1098" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,-4.5000114,803.252)" id="g1104" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 564.326,478.356 c 0.007,0.519 0.002,1.038 0.002,1.557 v 1.03 c 0,0.164 -0.091,-0.114 -0.085,-0.296 0.032,-0.962 0.214,-1.948 0.663,-2.807 0.224,-0.43 0.495,-0.82 0.846,-1.157 2.614,-2.504 4.236,2.156 4.401,4.091" style="fill:none;stroke:#000000;stroke-width:2px" id="path1102" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,-4.5000114,803.252)" id="g1108" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 582.114,481.415 c -0.07,-1.018 0.791,-2.158 1.172,-3.039 0.677,-1.568 1.04,-3.193 1.411,-4.853 0.115,-0.518 0.357,-2.767 1.199,-2.821 0.85,-0.055 1.29,2.321 1.439,2.807 0.744,2.443 1.93,5.184 2.059,7.758" style="fill:none;stroke:#000000;stroke-width:2px" id="path1106" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,-4.5000114,803.252)" id="g1112" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 584.548,477.438 c 0.725,-0.31 1.903,-0.268 2.688,-0.459" style="fill:none;stroke:#000000;stroke-width:2px" id="path1110" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,-4.5000114,803.252)" id="g1116" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 598.379,472.417 c -0.63,-0.27 -1.258,-1.163 -1.96,-1.066 -2.966,0.409 -4.496,5.839 -3.16,8.111 0.941,1.6 3.257,1.865 4.895,1.636" style="fill:none;stroke:#000000;stroke-width:2px" id="path1114" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,-4.5000114,803.252)" id="g1120" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 606.607,470.254 c -0.85,0.676 -0.936,1.979 -1.019,2.973 -0.21,2.517 -0.127,5.066 -0.127,7.589" style="fill:none;stroke:#000000;stroke-width:2px" id="path1118" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,-4.5000114,803.252)" id="g1124" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 606.574,478.076 c 0.812,-2.535 4.39,-3.381 6.617,-4.101" style="fill:none;stroke:#000000;stroke-width:2px" id="path1122" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,-4.5000114,803.252)" id="g1128" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 614.105,473.561 c -0.166,0.008 -0.342,0.074 -0.505,0.103" style="fill:none;stroke:#000000;stroke-width:2px" id="path1126" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,-4.5000114,803.252)" id="g1132" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 608.075,475.745 c -0.28,2.696 2.98,3.925 5.088,5.029" style="fill:none;stroke:#000000;stroke-width:2px" id="path1130" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,-4.5000114,803.252)" id="g1136" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 625.931,470.757 c -0.062,2.395 0.312,4.806 0.385,7.2" style="fill:none;stroke:#000000;stroke-width:2px" id="path1134" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,-4.5000114,803.252)" id="g1140" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 626.367,474.725 c 0.16,-1.491 3.053,-0.144 3.545,0.193 1.105,0.757 2.437,2.235 2.026,3.71 -0.509,1.828 -4.103,2.419 -5.406,1.309" style="fill:none;stroke:#000000;stroke-width:2px" id="path1138" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,-4.5000114,803.252)" id="g1144" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 635.799,477.121 c 0.934,0.27 2.677,-0.39 2.979,-1.372 0.282,-0.916 -0.916,-1.799 -1.719,-1.764 -2.003,0.086 -3.209,3.418 -2.185,4.95 0.838,1.254 2.718,0.985 3.983,0.854" style="fill:none;stroke:#000000;stroke-width:2px" id="path1142" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,-4.5000114,803.252)" id="g1148" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 647.998,468.062 c -0.591,0.027 -1.288,-0.14 -1.834,0.164 -1.23,0.686 -1.222,2.533 -1.093,3.722 0.259,2.38 1.872,5.481 0.945,7.841" style="fill:none;stroke:#000000;stroke-width:2px" id="path1146" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,-4.5000114,803.252)" id="g1152" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 643.042,474.273 c 1.687,-0.51 3.174,-0.886 4.965,-0.982" style="fill:none;stroke:#000000;stroke-width:2px" id="path1150" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,-4.5000114,803.252)" id="g1156" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 655.251,474.002 c -0.847,-0.111 -1.556,-0.557 -2.429,-0.12 -1.283,0.642 -3.101,3.586 -1.514,4.735 1.039,0.752 4.43,0.357 4.746,-1.184 0.173,-0.848 -0.319,-1.831 -0.535,-2.629 -0.036,-0.132 -0.348,-1.485 -0.262,-1.457" style="fill:none;stroke:#000000;stroke-width:2px" id="path1154" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,-4.5000114,803.252)" id="g1160" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 659.573,476.08 c -0.014,0.516 -0.069,1.031 -0.094,1.547 -0.022,0.462 -0.023,0.925 -0.023,1.388 0,0.034 -0.012,0.789 -0.022,0.785 -0.389,-0.129 -0.598,-2.193 -0.626,-2.492 -0.183,-1.919 1.279,-6.674 3.96,-5.816" style="fill:none;stroke:#000000;stroke-width:2px" id="path1158" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,-4.5000114,803.252)" id="g1164" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 666.689,475.467 c 0.142,0.317 0.843,0.259 1.11,0.259 1.051,0 2.514,-0.002 3.095,-1.081 0.27,-0.501 0.486,-1.631 -0.228,-1.901 -2.196,-0.831 -5.552,2.367 -4.734,4.53 1.005,2.656 4.594,2.593 6.885,2.481" style="fill:none;stroke:#000000;stroke-width:2px" id="path1162" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g1168" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 505.845,498.736 c 1.655,-1.797 5.833,-2.821 8.074,-3.534" style="fill:none;stroke:#000000;stroke-width:2px" id="path1166" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g1172" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 510.283,497.629 c -0.698,1.005 -0.165,2.606 -0.07,3.717 0.185,2.186 -0.013,4.349 -0.578,6.466" style="fill:none;stroke:#000000;stroke-width:2px" id="path1170" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g1176" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 507.196,508.093 c -0.241,-0.31 1.054,-0.421 1.091,-0.425 1.744,-0.212 3.492,-0.282 5.231,-0.546" style="fill:none;stroke:#000000;stroke-width:2px" id="path1174" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g1180" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 526.881,500.75 c 0.183,-0.141 -1.259,-0.528 -1.427,-0.495 -1.013,0.203 -2.142,1.144 -2.069,2.263 0.133,2.038 7.206,3.105 2.984,4.941 -0.4,0.174 -0.75,0.291 -1.171,0.401 -0.691,0.18 -1.321,0.263 -2.022,0.128" style="fill:none;stroke:#000000;stroke-width:2px" id="path1178" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g1184" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 531.584,503.81 c 0.104,0.505 0.971,0.328 1.299,0.248 2.053,-0.503 3.766,-2.237 0.946,-3 -0.383,-0.104 -0.788,-0.177 -1.187,-0.113 -4.085,0.654 -3.001,7.374 1.011,7.557" style="fill:none;stroke:#000000;stroke-width:2px" id="path1182" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g1188" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 538.277,503.289 c -0.243,0.704 -0.353,1.44 -0.421,2.181 -0.05,0.548 -0.044,0.485 -0.064,1.075 -0.02,0.613 -0.278,0.057 -0.289,-0.594 -0.022,-1.336 0.213,-2.935 1.236,-3.899 0.97,-0.914 2.185,-0.469 3,0.417 1.337,1.455 1.57,3.474 1.267,5.343" style="fill:none;stroke:#000000;stroke-width:2px" id="path1186" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g1192" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 552.711,503.122 c -1.128,-0.534 -2.08,-1.542 -3.448,-1.086 -1.859,0.62 -3.724,4.17 -2.326,5.872 1.433,1.744 4.41,0.208 5.166,-1.428 0.806,-1.741 -0.028,-4.152 -0.337,-5.92 -0.363,-2.081 -0.642,-4.173 -0.866,-6.273" style="fill:none;stroke:#000000;stroke-width:2px" id="path1190" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g1196" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 562.462,501.733 c 0.055,-0.034 0.13,1.129 0.134,1.17 0.107,1.056 0.243,2.109 0.385,3.16 0.049,0.364 0.096,0.728 0.144,1.091 0.019,0.144 0.084,0.574 0.048,0.433 -0.417,-1.668 0.148,-5.626 2.198,-5.968 1.751,-0.292 2.761,2.406 3.033,3.69 0.039,0.185 0.304,1.651 -0.145,1.701 -0.405,0.045 0.328,-2.296 0.369,-2.407 0.476,-1.268 1.482,-3.667 3.241,-3.337 2.572,0.482 1.773,4.565 1.572,6.305" style="fill:none;stroke:#000000;stroke-width:2px" id="path1194" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g1200" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 582.389,502.585 c -2.596,-1.931 -6.889,0.559 -5.482,4.008 0.86,2.109 4.568,2.152 5.695,0.272 0.775,-1.291 0.097,-3.108 -0.272,-4.428" style="fill:none;stroke:#000000;stroke-width:2px" id="path1198" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g1204" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 587.31,503.401 c -0.014,0.557 -0.151,1.107 -0.151,1.667 0,0.258 0.403,2.495 0.064,2.551 -0.32,0.054 -0.344,-2.144 -0.337,-2.326 0.072,-1.837 1.116,-5.321 3.45,-5.118" style="fill:none;stroke:#000000;stroke-width:2px" id="path1202" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g1208" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 594.003,502.637 c 0.744,1.392 4.52,0.656 5.365,-0.136 0.629,-0.588 0.747,-1.746 -0.048,-2.262 -1.602,-1.039 -3.882,0.462 -4.621,1.893 -1.823,3.535 0.59,6.53 4.092,6.386" style="fill:none;stroke:#000000;stroke-width:2px" id="path1206" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0.136055,803.592)" id="g1212" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 499.854,444.522 c -4.548,2.312 -8.671,7.627 -9.109,12.758 -0.229,2.69 2.228,9.092 6.197,7.505 0.183,-0.073 -0.395,0.037 -0.579,0.107 -0.239,0.092 -0.919,0.431 -1.093,0.536 -0.905,0.546 -1.697,1.276 -2.423,2.037 -2.129,2.231 -3.739,5.025 -4.503,8.02 -1.143,4.478 -0.581,9.712 3.666,12.33 0.886,0.546 3.683,0.885 4.268,1.587 0.064,0.077 -2.19,2.467 -2.316,2.637 -1.174,1.581 -2.034,3.339 -2.509,5.254 -1.786,7.198 1.684,14.837 8.234,18.334 3.074,1.641 6.576,1.874 9.821,0.6 1.04,-0.408 3.319,-2.573 4.353,-2.573 0.041,0 1.04,1.363 1.094,1.415 1.457,1.428 3.173,2.675 4.953,3.667 5.06,2.819 11.014,3.242 16.64,2.273 3.737,-0.644 7.468,-1.922 10.571,-4.139 1.525,-1.089 3.527,-3.646 5.597,-3.602 2.038,0.043 3.925,2.724 5.382,3.86 2.901,2.259 6.244,3.573 9.886,4.031 3.896,0.49 8.344,0.553 12.094,-0.772 2.482,-0.877 5.278,-4.177 7.955,-4.117 1.929,0.043 3.859,2.453 5.425,3.367 2.383,1.39 5.078,2.025 7.806,2.273 4.879,0.443 10.472,0.148 15.01,-1.845 3.174,-1.393 5.882,-4.465 9.242,-5.339 1.875,-0.488 3.605,1.39 5.146,2.144 2.88,1.409 6.151,1.823 9.328,1.694 5.527,-0.224 10.397,-2.33 14.324,-6.218 1.49,-1.475 2.66,-4.139 4.31,-5.34 0.392,-0.285 1.724,0.256 2.123,0.322 1.994,0.328 4.032,0.378 6.047,0.279 7.453,-0.368 13.645,-4.464 15.932,-11.772 0.611,-1.952 0.884,-4.112 -0.407,-5.854 -0.559,-0.755 -2.392,-1.746 -2.273,-2.852 0.126,-1.179 1.212,-2.414 1.673,-3.474 1.33,-3.061 2.829,-7.207 2.38,-10.593 -0.627,-4.724 -6.531,-2.208 -8.063,-5.361 -1.057,-2.177 -1.163,-4.772 -2.123,-7.012 -2.078,-4.849 -6.515,-7.735 -11.3,-9.542 -2.203,-0.832 -4.482,-1.297 -6.819,-1.544 -1.233,-0.13 -2.913,0.076 -4.01,-0.643 -1.541,-1.01 -2.525,-3.279 -3.817,-4.589 -3.007,-3.049 -6.819,-4.61 -11.043,-5.082 -6.216,-0.695 -11.892,2.582 -17.862,2.208 -1.999,-0.125 -3.729,-1.699 -5.469,-2.53 -2.663,-1.272 -5.612,-1.766 -8.555,-1.501 -3.191,0.288 -6.227,1.38 -9.092,2.766 -1.455,0.704 -3.417,2.406 -5.104,2.466 -1.41,0.051 -2.923,-1.337 -4.181,-1.822 -2.2,-0.848 -4.692,-1.117 -7.034,-0.987 -3.198,0.178 -6.134,1.174 -8.899,2.766 -1.544,0.89 -3.287,2.729 -5.082,3.088 -1.234,0.247 -2.926,-1.859 -3.924,-2.38 -2.488,-1.3 -5.464,-1.461 -8.191,-0.986 -2.945,0.512 -5.542,1.875 -8.02,3.495 -0.302,0.198 -2.096,1.802 -2.53,1.694 -0.059,-0.015 -0.287,-1.177 -0.322,-1.265 -0.274,-0.705 -0.694,-1.301 -1.265,-1.802 -1.971,-1.729 -4.888,-2.047 -7.398,-1.758 -2.411,0.278 -4.586,1.219 -6.283,2.981 -0.136,0.141 -1.712,1.826 -1.415,2.123 0.651,0.651 1.061,-2.41 0.643,-3.024 -1.331,-1.957 -3.949,-2.903 -6.175,-3.281 -6.827,-1.157 -16.296,-2.606 -20.912,2.982 z" style="fill:none;stroke:#000000;stroke-width:2px" id="path1210" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g1216" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 471.585,439.073 c 2.543,-2.342 5.855,2.365 4.343,4.805 -2.468,3.981 -6.929,-0.282 -4.718,-3.967" style="fill:none;stroke:#000000;stroke-width:2px" id="path1214" /> </g> <g transform="matrix(0.750002,0,0,-0.750002,0,803.252)" id="g1220" style="clip-rule:evenodd;fill-rule:evenodd;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:1.5"> <path d="m 482.152,439.905 c 1.283,-3.321 7.856,0.805 5.698,3.887 -3.427,4.896 -9.283,-1.425 -4.588,-4.675" style="fill:none;stroke:#000000;stroke-width:2px" id="path1218" /> </g> </g> </svg> </html>

The result is exactly the behaviour we've been seeing in this bug:

<ol> <li> The client opens the connection. </li> <li> The client sends some data (the start of a POST request) </li> <li> The server waits to ACK (because it's using delayed ACKs) </li> <li> The client waits to send more packets (because it's using Nagle's algorithm) </li> <li> Oh no! The server is waiting for the client and client is waiting for the server! We're stuck!!!! </li> <li> After 40ms (that number we've been seeing everywhere!!), the server finally gives up and sends an ACK packet</li> <li> The client sends more data and everything continues </li> </ol>

<div class="nav"> [[Figure out what to do about it->What to do]] </div> You ssh to the server to do some more investigation.

<div class="nav"> [[Use strace -> strace server]] [[Look at top -> top]] [[Go back -> Reproduce]] </div> You run <code>htop</code> to see if the server is using a lot of CPU.

<pre> $ htop CPU[| 0.7%] Tasks: 31, 23 thr; 1 running Mem[|||||||||||||||||||||||||||||||||||||||||||||||||||||||| 152M/981M] Load average: 0.00 0.00 0.00 Swp[ 0K/0K] Uptime: 17 days, 21:00:01 PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command 99525 root 20 0 8384 4056 3172 R 0.7 0.4 0:00.04 htop 1 root 20 0 164M 10792 6104 S 0.0 1.1 0:32.69 /sbin/init 328 root 19 -1 43508 10420 9164 S 0.0 1.0 0:26.54 /lib/systemd/systemd-journald 425 root RT 0 273M 17980 8188 S 0.0 1.8 0:15.93 /sbin/multipathd -d -s 426 root RT 0 273M 17980 8188 S 0.0 1.8 0:00.00 /sbin/multipathd -d -s 427 root RT 0 273M 17980 8188 S 0.0 1.8 0:02.51 /sbin/multipathd -d -s 428 root RT 0 273M 17980 8188 S 0.0 1.8 1:29.70 /sbin/multipathd -d -s 429 root RT 0 273M 17980 8188 S 0.0 1.8 0:00.00 /sbin/multipathd -d -s 430 root RT 0 273M 17980 8188 S 0.0 1.8 0:00.00 /sbin/multipathd -d -s 424 root RT 0 273M 17980 8188 S 0.0 1.8 2:07.36 /sbin/multipathd -d -s 449 systemd-t 20 0 90396 4104 3232 S 0.0 0.4 0:00.00 /lib/systemd/systemd-timesyncd 446 systemd-t 20 0 90396 4104 3232 S 0.0 0.4 0:02.51 /lib/systemd/systemd-timesyncd 491 systemd-n 20 0 18680 4892 3852 S 0.0 0.5 0:11.27 /lib/systemd/systemd-networkd 494 systemd-r 20 0 24052 9652 5528 S 0.0 1.0 0:02.52 /lib/systemd/systemd-resolved 540 root 20 0 18976 4348 2972 S 0.0 0.4 0:06.09 /lib/systemd/systemd-udevd 656 root 20 0 8536 2956 2680 S 0.0 0.3 0:02.84 /usr/sbin/cron -f 657 messagebu 20 0 7716 4608 3640 S 0.0 0.5 0:03.53 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation 667 root 20 0 29256 12252 4616 S 0.0 1.2 0:00.11 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers 684 syslog 20 0 219M 4288 2728 S 0.0 0.4 0:02.41 /usr/sbin/rsyslogd -n -iNONE 685 syslog 20 0 219M 4288 2728 S 0.0 0.4 0:00.00 /usr/sbin/rsyslogd -n -iNONE 686 syslog 20 0 219M 4288 2728 S 0.0 0.4 0:02.41 /usr/sbin/rsyslogd -n -iNONE 669 syslog 20 0 219M 4288 2728 S 0.0 0.4 0:04.94 /usr/sbin/rsyslogd -n -iNONE 671 root 20 0 18260 7508 5012 S 0.0 0.7 0:03.57 /lib/systemd/systemd-logind 673 daemon 20 0 3792 2336 2164 S 0.0 0.2 0:00.02 /usr/sbin/atd -f 719 root 20 0 7352 2100 1976 S 0.0 0.2 0:00.00 /sbin/agetty -o -p -- \u --keep-baud 115200,38400,9600 ttyS0 vt220 729 root 20 0 5828 1780 1664 S 0.0 0.2 0:00.02 /sbin/agetty -o -p -- \u --noclear tty1 linux 740 root 20 0 230M 3996 3092 S 0.0 0.4 0:00.00 /usr/lib/policykit-1/polkitd --no-debug 743 root 20 0 230M 3996 3092 S 0.0 0.4 0:00.21 /usr/lib/policykit-1/polkitd --no-debug </pre>

What do you think? Is this happening because the server is too overloaded?

<div class="yesno"> [[Yes->top 2]] [[No->top 2]] </div> CPU usage is basically at 0%. It's not happening because the server is overloaded.

<div class="nav go-back"> [[Go back -> Server]] </div> You decide to strace the server process.

<<ask "What command do you need to run to strace the server?" "$strace_cmd" "strace server 2">><<you-said "You said:" $strace_cmd>>

This game only knows one way to do it so here's what we're going to do. To strace the server, first you need to find its PID. You know that the program is called <code>server.py</code>.

<<ask "What command can you run to find the server's PID?" "$server_pid_cmd" "strace server 3">><<you-said "Your idea for the comand to run to find the server's PID:" $server_pid_cmd>>

There are lots of ways to do this, one ways is to run <code>ps aux | grep server.py</code>.

<pre> # ps aux | grep server.py root 96612 0.2 0.9 16168 9416 pts/0 S+ 14:35 0:00 python3 server.py root 96700 0.0 0.0 8164 736 pts/1 S+ 14:36 0:00 grep --color=auto server.py </pre>

<<ask "What's the server's PID?" "$server_pid" "strace server 4">><<you-said "You said the server's PID is:" $server_pid>>

The PID is 96612. You can see it here:

<pre> root 96612 0.2 0.9 16168 9416 pts/0 S+ 14:35 0:00 python3 server.py </pre>

Now that we know the PID, we can strace the server:

<pre> $ strace -f -p 96612 strace: Process 96612 attached accept4(3, </pre>

<div class="nav"> [[Make a request to the server->strace server 5]] </div> You make a request to the server by running <code>node slow.js</code>, and strace prints out a lot of output:

<pre> $ strace -tt -f -p $(pgrep python) 15:29:44.894106 accept4(3, {sa_family=AF_INET, sin_port=htons(48560), sin_addr=inet_addr("5.6.7.8")}, [16], SOCK_CLOEXEC) = 4 15:29:50.008666 getsockname(4, {sa_family=AF_INET, sin_port=htons(8000), sin_addr=inet_addr("1.2.3.4")}, [128->16]) = 0 15:29:50.008986 recvfrom(4, "POST / HTTP/1.1\r\nHost: mysterybo"..., 4096, 0, NULL, NULL) = 107 15:29:50.013696 recvfrom(4, "buy some milk\n", 4096, 0, NULL, NULL) = 14 15:29:50.055463 sendto(4, "HTTP/1.1 200 OK\r\nContent-Length:"..., 41, 0, NULL, 0) = 41 15:29:50.055787 close(4) = 0 </pre>

<<ask "Which part of this is the slow part?" "$strace_slow_2" "strace server 6">><<you-said "You said:" $strace_slow_2>>

It's actually hard to tell just from this!

There's a 41ms gap here:

<pre> 15:29:50.013696 recvfrom(4, "buy some milk\n", 4096, 0, NULL, NULL) = 14 15:29:50.055463 sendto(4, "HTTP/1.1 200 OK\r\nContent-Length:"..., 41, 0, NULL, 0) = 41 </pre>

But it's not easy to tell what causes it -- did the <code>recvfrom</code> system call take a long time to return, or were there 42ms of calculations in between the <code>recvfrom</code> and the <code>sendto</code>?

To figure it out, you can use the <code> -T</code> flag to strace which tells us how long each system call takes. The number in angle brackets (<code><......></code>) at the end of each line is how long that system call took, in seconds. Here's what that looks like:

<pre> $ strace -tt -T -f -p $(pgrep python) 15:56:15.517327 accept4(3, {sa_family=AF_INET, sin_port=htons(48564), sin_addr=inet_addr("5.6.7.8")}, [16], SOCK_CLOEXEC) = 4 <1.889267> 15:56:17.407185 getsockname(4, {sa_family=AF_INET, sin_port=htons(8000), sin_addr=inet_addr("1.2.3.4")}, [128->16]) = 0 <0.000018> 15:56:17.407481 recvfrom(4, "POST / HTTP/1.1\r\nHost: mysterybo"..., 4096, 0, NULL, NULL) = 107 <0.002145> 15:56:17.410349 recvfrom(4, "buy some milk\n", 4096, 0, NULL, NULL) = 14 <0.040266> 15:56:17.451423 sendto(4, "HTTP/1.1 200 OK\r\nContent-Length:"..., 41, 0, NULL, 0) = 41 <0.000031> 15:56:17.451710 close(4) = 0 <0.000055> </pre>

<<ask "What's causing the slowness?" "$strace_slow_3" "strace server 7">><<you-said "You said:" $strace_slow_3>>

This <code>recvfrom</code> is the slow system call: <pre> 15:56:17.410349 recvfrom(4, "buy some milk\n", 4096, 0, NULL, NULL) = 14 <0.040266> </pre>

<<newinfo "The server spends 40ms waiting for the client to finish the POST request">>

<div class="nav go-back"> [[Go back -> Server]] </div> Now that you understand what's happening, you need to figure out what to do about it.

<<ask "How do you think we should fix this?" "$how_fix" "how fix">><<you-said "You said:" $how_fix>>

This game isn't very smart, so it only knows about 3 options. Hopefully one of these is what you said:

<div class="nav"> [[Disable Nagle's algorithm -> disable nagle]] [[Disable delayed ACKs -> disable delayed ACKs]] [[Don't split the POST request into 2 packets -> send 1 packet]] </div> You decide to fix this by disabling Nagle's algorithm. You look up "disable nagle node js" and find out that you need to set the <code>TCP_NODELAY</code> socket option.

You just add one line of code to your test script: <pre> req.setNoDelay(true); </pre>

<div class="nav"> [[Run slow.js again -> win]] </div> You decide to fix this by disabling delayed ACKs on the server. You google "disable delayed ACKs" and you see that setting the <code>TCP_QUICKACK</code> socket option to 1 will disable delayed ACKs, but also some people are saying that the Linux TCP stack will sometimes turn off that setting, so it's not reliable.

You spend another ten minutes Googling and trying to figure out how to disable delayed ACKs but it doesn't seem that clear how to do it.

[[Try a different solution -> how fix]] One way to look at this is that this problem is happening because the client is splitting its POST request into 2 parts -- if it were only sending 1 packet, then it wouldn't have to wait for the second part to get sent.

This headers are being sent in a separate packet because of the <code>req.flushHeaders()</code> line in the uploadFile function -- it forces the headers to be sent right away.

<pre> async function uploadFile(filename) { const req = http.request({ hostname: 'mysterybox.local', port: 8000, method: 'POST'}, res => { console.log("Time elapsed:", new Date() - start, "milliseconds"); }) const stats = await fs.stat(filename) start = new Date(); req.setHeader('Content-Type', 'text/plain'); req.setHeader('Content-Length', stats.size); req.flushHeaders(); const data = await fs.readFile(filename, 'utf8'); await new Promise(resolve => {req.write(data, resolve)}); } uploadFile("test.txt") </pre>

You remove the <code>req.flushHeaders()</code> line and try running the code again.

<div class="nav"> [[Run slow.js again -> win]] </div>Now the request in the test script only takes 10ms, down from 50ms -- it's 5 times faster! Amazing! Not so coincidentally, the change of time is exactly 40ms.

<a href="https://www.freepik.com/vectors/background">image by brgfx</a>

<div class="nav"> [[Try a different solution -> how fix]] [[Epilogue -> Epilogue]] </div> This bug seems obscure if you haven't seen it before, but it's relatively common! It can happen any time a client is sending two small packets immediately followed by reading a response.Here are 5 blog posts talking about the same bug:

<ul> <li> <a href="https://rachelbythebay.com/w/2020/10/14/lag/"> 40 milliseconds of latency that just would not go away</a></li> <li> <a href="https://jvns.ca/blog/2015/11/21/why-you-should-understand-a-little-about-tcp/"> Why you should understand (a little) about TCP</a></li> <li> <a href="https://gocardless.com/blog/in-search-of-performance-how-we-shaved-200ms-off-every-post-request/"> In search of performance - how we shaved 200ms off every POST request</a></li> <li> <a href="https://vorner.github.io/2020/11/06/40-ms-bug.html"> 40 millisecond bug</a></li> <li> <a href="http://www.stuartcheshire.org/papers/NagleDelayedAck/">TCP Performance problems caused by interaction between Nagle’s Algorithm and Delayed ACK </a></li> </ul>

A "magic number" for this bug is 40 milliseconds -- that's the delayed ACK timeout. But like you can see in the "how we shaved 200ms off every POST request" blog post above, it can also manifest as a 200ms delay in some cases.

Because of this issue, a lot of programming languages (including Python, Ruby, and Go) set <code>TCP_NODELAY</code> on every TCP connection by default to disable Nagle's algorithm.

Also, the Linux kernel doesn't always enable delayed ACKs -- to reproduce this I actually had to write a Python program that explicitly turns them on. I haven't been able to find a clear explanation of exactly when delayed ACKs are used.

The reason the example code here uses Node is that node.js (as of May 2021) doesn't set <code>TCP_NODELAY</code> by default. The <code>req.flushHeaders()</code> thing is admittedly a little contrived.

<html> <script async data-uid="353d410db9" src="https://computer-wizard-industries.ck.page/353d410db9/index.js"></script> </html><<widget "newinfo">> <<if !$knowledge.has($args[0])>> <div class="newinfo"> <h2> you've found new information!! </h2> <<run $knowledge.add($args[0])>> Adding "<<print $args[0]>>" to your "What you know" list. </div> <</if>> <</widget>>

<<widget "init">> <<if ndef $knowledge>> <<set $knowledge = new Set()>> <</if>> <</widget>>

<<widget "ask">> <div style="margin-top: 3em;"> <<print $args[0]>> <<textbox $args[1] "" $args[2]>> <div style="display: flex; margin-top: .5em;"> <<link "Submit" $args[2]>><</link>> <<link "I don't know" $args[2]>><</link>> </div> </div> <</widget>>

<<widget "ask-area">> <div style="margin-top: 3em;"> <<print $args[0]>> <<textarea $args[1] "" $args[2]>> <div style="display: flex; margin-top: .5em;"> <<link "Submit" $args[2]>><</link>> <<link "I don't know" $args[2]>><</link>> </div> </div> <</widget>>

<<widget "you-said">> <<if !$args[1] eq "">><<print $args[0]>> "<<print $args[1]>>".<</if>> <</widget>><div style="text-align: left"> What you know: <<if $knowledge.size eq 0>> nothing yet! <<else>> <ol> <<for _i, _name range $knowledge>> <li> _name</li> <</for>> </ol> <</if>> </div>

The Case of the 50ms request

What you know: nothing yet!

<< Back to all puzzles

Hello! This is a mystery where your goal is to solve a debugging problem! You'll collect clues, interpret evidence, and ultimately solve the Case of the 50ms Request!

In the sidebar, you'll notice a "What you know" list. As you collect more clues, the list will update with what you've learned.

Click "Start" to get started.

Start

The Case of the 50ms request

The Case of the 50ms request

Recommend

Three interesting refrains of Coders at Work

《福布斯》潜在收购者概述区块链媒体策略

The FreeBSD Project | Introduction

老韭菜怎样玩转Chia挖矿？？？

FOX狐狸币上线在即，豪领空投1000w！

流量背后的焦虑，薇娅也在“另谋出路”

Using Webpacker in Your Ruby on Rails Application — a Deep Dive

How can we avoid coded bias in facial recognition tech?

如何整合 AdminLTE 3 和 Angular X

Branch predictor: How many "if"s are too many? Including x86 and M1 be...

About Joyk