Conill: How efficient can cat(1) be?
source link: https://lwn.net/Articles/901707/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
Conill: How efficient can cat(1) be?
The first possible option is the venerable sendfile syscall, which was originally added to improve the file serving performance of web servers. Originally, sendfile required the destination file descriptor to be a socket, but this restriction was removed in Linux 2.6.33. Unfortunately, sendfile is not perfect: because it only supports file descriptors which can be memory mapped, we must use a different strategy when using copying from stdin.
(Log in to post comments)
Conill: How efficient can cat(1) be?
Posted Jul 18, 2022 19:00 UTC (Mon) by qyliss (guest, #131684) [Link]
Since version 9.1, cat in GNU coreutils will use copy_file_range
if possible. Surprisingly, it's actually the first program in coreutils to use that API. Which means that, for a disk image build I have, that requires assembling individual partition images into a combined GPT image, it's significantly faster to write each partition into place with cat
than it is to use dd
, which I'd have expected to be the better tool for the job.
Conill: How efficient can cat(1) be?
Posted Jul 19, 2022 10:45 UTC (Tue) by ddevault (subscriber, #99589) [Link]
Here's my implementation of cat in Hare, for general interest. io::copy uses sendfile when possible, and perhaps splice could also be added, but the extra pipe(2) is not great IMO, could fail if file descriptors are exhausted and has other side-effects besides. Simplicity trumps efficiency for tools like this in my book.
use fmt; use fs; use getopt; use io; use main; use os; export fn utilmain() (main::error | void) = { const cmd = getopt::parse(os::args, ('u', "POSIX compatibility, ignored"), "[file...]"); defer getopt::finish(&cmd); if (len(cmd.args) == 0) { io::copy(os::stdout, os::stdin)?; return; }; for (let i = 0z; i < len(cmd.args); i += 1z) { const file = open(cmd.args[i]); match (io::copy(os::stdout, file)) { case size => void; case let err: io::error => io::close(file): void; return err; }; io::close(file)?; }; }; fn open(path: str) io::handle = { if (path == "-") { return os::stdin; }; match (os::open(path)) { case let err: fs::error => fmt::fatal("Error opening '{}': {}", path, fs::strerror(err)); case let file: io::file => return file; }; };
Conill: How efficient can cat(1) be?
Posted Jul 19, 2022 14:06 UTC (Tue) by mathstuf (subscriber, #69389) [Link]
Conill: How efficient can cat(1) be?
Posted Jul 19, 2022 14:12 UTC (Tue) by ddevault (subscriber, #99589) [Link]
Conill: How efficient can cat(1) be?
Posted Jul 19, 2022 14:43 UTC (Tue) by zdzichu (subscriber, #17118) [Link]
Nevertheless, the original blog post was about efficiency. Could you please benchmark the code from blog and your implementation?
Conill: How efficient can cat(1) be?
Posted Jul 19, 2022 15:07 UTC (Tue) by ddevault (subscriber, #99589) [Link]
My point is to suggest that efficiency is not always the correct measure. I am questioning the article's premise. There are vanishingly few situations where cat is your bottleneck, and you would be better off with a simpler (and thus likely to be less buggy!) implementation which can be relied upon to work and be understood.
But sure. I'm not on a particularly good laptop here, so I'm working with a 2G test file instead of 4G. Re-running the benchmark from the programs in the blog post in order, I get 2.7 GB/s, 6.4 GB/s, and 7.1 GB/s. My implementation gets 6.4 GB/s. This is not really surprising since io::copy uses the same sendfile approach as the second program in the blog post and the bottlenecks have everything to do with the syscalls and almost nothing to do with the userspace code.
But what's important is that all of these numbers are way more than "fast enough" for essentially any use-case of cat.
Conill: How efficient can cat(1) be?
Posted Jul 19, 2022 15:27 UTC (Tue) by Wol (subscriber, #4433) [Link]
Which may be true for code that is not run that often.
But for a utility like cat that is used again and again, while the gain may be minimal per invocation, it is also colossal over time. A tiny gain in heavily used code can easily trump a huge gain in rarely used code.
And if the code is heavily used, there are (hopefully) a lot of people who understand it. Just because you think the gain isn't worth it - for you - there are plenty of people who will think the gain worth a lot. Especially in datacentres where those few milliseconds add up to lots of £££ ...
Cheers,
Wol
Conill: How efficient can cat(1) be?
Posted Jul 19, 2022 15:33 UTC (Tue) by ddevault (subscriber, #99589) [Link]
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK