This is a continuation of the first post, Hacker School Day 40 – Making pstree Faster/Ugly. The last post was where I removed the regex.
After removing the regex, the next place pprof pointed me to was how I was reading files. I originally used io/ioutil’s ReadAll method, which reads the whole file in. If you look at the source for ReadAll, it’s just calling a private read method, which is just calling bufio’s ReadFrom method. It also sets up a handler for panics in case the file contents are bigger than memory. /proc/pid/stat files are small — under 512 bytes (averaging ~212 bytes on my system).
Weird side note:
I’ve held on to this One Weird Trick for doing quick sums of columns of numbers on the command line. To get the average /proc/pid/stat size on my machine, I started by getting the size of each file:
find /proc -mindepth 2 -maxdepth 2 -type f -name stat -exec wc -c {} \;
Then I grabbed the first column with cut, and used the under-appreciated paste utility:
find /proc -mindepth 2 -maxdepth 2 -type f -name stat -exec wc -c {} \; | sort -gk1 | cut -d' ' -f1 | paste -s -d+
Paste is like a string Join method. I can specify the delimiter to join with, in this case, a plus sign. I get a string like “200+344+211…”, which can go right into bc:
find /proc -mindepth 2 -maxdepth 2 -type f -name stat -exec wc -c {} \; | sort -gk1 | cut -d' ' -f1 | paste -s -d+ | bc -l
Paste is neat. If you do this stuff often, you should be using num-utils, which is awesome. I don’t use it enough to remember it before paste, so I ended up dividing the sum by the count of files.
Anyway
I switched to bufio’s ReadFrom method for some savings. At this point it’s important to note that it would definitely be better to stick with ioutil because of the nice panic handler. This set of posts is just outlining some fun and dangerous ways I sped up some benchmarks.
The source code for ReadFrom is essentially calling the os.File’s Read method in a while loop, waiting for the end of the file. Since I know how about how big the file is going to be, I might as well cut straight to os.File’s Read! The code for Read just does a couple sanity checks before (indirectly) calling the read syscall. The documentation for File.Read is slightly confusing:
EOF is signaled by a zero count with err set to io.EOF.
If EOF is encountered at the end of reading a file, you’ll still get a real count, and no error. But, if the only character you read is EOF, you’ll get the above. I was reading with a buffer of 512 bytes, and getting 212 bytes back with no error. It was only until I tested with reading an empty file that I got the documented EOF response. This might need to be clarified in the docs, or maybe I was just easily confused.
It’s faster. It’s a lot more delicate too — it’s not very useful if the stat file suddenly got larger. The code is kind of safe in this case, since I only care about the span of the first four fields anyway.
Lastly, I changed the call to “defer fh.Close()” to just closing the filehandle there. I knew I was done with it, and there’s no point in telling Go to clean up later.
The last part is when I swapped out a call to filepath.Glob!