recoset

Efficient log processing

I've recently learned a couple of neat tricks to process large amounts of text files more efficiently. Our use-case is efficiently going through tens of gigabytes of logs to extract specific lines and do some operation on them. Here are a couple of things we've done to speed things up.