Bits of py

I really like using Python's generators . Watching/reading David Beazley's talk on the topic of generators has made me try use them where ever practical. Recently I've had to deal with streaming and capturing massive amounts of data. Recording all traffic from a high baud rate CAN bus incidentally. Anyway I have Packets of unknown type and length coming in and want to store the sequence in such a way that I can recreate the full python objects on another computer. A quick solution is to serialize the data and store it to a file. The built in module for this is pickle . Unfortunately even with 12GB of RAM my work PC couldn't store the entire sequence in memory waiting to do a single serialization dump with pickle. After a bit of research I found a Python 2 implementation of streaming-pickle by Philip Guo . His solution didn't support bytearray objects and also suffered from a content boundary separation problem - multiple newlines within the pickled data coul

One of the projects I've been involved in at Dynamic Controls is building a Dyno Bench - essentially a set of motors that are very well instrumented. With this bench we can measure all sorts of things like Motor Speeds, Voltages, Current, and temperatures. Recently I've been investigating more and more in a live fashion - wanting to see changes in real time while I tweak parameters or inputs. I required a quick an easy way to see the frequencies present in an arbitrary signal. This is usually calculated after the test has run with a large Fast Fourier Transform . These calculations are quite expensive but with a small enough window and a powerful enough computer they can easily run in real time. By considering the frequencies of interest, as well as the computation power you have to play with you can make the trade off between a small enough sampling time dt and a large enough sample size n For a problem on Friday I was interested in frequencies in the 10 - 100 Hz range,