I put the task of accelerating the color enhancement filter and removing flicker aside a bit, and ported Autostretch HSV, a very similar GIMP filter that does histogram stretching directly on the saturation and value channels of a HSV image. I've just pushed the new code to github. Its current problems are the same as with the color enhancement filter: flicker, and speed. On my test video, the effect (and problems) were clearly observable with contrast and saturation reduced by a factor of 0.2 using the videobalance plugin. (Also, this was a nice occasion to finally try the tee element. :-) )
For the flickering introduced by maxima and minima being calculated and used directly for every frame, I tried to keep a moving average of them from 50 frames. That didn't work out very well.
For speed, I'm still looking for options. I currently don't see anything substantially improvable in the code, aside from an efficient YUV to HSV conversion algorithm. If such an algorithm exists, we could skip the YUV-RGB conversions.