The Journey Continues: Creating My Own Image Processing Library

Introduction During my study of Machine Learning, I learned about a method for improving the generalization capability of Neural Networks for image-based datasets called Data Augmentation. Now, I suspected at the time that there existed at least one library in Julia that focused on DA (which turned out to be called Augmentor.jl, though there are probably others), but I wanted to see if I could figure out for myself how to do at least one image processing operation without any outside help, in particular, image rotation. So, one day while at work at my manufacturing job I started visualizing matrices representing images in my head, and after about 2 1/2 hours of thinking about it while working, I felt that I'd figured out how to do digital image rotation through arbitrary degree. I went home after work and coded it up in Julia, and, lo and behold, it worked! Now, it was extremely slow at first (about 1000 ms to rotate a 552x736 grayscale image) and used A LOT of memory, but long story short, I eventually got the time down to around 80 ms for the same operation. Nevertheless, that was still almost 10x as long as the image rotation function in Augmentor. This puzzled me and frustrated me for a while, and it took me a fairly long time to figure out what made Augmentor so much faster than my own method, but somehow I stumbled upon a writeup about backwards warping, or inverse mapping. I already knew about interpolation, though I didn't really get it at first, but after I learned about inverse mapping the concept of interpolation and why it's used for certain image processing operations finally clicked. So, I eventually abandoned (for the time being) my own image rotation method and adopted the inverse mapping + bilinear interpolation method that I suspect Augmentor uses, which was much faster and used FAR less memory. After a period of tinkering with the code I finally achieved what I'd set out to achieve when I first tested Augmentor: my rotation function was now faster than theirs! After transitioning to the updated rotation method and implementing it in code both on the CPU and GPU, I decided I would try to figure out all the other image processing operations Augmentor has in its library and implement those in code as well, both for a single image and for a batch of images, thus creating my own image processing library to be used for Machine Learning and other purposes. Here are the image transformation operations I have implemented, all of which I figured out how to do myself (with the exception of inverse mapping and bilinear interpolation¹, of course): distortion filtering flipping rotation scaling shearing zooming Four out of the seven operations use inverse mapping + bilinear interpolation: distortion, rotation, scaling, and zooming. Flipping doesn't use IM + BI because it doesn't need it, since the pixel coordinates for the relevant axis simply get reversed. Image filtering doesn't use bilinear interpolation because it's a different type of operation from the others (I don't know the exact theoretical details on what delineates the two types of operations, just that they're different). Finally, shearing doesn't use bilinear interpolation because I reasoned that, since the image only changed in one dimension, linear interpolation was sufficient. Again, I don't have a strong theoretical justification for doing this, just a logical one. Qualitative comparison with Augmentor While I was inspired by Augmentor and still highly regard it as a generally well-designed and well-organized library, I do have some issues with it. Issues I have with Augmentor The main issue I have with Augmentor is that it hasn't been updated since 2022. And because it has several external library dependencies, it presents problems when adding it to a Julia work environment. One can see all kinds of red and purple text where packages are being removed and downgraded after adding Augmentor to the environment. In addition, it causes the Plots package to error on my system, rendering it unusable. This is why I have a separate work environment just for testing my functions against Augmentor's. The Augmentor batch functions require that the user know the size of the output images ahead of time, since they have to be pre-allocated and passed to the functions. My batch functions don't require pre-allocation. They figure out the size of the output images within the functions, the same as my single image functions. The filtering function is GaussianBlur only. No option for a different filtering type, whereas mine takes a conv kernel as input so that any conv operation under the sun can be performed on an image. It appears that there is only one single-image function in Augmentor that is multithreaded, the GaussianBlur function, whereas all of my single-image functions are multi-threaded. Where my library comes up short compared to Augmentor Mine has limited cropping ability, whereas Augmentor has many different options for cropping. This can be changed before publishing, though, if necessary. The random variation in the rotation and other relevant functions are based on a simple seed; Augmentor's has more options. But again, this can be changed before publishing, if necessary. Other features are missing that Augmentor has, particularly useful utility functions. For more details, please see the Augmentor documentation. It was written by a self-taught non-professional, so it might not look as polished as those written by pros. Also, since I'm self-taught, etc., it might be more difficult for teams to consider using my library once published. But I don't want to get the cart ahead of the horse and assume something that may turn out to not be true. I don't yet have a method to chain together various image manipulation operations, though one is in the works, whereas Augmentor has very robust operation chaining capabilities and is really one of its most powerful and useful features. Where my functions shine They're generally pretty fast², usually faster than Augmentor's. They usually don't use as much memory as Augmentor's, though the difference is generally small, with some exceptions. My library has no major external library dependencies apart from what can be found in Base and other basic libraries such as Random. Each CPU image manipulation function has a GPU counterpart using custom GPU kernels for highest performance, whereas Augmentor is CPU only. My rotation, scaling, and shearing functions have an option to allow the user to choose which type of background to use, either :border or :value (though this may not be all that useful in practice). Did I mention that they're fast? Example quantitative comparison with Augmentor As an example of what my functions are capable of performance-wise, using my own image distorion function it takes my 1660ti MaxQ laptop GPU about 115 ms to performm 16 augmentations of the MNIST Training set that consists of 60000 28x28 grayscale images, which equates to about 7.2 ms per augmentation. The multithreaded CPU distortion function, by contrast, takes around 180 ms multi-threaded (~700 ms single threaded) to perform a single augmentation operation of the MNIST Training dataset, so 16 augmentations would take about 3.2 s multi-threaded (11 s single threaded). But this is a far cry from Augmentor, which takes about 1.8 s on my system per augmentation when multi-threaded to perform the same operation (11 s otherwise), and so would take about 29 s to do 16 augmentations (around 3 m single threaded). Again, though imperfect and a little unpolished, I used the distortion function under discussion to attain a max accuracy of 99.46% on the MNIST Test dataset in my Machine Learning program. Since this blog post is getting a little long, a follow-up post will present numbers from benchmarks to backup my claims that my functions are fast, or at least, that they in general outperform those in Augmentor as well as the Images.jl imrotate, imresize, and imfilter² functions on the CPU. I will also provide benchmark results for my GPU functions. Small but notable achievements Note: Some of these are a rehash of what I've already discussed. I came up with my own image rotation method, but it was slow and used a lot of memory, since it required repeating pixels, so it was eventually superseded by a better method (inverse mapping + bilinear interpolation). My image rotation method is only suitable for very high-resolution images, since they require no repetition of pixels to get good results, and is still available as a convenience function. I came up with my own formula for determining the dimensions of the output image for the convolution/cross-correlation operation for arbitrary stride and padding. (Why? Mainly just to see if I could, I guess. I like figuring things out on my own when I can.) My formula was tested against the existing formula on thousands of different image sizes, with each image using numerous different strides, and my formula always agreed with the existing formula. While this does not constitute a proof that they are equivalent, it does provide me with some confidence that it will give correct results for the vast majority of image sizes and kernel sizes that it will be used on. But a proof of their equivalence would probably be a requirement if I intended for my library be used in production. I came up with my own image distortion method, which allowed me to obtain 99.46% accuracy on the MNIST Test dataset after augmentation. While the images it produces are not as consistently good as Augmentor's elastic distortion method, and I haven't yet figured out how to keep all the pixels from the original image from going out of bounds, it's pretty darn fast and in fact keeps up with my other image transformation functions, whereas the Augmentor distortion method lags behind the performance of its other functions. I worked out how to do all the other image transformation operations: filtering (for which I'm most proud and the first one I did after image rotation, since I intended to eventually implement a CNN in code), flipping (the easiest), scaling (the largest function and the biggest headache, and probably the one most in need of refactoring), shearing (a bit more challenging than flipping, but not as challenging as some of the others), and zooming (which I eventually realized was just a special case of scaling, i.e., scaling with cropping). I figured out a method to determine when a convolution kernel is separable (well, any matrix that is capable of being separable, I guess), though it has its limitations and needs to be modified to handle certain kinds of kernels. This is also a work in progress and isn't ready for production use. TODOs Develop my own method of chaining together various image transformation operations. I have some ideas written down; I just haven't implemented them in code yet. Make my code production ready. Create a legitimate image processing library for data augmentation and eventually get it published as a registered library. Wrap-up Your comments and suggestions are appreciated. Thanks for reading. Notes ¹ I did try my hand at working out the math for bilinear interpolation, but as is normally the case I made an error, which after referencing the literature and correcting my mistake resulted in image quality comparable to Augmentor and imrotate and imfilter. ² The performance of my filtering functions falls far behind that of imfilter for larger kernels. It's hard to say, though, precisely when the inverse is true because it changes based on the image size. Apparently, imfilter uses a heuristic (don't know for sure) to determine when to apply multithreading, when to use FFT for convolutions, etc. and this changes based, no doubt, on several factors, one of which is image size. Mine, on the other hand, currently uses the same method of convolution regardless of kernel size and image size. But I think for the vast majority of non-separable kernels and image combinations that will be used in practice, especially for CNN's, mine is probably usually faster. Will have to do more testing to be sure. Separable kernels are another matter and is another area in which my functions need to be improved, since I don't yet have a method to more efficiently handle them. (Luckily, my GPU filtering functions are always faster than imfilter except for small images/arrays.)