Video editing in Julia for SF2526
(There is also a MATLAB version of this tutorial: Video editing in MATLAB for SF2526)
This is a tutorial on video and image processing in Julia. You may need to install the Images package and other packages used below. To install packages in Julia:
julia> ]
(@v1.7) pkg> add Images
When done, use backspace when the line is empty or press Ctrl+C to return to the normal julia> prompt.
Documentation for the Images package can be found at https://juliaimages.org/latest/
Links to an external site.
Loading images
To load images in Julia, one can use the load command found in the Images package:
julia> using Images;
julia> fname = "SF2526_numerics_for_data_science.png";
julia> img = load(fname);
julia> size(img)
(768, 1145)
julia> typeof(img)
Matrix{RGB{N0f8}} (alias for Array{RGB{Normed{UInt8, 8}}, 2})
Julia stores images as an array of pixels, where each pixel encodes a color. The image we loaded contains 768×1145 pixels (height×width). In this example, each pixel is an RGB{N0f8} object, which means that it uses three numbers between 0 and 1 to encode the amount of red, green and blue. The individual color channels can be extracted using red(), green() and blue(), which can act either on a single pixel value, or on the whole image (then a dot is needed after the function name):
julia> img[1,1] # pixel in top left corner
RGB{N0f8}(0.98,0.98,0.98)
julia> red(img[1,1])
0.98N0f8
julia> red.(img)
768×1145 Array{N0f8,2} with eltype N0f8:
0.98 0.98 0.98 0.98 0.98 0.98 … 0.98 0.98 0.98 0.98 0.98
0.98 0.98 0.98 0.98 0.98 0.98 0.98 0.98 0.98 0.98 0.98
⋮ ⋮ ⋱ ⋮
0.98 0.98 0.98 0.98 0.98 0.98 0.98 0.98 0.98 0.98 0.98
Images can be displayed using the command plot from the Plots package:
julia> using Plots;
julia> plot(img) # No semicolon here!
It is easy to change the pixel format. For example, Gray.(img) converts the image to grayscale (common pixel formats are RGB, Gray and Lab):
An alternative to doing red.(img), green.(img) and blue.(img) is to use the function channelview:
julia> ch = channelview(img);
julia> size(ch)
(3, 768, 1145)
julia> typeof(ch)
Base.ReinterpretArray{N0f8, 3, RGB{N0f8}, Matrix{RGB{N0f8}}, true}
The array ch is now of size 3×768×1145, where the first index is the color channel. This is similar to how MATLAB stores images (although MATLAB has the color channel as the third index). Doing plot on a channelview (like ch) will not work. To convert individual channels back to an image (that can be plotted or saved to disk), use colorview:
julia> img2 = colorview(RGB, ch)
768×1145 Array{RGB{N0f8},2} with eltype RGB{N0f8}:
RGB{N0f8}(0.98,0.98,0.98) RGB{N0f8}(0.98,0.98,0.98) … RGB{N0f8}(0.98,0.98,0.98)
RGB{N0f8}(0.98,0.98,0.98) RGB{N0f8}(0.98,0.98,0.98) RGB{N0f8}(0.98,0.98,0.98)
⋮ ⋱
RGB{N0f8}(0.98,0.98,0.98) RGB{N0f8}(0.98,0.98,0.98) RGB{N0f8}(0.98,0.98,0.98)
julia> img2 = colorview(RGB, ch[1,:,:], ch[2,:,:], ch[3,:,:]); # can also give channels separately
Let's say we want to make the top left corner of the image red. This can be done as follows (note the dot before the equals sign):
julia> img[1:100,1:100] .= RGB(1,0,0);
julia> plot(img)
To save an image, use save(filename, img), where filename is a string.
Vectorizing the image into a vector
A video is a sequence of images (called frames), and we want the sequence of images to be stored in a matrix. For this reason we need to turn our image into a vector. We will do this by stacking the elements vertically (also known as vectorizing or reshaping the image). The resulting vector is of size 3*prod(size(img)):
julia> sz = size(img)
(768, 1145)
julia> R = red.(img);
julia> G = green.(img);
julia> B = blue.(img);
julia> v = vcat(vec(R), vec(G), vec(B));
julia> size(v)
(2638080,)
julia> n = 3*prod(sz)
2638080
Later we want to reverse this transformation, which can be done with the reshape command:
julia> vv = reshape(v, sz[1], sz[2], 3);
julia> R = vv[:,:,1];
julia> G = vv[:,:,2];
julia> B = vv[:,:,3];
julia> img_new = colorview(RGB, R, G, B);
Loading a sequence of images (video)
In the course files area you will find videos stored both as avi/mkv-files viewable directly in your browser, and as snapshot zip-files. In the zip-archive you will find png-files with names like testbild_snapshot_0004.png, which represents frame 4 in the video. In your for-loop where you turn the sequence of snapshots into a matrix, you need to create the filename. For this filname format, the name can be constructed with the @sprintf macro from the Printf package:
julia> using Printf;
julia> k = 4;
julia> filename = @sprintf("testbild_snapshot_%04d.png", k)
"testbild_snapshot_0004.png"
julia> frame_k = load(filename);