Breathing new life into iconic moments

Share this post:

Author: Adam Makarucha, AI Practitioner and IBM Q Ambassador, IBM Systems

The 70s and 80s were full of iconic moments, religious and royal visits, unique styles (who can forget the flares) and vibrant colour. Looking back, however, some of these historic Australian moments were only ever captured on film in black and white. Not because colour photography was unavailable at the time. But because newspapers were not yet printing in colour, so why go to the effort and cost of developing colour photos!

The Sunday Telegraph, News Corp. May 31st, 2020

On May 31st, six of these historic and iconic images were brought to life and published in colour for the very first time in The Sunday Telegraph. This continues the partnership with Jeff Darmanin, Head of Vision and Picture Editor of The Sunday Telegraph. The partnership started with our work to remaster photographs of Anzac heroes using state-of-the-art AI for a special Anzac Day lift-out (see my colleague Ben Swinney’s blog here). The amazing outcome of these projects is a testament to the capability of modern AI techniques coupled with IBM’s Power Systems hardware.

“Remaster” is a pretty loose term and could mean anything so let’s start by defining what we actually did to remaster these photos. We applied three AI-based approaches to the photo enhancement processes. These included denoising, upscaling and colourisation. What each model tries to achieve is in the name, but the source of the issues we were trying to correct may not be so obvious.

Singer Tina Turner with Players Allan Langer, Andrew Ettingshausen and Wayne Pearce during filming for the ‘Simply the Best’ advertising campaign for the 1990 rugby league season. Credit – Peter Muhlbock, News Corp

Firstly, the process of denoising is aimed at removing the noise (unwanted variation in colour and brightness). This noise comes from the original camera (which introduced film grain noise), the scanner that digitised the photo (luma and chroma noise) and then the digital compression to store the images in jpeg format (random noise). The photos we remastered were taken some 30 to 40 years ago and were digitised at some point between now and then. As a result, the black and white digital copies of the photos that News Corp had, came in varied resolutions with most being under 3MP (where your average iPhone takes photos at 12MP).

The second process we undertook was to take the black and white digital still images and upscale them to as high a resolution as possible. This was important to ensure, when printed at full page size in the newspaper, they would retain as much detail and clarity as possible. This process is known as super-resolution in the AI community.

The final image enhancement, and perhaps the most obvious, was that of colourisation to recreate the images in colour.

Gough Whitlam on the steps of Parliament House in 1975 listens as an official reads a statement to the press that Sir John Kerr had dismissed the Labor government. Credit – Maurie Wilmott, News Corp.

We also used a family of AI models known as generative deep learning which are developed specifically to generate new data. They take the input image and attempt to create a new image that has less noise, is a higher resolution or is accurately colourised.

To train a generative model to create new images, we need a dataset to train on, with input and output pairs, just like any other AI model. Creation of this dataset is easy – for the super-resolution task you basically take a high-resolution image (our output image), copy it and then downscale the copy to a lower resolution (our input image). Literally, anyone with an image editor can do this. You just have to repeat for millions of images, so we automate this part, and boom, you now have a dataset with millions of images that can be used to train a model to upscale images. We generate two more datasets for the two other process. One where we take sharp and clear images (output) then add noise to the image (input), and one where we take colour images (output) and turn them into black and white (input).

Salesman and TV personality ‘The Demtel Man’ Tim Shaw kisses Princess Diana’s hand at Sydney’s Bayview Wharf when she toured Australia in 1988. Credit – News Corp.

Once you have a dataset, you can start training the AI, which can take days. This is where having some advanced GPU accelerated hardware comes in pretty handy to speed things up. When we train a generative model, say the colourisation model, we take the dataset we produced earlier showing the AI random black and white images from this dataset, and it attempts to generate colour images that reproduce the colour of the original image. Initially, it’s terrible and produces the sort of colourisation my newborn son would think was appropriate.

However, after millions of attempts on millions of images, it eventually becomes the Leonardo da Vinci of the AI world, learning how to colour the images in style indistinguishable from real life. In fact, it gets so good it can recognise that a person is a person, a tree is a tree and buildings are buildings and so on. With this training, it can colour these objects and people accurately, even if they’re wearing different styles of clothing or are in completely different locations.

What was unique about our process is that we leveraged some open source code produced by IBM known as Large Model Support (GitHub – IBM/pytorch-large-model-support: Large Model Support in PyTorch) to enable us to push the limits of resolution and size of the AI models. We started with a terrific project called DeOldify produced by Jason Antic, adding Large Model Support (LMS) and training models on the IBM Power System AC922.

Pope John Paul II takes a drive around  Sydney harbour foreshores in the famous ‘Popemobile’ during his tour of Australia in 1986. Credit – Ray Strange, News Corp.

The outcome of this training is a model that can produce, in seconds, the clear, high resolution and colour images you see pictured here from the original low resolution black and white photographs.

Stay tuned for a follow-up to this blog where I’ll dive into what changes we made to the underlying Generative Adversarial Networks and Autoenconder models used.