Home Projects

Machine Learning/Deep Learning (MLDL)

Overview

What is Machine Learning?

Machine learning is a subset of artificial intelligence in the field of computer science that often uses statistical techniques to give computers the ability to "learn" (i.e., progressively improve performance on a specific task) with data, without being explicitly programmed.
If you really want to learn more about this, I strongly recommend the Cousera online class on Machine Learning taught by Professor Andrew Ng of Stanford University.

Why Am I Interested?

One of my main interests is in photography. I've been working on a custom photo editor on and off for a while now and many of the 'automatic' functions are extremely complex to program. Machine Learning provides an approach where I can take models developed in research applications and transfer them to 'real' apps and use them to automatically adjust your own photos.

Interesting Links (Photography-related)

Title/Link
Description
Automatic image retouching on your phone System can apply a range of styles in real-time, so that the viewfinder displays the enhanced image.
starcolon/photo-auto-balance photo-auto-balance - :sunrise: Automated image color correction / enhancement with Deep Learning (CNN) ***(NO LONGER MAINTAINED)
Automatic Tone Mapping Results
This page shows several results produced by our automatic tone mapping algorithm, inspired by Ansel Adams' Zone System. Reinhard's global tone mapping curve was used to assign a target exposure value for each zone. We compare our results to those produced by simply applying Reinhards tone ...
RT Tone Mapping - Master's Thesis
GPGPU Techniques for Real Time Tone Mapping in the Context of Virtual Night Driving
Color Balancing Algorithms | Illuminant Estimation | Simplest Color Balance This is the color balancing technique used in Adobe Photoshop's "auto levels" command. The idea is that in a well balanced photo, the brightest color should be white and the darkest black. Thus, we can remove the color cast from an image by scaling the histograms of each of the R, G, and B channels ...
Learning to See in the Dark
A pipeline for processing low-light images, based on end-to-end training of a fully-convolutional network.
Fast Neural Style Transfer
Justin Johnson github repo
Deep Photo Style Transfer This paper introduces a deep-learning approach to photographic style transfer that handles a large variety of image content while faithfully transferring the reference style.
luanfujun/deep-photo-styletransfer Code and data for paper "Deep Photo Style Transfer"
Deep Image Matting Our deep model has two parts. The first part is a deep convolutional encoder-decoder network that takes an image and the corresponding trimap as inputs and predict the alpha matte of the image. The second part is a small convolutional network that refines the alpha matte predictions of the first network to have more accurate alpha values and sharper edges
Deep Painterly Harmonization
Copying an element from a photo,  pasting it into a painting, and matching the style of the painting
Image Denoising State-of-the-art
A curated list of image denoising resources and a benchmark for image denoising approaches.


Implementation links

The following links provide info related to implementing MLDL apps on iOS. Of course, Apple doesn't support any of the commonly used MLDL frameworks directly (quelle surprise) so we have to play some games to take an existing model and get it running in an iOS app.

Creating and Using CoreML Models

A neat thing about the CoreML framework provided by Apple is that you can just drag an existing pre-trained model into your Xcode project and it will create accessor classes for you to use the model, and the underlying framework will take care of using CPU vs GPU etc. If you also use the Vision framework, then that will take care of converting image formats for you as well.

Converting models to the CoreML format:

There is a python package called coremltools provided that allows you to save various types of models as a CoreML model. Unfortunately, Tensorflow is notably not supported, and curently only keras 1.2 (not 2.x) is supported. The two main formats that will be encountered in this problem space are caffe and keras. Examples are given below:

Keras:


import coremltools
# ...load the model that you want to convert (kmodel here)...
coreml_model = coremltools.converters.keras.convert('kmodel.h5')
coremltools.utils.save_spec(coreml_model, 'my_model.mlmodel')
        

Caffe:


import coremltools
# ...load the model that you want to convert (cmodel here)...
coreml_model = coremltools.converters.caffe.convert('cmodel.caffemodel')
coremltools.utils.save_spec(coreml_model, 'my_model.mlmodel')

Using a Model in Code

To add a model to your project, just drag the model file into the Xcode window. There should be a viewer that let's you examine the model and determine the class name and the names/types of the input and output variables. Xcode also uses information about the model’s inputs and outputs to automatically generate a custom programmatic interface to the model, which you use to interact with the model in your code. For MyModel.mlmodel, Xcode generates interfaces to represent the model (MyModel), the model’s inputs (MyModelInput), and the model’s output (MyModelOutput). Use the generated MyModel class’s initializer to create the model:

let model = MyModel()

let srcImage = UIImage(named: "myImage.jpg")!

guard let myModelOutput = try? model.prediction(srcImage: image) else {
    fatalError("Unexpected runtime error.")
}

let outImage = myModelOutput.outImage

 
and the equivalent code using Vision is:

let model = MyModel()

let image = UIImage(named: "yImage.jpg")!

guard let visionModel = try? VNCoreMLModel(for: model.model) else {
  fatalError("whoops")
}

let request = VNCoreMLRequest(model: visionModel) { request, error in
if let observations = request.results as? [VNCoreMLFeatureValueObservation],
    let output = observations.first?.featureValue.multiArrayValue {
    // do something with output
    }
}

let handler = VNImageRequestHandler(cgImage: image.cgImage!)
try? handler.perform([request])