Home Projects

phixer - Phil's Photo Fixing App

Overview

The idea of this app is to provide something that can edit a photo and apply Machine Learning algorithms to help 'fix' the photo. It will also provide more standard features such as allowing the user to manually adjust exposure etc. and apply various filters and transforms.

This is essentially a port of the philter app to Apple's Metal framework (instead of OpenGLES), with Machine Learning algorithms added for the 'auto' functions.

Note: Info on the current page relates to approaches that might be usable in this app. For info purely related to implementing Machine Learning on iOS, check out the Machine Learning page.

General Functionality

In general, the app will provide the following types of functions (not including 'auto' functionality, which I'll list separately):

Auto (ML) Functionality

The intent is to review academic and open source Machine Learning (ML) papers and algorithms that provide various 'smart' or 'automatic' image adjustments or transformations. I will then port those algorithms to Keras (a Python-based ML framework that sits above Tensorflow or Theros) and verify on a GPU-enabled AWS server. Once verified, I will then port the Keras model to a Metal model, which can then be used on an iOS platform. Another approach would be to build TensorFlow into the app and call TensorFlwo directly.

The reason for the many steps is that the ML algorithms require a training (and verification) phase which is computationally very expensive, and best done on high end platforms that are easily available from Amazon (for example). Also, it is much easier to port the general training algorithm to a Python-based environment like Keras than it would be to port it directly to iOS, plus you can use the many Python-based libraries and tools to process the data and tune the algorithm. Once trained, the model is relatively easy to use in an app. BTW, the reason I chose Keras is that the iOS Metal framework is able to process Keras models.

The types of functions that I plan to implement include:

MLDL - Auto Fix

So, there are lots of algorithms out there that implement various forms of "auto-fixing" photos. The ones that seem most promising to me include:

and here is a list of some potentially useful algorithms related to this problem space:

MLDL - Auto White Balance

This is an operation something like Photoshop's Auto White Balance function, i.e. automatically adjusting the histogram to eliminate colour casts.
There are several ways to do this:

MLDL - Low Light Enhancement

MLDL - Style Transfer

MLDL - De-noising

MLDL - Foreground Matting

Implementation Notes

Here are some random notes and links related to implementation approaches and procedures