Egyptian Hieroglyphics and Google Cloud AutoML Vision

Lately I've been on a ancient history kick and while daydreaming about visiting Egypt in person I was curious to find out that only small percentage of hieroglyphics have been translated. My thoughts wandered to a mobile app that uses the camera and machine learning to interpret the hieroglyphics in real time and a quick google search found I'm not the only one who has had the idea. I found an image dataset, complementary to a paper named "Automatic egyptian hieroglyph recognition by retrieving images as texts" by Morris Franken and Jan C van Gemert. 2013. In ACM Multimedia Conference. ACM, 765–768.

Morris has made some contributions on GitHub and uses python to train a data model using a dataset of images of hieroglyphics found in the Pyramid of Unas.  His python script then uses the trained model to classify an image of a hieroglyph and return the Gardiner labels.

I immediately start looking at TensorFlow.js and look at reusing the data model created by Morris's python script. Combining the ability to do image classification in Javascript with React Native's ability to access a mobile device's camera is a fast path to making a real time hieroglyph translating mobile app into a reality.

My first attempts were to convert the data model created by Morris into the model type needed by TensorFlow.js however there seems to be a lack of tooling to convert machine learning data models to different ML libraries.  TensorFlow.js provides a pretty robust walkthrough on classifying handwriting images and while this is the logical path forward I wanted to experiment with the AutoML service currently in beta in Google Cloud Platform.

I will not be going over how to create a new project, enable billing, and enabling APIs for AutoML in Google Cloud Platform, however, when you browse to this service in the Google Cloud Platform console you are presented with a walkthrough to perform these tasks.

After the enabling the proper services you are prompted to create a dataset.  I entered a dataset name and when I moused over the help icon next to Upload images from your computer I found that if you organize the folder structure inside a zip file of images it would automatically label my images!!! After spending some time reorganizing my image dataset I found that this feature is not yet active :(

After uploading the reorganized dataset I now have all of my images uploaded but none of the images have labels and Google currently provides 3 options for labeling the images:

1. Manually create labels and then manually associate your images with a label
2. Human help is provided by Google
3. CSV import using images stored in Google Cloud storage

This dataset has over 4000 images so I wont be manually labeling and while I believe Google is offering Human labeling free of charge while in beta I believe I can construct a CSV in the proper format in short order.

The first step to create the CSV was using the Export Data option in the AutoML console.  

When I imported my images the console uploads them to a google storage bucket so by exporting this spreadsheet we have the CSV in the format we need and only need to add labels to the C column.

As previously stated, before uploading I restructured the images into folders named with the images' labels so the label in part of the google storage location path. I will use a formula in the D column to parse the label from the path and then copy and paste those values into the C column.

I used this formula for extracting a string from cell contents. It basically says give me the 6 characters starting at the 65th character position from the contents of cell B1 and then search that for the "/" character and only return the characters before. Otherwise it returns "UNKNOWN".

=LEFT(MID(B1,65,6), SEARCH("/", MID(B1,65,6))-1)

After saving this CSV you will need to upload it into the google storage location where your images are stored.  In my case this path is gs://glyphy-222400-vcm/PyramidOfUnas/uploads/unzip/FolderLabels/PyramidOfUnasDataset.csv

After uploading the CSV to google storage browse to the dataset and click on Select CSV file on Cloud Storage and put in the path to the CSV file we added our labels to and click Import.

This will reimport the images with labels and thankfully does not duplicate the images.  After importing I now have all 4210 images labeled!

Now we are ready to move on to training so I click the Train button but am presented with a prompt to label more images. In this dataset there are several Gardiner labels that have less than 10 example images so at this point I have slimmed down the spreadsheet to include only those Gardiner labels who have more than 10 example images and created a new dataset from this new spreadsheet which has 3803 images and thanks to the CSV import they all have associated labels.

Now each label should have at least 10 images assigned before you can train a model. So now I'll continue on and train the model.

After training the model I visit the evaluate tab to analyze some statistics on classification precision.

And finally on the Predict tab we can upload an image from our dataset to test out our model.

At the bottom of the predict page you can choose between implementing the model we've just trained by using python or a REST API. So in conclusion we have taken the hieroglyph dataset assembled by Morris Franken and uploaded it with labels to the Google Cloud AutoML engine to train a data model.

Soon I'll be working on creating a React Native app that utilizes the camera to take a picture and submit it to our AutoML data model's REST API for classification and return the english translation of the Gardiner label.