Modify your OT-2 to be a colony picker costing less than $300
Fig.1 - The OT-2 equipped with the CPU (Colony Picking Unit).
Opentrons OT-2 is currently the cheapest liquid handling robot on the market and each year many iGEM teams
get their
hands on one through the Opentrons Robot Challenge. This pipetting robot is not only interesting for iGEM
teams, but
also for start-up companies (Link to Doulix IHP). To exploit the full potential of the robot and facilitate
the
workflow of cloning, we decided to turn the OT-2 into a fully functioning colony picker for a relatively low
price and
open-source based.
Upgrade your OT-2 with the CPU (Colony Picking Unit)
To enable colony picking with the OT-2 we developed a package of custom-made modular hardware, called the
Colony
Picking Unit (CPU), which fits perfectly into the slots of the OT-2. All devices are simple to install, easy
to
use
and low cost solutions, while maintaining a high quality
Fig.2 - The Raspberry Pi 4 and the ArduCAM fixed in the 3D printed case
attached
to the arm of the OT-2.
Fig.3 - Our custom made light table in action while taking a picture of the 90 mm agar plate with colonies
The CPU consists of the light table, containing an Arduino and LEDs making it possible to
automatically
switch
the light on and off while taking pictures of your plate, and the mount for the OT-2 arm, where the
Raspberry Pi
4 Model B with an ArduCAM will be mounted, taking and subsequently analysing the corresponding
picture.
Of course, all the files required for 3D printing and instructions on how to install the electronic
devices
are
provided open-source (link to hardware and
github).
For analyzing the taken picture and identifying colonies we decided on using a neural network based
artificial
intelligence (AI) instead of a hard coded image processing approach This ensures a flexible solution and
makes it
possible to extend the functionalities in the future even further.
To train the MCoINN, many pictures of agar plates containing E. coli colonies were taken, not only by our
team,
but
also by other iGEM teams in the context of a collaboration (Link to collaboration). Each individual colony
which
seemed suitable for picking was manually labelled, serving as training data for MCoINN to define the
characteristics
of a suitable colony.
Fig.4 - Picture of the agar plate that was taken with the ArduCAM before
(left)
and after (right) being processed by the AI. The identified colonies are marked in green.
MCoINN was trained with around 200 pictures, of which every single one contained multiple colonies. The
resulting
colony prediction was very reflective of the training data we trained MCoINN with. This was very surprising
because we
expected it to work with much more training data (~1000 pictures) that we did not even need to do image
augmentation
for the training data. With much more time it will be interesting to retrain the algorithm with augmented
images
that
will multiply the quantity of the data by at least a factor of 4. Nevertheless this is a very welcomed
development,
because this lowers the threshold for a custom AI algorithm even lower. The next time any user will only
need
around
50 colony pictures and image augmentation algorithm to train a functional AI. Moreover, other users who are
interested
in building their own AI will also benefit from our dataset, which can provide basic structures for the AI
regardless
of however their colonies look like. We got the pictures from our colony picking pictures collaboration and
this
is
very beneficial for the training because the AI got exposed to various kinds of real lab conditions. More
details
on
the implementation of the AI can be found in
github1
and
github2
Fig.5 - Visualization of the training via TensorBoard. On the x-axis is the run number and the loss values
on
the
y-axis.
Let the OT-2 pick them for you
Combining the designed hardware with the trained AI will turn the OT-2 in a low-budget colony picker. The
only
thing
we had to do, was putting the light table in the OT-2 and connecting it with the robot, mounting the
Raspberry
Pi
4
and the ArduCAM onto the arm, putting an agar plate onto the light table and then taking a picture of that
specific
plate. The trained AI would do the rest by identifying the colonies, translating the coordinates from the
picture
into
coordinates of the OT-2 system, to eventually move to that certain position and pick the identified
colony.
Fig.6 - Colony picking in action - the OT-2 in the progress of picking a colony with the P10.
Via the GUI the OT-2 will take a picture of the colony laid on the light table, which will then be processed
by
the
MCoINN. After analysis coordinates of the colonies from the picture is extracted and translated into
coordinates
that
are intelligible for the OT-2. Via the move_to function from the robot library provided by Opentrons, the
pipette
will
be moved to the target colony, which will be picked.
All done on the Graphical User Interface for Directed Engineering (GUIDE)
We at iGEM Marburg are serious about bringing AI and robotic revolution to biology labs. For this the
system
has
to be
user friendly and therefore we created a GUI. It will increase the accessibility of the software: with the
GUI
people
with less technical experience can also operate the colony picker. Moreover the AI training is packaged
into
the
GUI
and will allow people to retrain their colony picker according to their own individual needs. Using AI as
a
backend
allows our system to be flexible and using a GUI makes it more accessible.
The GUI is written on Python using the Kivy library. On the backend it connects to a server/ client on
both
the
camera
module and the Raspberry Pi inside the OT-2 and acts as a man in the middle communication between both.
The
GUI
enables calibration of the OT-2, labware deck overview, and colony picking within a few clicks to
guarantee
accuracy
and intuitive colony picking experience.
Fig.8 - GUI
Build AI for Your own Need
Our AI is trained for detecting white E. coli colonies, but we understand that other people’s needs will
differ.
That
is why we chose AI instead of hard-coded algorithm: so that other potential users can build one according
to
their
needs. Some would like to pick bigger, or smaller, or green colonies; you name it. We designed our GUI so
that
other
users can easily train their own AI that will suit their needs.
We contained all of our work on AI within a docker file, which enables training a new custom AI as easy as
implementing a few commands. Users will just need to provide a folder containing training and test images
according to
their own needs. The AI can then be trained on a local computer and will output frozen inference graph at
the
end,
which then can be utilized according to each user’s needs. More details on the implementation of the AI
can be
found
here
and
here
How does it all work?
Here is a graph that explains our project:
Conceptual flowchart of the CPU. Red denotes steps that involves the
GUI.
