When farmers sell their canola crop to grain elevators, the price is dictated by the grade of their harvested canola seeds. The grading of canola seeds is currently a manual process whereby human eyes examine and grade samples of canola seeds. While thorough, this grading methodology is highly prone to subjectivity, sometimes leading to inconsistent gradings for the same crop of canola. With drastic price differences between grades of canola, unpredictability is a major concern to canola growers.
Therefore, created an objective seed sample grading system that attempts to consistently provide accurate seed grades by utilizing a synthesis of hardware and software tools. The system is comprised of a physical lightbox (named the Mean Green Machine (MGM)) with an embedded or external camera for the standardization of seed sample photos, with software (named GreatGrader) to analyze and automatically grade the given images. The utilization of this tool by farmers, grain elevators, and oil producers will improve the consistency of canola seed gradings, lowering uncertainty for all parties involved in canola transactions.
Figure 1. Our standardized seed grading system: Mean Green Machine (left) and GreatGrader (right)
The Importance of Seed Grade
“Green is the difference between profit and loss.” - Craig Shand
In one of our first talks with canola farmers it was identified that we had to do something about canola grading. For example, in a meeting with Craig Shand, a Southern Alberta farmer, he recounted how two different grain elevators had given him two different grades for the same crop of canola seeds. Additionally, he revealed that his experience was not an uncommon one among farmers.
As explained by Craig Shand, this issue has an enormous impact on farmers as selling grade No. 1 seeds yields enough revenue for farmers to make a profit, but grade No. 2 seeds net around $10-$20 less per tonne of canola, potentially halving earnings. This is due to the fact that Grade No. 2 Seeds and lower (like with grade No. 3 seeds) can only really be used to produce meal due to their chlorophyll content. Therefore, a farmer that produces anything but grade No. 1 seeds will risk heavy losses.
Addressing Seed Grading
Seeds are sampled randomly, but are graded by the human eye. The human eye is a marvel, but correctly identifying exact colours of all seeds perfectly is difficult, especially when large volumes of seeds must be graded for hours on end.
Figure 2. Manual seed grading
To help farmers, and all those affected by inconsistent grading practices, we built a tool with the intention of standardizing the grading of the seeds. If successfully implemented in industry, farmers and grain elevators will be able to get the exact same grade for the same canola seed samples every single time.
To accomplish our goal, we had to do 4 main tasks.
- Investigate what has already been done.
- Determine evaluation and deployment methods.
- Build the hardware and standardize the conditions.
- Create osftware that can detect a seed and its colour.
1. Investigate Previous Designs
To understand what solutions had already been tried, we talked to the Canadian Grain Commission (CGC) and their chief scientists/graders. We learned that there had been previous attempts to create a tool to measure chlorophyll accurately and quickly.
For example, they developed a tool that necessitated grinding seeds in a sample before running them through an NIR spectrometer. By homogenizing the seeds before measuring the amount of green in the mixture with the spectrometer, this method achieved a high degree of precision. However, the need for grinding the seeds resulted in their process becoming much slower overall, causing it to take several minutes to run. Additionally, NIR spectrometers cost around $22 000 CAD, a steep price to pay.
They also explored camera-based options of evaluating green seed, where computer vision was utilized to grade photos of seed samples. However, they ran into problems with different lighting standards for their scanners decreasing the reliability of their tool. Also, different imperfections with seed samples, such as hulls being on top of seed smears, occurrence of disease, and other factors that require discarding seeds from evaluations, were raised as issues.
Through our talks with the fine folks at the Canadian Grain Commission, we learned that to have a successful product, it must robustly handle the issues of different lighting conditions and seed aberrations. Additionally, it was revealed that the speed of the operation takes precedence over the cost of the tool, as processing times must remain short to facilitate the large inflow of seeds grain elevators and canola refineries require.
2. Designing the System
Our project began with developing a tool consisting of simple cameras within a standardized lightbox, wherein seed samples are placed to be graded. Therefore, there was a level of software and a level of firmware to consider in this project. Our primary concern was creating the software to analyze the pictures taken by the camera, count the number of open seeds, count the number of distinctly green seeds, and then output the sample’s distinctly green percentage on a computer screen. Secondly, there was also the firmware of the camera to consider. This very low-level firmware can have drastic impacts on picture quality and calculations. Camera firmware standardizes the brightness of the whole picture, while also applying white balancing to deal with the subtle tints of various light sources. An additional feature of the tool to be pursued was the ability to allow for identifying suspect seeds for more careful examination, expediting the grading process while also providing greater detail in the sample analysis. Additionally, this would enable increased grading reliability through larger seed sample sizes.
We will first try to implement our method as an adjacent seed sample grading service for farmers, wherein farmers will mail in seed samples for evaluation, like with the Canadian Grain Commission’s (CGC) Harvest Sample Program, which “offers unofficial grade and quality information that could help producers make delivery decisions.” (CGC, 2018). This methodology will allow farmers to acclimate to the service, expanding our user base while also assisting in iterating our project for improved usability, correctness, and robustness. By comparing the gradings of our system with the gradings from the CGC for the same canola seed samples from farmers, the validity of our system can be evaluated and improved over time.
Further iterations of the system may use a machine vision camera, which can be customized with low-level image processing functions to increase its adaptability in any lighting environment it is in. This approach is quite appropriate due to DGR classification being a qualitative problem. Additionally, other canola seed grading factors such as being heated, diseased, or damaged could be tackled by future versions of the standardized grading tool.
The end goal for this project is to gain approval from the CGC for the tool’s use as an automated grading system in the canola industry. In future expansions of this project, our team wishes to develop and receive approval for additional standardized seed grading tools for wheat, palm seeds, and soybeans, as all of these seeds suffer from visually discernible damage.
If these goals are achieved, the monetization of the tool will likely be as a rental service or subscription service. The reasoning behind this format is that for the target demographic of farmers and grain elevators, their need for the tool exists only in a short time window in the fall, as canola crops are harvested in that time. With the rental plan, the boxes would be delivered to farmers and grain elevators for use in the harvest season, before being returned. This would allow us to repair, modify, and update the boxes with relative ease. The subscription plan would work similarly, but with the end users deciding when to send their boxes in for maintenance and updates.
3. Developing the Hardware
The general concept of our hardware is a lightbox that will standardize the conditions used to take pictures, such as lighting, white balance, camera angle, and camera distance. Additionally, the lightbox will come with a palette to crush seeds on, standardizing the location of seeds in photos. The name given to this lightbox portion of the project is Mean Green Machine.
There will be a camera to take pictures of the seeds, linked with a computer for photo storage and analysis. The computer will also run the GUI for allowing users to interface with the tool and examine the grading results.
To build the hardware, we approached it from a quick prototyping perspective, using simple materials such as 3D prints, LEDs, and wood, which are easy to acquire and construct with. While initially Raspberry PIs were explored as platforms for firmware, the computational hardware utilized was decided to be that of a standard computer, due to their ability to run python code and interface with a variety of USB cameras. Later iterations will not be using these simple materials, as they will need to be built sturdier and more easily repairable to survive heavy user wear and tear.
Figure 3. Lightbox without an integrated camera
In the initial prototype, the light box was designed as a lid for the container housing the crushed seeds. An LED strip lined the inside walls which were covered in tin foil to increase light reflection. Wax paper on acrylic was used to dampen the LEDs to avoid harsh lighting on the seeds and reflection off of the tape. In this prototype, the camera was fixed inside of the light box using a 3D printed stand and a hole was cut in the acrylic to fit the camera. The Raspberry Pi sat outside of the housing where it was accessible for input/output devices.
Figure 4. 3D printed camera stand fixed to the lightbox and PI camera.
Figure 5. Acrylic cover with a hole cut for a PI camera
Early testing revealed a number of issues with this prototype. Firstly, the Pi camera had a very low resolution, as well as a focal distance that was much longer than the dimensions of the lightbox, resulting in poor image quality and difficult colour detection. Furthermore, the camera stopped working partway through testing. In addition, the Raspberry Pi was resistant to the installation of the computer vision packages necessary to run the green seed detection algorithm. To alleviate these issues, the team planned to interface the camera and algorithm with a laptop, and purchase a USB camera with higher resolution and colour sensitivity. Secondly, the light box had to be removed every time the user wanted to place the colour chip or seed strip inside the box. The light box had pieces of hardware inside that would be shifted each time it was moved, and removing such a large lid was a cumbersome process. To avoid these issues, the team began testing different methods of inserting the colour chip and seed strips.
Figure 6. Lightbox interior with LED arrays
In our second prototype we wanted to erase the errors made in the first prototype and make it better overall. After deliberating on wanted features such as: good looking design, white painted inner walls, sturdiness, light-blocking, easy-to-assemble and dissassemble, gull-wing door, easy to modify the camera (modular), and a clear identity.
While we could do a lot of these, building it out of stronger materials would likely require enlisting a machine shop to help build it. In comes the Schulich School of Engineering and Sean Mason.
The Schulich School of Engineering allows Schulich teams to order machining at a great discount. But to take advantage of this, we must be able to create the drawings required. Sean Mason's help as a Machining mentor was invaluable as he both had access to CAD tools as well as expertise in choosing the materials for each component. He chose to make the project out of quarter inch lexan, as it is an extremely durable, easy to work plastic.
Close work with Sean Mason helped us design Mean Green Machine in such a way that is easy to machine and assemble.
Figure 7. Design of prototype 2
The design had to be simple enough for us to put into Solidworks and be easy to assemble. With this in mind, having static interlocking pieces would be a lot easier to design than dynamic parts. In the following images we can see the interlocking groove system applied to make it easy to assemble.
To minimize time in the machine shop, the majority of the design can be cut with a waterjet table. WHilst lexan loses its perfect transparency in the process, we were painting it anyways so it didnt matter.
Once everything was cut in the machine shop, we still had to glue it, and assemble it.
The polycarbonate glue used is very dangerous and a carcinogen. Used it with gloves, and proper ventilation.
Parts had to set with the glue over several days.
Significant filing was required as certain portions of the design had to be light-tight; precutting them may leave too much tolerance.
First assembly with glued components.
Near final iteration of mean green machine, painted and functional, missing 2nd coat of paint and log.
With the Logo
4. Developing the Software
Seed Grading Algorithm
The main purpose of the device is to give an objective, consistent, and accurate canola seed sample grading, merely from images of the seed sample and the calibration colour chip. In order to accomplish this, various computer vision and image processing techniques were employed to create a comprehensive seed grading algorithm, which was then implemented in a GUI. The overall software part of this project was given the moniker of GreatGrader.
The inputs for the software include a picture of the crushed seed sample being analyzed/graded. It is assumed by the software that the picture of the seed sample can be subdivided into a grid of individual crushed seeds.
Figure 8. An example seed sample subdivided into a 12x8 grid
Each cell of the picture is analyzed one by one as its own individual seed smear, independent of the others.
Figure 9. The seed highlighted in red in the seed sample image
In order to separate the image of the seed in the foreground from the background, the image processing algorithm known as marker-controlled image segmentation with watershed is employed.
Image Segmentation with Watershed
Figure 10. Watershed visualization
Image segmentation algorithms are utilized to identify groups of pixels in an image which correspond to separate entities. watershed is an image segmentation algorithm that first treats 2D images like 3D terrain, where lighter pixels are lower in altitude, as contrast in objects often manifests in differences in brightness (Beucher & Meyer, 1993). Subsequently, by filling the 3D terrain image with markers in the same way water fills a container, i.e. with the brightest parts of the image filled first, bodies of “water” can form. The borders of segments of the image form where these bodies of “water” meet. Additionally, different bodies of “water” have different identities, allowing for their pixel groups to be isolated from one another.
Marker-controlled Image Segmentation with Watershed
Marker-based watershed segmentation is a modification of the standard watershed algorithm. Instead of the bodies of “water” first forming from the places of the highest brightness, different foreground markers placed on the original image can be considered as sources of “rainfall,” forming the bodies of “water” on the image. Additionally, special background markers can be used to identify regions of the image where no entities are imaged, forming the background of the image.
In marker-controlled image segmentation with watershed, a number of pixels in the image are evaluated to determine if they should be marked as foreground or background. A pixel is identified as background if it is not saturated enough, although pixels that are green in hue can avoid being identified as background if they are dark enough in brightness. A pixel is identified as foreground if it is not identified as background. Unfortunately, sometimes false positives develop as the insides of seeds are sometimes falsely identified as background.
Figure 11. The seed being marked with white foreground markers and dark background markers for watershed segmentation.
With the image now marked, watershed can be employed to separate the foreground from the background, creating a foreground mask.
Figure 12. The initial watershed mask. White = foreground, black = background.
Unfortunately, the effects of the previous false positives in the marking stage have resulted in holes appearing in the seed mask.
Modified Flood Fill
In order to resolve this, a modified flood fill algorithm is employed. Flood fill algorithms are employed to fill in pixel areas inside of defined boundaries. The Flood fill algorithm utilized in this stage is designed to only fill in holes inside of the seed, while ignoring the voids that exist between the borders of the image and the contained seed.
Figure 14. Modified flood fill
Figure 15: The final Watershed mask. White = foreground, black = background
After the flood fill stage, the seed has successfully been separated from its background. However, seed images contain lots of irrelevant information, mostly consisting of seed hulls. The desired information lies in identifying the entire area of the smear coming from inside the crushed canola seeds.
Seed Smear Segmentation
Seed smears range in hue from yellow to green, which covers a range of approximately. In order to accomplish this, the hue of each pixel is examined to see if it is in the yellow to green region of the HSV color space. The benefit of using the HSV color space in this situation is that the only examined variable here is the hue of the pixels, irrespective of the lighting conditions or intensity of the pixels, which are quantified in the saturation and brightness variables of the HSV colour space.
Figure 16. The conical HSV colour space
Knowing where the smear lies in the seed image is important for determining what proportion of the seed is actually distinctly green, while also narrowing down the area to search for distinctly green seeds.
Figure 17. The inside smear of the crushed seed is removed from the full seed image
Distinctly Green Seed Computation
When determining what pixels count as distinctly green, it is important to factor in lighting conditions, as those can affect the colour intensity of pixels. After some discussion on different colour spaces with Dr. Alim, we decided that this stage of distinctly green classification should utilize the Lab color space. The Lab colour space utilizes 2 variables, a and b, to represent colour, while the L variable represents the lightness of the image. The green volume of the Lab colour space can be approximately represented as such:
Figure 18. The green volume of the Lab colour space
As seed smears can consist of numerous different colours of green, it is most important to calibrate the classification system to the lighting conditions of the photos. Therefore, the aforementioned calibration chip is examined to determine its lightness (L) value. The system assumes that the lightness of the green in the calibration chip negatively corresponds with the intensity and concentration of the chlorophyll content within seeds.
For example, the seed sample being examined in the above figures had a colour chip with an L value of 50.
Therefore, the pixels that lie roughly within this subsection of the identified green volume of the Lab colour space are marked as distinctly green.
Figure 19. The DGR pixels are selected from the smear of the inside seed
However, even the most apparently distinctly green seeds do not achieve 100% distinctly green pixels. Therefore, a threshold percentage of around 50% is used as a standardized, albeit somewhat arbitrary, method of classifying green seeds. This evaluation results in fairly fast and accurate seed sample gradings.
Figure 20. A graded seed sample. Seeds highlighted in yellow are fine, while seeds highlighted in red have been classified as distinctly green
However, because the system is not perfect, it is important to identify seeds that should be examined more closely by human eyes. Therefore, a confidence heatmap is constructed to identify seeds whose classifications are of low confidence.
Figure 21. The confidence of GreatGrader’s classifications. Black = low confidence. Green = high confidence
Seeds that possess a distinctly green pixel percentage near the threshold value are made with low confidence, while seeds with extremely low or extremely high (barring some extremely dark seeds that consist mostly of hull) distinctly green pixel percentages can be classified with a high degree of confidence.
In order to make it easier for end users to use the software, a GUI was developed. The GUI allows the user to take calibration images using a USB camera or previously saved images. Additionally, the user can crop and subdivide those images. Following these settings being set, the seed sample image can be graded, with saveable results. Also, the supplementary analysis images can be toggled between, with individual seed information being selectable by mouse.
Figure 22. The first GUI for the seed classification tool
When additional testing revealed that the USB camera resolution was insufficient (around 5 MP), the use of higher resolution phone cameras was explored. Through the use of IP Webcam, the program gained the ability to use a broadcasting phone as a camera. However, differences in phone cameras necessitated some way to calibrate the software to each phone. Subsequently, an advanced settings options menu was added, giving users the ability to adjust and save the variables in the threshold and colour space distance calculations used in the software. Finally, after meeting with the CGC, their feedback on the GUI lead to wider image panels, in addition to an expanded view of the seed sample.
Figure 23. The second GUI for the seed classification tool
The GUI and accompanying software are available for download from our GitHub:
In order to use MGM, after assembling the lightbox and powering the LEDs, one must decide on how to acquire the required seed grading images. The potential methods include:
- Connecting a USB camera to a computer before running the GreatGrader software.
- Connecting to and accessing a video stream from an android phone using the app IP Webcam and the GreatGrader software.
- Using a standalone camera and transferring the images (preferably without data loss) to the file system of the computer running the GreatGrader software.
Figure 24. The test gradings our team conducted utilized the above method
Once the GreatGrader software is set up, the calibration colour chip photo must be taken or loaded. If taking a picture of the calibration chip, it must be placed roughly where the seed samples are placed, to reflect the lighting and camera conditions of the seed sample. Appropriate cropping should be applied so that only the distinctly green parts of the calibration image are used.
Figure 25. The placement of the colour chip in MGM
Seed Sample Preparation
In order to create a seed sample in the canola industry, canola seeds are arranged on a seed stick, covered with masking tape, and then crushed with a roller.
Figure 26. The basic seed sampling process
In our system, the taped seed samples should be arranged side by side as parallel as possible. In the future, the use of one large seed stack would eliminate the need for arranging different pieces of masking tape, thereby ensuring an even distribution of seeds, which are easily divided into an array.
The masking tape seed sample should then be placed into the lightbox, with special care to ensure that the seeds are parallel to the edges of the camera picture. After taking an appropriately cropped and divided seed sample picture, the GreatGrader software can now grade the sample, and display the resulting grade.
With a calibration photo and a seed sample picture now taken, the GreatGrader software can now analyze it after following the following steps in GreatGrader:
- Crop the calibration photo so that only the
DGRcolour is being considered.
- Crop the seed sample photo so that as much background as possible is removed to lower processing time.
- Set the number of rows and columns of the seed array to overlay onto the seed sample photo.
- If necessary, edit the advanced settings of GreatGrader to calibrate it to your camera.
- Grade the seed sample. Processing times scale with image size.
In order to evaluate the accuracy of the automatic seed grading tool, we went to a CGC office to get some graded canola seed samples. The grades given by the CGC were compared against the grades of the GreatGrader software. Additionally, we also evaluated the performance of our system on samples provided from Richardson Oilseed and on canola seeds provided courtesy of our TA Jacob Grainger. With minor calibrations, especially with regards to minimum seed smear area, the team’s GreatGrader software was able to accurately grade canola seed samples.
Figure 27. CGC Seed Sample 1. Graded by the CGC as 6.4% DGR
Figure 28. CGC Seed Sample 1. Graded by GreatGrader as 7.444% DGR
Figure 29. CGC Seed Sample 2. Graded by the CGC as 7.0% DGR
Figure 30. CGC Seed Sample 2. Graded by GreatGrader as 7.739% DGR
Figure 31. Jacob Grainger Seed Sample. Graded by the Team as between 3.6% and 4.7% DGR
Figure 32. Jacob Grainger Sample. Graded by GreatGrader as 5.734%.
Figure 33. Richardson Seller 8 Sample. Graded by the CGC as 0.308% DGR
Figure 34. Richardson Seller 8 Sample. Graded by GreatGrader as 0.305% DGR
In order to more comprehensively evaluate our standardized seed grading, we not only compared the overall distinctly green % of our system to real values, but we also compared the distinctly green % of different sections of our samples, as real seed gradings are composites of gradings of smaller samples. This gave our team the additional benefit of being able to compute and compare the standard deviation of our standardized gradings to human gradings. The results of our tests are displayed below.
|Sample:||Metric:||Human Grading||Standardized Seed Grading||Absolute Error||Time to Grade:|
|CGC #1||Overall Grade||Grade No. 3||Grade No. 3||No Error||164.287 seconds|
|Section 1 DGR%||10.800%||13.412%||2.612%|
|Section 2 DGR%||3.000%||2.306%||0.694%|
|Section 3 DGR%||3.800%||4.632%||0.832%|
|Section 4 DGR%||8.000%||9.072%||1.072%|
|CGC #2||Overall Grade||Grade No. 3||Grade No. 3||No Error||189.261 seconds|
|Section 1 DGR%||6.400%||6.122%||0.278%|
|Section 2 DGR%||7.600%||9.036%||1.436%|
|Jacob Grainger Seeds (Using average of minimum and maximum grades)||Overall Grade||Grade No. 2||Grade No. 2||No Error||289.408 seconds|
|Section 1 DGR%||6.333%||7.333%||1.000%|
|Section 2 DGR%||5.250%||7.767%||2.517%|
|Section 3 DGR%||2.250.%||3.415%||1.165%|
|Section 4 DGR%||3.000%||4.545%||1.545%|
|Richardson Oilseed Seller 8||Overall Grade||Grade No. 1||Grade No. 1||No Error||267.353 seconds|
|Section 1 DGR%||0%||0%||0%|
|Section 2 DGR%||0.500%||0.518%||0.02%|
|Section 3 DGR%||0.500%||0.476%||0.024%|
|Section 4 DGR%||0%||0%||0%|
Table 1. Seed sample grade comparisons
A summary of the takeaways from our results is displayed below:
|Accurate Grade||Farmers||Grade determines farmers' ROI for canola.||Possibly achieved||Much more testing is required, especially for edge cases.|
|Accurate Sample DGR %||Oil Refineries||Seed grades are not specific enough for determining vital input concentrations.||Somewhat achieved||Much more refinement in areas such as seed partitioning is needed.|
|Grade Speed < 5-10 minutes||Oil Refineries||Seed grade times must be fast to process frequent seed shipments.||Tenuously achieved... for now||As camera resolution improves, the speed of the program will suffer. Improvements must be made to achieve desirable speeds.|
|Relatively Affordable||All seed graders||Economics.||Achieved||Multiple methods of image taking are compatible with the system, including the use of ubiquitous phones.|
|Robustly handles different camera and lighting conditions||All seed graders||Inconsistent lighting conditions can decrease grading accuracy and consistency.||Not achieved.||Automating calibration and exploring higher quality USB cameras for standardization.|
|Handles increased sample size||All seed graders.||Larger samples are more representative, and result in fairer grades.||Achieved (The system was able to grade samples of around 2000 seeds)||Creating a seed stick of 1000 or 2000 divots to standardize large seed sample arrangement.|
Table 2. Requirements Progress
With an overall absolute error rate of 0.843 percentage points in our tests, our standardized seed grading tool, comprised of the physical Mean Green Machine (MGM) for standardizing lighting conditions and the software tool GreatGrader, was developed to a fine level for a proof of concept. At our final meeting with the Canadian Grain Commission (CGC), they expressed informal support for the project and suggested multiple future directions to ensure the continued success of the project. The team received encouragement to continue refining the software, in order to improve the accuracy and speed of the program. Also, they advocated for the implementation of more lighting standardization, through equipment and design choices like grey background paint, light diffusers, and better-quality cameras. Additionally, they propounded the use of infrared cameras to gather additional information for informing seed grading. The CGC seed grading experts we talked to also recommended that we expand our project into tackling even more difficult seed grading issues, like with evaluating frost damage on wheat. You can read more about our interactions with the Canadian Grain Commission here. With exciting and challenging future directions on the horizon, our standardized seed grading project shows promise in addressing canola and potentially other staples of Canadian agriculture.
Beucher. S., & Meyer. F. (1993). The morphological approach to segmentation: The watershed transform. In E. R. Dougherty (Ed.), Mathematical Morphology in Image Processing (pp. 433-481). New York, NY: Marcel Dekker
Canadian Grain Commission. (2018, October 25). Canadian Grain Commission extends Harvest Sample Program deadline [News release]. Retrieved from https://www.canada.ca/en/grain-commission/news/2018/10/canadian-grain-commission-extends-harvest-sample-program-deadline.html
Canadian Grain Commission. (2019, August 1). Canola and rapeseed: Primary grade determinants tables. Retrieved from https://www.grainscanada.gc.ca/en/grain-quality/official-grain-grading-guide/10-canola-rapeseed/primary-grade-determinants-tables.html