6.7   Voice Recognition for Concurrent Field Scouting and Machine Operation

Investigators

John P. Fulton, Biosystems and Agricultural Engineering, jfulton@bae.uky.edu
Scott A. Shearer, Biosystems and Agricultural Engineering, shearer@bae.uky.edu
Thomas G. Mueller, Agronomy, mueller@pop.uky.edu
Sam McNeil, Biosystems and Agricultural Engineering, smcneil@ca.uky.edu

Cooperators

Mike Ellis, Worth and Dee Ellis Farms, Shelby County, Kentucky
Tim Karn, Karn Farms, Ohio County, Kentucky
Milton Cook, Cook Bros Farms, Caldwell County, Kentucky

Introduction

            Precision agriculture provides new technology enabling the collection of spatial data for management of agricultural cropland.  The advent of GPS has provided farmers with a system to geo-reference data and a means to automatically control field operations.  Farmers are now able to collect more information on fields than ever before.  With managers or producers running farming operations from an office, these individuals seldom observe crop and field conditions that machinery operators observe during spraying and harvest operations.  Consequently, this valuable source of information remains untapped and could be of real profit for making current and future management decisions.

The most widely adopted precision agricultural practices so far have been grid soil sampling and yield monitoring.  Both give insight to the variability that exist within and across fields, but typically do not explain yield variations exhibited by many yield maps.  Insect and weed infestation, nutrient deficiency, and water drainage problems are a few factors influencing crop performance and need to be captured.  Automated sensors for identifying weeds, soil fertility and other field and crop conditions are under development.  However, machinery operators might be the most economical and accurate source for scouting and assessing crop condition.  While transcribing notes provides a viable option, this takes time, hard to perform, and most hired help do not want to take the time to write notes.  Yield monitors provide ‘flagging’ capabilities to mark particular areas and points of interest.  However, flagging requires the operator to redirect attention from harvest and spraying operations to press a key on the yield monitor.  During harvesting, the operator has a lot of other activities to oversee rather than take the time to record data through a keypad on a yield monitor.  Operators must continually view the header, which can range from 20 to 30 feet in width and simultaneously steer the combine obstacles such as rocks and sink holes.   The combine also contains other displays that must be occasionally monitored for proper operation such as threshing unit speed.  With all the attention the operator must spend on maintaining forward progress, entering scouting data on a yield monitor or through field notes diverts attention from machine operation and increases operator fatigue.

Spraying operations provide the same opportunity to obtain scouting information as well as having similar drawbacks.  However, the machine is operated at much higher speed reducing the time required to make proper path adjustments when obstacles are present.  The operator must also monitor the spray boom to ensure all nozzles are operating while also maintaining a parallel path to the adjacent pass to minimize spray under- and over-lap.  The addition of a monitor needing keyboard entry for scouting to flag and mark problem areas only generates more activities for the operator to oversee and divert the attention away from driving the sprayer.  Also, making field notes creates a problem.  Therefore, voice recognition in combination with GPS is an attractive alternative for collecting this spatial data on fields.  Voice recognition enables operators to maintain a focus on control of machine functions, and at the same time collect geographically referenced data.  The operator can record weed patches, nutrient deficiencies, erosion problems, insect infestations, plant stand, etc. by simply speaking into a microphone and allowing the software to tag the voice information with a GPS location.   Field scouting can be performed simultaneously with field operations to provide additional input to spatial databases.  Additionally, the data provides farm managers with field data to assist in management decisions.

Objectives

1.       To evaluate voice recognition software as a tool for use with field machinery to collect spatial information on cropland.

2.       To develop an interface for organizing and integrating voice recognition with GPS tags into a spatial database.

3.       To field test the field scouting program with Kentucky grain producers.

Background

            Voice recognition provides a natural interface for inputting data into a computer for the user (Robinson, 1992).  Industry has been using voice recognition to improve efficiency.  The use of voice recognition in quality control and parts inspection allowed users the ability to maintain focus on the job and enter information more quickly by eliminating the keyboard (Sharp, 1995).  The Saturn Corporation has used voice recognition systems for several years to monitor paint defects.  Inspectors continually use their hands and eyes to concentrate on the paint while recording information simultaneously.  Inspection time is reduced while accuracy is increased (Hemphill, 1991).

The use of voice recognition on agriculture machinery will require noise reducing or canceling microphones and software packages capable of filtering background noise emitted by the machinery.  King (1993) showed that voice recognition could be used in a lumber mill to grade logs under noisy conditions.  The user was able to visually inspect the logs and grade them within four seconds while keeping a marking pen in one hand and a measuring stick in the other.  Noise levels were measured between 90 and 155 db, which is equivalent to the range of noise typical of field machinery.  Noise levels between 70 and 85 db can be experienced in tractors with cabs whereas operators on tractors without cabs typically encounter noise levels between 85 and 100 db (Dux et al., 1997).  Some New Holland model combines register cab noise levels around levels 76 db while some John Deere combine cabs experience noise levels between 80 to 82 db (Dux et al., 1997).   The success of this system in a noisy environment shows that voice recognition has potential in agricultural settings.  Providing users speak loudly enough, voice recognition systems can distinguish between machine noise and the voice of the operator (Sato et al., 1993).

Jones (1992) outlined speech recognition as a means to collect data for dairy herd management.  He compared various devices for communicating with computers.  Voice recognition was considered a viable data input technique if used with a limited vocabulary to improve word recognition.

Procedures

A speech recognition system will be developed for mounting on an all-terrain vehicle (ATV), tractor, sprayer, combine or other field machinery to collect spatial crop performance data during field operations.  Several speech recognition software packages along with multiple microphone headsets will be evaluated.  Voice recognition software packages have been on the market for a few years and have been well devised for use with laptops and personal computers.  However, machinery environment creates a different and noisy atmosphere for the use of voice-recognition and will required filtering techniques for effective use in farm machinery.  The combination of software and hardware packages will be tested under noisy conditions within the laboratory to determine the best combination for use on agricultural machinery.  The Biosystems and Agricultural Engineering Department maintains ATVs, tractors and other farm machinery for preliminary testing.

Each software package will be trained to recognize a user prior to mounting on test machines.  To minimize assimilation errors, a limited vocabulary will be used to improve recognition accuracy.  The vocabulary will consist of words, which describe field and crop conditions.  Results from testing on different machinery should yield the best combination of software-headset combinations for field-testing.

Once appropriate software-headset combinations are identified (above 95% accuracy of identifying input commands) the next step is to tag the voice recognition data with geographical locations.  GPS will be used to generate spatial tags.  A software program will be coded using Visual Basic (VB) for manipulation of voice and GPS data.  VB provides an easy to use programming language and a development environment for embedding MapObjects by ESRI.  MapObjects adds the necessary GIS capabilities into the VB code to collect and record GPS and voice information. The program will accumulate pertinent voice commands from the voice recognition software while at the same time recording GPS position fixes for these commands.  The number of voice commands will be limited to help increase voice detection accuracy.  Commands will include the minimal set of words required to describe a field or crop condition and will need to be spoken in a particular order for the software to properly identify the condition, area size and shape.  For example, a user might pass though an area containing foxtail and the voice commands might be as follows: WEED, FOXTAIL, CIRCLE, FIFTY FEET.  The program would identify the problem as weeds, immediately capture a GPS tag, label the problem as foxtail and specify a circular area with a 50 ft radius for the infestation.  After deciphering this information, MapObjects is able to build a database containing the GPS and other information, and then create ‘Shape Files’ for each field, which can be read and displayed using most agricultural mapping software.

            Testing the voice recognition software and gathering GPS data will be performed in the laboratory to debug code and identify any problems.  Once the software and system are debugged, complete systems will be provided with appropriate training for each cooperator.  The system will include a laptop computer with the required software, DGPS receiver and headset.  A minimum of five fields per cooperator will be scouted each year.  The system will be implemented with spraying and harvesting operations during the first year. Each cooperator will be expected to provide feedback regarding system effectiveness during field operations.  Additionally, they will be asked to provide yield maps along with scouting data for these fields.  Limiting the use of the system during spraying and harvesting operations will provide ample off time during the first year for system improvements.  Performance testing in conjunction with other field practices (i.e. soil sampling and planting) will be explored during the remaining two years.

            Accuracy of the system will be evaluated by scouting fields before and after using the system.  Preliminary scouting will be performed on foot.   Areas of interest will be located and mapped to determine exact location, size, and shape.  These ground-truth maps will then be overlaid with the maps created by voice recognition system to determine system accuracy.  In addition, voice commands will be recorded on tape to verify recognition of the user’s spoken commands.

Expected Benefits

Voice recognition combined with GPS will provide farmers a quick and easy method for capturing field and crop condition information during field operations.   It has the potential to reduce scouting performed during the growing season by combining it with existing field operations.  Voice recognition will allow equipment operators to focus on the primary task at hand, control and operation of field machinery while also recording valuable crop scouting information.  Safety will be improved over traditional approaches by eliminating hand notes and keyboard entry information found on most yield monitors or transcribing information on a note pad.  A voice recognition system also provides a means to record unlimited information regarding crop performance and field conditions and report this important information back to crop managers to determine if corrective actions are warranted to improve economic returns.  Machinery operators are more likely to collect and assimilate scouting information if it does not impede field operations.

Deliverables

Software developed for this project will become public domain so that is can be used by producers to implement voice recognition.  This software package will be made available to producers via the Internet in downloadable form through the Precision Agriculture web page.  The program will enable users to implement voice recognition with GPS location tags and create ‘Shape Files’ which can be imported directly into most GIS-driven management software.  Producers will need to only purchase a microphone headset and the appropriate voice recognition software.

During the second year of work, a presentation will be given at the Annual International ASAE (American Society of Agricultural Engineers) Meeting.  It will include a description of the system, problems incurred during development, and preliminary results.  Further, this meeting paper will be submitted for publication in an appropriate journal.  This same information will be made available on the University of Kentucky’s Precision Agriculture web page.   Upon completion of the project, a final report will be written to summarize detailed research findings.  Extension presentations will be provided at the request of agents, colleagues in the College of Agriculture and others within the agricultural community.