A COMPUTER VISION SYSTEM FOR TERRAIN RECOGNITION AND OBJECT DETECTION TASKS IN MINING AND CONSTRUCTION ENVIRONMENTS CONSTRUCTION ENVIRONMENTS
ABSTRACT
Recent studies towards dragline excavation efficiency have
focused on incrementally achieving automation of the entire excavation
cycle. Initial efforts resulted in the development of an automated
dragline swing system, which optimizes the swing phase time.
However, the system still requires human operation for collision
avoidance. For full dragline autonomy, a machine vision system is
needed for collision prevention and big rock handling during the
‘swinging’ and ‘digging’ phases of the excavation operation. Previous
attempts in this area focused on collision avoidance vision models
which estimated the location of the bucket in space in real-time.
However, these previous models use image segmentation methods
that are neither scalable nor multi-purpose. In this study, a scalable
and multi-purpose vision model has been developed for draglines
using Convolutional Neural Networks. This vision system averages
82.6% classification accuracy and 91% detection in collision
avoidance. It also achieves an 87.32% detection rate in bucket pose
estimation tasks. In addition, it averages 80.9% precision and 91.3%
recall performance across terrain recognition and oversized rock
detection tasks. With minimal modification, the proposed vision system
can be adjusted for other automated excavators.
Keywords: Surface mining, dragline bucket, deep learning, machine
learning, oversized rock, convolutional neural network, machine vision,
object detection, earthmoving equipment.
INTRODUCTION
Complete excavator automation is widely regarded as the next
phase in efforts toward improving earthmoving efficiency. Earthmoving
often involves complex, forceful interactions between an excavator and
the ground. The nature of these interactions depends largely on the
type, properties and physical characteristics of the earth material. This
process is further complicated by the random occurrence of tree roots,
boulders and other such obstructions. Therefore, the ability of a
controller to detect changes in the operating conditions, adjust the
digging strategy and respond in real-time is of utmost importance.
Early studies into autonomous excavation identified some key
performance criteria which included the following [1]:
• The autonomous excavator must be able to work in any type
of earth material.
• Its excavation accuracy must be within 50mm.
• It should be able to handle different surface and
underground obstacles autonomously.
• It should be able to operate at the speed of the average
operator in any condition.
• Its operation should be capable of safe integration with other
site systems.
Up to date, only partial successes have been reported for
autonomous excavator model developments. These models use rule-
based algorithms to define the digging trajectory when the excavator
encounters obstructions. The simplest of these systems use pre-set
force thresholds to pre-define the excavator response. Gocho [2]
presented an autonomous model for the wheel loader. The model was
able to achieve loading by driving the bucket into a muckpile until a
pre-defined hydraulic pressure threshold is reached. At that point, the
loader scoops the material and moves towards the dumping area. A
similar model has been proposed for the back-hoe excavator. In their
model, Bullock and Oppenheim [3] used strain gauges to monitor strain
measurements as the back-hoe travelled through a prescribed
trajectory until a preset threshold was exceeded. However, both
models stop abruptly and fail to complete digging when they encounter
big rock obstructions.
Shi et al. [4] and Huang and Bernold [5] later extended the
digging controls in the wheel loader and back-hoe models to
accommodate the presence of big rock obstructions. While both
models complete digging cycles successfully, they only achieve this by
altering the digging trajectory to avoid these obstructions after an
encounter (Figure 1). However, this approach is unacceptable in the
case of mining and construction excavations where the complete
removal of such obstructions is necessary.
Corke and a team of researchers [6-9] developed a semi-
autonomous dragline model to move 200,000 tons of material in over
12,000 cycles. The model included an automated hoisting, dumping
and swing assist system. However, bucket paths had to be manually
controlled to prevent collisions. Also, the model is unable to
automatically adjust to ground obstructions. Therefore, the digging
phase still had to be human-operated to deal with these obstructions.
Recent studies towards dragline excavation efficiency have
focused on incrementally achieving automation of the entire excavation
cycle. Initial efforts resulted in the development of an automated
dragline swing system, which optimizes the swing phase time.
However, the system still requires human operation for collision
avoidance. For full dragline autonomy, a machine vision system is
needed for collision prevention and big rock handling during the
‘swinging’ and ‘digging’ phases of the excavation operation. Previous
attempts in this area focused on collision avoidance vision models
which estimated the location of the bucket in space in real-time.
However, these previous models use image segmentation methods
that are neither scalable nor multi-purpose. In this study, a scalable
and multi-purpose vision model has been developed for draglines
using Convolutional Neural Networks. This vision system averages
82.6% classification accuracy and 91% detection in collision
avoidance. It also achieves an 87.32% detection rate in bucket pose
estimation tasks. In addition, it averages 80.9% precision and 91.3%
recall performance across terrain recognition and oversized rock
detection tasks. With minimal modification, the proposed vision system
can be adjusted for other automated excavators.
Keywords: Surface mining, dragline bucket, deep learning, machine
learning, oversized rock, convolutional neural network, machine vision,
object detection, earthmoving equipment.
INTRODUCTION
Complete excavator automation is widely regarded as the next
phase in efforts toward improving earthmoving efficiency. Earthmoving
often involves complex, forceful interactions between an excavator and
the ground. The nature of these interactions depends largely on the
type, properties and physical characteristics of the earth material. This
process is further complicated by the random occurrence of tree roots,
boulders and other such obstructions. Therefore, the ability of a
controller to detect changes in the operating conditions, adjust the
digging strategy and respond in real-time is of utmost importance.
Early studies into autonomous excavation identified some key
performance criteria which included the following [1]:
• The autonomous excavator must be able to work in any type
of earth material.
• Its excavation accuracy must be within 50mm.
• It should be able to handle different surface and
underground obstacles autonomously.
• It should be able to operate at the speed of the average
operator in any condition.
• Its operation should be capable of safe integration with other
site systems.
Up to date, only partial successes have been reported for
autonomous excavator model developments. These models use rule-
based algorithms to define the digging trajectory when the excavator
encounters obstructions. The simplest of these systems use pre-set
force thresholds to pre-define the excavator response. Gocho [2]
presented an autonomous model for the wheel loader. The model was
able to achieve loading by driving the bucket into a muckpile until a
pre-defined hydraulic pressure threshold is reached. At that point, the
loader scoops the material and moves towards the dumping area. A
similar model has been proposed for the back-hoe excavator. In their
model, Bullock and Oppenheim [3] used strain gauges to monitor strain
measurements as the back-hoe travelled through a prescribed
trajectory until a preset threshold was exceeded. However, both
models stop abruptly and fail to complete digging when they encounter
big rock obstructions.
Shi et al. [4] and Huang and Bernold [5] later extended the
digging controls in the wheel loader and back-hoe models to
accommodate the presence of big rock obstructions. While both
models complete digging cycles successfully, they only achieve this by
altering the digging trajectory to avoid these obstructions after an
encounter (Figure 1). However, this approach is unacceptable in the
case of mining and construction excavations where the complete
removal of such obstructions is necessary.
Corke and a team of researchers [6-9] developed a semi-
autonomous dragline model to move 200,000 tons of material in over
12,000 cycles. The model included an automated hoisting, dumping
and swing assist system. However, bucket paths had to be manually
controlled to prevent collisions. Also, the model is unable to
automatically adjust to ground obstructions. Therefore, the digging
phase still had to be human-operated to deal with these obstructions.
留言
張貼留言