Determine the location an object by image processing used for controlling autonomous vehicles

Phạm Đức Long Tạp chí KHOA HỌC & CÔNG NGHỆ 116 (02): 35 - 40 35 DETERMINE THE LOCATION AN OBJECT BY IMAGE PROCESSING USED FOR CONTROLLING AUTONOMOUS VEHICLES Pham Duc Long* College of Information and Communication Technology - TNU SUMMARY In this paper we propose a new method to locating of objects by a camera CNN Bi-i V2. These objects are on the ground and they have any position with camera. It is method to locate objects based on geometric relationships of real size of object with its image in camera. The hybrid algorithms are performed in Bi-i that includes parallel and serial tasks of processing. The values of distance and orient are sent to an embedded computer on the robot for controlling. Experiments are successfully implemented on the robot Pioneer 3DX. Keywords: Estimating location by image processing; camera pinhole INTRODUCTION * Today, the services of goods transportation (LOGISTICS) is growing due to the circulation of goods across the globe are on the rise. To response this development the topic of automation for warehouse, using automatic car for transport, ... is interest in researchers. Have appeared the AGV - Automated Guided Vehicle. The chain of freight car is researched. They can automatically transport, load and unload goods. These vehicles are usually automated using ultrasonic sensors or camera image and encoder to estimate location in the warehouse. When using a camera and image processing program researchers have used the method with one or more cameras [1, 2, 3, 4]. Estimating location of a object by using two camera is the results better but longer computation time. In the technique to successfully a same task if that plan is simpler, using less equipment than going for many more outstanding features. The mission that research done in this paper is to use a Bi-i V2 CNN camera fixed on a Pioneer 3DX robot captured the image and processing to compute for estimating location parameters (distance and angle orient) of a model of pallet and send these parameters to industrial PC for controlling the robot moves it to the pallet, lifting and moving pallet to the required position - "region target". In the Fig. * Tel: 0912 551589, Email: pdlong@ictu.edu.vn 1 the robot lies on the root of system co- ordinate that is the point O. The Bi-i camera fixed on the robot can observation an region in which the captured image is used for computed. The position of the object (pallet model) was placed in the observation space. This position can be changed from a host computer (e.g. from a notebook) via wireless network. Fig. 1 Robot approaching pallet and take it to area target The estimation of location of pallet is implemented in two cases: When pallet lies in an area, its real distance bounded is known. When pallet lies in the any space. ESTIMATING LOCATION OF OBJECT BY IMAGE PROCESSING From a stationary camera to estimate location for any object in viewable space of the camera We use a camera CNN Bi-i V2. It is placed on a robot. Object's image is captured by camera in the Fig.2. When robot approaching to object on direction P - P it uses the point K as target (Fig. 3). Program of image processing have to computing parameters and sends it to Industrial PC: Phạm Đức Long Tạp chí KHOA HỌC & CÔNG NGHỆ 116 (02): 35 - 40 36 The ordinates of point K (XK and YK), and The angle orient (angle of line P-P and axis X) Fig. 2 Object and its image The point M is a black pixel its ordinate Y is biggest in the image. In the Fig. 3: TT is line through optical axis of lens. NN is line perpendicular with TT. The camera leans to axis X angle pan. If we compute to get real distance O1M from pinhole point O1 to the point M (next step get the XK and YK), and we already have real size of the object then we can to compute to get YK and XK. The point M is not always on the optical axis of lens that is often lop-sided (to right or left of the optical axis - on the Fig. 3 is being skewed to the right at an angle R). The perform as follows: Determining location of the point M: O1M=O1H*tang( ). (1) = { R, L} If the point M is on the right of symmetrical axis of image (that is XM > 64 pixels): In real plane then XM = O1M*cos(pan - R). (2a) and YM = O1M*sin(pan - R). If the point M is on the left of symmetrical axis of image (that is XM < 64 pixels): In real plane then: XM = O1M*cos(pan+ L). (2b) and YM = O1M*sin(pan+ L). If the point M is on the symmetrical axis of image (that is XM = 64 pixels) : In real plane then XM = O1M*cos(pan). (2c) and YM = O1M*sin(pan). After computed the angle orient XK = XM+(half size of width of the box)*cos(orient); (3) YK = YM+(half size of width of the box)*sin(orient); Because the root of system co-ordinate O does not coincide with center of lens O1 therefore XK=XK+70; (4) YK=YK+200; Fig. 3 Model of estimating location from robot to pallet Fig. 4 Real side of object is ON’ and its image (distance ON in the line, which is perpendicular to axis optical through points CP’, is angel view of lens) Fig. 5 Relation of the real side M'N' and its image MN in the case plane of image is un-perpendicular with axis optical Phạm Đức Long Tạp chí KHOA HỌC & CÔNG NGHỆ 116 (02): 35 - 40 37 In the model of Fig.5 we knew in real size of: - height from ground to camera, declination of camera tilt, the bounds viewed of camera (CO, CQ' are knew). Like this the angle of OCQ' is knew - Ordinate in direction Y of point Q (YQ). - Dimension ON keeps takes 128 pixels (because resolution of camera equal 128x128). - Dimension OM keeps (128-YQ) pixel. In image we have the point M far the point O a distance i pixel. The distance in image is OM and corresponding real distance is O1M'. Angle of OCM (with top is point C) has degree: i*(angle OCQ'/number of pixel in the distance OQ) (5) Real distance from point O1 to point M is O1M'=height*tang(((tilt - 2 ) + i)*(góc OCQ'/number of pixel in the distance OQ)) (6) Corresponding to Fig.1 we computing the distance O1M. Using formula (4) we get XK and YK Fig. 6 The variant of real dimension in horizontal direction. We need computation HM in Fig. 6. We have: = atan(((Maxsize-Minsize)/2)/Length) From Minsize to Maxsize in the view region size in horizotal direction is increasing a number Maxzide-Minsize. After each horizontal pixel size will increase: (Maxsize-Minsize)/ Length For this reason, coefficient to correct for get real dimension in horizontal direction as: heso_thuc = (Minsize+(128- MAXy)*((Maxsize-Minsize)/ Length))/128 Fig. 7 Model for computing angel orient. Let examining case of when object is (and also other cases when object is not) on the optical axis of camera: L and R is denoted the real sloping to the left and right angle side of object, respectively. In the image these angles are 'L and 'R. If optical axis of camera is perpendicular to object's plane then L = 'L and R = 'R. In fact, optical axis of camera often is not perpendicular to object's plane. Therefore, when by image processing we compute to get angles 'L and 'R , but their value is not equal real values of angles. For get real value of the L and R we have multiply 'L and 'R with a correct coefficient k. The value of k is dependent on the parameters height and tilt (in Fig. 5). Formula for computing it as follows: Because L + R = 90 0 then in the image if ( ’L + ’R) < 90 0 then k = 90 0 /( ’L + ’R). This reason is logical because relations of L on 'L, and R on 'R are linear (however we are not expressing this relation). orient = pan- L or (7) orient = pan+ R - 90 0 Fig. 8 The variant of angles L and R when nearest point of the object is not on the vertical symmetrical axis of image Phạm Đức Long Tạp chí KHOA HỌC & CÔNG NGHỆ 116 (02): 35 - 40 38 Estimating location of an object in any space (when camera is moving) In this case the real view region of camera is unknown. We knew: + Real size of the object (by millimeter), and + Parameters of array optical sensor in our camera (width, height by millimeter and degree resolution of camera), and + Len's length of focus. Using model camera pinhole we can calculate distance from camera to object in two cases: When plane of object is parallel to plane that containing sensors of camera, and optical axes of camera is identical with the normal of object's plane in the geometrical center. This is standard model of camera pinhole. This case often is in some cases using image processing receiving control information in the industrial robot arm. The case of when object's plane is not parallel to plane that containing sensors of camera, and optical axes of camera is sloping to object's plane an any angle. In practice, this case is when a camera is placed on an autonomous robot. The resolution of the cases i) and ii) is given in the paper [5]. EXPERIMENTS The real parameters of a camera (for instance: length of focus, distort image in the corner of image,...) often are not accurate with parameters of the manufacturer; To adjusting these parameters one realize calibrating for camera by software Calibration of MATLAB. The algorithm Algorithm of the case 2.1 is shown in Fig. 9 Fig. 9 Algorithm in the case of 2.1 Results Experiments are realized with two models of the object: The box size of 160x160x130 and model of Korea pallet in the ratio of 1:4 Fig. 10 The model of objects in the experiment Fig. 11 Integrating an Autonomous Vehicle: : Router TP LINK W8951ND; : Camera Bi-i : IPC : Robot Pioneer 3DX Fig. 12 The robot approaching an object     Phạm Đức Long Tạp chí KHOA HỌC & CÔNG NGHỆ 116 (02): 35 - 40 39 The lens type of 1:1.4 25 mm is used in the experiments. The speed captured and processing image is 1000fps. In the experiments program image processing exactly computed the location of model pallet and then sending these information via wireless communication to an IPC that is placed on the robot to driving accurately the robot to model of pallet, lifting and carrying it to the required position. It has confirmed the correctness of the proposed method. CONCLUSION Using a CNN camera type Bi-i V2 placed on a mobile robot with wheels can captured images of an object in the search space to locating position of the object and provides these information for the IPC in the robot to driving it move to object to perform the next task. When using the camera CNN a lot of image processing tasks have been parallel implemented that causes to reducing time for computing in every task and also reducing time for computing overall mission. Now, the trend of computer vision is using stereo- camera for locating the object. The continue study of image processing algorithms to locating the robot will be used 2 CNN camera. With this approach we can be obtained more significant results in the near future. REFERENCES 1. Giovanni Garibotto, Stefano Masciangelo, Marco Ilic, Paolo Bassino, ROBOLIFT: a Vision Guided Autonomous Fork-Lift for Pallet Handling, IEEE, pp. 656-663, 1996. 2. Michael Seelinger, John-David Yoder, Automatic Pallet Engagment by a Vision Guided Forklift, Internatinal Conference on Robotics and Automation Baccelona, Spain, pp 4068-4073, April 2005. 3. Daniel Lecking, Oliver Wulf, Bernardo Wagner, Variable Pallet Pick-Up for Automatic Guided Vehicles in Industrial Environments, IEEE, pp. 1169-1174, 2006. 4. Sungmin Byun and Minhwan Kim, Real-Time Positioning and Orienting of Pallets Based on Monocular Vision, 20th IEEE International Conference on Tools with Artificial Intelligence, IEEE, pg 505-508, 2008. 5. Phạm Duc Long, Pham Thuong Cat, Using Image processing for computation distance and orient from pallet to autonomous forklif, Proceedings of the VCCA 2011, Hanoi 12-2011. TÓM TẮT XÁC ĐỊNH VỊ TRÍ ĐỐI TƢỢNG BẰNG XỬ LÝ ẢNH ỨNG DỤNG TRONG ĐIỀU KHIỂN XE TỰ HÀNH Phạm Đức Long* Trường Đại học Công nghệ Thông tin và Truyền thông - ĐH Thái Nguyên Trong bài viết này, chúng tôi đƣa ra một phƣơng pháp mới xác định vị trí của các đối tƣợng bằng xử lý ảnh thu đƣợc từ một camera CNN kiểu Bi-i V2. Các đối tƣợng nằm trên mặt phẳng và có vị trí bất kỳ đến camera. Đây là phƣơng pháp xác định vị trí đối tƣợng dựa trên cơ sở các quan hệ hình học trong thực tế và ảnh thu đƣợc. Thuật toán lai xử lý trên thiết bị Bi-i bao gồm các bƣớc xử lý song song dùng CNN và các bƣớc xử lý nối tiếp trên DSP. Thông tin vị trí đƣợc gửi tới máy tính trên xe tự hành để điều khiển xe tiếp cận và nâng đối tƣợng di chuyển tới một vị trí yêu cầu mới. Thực nghiệm đã đƣợc thực hiện thành công trên robot thí nghiệm kiểu Pioneer 3DX. Từ khóa: Xác định vị trí đối tượng bằng xử lý ảnh; Camera pinhole Ngày nhận bài:25/01/2014; Ngày phản biện:10/02/2014; Ngày duyệt đăng: 26/02/2014 Phản biện khoa học:TS. Nguyễn Văn Huân – Trường ĐH Công nghệ Thông tin & Truyền thông - ĐHTN * Tel: 0912 551589, Email: pdlong@ictu.edu.vn