The proposed virtual assistive devise aims to develop a wearable hardware embedded devise with high features, less weight and easily understandable instructions for blind or visually impaired people. Figure 1. shows the flow diagram of working of the proposed wearable light weight apron for visually impaired and blind people.
It includes various visual components required to assist the blind people such as obstacle detection, potholes detection, and object detection along with location tracking using geo location for care takers. The proposed model is a smart wearable device which uses a total of 5 volts 3 amperes supply with rechargeable power unit. It has one micro HDMI output port coming from the raspberry pi 4 to configure the settings of raspberry pi by connecting to a monitor and a single charging port that is connected to rechargeable unit. The model uses VL53L1X Time-of-Flight sensor to measure the time. It considers the time as the light beams emitting from the sensor to reflect and reach back the sensor again. This time value is then sent to the Arduino nano for processing and calculating the distance at which the light beam got reflected. A total of ten VL53L1X sensors are used in the proposed model attached to the different locations of the body and each sensor is provided with a separate Arduino nano to increase the response time of the vibrating motors that are connected to each sensor. The vibrating motors start vibrating as soon as the Arduino nano detects the distance of the reflecting point of the light beam less than 30 cm. These vibrations at different parts of the body can easily help the blind person to predict the size of the obstacle ahead without any manual effort.
This feature makes the proposed design unique and stand out of line because no other design was able to help the blind person to detect and predict the height and size of the obstacles around him simultaneously without manual effort.
The potholes detection is designed with a single VL53L1X sensor. It is used to detect any uneven surfaces on the ground like potholes, speed breakers etc. This sensor along with the switching circuit which is used to switch off and, on the sensors, and raspberry pi is controlled by Arduino mega. A GPU module is used to track the location of the blind person which is controlled by raspberry pi. The GPU module sends the data to the Firebase using Raspberry pi and it’s in built Wifi module and the caretakers can track the location of the blind people by importing the data from the firebase to a simple tracking application.
Object detection is designed with a Pi camera, raspberry pi, Google cloud API. Pi camera is used to capture the pictures intended by the blind person and that is sent to Raspberry pi where the pictures are sent to Google Cloud using Vision API services and the picture is processed in the cloud to detect objects in the image and the results are sent back by the cloud pub/sub service to the raspberry pi. The results are then informed to the blind person using voice output which is in-build in Raspberry pi 4. The design of the proposed model is shown in Fig. 2. It involves a security feature where the user of the product should enter a four-digit pin by clicking the four sequential buttons that are provided at the side of the main body.
These buttons are connected to Arduino mega and the functionalities of the Arduino mega such as switching on the sensors and Raspberry pi will only work if the right pin is entered by pressing the buttons. Once the right pin is entered, it makes a pin in the Arduino nano which is connected to that four buttons to go to HIGH state which is essential for the Arduino mega to perform its functionalities. This security feature is added mainly because to avoid unauthenticated users to operate the system.
3.1. Guidelines for Graphics Preparation and Submission
The design guidelines of graphic presentation are elaborated in 5 components such as obstacle detection, uneven surface detection, slope and downward steps detection, pothole detection, hallow object detection.
3.1.1 VL53L1X Time-of-Flight Sensors and obstacle detection
The selection of appropriate sensors depends on different factors such as cost of the sensor, the type of obstacles to be detected, precision of measurement and detection range. Time-of-Flight sensors are more precise and quicker in measuring the distances and the detection range is high when compared to the traditional ultrasonic sensors that have been used [2,3,4,5]. They also weigh extremely less when compared to ultrasonic sensors which is important because it adds to the comfort of the blind person. A total of ten such sensors are used in the proposed design located at different positions of the body. Each sensor weighs around 0.4 to 0.7 grams and will be operating at 15mA supply current. So, the reason for selecting the VL53L1X Time-of-Flight distance sensors is mainly because of size and accuracy. These sensors are extremely small when compared to other distance measuring sensors like ultrasonic and IR sensors. Also, their accuracy is extremely precise which is definitely necessary since the product cannot compromise on accuracy of sensors as that could ultimately put the user in danger.
Table 2
Comparison Among Different factors of Sensors.
Factors to be considered for sensor selection
|
Type of sensor
|
Ultrasonic
|
IR
|
Time of flight
|
Long Range Sensing
|
No
|
No
|
Yes
|
High reading Frequency
|
No
|
No
|
Yes
|
Cost
|
Low
|
Low
|
Moderate
|
3D imaging Compatible
|
No
|
No
|
Yes
|
Sensitive to external conditions
|
Yes
|
No
|
No
|
Accuracy
|
Low
|
Moderate
|
High
|
Table 2 shows the comparison among three different distance measuring sensors. These comparisons conclude that VL53L1X Time-of-Flight sensor is much better when compared to Ultrasonic or IR sensor for the proposed design. The other feature of VL53L1X is that additional components include voltage regulator and level shifter circuit. Voltage regulator will make the sensor work at 2.2V − 5.5V and the level shifter circuit is used to allow I2C communication. But I2C communication is not used in our current model because the need is insignificant. The range of these sensors is 4 meters, and the working of these sensors is based on the time difference between the emission of light pulses and its return to the sensor. The sensor also offers three distance modes: short, medium, and long. Long distance mode allows the user to measure the longest possible ranging distance of 4 m, but the maximum range will always be significantly affected by ambient light. Short distance mode is generally immune to ambient light, but the maximum ranging distance is mostly limited to 1.3 meters. The maximum sampling rate present in the short distance mode is 50 Hz while the maximum sampling rate that is present in the medium and long-distance modes is 30 Hz. The advantage of having these different modes can be used in the product to set the working of the sensors based on the size of the environment the user is currently interacting. Even though the VL53L1X is not sensitive to external conditions, the main aim of the proposed design is to bring accuracy, comfort as well as fast responsiveness in the system.
Figure 3 shows a situation where the blind can easily judge the height of different obstacles by wearing the apron. Once the blind comes closer to an obstacle with a particular height, the vibrating motors on the system will vibrate at specific positions where the VL53L1X is exposed to the obstacle.
3.1.2 Potholes and Uneven Surface Detection
The potholes detection is done with the help of sensor based method of framework, by considering convolution neural network as obstacle identification [22].
A VL53L1X sensor is placed at the abdomen level of the user. The sensor calculates the time difference between the light pulses hitting the ground surface and reflecting to the sensor. This difference is sent to Arduino mega. According to the instructions, mega will either activate or deactivate Pothole indication.
The algorithm 1 indicates that the sensor takes a couple of seconds to configure the initial reference value to a particular number through which the uneven surfaces are detected based on the difference between the reference value as mentioned in [10] and the subsequent values read by the sensor. The Figs. 4 & 5 shows the pothole, steps detection and measuring distance. This configuration is done based on the height of the user. The reference value will be constant until the user reset the system. This value is basically the distance measured from the user's abdomen level to the ground at a particular angle, making a hypotenuse from the user’s abdomen to a point on the ground. This reference value is considered as ‘d’.
Now, if any Pothole or slope is present ahead of the user, the value of this hypotenuse length measured by the sensor will either increase or go out of range. So if calculated_distance > d or calculated_distance = Out of Range, then the proposed system will conclude that there is uneven road ahead and alert the user on the same.
These sensors are also used to detect objects with hollow bottoms like Tables and Chairs. For such objects, this hypotenuse length will fall drastically within seconds if the user tries to walk against these objects. The processing unit can identify such sudden drop and alert the user on the same. Since the initial threshold value is configured for the sensor, once it comes across an object above the ground level with some height, the distance measured by these sensors will fall instantaneously. This drop can be detected by the microcontrollers and indicate the user regarding the event so that the user can take the measures to avoid the danger.
In Figure. 5, the sensor was encountered with a pothole within a time interval of 6 to 8 seconds thereby increasing the value of calculated distance from the user to the ground surface. In Fig. 6, the sensor was encountered with a dining table in front of the user. So, as soon as the user came close to the dining table, the distance measured kept dropping from 7th second to 10th second and later after walking away from the table, it restored back to the reference value. In Fig. 5, the user marched towards the downward steps and initially at 7th second the readings started to increase but at 9th second the readings of the sensor went out of range.
3.1.3 Pi camera and Vision API services
The Fig. 7 shows the work flow of object detection. This comprises of pi camera for input images and respective API components for further control of flow are identified. Pi camera captures pictures of the surroundings and outputs the image to raspberry pi. Raspberry pi will then send the image data to the cloud. In the cloud, the image is processed for object, features, landmark and logo detection using pre-trained models and the results are given back to the "text to speech conversion" module of the Raspberry pi. Then the results are recited for the blind to hear and judge the surroundings.
Once the image data is received by google cloud, it uses Vision API services to detect the objects, features etc. in the image. The Cloud Pub/Sub acts as a real-time messaging service which acts as an interface between the input image data and Vision API. It is an asynchronous messaging service that helps in decoupling the services that produce events from services that process the events. It simply allows us to create one to many or many to one or even one to one publisher subscriber channels. In Vision API services, we can use google pre trained prediction model trained with thousands of datasets and predict the objects, features and many other entities in the image. Auto ML services is used to train our custom prediction model using our own datasets. After the prediction by the model, the results are sent to the raspberry pi for letting the user know about the object captured with the help of pub/sub.
Due to the most flexible features that google provide in Vision API such as Object detection, Feature detection, Landmark detection, Logo Detection, Optical Character Recognition. Google vision API is known for its fast and accurate responses. Since blind people cannot afford delay in knowing the obstacles around them by using other GPU intensive deep learning techniques, Vision API will be a better choice.
3.1.4 GPS module and location tracking
Similar to the concept used in [8], the proposed design also uses location tracking system by using Ada fruit Ultimate GPS Module, which is connected with the Raspberry Pi. The key difference is that here, the GPS module is used not to guide the Blind, but to help the caretakers track the blind person’s movements. The following steps are included in the process of location tracking,
- The GPS module sends the live location to the Raspberry Pi.
- As soon as the location data is received, Raspberry Pi sends the data to the Google Firebase DB after proper authentication.
- A mobile application is connected to this database with proper API key and database URL.
- Once the location data from Raspberry Pi is sent to the Firebase, the same location data will be displayed on the mobile application, helping the caretakers to track the blind person’s current location.
Figure 8 shows the circuit diagram of GPS module with indicators of pins such as GPS Vin to 3.3V (red wire), GPS Ground to Ground (black wire), GPS RX to TX (green wire), GPS TX to RX (white wire).
3.1.5 Vibrating Motor
This is a Direct Current vibration motor which is utilized in the mobile/cell phones. It generally requires around 125 mA current and 3V to 5V voltage supply. By applying the Pulse Width Modulation (PWM) method the speed control of these types of motors can be programmed, although the default speed is used in the proposed system. The motor diameter and thickness are 0.5 cm and 2.5mm respectively.
3.1.6 Four Pin Security Lock
The inbuilt security system is provided with the help of the Arduino board. The circuit as shown in Fig. 9 is designed in such a way that the user is provided with four different buttons along with a reset button, each button is assigned with numerical from 1 to 4. One terminal of the button switch is connected to its respective digital pin in arduino and the other is grounded. When the switch pressed the respective digital pin in the arduino turns low. Based on the pattern in which the pins are getting to lower states, the digital pin 7 will get its value. If the switches are pressed in correct order the output of digital pin 7 is high else the output is low. Once pin 7 is in its higher state, that pin sends voltage signals to all the reading pins of the remaining microcontrollers which instructs them to follow the switching instructions from the user. Once the user presses the reset button, the system gets locked again. A simple LED is turned ON if the entered pin is correct.
3.1.7 Communication in the Blind’s Apron
A wireless communication module is developed with “frequency modulation” (FM) to interact the blind people for finding the system location from them. The generation of radio frequency is done by an RF transmitter in its circuits as depicted in Fig. 10. The carrier signal is modulated, and the information is added to this 'carrier signal'. This integrated carrier and information which forms a composite signal will be then fed to the antenna. The electric and magnetic fields are being changed from its own antenna after the signal from the atmosphere with same frequency will be received by an RF receiver. The receiver circuits will then strip the information part from the composite signal and amplify this to a useful level for getting the audio signals which then could be used by the blind person or even the caretakers to find the system.