Evaluation

We have used testing to guide us in some important decisions along the progress of each subsystem and as a way of evaluating if the systems work well enough.

On this page you can find the evaluation of each separate subsystem:

The footprint and parameters of Navigation
Success of identification, choosing a tracking algorithm, and evaluating threading for the Vision system
User experience for the Web App, the users' first point of contact
Finding the sweet spot between quick book retrieval and silent operation and choosing the effort threshold values for The Arm

As we did not have a chance to finish the robot, the systems have not been tested together adequately. Main areas of improvement for our system can also be found here.

Navigation

Click on the images to enlarge them

The old footprint

The new footprint

Test results using the updated footprint

Test results of xy_goal_tolerance

The navigation process is optimised by tuning of the Navigation Stack parameters. The parameters we have tested include footprint, xy_goal_tolerance, max_vel_x and min_vel_x.

Footprint

Footprint is the contour of the mobile base of the robot. In ROS, it is represented by a two-dimensional array of the form [[x0, y0], [x1, y1], [x2, y2], ...] where the origin of the coordinates is the centre of the robot. We are able to provide accurate robot size and shape by specifying its footprint. With the old footprint, the robot arm and the book storage compartment were regarded as obstacles in LiDAR scans. BookBot wasn’t aware of the arm and the container at the back of it which led to collisions sometimes. We updated the footprint with the coordinate values of six points around the robot's body and two points on the arm.

To evaluate the impact of the new footprint, tests were performed on navigation with the original footprint or the updated footprint. During each test, BookBot moved from the same Point A to Point B. BookBot performed much better with the new footprint, which led to fewer collisions and failures.

Goal tolerance

The parameter xy_goal_tolerance is the x, y distance allowed when the robot reaches its goal pose. To successfully grab a book, we have to ensure that the distance between the bookshelf and the front wheel on the left is 13cm and the goal could be reached in a reasonable time. The average deviation is the difference between the real distance and 13cm.

Four different values are tested in tuning. In each test, the robot moves from the initial position (robot station) to the bookshelf. After balancing between time cost and the distance deviation, we finally set the tolerance value as 1.

Minimum and maximum velocities

Translational velocity (measured in m/s) is the velocity when the robot is moving in a straight line. The parameter max_vel_x sets the maximum value of translational velocity. And the parameter min_vel_x sets the minimum value of translational velocity. After testing with different values, we decreased max_vel_x to 0.13 m/s to prevent collisions and set min_vel_x to -0.13 m/s to allow the robot to move backwards.

Navigation

Vision

Testing identification

Five books from the library were placed in five different orders and identified. The labels of all five books were correctly detected on all test runs. However, in all tests one of the books was not identified correctly. In the third test run, two 'phantom books' were detected. These are bright regions in the image our program identified as book labels and incorrectly matched it to one of the stored books, because Tesseract translated noise into text.

Testing tracking

The tracker algorithms available in OpenCV were evaluated using a laptop and the webcam. A specified book label was found and the label was tracked and the book spine detected. The most successful tracking method was CSRT (Discriminative Correlation Filter with Channel and Spatial Reliability).

The frame rate for each tracking algorithm captured on a pre-recorded video and running on the Raspberry pi 3 are also presented on the right.

Improving performance with threading

Threading has halved the time needed for book identification on a still frame (~10s to ~5s on the Raspberry Pi 4), which significantly reduces the time for aligning the robot overall as identification may run multiple times until a book is found on the shelf.

For book spine tracking, using a separate thread for the camera video stream and for publishing ROS messages has increased the rate at which frames are being processed from 0.7 fps to 2.5 fps, on average, when on battery power and from 1.5 fps to 6 fps when the Raspberry Pi is connected to the mains.

Testing alignment

We have conducted 20 runs of the full book alignment system after implementing the threading (including searching for the book, identification, tracking and alignment) and measured the runtime, as well as whether the alignment was successful or not. Alignment is successful when the robot is aligned with the correct book so that the lens of the camera is directly in front of the book. It can be unsuccessful if it fails to identify the book correctly, fails to track the book spine or fails to move the robot correctly. From the 20 runs, 19 were successful. On the one unsuccessful run, the system failed to identify the book. The robot starts scanning the book shelf from the left hand side of the shelf. Hence, the runtime depends on the position of the target book on the shelf.

Frame rate of OpenCV's tracking algorithms without the use of threading

Success of OpenCV's tracking algorithms

Identifying 5 library books

Runtime of full alignment procedure, including identifying the book

Click on the images to enlarge them

Vision

User Experience Tests (Web app)

We evaluated our product by testing it on team members from different subgroups (excluding the web app group).

Users have different familiarity with the user interface. This difference is shown as "Familiarity" ranging from 0 to 5, where 0 means knowing nothing about the web app and 5 means being very familiar with the web app. The familiarity value of the web developers is 5, though they are not in the test. Users were asked to complete 4 tasks in the test: sign up, log in, search and get a book.

"Success" means that the user was able to achieve the task with no assistance. "Had Struggles" means that the user had to ask for a small amount of help with the task. "Failed to complete the task" means that the user could not achieve the task and had to be explicitly shown how to do it.

The testing result on the left-hand side shows that all the users succeeded in signing up and logging in with little to no help and one user was not able to search for a book and therefore was not able to get a book.

Given that the goal project was to produce a proof of concept, functionality of the system was prioritised over fine-tuning the user experience (e.g. via multiple iterations of more extensive user testing). However, with the results from the first iteration being largely positive, a solid base for a UI has been discovered.

User Experience Evaluation results

Webapp

The Arm

Speed vs Noise Levels

Motivation

When setting the speed of the arm, we wanted to find the sweet spot between quick retrieval and quiet performance. The latter of this was incredibly important, as we want to ensure that BB will not disturb the ambience of the library.

The Data

The speed value of the arm can be set from a range of -1 to 1, where -1 is the slowest and 1 is the fastest. However, we chose to limit our testing to values between -0.8 and 0, as we generally found that anything beyond this range was either unreasonably loud or slow.

The Results

The table corresponding with this test features the 3 most prominent speed values used in our testing. We ran 5 individual tests for each speed value and found the average time taken to run the entire book gabbing algorithm, as well as the average noise level produced by the arm movements. As the average noise level of a quiet library is 40 decibels, we wanted to ensure that the noise levels produced by BB were not much louder than this.

Conclusion

Based on this data, we chose to set the speed to -0.5. We felt this was the best compromise, as there is only a 1 decibel difference between the average noise level of a library vs BB. Furthermore, 19 seconds is still relatively fast for BB to retrieve a book from the shelf.

Disclaimer

We measured BB’s noise level using a sound meter app on a mobile device, as we did not have access to a Sound Level Meter (SLM). While this was tested in a quiet room, we were not able to test this in a real library environment, so there may have been more ambient noise than we were aware of. For this reason, our test results may not be 100% accurate. However, we do feel that they give a relatively good representation of the low levels of noise that we have experienced BB to produce.

Effort

Finding Effort Values

Effort is the estimated force applied on a joint’s motor. A high value here indicates high force, whilst a low value indicates low force. The sign(+ve or -ve) shows the direction of the force. A free standing arm in its initial position will have some value of effort, as some force is required to keep it in the current position.

We used effort to tell us the current state of the gripper (more information can be found on the 'How does it work?' page). Multiple values of effort were tested on the gripper and shoulder combination. After running the tests 5 times for each combination, we found 3 effort combinations which leads to a successful grab.

Choosing Effort Values

We further tested the 3 combinations found above 7 times. After balancing the success rate and average time cost, we set the effort threshold values to be {-100, 100}.

Value of speed compared to the average time taken and noise produced for the arm

Effort values compared to time taken to run and number of successes produced

Results based on the effort threshold chosen

Click on the images to enlarge them

Arm

Improvements

Main areas of improvement

As we, like every other group, had to end our project early, there are still some things we had planned to implement or improve for our final proof of concept:

Putting the whole system together: Currently, each subsystem of our product works individually, however never had the chance to coordinate them to work as one unit. For example, the vision system will currently align itself with the lens of the camera directly in front of the target book, but we have not tested how to align it with the arm to grab the book.

Networking: As it stands, our networking set-up not been fully implemented. It currently works for the web app, vision and navigation, but we did not have the time to complete it with book grabbing. The communications between the web app and the other servers have also not been tested.

Searching for missing books: We had planned to implement the ability for BB to traverse a shelf, get the list of books which would normally be on that shelf from the database (excluding those currently loaned), and match all the books from the shelf to one book on the list. If the robot could not find a book, it would notify a staff member to check, and update the database to set the book as being ‘missing’. This would stop users from potentially ordering a book which is not actually available.

Grabbing books from different shelf levels: currently, BB is only able to grab a book from the top section of our prototype bookshelf. We had planned to have a solution for it to grab books from the bottom shelf implemented for the last demo. For a real product, we would also implement some form of lift system for BB to be able to reach higher levels of shelves.

Tightly packed bookshelf problem: Our current solution of grabbing books from a tightly packed shelf, relies on there being at least an extra 1 cm of space in the shelf overall. While we have found this to normally be the case with most bookshelves, as it is rare you will get books to perfectly occupy all the space on a shelf; we do have to acknowledge that in the very rare occasion where a shelf is too tightly packed, BB’s current method of retrieving books would fail. For this reason, we would have to try and design an alternate solution which would work in every scenario.

Reliable book retrieval: Due to the lack of physical access to the robot, we were unable to spend adequate time testing the grabbing software, meaning it is still a little bit unreliable. With more time, we would have likely changed the book grabbing software a lot, to make it more stable.