I’ve gotten into the habit of doing tests with MARS on the weekends. I like to use the giant parking lot outside of our building, which is usually pretty quiet outside of work hours. Typically, I set up a simple autonomous navigation program that drives the robot in a circle, and run it until either something breaks or it’s about to crash into a parked car. I’ve racked up fair number of hours doing this, which probably presents a pretty damning indictment of my social life.
A while ago we discovered a pernicious electrical issue with MARS caused by back-EMF from the motors. This excess voltage, when not properly dissipated, was damaging all of the sensitive electrical components on the robot. Until this issue was fully resolved, I elected not to do any more parking lot tests. We ordered some shunts to deal with the voltage problem, but of course, there were issues with UF purchasing, and then my initial supplier ghosted me, and then I had to wait for the USPS to begrudgingly work its magic. All told, I ended up not being able to do any tests for a whole month.
But last week, the shunts finally came! They were pretty easy to install, and they seem to have fixed the problem. That means that this weekend was inevitably a robot test day:
This test was probably the most successful one yet. I was unable to replicate the navigation issues I was having before, which makes me inclined to think that they were caused by a poor GPS signal. (I was forced to test close to the building on that day.) I still encountered a couple of problems with the motors, however.
More Arduino Woes
First up, I noticed that one side of the robot would sometimes randomly stop responding to motor commands. The motors did not appear to be in a fault state. I had a hunch that this was an Arduino problem, and, sure enough, checking the logs for one of them revealed that it was repeatedly failing to forward CAN frames from the other side to the controller. After some debugging, I figured out that the cause was probably a feature of the ESP32 CANBus implementation which automatically disconnects a node from the bus if it encounters too many CAN errors in too short of a period.
The solution is clear: I need to implement the proper recovery behavior in the firmware. (I added a basic version of this feature in the field, and it seemed to work okay.) Annoyingly, the Arduino CAN driver has the code for this recovery feature, but nobody ever bothered to cut a new release for it. (Hey Arduino CAN driver maintainer, if you’re reading this, could you do that? Thanks!) I had to clone the repository manually to my Arduino libraries folder.
Of course, the bigger question is why we’re sometimes getting enough CAN errors in a short period of time to trigger the error handling feature. I don’t really know, but some very basic tests suggest that the robot operates error-free most of the time, and the errors happen in bursts. I suspect that these bursts coincide with times when there’s a lot of congestion on the bus, such as when the controller is sending a storm of SDO requests during the motor initialization process. (This can also happen during error recovery.)
Motor Errors
MARS uses Moog Animatics SmartMotors for all of its joints. These motors can detect a variety of fault conditions, and their default response is to immediately shut themselves off, halting all motion. One exception to this is the intermittent over-current fault, which the motor will report, but is not supposed to affect the motion.
Reader, this is a lie.
As near as I can tell, over-current errors do cause the halting behavior. (I’m hedging a little because I’m not 100% certain that this is not triggered by some over-zealous error handler in the driver code.) This wouldn’t be a problem, except that over-current faults happen fairly regularly on the drive motors when performing certain maneuvers. This is probably, at root, an issue with the kinematics being slightly off, but I’m not sure how precisely we can dial that in.
In the meantime, we need a graceful way to handle over-current errors without causing wheels to randomly jerk to a stop while the robot is moving. I have some thoughts on the matter, but I’ll have to test them more thoroughly to see if anything works adequately.
Overall, I was rather pleased with the outcome of this test. Nothing exploded, the robot didn’t have to be pushed back into the lab, and I didn’t get caught in a surprise downpour.
Although to be honest, the bar isn’t exactly high.
No responses yet