So, this happened:
The scourge of back EMF strikes again. Most of us are probably familiar with back EMF in the form of the large voltage spikes we see if a motor suddenly stops or gets back-driven. Large enough to fry anything else connected to the same power source, apparently.
MARS-X has 8 large servo motors and two separate battery packs. 4 motors each are connected to each battery. In order to power the various control electronics, there are also a variety of DC-DC regulators that are connected directly to the battery bus.
Cascade Failure
The regulator shown above was powering the Arduino inside one of the battery modules. It’s one of those LM2596 munchkins you can get on Amazon for, like, $5. As you might expect given the price, these regulators are not electrically isolated at all, and they tend to fail in one of two exciting ways: If you’re lucky, they fail open, and you lose the regulator. If you’re unlucky, they fail closed, and everything downstream of them ends up connected directly to the input voltage. This is the fate that befell the poor Arduino, which found itself briefly subjected to the full 40V from the battery pack. Consequently, the built-in regulator on that board is now a charcoal briquet:
This has happened before, and it means that I’ll have to go through the tremendously annoying process of de-soldering the Arduino from its carrier board and soldering in a new one. I’m starting to think those things should be socketed.
The other Arduino, luckily, survived. For somewhat accidental reasons, that one is connected to an expensive Meanwell power supply that we originally bought for something else. This power supply is isolated and seems to handle the voltage spikes much more elegantly. Rui already ordered a second one for the other Arduino, so I suppose the lesson has been learned.
Always use protection, boys and girls. Electrical protection, that is.
Motor Errors
Awhile ago, I added code to the motor controller that keeps and eye on the motor’s error registers, and puts up a fuss if it sees errors. A few days later, I took the robot out to do some autonomous navigation tests. Lo and behold, my motor error checker code started fussing. I could barely even start autonomous navigation before one of the motors would encounter a fault. I tried to do some debugging out in the field, but it was too cold to type and eventually it started to rain.
Later on, I identified the exact cause of the errors as a voltage spike that was causing the motors to protectively shut down. At first, it seemed like it was happening at random, but after a morning of abusing the robot, Rui and I settled on a working theory that this was being caused by sudden deceleration. The kinematic controller code for the robot is rather trusting; it will let you push the robot harder than you really should.
Unfortunately, the tests we came up with to trigger voltage faults turned out to be an excellent way to destroy voltage regulators. In hindsight, that should have been entirely predictable, but by the time the dust settled, we were looking at two dead regulators, a damaged pair of fans, and a destroyed Arduino. It might not sound like much, but once I get the parts in, it will probably take me several hours with a soldering iron to replace all that.
Solutions and Mitigations
The best mitigation for back EMF is to use some kind of shunt. It’s basically just a device that monitors the voltage on the battery bus, and connects a large resistor in parallel if the voltage gets too high in order to dissipate the excess power. (The BMS we’re using is supposed to be able to feed excess energy back into the battery pack, but this doesn’t seem to work if the battery is near full.)
I’ve also worked on improving the kinematics. Specifically, I put much harder limits on the maximum acceleration that the robot can sustain, and modified the controller to enforce this. This is a 500 lb robot after all, it’s not the swerve drive you built in high school robotics. Nobody is going to get mad if it handles like a piece of large farm equipment.
Dr. Li is concerned about MARS reliability in general. He has a point: so far, we’ve been working on the robot for three years, and there’s been a seemingly never-ending parade of issues. At the same time, we’ve recently made many ambitious upgrades, including a new adjustable frame, and incorporating CANBus. Also, I personally believe that the reliability of certain components, such as the sensing system, has improved greatly. Any engineer knows that when a platform is changing rapidly, reliability is bound to suffer. Hopefully now we can settle on a hardware platform and really focus on making it work.
But then I guess I might not have anything to write about here.
No responses yet