We continually see integrators and users getting into trouble with their video analytics choices because they thought they were making logical choices. What seemed reasonable turned out to be wrong. Unfortunately, this has led to a lot of disappointment.

For example, here are some of the common mistakes made, that seem logical, but often turn out to be false:

  1. Having more manual control over how the analytics work will give you better detection accuracy.
  2. The longer the list of analytics behaviors, the more advanced and better the system is.
  3. The company that is known best in a market will have the best technology, or at least very good technology.
  4. Calibration adds accuracy, and therefore such systems work better than those that don’t require calibration.
  5. A system tweaked and tuned by a well-trained technician or engineer will work better than another system installed by someone with little training.

All of the above seem to make sense. They certainly would not lead to big problems if you used that kind of reasoning when choosing most technologies or products. However, they fail when it comes to video analytics. Why?

Let’s review the above 5 cases. Then the answer will become obvious.

1.  Most analytics systems require tuning and tweaking to reduce false alarms. Performance is horrible without those knobs and adjustments. However, this is actually a weakness, not a strength. When we introduced our first VideoIQ products, back in 2003, we had a system that worked fairly well out of the box without any adjustments, but we included some calibration tools as well, to help improve detection accuracy further. What we soon discovered, however, was that calibrating the size of objects often created more problems than it solved. Our biggest users stopped using the calibration and got better results. In other words, it is easy to make things worse, and it can take lots of training to know when to use which knobs. In 2007, when we relaunched the technology, we made the calibration process fully automatic. This has been a huge advantage and leads to far more successful deployments than we’d seen before.

This doesn’t mean that having added controls is a bad idea. They can, if used correctly, help. But this doesn’t mean a technology with lots of tuning knobs is better than one with less. From what we’ve seen, the opposite is true.

2.  Generally, a long list of capabilities is a good thing, even if you don’t need them all. Comparing products, why shouldn’t you choose one that can do more? When it comes to analytics, the reason is simple: Because it is far more important that the product can work well enough to be usable. For most video analytics systems, this simply isn’t true.

We’ve seen serious money put into offerings by big name companies, that have long lists of behaviors and some beautiful software, but they are disasters in the field. Why would companies spend so much money, and risk their reputation, to offer a solution that was not even usable in most cases? Because they are following other market leaders. This is the blind leading the blind.  It might look like it works in a lab, but the real world is far more difficult. Don’t worry about a long list of behaviors. Look first for something that really works  – at least well enough to be usable. That’s hard enough to find in the analytics space today.

3.  The company who has had the best market awareness is ObjectVideo. Unfortunately, they have also created more problems for the industry than anyone. This is just about unanimously agreed on by all the video analytics companies I’ve talked with. Their technology has serious problems with detection accuracy. Even ex-employees of their company have admitted this to me. They achieved such wide attention, not from so many happy customers, but from having raised $60 million dollars early on, and using those dollars to market and promote themselves into the industry recognized leader. Their sales grew rapidly, but then plummeted as problems became rampant. Seeing the problems that have been created, we’ve taken the opposite approach. We  spend hardly any money on advertising in publications, because we know that people first need to see it work. So, we rely largely on word of mouth, and helping people try our products. As far as we know, we are by far the fastest growing analytics solution on the market.

4.  I’ve blogged about the problems with manual calibration already, so I don’t need to go through all of this again. Calibration does improve accuracy, but the problem is that manual calibration also introduces problems. Automatic calibration that continually adapts as the scene changes over time, is better. But this isn’t the most important issue. The underlying accuracy of detection is far more important. While calibration does improve performance, the question you should ask is: Where is the accuracy starting point, before this improvement? How well does it work without any calibration? That’s far more important when comparing technologies these days. Why? Because most are horrible without calibration, and it shows how bad their underlying accuracy really is. Calibration is only going to filter out a limited number of false alarms. Try this test when comparing video analytics: Run them without any calibration and see how they compare. It’s a real eye opener.

5.  The logic for this one is simple: Installing a system by a well-trained engineer should always perform better than when installed by someone without any training. Seems obvious. So, then, buying products that will be installed by an engineer from the factory should be better than systems installed by integrator who are not experts. The logic fails, however, because the underlying accuracy of the technology is far more important. All the filters and tuning knobs and calibration can only patch up so many holes. We saw one system, with over a hundred cameras, where factory engineers spent months tuning and optimizing the system, but it still produced 10 false alarms per day per camera. That was an improvement, since it started off producing twice as many when it was first installed. As a comparison, they took our cameras and installed four of them in the worst locations. They produced 0.5 false alarms per day. This was without any tuning or tweaking.

Simple logic seems to fail because there are way too many companies selling what they call analytics, but they aren’t even close to being good enough. John Honovich, who has been making a concerted effort to test as many video analytics systems as he can, recently posted in a discussion with me that he was coming to the conclusion that out of 40-50 products on the market, maybe 3-4 at most were usable.

So, the first new rule of logic to use, in a market like this, is to pick something that really works, and ignore all the hype, the long lists of features, the fancy software, the big brand names, etc. Someday, the technology will be advanced enough and widespread enough, that everything will be good enough. But today the market is flooded by products that are so bad that they are only usable in very limited applications, even with careful adjustments and tuning. Even from leading companies.

It’s not always logical when it comes to analytics, because detection accuracy in the real world is far more complex and challenging than it seems. Only a few of the most advanced technologies are good enough. And the companies with the strongest market recognition have been some of the worst. It’s a strange problem that you don’t usually see. Fortunately, more and more companies are finding success with video analytics, because they are first finding something that really works, and then learning the best places to use it.

 

Advertisement