Enhancing my ordinary IP security cameras with AI

Artificial Intelligence is quickly becoming an important ingredient for IoT projects’ success, a requirement for unlocking its full potential and providing a competitive edge to those that embrace it. AI naturally integrates into existing connected sensor/actuator networks and immediately adds measurable value. It was hard to imagine until recently, that AI will be so easily accessible, with just a few libraries installed we can take benefit of this amazing technology.

Let me illustrate with a simple example – enhancing ordinary IP security cameras with AI. The goal is that the cameras will recognize the objects they see and publish the recognized object’s data to an MQTT topic. I am interested in the detected object’s type, location within the captured frame and recognition confidence level. The applications I am considering are:

  • Object following via PTZ (keeping the object of interest e.g. human in the middle of the frame)
  • PTZ camera blind spot avoidance with opposing cameras providing each other that object of interest is beyond their current viewing angle
  • Smart movement detection i.e. only trigger alarm event if certain object types are detected
  • Monitor our presence at home vs detected object of risky type, i.e. I don’t want to see humans wandering the yard while we are at work, or during nights while the security system is armed
  • Providing the object’s data over MQTT so other IoT nodes can make decisions/trigger actions

The project relies on image classification with deep convolutional neural networks using the darkflow library. Setting up darkflow is pretty trivial, simply follow the instructions in the repository. I used Python, basically, it starts an FFmpeg background process that captures a single frame from the RTSP stream every 5 seconds and passes it to darkflow for analysis. The analysis results are then published to an MQTT topic, and as an option, a screenshot of the detected objects is taken.

The reason why I used FFmpeg rather than OpenCV to capture the RTSP frames is that OpenCV implements RTSP over TCP by default. My specific camera model only provides RTSP over UDP. I tried to recompile with UDP support, but could not get it to work reliably. I use a RAM drive to avoid excessive writes on my SSD. Ideally, this setup will run on a Raspberry Pi 3 B+, I have one in order.

My code is available here for you to try out.

Couple screenshots, note how the detected objects are outlined, labeled and confidence level visualised:


Page views: 5719

23 thoughts on “Enhancing my ordinary IP security cameras with AI

  1. Hi Martin, great project ! A couple of questions if I may ?
    1. How would use the RasPi camera locally ?
    2. In your python script I notice the yolo.weights file is stored off-box. Why is that ?
    3. Is all processing done by the raspi ? or is the raspi just grabbing the images and handing them to a much larger box for actual processing ?


  2. Awesome project! I’m a cctv/access control/voice/data/alarm(low voltage) field tech. Have you thought of adding cloud recording ability via another dedicated remote RPi0 or a cloud service such as Dropbox?

    I will definitely be keeping track on this project.

    Thanks again!

  3. Why 1 frame every 5 seconds? Would this not greatly limit it’s detection ability? A quick moving car or person, would likely be able to zip past the camera’s field of view, before it cycles for another picture?

  4. When you say follow the instructions in the repository you mean the readme file for darkflow? It was not so trivial for me. I guess I don’t quite pass the knowledge required what is really going on there.

      • I might use your help here if you don’t mind. I have a Raspberry PI 3 with more-or-less standard raspbian image with node-red I am using for “home automation”. My Linux understanding is fairly basic, I am a low level user mostly playing in node-red. Let me know if my understanding is correct:
        First I need to install Darknet: I just follow the steps here: https://pjreddie.com/darknet/install/. Should I pick OpenCV or CUDA? I can’t pick which would be better in this case.
        Next I install Yolo using the steps here: https://pjreddie.com/darknet/yolo/
        Next I download darkflow: I just download this to my /home/pi/darkflow folder? I also can’t really tell which compile option to use? Just use the inplace option?
        At this point I assume I can download one of the pre-trained files weights and use your python example to process IP camera images. That should be all, right?

          • I installed openCV. Downloaded the darkflow to my .flow folder and tried to compile it. I got the following message:
            File “setup.py”, line 3, in
            from Cython.Build import cythonize
            ImportError: No module named ‘Cython’
            I installed Cython (pip install –user Cython). Installation was successful, but the same message on the darkflow build persists.

  5. Hello, I’m trying your project using RPI 3B and IP Cam, but I got an error in python..

    Traceback (most recent call last):
    File “/home/pi/IPcam/AI-ipcam-master/ai-ipcam.py”, line 101, in

    I can control PTZ using python and I also created a ramdisk on RPI.
    I think my cam frame is not ready, but I don’t know the reason..
    I’d appreciate it if you can tell me how to solve it.

    • Hi,
      that error message doesn’t give out much what the issue is. I suggest printing out more debug info to help narrow the issue. You can easily spot, for example, if the screengrabs are being stored into the temp ramdrive.

  6. Greetings, I am trying to get this up and running on my pi now but I am getting the error

    Traceback (most recent call last):
    File “”, line 1, in
    ImportError: cannot import name ‘TFNet’

    Are you actually using “tensorflow1.0” as described in the requirements? the earliest version I can find on the tensorflow repository is 1.7 and the current version is 1.9.

    Thanks, can’t wait to get this working!

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.