Agent integrates with DeepStack AI and senseAI to provide intelligent object detection and Facial Recognition services. Agent DVR uses AI object detection to apply an intelligent filter to alerts to reduce the number of false alert events to near zero. You configure Agent DVR as usual with a motion detector/ alert configuration and then add AI alert filtering to ignore alerts that don't have specific objects in the frame - like car, person, dog etc.

To get started, install an AI Server for your platform (Install DeepStack or Install senseAI). We currently recommend using senseAI as it seems to be more actively developed than DeepStack.

DeepStack Notes:

Use the GPU version if possible (you'll need an nvidia card for GPU acceleration). Make sure it's running in Vision Detection mode. To do that via the command line pass in "-e VISION-DETECTION=True".

If you want to use facial recognition as well as object detection then start DeepStack with "-e VISION-DETECTION=True VISION-FACE=True".


When you have the AI server up and running, switch back to Agent DVR and go to Server Menu - Settings - Intelligence:

  • AI Server URL: Enter the URL of your AI server. For example: http://localhost:5000
  • Endpoint: The path to the vision model endpoint on the AI server. Only change this if you are using a custom model. Default is /v1/vision/detection
  • Object Classes: This is the list of objects that the AI Server can find. Don't modify this unless you have loaded a custom model.
  • Timeout: Timeout in seconds for the requests to the server.
  • Auto Lock Tags Configure a list of tags that when applied will automatically lock the files (prevent them from being deleted by the automatic storage management system).

Edit a camera and go to the Alert Filter tab (or the Intelligence option on the Alerts tab in previous versions of Agent DVR):

  • Enabled: Switch AI Filtering on or off. You can also do this via the Schedule
  • Filter on Failure: If DeepStack doesn't respond you can choose to allow or block the alert here. With this on alerts will be blocked if DeepStack doesn't respond.
  • Override URL: You can specify an alternate AI server here which will override the main server in server settings. Note that this server will then be used by this camera for all processing (face/ object recognition/ alert filtering etc)
  • Configure...: Click this to configure the Alert Filter

Click the button to configure the filter option:

AI Server Configuration
  • Mask: Define an area to IGNORE (red) - Use the tools at top right to create a mask to ignore. Important: The red area is ignored.
    This mask is used by both the alert filter and the object recognition system
  • Find: Choose from a list of objects to look for. Example: car, dog, person
  • Confidence: Minimum confidence of result. Anything below this value will be discarded (0-100)
  • Photos: Check this to take a photo. These photos will be saved to Photos in Agent DVR.
  • Label: Check this to label the photo with results.
  • Prevent Repeat: Check this to stop Agent from generating repeat alerts about objects in the same place they were last seen.
  • Request Interval (ms): Milliseconds to wait after getting a response before sending another request to DeepStack - can help reduce server load. 1 second = 1000 ms.
  • Alert Condition: Choose from More Than, Less Than or Equal to (applies to Alert Number)
  • Alert Number: The number to use in the condition. For example, Find car More Than 0 will alert if any car is found. Find car Less Than 1 will alert if a car is not found.
  • Region Match: Enable this to only alert when the area of detected movement overlaps the detected object. Only works if the motion detector is a tracking detector (speed, tracking or tripwires).

Agent will generate motion events as normal (according to the motion detection tab) and when it does it will use the AI Server to look for relevant objects in the scene and decide if this is a real alert or not.

For Agent to apply the AI filter to new recordings, make sure Mode on the Recording tab when you edit the camera is set to Alert and not Detect.

Note: The first request into the AI server can take a while to complete whilst it is initializing. You may see an error message about it not responding. Usually the second request will complete OK.

Configure the options on the screen. In this example we're blocking out the office using the Mask option but looking for people in the corridor. If a person is detected an alert will be generated.

When you are done click OK and then toggle the Enabled switch to start using intelligent filtering.

Tip: You can specify multiple AI servers by using a URL override setting in the Alert Filter settings.

General Object Recognition

In addition to the intelligent alert filter Agent also offers general object recognition. This will generate events like "Object Found" and "Object Not Found" that you can perform actions on. General object recognition can be triggered by motion detection, alerts or via API

After configuring AI, edit a camera and go to the Object Recognition tab:

Configure Object Recognition

  • Enabled: Switch Object Recognition on or off. You can also do this via the Schedule
  • Mode: Choose between Motion, Alert, Interval or None. Motion and alert only process images when something is happening. Interval will constantly send images to the AI Server. Use None if you only want to process images using the API.
  • Minimum Interval: How often to process frames when triggered.
  • Confidence: Minimum confidence of result. Anything below this value will be discarded (0-100)
  • Photos: Check this to take a photo and overlay results. These photos will be saved to Photos in Agent DVR.
  • Prevent Repeat: Check this to ignore objects repeatedly detected in the same location.
  • Find: Enter the objects to look for in the scene (from the list) - example: car,dog. Make sure the labels match exactly.

Using Custom Models (DeepStack)

You can change the model that deepstack uses for object detection quite easily. There are several models you can use created by third parties. A good one, which has been trained specifically for security cameras is:

https://github.com/MikeLud/DeepStack-Security-Camera-Models/blob/main/Models/combined.pt

To install this into deepstack see the deepstack docs. Basically you create a folder on your computer. Copy the .pt file to it and start deepstack with the path to the folder: deepstack --MODELSTORE-DETECTION "C:/path-to-custom-models-folder"

Once your deepstack instance is running with the new model you'll need to update the available tags in Agent. In this example (the combined example linked above) the labels are: "person,bicycle,car,motorcycle,bus,truck,bird,cat,dog,horse,sheep,cow,bear,deer,rabbit,raccoon,fox,skunk,squirrel,pig". Copy this list (excluding the quotes) and paste it into Object Classes.

You'll then be able to use these new classes in the alert filter and object detection features of Agent DVR.

Be sure to update the Endpoint setting in Server Settings - Intelligence to use the new model

Changing the AI language

To change the language that Agent uses when displaying labels and tagging recordings see Server Menu - Settings - Playback - Default Language.


Adding Object Recognition Actions

When Object Recognition runs Agent will generate events for "Object Found" and "Object Not Found" under Actions.

You can use these events to trigger actions such as calling a URL with the object labels or running an executable or publishing a message to an MQTT server. You can use the tags {AI} for the labels (if more than one they will be in a CSV list) or {AIJSON} for the full response JSON object from DeepStack.


AI Server Troubleshooting

If AI isn't filtering your recordings:

Check your Find setting exactly matches one or more of the Available options.

  • If you are getting errors (code 500 server errors) from DeepStack on Windows you may need to install https://aka.ms/vs/16/release/vc_redist.x64.exe
  • Check the master alert switch at the top left of Agent - It should show a closed padlock meaning Alerts are active. Click it to toggle.
  • Check that the recording mode on the recording tab when you edit the camera is set to Alert not Detect. AI filtering only filters alert events.
  • Check that filtering is enabled on the Alert Filter tab.
  • Check that DeepStack is running in DETECTION mode (VISION-DETECTION=True on the command line).
  • Try adjusting the Confidence level down in Agent as results may be being filtered
  • Check the logs at /logs.html for error messages from the server. Possibly the AI service is running on a different port or a firewall or antivirus is blocking access to it. If Agent DVR can't contact the AI server it will record the event.
  • Check the alert condition. If, for example you have it set to Alert when objects detected is less than 2 then it will alert when nothing is detected.
  • Check that the AI server isn't overloading your computer. If the AI server can't process the images quickly then it will time-out and Agent DVR will give up waiting for a result. If the AI server is timing out you will see errors in the logs. You may need to move the AI service to another computer with more resources or a compatible GPU.
  • If DeepStack is detecting every class of object this usually means it's not working with your GPU. Check your GPU drivers and try the CPU version of DeepStack instead.