Agent integrates with DeepStack AI and CodeProject.AI to provide intelligent object detection and Facial Recognition services. Agent DVR uses AI object detection to apply an intelligent filter to alerts to reduce the number of false alert events to near zero. You configure Agent DVR as usual with a motion detector/ alert configuration and then add AI alert filtering to ignore alerts that don't have specific objects in the frame - like car, person, dog etc.

To get started, install an AI Server for your platform (Install DeepStack or Install CodeProject.AI). We currently recommend using CodeProject.AI as it seems to be more actively developed than DeepStack.

These instructions are for agent v4.4.6.0 and below. For v4.4.7.0+ See AI Settings.
CodeProject have put a great guide together for configuring Agent DVR with CodeProject.AI.
DeepStack Notes:

Use the GPU version if possible (you'll need an nvidia card for GPU acceleration). Make sure it's running in Vision Detection mode. To do that via the command line pass in "-e VISION-DETECTION=True".

If you want to use facial recognition as well as object detection then start DeepStack with "-e VISION-DETECTION=True VISION-FACE=True".

When you have the AI server up and running, switch back to Agent DVR and go to Server Menu - Settings - AI Settings and click the configure button then Add to add it to Agent DVR:

You can add multiple AI servers here or multiple instances of the same server using different models.

  • Name: Give your server a name, eg: Cat Catcher (animal model)
  • AI Server URL: Enter the URL of your AI server. For example: http://localhost:5000
  • Discover: Click Get Models to retrieve the available models from the server (CodeProject.AI only)
  • Model Endpoint: The path to the vision model endpoint on the AI server. Only change this if you are using a custom model. Default is /v1/vision/detection
  • Object Classes: This is the list of objects that the AI Server model is generated for. Don't modify this unless you have loaded a custom model.
  • API Key: Enter your key if you have set one up.
  • Timeout: Timeout in seconds for the requests to the server.
  • Retry delay: Time in seconds before retrying requests to this server if they fail.

Also see AI Settings:

    Auto Lock Tags Configure a list of tags that when applied will automatically lock the files (prevent them from being deleted by the automatic storage management system).

Edit a camera and go to the Alert Filter tab (or the Intelligence option on the Alerts tab in previous versions of Agent DVR):

  • Enabled: Switch AI Filtering on or off. You can also do this via the Schedule
  • Filter on Failure: If DeepStack doesn't respond you can choose to allow or block the alert here. With this on alerts will be blocked if DeepStack doesn't respond.
  • Override URL: You can specify an alternate AI server here which will override the main server in server settings. Note that this server will then be used by this camera for all processing (face/ object recognition/ alert filtering etc). This is deprecated in 4.2.5.0+ (add multiple AI servers instead).
  • Configure...: Click this to configure the Alert Filter

Click the button to configure the filter option:

AI Server Configuration
  • Mask: Define an area to IGNORE (red) - Use the tools at top right to create a mask to ignore. Important: The red area is ignored.
    This mask is used by both the alert filter and the object recognition system
  • Find: Choose from a list of objects to look for. Example: car, dog, person
  • Confidence: Minimum confidence of result. Anything below this value will be discarded (0-100)
  • Photos: Check this to take a photo. These photos will be saved to Photos in Agent DVR.
  • FTP: Upload the images to the devices FTP server.
  • Label: Check this to label the photo with results.
  • Crop: Cut out detected objects instead of saving full image.
  • Use Snapshot URI: Get a jpeg image from the devices snapshot URI instead of using the live frame (useful if the camera has a high resolution endpoint for still images).
  • Prevent Repeat: Check this to stop Agent from generating repeat alerts about objects in the same place they were last seen.
  • Request Interval (ms): Milliseconds to wait after getting a response before sending another request to DeepStack - can help reduce server load. 1 second = 1000 ms.
  • Alert Condition: Choose from More Than, Less Than or Equal to (applies to Alert Number)
  • Alert Number: The number to use in the condition. For example, Find car More Than 0 will alert if any car is found. Find car Less Than 1 will alert if a car is not found.
  • Region Match: Enable this to only alert when the area of detected movement overlaps the detected object. Only works if the motion detector is a tracking detector (speed, tracking or tripwires).

Agent will generate motion events as normal (according to the motion detection tab) and when it does it will use the AI Server to look for relevant objects in the scene and decide if this is a real alert or not.

For Agent to apply the AI filter to new recordings, make sure Mode on the Recording tab when you edit the camera is set to Alert and not Detect.

Note: The first request into the AI server can take a while to complete whilst it is initializing. You may see an error message about it not responding. Usually the second request will complete OK.

Configure the options on the screen. In this example we're blocking out the office using the Mask option but looking for people in the corridor. If a person is detected an alert will be generated.

When you are done click OK and then toggle the Enabled switch to start using intelligent filtering.

Tip: You can specify multiple AI servers by using a URL override setting in the Alert Filter settings.

General Object Recognition

In addition to the intelligent alert filter Agent also offers general object recognition. This will generate events like "Object Found" and "Object Not Found" that you can perform actions on. General object recognition can be triggered by motion detection, alerts or via API

After configuring AI, edit a camera and go to the Object Recognition tab:

Configure Object Recognition

  • Enabled: Switch Object Recognition on or off. You can also do this via the Schedule
  • Mode: Choose between Motion, Alert, Interval or None. Motion and alert only process images when something is happening. Interval will constantly send images to the AI Server. Use None if you only want to process images using the API.
  • Minimum Interval: How often to process frames when triggered.
  • Confidence: Minimum confidence of result. Anything below this value will be discarded (0-100)
  • Photos: Check this to take a photo and overlay results. These photos will be saved to Photos in Agent DVR.
  • FTP: Upload the images to the devices FTP server.
  • Label: Check this to label the photo with results.
  • Crop: Cut out detected objects instead of saving full image.
  • Use Snapshot URI: Get a jpeg image from the devices snapshot URI instead of using the live frame (useful if the camera has a high resolution endpoint for still images).
  • Prevent Repeat: Check this to ignore objects repeatedly detected in the same location.
  • Find: Enter the objects to look for in the scene (from the list) - example: car,dog. Make sure the labels match exactly.
Using Custom Models (DeepStack)

You can change the model that deepstack uses for object detection quite easily. There are several models you can use created by third parties. A good one, which has been trained specifically for security cameras is:

https://github.com/MikeLud/DeepStack-Security-Camera-Models/blob/main/Models/combined.pt

To install this into deepstack see the deepstack docs. Basically you create a folder on your computer. Copy the .pt file to it and start deepstack with the path to the folder: deepstack --MODELSTORE-DETECTION "C:/path-to-custom-models-folder"

Once your deepstack instance is running with the new model you'll need to update the available tags in Agent. In this example (the combined example linked above) the labels are: "person,bicycle,car,motorcycle,bus,truck,bird,cat,dog,horse,sheep,cow,bear,deer,rabbit,raccoon,fox,skunk,squirrel,pig". Copy this list (excluding the quotes) and paste it into Object Classes.

You'll then be able to use these new classes in the alert filter and object detection features of Agent DVR.

Be sure to update the Endpoint setting in Server Settings - Intelligence to use the new model
Changing the AI language

To change the language that Agent uses when displaying labels and tagging recordings see Server Menu - Settings - Playback - Default Language.

Adding Object Recognition Actions

When Object Recognition runs Agent will generate events for "Object Found" and "Object Not Found" under Actions.

You can use these events to trigger actions such as calling a URL with the object labels or running an executable or publishing a message to an MQTT server. You can use the tags {AI} for the labels (if more than one they will be in a CSV list) or {AIJSON} for the full response JSON object from DeepStack.

AI Server Troubleshooting

If AI isn't filtering your recordings:

Check your Find setting exactly matches one or more of the Available options.

  • If you are getting errors (code 500 server errors) from DeepStack on Windows you may need to install https://aka.ms/vs/16/release/vc_redist.x64.exe
  • Check the master alert switch at the top left of Agent - It should show a closed padlock meaning Alerts are active. Click it to toggle.
  • Check that the recording mode on the recording tab when you edit the camera is set to Alert not Detect. AI filtering only filters alert events.
  • Check that filtering is enabled on the Alert Filter tab.
  • Check that DeepStack is running in DETECTION mode (VISION-DETECTION=True on the command line).
  • Try adjusting the Confidence level down in Agent as results may be being filtered
  • Check the logs at /logs.html for error messages from the server. Possibly the AI service is running on a different port or a firewall or antivirus is blocking access to it. If Agent DVR can't contact the AI server it will record the event.
  • Check the alert condition. If, for example you have it set to Alert when objects detected is less than 2 then it will alert when nothing is detected.
  • Check that the AI server isn't overloading your computer. If the AI server can't process the images quickly then it will time-out and Agent DVR will give up waiting for a result. If the AI server is timing out you will see errors in the logs. You may need to move the AI service to another computer with more resources or a compatible GPU.
  • If DeepStack is detecting every class of object this usually means it's not working with your GPU. Check your GPU drivers and try the CPU version of DeepStack instead.