Duo Security is one of the most popular 2-factor authentication applications on the market today. All of the authentication and administrative logs are stored in the admin portal located at https://admin.duosecurity.com. Up until recently, if you wanted to view the logs, you either had to log into the admin portal or use the Duo API to query your Duo instance and manually pull the logs. There have been some scripts that individuals have released to perform this API query, but recently, Duo released their own official version: https://github.com/duosecurity/duo_log_sync/.

If you want to use the Duo Log Sync to stream the logs to a SIEM, it works well if your SIEM supports the json TCP stream. However, if your SIEM only supports log file reads or syslog, you need to do some extra work. In this article, we’ll discuss how to use another component, fluentd, to get syslog running with the Duo Log Sync and we’ll also give you regex rules to parse the Duo data. We’ll use a Windows server in our example, but the similar rules apply for Linux.

Set up Duo Admin API

First, you’ll need to add the Duo Admin API to your Duo instance.

• Log into the admin portal located at https://admin.duosecurity.com
• Click on Applications -> Protect an Application
• Select the “DUO Admin API”

• Copy the keys – Once the “Duo Admin API” application is created, you’ll need to copy the hostname and key values to use in the Duo Log Sync configuration.

Use the Integration Key (ikey), Secret key (skey) and API hostname (hostname) values here to populate the configuration script

Set up the Duo Log Sync

Follow the instructions from the README on https://github.com/duosecurity/duo_log_sync/

• Install Python3 on the server if it’s not already installed
• Download the duo_log_sync program from https://github.com/duosecurity/duo_log_sync/. If you’re on linux, you can just clone the repository. On Windows, if you don’t have GIT installed, just download the zip and extract to the root directory.
• Go to the duo_log_sync folder and from the command-line type:

python3 setup.py install

• Create a temp folder at c:\temp to store the log files. On Linux, you’ll already have a /tmp folder that can be used.
• Create/Edit the config.yml file located here c:\duo_log_sync\duologsync\config.yml Use the Example configuration from the README file as a guide. On Windows, you’ll need to escape the directory references and put the full path (see example config.yml below). Ensure you enter the skey, ikey, and host values from your Admin API application in the Duo Admin Management panel (see screenshot above).
• Send data to SIEM or log device If your SIEM supports receiving data via TCP Streams, then all you need to do is put the IP information for the SIEM in the “transport” section of the config.yml file. Unfortunately, some SIEMs do not support this and require either a file dump or syslog. So if you can do the TCP streams, you can skip to the scheduled task section and ignore all of the fluentd steps. Otherwise, proceed to the next section.

Linux config.yml file:

duoclient:
  skey: "ENTER-SECRET-KEY-HERE"
  ikey: "ENTER-INTEGRATION-KEY-HERE"
  host: "ENTER-API-HOSTNAME-HERE"

logs:
  logDir: "/tmp"
  endpoints:
    enabled: ["auth", "telephony", "adminaction"]
  polling:
    duration: 5
    daysinpast: 1
  checkpointDir: "/tmp"

transport:
  protocol: "TCP"
  host: "localhost"
  port: 8888
  certFileDir: "/tmp"
  certFileName: "selfsigned.cert"

recoverFromCheckpoint:
  enabled: False

Windows config.yml file:

duoclient:
  skey: "ENTER-SECRET-KEY-HERE"
  ikey: "ENTER-INTEGRATION-KEY-HERE"
  host: "ENTER-API-HOSTNAME-HERE"

logs:
  logDir: "c:\\temp"
  endpoints:
    enabled: ["auth", "telephony", "adminaction"]
  polling:
    duration: 5
    daysinpast: 1
  checkpointDir: "c:\\temp"

transport:
  protocol: "TCP"
  host: "localhost"
  port: 8888
  certFileDir: "c:\\temp"
  certFileName: "selfsigned.cert"

recoverFromCheckpoint:
  enabled: False

Setup Fluentd

Fluentd (https://www.fluentd.org/) can be used to accept the TCP stream data from the Duo_log_sync script and send it to your SIEM via syslog. We need to install and configure this to act as the “proxy” between the duo_log_sync script and your SIEM. This can be installed and configured on the same system that you configured the duo_log_sync, but it can also be installed on a different system. If you’re installing on the same system, then the duo_log_sync config.yml file should contain “localhost” for the “host” field in the “transport” section. If you set up fluentd on a separate system, then add the necessary host information in the config.yml file.

• Download fluentd from https://www.fluentd.org/download. We’re using the Windows version here, but you can use any version. Use the respective installation guide per your OS version.
• Install the application
• From the start menu, run the td-agent command

• Install the syslog plugin.

1. The plugin details are here: https://github.com/dlackty/fluent-plugin-remote_syslog
2. To install, from the td-agent prompt, type the following:

fluent-gem install fluent-plugin-remote_syslog

• Edit the td-agent.conf file located here on Windows: C:/opt/td-agent/etc/td-agent/td-agent.conf

• a. Go to the “built-in TCP input” section and put the following while commenting out the other lines if necessary with “#” characters. It should look like this:

• b. (optional) Test the log delivery by sending the Duo logs to the screen. Add the lines below in the script. Otherwise, just skip to the next step to send files to your SIEM.

• c. To send the logs to the SIEM, use the configuration below. Replace the host information with the hostname of your SIEM.

Test sending logs

• Start the td-agent process

• a. At the td-agent command prompt, type:

fluentd -c etc\td-agent\td-agent.conf

• Run the duo_log_sync script

• a. Open an admin command prompt. The duo script will NOT work from a non-admin prompt so you need to right-click on the CMD icon and click “run as administrator”

• b. Enter the command to run:

duologsync c:\duo_log_sync\duologsync\config.yml

• The command batches 1000 lines at a time and will continuously run. Note, there is a 5 minute delay from when it actually starts collecting logs.
• Check progress at c:\temp\duologsync.log
• Everything should now work and your logs should be going to the SIEM!

Schedule Task

Once everything works, we’ll want to schedule tasks for this to run automatically. First cancel the current running duo_log_sync and td-agent processes if they are running.

Fluentd as a service

• Open a td-agent command window. We’ll follow the steps on the Windows page to register this as a service: https://docs.fluentd.org/installation/install-by-msi
• Type the commands below to register tg-agent as a service

fluentd --reg-winsvc i
fluentd --reg-winsvc-fluentdopt '-c C:/opt/td-agent/etc/td-agent/td-agent.conf -o C:/opt/td-agent/td-agent.log'

• Using the Windows GUI, go to Control Panel -> System and Security -> Administrative Tools -> Services, and you’ll see Fluentd Windows Service
• Start the Fluentd Windows Service and set it to run Automatically

Duo Log Sync as a service

• Open the Task Scheduler
• Click on “Create Task”
• Name: Duo Log Sync
• General -> Security Options. Run the task as your designated service account. Ensure the user does not need to be logged in.
• Triggers Tab

• a. New
• b. Set to run at startup

• Action Tab

• a. Set this to start the Duo program
• b. Use the full path to the duologsync.exe file
• c. Arguments should be the location of the config.yml file.

• Settings

• a. Allow task to be run on demand

• Manually kick off the task to start it. It’ll run continuously.

You’re done! You can manually kick off the task to ensure it runs properly.

Duo Log Regex

There are three basic logs in Duo, the Admin logs, auth logs, and telephony logs. To save you some time on parsing these events, here are the event types and associated regex commands.

Auth Log regex

{"access_device":{"browser":"?([^",]+)"?,.+hostname":"?([^",]+)"?,"ip":"?([^",]+)"?,.+location":{"city":"?([^",]+)"?,"country":"?([^",]+)"?,"state":"?([^",]+)"?},"os":"?([^",]+)"?,.+name":"?([^",]+)"?},"auth_device":{"ip":"?([^",]+)"?,"location":{"city":"?([^",]+)"?,"country":"?([^",]+)"?,"state":"?([^",]+)"?},"name":"?([^}"]+)"*},\s*"email":"?([^",]+)?"?,"event_type":"?([^",]+)"?,"factor":"?([^",]+)"?,"isotimestamp":"?([^",\+\.]+).+"?,"reason":"?([^",]+)"?,"result":"?([^",]+)"?.+user":{"groups":\["?([^\]]+)?"?\],"key":"?([^",]+)"?,"name":"?([^",]+)"?.+

Admin Log regex

{"action":"?([^",]+)"?,"description":"{(.+)?}","isotimestamp":"?([^",\+]+).+"?,"object":"?([^",]+)"?.+username":"?([^",]+)"?,"eventtype":"?([^",]+)"?.+

Telephony Log regex

{"context":"?([^",]+)"?,.+isotimestamp":"?([^",\+]+).+"?,"phone":"?([^",]+)"?.+"type":"?([^",]+)"?,"eventtype":"?([^",]+)"?.+