High Level Overview
Let’s suppose that you have a list of Indicator’s of Compromise such as IPv4 being updated on a weekly basis on a publicly accessible URL or this URL could also be used by your team to push the indicators of compromise into Azure Sentinel.
I will detail the steps that are used to ingest these indicators of compromise into the native ThreatIntelligence table.
Pre-Requisites
Threat intelligence table needs to exist in the Sentinel Workspace.
You can install the solution from Sentinel -> Content Hub -> Threat Intelligence
Once the solution has been installed, easiest way is to connect Microsoft Defender Threat Intelligence data connector. You just need to configure the settings and hit the Connect button!
URL's
Following has a good curated list of open source Threat Intel.
However, for this logic app, I am using the following URL
You might want to try with a smaller list of indicators of compromise for testing purposes as you can run into rate limit issues with the above URL. You can also create your own sample Ipv4 list on github public repository.
High Level Diagram
Legend – Oval/Round indicates trigger, Rectangle indicates Action and triangle indicates condition.
Green arrow indicates Next Step in the flow, Blue arrow indicates true branch and Red arrow indicates false branch
Low Level Overview
As I mentioned earlier, we use a Recurrence trigger and here, we can set the interval and the time at which the logic app should run. After that, I am just initializing a few variables and setting initial values in these variables. For example, use your subscription Id when initializing the variable.
Make sure you populate the URL variable with your Github endpoint/URL.
After this, we do the following. This is very simple, I am just using HTTP Action to get data from our Github URL.
After this, I use a condition to evaluate the status code.
If status code returns anything other than 200, then we can send a notification either via Teams or Email. Following is what I used for the false branch.
We then evaluate the status code, 200 means a successful code and it will go into the True branch.
Let us have a look at the next steps.
In Compose FilterList, you can type the following expression. In this example, I am faced with one escape character namely \n
“@split(outputs,(‘Compose’),’\n’)”
Example if faced with a semicolon instead of escape characters.
Note - If you are facing difficulty in this step then please edit via Logic App Code.
Now, lets move on to the next set of steps,
We use a for each output control and in Compose3 we use one more Compose action in order to get rid of double quotes.
Following is how the logic app will look until For Each Loop.
If you cannot find Current item, you can manually type in the following expression.
"@item()"
Note - If you are facing difficulty in this step then please edit via Logic App Code.
Anyways, the action that follows after ForEach is as shown below.
We then run a query to check if this IoC is already existing in ThreatIntelligence table or not.
Note- We do this so that we can avoid any duplicate ingestion.
After that, we then pass the output of our query to Compose action once more.
We add a condition namely if length of value output is equal to zero which means that the Indicator of Compromise does not exist in ThreatIntelligence table.
If true, following HTTP will be used to push the IoC to the table. You can define your parameters are required here or you can use the following as a baseline.
I am leveraging the REST API provided by Microsoft Sentinel at
For the valid until field, I am using the expression addDays(utcNow(),90) in order to add an expiration date of 90 days from date and time of being ingested.
Following is the body section of the REST API, you can add or modify details as required.
JSON Code Snippet is also given below
{
"kind": "indicator",
"properties": {
"confidence": 90,
"createdByRef": "Aniket",
"description": "Indicator of Compromise extracted from Github, possible threat",
"displayName": "Github",
"externalReferences": [],
"granularMarkings": [],
"killChainPhases": [],
"labels": [],
"modified": "",
"pattern": "[ipv4-addr:value = '@{outputs('Compose_CurrentItem')}']",
"patternType": "ipv4-addr",
"revoked": false,
"source": "Github",
"threatIntelligenceTags": [
"honeypot"
],
"threatTypes": [
"Botnet"
],
"validFrom": "@{utcNow()}",
"validUntil": "@{addDays(utcNow(),90)}"
}
}Logic App in action
Before we check our run, make sure that a System Managed Identity is assigned to the logic app and that role assignment of Contributor has been provided at resource group level.
You can do this by navigating to Logic App -> Identity -> System Assigned (Enable it and click on role assignments to assign role at Resource Group level example Contributor).
Now, you can confirm the same by navigating to Microsoft Sentinel -> Logs and using the following query.
ThreatIntelligence
| where NetworkIP == ‘XXX’
Replace XXX with the Ipv4/IoC that you are looking for.
Logic App Code Reference
Following is a code snip, make sure to replace connection parameters and variables such as subscription id, resource group and resource name accordingly.
{
"definition": {
"$schema": "https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#",
"actions": {
"Condition_HTTPStatusCode": {
"actions": {
"Compose": {
"inputs": "@variables('ListOfIoc')",
"runAfter": {
"Set_variable_ListOfIoc": [
"Succeeded"
]
},
"type": "Compose"
},
"Compose_FilterList": {
"inputs": "@split(outputs('Compose'),'\n')",
"runAfter": {
"Compose": [
"Succeeded"
]
},
"type": "Compose"
},
"For_each": {
"actions": {
"Compose_CurrentItem": {
"inputs": "@item()",
"runAfter": {},
"type": "Compose"
},
"Compose_QueryOutput": {
"inputs": "@body('Run_query_and_list_results_CheckIfIocExists')?['value']",
"runAfter": {
"Run_query_and_list_results_CheckIfIocExists": [
"Succeeded"
]
},
"type": "Compose"
},
"Condition": {
"actions": {
"HTTP_PushIndicatorOfCompromise": {
"inputs": {
"authentication": {
"audience": "https://management.azure.com",
"type": "ManagedServiceIdentity"
},
"body": {
"kind": "indicator",
"properties": {
"confidence": 90,
"createdByRef": "XXX",
"description": "Indicator of Compromise extracted from Github, possible threat",
"displayName": "Github",
"externalReferences": [],
"granularMarkings": [],
"killChainPhases": [],
"labels": [],
"modified": "",
"pattern": "[ipv4-addr:value = '@{outputs('Compose_CurrentItem')}']",
"patternType": "ipv4-addr",
"revoked": false,
"source": "Github",
"threatIntelligenceTags": [
"honeypot"
],
"threatTypes": [
"Botnet"
],
"validFrom": "@{utcNow()}",
"validUntil": "@{addDays(utcNow(),90)}"
}
},
"headers": {
"Content-Type": "application/json"
},
"method": "POST",
"uri": "https://management.azure.com/subscriptions/@{variables('SubscriptionId')}/resourceGroups/@{variables('ResourceGroup')}/providers/Microsoft.OperationalInsights/workspaces/@{variables('LAW')}/providers/Microsoft.SecurityInsights/threatIntelligence/main/createIndicator?api-version=2023-02-01"
},
"runAfter": {},
"type": "Http"
}
},
"expression": {
"or": [
{
"equals": [
"@length(body('Run_query_and_list_results_CheckIfIocExists')?['value'])",
0
]
}
]
},
"runAfter": {
"Compose_QueryOutput": [
"Succeeded"
]
},
"type": "If"
},
"Run_query_and_list_results_CheckIfIocExists": {
"inputs": {
"body": "ThreatIntelligenceIndicator | where NetworkIP == '@{outputs('Compose_CurrentItem')}' or NetworkSourceIP == '@{outputs('Compose_CurrentItem')}'",
"host": {
"connection": {
"name": "@parameters('$connections')['azuremonitorlogs']['connectionId']"
}
},
"method": "post",
"path": "/queryData",
"queries": {
"resourcegroups": "@variables('ResourceGroup')",
"resourcename": "@variables('LAW')",
"resourcetype": "Log Analytics Workspace",
"subscriptions": "@variables('SubscriptionId')",
"timerange": "Last 30 days"
}
},
"runAfter": {
"Compose_CurrentItem": [
"Succeeded"
]
},
"type": "ApiConnection"
}
},
"foreach": "@outputs('Compose_FilterList')",
"runAfter": {
"Compose_FilterList": [
"Succeeded"
]
},
"type": "Foreach"
},
"Send_an_email_(V2)_Succeeded": {
"inputs": {
"body": {
"Body": "<p>Fetching Indicators of Compromise from @{variables('URL')} has succeeded with Status Code @{outputs('HTTP_FetchIndicatorOfCompromise')['statusCode']}<br>\n<br>\nNo Action required from your end, this is for informational purposes only.</p>",
"Importance": "Normal",
"Subject": "Logic App OpenSourceTI - Data Fetch Succeeded",
"To": "XXX"
},
"host": {
"connection": {
"name": "@parameters('$connections')['outlook']['connectionId']"
}
},
"method": "post",
"path": "/v2/Mail"
},
"runAfter": {
"For_each": [
"Succeeded"
]
},
"type": "ApiConnection"
},
"Set_variable_ListOfIoc": {
"inputs": {
"name": "ListOfIoc",
"value": "@{body('HTTP_FetchIndicatorOfCompromise')}"
},
"runAfter": {},
"type": "SetVariable"
}
},
"else": {
"actions": {
"Send_an_email_(V2)_Failed": {
"inputs": {
"body": {
"Body": "<p>Fetching Indicators of Compromise from @{variables('URL')} has failed with Status Code @{outputs('HTTP_FetchIndicatorOfCompromise')['statusCode']}<br>\n<br>\nPlease check whether URL is up.</p>",
"Importance": "High",
"Subject": "Logic App OpenSourceTI - Data Fetch Failed",
"To": "XXX"
},
"host": {
"connection": {
"name": "@parameters('$connections')['outlook']['connectionId']"
}
},
"method": "post",
"path": "/v2/Mail"
},
"runAfter": {},
"type": "ApiConnection"
}
}
},
"expression": {
"and": [
{
"equals": [
"@outputs('HTTP_FetchIndicatorOfCompromise')['statusCode']",
200
]
}
]
},
"runAfter": {
"HTTP_FetchIndicatorOfCompromise": [
"Succeeded"
]
},
"type": "If"
},
"HTTP_FetchIndicatorOfCompromise": {
"inputs": {
"method": "GET",
"uri": "https://raw.githubusercontent.com/stamparm/ipsum/master/levels/6.txt"
},
"runAfter": {
"Initialize_variable_URL": [
"Succeeded"
]
},
"type": "Http"
},
"Initialize_variable_ListOfIoc": {
"inputs": {
"variables": [
{
"name": "ListOfIoc",
"type": "string"
}
]
},
"runAfter": {
"Initialize_variable_LogAnalyticsWorkspace": [
"Succeeded"
]
},
"type": "InitializeVariable"
},
"Initialize_variable_LogAnalyticsWorkspace": {
"inputs": {
"variables": [
{
"name": "LAW",
"type": "string",
"value": "XXX"
}
]
},
"runAfter": {
"Initialize_variable_ResourceGroup": [
"Succeeded"
]
},
"type": "InitializeVariable"
},
"Initialize_variable_ResourceGroup": {
"inputs": {
"variables": [
{
"name": "ResourceGroup",
"type": "string",
"value": "XXX"
}
]
},
"runAfter": {
"Initialize_variable_SubscriptionId": [
"Succeeded"
]
},
"type": "InitializeVariable"
},
"Initialize_variable_SubscriptionId": {
"inputs": {
"variables": [
{
"name": "SubscriptionId",
"type": "string",
"value": "XXX"
}
]
},
"runAfter": {},
"type": "InitializeVariable"
},
"Initialize_variable_URL": {
"inputs": {
"variables": [
{
"name": "URL",
"type": "string",
"value": "https://raw.githubusercontent.com/stamparm/ipsum/master/levels/6.txt"
}
]
},
"runAfter": {
"Initialize_variable_ListOfIoc": [
"Succeeded"
]
},
"type": "InitializeVariable"
}
},
"contentVersion": "1.0.0.0",
"outputs": {},
"parameters": {
"$connections": {
"defaultValue": {},
"type": "Object"
}
},
"triggers": {
"Recurrence": {
"evaluatedRecurrence": {
"frequency": "Day",
"interval": 7
},
"recurrence": {
"frequency": "Day",
"interval": 7
},
"type": "Recurrence"
}
}
},
"parameters": {
"$connections": {
"value": {
"azuremonitorlogs": {
"connectionId": "/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Web/connections/azuremonitorlogs",
"connectionName": "azuremonitorlogs",
"id": "/subscriptions/XXX/providers/Microsoft.Web/locations/{region}/managedApis/azuremonitorlogs"
},
"outlook": {
"connectionId": "/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Web/connections/outlook",
"connectionName": "outlook",
"id": "/subscriptions/XXX/providers/Microsoft.Web/locations/{region}/managedApis/outlook"
}
}
}
}
}
Closing Thoughts
This might look complicated due to how data formatting works and parsing the data in logic app does not seem very easy right now. We will soon look at a much simpler approach using Kusto Query Language.
Stay Tuned In!