Introduction – What are Log Analytic Search Jobs ?

Azure log analytics search jobs enable you to query log analytics data across long timespans and across huge datasets. It includes the logs available for interactive querying as well as those in retention. Usefulness of search jobs  – examples:

• Audit & Compliance Logs -> Compliance checks over several months. Search for user activities, privileged identity management logs or Azure policy violations over long periods of time
• Security & Incident Investigation -> analyze past attacks or anomalies
• Application & Performance Logs -> analyze resource utilisation  over time
• Custom Logs ..

How do Search Jobs work and what seperates them from Restore

Search Job Queries are executed asynchronously. (The default log analytic query timeout of 10 minutes doesnt apply. Supports long running searches up to a 24-hour time-out.) The results are written into a new table. Search Jobs allow you to access data that is beyond the ‘retention period’. (Search date range is up to one year.) The retention period varies depending on your configuration and can be set between 4 days and 2 years (costly $$$). https://learn.microsoft.com/en-us/azure/azure-monitor/logs/data-retention-configure
‘Search jobs’ and ‘Restore’ are two seperate functions. Restore is used to move data from the cold tier to the hot tier. Search jobs access data that match a specific criteria.

Hands-on: Using Azure Log Analytics Search Jobs Effectively

👉 Suppose a VM has been compromised and has established unauthorised connections to the Internet. To analyse the incident, we use the network and application rule logs of the Azure firewall that protects the VNET. The aim is to find out how long the attack was active.
To be able to determine how long the attack lasted, the occurances of a certain communication shall be traced in the logs. The initial search job query should be as specific as possible, in order to minimize query time and result data that has to be ingested(cost). Initially we want to extract all relevant communiction info for the specific host. We do know the source IP of the VM. We can exclude ICMP traffic and we can exclude FlowTraceLogs as they do not include the URL info. Example of basic search job query:

AzureDiagnostics
| where ResourceType == "AZUREFIREWALLS"
| where AdditionalFields.SourceIp == "10.10.10.10"
| where AdditionalFields.Protocol <> "ICMP Type=8"
| where Category <> "AZFWFlowTrace"


In the Log Analytic Workspace that contains the Firewall Logs open the Query UI for the table.   In (1.) we do enable the “search job” Option. This will make the extended duration in (2.) available. Select a Year.

It is also possible to specify the duration in the query. Once the search job is finished the result table is available under “Search results” in the tablelisting. (While the Search jobs is being exectued the result table can be queried after a while.)

After the search job is completed locate the result table in the “Search results” Folder
Further research has shown that the compromised Host accessed URLs that matches a pattern http://IP/filestreamingservice/.. We query the result table for that string and extract two columns of relevance.

IPtrace_SRCH
| where AdditionalFields.ActionReason contains "filestreaming"
| project _OriginalTimeGenerated, AdditionalFields.ActionReason


The result shows that the suspicous activity occurred from 26th September to 17th November. This information will help to narrow down further analysis timewise.

Things to consider when using Azure Log Analytic Search Jobs

⚠️ Important restrictions for search jobs:

❌ Omit using contains as operator, it will not be objected but the query might run endlessly.
✅ Allowed Operators: where, extend, project, project-away, project-keep, project-rename, project-reorder, parse, parse-where – compute intensive functions are excluded
⏳ Maximum results: A search job returns a maximum of 1 million data records.
🔄 limited simulatenous queries: concurrent execution is limited to five search jobs per workspace.

So what do you think about Search Jobs – Do you have any practical application or performance tweaks ?Let’s discuss in the comments below or connect on LinkedIn to exchange ideas!