[FR] [DAC] Initial Yaml Support

[eric-forte-elastic]

Pull Request

Issue link(s):

Summary - What I changed

Exploring adding yaml output support when exporting rules either from repo or form Kibana. This may be useful in supporting Terraform as it cannot directly pull detection contents from a stack. In a DaC implementation, Terraform will have a major gap in this way. However, adding this means that Terraform can be used in full CI/CD sync workflows enabling customers/community members who use Terraform to also use it for bidirectional DaC.

How To Test

Test Example Commands:

python -m detection_rules export-rules-from-repo -e -ac -syd yaml_output/

python -m detection_rules kibana --space dac-dev-test export-rules -e -ac -sv -sy -d yaml_output/

Checklist

• Added a label for the type of pr: bug, enhancement, schema, maintenance, Rule: New, Rule: Deprecation, Rule: Tuning, Hunt: New, or Hunt: Tuning so guidelines can be generated
• Added the meta:rapid-merge label if planning to merge within 24 hours
• Secret and sensitive material has been managed correctly
• Automated testing was updated or added to match the most common scenarios
• Documentation and comments were added for features that require explanation

Contributor checklist

• Have you signed the contributor license agreement?
• Have you followed the contributor guidelines?

[github-actions]

Enhancement - Guidelines

These guidelines serve as a reminder set of considerations when addressing adding a feature to the code.

Documentation and Context

• Describe the feature enhancement in detail (alternative solutions, description of the solution, etc.) if not already documented in an issue.
• Include additional context or screenshots.
• Ensure the enhancement includes necessary updates to the documentation and versioning.

Code Standards and Practices

• Code follows established design patterns within the repo and avoids duplication.
• Ensure that the code is modular and reusable where applicable.

Testing

• New unit tests have been added to cover the enhancement.
• Existing unit tests have been updated to reflect the changes.
• Provide evidence of testing and validating the enhancement (e.g., test logs, screenshots).
• Validate that any rules affected by the enhancement are correctly updated.
• Ensure that performance is not negatively impacted by the changes.
• Verify that any release artifacts are properly generated and tested.
• Conducted system testing, including fleet, import, and create APIs (e.g., run make test-cli, make test-remote-cli, make test-hunting-cli)

Additional Checks

• Verify that the enhancement works across all relevant environments (e.g., different OS versions).
• Confirm that the proper version label is applied to the PR patch, minor, major.

[eric-forte-elastic]

Based on testing done by Marc-Antoine Leclercq this PR is ready for review. The yaml format output is compatible for use in custom tooling around https://github.com/elastic/terraform-provider-elasticstack.

[Mikaayenson]

a couple nits. otherwise lgtm

CLI Testing

detection_dev) ➜  detection-rules git:(yaml_support) ✗ uv run python -m detection_rules export-rules-from-repo -f rules/linux/defense_evasion_base64_decoding_activity.toml --save-yaml-dir ./mikas_folder/yaml_export_test && ls -la ./mikas_folder/yaml_export_test && wc -c ./mikas_folder/yaml_export_test/*.yaml && head -n 30 ./mikas_folder/yaml_export_test/unusual_base64_encoding_decoding_activity.yaml

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

WARNING: Rule path does not match required path: defense_evasion_base64_decoding_activity.toml != defense_evasion_unusual_base64_encoding_decoding_activity.toml
Exported 1 rules into mikas_folder/yaml_export_test
total 24
drwxr-xr-x@  3 stryker  staff     96 Apr  9 13:11 .
drwxr-xr-x  11 stryker  staff    352 Apr  9 13:11 ..
-rw-r--r--@  1 stryker  staff  11355 Apr  9 13:11 unusual_base64_encoding_decoding_activity.yaml
   11355 ./mikas_folder/yaml_export_test/unusual_base64_encoding_decoding_activity.yaml
author:
- Elastic
description: This rule leverages ESQL to detect unusual base64 encoding/decoding activity
  on Linux systems. Attackers may use base64 encoding/decoding to obfuscate data,
  such as command and control traffic or payloads, to evade detection by host- or
  network-based security controls. ESQL rules have limited fields available in its
  alert documents. Make sure to review the original documents to aid in the investigation
  of this alert.
from: now-61m
interval: 1h
language: esql
license: Elastic License v2
name: Unusual Base64 Encoding/Decoding Activity
note: '## Triage and analysis


  > **Disclaimer**:

  > This investigation guide was created using generative AI technology and has been
  reviewed to improve its accuracy and relevance. While every effort has been made
  to ensure its quality, we recommend validating the content and adapting it to suit
  your specific environment and operational needs.


  ### Investigating Unusual Base64 Encoding/Decoding Activity

  Base64 encoding is a method to convert binary data into ASCII text, often used for
  data transmission. Adversaries exploit this to obfuscate malicious payloads or commands,
  bypassing security controls. The detection rule identifies suspicious Base64 activity
  on Linux by monitoring specific processes and command patterns, flagging anomalies
(detection_dev) ➜  detection-rules git:(yaml_support) ✗ 

[eric-forte-elastic]

Additional testing for new flag changes:

❯ python -m detection_rules kibana --space test export-rules -e -ac -sv -sy -d yaml_output/
Loaded config file: /home/forteea1/tmp/tmp/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

1 results exported
1 rules converted
0 exceptions exported
0 action connectors exported
1 rules saved to yaml_output
0 exception lists saved to /home/forteea1/tmp/tmp/detection-rules/custom_rules/exceptions
0 action connectors saved to /home/forteea1/tmp/tmp/detection-rules/custom_rules/action_connectors

[shashank-elastic]

Post code optimisations, this looks much cleaner 👍