Repository navigation
Detect DGA-like patterns in HTTP URL paths (fixes #2089) - #3219
Closed
RajaMuhammadAwais wants to merge 5 commits into
Closed
RajaMuhammadAwais wants to merge 5 commits into
RajaMuhammadAwais wants to merge 5 commits into
Conversation
Some malware families (e.g. Emotet) encode DGA-generated tokens directly in the HTTP URL path instead of the hostname, bypassing the existing hostname- based DGA checks. This commit adds ndpi_check_dga_url_path(), which applies a refined version of the DGA heuristic to URL path segments: - Segments are stripped of file extensions, split into sub-tokens by '_', '-' and '.', and empty sub-tokens are ignored. - Structured identifiers are excluded: hex hashes/numeric IDs, short alphanumeric resource IDs (alpha prefix followed by hex body), short tokens (at most two non-hex letters plus digits, a single letter group after up to four leading digits, and ASIN-style digit pairs followed by letters), and purely letter-based names. - Remaining segments are flagged as DGA-like when they contain at least eight alphabetic characters with at most two vowels, or an identical- character run of eight or more, consistent with the hostname checks in ndpi_check_dga_name(). The check is invoked from the HTTP dissector when validating URLs and emits the NDPI_SUSPICIOUS_DGA_DOMAIN risk with a descriptive info string. Unit tests pass; a positive/negative test pcap pair for an Emotet-style URL-path DGA pattern is included, and existing regression pcaps remain byte-identical to the committed baselines.
Member
|
@RajaMuhammadAwais, could you sign the CLA , please? |
Contributor
Author
|
Hi Ivan, sure! I’ve signed the CLA. Thanks for letting me know. |
Member
|
@RajaMuhammadAwais, please update all unit tests results (see CI results) |
…etection (PR ntop#3219) The URL-path DGA heuristic increases risk detections in the default configuration, changing the Automa and Patricia risk mask counters: - Automa risk mask: 0/0 -> 3/3 (search/found) - Patricia risk mask: 2/0 -> 8/0 (search/found) Regenerated with the exact CI command: ndpiReader --cfg=filename.config,config.txt -A -p protos.txt -c categories.txt -r risky_domains.txt -j ja4_fingerprints.csv -S sha1_fingerprints.csv -G lists -L lists/public_suffix_list.dat -q -K JSON -k /dev/null -t -v 2 Verified: full unit test suite (do.sh) passes with 0 failures. do-dga.sh baseline metrics (Accuracy=0.69, Precision=0.97, Recall=0.40) are unaffected.
Contributor
Author
|
Thanks Ivan, done. I updated the expected unit test results for the failing starcraft_battle.pcap test (Automa risk mask 3/3, Patricia risk mask 8/0) and verified the full test suite passes locally with 0 failures. The CI Build workflow is waiting for maintainer approval to run again. |
added 3 commits
August 16, 2026 21:12
…s (issue ntop#3062) Prefixes tagged with multiple AWS services in ip-ranges.json (such as AMAZON, EC2 and S3 at the same time) were ending up in several nDPI lists, which made the protocol matchers ambiguous. This commit gives every prefix exactly one owner through a fixed precedence chain (EC2_INSTANCE_CONNECT > EC2 > S3 > CLOUDFRONT > DYNAMODB > API_GATEWAY > KINESIS_VIDEO_STREAMS > AMAZON_AWS) in the new ndpi_aws_ip_dedup.py, and it adds ndpi_aws_prune_overlaps.py, a post-processor that removes networks in lower-precedence lists when they are fully covered by a higher-precedence list. The pruning matters because netaddr.cidr_merge widens adjacent prefixes into supernets that can swallow entries kept by another list (for example a merged CloudFront /19 covering the generic list's /23). Both helpers are wired into utils/aws_ip_addresses_download.sh, so regenerating the include files stays a single script invocation. Verified locally: zero duplicate entries across the seven generated include files, unit test suite passes, and shell and Python syntax checks are clean. The deduplicated service lists shrink because many API Gateway and Kinesis prefixes in the AWS data are themselves contained inside EC2 and S3 supernets, so those address ranges keep matching through the owning higher-precedence detector.
SonarCloud flagged the CLI-supplied paths in ndpi_aws_ip_dedup.py and ndpi_aws_prune_overlaps.py as a path traversal risk on new code. resolve_path() now resolves every CLI-provided path with os.path.realpath() and refuses anything that escapes the working directory, and the per-service paths built by the pruner are checked to remain inside the merged directory before they are read or rewritten. Behaviour is unchanged for the supported invocation from aws_ip_addresses_download.sh. Verified locally: the regeneration still ends with 0 duplicate entries across the seven generated include files.
The pruner only ever accepts the suffix values produced by the
pipeline itself ('' for the dedup output and 'm' for the merged
output), so the suffix argument is now validated against that fixed
set before it can reach the filesystem. The constructed per-service
paths are resolved and checked against the merged directory base
before any open(), so a hostile merged_dir argument can no longer
escape the intended directory.
The dedup helper now resolves and bounds both CLI paths (input JSON
and output directory) before the first open() or makedirs() call.
Verified locally: the full regeneration still ends with 0 duplicate
entries across the seven generated include files.
|
RajaMuhammadAwais
added a commit
to RajaMuhammadAwais/nDPI
that referenced
this pull request
Aug 18, 2026
…etection (PR ntop#3219) The URL-path DGA heuristic increases risk detections in the default configuration, changing the Automa and Patricia risk mask counters: - Automa risk mask: 0/0 -> 3/3 (search/found) - Patricia risk mask: 2/0 -> 8/0 (search/found) Regenerated with the exact CI command: ndpiReader --cfg=filename.config,config.txt -A -p protos.txt -c categories.txt -r risky_domains.txt -j ja4_fingerprints.csv -S sha1_fingerprints.csv -G lists -L lists/public_suffix_list.dat -q -K JSON -k /dev/null -t -v 2 Verified: full unit test suite (do.sh) passes with 0 failures. do-dga.sh baseline metrics (Accuracy=0.69, Precision=0.97, Recall=0.40) are unaffected.
Member
|
New version available in #3228 |
Contributor
Author
|
fixed #3228 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Some malware families (e.g. Emotet) encode DGA-generated tokens directly in the HTTP URL path instead of the hostname, bypassing the existing hostname- based DGA checks. This commit adds ndpi_check_dga_url_path(), which applies a refined version of the DGA heuristic to URL path segments:
The check is invoked from the HTTP dissector when validating URLs and emits the NDPI_SUSPICIOUS_DGA_DOMAIN risk with a descriptive info string. Unit tests pass; a positive/negative test pcap pair for an Emotet-style URL-path DGA pattern is included, and existing regression pcaps remain byte-identical to the committed baselines.
Please sign (check) the below before submitting the Pull Request:
Link to the related issue:
Describe changes: