feat(unstructured): add Transform MCP tool

[SudSampath]

Summary

• add a Transform Document tool to the existing official Unstructured plugin
• accept Dify files or public URLs and return Markdown, JSON, HTML, or text
• support parsing, OCR/enrichment, chunking, and embedding for RAG preprocessing
• stream result downloads with a documented 50 MB inline limit and durable output references
• preserve the existing Partition tool for legacy and self-hosted deployments

Unstructured Transform converts PDFs, DOCX, PPTX, HTML, images, and scanned pages into clean structured output.

Marketplace behavior

This updates the existing langgenius/unstructured Marketplace plugin. Once approved and merged, Dify's repository workflow packages and uploads version 0.0.10 to the Marketplace. Users will discover and install it from the existing Unstructured listing, then authenticate at runtime with their own Unstructured account and API key. No shared Unstructured credential is included in the plugin.

Documentation

• Unstructured Dify installation guide

Validation

• uv run pytest -q tests/test_transform.py (25 passed)
• uvx ruff check tools/transform.py tests/test_transform.py
• uvx black --check tools/transform.py tests/test_transform.py
• YAML parsing for the manifest, provider, and tool definitions
• git diff --check
• dify plugin package ./unstructured
• authenticated smoke test against https://mcp.transform.unstructured.io: credential validation, signed upload, start_transform_job, check_job_status, get_job_results, and Markdown result download all passed with a synthetic document
• the test credential was kept private, removed after validation, and is not included in the plugin

[Copilot]

Pull request overview

Adds the hosted Unstructured Transform MCP workflow while retaining the legacy Partition tool.

Changes:

• Adds document transformation with multiple output formats and RAG preprocessing.
• Expands provider credentials, documentation, and privacy disclosures.
• Adds MCP dependencies and focused helper tests.

Reviewed changes

Copilot reviewed 11 out of 12 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
.difyignore	Excludes tests from packages.
PRIVACY.md	Documents data handling.
README.md	Documents Transform and Partition tools.
main.py	Extends request timeout.
manifest.yaml	Releases version 0.0.10.
provider/unstructured.py	Routes credential validation.
provider/unstructured.yaml	Registers Transform credentials and tool.
pyproject.toml	Adds MCP and test dependencies.
tests/test_transform.py	Tests payload and stage helpers.
tools/transform.py	Implements the Transform workflow.
tools/transform.yaml	Defines the Transform interface.
uv.lock	Locks added dependencies.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[crazywoola]

Please resolve the comments if you think it's irrelvant.

[SudSampath]

Thanks! I addressed all six review comments in 000a715 and resolved the threads. The update enforces HTTPS, validates the full four-tool MCP protocol, redacts signed transfer URLs, retries the job_not_complete consistency window, and prevents Partition from using Transform credentials. Validation: 9 tests pass, Ruff format/lint pass, YAML parses, and the plugin packages successfully.

[Copilot]

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

tools/unstructured/tools/transform.py:24

• The hosted endpoint's current canonical integration reference (updated July 20) lists the live tools as start_transform_job, check_job_status, and get_job_results, not these three legacy names. Requiring the legacy set makes credential validation reject the advertised endpoint before this tool can run, and the invocation sites below also call the stale protocol. Update the required names and adapt the corresponding request/response handling to the current schemas.

_REQUIRED_TRANSFORM_TOOLS = {
    "request_file_upload_url",
    "transform_files",
    "check_transform_status",
    "get_transform_results",
}

[SudSampath]

Latest update is in e56a388:

• synced the plugin to the live hosted MCP names (start_transform_job, check_job_status, and get_job_results)
• resolved the output decoding comment and added the 50 MB local-file guard
• added focused local-file and public-URL orchestration tests (12 tests passing)
• ran a private end-to-end smoke test against the hosted Transform MCP: credential validation, signed upload, job polling, results, and Markdown download all passed
• updated the matching Unstructured Dify guide

The test key stayed private and was removed after the run. Would appreciate another look when you have a chance.

[Copilot]

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated 11 comments.

Comments suppressed due to low confidence (8)

tools/unstructured/tools/transform.yaml:39

• The Portuguese word is missing its diacritic: publica should be pública.

      pt_BR: URL publica do arquivo

tools/unstructured/tools/transform.yaml:44

• The Portuguese word is missing its diacritic: publica should be pública.

      pt_BR: Uma URL HTTP(S) publica do documento. Informe este campo ou um arquivo.

tools/unstructured/tools/transform.yaml:89

• The Portuguese copy is missing diacritics in Estrategia de analise.

      pt_BR: Estrategia de analise. Use hi_res para enriquecimento e documentos digitalizados ou com layout complexo.

tools/unstructured/tools/transform.yaml:132

• The Portuguese word is missing its diacritic: virgula should be vírgula.

      pt_BR: Enriquecimentos opcionais separados por virgula, como table_to_html,image_description,ner.

tools/unstructured/tools/transform.yaml:142

• The Portuguese word is missing its diacritic: Estrategia should be Estratégia, and fragmentacao should be fragmentação.

      pt_BR: Estrategia de fragmentacao

tools/unstructured/tools/transform.yaml:177

• The Portuguese copy is missing diacritics in maximo, estrategia, and e.

      pt_BR: Tamanho maximo do fragmento quando uma estrategia e selecionada.

tools/unstructured/tools/transform.yaml:193

• The Portuguese word is missing its diacritic: estrategia should be estratégia, and fragmentacao should be fragmentação.

      pt_BR: Anexe embeddings para RAG. Recomendado com uma estrategia de fragmentacao.

tools/unstructured/tools/transform.py:147

• The DNS preflight does not constrain the later HTTPX connection: HTTPX resolves the hostname again for the actual request. A rebinding/alternating DNS record can return a global address here and an internal address on the second lookup, bypassing this SSRF guard. Pin the validated address in the transport while preserving TLS SNI/Host, or restrict signed transfers to an allowlist of trusted storage domains.

            resolved = await asyncio.to_thread(
                socket.getaddrinfo,
                hostname,
                port or 443,
                type=socket.SOCK_STREAM,
            )

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[SudSampath]

Reviewed Copilot's suppressed DNS-rebinding suggestion as well. I am not pinning resolved IPs or adding a storage-domain allowlist in this plugin: signed transfer URLs are issued by the locked hosted Transform MCP endpoint rather than supplied by the end user, redirects are disabled, and the existing global-address check remains defense in depth. IP pinning would require custom TLS/SNI transport behavior, while an allowlist would be brittle without a documented stable storage-domain contract. The hosted service remains the authoritative security boundary for source-URL fetching.

[SudSampath]

@crazywoola All review threads are addressed and resolved in the latest head (42144a4). The focused suite now has 25 passing tests, and lint, YAML validation, and plugin packaging all pass. Could you take another look when you have a chance?

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Copilot]

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated no new comments.

Suppressed comments (1)

tools/unstructured/manifest.yaml:35

• created_at is the plugin's original creation timestamp, not the release timestamp. Previous Unstructured releases (including 0.0.7 and 0.0.9) retained 2025-04-23T17:46:21.256715+08:00 while only incrementing version; replacing it here rewrites the Marketplace metadata for an existing listing. Please preserve the original timestamp.

created_at: 2026-07-14T00:00:00.000000+00:00