cannot fulfill requested CSI volume mounts on 1 node

[masterjg]

Hey guys,

So I was trying to setup Docker Swarm cluster with CSI plugin (ghcr.io/cloud-skeleton/democratic-csi-swarm).

I have installed it on all nodes:

ID             NAME                          DESCRIPTION                              ENABLED
2a8d168b768a   democratic-csi-swarm:latest   Democratic CSI plugin for Docker Swarm   true

On manager node I can create volume directly with cli:

docker volume create \
  -d democratic-csi-swarm \
  --group system \
  --label eu.cloudskeleton.volume=true \
  --label eu.cloudskeleton.volume.type=system \
  --scope multi \
  --sharing all \
  --required-bytes 10737418240 \
  --topology-required eu.cloudskeleton.node=true \
    proxy

and all's good, HOWEVER I have tried everything and I can not achieve same result with compose.yml:

services:
  test:
    ...
    volumes:
      - source: proxy
        target: /data/traefik/configs
        type: cluster

volumes:
  proxy:
    driver: democratic-csi-swarm
    driver_opts:
      group: system
      required-bytes: 10737418240
      scope: multi
      sharing: all
      topology-required: eu.cloudskeleton.node=true
    labels:
      eu.cloudskeleton.volume: "true"
      eu.cloudskeleton.volume.type: system

Deployment would always fail with:

cannot fulfill requested CSI volume mounts on 1 node

[masterjg]

It looks like Docker doesn't even TRY to create volume for some reason

[ppignet]

As far as I understand the code, there is no translation between compose-defined cluster volume definitions and actual cluster volumes as of yet.

Service-level volume entries can hold a reference to a cluster volume by using the type cluster, but inlining cluster volume options is left out: https://github.com/docker/cli/blob/7b4cde69673ee7c1d6929749d1a639d748749804/internal/volumespec/types.go#L38-L40

Top-level volume definitions do have a key that is related to cluster volume options (x-cluster-spec, https://github.com/docker/cli/blob/7b4cde69673ee7c1d6929749d1a639d748749804/cli/compose/types/types.go#L478-L486), but specifying that in compose is explicitly ignored by the extensions syntax rule: https://docs.docker.com/reference/compose-file/extension/

So it looks like cluster volumes can neither be managed or used through compose files.

In any case, I believe that passing cluster volume options through regular driver_opts is not correct.
The "technically correct according to the source, but not-yet-implemented" syntax would be:

volumes:
  my-volume:
    driver: my-csi-driver
    x-cluster-spec:
      group: <group>
      access_mode:
        scope: <scope>
        sharing: <sharing>
        mount_volume:
          fs_type: <fstype>
      capacity_range:
        required_bytes: <req_bytes>
        limit_bytes: <limit_bytes>
      secrets:
        - key: <some_key>
          secret: <some_docker_secret_id>
      #availability: "active"    # not sure if needed in spec or only at runtime
    driver_opts:
      driver_specific_opt_A: <some_value>
      driver_specific_opt_B: <some_value>

...and then to use it:

 services:
  my-service:
    ...
    volumes:
      - type: cluster
        source: <cluster_volume_name_or_id>
        target: "/target/path/in/container"