# Introduction (/docs)

Veloxpack is designed as a modular ecosystem for efficient video workflows, offering several core components that work together to handle everything from data transfer to orchestration.

<Cards>
  <Card title="CSI Rclone Driver Plugin" href="/docs/csi-driver-rclone">
    Integrates Rclone as a storage driver for flexible and efficient data movement across various backends enabling seamless interaction with cloud and local storage systems.
  </Card>

  <Card title="Core Operator" href="/docs/core-operator">
    Manages and orchestrates the core services of Veloxpack, ensuring high availability, scalability, and automation for complex media processing pipelines.
  </Card>
</Cards>


# Introduction (/docs/core-operator)

## Overview

Content coming soon.


# Development (/docs/csi-driver-rclone/development)

## Prerequisites

* Go 1.21+
* Kubernetes 1.20+
* Docker or compatible container runtime
* kubectl configured
* FUSE support (for local testing)
* [Skaffold](https://skaffold.dev/) (recommended for development)

## Repository Structure

```
csi-driver-rclone/
├── cmd/rcloneplugin/          # Main application entry point
├── pkg/rclone/                # Core driver implementation
│   ├── server.go              # gRPC server setup
│   ├── identityserver.go      # CSI Identity service
│   ├── controllerserver.go    # CSI Controller service
│   ├── nodeserver.go          # CSI Node service
│   ├── rclone.go              # Driver initialization
│   ├── utils.go               # Utility functions
│   ├── version.go             # Version information
│   └── mount_options_mapper.go # Mount options parsing
├── deploy/                    # Kubernetes manifests
├── charts/                    # Helm charts
├── test/                      # Test suites
└── docs/                      # Documentation
```

## Quick Development with Skaffold

For rapid development, [Skaffold](https://skaffold.dev/) is the recommended approach as it automatically rebuilds and deploys to your local cluster whenever you make changes in code.

```bash
# Clone repository
git clone https://github.com/veloxpack/csi-driver-rclone.git
cd csi-driver-rclone

# Download and tidy dependencies
go mod tidy

# Start development mode with auto-rebuild and deploy
skaffold dev
```

Skaffold will:

* Watch for code changes
* Automatically rebuild the container image
* Deploy to your local Kubernetes cluster
* Stream logs from all components
* Clean up resources when you stop (Ctrl+C)

## Building from Source

```bash
# Clone repository
git clone https://github.com/veloxpack/csi-driver-rclone.git
cd csi-driver-rclone

# Download and tidy dependencies
go mod tidy

# Build binary
make build

# Build container image
make container

# (optional) Push to registry
make push
```

## Development Workflow

### 1. Local Development

```bash
# Make code changes
vim pkg/rclone/nodeserver.go

# Run tests
make test

# Run linter
make lint

# Build and test
make build
make container
```

### 2. Testing with csc Tool

Install the CSI test client:

```bash
go install github.com/rexray/gocsi/csc@latest
```

Test the driver locally:

```bash
# Build and run driver locally
make build
./bin/rcloneplugin --endpoint unix:///tmp/csi.sock --nodeid CSINode -v=5

# In another terminal, test with csc
export cap="1,mount,"
export volname="test-$(date +%s)"
export volsize="2147483648"
export endpoint="unix:///tmp/csi.sock"
export target_path="/tmp/targetpath"
export params="remote=s3,remotePath=test-bucket,configData=[s3]\ntype = s3\nprovider = Minio\nendpoint = http://localhost:9000\naccess_key_id = minioadmin\nsecret_access_key = minioadmin"

# Test operations
csc identity plugin-info --endpoint "$endpoint"
csc controller new --endpoint "$endpoint" --cap "$cap" "$volname" --req-bytes "$volsize" --params "$params"
csc node publish --endpoint "$endpoint" --cap "$cap" --vol-context "$params" --target-path "$target_path" "$volumeid"
csc node unpublish --endpoint "$endpoint" --target-path "$target_path" "$volumeid"
csc controller del --endpoint "$endpoint" "$volumeid"
```

### 3. Deploy to Cluster

```bash
# Update image references
export REGISTRY=your-registry.com
export IMAGE_VERSION=latest
sed -i "s|image: .*|image: ${REGISTRY}/csi-rclone:${IMAGE_VERSION}|g" deploy/csi-rclone-controller.yaml
sed -i "s|image: .*|image: ${REGISTRY}/csi-rclone:${IMAGE_VERSION}|g" deploy/csi-rclone-node.yaml

# Deploy to cluster
kubectl apply -k deploy/
```

## Testing

### Unit Tests

```bash
# Run all tests
go test ./pkg/rclone/...

# Run specific test
go test ./pkg/rclone -run TestNodePublishVolume

# Run with verbose output
go test -v ./pkg/rclone/...
```

### Integration Tests

```bash
# Run integration tests
go test ./test/integration/...

# Run e2e tests
go test ./test/e2e/...
```

### Linting

```bash
# Run linter
./bin/golangci-lint run --config .golangci.yml ./...

# Fix linting issues
./bin/golangci-lint run --config .golangci.yml --fix ./...
```

## Debugging

### Debug Commands

```bash
# Check driver pods
kubectl get pods -n veloxpack -l app=csi-rclone-controller
kubectl get pods -n veloxpack -l app=csi-rclone-node

# Tail logs
kubectl logs -l app=csi-rclone-controller -f -n veloxpack
kubectl logs -l app=csi-rclone-node -f -n veloxpack

# Check CSIDriver
kubectl get csidriver rclone.csi.veloxpack.io

# Check events
kubectl get events --sort-by=.metadata.creationTimestamp

# Check mount points
kubectl exec -n veloxpack -l app=csi-rclone-node -- mount | grep rclone
```

## Architecture Details

### Driver Components

1. **Identity Server**: Implements CSI Identity service
   * `GetPluginInfo`: Returns driver name and version
   * `Probe`: Health check endpoint
   * `GetPluginCapabilities`: Reports supported capabilities

2. **Controller Server**: Implements CSI Controller service
   * `CreateVolume`: Validates parameters and creates volume context
   * `DeleteVolume`: No-op (rclone doesn't require cleanup)
   * `ValidateVolumeCapabilities`: Validates volume capabilities

3. **Node Server**: Implements CSI Node service
   * `NodePublishVolume`: Mounts rclone filesystem using FUSE
   * `NodeUnpublishVolume`: Unmounts and cleans up
   * `NodeGetCapabilities`: Reports node capabilities

### Key Design Decisions

1. **No Staging**: Rclone volumes don't require staging
2. **Direct Rclone Integration**: Uses rclone's Go library directly
3. **Remote Creation**: Creates temporary remotes for each mount
4. **VFS Caching**: Leverages rclone's VFS for improved performance
5. **Template Variable Support**: Dynamic path substitution using PVC/PV metadata

### Mount Process

1. **Parameter Processing**: Loads secrets, merges with volume context
2. **Config Parsing**: Parses INI format configData if provided
3. **Remote Creation**: Creates temporary rclone remote configuration
4. **Filesystem Initialization**: Initializes rclone filesystem
5. **Mount Options Parsing**: Converts Kubernetes mount options to rclone options
6. **FUSE Mount**: Mounts using rclone's FUSE implementation

For a complete list of available mount options, see the [rclone mount documentation](https://rclone.org/commands/rclone_mount/).

## Contributing

### Code Style

* Follow Go conventions
* Use `golangci-lint` for linting
* Add tests for new functionality
* Update documentation

### Pull Request Process

1. Fork the repository
2. Create a feature branch
3. Make changes and add tests
4. Run tests and linter
5. Submit pull request

### Commit Messages

Use conventional commit format:

```
feat: add support for new storage backend
fix: resolve mount option parsing issue
docs: update installation guide
test: add integration tests for S3
```

## Performance Considerations

### VFS Cache Configuration

```yaml
mountOptions:
  - vfs-cache-mode=writes          # Cache writes for better performance
  - vfs-cache-max-size=10G         # Limit cache size
  - vfs-cache-max-age=1h           # Cache expiration
  - dir-cache-time=30s             # Directory cache time
```

### Resource Limits

```yaml
resources:
  requests:
    memory: "256Mi"
    cpu: "100m"
  limits:
    memory: "1Gi"
    cpu: "1000m"
```

### Monitoring

Monitor these metrics:

* Driver pod resource usage
* Mount/unmount operation times
* VFS cache hit rates
* Storage backend latency

## Troubleshooting

### Common Issues

1. **Build failures**: Check Go version and dependencies
2. **Image push failures**: Verify registry credentials
3. **Driver won't start**: Check FUSE installation and permissions
4. **Volume mount fails**: Verify rclone configuration and network connectivity

### Debug Commands

```bash
# Check build logs
make build 2>&1 | tee build.log

# Check container logs
docker logs csi-rclone-container

# Check Kubernetes logs
kubectl logs -l app=csi-rclone-controller --tail=100
kubectl logs -l app=csi-rclone-node --tail=100

# Check events
kubectl get events --sort-by=.metadata.creationTimestamp
```

## Uninstallation

### Remove Development Installation

```bash
# Delete all resources
kubectl delete -k deploy/

# Or delete individually
kubectl delete -f deploy/csi-rclone-driverinfo.yaml
kubectl delete -f deploy/csi-rclone-node.yaml
kubectl delete -f deploy/csi-rclone-controller.yaml
kubectl delete -f deploy/rbac-csi-rclone.yaml
kubectl delete -f deploy/namespace-csi-rclone.yaml
```

### Clean Up Local Development

```bash
# Stop local driver
pkill -f rcloneplugin

# Clean up test files
rm -rf /tmp/targetpath
rm -f /tmp/csi.sock
```


# Driver Parameters (/docs/csi-driver-rclone/driver-parameters)

## Overview

The rclone CSI driver supports various parameters for configuring storage backends, mount options, and VFS caching. Parameters can be specified in StorageClass, PersistentVolume, or Kubernetes Secrets.

## Core Parameters

### Required Parameters

| Parameter    | Description                    | Example                  | Required |
| ------------ | ------------------------------ | ------------------------ | -------- |
| `remote`     | Rclone remote name             | `s3`, `gcs`, `azureblob` | Yes      |
| `remotePath` | Path within the remote storage | `my-bucket`, `/data`     | Yes      |

### Optional Parameters

| Parameter    | Description                              | Example           | Required |
| ------------ | ---------------------------------------- | ----------------- | -------- |
| `configData` | Inline rclone configuration (INI format) | `[s3]\ntype = s3` | No       |

## Template Variables

The driver supports dynamic path substitution using template variables:

| Variable                    | Description   | Example            |
| --------------------------- | ------------- | ------------------ |
| `${pvc.metadata.name}`      | PVC name      | `my-pvc-12345`     |
| `${pvc.metadata.namespace}` | PVC namespace | `default`          |
| `${pv.metadata.name}`       | PV name       | `pv-rclone-abc123` |

### Example Usage

```yaml
parameters:
  remote: "s3"
  remotePath: "buckets/${pvc.metadata.namespace}/${pvc.metadata.name}"
```

## Mount Options

Mount options are specified in `mountOptions` field of StorageClass or PersistentVolume. For a complete list of available mount options, see the [rclone mount documentation](https://rclone.org/commands/rclone_mount/).

### VFS Cache Options

| Option                     | Description                | Values                             | Example                       |
| -------------------------- | -------------------------- | ---------------------------------- | ----------------------------- |
| `vfs-cache-mode`           | Cache mode                 | `off`, `minimal`, `writes`, `full` | `vfs-cache-mode=writes`       |
| `vfs-cache-max-size`       | Maximum cache size         | Size with unit                     | `vfs-cache-max-size=10G`      |
| `vfs-cache-max-age`        | Max time since last access | Duration                           | `vfs-cache-max-age=1h`        |
| `vfs-cache-min-free-space` | Minimum free space         | Size with unit                     | `vfs-cache-min-free-space=1G` |
| `dir-cache-time`           | Directory cache time       | Duration                           | `dir-cache-time=30s`          |

### Mount Flags

| Option            | Description                             | Type    | Example           |
| ----------------- | --------------------------------------- | ------- | ----------------- |
| `debug-fuse`      | Debug FUSE internals                    | Boolean | `debug-fuse`      |
| `allow-other`     | Allow access to other users             | Boolean | `allow-other`     |
| `allow-non-empty` | Allow mounting over non-empty directory | Boolean | `allow-non-empty` |
| `async-read`      | Use asynchronous reads                  | Boolean | `async-read`      |
| `direct-io`       | Use Direct IO (disables caching)        | Boolean | `direct-io`       |
| `read-only`       | Read-only access                        | Boolean | `read-only`       |

### VFS Options

| Option                 | Description                        | Type    | Example                |
| ---------------------- | ---------------------------------- | ------- | ---------------------- |
| `vfs-case-insensitive` | Case insensitive file matching     | Boolean | `vfs-case-insensitive` |
| `vfs-links`            | Translate symlinks                 | Boolean | `vfs-links`            |
| `vfs-refresh`          | Refresh directory cache on start   | Boolean | `vfs-refresh`          |
| `no-seek`              | Don't allow seeking in files       | Boolean | `no-seek`              |
| `no-modtime`           | Don't read/write modification time | Boolean | `no-modtime`           |
| `no-checksum`          | Don't compare checksums            | Boolean | `no-checksum`          |

## Backend-Specific Parameters

### Amazon S3

| Parameter           | Description       | Example                                    |
| ------------------- | ----------------- | ------------------------------------------ |
| `type`              | Backend type      | `s3`                                       |
| `provider`          | S3 provider       | `AWS`, `Minio`, `DigitalOcean`             |
| `access_key_id`     | Access key ID     | `AKIAIOSFODNN7EXAMPLE`                     |
| `secret_access_key` | Secret access key | `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY` |
| `region`            | AWS region        | `us-east-1`                                |
| `endpoint`          | Custom endpoint   | `https://s3.amazonaws.com`                 |

### Google Cloud Storage

| Parameter                     | Description               | Example                          |
| ----------------------------- | ------------------------- | -------------------------------- |
| `type`                        | Backend type              | `google cloud storage`           |
| `project_number`              | GCP project number        | `12345678`                       |
| `service_account_file`        | Service account file path | `/path/to/service-account.json`  |
| `service_account_credentials` | Service account JSON      | `{"type":"service_account",...}` |

### Azure Blob Storage

| Parameter  | Description          | Example                                           |
| ---------- | -------------------- | ------------------------------------------------- |
| `type`     | Backend type         | `azureblob`                                       |
| `account`  | Storage account name | `mystorageaccount`                                |
| `key`      | Storage account key  | `base64encodedkey`                                |
| `endpoint` | Custom endpoint      | `https://mystorageaccount.blob.core.windows.net/` |

## Parameter Processing

The driver processes parameters in this order:

1. **Secrets**: Loaded as defaults from `csi.storage.k8s.io/node-publish-secret-name`
2. **Volume Context**: Overrides secrets (from StorageClass parameters or PV volumeAttributes)
3. **ConfigData**: Parsed INI format and merged with other parameters
4. **Parameter Sanitization**: Remote prefixes removed, hyphens converted to underscores

### Parameter Sanitization

Parameters are sanitized for consistency:

* `s3-endpoint` → `endpoint` (when remote is "s3")
* `--cache-mode` → `cache_mode`
* `EndPoint` → `endpoint`

## CSI Parameters

| Parameter                                          | Description                 | Example         |
| -------------------------------------------------- | --------------------------- | --------------- |
| `csi.storage.k8s.io/node-publish-secret-name`      | Secret name for credentials | `rclone-secret` |
| `csi.storage.k8s.io/node-publish-secret-namespace` | Secret namespace            | `default`       |

## Examples

### Basic S3 Configuration

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: rclone-s3-secret
type: Opaque
stringData:
  remote: "s3"
  remotePath: "my-bucket"
  configData: |
    [s3]
    type = s3
    provider = AWS
    access_key_id = YOUR_ACCESS_KEY_ID
    secret_access_key = YOUR_SECRET_ACCESS_KEY
    region = us-east-1
```

### Multi-tenant Configuration

```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rclone-multitenant
provisioner: rclone.csi.veloxpack.io
parameters:
  remote: "s3"
  remotePath: "buckets/${pvc.metadata.namespace}/${pvc.metadata.name}"
mountOptions:
  - vfs-cache-mode=writes
  - vfs-cache-max-size=10G
  - dir-cache-time=30s
csi.storage.k8s.io/node-publish-secret-name: "rclone-secret"
csi.storage.k8s.io/node-publish-secret-namespace: "default"
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
```

### Performance Tuning

```yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-rclone-performance
spec:
  mountOptions:
    - vfs-cache-mode=full
    - vfs-cache-max-size=50G
    - vfs-cache-max-age=24h
    - dir-cache-time=5m
    - async-read
    - vfs-read-ahead=1M
  csi:
    driver: rclone.csi.veloxpack.io
    volumeHandle: performance-volume
    volumeAttributes:
      remote: "s3"
      remotePath: "my-bucket"
      configData: |
        [s3]
        type = s3
        provider = AWS
        access_key_id = YOUR_ACCESS_KEY_ID
        secret_access_key = YOUR_SECRET_ACCESS_KEY
        region = us-east-1
```


# Examples (/docs/csi-driver-rclone/examples)

## rclone-secret.yaml

```yaml
---
apiVersion: v1
kind: Secret
metadata:
  name: rclone-secret
  namespace: default
type: Opaque
stringData:
  # Default remote configuration
  # These values can be overridden by volumeAttributes in PV/StorageClass
  remote: "s3"
  remotePath: "mybucket"
  configData: |
    [s3]
    type = s3
    provider = Minio
    endpoint = http://localhost:30900
    access_key_id = admin
    secret_access_key = password
---
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: rclone.csi.veloxpack.io
  name: pv-nginx
  namespace: default
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  csi:
    driver: rclone.csi.veloxpack.io
    volumeHandle: data-id
    nodePublishSecretRef:
      name: rclone-secret
      namespace: default
    # volumeAttributes:
    #   remote: "s3"
    #   remotePath: "mybucket"
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-nginx
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  volumeName: pv-nginx
  storageClassName: ""
---
apiVersion: v1
kind: Pod
metadata:
  name: nginx-rclone-example
  namespace: default
spec:
  containers:
    - image: nginx
      name: nginx
      ports:
        - containerPort: 80
          protocol: TCP
      resources:
        requests:
          cpu: 100m
          memory: 128Mi
        limits:
          cpu: 500m
          memory: 512Mi
      volumeMounts:
        - name: pvc-nginx
          mountPath: /usr/share/nginx/html
          readOnly: false
  volumes:
    - name: pvc-nginx
      persistentVolumeClaim:
        claimName: pvc-nginx
```

## rclone-pv-example.yaml

```yaml
---
# Rclone CSI Driver - PersistentVolume Example
# This example demonstrates how to create a PersistentVolume with inline rclone configuration
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: rclone.csi.veloxpack.io
  name: pv-rclone-example
  namespace: default
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteMany  # Rclone supports ReadWriteMany for cloud storage
  persistentVolumeReclaimPolicy: Delete
  mountOptions:
    - debug-fuse
  csi:
    driver: rclone.csi.veloxpack.io
    volumeHandle: rclone-pv-example-volume
    volumeAttributes:
      remote: "s3"
      remotePath: "mybucket"
      # Inline rclone configuration for MinIO S3-compatible storage
      configData: |
        [s3]
        type = s3
        provider = Minio
        endpoint = http://localhost:30900
        access_key_id = admin
        secret_access_key = password
---
# PersistentVolumeClaim that references the above PV
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-rclone-example
  namespace: default
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
  volumeName: pv-rclone-example
  storageClassName: ""  # Empty string means no dynamic provisioning
---
# Example application pod using the rclone volume
apiVersion: v1
kind: Pod
metadata:
  name: rclone-test-app
  namespace: default
spec:
  containers:
    - image: nginx
      name: web-server
      ports:
        - containerPort: 80
          protocol: TCP
      resources:
        requests:
          cpu: 100m
          memory: 128Mi
        limits:
          cpu: 500m
          memory: 512Mi
      volumeMounts:
        - name: rclone-storage
          mountPath: /usr/share/nginx/html
          readOnly: false
      # Example: Create a test file to verify rclone mount works
      command: ["/bin/sh"]
      args: ["-c", "echo 'Hello from rclone CSI driver!' > /usr/share/nginx/html/index.html && nginx -g 'daemon off;'"]
  volumes:
    - name: rclone-storage
      persistentVolumeClaim:
        claimName: pvc-rclone-example
```

## nginx-dynamic-path.yaml

```yaml
# Example: Dynamic Path Generation with StorageClass
#
# This example demonstrates the new template variable syntax for multi-tenant storage.
# The CSI driver will automatically substitute PVC/PV metadata into the remotePath.
#
# Supported template variables:
#   ${pvc.metadata.name}       - Name of the PersistentVolumeClaim
#   ${pvc.metadata.namespace}  - Namespace of the PersistentVolumeClaim
#   ${pv.metadata.name}        - Name of the PersistentVolume

---
# StorageClass with dynamic path generation
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rclone-s3-multitenant
provisioner: rclone.csi.veloxpack.io
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: false
parameters:
  # Remote configuration (Minio S3)
  remote: "s3"

  # Dynamic path with namespace and PVC name isolation
  # Each PVC gets its own isolated directory: buckets/<namespace>/<pvc-name>
  remotePath: "buckets/${pvc.metadata.namespace}/${pvc.metadata.name}"

  # Rclone configuration (for demo - use secrets in production!)
  configData: |
    [s3]
    type = s3
    provider = Minio
    endpoint = http://localhost:9000
    access_key_id = admin
    secret_access_key = password

---
# PersistentVolumeClaim in the 'production' namespace
# Will result in remotePath: buckets/production/data-store
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-store
  namespace: production
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: rclone-s3-multitenant
  resources:
    requests:
      storage: 10Gi

---
# PersistentVolumeClaim in the 'staging' namespace
# Will result in remotePath: buckets/staging/cache-volume
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cache-volume
  namespace: staging
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: rclone-s3-multitenant
  resources:
    requests:
      storage: 5Gi

---
# Pod using the production PVC
apiVersion: v1
kind: Pod
metadata:
  name: nginx-production
  namespace: production
spec:
  containers:
    - name: nginx
      image: nginx:latest
      ports:
        - containerPort: 80
      resources:
        requests:
          cpu: 100m
          memory: 128Mi
        limits:
          cpu: 500m
          memory: 512Mi
      volumeMounts:
        - name: storage
          mountPath: /usr/share/nginx/html
  volumes:
    - name: storage
      persistentVolumeClaim:
        claimName: data-store

---
# Pod using the staging PVC
apiVersion: v1
kind: Pod
metadata:
  name: nginx-staging
  namespace: staging
spec:
  containers:
    - name: nginx
      image: nginx:latest
      ports:
        - containerPort: 80
      resources:
        requests:
          cpu: 100m
          memory: 128Mi
        limits:
          cpu: 500m
          memory: 512Mi
      volumeMounts:
        - name: cache
          mountPath: /var/cache/nginx
  volumes:
    - name: cache
      persistentVolumeClaim:
        claimName: cache-volume
```


# CSI Driver Rclone (/docs/csi-driver-rclone)

## Overview

The Rclone CSI Driver enables Kubernetes to mount 50+ cloud storage backends as persistent volumes using rclone as a Go library. It provides seamless integration between Kubernetes storage and cloud providers without requiring external rclone binaries.

### Key Features

* **50+ Storage Providers**: Supports Amazon S3, Google Cloud Storage, Azure Blob, Dropbox, SFTP, and many more
* **No External Dependencies**: Uses rclone as a Go library directly - no rclone binary installation required
* **No Process Overhead**: Direct library integration means no subprocess spawning or external process management
* **Dynamic Volume Provisioning**: Create persistent volumes via StorageClass
* **Secret-based Configuration**: Secure credential management using Kubernetes secrets
* **Inline Configuration**: Direct configuration in StorageClass parameters
* **Template Variable Support**: Dynamic path substitution using PVC/PV metadata
* **VFS Caching**: High-performance caching with configurable options
* **No Staging Required**: Direct mount without volume staging
* **Flexible Backend Support**: Choose between minimal or full backend support for smaller images

### Architecture

The driver implements the CSI specification with three main components:

* **Identity Server**: Plugin metadata and health checks
* **Controller Server**: Volume lifecycle management (create/delete)
* **Node Server**: Volume mounting/unmounting on nodes

**Key Design Decisions:**

1. **No Staging**: Rclone volumes don't require staging
2. **Direct Rclone Integration**: Uses rclone's Go library directly
3. **Remote Creation**: Creates temporary remotes for each mount
4. **VFS Caching**: Leverages rclone's VFS for improved performance
5. **Template Variable Support**: Dynamic path substitution using PVC/PV metadata

### Supported Kubernetes Versions

| Driver Version | Kubernetes Version | Status |
| -------------- | ------------------ | ------ |
| main branch    | 1.20+              | GA     |
| v0.1.0         | 1.20+              | GA     |

### Requirements

* Kubernetes 1.20 or later
* CSI node driver registrar
* FUSE support on nodes (for mounting)
* **No rclone installation required** - uses rclone as Go library


# Install (/docs/csi-driver-rclone/install)

## Prerequisites

* Kubernetes 1.20+
* kubectl configured
* Container registry (for custom images)

## Build From Source (Optional)

```bash
git clone https://github.com/veloxpack/csi-driver-rclone.git
cd csi-driver-rclone

# Build binary and image
make build
make container

# (optional) Push to your registry
make push
```

## Install with kubectl (Kustomize)

```bash
# If using a custom image, update image refs first
sed -i "s|image: .*|image: ${REGISTRY}/csi-rclone:${IMAGE_VERSION}|g" deploy/csi-rclone-controller.yaml
sed -i "s|image: .*|image: ${REGISTRY}/csi-rclone:${IMAGE_VERSION}|g" deploy/csi-rclone-node.yaml

# Apply controller, node, RBAC, driverinfo, namespace
kubectl apply -k deploy/
```

## Install with Helm (Recommended)

Install directly from the OCI registry:

```bash
# Install with default configuration
helm install csi-rclone oci://registry-1.docker.io/veloxpack/csi-driver-rclone-charts

# Install in a specific namespace
helm install csi-rclone oci://registry-1.docker.io/veloxpack/csi-driver-rclone-charts \
  --namespace veloxpack --create-namespace
```

Verify the installation:

```bash
# Check release status
helm list -n veloxpack

# Verify pods are running
kubectl get pods -n veloxpack -l app.kubernetes.io/name=csi-driver-rclone
```

### Custom Configuration

Create a `values.yaml` for custom configuration:

```yaml
image:
  repository: your-registry.com/csi-rclone
  tag: latest
  pullPolicy: Always

controller:
  replicas: 2
  resources:
    requests:
      memory: "256Mi"
      cpu: "100m"
    limits:
      memory: "512Mi"
      cpu: "500m"

node:
  resources:
    requests:
      memory: "256Mi"
      cpu: "100m"
    limits:
      memory: "512Mi"
      cpu: "500m"
```

```bash
helm install csi-rclone oci://registry-1.docker.io/veloxpack/csi-driver-rclone-charts \
  -f values.yaml --namespace veloxpack --create-namespace
```

## Verify Installation

```bash
# Check driver pods (for Helm installation)
kubectl get pods -n veloxpack -l app.kubernetes.io/name=csi-driver-rclone

# Check driver pods (for kubectl installation)
kubectl get pods -n veloxpack -l app=csi-rclone-controller
kubectl get pods -n veloxpack -l app=csi-rclone-node

# Check CSIDriver
kubectl get csidriver rclone.csi.veloxpack.io

# Check driver logs (for Helm installation)
kubectl logs -n veloxpack -l app.kubernetes.io/name=csi-driver-rclone

# Check driver logs (for kubectl installation)
kubectl logs -n veloxpack -l app=csi-rclone-controller
kubectl logs -n veloxpack -l app=csi-rclone-node
```

## Driver Configuration

The driver supports various configuration options via command-line flags:

### Mount Options

* `--allow-non-empty`: Allow mounting over non-empty directories
* `--allow-other`: Allow access to other users
* `--async-read`: Use asynchronous reads
* `--debug-fuse`: Debug FUSE internals
* `--direct-io`: Use Direct IO (disables caching)

### VFS Options

* `--vfs-cache-mode`: Cache mode (off|minimal|writes|full)
* `--vfs-cache-max-size`: Maximum cache size
* `--vfs-cache-max-age`: Max time since last access
* `--dir-cache-time`: Directory cache time
* `--read-only`: Read-only access

### Default Parameters

The driver sets these defaults:

```yaml
cache-info-age: "24h"
cache-chunk-clean-interval: "5m"
cache-dir: "/tmp/rclone-vfs-cache/"
```

## Troubleshooting

### Common Issues

1. **Driver won't start**: Check FUSE installation and permissions
2. **Mount fails**: Verify rclone configuration and network connectivity
3. **Performance issues**: Adjust VFS cache settings
4. **Permission errors**: Check node permissions and FUSE setup

### Debug Commands

```bash
# Check driver status (for Helm installation)
kubectl get pods -n veloxpack -l app.kubernetes.io/name=csi-driver-rclone

# Check driver status (for kubectl installation)
kubectl get pods -n veloxpack -l app=csi-rclone-controller
kubectl get pods -n veloxpack -l app=csi-rclone-node

# Check driver logs (for Helm installation)
kubectl logs -n veloxpack -l app.kubernetes.io/name=csi-driver-rclone --tail=100

# Check driver logs (for kubectl installation)
kubectl logs -n veloxpack -l app=csi-rclone-controller --tail=100
kubectl logs -n veloxpack -l app=csi-rclone-node --tail=100

# Check CSIDriver
kubectl get csidriver rclone.csi.veloxpack.io -o yaml

# Check events
kubectl get events --sort-by=.metadata.creationTimestamp
```

### Enable Debug Logging

```yaml
# In controller deployment
args:
  - "--v=5"
  - "--logtostderr=true"
  - "--stderrthreshold=INFO"

# In node deployment
args:
  - "--v=5"
  - "--logtostderr=true"
  - "--stderrthreshold=INFO"
```


# Quick Start (/docs/csi-driver-rclone/quick-start)

## Prerequisites

* Kubernetes 1.20+
* kubectl configured
* FUSE support on nodes

## 1. Deploy the Driver

```bash
kubectl apply -k deploy/
```

This installs:

* CSI Controller (StatefulSet)
* CSI Node Driver (DaemonSet)
* RBAC permissions
* CSIDriver CRD

## 2. Create Storage Secret

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: rclone-secret
  namespace: default
type: Opaque
stringData:
  remote: "s3"
  remotePath: "my-bucket"
  configData: |
    [s3]
    type = s3
    provider = AWS
    access_key_id = YOUR_ACCESS_KEY_ID
    secret_access_key = YOUR_SECRET_ACCESS_KEY
    region = us-east-1
```

## 3. Create StorageClass

```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rclone-csi
provisioner: rclone.csi.veloxpack.io
parameters:
  remote: "s3"
  remotePath: "my-bucket"
csi.storage.k8s.io/node-publish-secret-name: "rclone-secret"
csi.storage.k8s.io/node-publish-secret-namespace: "default"
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
```

## 4. Create PVC and Pod

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-rclone
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
  storageClassName: rclone-csi
---
apiVersion: v1
kind: Pod
metadata:
  name: nginx-rclone
spec:
  containers:
  - name: nginx
    image: nginx
    volumeMounts:
    - name: data
      mountPath: /data
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: pvc-rclone
```

## 5. Verify Installation

```bash
# Check driver pods
kubectl get pods -n veloxpack -l app=csi-rclone-controller
kubectl get pods -n veloxpack -l app=csi-rclone-node

# Check CSIDriver
kubectl get csidriver rclone.csi.veloxpack.io

# Check PVC status
kubectl get pvc pvc-rclone

# Test mount
kubectl exec nginx-rclone -- ls -la /data
```


# Rclone Configuration (/docs/csi-driver-rclone/rclone-configuration)

## Overview

The CSI driver supports 50+ storage backends through rclone configuration. Store sensitive credentials in Kubernetes secrets and reference them in StorageClass or PersistentVolume.

## Configuration Methods

### Method 1: Kubernetes Secrets (Recommended)

Store credentials in secrets and reference them in StorageClass:

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: rclone-secret
  namespace: default
type: Opaque
stringData:
  remote: "s3"
  remotePath: "my-bucket"
  configData: |
    [s3]
    type = s3
    provider = AWS
    access_key_id = YOUR_ACCESS_KEY_ID
    secret_access_key = YOUR_SECRET_ACCESS_KEY
    region = us-east-1
```

### Method 2: Inline Configuration

Include configuration directly in StorageClass parameters:

```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rclone-s3
provisioner: rclone.csi.veloxpack.io
parameters:
  remote: "s3"
  remotePath: "my-bucket"
  configData: |
    [s3]
    type = s3
    provider = AWS
    access_key_id = YOUR_ACCESS_KEY_ID
    secret_access_key = YOUR_SECRET_ACCESS_KEY
    region = us-east-1
```

### Method 3: PersistentVolume Configuration

Configure directly in PersistentVolume volumeAttributes:

```yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-rclone
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteMany
  csi:
    driver: rclone.csi.veloxpack.io
    volumeHandle: rclone-volume
    volumeAttributes:
      remote: "s3"
      remotePath: "my-bucket"
      configData: |
        [s3]
        type = s3
        provider = AWS
        access_key_id = YOUR_ACCESS_KEY_ID
        secret_access_key = YOUR_SECRET_ACCESS_KEY
        region = us-east-1
```

**Priority**: volumeAttributes > StorageClass parameters > Secrets

## Dynamic Path Substitution

Use template variables in `remotePath` for multi-tenant isolation:

| Variable                    | Description   | Example            |
| --------------------------- | ------------- | ------------------ |
| `${pvc.metadata.name}`      | PVC name      | `my-pvc-12345`     |
| `${pvc.metadata.namespace}` | PVC namespace | `default`          |
| `${pv.metadata.name}`       | PV name       | `pv-rclone-abc123` |

Example:

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: rclone-multitenant-secret
type: Opaque
stringData:
  remote: "s3"
  remotePath: "buckets/${pvc.metadata.namespace}/${pvc.metadata.name}"
  configData: |
    [s3]
    type = s3
    provider = AWS
    access_key_id = YOUR_ACCESS_KEY_ID
    secret_access_key = YOUR_SECRET_ACCESS_KEY
    region = us-east-1
```

## VFS Cache Options

Configure caching for better performance using `mountOptions`:

```yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-rclone-performance
spec:
  mountOptions:
    - vfs-cache-mode=writes
    - vfs-cache-max-size=10G
    - dir-cache-time=30s
  csi:
    driver: rclone.csi.veloxpack.io
    volumeHandle: performance-volume
    volumeAttributes:
      remote: "s3"
      remotePath: "my-bucket"
      configData: |
        [s3]
        type = s3
        provider = AWS
        access_key_id = YOUR_ACCESS_KEY_ID
        secret_access_key = YOUR_SECRET_ACCESS_KEY
        region = us-east-1
```

## Supported Backends

The driver supports all rclone backends, including:

* **Amazon S3** and S3-compatible storage (MinIO, DigitalOcean Spaces, etc.)
* **Google Cloud Storage**
* **Azure Blob Storage**
* **Dropbox**
* **SFTP/SSH**
* **Google Drive**
* **OneDrive**
* **Box**
* **Backblaze B2**
* **WebDAV**
* **FTP**
* **And 50+ more backends**

## Parameter Processing

The driver processes parameters in this order:

1. **Secrets**: Loaded as defaults from `csi.storage.k8s.io/node-publish-secret-name`
2. **Volume Context**: Overrides secrets (from StorageClass parameters or PV volumeAttributes)
3. **ConfigData**: Parsed INI format and merged with other parameters
4. **Parameter Sanitization**: Remote prefixes removed, hyphens converted to underscores

### Parameter Sanitization

Parameters are sanitized for consistency:

* `s3-endpoint` → `endpoint` (when remote is "s3")
* `--cache-mode` → `cache_mode`
* `EndPoint` → `endpoint`

## Security Best Practices

1. **Use Secrets**: Store sensitive credentials in Kubernetes secrets
2. **RBAC**: Ensure proper RBAC permissions are configured
3. **Network Policies**: Consider using network policies to restrict access
4. **Image Security**: Use trusted container images
5. **Credential Rotation**: Regularly rotate storage backend credentials
6. **Minimal Permissions**: Grant only necessary permissions to storage backends

## Troubleshooting Configuration

### Common Issues

1. **Authentication failures**: Verify credentials in secrets or configData
2. **Network connectivity**: Ensure nodes can reach the storage backend
3. **Permission errors**: Check that credentials have proper access rights
4. **Configuration format**: Ensure configData is valid INI format
5. **Resource constraints**: Verify sufficient memory and disk space

### Debug Commands

```bash
# Check secret contents
kubectl get secret rclone-secret -o yaml

# Decode secret data
kubectl get secret rclone-secret -o jsonpath='{.data.configData}' | base64 -d

# Check driver logs
kubectl logs -l app=csi-rclone-node -n veloxpack

# Check mount options
kubectl describe pv pv-rclone
```


# StorageClass (/docs/csi-driver-rclone/storageclass)

## Overview

StorageClass defines how volumes are dynamically provisioned. The rclone CSI driver supports dynamic provisioning with various storage backends through StorageClass parameters.

## Basic StorageClass

```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rclone-csi
provisioner: rclone.csi.veloxpack.io
parameters:
  remote: "s3"
  remotePath: "my-bucket"
csi.storage.k8s.io/node-publish-secret-name: "rclone-secret"
csi.storage.k8s.io/node-publish-secret-namespace: "default"
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
```

## StorageClass Parameters

### Required Parameters

| Parameter    | Description                    | Example                                 |
| ------------ | ------------------------------ | --------------------------------------- |
| `remote`     | Rclone remote name             | `s3`, `gcs`, `azureblob`, `dropbox`     |
| `remotePath` | Path within the remote storage | `my-bucket`, `/data`, `buckets/tenant1` |

### Optional Parameters

| Parameter    | Description                 | Example                           |
| ------------ | --------------------------- | --------------------------------- |
| `configData` | Inline rclone configuration | `[s3]\ntype = s3\nprovider = AWS` |

### CSI Parameters

| Parameter                                          | Description                 | Example         |
| -------------------------------------------------- | --------------------------- | --------------- |
| `csi.storage.k8s.io/node-publish-secret-name`      | Secret name for credentials | `rclone-secret` |
| `csi.storage.k8s.io/node-publish-secret-namespace` | Secret namespace            | `default`       |

## Dynamic Path Substitution

Use template variables for multi-tenant storage isolation:

```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rclone-multitenant
provisioner: rclone.csi.veloxpack.io
parameters:
  remote: "s3"
  # Each PVC gets its own isolated directory
  remotePath: "buckets/${pvc.metadata.namespace}/${pvc.metadata.name}"
csi.storage.k8s.io/node-publish-secret-name: "rclone-secret"
csi.storage.k8s.io/node-publish-secret-namespace: "default"
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: false
```

### Supported Template Variables

| Variable                    | Description   | Example            |
| --------------------------- | ------------- | ------------------ |
| `${pvc.metadata.name}`      | PVC name      | `my-pvc-12345`     |
| `${pvc.metadata.namespace}` | PVC namespace | `default`          |
| `${pv.metadata.name}`       | PV name       | `pv-rclone-abc123` |

## Performance Tuning

### VFS Cache Configuration (via mountOptions)

Configure VFS cache using `mountOptions` (not StorageClass `parameters`):

```yaml
apiVersion: v1
kind: StorageClass
metadata:
  name: rclone-performance
provisioner: rclone.csi.veloxpack.io
parameters:
  remote: "s3"
  remotePath: "my-bucket"
mountOptions:
  - vfs-cache-mode=writes
  - vfs-cache-max-size=10G
  - dir-cache-time=30s
csi.storage.k8s.io/node-publish-secret-name: "rclone-secret"
csi.storage.k8s.io/node-publish-secret-namespace: "default"
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
```

### Mount Options

```yaml
apiVersion: v1
kind: StorageClass
metadata:
  name: rclone-debug
provisioner: rclone.csi.veloxpack.io
parameters:
  remote: "s3"
  remotePath: "my-bucket"
mountOptions:
  - debug-fuse
  - vfs-cache-mode=writes
  - vfs-cache-max-size=5G
csi.storage.k8s.io/node-publish-secret-name: "rclone-secret"
csi.storage.k8s.io/node-publish-secret-namespace: "default"
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
```

For a complete list of available mount options, see the [rclone mount documentation](https://rclone.org/commands/rclone_mount/).

## Volume Binding Modes

### Immediate Binding

```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rclone-immediate
provisioner: rclone.csi.veloxpack.io
parameters:
  remote: "s3"
  remotePath: "my-bucket"
volumeBindingMode: Immediate
csi.storage.k8s.io/node-publish-secret-name: "rclone-secret"
csi.storage.k8s.io/node-publish-secret-namespace: "default"
reclaimPolicy: Delete
allowVolumeExpansion: true
```

### WaitForFirstConsumer Binding

```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rclone-wait
provisioner: rclone.csi.veloxpack.io
parameters:
  remote: "s3"
  remotePath: "my-bucket"
volumeBindingMode: WaitForFirstConsumer
csi.storage.k8s.io/node-publish-secret-name: "rclone-secret"
csi.storage.k8s.io/node-publish-secret-namespace: "default"
reclaimPolicy: Delete
allowVolumeExpansion: true
```

## Reclaim Policies

### Delete (Default)

```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rclone-delete
provisioner: rclone.csi.veloxpack.io
parameters:
  remote: "s3"
  remotePath: "my-bucket"
reclaimPolicy: Delete
csi.storage.k8s.io/node-publish-secret-name: "rclone-secret"
csi.storage.k8s.io/node-publish-secret-namespace: "default"
volumeBindingMode: Immediate
allowVolumeExpansion: true
```

### Retain

```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rclone-retain
provisioner: rclone.csi.veloxpack.io
parameters:
  remote: "s3"
  remotePath: "my-bucket"
reclaimPolicy: Retain
csi.storage.k8s.io/node-publish-secret-name: "rclone-secret"
csi.storage.k8s.io/node-publish-secret-namespace: "default"
volumeBindingMode: Immediate
allowVolumeExpansion: true
```

## Volume Expansion

Enable volume expansion for PVCs:

```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rclone-expandable
provisioner: rclone.csi.veloxpack.io
parameters:
  remote: "s3"
  remotePath: "my-bucket"
allowVolumeExpansion: true
csi.storage.k8s.io/node-publish-secret-name: "rclone-secret"
csi.storage.k8s.io/node-publish-secret-namespace: "default"
reclaimPolicy: Delete
volumeBindingMode: Immediate
```

## Inline Configuration

Store configuration directly in StorageClass (not recommended for production):

```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rclone-inline
provisioner: rclone.csi.veloxpack.io
parameters:
  remote: "s3"
  remotePath: "my-bucket"
  configData: |
    [s3]
    type = s3
    provider = AWS
    access_key_id = YOUR_ACCESS_KEY_ID
    secret_access_key = YOUR_SECRET_ACCESS_KEY
    region = us-east-1
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
```

## Troubleshooting StorageClass

### Common Issues

1. **Provisioning fails**: Check secret exists and has correct data
2. **Volume binding fails**: Verify volumeBindingMode and node capabilities
3. **Mount fails**: Check rclone configuration and network connectivity
4. **Performance issues**: Adjust VFS cache settings

### Debug Commands

```bash
# Check StorageClass
kubectl get storageclass rclone-csi -o yaml

# Check CSIDriver
kubectl get csidriver rclone.csi.veloxpack.io

# Check PVC events
kubectl describe pvc my-pvc

# Check driver logs
kubectl logs -l app=csi-rclone-controller -n veloxpack
kubectl logs -l app=csi-rclone-node -n veloxpack
```

## Best Practices

1. **Use Secrets**: Store credentials in Kubernetes secrets, not inline
2. **Namespace Isolation**: Use template variables for multi-tenant setups
3. **Resource Limits**: Set appropriate resource limits for driver pods
4. **Monitoring**: Monitor driver logs and metrics
5. **Testing**: Test StorageClass configurations in non-production environments
6. **Documentation**: Document custom StorageClass configurations


# Troubleshooting (/docs/csi-driver-rclone/troubleshooting)

## Overview

This guide covers common issues, debugging techniques, and solutions for the rclone CSI driver.

## Check Driver Status

### Verify Driver Pods

```bash
# Check controller pods
kubectl get pods -n veloxpack -l app=csi-rclone-controller

# Check node pods
kubectl get pods -n veloxpack -l app=csi-rclone-node

# Check pod status
kubectl describe pod -n veloxpack -l app=csi-rclone-controller
kubectl describe pod -n veloxpack -l app=csi-rclone-node
```

### Check CSIDriver Resource

```bash
# Check CSIDriver
kubectl get csidriver rclone.csi.veloxpack.io

# Get detailed information
kubectl describe csidriver rclone.csi.veloxpack.io
```

### Verify Driver Functionality

```bash
# Check if the driver is working correctly
kubectl exec -n veloxpack -l app=csi-rclone-node -- /rcloneplugin --help

# Check driver version information
kubectl logs -n veloxpack -l app=csi-rclone-node --tail=10 | grep "DRIVER INFORMATION" -A 10
```

## Common Issues

### 1. Driver Pods Not Starting

**Symptoms:**

* Pods stuck in `Pending` or `CrashLoopBackOff`
* Driver not responding to CSI calls

**Causes:**

* FUSE not installed on nodes
* Insufficient permissions
* Resource constraints
* Image pull issues

**Solutions:**

```bash
# Check node capabilities
kubectl describe node <node-name>

# Check if FUSE is available
kubectl exec -n veloxpack -l app=csi-rclone-node -- ls /dev/fuse

# Check resource limits
kubectl describe pod -n veloxpack -l app=csi-rclone-controller

# Check image pull
kubectl describe pod -n veloxpack -l app=csi-rclone-controller | grep -i image
```

### 2. Volume Mount Failures

**Symptoms:**

* PVC stuck in `Pending`
* Mount operations failing
* Pods can't access mounted volumes

**Causes:**

* Invalid rclone configuration
* Network connectivity issues
* Authentication failures
* Permission errors

**Solutions:**

```bash
# Check PVC events
kubectl describe pvc <pvc-name>

# Check pod events
kubectl describe pod <pod-name>

# Check driver logs
kubectl logs -n veloxpack -l app=csi-rclone-node --tail=100

# Verify secret contents
kubectl get secret <secret-name> -o yaml
kubectl get secret <secret-name> -o jsonpath='{.data.configData}' | base64 -d
```

### 3. Authentication Failures

**Symptoms:**

* Mount operations fail with authentication errors
* Driver logs show credential issues

**Causes:**

* Invalid credentials in secrets
* Expired tokens
* Incorrect configuration format

**Solutions:**

```bash
# Check secret data
kubectl get secret rclone-secret -o jsonpath='{.data.configData}' | base64 -d

# Verify credentials manually
kubectl exec -n veloxpack -l app=csi-rclone-node -- /rcloneplugin --help

# Test configuration
kubectl exec -n veloxpack -l app=csi-rclone-node -- sh -c 'echo "[s3]
type = s3
provider = AWS
access_key_id = YOUR_KEY
secret_access_key = YOUR_SECRET
region = us-east-1" > /tmp/test.conf && rclone lsd s3:test-bucket --config /tmp/test.conf'
```

### 4. Performance Issues

**Symptoms:**

* Slow file operations
* High memory usage
* Timeout errors

**Causes:**

* Inadequate VFS cache configuration
* Network latency
* Resource constraints

**Solutions:**

```bash
# Check VFS cache settings
kubectl describe pv <pv-name> | grep -i mount

# Monitor resource usage
kubectl top pods -n veloxpack -l app=csi-rclone-node

# Adjust cache settings
# Add to StorageClass mountOptions:
# - vfs-cache-mode=writes
# - vfs-cache-max-size=10G
# - dir-cache-time=30s
```

### 5. Network Connectivity Issues

**Symptoms:**

* Timeout errors
* Connection refused
* Slow operations

**Causes:**

* Network policies blocking access
* DNS resolution issues
* Firewall rules

**Solutions:**

```bash
# Test connectivity from driver pod
kubectl exec -n veloxpack -l app=csi-rclone-node -- nslookup s3.amazonaws.com

# Check network policies
kubectl get networkpolicies

# Test from node
kubectl debug node/<node-name> -it --image=busybox -- nslookup s3.amazonaws.com
```

## Debug Commands

### Check Driver Logs

```bash
# Controller logs
kubectl logs -n veloxpack -l app=csi-rclone-controller --tail=100

# Node logs
kubectl logs -n veloxpack -l app=csi-rclone-node --tail=100

# Follow logs
kubectl logs -n veloxpack -l app=csi-rclone-node -f

# Previous container logs
kubectl logs -n veloxpack -l app=csi-rclone-node --previous
```

### Check Mount Points

```bash
# List mount points
kubectl exec -n veloxpack -l app=csi-rclone-node -- mount | grep rclone

# Check mount options
kubectl exec -n veloxpack -l app=csi-rclone-node -- cat /proc/mounts | grep rclone

# Check FUSE mounts
kubectl exec -n veloxpack -l app=csi-rclone-node -- ls -la /dev/fuse
```

### Check Volume Status

```bash
# Check PVC status
kubectl get pvc <pvc-name> -o yaml

# Check PV status
kubectl get pv <pv-name> -o yaml

# Check pod volume mounts
kubectl describe pod <pod-name> | grep -A 10 "Volumes:"
```

### Check Events

```bash
# All events
kubectl get events --sort-by=.metadata.creationTimestamp

# Events for specific resource
kubectl get events --field-selector involvedObject.name=<resource-name>

# Recent events
kubectl get events --sort-by=.metadata.creationTimestamp --field-selector type=Warning
```

## Enable Debug Logging

### Driver Logging

```yaml
# In controller deployment
args:
  - "--v=5"
  - "--logtostderr=true"
  - "--stderrthreshold=INFO"

# In node deployment
args:
  - "--v=5"
  - "--logtostderr=true"
  - "--stderrthreshold=INFO"
```

### FUSE Debugging

```yaml
# Add to StorageClass mountOptions
mountOptions:
  - debug-fuse
  - v=5
```

## Performance Tuning

### VFS Cache Configuration

```yaml
# High performance configuration
mountOptions:
  - vfs-cache-mode=full
  - vfs-cache-max-size=50G
  - vfs-cache-max-age=24h
  - dir-cache-time=5m
  - vfs-read-ahead=1M
```

### Resource Limits

```yaml
# Controller resources
resources:
  requests:
    memory: "256Mi"
    cpu: "100m"
  limits:
    memory: "1Gi"
    cpu: "1000m"

# Node resources
resources:
  requests:
    memory: "512Mi"
    cpu: "200m"
  limits:
    memory: "2Gi"
    cpu: "2000m"
```

## Monitoring

### Key Metrics to Monitor

1. **Driver Health**
   * Pod status and restarts
   * Memory and CPU usage
   * Log error rates

2. **Volume Operations**
   * Mount/unmount success rates
   * Operation latency
   * Error rates

3. **Storage Backend**
   * API call latency
   * Error rates
   * Throughput

### Prometheus Metrics

```yaml
# Add to driver deployment
args:
  - "--metrics-address=:8080"
  - "--metrics-path=/metrics"
```

## Recovery Procedures

### Restart Driver Pods

```bash
# Restart controller
kubectl rollout restart deployment/csi-rclone-controller -n veloxpack

# Restart node daemonset
kubectl rollout restart daemonset/csi-rclone-node -n veloxpack
```

### Clean Up Corrupted Mounts

```bash
# Force unmount on specific node
kubectl exec -n veloxpack -l app=csi-rclone-node -- umount -f /var/lib/kubelet/pods/*/volumes/kubernetes.io~csi/*/mount

# Restart node daemonset
kubectl rollout restart daemonset/csi-rclone-node -n veloxpack
```

### Reset Driver State

```bash
# Delete CSIDriver resource
kubectl delete csidriver rclone.csi.veloxpack.io

# Recreate
kubectl apply -f deploy/csi-rclone-driverinfo.yaml
```

## Getting Help

### Log Collection

```bash
# Collect logs for debugging
kubectl logs -n veloxpack -l app=csi-rclone-controller > controller.log
kubectl logs -n veloxpack -l app=csi-rclone-node > node.log
kubectl get events --sort-by=.metadata.creationTimestamp > events.log
```

### Support Resources

* [GitHub Issues](https://github.com/veloxpack/csi-driver-rclone/issues)
* [GitHub Discussions](https://github.com/veloxpack/csi-driver-rclone/discussions)
* [Rclone Documentation](https://rclone.org/)
* [CSI Specification](https://github.com/container-storage-interface/spec)