fe-filesystem

Doris FE Filesystem Module

The fe-filesystem module provides a pluggable filesystem abstraction for the Doris FE (Frontend). It decouples the query engine (fe-core) from concrete storage backends (S3, HDFS, Azure Blob, etc.) at both the Maven dependency level and the runtime classpath level.

Each storage backend is a self-contained plugin that is discovered and loaded at FE startup—no modification to fe-core is required to add a new storage backend.

Module Structure

fe-filesystem/                        (aggregator POM — no Java code)
│
├── fe-filesystem-api/                [API]  Core abstractions (FileSystem, Location, …)
├── fe-filesystem-spi/                [SPI]  Provider interface + object-storage contracts
│
├── fe-filesystem-s3/                 [IMPL] AWS S3 / S3-compatible (MinIO, …)
├── fe-filesystem-oss/                [IMPL] Alibaba Cloud OSS  (delegates to S3)
├── fe-filesystem-cos/                [IMPL] Tencent Cloud COS  (delegates to S3)
├── fe-filesystem-obs/                [IMPL] Huawei Cloud OBS   (delegates to S3)
├── fe-filesystem-azure/              [IMPL] Azure Blob Storage
├── fe-filesystem-hdfs/               [IMPL] HDFS / ViewFS / OFS / JFS
├── fe-filesystem-local/              [IMPL] Local filesystem   (testing only)
└── fe-filesystem-broker/             [IMPL] Doris Broker process (Thrift RPC)

Layering

Layer	Module	Compiled into fe-core?	Deployed as plugin?
API	`fe-filesystem-api`	✅ Yes	❌ No
SPI	`fe-filesystem-spi`	✅ Yes	❌ No
IMPL	`fe-filesystem-s3`, `-hdfs`, …	❌ No	✅ Yes

API — Pure-JDK interfaces and value types consumed by fe-core (zero third-party dependencies). Defines FileSystem, Location, FileEntry, DorisInputFile, DorisOutputFile, etc.
SPI — The FileSystemProvider interface (extends PluginFactory from fe-extension-spi) plus the object-storage layer (ObjStorage, ObjFileSystem, HadoopAuthenticator). Also compiled into fe-core.
IMPL — Concrete backends. Each one depends on fe-filesystem-spi (and transitively on fe-filesystem-api). S3-delegating backends (OSS, COS, OBS) also depend on fe-filesystem-s3 to reuse S3FileSystem. They must not depend on fe-core, fe-common, or fe-catalog.

How It Works

Plugin Discovery & Loading

At FE startup, Env.initFileSystemPluginManager() creates a FileSystemPluginManager and loads providers in two phases:

ServiceLoader scan — Discovers FileSystemProvider implementations already on the classpath (built-in providers, test overrides).
Directory plugin scan — Uses DirectoryPluginRuntimeManager to scan the directory configured by Config.filesystem_plugin_root (default: ${DORIS_HOME}/plugins/filesystem). Each direct child directory is treated as an unpacked plugin directory. The runtime manager resolves pluginDir/*.jar (root-level jars, scanned for ServiceLoader registration) and pluginDir/lib/*.jar (dependency jars, available for class loading only).

Classpath providers have higher priority than directory-loaded providers.

Provider Selection

When fe-core needs a filesystem, it calls FileSystemFactory.getFileSystem(properties):

Map<String, String> properties
        │
        ▼
FileSystemPluginManager.createFileSystem(properties)
        │
        │  for each registered FileSystemProvider:
        │    1. provider.supports(properties)  ← cheap, no I/O
        │    2. if true → provider.create(properties) → return FileSystem
        │
        ▼
First matching provider wins

Each provider's supports() method examines property keys to decide if it can handle the request. For example:

Provider	Matching Logic
S3	`AWS_ACCESS_KEY` + (`AWS_ENDPOINT` or `AWS_REGION`)
OSS	Endpoint contains `aliyuncs.com` or `_STORAGE_TYPE_` = `"OSS"`
HDFS	`_STORAGE_TYPE_` = `"HDFS"` or URI scheme is `hdfs`/`viewfs`/`ofs`/`jfs`/`oss`
Azure	`AZURE_ACCOUNT_NAME` or endpoint contains `blob.core.windows.net`
Local	URI starts with `file://` or `local://`
Broker	`_STORAGE_TYPE_` = `"BROKER"` and `BROKER_HOST` present

Plugin Packaging & Class Loading

Each implementation module uses maven-assembly-plugin to produce a zip build artifact. The zip must be unpacked before deployment. At runtime, DirectoryPluginRuntimeManager expects each plugin to be an unpacked directory under filesystem_plugin_root:

${DORIS_HOME}/plugins/filesystem/
├── s3/                                   ← one directory per plugin
│   ├── doris-fe-filesystem-s3.jar        ← plugin jar at root (scanned for ServiceLoader)
│   └── lib/
│       ├── aws-sdk-*.jar                 ← third-party dependencies
│       └── ...
├── hdfs/
│   ├── doris-fe-filesystem-hdfs.jar
│   └── lib/...
└── ...

The Maven build produces a zip with the same layout. To deploy, unzip it into the appropriate subdirectory (e.g., unzip doris-fe-filesystem-s3.zip -d plugins/filesystem/s3/). Dropping the raw .zip file into the directory will not work — DirectoryPluginRuntimeManager resolves pluginDir/*.jar and pluginDir/lib/*.jar from unpacked directories only.

Jars that are already on the fe-core classpath (fe-filesystem-api, fe-filesystem-spi, fe-extension-spi) are excluded from the zip to avoid duplication.

At runtime, DirectoryPluginRuntimeManager creates an isolated ClassLoader per plugin. A ClassLoadingPolicy ensures that shared framework classes (org.apache.doris.filesystem.*, software.amazon.awssdk.*, org.apache.hadoop.*) are loaded parent-first to avoid cross- ClassLoader cast failures.

S3 Delegation Pattern

Several cloud providers (OSS, COS, OBS) are S3-compatible. Instead of duplicating the S3 implementation, they follow a delegation pattern:

Translate cloud-native property keys to S3-compatible keys.
Extend ObjStorage to override cloud-specific operations (pre-signed URLs, STS tokens).
Delegate core I/O to S3FileSystem (from fe-filesystem-s3).

CosFileSystemProvider
  └─→ creates S3FileSystem(CosObjStorage)
        │                     │
        │  FileSystem ops     │  Overrides getPresignedUrl(), getStsToken()
        └─────────────────────┘

Relationship to Other Modules

┌──────────────────────────────────────────────────────────┐
│                        fe-core                           │
│  Uses: FileSystem, Location, FileEntry (from API)        │
│  Depends on: fe-filesystem-api, fe-filesystem-spi        │
│  Loads plugins from: ${DORIS_HOME}/plugins/filesystem    │
└────────────┬────────────────────────┬────────────────────┘
             │ compile-time           │ runtime (plugin)
             ▼                        ▼
  ┌──────────────────┐    ┌─────────────────────────┐
  │ fe-filesystem-api│    │ fe-filesystem-s3 (zip)  │
  │ fe-filesystem-spi│    │ fe-filesystem-hdfs (zip)│
  │ (compiled in)    │    │ fe-filesystem-xxx (zip) │
  └──────────────────┘    └─────────────────────────┘

fe-core — Compile-time dependency on fe-filesystem-api and fe-filesystem-spi. Implementation modules are not Maven dependencies of fe-core; they are loaded at runtime from the plugin directory.
fe-extension-spi — Provides the PluginFactory / Plugin interfaces and the DirectoryPluginRuntimeManager class used by the plugin loading infrastructure.
fe-filesystem-local — Used in fe-core unit tests (test-scope dependency) so that tests can exercise FileSystem operations without cloud credentials.

Adding a New Filesystem Sub-Module

Follow these steps to add support for a new storage backend (e.g., Google Cloud Storage):

1. Create the Maven module

Create a new directory fe-filesystem/fe-filesystem-gcs/ with this structure:

fe-filesystem-gcs/
├── pom.xml
└── src/
    ├── main/
    │   ├── assembly/
    │   │   └── plugin-zip.xml
    │   ├── java/org/apache/doris/filesystem/gcs/
    │   │   ├── GcsFileSystemProvider.java
    │   │   ├── GcsFileSystem.java
    │   │   └── GcsObjStorage.java          (if object-storage based)
    │   └── resources/META-INF/services/
    │       └── org.apache.doris.filesystem.spi.FileSystemProvider
    └── test/
        └── java/org/apache/doris/filesystem/gcs/
            └── ...Test.java

2. Write `pom.xml`

<project>
    <parent>
        <groupId>org.apache.doris</groupId>
        <artifactId>fe-filesystem</artifactId>
        <version>${revision}</version>
    </parent>

    <artifactId>fe-filesystem-gcs</artifactId>
    <packaging>jar</packaging>
    <name>Doris FE Filesystem - GCS</name>

    <dependencies>
        <!-- REQUIRED: SPI contract (transitively includes API) -->
        <dependency>
            <groupId>org.apache.doris</groupId>
            <artifactId>fe-filesystem-spi</artifactId>
            <version>${revision}</version>
        </dependency>

        <!-- Cloud SDK -->
        <dependency>
            <groupId>com.google.cloud</groupId>
            <artifactId>google-cloud-storage</artifactId>
            <version>...</version>
        </dependency>

        <!-- Test -->
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <finalName>doris-fe-filesystem-gcs</finalName>
        <plugins>
            <!-- Assembly: produces the plugin zip -->
            <plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <configuration>
                    <appendAssemblyId>false</appendAssemblyId>
                    <descriptors>
                        <descriptor>src/main/assembly/plugin-zip.xml</descriptor>
                    </descriptors>
                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id>
                        <phase>package</phase>
                        <goals><goal>single</goal></goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

Important: Do not add dependencies on fe-core, fe-common, or fe-catalog.

3. Write the assembly descriptor

Copy plugin-zip.xml from an existing module (e.g., fe-filesystem-s3/src/main/assembly/). The key points:

Place the plugin jar at the root of the zip (for ServiceLoader discovery).
Place all runtime dependencies in lib/.
Exclude fe-filesystem-api, fe-filesystem-spi, and fe-extension-spi (they are already on the fe-core classpath).

4. Implement `FileSystemProvider`

package org.apache.doris.filesystem.gcs;

import org.apache.doris.filesystem.FileSystem;
import org.apache.doris.filesystem.spi.FileSystemProvider;
import java.io.IOException;
import java.util.Map;

public class GcsFileSystemProvider implements FileSystemProvider {

    // Public no-arg constructor — required by ServiceLoader
    public GcsFileSystemProvider() {}

    @Override
    public boolean supports(Map<String, String> properties) {
        // Must be cheap (no network calls) and deterministic.
        String type = properties.get("_STORAGE_TYPE_");
        return "GCS".equalsIgnoreCase(type);
    }

    @Override
    public FileSystem create(Map<String, String> properties) throws IOException {
        return new GcsFileSystem(properties);
    }

    @Override
    public String name() {
        return "GCS";
    }
}

5. Implement `FileSystem`

You have two choices:

Object-storage backend — Extend ObjFileSystem (from SPI) and implement ObjStorage<C>. This gives you exists(), close(), and cloud-specific delegates for free.
Custom backend — Implement FileSystem directly (like DFSFileSystem for HDFS).

At minimum, you must implement:

Method	Description
`exists(Location)`	Check if a file/directory exists
`mkdirs(Location)`	Create directories
`delete(Location, boolean)`	Delete file or directory
`rename(Location, Location)`	Rename/move
`list(Location)`	List directory contents
`newInputFile(Location)`	Open a file for reading
`newOutputFile(Location)`	Open a file for writing
`close()`	Release resources

6. Register via ServiceLoader

Create the file:

src/main/resources/META-INF/services/org.apache.doris.filesystem.spi.FileSystemProvider

With content:

org.apache.doris.filesystem.gcs.GcsFileSystemProvider

7. Register the module in the parent POM

Add your module to fe-filesystem/pom.xml:

<modules>
    ...
    <module>fe-filesystem-gcs</module>
</modules>

8. Add tests

Unit tests — Test your FileSystem and ObjStorage implementations with mocked cloud clients. Place in src/test/java/.
Environment tests — Tests that require real cloud credentials should be tagged with @Tag("environment"). They are excluded by default and can be enabled with -Dtest.excludedGroups=none.

9. Build and deploy

# Build the new module (must go through the reactor so sibling SNAPSHOTs resolve)
cd fe
mvn package -pl fe-filesystem/fe-filesystem-gcs --also-make -DskipTests

# The build produces a zip at:
#   fe-filesystem/fe-filesystem-gcs/target/doris-fe-filesystem-gcs.zip
# Deploy by unpacking into the plugin directory:
mkdir -p ${DORIS_HOME}/plugins/filesystem/gcs
unzip fe-filesystem/fe-filesystem-gcs/target/doris-fe-filesystem-gcs.zip \
  -d ${DORIS_HOME}/plugins/filesystem/gcs/

# The unpacked layout should be:
#   plugins/filesystem/gcs/
#   ├── doris-fe-filesystem-gcs.jar
#   └── lib/
#       └── *.jar
#
# NOTE: Do NOT drop the .zip file directly — it must be unpacked.

Checklist

Module depends only on fe-filesystem-spi (not fe-core/fe-common/fe-catalog)
FileSystemProvider has a public no-arg constructor
supports() is cheap — no network calls
META-INF/services file is present and correct
Assembly descriptor excludes fe-filesystem-api, fe-filesystem-spi, fe-extension-spi
Module is listed in fe-filesystem/pom.xml <modules>
Unit tests pass; environment tests are tagged @Tag("environment")

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

README.md

Doris FE Filesystem Module

Module Structure

Layering

How It Works

Plugin Discovery & Loading

Provider Selection

Plugin Packaging & Class Loading

S3 Delegation Pattern

Relationship to Other Modules

Adding a New Filesystem Sub-Module

1. Create the Maven module

2. Write `pom.xml`

3. Write the assembly descriptor

4. Implement `FileSystemProvider`

5. Implement `FileSystem`

6. Register via ServiceLoader

7. Register the module in the parent POM

8. Add tests

9. Build and deploy

Checklist

Name		Name	Last commit message	Last commit date
parent directory ..
fe-filesystem-api		fe-filesystem-api
fe-filesystem-azure		fe-filesystem-azure
fe-filesystem-broker		fe-filesystem-broker
fe-filesystem-cos		fe-filesystem-cos
fe-filesystem-hdfs		fe-filesystem-hdfs
fe-filesystem-http		fe-filesystem-http
fe-filesystem-local		fe-filesystem-local
fe-filesystem-obs		fe-filesystem-obs
fe-filesystem-oss		fe-filesystem-oss
fe-filesystem-s3		fe-filesystem-s3
fe-filesystem-spi		fe-filesystem-spi
ENV_TEST_README.md		ENV_TEST_README.md
README.md		README.md
pom.xml		pom.xml

Uh oh!

FilesExpand file tree

fe-filesystem

Directory actions

More options

Directory actions

More options

Latest commit

History

fe-filesystem

Folders and files

parent directory

README.md

Doris FE Filesystem Module

Module Structure

Layering

How It Works

Plugin Discovery & Loading

Provider Selection

Plugin Packaging & Class Loading

S3 Delegation Pattern

Relationship to Other Modules

Adding a New Filesystem Sub-Module

1. Create the Maven module

2. Write pom.xml

3. Write the assembly descriptor

4. Implement FileSystemProvider

5. Implement FileSystem

6. Register via ServiceLoader

7. Register the module in the parent POM

8. Add tests

9. Build and deploy

Checklist

2. Write `pom.xml`

4. Implement `FileSystemProvider`

5. Implement `FileSystem`