Filesystem over HTTP spec. Looking for review / comments

spec.md

HTTP Filesystem Protocol Specification

Overview

The HTTP Filesystem Protocol provides a RESTful interface for performing POSIX-like filesystem operations over HTTP. It enables hierarchical file and directory manipulation using standard HTTP methods with filesystem-specific metadata encoded in HTTP headers.

Protocol Design

Core Principles

• RESTful Interface: HTTP methods map directly to filesystem operations
• Metadata in Headers: File attributes encoded as HTTP headers
• Path-based URLs: Filesystem paths map directly to URL paths
• Directory Listings: Directory contents encoded as plain text
• Atomic Operations: Individual operations are atomic

URL Structure

Filesystem paths map directly to HTTP URLs:

• Base URL: https://example.com/fs
• File path: /path/to/file.txt → https://example.com/fs/path/to/file.txt
• Directory path: /path/to/dir → https://example.com/fs/path/to/dir

Note: Directory paths MAY end with / as a convenience to automatically set Content-Type: application/x-directory.

HTTP Methods

GET - Read File/Directory

Retrieves file content or directory listing.

Request:

GET /path/to/file.txt HTTP/1.1

Response (File):

HTTP/1.1 200 OK
Content-Type: application/octet-stream
Content-Length: 1024
Content-Mode: 33188
Content-Modified: 1641024000
Content-Ownership: 1000:1000

[file content]

Response (Directory):

HTTP/1.1 200 OK
Content-Type: application/x-directory
Content-Length: 45
Content-Mode: 16877
Content-Modified: 1641024000
Content-Ownership: 1000:1000

file.txt 33188
subdir 16877

HEAD - Get Metadata Only

Retrieves file/directory metadata without content.

Request:

HEAD /path/to/file.txt HTTP/1.1

Response:

HTTP/1.1 200 OK
Content-Type: application/octet-stream
Content-Length: 1024
Content-Mode: 33188
Content-Modified: 1641024000
Content-Ownership: 1000:1000

PUT - Create/Replace File/Directory

Creates or completely replaces a file or directory.

Request (File):

PUT /path/to/file.txt HTTP/1.1
Content-Type: application/octet-stream
Content-Length: 12
Content-Mode: 33188
Content-Modified: 1641024000
Content-Ownership: 1000:1000

Hello World!

Request (Directory):

PUT /path/to/dir HTTP/1.1
Content-Type: application/x-directory
Content-Length: 0
Content-Mode: 16877
Content-Modified: 1641024000
Content-Ownership: 1000:1000

Alternative (using trailing slash convenience):

PUT /path/to/dir/ HTTP/1.1
Content-Length: 0
Content-Mode: 16877
Content-Modified: 1641024000
Content-Ownership: 1000:1000

Response:

HTTP/1.1 200 OK

OK

PATCH - Update Metadata Only

Updates file/directory metadata without changing content.

Request:

PATCH /path/to/file.txt HTTP/1.1
Content-Mode: 33261

Response:

HTTP/1.1 200 OK

OK

DELETE - Remove File/Directory

Removes a file or directory.

Request:

DELETE /path/to/file.txt HTTP/1.1

Response:

HTTP/1.1 200 OK

OK

Filesystem Metadata Headers

Content-Mode

• Purpose: Unix file mode (permissions + type)
• Format: Decimal string representation of Unix mode
• Required: No (defaults applied)
• Examples:
  • 33188 - Regular file with 0644 permissions
  • 16877 - Directory with 0755 permissions
  • 33261 - Executable file with 0755 permissions

Content-Modified

• Purpose: Last modification timestamp
• Format: Unix timestamp (seconds since epoch) as decimal string
• Required: No (current time used if omitted)
• Example: 1641024000

Content-Ownership

• Purpose: File owner and group
• Format: uid:gid format
• Required: No (defaults to 0:0)
• Example: 1000:1000

Content-Type

• Purpose: MIME type indicator
• Values:
  • application/x-directory - Directory
  • application/octet-stream - Binary file (default)
  • Other standard MIME types as appropriate
• Required: No (auto-detected from path and content)

Content-Length

• Purpose: Size of content in bytes
• Format: Decimal string
• Required: Yes for PUT requests
• Behavior: Standard HTTP header

Directory Listing Format

Directory contents are encoded as plain text with the format:

filename mode
dirname mode

Characteristics:

• One entry per line
• Space-separated name and mode
• Lexicographically sorted
• Unix mode in decimal format
• Terminated with newline

Example:

.hidden 33188
README.md 33188
bin 16877
src 16877

File vs Directory Detection

1. Primary: Content-Type header (application/x-directory)
2. Secondary: Mode value (directory flag in Unix mode)
3. Convenience: Path ending with / automatically sets directory content-type

Error Responses

HTTP Status Codes

• 200 OK - Operation successful
• 404 Not Found - File/directory does not exist
• 405 Method Not Allowed - HTTP method not supported
• 412 Precondition Failed - Conditional request failed

Error Bodies

Error responses include plain text descriptions:

HTTP/1.1 404 Not Found

Object Not Found

HTTP/1.1 405 Method Not Allowed
Allow: GET, HEAD, PUT, PATCH, DELETE

Method Not Allowed

Operation Semantics

File Operations

Reading Files:

• GET retrieves content with metadata in headers
• HEAD retrieves only metadata
• Returns 404 if file doesn't exist

Writing Files:

• PUT creates/replaces entire file
• All metadata must be provided to preserve existing values
• Content-Length header required

Modifying Files:

• PATCH updates metadata only, content unchanged
• Only provided headers are updated
• Existing metadata preserved if not specified

Directory Operations

Reading Directories:

• GET returns directory listing as plain text
• Content-Type is application/x-directory
• Entries sorted lexicographically

Creating Directories:

• PUT with Content-Type: application/x-directory header
• Empty or minimal content body
• Alternatively, PUT with path ending in / automatically sets directory content-type

Directory Maintenance:

• Server automatically maintains parent directory listings
• Adding/removing files updates parent directory
• PATCH with Content-Mode updates entry in parent listing

Protocol Extensions

Conditional Requests

Standard HTTP conditional headers supported:

• If-Match / If-None-Match
• If-Modified-Since / If-Unmodified-Since

Range Requests

Standard HTTP range requests supported for partial file reads:

• Range: bytes=0-1023

Security Considerations

Authentication & Authorization

• Protocol is transport-agnostic regarding authentication
• Implementations should use standard HTTP authentication
• Access control is implementation-specific

Path Security

• Implementations should validate paths to prevent directory traversal
• Relative path components (., ..) require careful handling
• Path injection attacks should be prevented

Implementation Guidelines

Server Requirements

• MUST support GET, HEAD, PUT, PATCH, DELETE methods
• MUST detect directories via Content-Type: application/x-directory
• SHOULD treat paths ending with / as convenience for setting directory content-type
• MUST maintain directory listings automatically
• SHOULD support conditional requests
• SHOULD validate metadata format

Client Requirements

• MUST set Content-Type: application/x-directory for directory operations
• MAY use trailing / on directory paths as convenience
• MUST send required headers on PUT operations
• MUST parse directory listing format correctly
• SHOULD handle standard HTTP error responses
• SHOULD support conditional requests

Interoperability

• Metadata header names are case-insensitive (per HTTP)
• Directory listing format is strict (space-separated, sorted)
• Unix mode values are decimal integers
• Timestamps are Unix epoch seconds

Example Workflows

Creating a File

PUT /documents/readme.txt HTTP/1.1
Content-Type: text/plain
Content-Length: 13
Content-Mode: 33188
Content-Ownership: 1000:1000

Hello, World!

Reading a Directory

GET /documents HTTP/1.1

# Response:
readme.txt 33188
scripts 16877

Changing File Permissions

PATCH /documents/script.sh HTTP/1.1
Content-Mode: 33261

Checking File Existence

HEAD /documents/config.json HTTP/1.1

# 200 = exists, 404 = doesn't exist

Future Considerations

Planned Extensions

• Extended attributes (xattrs) support
• Symbolic link operations
• File locking mechanisms
• Bulk operations
• Directory watching/notifications

Limitations

• No atomic multi-file operations
• No recursive directory operations
• Access time not currently supported
• No built-in versioning or conflict resolution

Hey, this is cool, like a simpler webdav. I think I get why you wouldn't use it, but I do like how they make up their own HTTP methods like LOCK.

1. Is the application/x-directory text output a standard?

• What do you think about directories having a trailing slash? I suppose the content mode is all you need though. Here's a snippet of how a jekyll repo would look:

  _config.yml 33188
  _pages/ 16877
  assets/ 16877
  Gemfile 33188
  README.md 33188
  

2. Why do you have to create directories, instead of allowing something like PUT /foo/bar/baz.txt?
3. Just brainstorming: file moves, copies, and links could be something like POST /foo/bar/baz.txt?op=mv&dest=/new/loc.
4. At the risk of overloading POST: POST /foo/bar/baz.txt?lock=write&scope=shared&contact=URL-or-EMAIL
5. Does the API let you see the UIDs and GIDs? Does seem weird that PUT/PATCH let you set them, though. I would think it assumes all writes are from the authenticated user.
6. POST /foo/bar?notify=URL or GET /foo/bar;changes for a stream. But how do you structure the response without introducing a format? Maybe just send out [timestamp] [path] [mode]\n as changes happen?

Thanks for the comments!

(re WebDav + LOCK)

Since WebDAV deployments are still around, I wonder if it's safe to use LOCK with modern proxies and platforms. I'll look into it when I get around to locking.

Is the application/x-directory text output a standard?

Not that I know of. Old iterations used JSON, but seemed weird to use JSON for just this one thing. This is basically the simplest thing to satisfy io/fs.DirEntry.

What do you think about directories having a trailing slash?

They originally did! But it extended to the URL where you needed a trailing slash for directories. It started creating problems, so I removed it, and then that included in directory entries. I still support trailing slash as convenience, so maybe they can be supported here optionally as well. Since it is much easier to read as a human than the file mode.

Why do you have to create directories ...

That would be nice! You could ask the same thing of Unix. I didn't add this because my use case is actually mounting and using in a POSIX environment and you'd have to create them anyway.

file moves, copies, and links could be something like ...

Symlinks will be easy since its just a flag and the target path as the file contents (Linux used to do it this way).

My filesystem toolkit has fallbacks for move and copy, but it's also made to use the operation if it supports it. At the very least I want to support move/rename. I think the POST action pattern is reasonable, though ideally as a resource. Either way, I'd feel weird having only certain operations work this way.

COPY was added in WebDav right? If it turns out I can use custom verbs (specifically webdav), I'd use that for copy.

I could see PUT taking an argument for copy/move ... but also would the path be relative to the domain or the filesystem base path? Hmm.

Does the API let you see the UIDs and GIDs?

Content-Ownership is a work in progress placeholder. My current use case will be protected by auth and I'd love it if POSIX user permissions could be enforced and used, but it gets tricky! Also considering using strings like Plan 9. Maybe it can support both and its up to you to do any mapping.

(re change streams)

Definitely planning file watching. But I'm waiting to finalize the API I want to use in my toolkit first. I thought I started a discussion on Wanix for it, but I guess I haven't yet. Previous implementations at the HTTP level were just GET /path?watch to get change notifications. And at this point if there isn't a good format I have no reservations making something up if its "the simplest thing."

That would be nice! You could ask the same thing of Unix.

If they'd have consulted me, I'd have said the same thing (j/k!). I suppose I accept higher level convenience in web APIs than in filesystems.

Symlinks will be easy since its just a flag and the target path as the file contents (Linux used to do it this way).

Oh yeaaaah. I do like the idea of just uploading a file with a flag and a path. But the symlink approach seems to be the opposite of the COPY verb.

• # api mounted at https://foo.com/fs
  PUT /fs/path/to/symlink
  Source: https://foo.com/fs/path/to/orig
  

• COPY /fs/path/to/orig
  Destination: https://foo.com/fs/path/to/copy
  

• MOVE /fs/path/to/orig
  Destination: https://foo.com/fs/path/to/renamed
  

• SYMLINK /fs/path/to/orig
  Destination: https://foo.com/fs/path/to/symlink
  

I think headers should use full URLs, even if that's a little more work.

So I've implemented MOVE and COPY now based on WebDAV with the Destination header and so far it's going well. The spec wants full URLs but so far it's just relative, but I think it should ultimately be a full URL too. Also used the Overwrite header to let user decide if it should error if it exists. I did not implement the Depth header because it's not super helpful IMO. If it's a directory it's always recursive.

And I implemented symlinks as a file mode and content-type (similar to directory, since it specifies the meaning of the body). This makes the most sense to me because it's not doing anything special other than creating a file with file mode. Whereas COPY and MOVE are potentially compound operations, and they don't need the headers of PUT since they get that from the source file(s).