Skip to content

Instantly share code, notes, and snippets.

@brianellin

brianellin/atproto-llms.txt

Created June 11, 2025 16:14
Show Gist options
  • Save brianellin/47a0c16d36dfa6341b4fd616aa0e253f to your computer and use it in GitHub Desktop.
Save brianellin/47a0c16d36dfa6341b4fd616aa0e253f to your computer and use it in GitHub Desktop.
Download ZIP
atproto-llms.txt
Raw
atproto-llms.txt
# Home
# Welcome to the Atmosphere
The AT Protocol is an open, decentralized network for building social applications. {{ className: 'lead' }}
## Learn more
## Looking for the Bluesky API docs?
Go to [docs.bsky.app](https://docs.bsky.app) for Bluesky-specific documentation.
--------------------------------------------------------------------------------
# articles > atproto-ethos
# Atproto Ethos
*Apr 4, 2025*
*This post is a blogpost version of a [recent talk](https://www.youtube.com/watch?v=1A-0k58TfPo) that Daniel Holmgren gave at AtmosphereConf (March 2025).*
AT Protocol (or atproto) is a protocol for creating decentralized social applications.
It's not the first protocol with that aim to exist. In the history of decentralized social media protocols, atproto takes a unique approach which is still deeply influenced by technologies and movements that came before it.
The phrase “atproto ethos” often comes up during our protocol design discussions. It's a fuzzy term, but we use it to refer to the philosophical and aesthetic principles that underlie the design of the network.
In this post, we'll distill that ethos. First, we look at the movements in technology that have most directly influenced atproto.Then, we pull out the core innovations that atproto brought to the table. Finally, we highlight some opinionated ways of thinking that influenced the design.
## Atproto at the intersection of three movements
### The web
The first and most obvious influence on atproto is the web itself.
The web is an information system for publishing and accessing documents over the internet. It's permissionless to join and interact with. This guarantee is predicated on similar guarantees in the underlying internet. However, if the internet is functioning properly and you have a computer, an internet connection, and an IP address you can host a document on the web.
The permissionlessness of the web requires distributed authority of content. That is, there isn't a central authority that decides what is included in the web or if some content is “legitimate” or not. Rather, each computer that publishes and hosts a document is the authority for that document. This authority is **location-based**, meaning you can find out what the document hosted at a given site is by talking to that site and receiving the document. This sounds a little obvious when spelled out, but it's actually a radical idea. There's no need for distributors, document certifiers or central registries of documents.
While this openness is the defining governance feature of the web, the defining product feature is the hyperlink. A hyperlink is a link in a document that navigates you to another document. Notably, these hyperlinks can link *across authorities* without permission of the other authority. The result of this is that the web is not just a collection of unrelated documents published in isolation. Rather, documents on the web form a vast integrated network of interconnected data.
The web described here, the document-based web, is commonly referred to as Web 1.0. But Web 2.0, the web that we've been living with for the last 20 years, is quite different in its shape and function.
The modern web mostly takes place on a few large platforms like Facebook or Youtube where users have accounts and post content or view and interact with content from other accounts. These interactions and content often don't look like “documents” in the traditional sense. The content is all still deeply interconnected, but those connections are generally restricted to the boundaries of the platform that they are created through.
This new web is significantly easier to participate in — you don't need to know how to run a server or have a static IP address in order to publish content on the web. You can simply create an account. Perhaps most
The evolution of the document-based web into the modern web should not be seen as a failure of the web. Rather, it's a testament to its success. The idea, ethos, and primitives that the web was built on were powerful and flexible enough to facilitate the creation of essentially a second layer to the web. However, in this evolution, we also lost something.
### Peer-to-peer
The peer-to-peer (p2p) movement was in many ways a reaction to the emergence of the massive platforms that came to dominate the modern web. Idealistic and radical (sometimes to a fault), the p2p movement asked if we could remove backend applications from the mix entirely.
Instead of sending content and interactions to some coordinating backend service, users of social media platforms would send relevant data directly from their device to another device. If two users follow one another and one of them posts, the other would receive the post through a direct connection with the former's device. Of course, this requires that both parties are online at the same time for any data transfer to happen. P2p systems sought to get around this by allowing data to be replicated across many devices. Therefore a user may receive a post from the posting user or they could receive it from another friend that's currently online and in possession of the post.
With this, p2p moved away from the traditional location-based authority model of the web and towards “self-certifying data”. Data could come from a device that didn't originally create that data, and even if the content is coming from the canonical device, that device probably doesn't have a static IP associated with the account. Therefore p2p protocols moved the authority of data from the location of the data to the data itself.
P2p data is often addressed by the hash of the data and signed by a private key that the posting user holds. With this, content can be arbitrarily redistributed while maintaining full authority that it came from the user that posted it. Therefore a client can simply reach out to “the network” in the abstract to find some content with a known hash. When functioning well, content-addressed networks have this beautiful quality where the data seems to “just exist”, somewhere out there in this mesh of cooperating devices. Unlike the traditional location-addressed web where the data you receive is contingent on the service that you talk to, data in p2p systems has this essential quality to it - the data, not the location, is in primacy.
Of course the abstraction doesn't always work perfectly, and this new topology brought with it a host of problems - most notably data availability and discovery. As noted earlier, if the poster of some data is not online, you need to find that data from another device in the network. Discovering which device can serve you some data is a hard problem, especially with devices routinely coming on and offline at different addresses. And it may be the case that there is not a single online device that can send you the data that you're looking for! Even if everything is functioning well, these systems have to lean in hard to eventual consistency since network partitions are much more common and there is no service in the network that can purport to have a “global view” of all data.
The other major issue with these systems is that the rich social media sites that users had become accustomed to in the modern web rely on large compute and storage intensive indices. Getting servers out of the mix meant that all of that resource usage moved onto users' devices. Timeline generation, thread construction, aggregating interaction counts, social graph queries, and more all had to be done on relatively underpowered end user devices.
The end result is that products built on these p2p protocols often feel spotty - posts and interactions are missing, global algorithms are impossible to create, and running these applications takes a noticeable hit on the battery life and resource usage of the devices they run on. In short, it doesn't seem tractable to replicate the seamless dynamic feel of the massive platforms of the modern web.
### Data-intensive distributed systems
While peer-to-peer technology was being explored, the platforms that make up the modern web continued to grow in size. As the barriers to participating in the web lowered, millions and eventually billions of people started participating in and generating content on the web.
The content on these platforms isn't just documents that can be served statically. Rather these platforms process billions of interactions every day. These interactions in turn are composed into rich and dynamic views to serve to users: profile pages, news feeds, threads, like counts, and more.
Simply processing the data is not enough, as these platforms are very read heavy. For every piece of content that is created, platforms may have to serve hundreds or thousands of requests to other users. As these platforms became more encompassing, user expectations increased as well in regards to their availability and latency. Downtime and degradation of services became less tolerated and could cost companies millions of dollars.
All of these factors required new tools and architectures for building services. An overview of these techniques is maybe best captured in Martin Kleppman's book “Designing Data Intensive Applications”. While no two platforms are built exactly the same and there are many competing tools and methodologies, we can still sum up the general trends.
Because these platforms are so read heavy, generally they split apart read and write load so they can scale separately. A simple version of this is to introduce read replicas of a database. While there is still a single “leader” database, many replicas can help serve the outsized read load.
However, the sheer volume of data and the compute required to support these platforms cannot be handled by a single machine. Therefore even within a given platform's deployment, work must be split up across many machines. A common pattern is to peel off pieces of the application into smaller “micro-services” that can call into one another and then compose these micro-services into the larger platform. By isolating the concerns of each subsystem, teams can scale each according to its particular needs. For example, feed generation looks very different from video transcoding, and these workloads can benefit from being hosted on machines tailored to their workload.
The introduction of distributed systems also introduces distributed problems. Decomposed micro-services give up on the idea of maintaining a strongly consistent view of all the data on the platform. In some cases this means the use of eventually consistent databases tailored to high throughput, such as Scylla (which we use at Bluesky). Databases like Scylla give up the ACID guarantees and support for rich dynamic queries that more traditional SQL databases offer. Rather they require the indices and access patterns of the data to be known ahead of time.
The introduction of new database technologies and the general trend of micro-services brought even more problems. Micro-services often need access to the same data. If a micro-service goes down for a period of time, it needs a mechanism to catch up with everything that happened. And applications need a way to create new indices even over past data. An approach of stream processing helped to answer many of these issues. In a stream processing architecture - most prominently exemplified by Kafka - you maintain an event log of all events that flow through a system. Many consumers tap into this event log. If a consumer goes offline, when it comes back it can pick back up where it left off. And if a new index needs to be created, the entire event log can simply be replayed. These stream processing tools serve as a “backbone” of canonical data that enable the platform to coalesce around a single source of truth.
### Synthesis
Atproto is situated as the synthesis of these three movements.
1. **From the web:** an open, permissionless, and universal network of interconnected content.
2. **From peer-to-peer:** location-independent data, self-certifying data, and skepticism of centralized control of any aspect of the user’s experience.
3. **From data-intensive distributed systems:** a splitting of read and write load, application-aware secondary indices to facilitate high-throughput and low latency, streaming canonical data, and the decomposition of monoliths into microservices.
## Innovations
From this basis, atproto adds two core innovations: identity-based authority and the separation of data hosting from the rich applications built on top of it.
### Identity-based authority
The data model of atproto is very similar to that of p2p networks. The major difference is that rather than being simply content-addressed, a layer of indirection is added to make the data identity-addressed. Therefore an at-uri looks like `at://username/record-type/record-key`.
This identity-addressing of data is an inversion of the web-based model of location-addressed data. In the traditional web, the location (or the server) is the primary object; in atproto, the user is the primary object.
Identities are long-lived and untethered from any location where they may currently be housed. The primacy that data has in p2p networks is moved up the stack to identity. Identities exist independent of any given host or application and can move between them fluidly. Data in the network then flows from these identities with a well-defined mechanism of how to resolve the identity to some set of data and receive real time updates as new content is created.
Identities themselves resolve to key material that can verify the authenticity of that identity's data as well as a particular host that consumers can reach out to to get the canonical current state of the identity's data. These canonical hosts of user data are known as Personal Data Servers (PDSs).
While there is a canonical host for every identity, the data that comes out of that host is self-certifying and self-describing. This means that it can be arbitrarily rebroadcast out into the network. Consumers of user data do not ever need to reach out to the canonical host of the data. Instead, they can rely on external services that crawl the canonical data hosts and output a stream of updates for some well-defined set of the network.
The data that an identity outputs takes the form of a repository of “records”. Each record is a piece of structured data - a post, a like, a follow, a profile, etc. In the web, the primary object is the document and documents are linked together through hyperlinks. In atproto, the primary object is the record, and records are linked together through at-uri references.
### Generic hosting, Centralized product development
The data model (the repository) that PDSs adhere to is generic. The PDS as a whole actually has a very simple job: host the users data and manage their account/identity. This simplicity is intentional and
The other side of moving application-specific details out of the PDS is that it also gives applications the liberty to experiment and rapidly iterate on application semantics. Product development can occur in a centralized way. This centralization is critical to the ability to build new products in the network. We took a mentality of “no steps backwards” when it came to product development. Projects should be able to use modern tooling and product/application development patterns. Products benefit from the ability to rapidly iterate which requires clear ownership of application domains.
This ownership is reflected in the schema language for the atproto network where the notion of ownership is baked into the IDs of schemas. The owner of those schemas is free to iterate and evolve the data model for the application. Similarly, application owners are free to build their backends in whichever way they see fit. There is a boundary to the “weirdness” of the atproto network. Once identities and data cross that threshold, it is the application's decision how to index it and present it to the user.
Even though product development is centralized, the underlying data and identity remain open and universally accessible as a result of building on atproto. Put another way, ownership is clear for the evolution of a given application, but since the data is open, it can be reused, remixed, or extended by anyone else in the network.
## Opinionated takes
### Structure gives freedom
Atproto is a multi-party, low-coordination network. Services can join permissionlessly and operate under their own policies. While coordination pains are to some degree inevitable, atproto tries to head off the worst of it by being a very structured protocol. You'll see this reflected in many design decisions in the protocol: records encoded as canonical cbor, multiformats everywhere, a unicit (deterministic insertion-order-independent construction) data structure in the repository, a constrained set of allowed dids/hash functions, even things like requiring DPoP in OAuth.
While there's something empowering about the idea of being able to do *anything*, it's also easy for this to fall into the tyranny of structurelessness - a collapse in coordination that prevents anything from actually getting done. Without structure in the network, energy that could go into novel development gets redirected into facilitating interoperation, fixing edgecases between implementations, building up defenses to bad actors or security issues from other parties, and trying to coordinate evolution without a clear leader.
This structure is probably best exemplified by the schema language of atproto: Lexicon. Unlike other approaches to interoperation which attempt to create a generalized way of expressing properties on data (RDF) or to translate data between different domains to facilitate interoperation (lenses), Lexicon provides a way to determine if two things are speaking the same language, and to give a very structured mechanism for ensuring that the data is formatted properly.
With this, atproto avoids the gray area of “we're sort of talking about the same thing but not really” and forces applications into the discrete choice that if you're talking about the same thing then you're talking about the *exact* same thing. Instead of trying to posit universality of semantics, atproto facilitates specific semantics and then allows for collaboration and extension of those semantics.
### Lazy trust
The web requires a fair bit of trust with the services you interact with. While the transport layer is secured, the data that you receive from a given server is attested to solely by the fact that you received it from that server. If you load a post from a friend on Facebook for instance, you have to trust Facebook that they're serving you the correct post text. Similarly you have to trust Facebook to be a good custodian of your account and your data.
P2p networks took a different approach that was radically trust-less. You have to trust your client device of course, but that's it. Because the software runs on your client and the trust is imbued into the data, no trust in any other service or actor in the network is required.
Atproto takes a middle approach that we sometimes refer to as “lazy trust”. Like p2p networks, trust is imbued into the data. However most often when browsing the network, users are seeing computed views not the canonical data. When viewing a timeline for instance, clients are likely not checking the signatures on every post that comes down. This is mitigated by the fact that the canonical data is always at hand. Therefore the service that offers these computed views is *staking its reputation* on every view that it serves. If a service starts serving invalid views, this can be verifiably proven. And since the data is locked open, users can migrate to another service that is better behaved.
As well, unlike p2p networks, users maintain a relationship with a persistent backend (the PDS) that hosts their data and information about their identity. Most notably (while not a hard requirement in atproto), we made the decision to move signing keys up to the backend. This prevents the complex key management UX issues that come with client-side keys. However, there is an obvious loss in user-control by doing so. This is mitigated once again by lazy trust. Trust is moved a layer up into the identity system such that a user can hold a recovery key for their identity that allows them to migrate away from a bad PDS.
Operating in a trusted manner is by nature more efficient than operating trustlessly. Generally, this speaks to a philosophy of atproto which is to take advantage of the performance and UX gains that come with operating in a high-trust manner. However, when doing so always maintain the ability for credible exit and a system of checks and balances to hedge against bad actors and shoddy service providers.
## Conclusion
Taking all of this into account, we can see a general shape for a protocol and network emerge.
AT Protocol builds on the philosophy of the web with the technology of peer-to-peer protocols and the practices of data-intensive distributed systems.
We hold identity in primacy and center it in the protocol design. Canonical user data exists in a fluid and commoditized hosting network that allows rich applications to be built on it.
And we approach the design of this network erring on the side of structure and hoping to take advantage of high-trust environments where possible, but always allowing for credible exit if that relationship turns sour.
If this sounds exciting to you, you can dive in further by checking out the [protocol specs](https://atproto.com/specs/atp), or getting hands on with the [Statusphere](https://github.com/bluesky-social/statusphere-example-app) example app built on the protocol.
--------------------------------------------------------------------------------
# articles > atproto-for-distsys-engineers
# ATProto for distributed systems engineers
*Sep 3, 2024*
AT Protocol is the tech developed at [Bluesky](https://bsky.app) for open social networking. In this article we're going to explore AT Proto from the perspective of distributed backend engineering.
If you've ever built a backend with [stream-processing](https://milinda.pathirage.org/kappa-architecture.com/), then you're familiar with the kind of systems we'll be exploring. If you're not — no worries! We'll step through it.
## Scaling the traditional Web backend
The classic, happy Web architecture is the “one big SQL database” behind our app server. The app talks to the database and handles requests from the frontend.
<Container>
</Container>
As our application grows, we hit some performance limits so we toss some caches into the stack.
<Container>
</Container>
Then let's say we scale our database horizontally through sharding and replicas.
<Container>
</Container>
This is pretty good, but we're building a social network with hundreds of millions of users; even this model hits limits. The problem is that our SQL database is “[strongly consistent](https://en.wikipedia.org/wiki/Strong_consistency)” which means the state is kept uniformly in sync across the system. Maintaining strong consistency incurs a performance cost which becomes our bottleneck.
If we can relax our system to use “[eventual consistency](https://en.wikipedia.org/wiki/Eventual_consistency),” we can scale much further. We start by switching to a NoSQL cluster.
<Container>
</Container>
This is better for scaling, but without SQL it's becoming harder to build our queries. It turns out that SQL databases have a lot of useful features, like JOIN and aggregation queries. In fact, our NoSQL database is really just a key-value store. Writing features is becoming a pain!
To fix this, we need to write programs which generate precomputed views of our dataset. These views are essentially like cached queries. We even duplicate the canonical data into these views so they're very fast.
We'll call these our View servers.
<Container>
</Container>
Now we notice that keeping our view servers synced with the canonical data in the NoSQL cluster is tricky. Sometimes our view servers crash and miss updates. We need to make sure that our views stay reliably up-to-date.
To solve this, we introduce an event log (such as [Kafka](https://kafka.apache.org/)). That log records and broadcasts all the changes to the NoSQL cluster. Our view servers listen to &mdash; and replay &mdash; that log to ensure they never miss an update, even when they need to restart.
<Container>
</Container>
We've now arrived at a [stream processing architecture](https://milinda.pathirage.org/kappa-architecture.com/), and while there are a lot more details we could cover, this is enough for now.
The good news is that this architecture scales pretty well. We've given up strong consistency and sometimes our read queries lag behind the most up to date version of the data, but the service doesn't drop writes or enter an incorrect state.
In a way, what we've done is custom-built a database by [turning it inside-out](https://www.youtube.com/watch?v=fU9hR3kiOK0). We simplified the canonical storage into a NoSQL cluster, and then built our own querying engine with the view servers. It's a lot less convenient to build with, but it scales.
## Decentralizing our high-scale backend
The goal of AT Protocol is to interconnect applications so that their backends share state, including user accounts and content.
How can we do that? If we look at our diagram, we can see that most of the system is isolated from the outside world, with only the App server providing a public interface.
<Container>
</Container>
Our goal is to break this isolation down so that other people can join our NoSQL cluster, our event log, our view servers, and so on.
Here's how it's going to look:
<Container>
</Container>
Each of these internal services are now external services. They have public APIs which anybody can consume. On top of that, anybody can create their own instances of these services.
Our goal is to make it so anybody can contribute to this decentralized backend. That means that we don't just want one NoSQL cluster, or one View server. We want lots of these servers working together. So really it's more like this:
<Container>
</Container>
So how do we make all of these services work together?
## Unifying the data model
We're going to establish a shared data model called the [“user data repository.”](/guides/data-repos)
<Container>
</Container>
Every data repository contains JSON documents, which we'll call “records”.
<Container>
</Container>
For organizational purposes, we'll bucket these records into “collections.”
<Container>
</Container>
Now we're going to opinionate our NoSQL services so they all use this [data repository](/guides/data-repos) model.
<Container>
</Container>
Remember: the data repo services are still basically NoSQL stores, it's just that they're now organized in a very specific way:
1. Each user has a data repository.
2. Each repository has collections.
3. Each collection is an ordered K/V store of JSON documents.
Since the data repositories can be hosted by anybody, we need to give them [URLs](/specs/at-uri-scheme).
<Container>
</Container>
While we're at it, let's create a [whole URL scheme](/specs/at-uri-scheme) for our records too.
<Container>
</Container>
Great! Also, since we're going to be syncing these records around the Internet, it would be a good idea to cryptographically sign them so that we know they're authentic.
<Container>
</Container>
## Charting the flow of data
Now that we've set up our high-scale decentralized backend, let's map out how an application actually works on ATProto.
Since we're making a new app, we're going to want two things: an app server (which hosts our API & frontend) and a view server (which collects data from the network for us). We often bundle the app & view servers, and so we can just call it an “Appview.” Let's start there:
<Container>
</Container>
A user logs into our app using OAuth. In the process, they tell us which server hosts their data repository, _and_ they give us permission to read and write to it.
<Container>
</Container>
We're off to a good start &mdash; we can read and write JSON documents in the user's repo. If they already have data from other apps (like a profile) we can read that data too. If we were building a singleplayer app, we'd already be done.
But let's chart what happens when we write a JSON document.
<Container>
</Container>
This commits the document to the repo, then fires off a write into the event logs which are listening to the repo.
<Container>
</Container>
From there, the event gets sent to any view services that are listening &mdash; including our own!
<Container>
</Container>
Why are we listening to the event stream if we're the one making the write? Because we're not the only ones making writes! There are lots of user repos generating events, and lots of apps writing to them!
<Container>
</Container>
So we can see a kind of circular data flow throughout our decentralized backend, with writes being committed to the data repos, then emitted through the event logs into the view servers, where they can be read by our applications.
<Container>
</Container>
And (one hopes) that this network continues to scale: not just to add capacity, but to create a wider variety of applications sharing in this open applications network.
<Container>
</Container>
## Building practical open systems
The AT Protocol merges p2p tech with high-scale systems practices. Our founding engineers were core [IPFS](https://en.wikipedia.org/wiki/InterPlanetary_File_System) and [Dat](https://en.wikipedia.org/wiki/Dat_(software)) engineers, and Martin Kleppmann &mdash; the author of [Data Intensive Applications](https://www.oreilly.com/library/view/designing-data-intensive-applications/9781491903063/) &mdash; is an active technical advisor.
Before Bluesky was started, we established a clear requirement of “no steps backwards.” We wanted the network to feel as convenient and global as every social app before it, while still working as an open network. This is why, when we looked at federation and blockchains, the scaling limits of those architectures stood out to us. Our solution was to take standard practices for high scale backends, and then apply the techniques we used in peer-to-peer systems to create an open network.
--------------------------------------------------------------------------------
# guides > account-lifecycle
# Account Lifecycle Best Practices
This document complements the [Account Hosting](/specs/account) specification, which gives a high-level overview of account lifecycles. It summarizes the expected behaviors for a few common account lifecycle transitions, and what firehose events are expected in what order. Software is generally expected to be resilient to partial or incorrect event transmission.
**New Account Creation:** when an account is registered on a PDS and a new identity (DID) is created.
- the PDS will generate or confirm the existence of the account's identity (DID and handle). Once the DID is in a confirmed state that can be resolved by other services in the network, and points to the current PDS instance, the PDS emits an `#identity` event. It is good, but not required, to wait until the handle is resolvable by third parties before emitting the event (especially, but not only, if the PDS is providing a handle for the account). The account status may or may not be `active` when this event is emitted.
- once the account creation has completed, and the PDS will respond with `active` to API requests for account status, an `#account` event can be emitted
- when the account's repository is initialized with a `rev` and `commit`, a `#commit` message can be emitted. The initial repo may be "empty" (no records), or may contain records.
- the specific order of events is not formally specified, but the recommended order is: `#identity`, `#account`, `#commit`
- downstream services process and pass through these events
**Account Migration:** is described in more detail below, but the relevant events and behaviors are:
- the new PDS will not emit any events on initial account creation. The account state will be `deactivated` on the new PDS (which will reply as such to API requests)
- when the identity is updated and confirmed by the new PDS, it should emit an `#identity` event
- when the account is switched to `active` on the new PDS, it should emit an `#account` event and a `#commit` event; the order is not formally required, but doing `#account` first is recommended. Ideally the `#commit` event will be empty (no new records), but signed with any new signing key, and have a new/incremented `rev`.
- when the account is deactivated on the old PDS, it should emit an `#account` event, indicating that the account is inactive and has status `deactivated`.
- Relays should ignore `#account` and `#commit` events which are not coming from the currently declared PDS instance for the identity: these should not be passed through to the output firehose. Further, they should ignore `#commit` events when the local account status is not `active`. Overall, this means that account migration should result in three events coming from the relay: an `#identity` (from new PDS), an `#account` (from new PDS), and a `#commit` (from new PDS). The `#account` from the old PDS is usually ignored.
- downstream services (eg, AppView) should update their identity cache, and increment the account's `rev` (when the `#commit` is received), but otherwise don't need to take any action.
**Account Deletion:**
- PDS emits an `#account` event, with `active` false and status `deleted`.
- Relay updates local account status for the repo, and passes through the `#account` event. If the Relay is a full mirror, it immediately stop serving `getRepo`, `getRecord`, and similar API requests for the account, indicating the reason in the response error. The Relay may fully delete repo content locally according to local policy. The firehose backfill window does not need to be immediately purged of commit events for the repo, as long as the backfill window is time-limited.
- the PDS should not emit `#commit` events for an account which is not "active'. If any further `#commit` messages are emitted for the repo (eg, by accident or out-of-order processing or delivery), all downstream services should ignore the event and not pass it through
- downstream services (eg, AppView) should immediately stop serving/distributing content for the account. They may defer permanent data deletion according to local policy. Updating aggregations (eg, record counts) may also be deferred or processed in a background queue according to policy and implementation. Error status messages may indicate either that the content is "gone" (existed, but no longer available), or that content is "not found" (not revealing that content existed previously)
Account takedowns work similarly to account deletion.
**Account Deactivation:**
- PDS emits an `#account` event, with `active` false and status `deactivated`.
- similar to deletion, Relay processes the event, stops redistributing content, and passes through the event. The Relay should not fully purge content locally, though it may eventually delete local copies if the deactivation status persists for a long time (according to local policy).
- similar to deletion, `#commit` events should not be emitted by the PDS, and should be ignored and not passed through if received by Relays
- downstream services (eg, AppViews) should make content unavailable, but do not need to delete data locally. They should indicate account/content status as "unavailable"; best practice is to specifically indicate that this is due to account deactivation.
Account suspension works similarly to deactivation.
**Account Reactivation:**
- PDS emits an `#account` event, with `active` status.
- Relay verifies that the account reactivation is valid, eg that it came from the current PDS instance for the identity. It updates local account status, and passes through the event.
- any downstream services (eg, AppViews) should update local account status for the account.
- any service which does not have any current repo content for the account (eg, because it was previously deleted) may fetch a repo CAR export and process it as a background tasks. An “upstream” host (like a relay) may have a repo copy, or the service might connect directly to the account’s PDS host. They are not required to do so, and might instead wait for a `#commit` event.
- if the account was previously deleted or inactive for a long time, it is a best practice for the PDS to emit an empty `#commit` event after reactivation to ensure downstream services are synchronized
--------------------------------------------------------------------------------
# guides > account-migration
# Account Migration Details
This document complements the [Account Hosting](/specs/account) specification, which gives a high-level overview of account lifecycles. It breaks down the individual migration steps, both for an "easy" migration (when both PDS instances participate), and for account recovery scenarios. Note that these specific mechanisms are not a formal part of the protocol, and may evolve over time.
## Creating New Account
To create a PDS account with an existing identity, it is necessary to prove control the identity.
For an active account on another PDS, this is done by generating a service auth token (JWT) signed with the current atproto signing key indicated in the identity DID document. This can be requested from the previous PDS instance using the `com.atproto.server.getServiceAuth` endpoint (or an equivalent interface/API).
For an independently-controlled identity (eg, `did:web`, or a `did:plc` with old PDS offline or uncooperative), this may involve updating the identity to include a self-controlled atproto signing key, and generating the service auth token offline.
The service auth token is provided along with the existing DID when creating the account with `com.atproto.server.createAccount` (or an equivalent interface/API) on the new PDS.
The new account will be in a `deactivated` state. It should be possible to directly login and authenticate, but not to participate in the network. From the perspective of other services in the network, the old PDS account is still current, and the new PDS account is not yet active or valid. Functionality like OAuth or proxied requests using service auth will not yet work with the new PDS.
## Migrating Data
Some categories of data that are typically migrated are:
- public repository
- public blobs (media files)
- private preferences
At any stage of migration, the authenticated `com.atproto.server.checkAccountStatus` endpoint can be called on either the old or new PDS instance to check statistics about currently indexed data.
A copy of the repository can be fetched as a CAR file from the old PDS using the public `com.atproto.sync.getRepo` endpoint. If the old PDS is inaccessible, a mirror might be available from a public relay, or a local backup might be available. If not, the new account will still function with the same identity, but old content would be missing.
A CAR file can be
Blobs (media files) are download and re-uploaded one by one. They should not be uploaded to the new PDS until the repository has been
Private account preferences can be exported from the old PDS using the authenticated like `app.bsky.actor.getPreferences` endpoint, then
## Updating Identity
Once content has been migrated, the identity (DID and handle) can be updated to indicate that the new PDS is the current host for the account.
"Recommended" DID document parameters can be fetched from the new PDS using the `com.atproto.identity.getRecommendedDidCredentials` endpoint. This will include the DID service hostname, local handle (as requested during account creation), a PDS-managed atproto signing key (public key), and (if relevant) PLC rotation keys (public keys).
For users who are able to securely manage a private cryptographic keypair (eg, store in a password manager or digital wallet), it is recommended to include a self-controlled PLC rotation key (public key) in the PLC operation.
For a self-controlled identity (eg, `did:web`, or `did:plc` with local rotation key), the identity update can be done directly by the user.
For an account with a `did:plc` managed by the old PDS, a PLC "operation" is signed by the old PDS, then submitted via the new PDS. The motivation for having the new PDS submit the PLC operation instead of having the user do so directly is to give the new PDS a chance to validate the operation and do safety check to prevent the account from getting in a broken state.
Because identity operations are sensitive, they require an additional security token as an additional "factor". The token can be requested via `com.atproto.identity.requestPlcOperationSignature` on the old PDS, and will be delivered by email to the verified account email by default.
This token is included as part of a call to `com.atproto.identity.signPlcOperation` on the old PDS, along with the requested DID fields (new signing key, rotation keys, PDS location, etc). The old PDS will validate the request, sign using the PDS-managed PLC rotation key, and return the signed PLC operation. The operation will not have been submitted to any PLC directory at this point in time.
The user is then recommended to submit the operation to the new PDS (using the `com.atproto.identity.submitPlcOperation` endpoint), which will validate that the changes are "safe" (aka, that they enable the the PDS to help manage the identity and atproto account), and then submit it to the PLC directory.
With the identity successfully updated, the new PDS is now the "current" host for the account from the perspective of the entire network. This will be immediately apparent to new services which resolve the identity. Existing services that consume from the firehose will be alerted by the `#identity` event. Other services, which may have now-stale cached identity metadata for the account, will either refresh when the cache expires, or should refresh their cache when they encounter errors (such as invalid service auth signatures).
However, the new account is not yet "active".
## Finalizing Account Status
At this point, the user is still able to authenticate to both PDS instances. The new PDS knows that it is current for the account, but still has the account marked as "deactivated". The old PDS may not realize that it is no longer current.
It may be worth double-checking with `com.atproto.server.checkAccountStatus` on both PDS instances to confirm that all the expected content has been migrated.
The user can activate their account on the new PDS with a call to `com.atproto.server.activateAccount`, and deactivate their account on the old PDS with `com.atproto.server.deactivateAccount`.
At this point the migration is complete. New content can be published by writing to the repo, preferences can be updated, and inter-service auth and proxying should work as expected. It may be necessary to log out of any clients and log back in. In some cases, if services have aggressive identity caching and do not refresh on signature failure, service auth requests could fail for up to 24 hours.
It will still be possible to login and authenticate with the old PDS. The user may wish to fully terminated their old account eventually. This can be automated with the `deleteAfter` parameter to the `com.atproto.server.deactivateAccount` request. Note that the old PDS may be able to assist with PLC identity recovery during a fixed 72hr window, but only if the account was not fully deleted during that window.
--------------------------------------------------------------------------------
# guides > applications
# Quick start guide to building applications on AT Protocol
<Link href="https://github.com/bluesky-social/statusphere-example-app" className="not-prose flex items-center gap-2 bg-blue-100 dark:bg-blue-950 dark:text-white px-4 py-3 text-base rounded-lg hover:underline">
<svg viewBox="0 0 20 20" aria-hidden="true" className="h-6 dark:text-white">
<path
fill="currentColor"
fillRule="evenodd"
clipRule="evenodd"
d="M10 1.667c-4.605 0-8.334 3.823-8.334 8.544 0 3.78 2.385 6.974 5.698 8.106.417.075.573-.182.573-.406 0-.203-.011-.875-.011-1.592-2.093.397-2.635-.522-2.802-1.002-.094-.246-.5-1.005-.854-1.207-.291-.16-.708-.556-.01-.567.656-.01 1.124.62 1.281.876.75 1.292 1.948.93 2.427.705.073-.555.291-.93.531-1.143-1.854-.213-3.791-.95-3.791-4.218 0-.929.322-1.698.854-2.296-.083-.214-.375-1.09.083-2.265 0 0 .698-.224 2.292.876a7.576 7.576 0 0 1 2.083-.288c.709 0 1.417.096 2.084.288 1.593-1.11 2.291-.875 2.291-.875.459 1.174.167 2.05.084 2.263.53.599.854 1.357.854 2.297 0 3.278-1.948 4.005-3.802 4.219.302.266.563.78.563 1.58 0 1.143-.011 2.061-.011 2.35 0 .224.156.491.573.405a8.365 8.365 0 0 0 4.11-3.116 8.707 8.707 0 0 0 1.567-4.99c0-4.721-3.73-8.545-8.334-8.545Z"
/>
</svg>
<span>
Find the source code for the example application on GitHub.
</span>
</Link>
In this guide, we're going to build a simple multi-user app that publishes your current "status" as an emoji. Our application will look like this:
We will cover how to:
- Signin via OAuth
- Fetch information about users (profiles)
- Listen to the network firehose for new data
- Publish data on the user's account using a custom schema
We're going to keep this light so you can quickly wrap your head around ATProto. There will be links with more information about each step.
## Introduction
Data in the Atmosphere is stored on users' personal repos. It's almost like each user has their own website. Our goal is to aggregate data from the users into our SQLite DB.
Think of our app like a Google. If Google's job was to say which emoji each website had under `/status.json`, then it would show something like:
- `nytimes.com` is feeling 📰 according to `https://nytimes.com/status.json`
- `bsky.app` is feeling 🦋 according to `https://bsky.app/status.json`
- `reddit.com` is feeling 🤓 according to `https://reddit.com/status.json`
The Atmosphere works the same way, except we're going to check `at://` instead of `https://`. Each user has a data repo under an `at://` URL. We'll crawl all the user data repos in the Atmosphere for all the "status.json" records and aggregate them into our SQLite database.
> `at://` is the URL scheme of the AT Protocol. Under the hood it uses common tech like HTTP and DNS, but it adds all of the features we'll be using in this tutorial.
## Step 1. Starting with our ExpressJS app
Start by cloning the repo and installing packages.
```bash
git clone https://github.com/bluesky-social/statusphere-example-app.git
cd statusphere-example-app
cp .env.template .env
npm install
npm run dev
# Navigate to http://localhost:8080
```
Our repo is a regular Web app. We're rendering our HTML server-side like it's 1999. We also have a SQLite database that we're managing with [Kysely](https://kysely.dev/).
Our starting stack:
- Typescript
- NodeJS web server ([express](https://expressjs.com/))
- SQLite database ([Kysely](https://kysely.dev/))
- Server-side rendering ([uhtml](https://www.npmjs.com/package/uhtml))
With each step we'll explain how our Web app taps into the Atmosphere. Refer to the codebase for more detailed code &mdash; again, this tutorial is going to keep it light and quick to digest.
## Step 2. Signing in with OAuth
When somebody logs into our app, they'll give us read & write access to their personal `at://` repo. We'll use that to write the status json record.
We're going to accomplish this using OAuth ([spec](https://github.com/bluesky-social/proposals/tree/main/0004-oauth)). Most of the OAuth flows are going to be handled for us using the [@atproto/oauth-client-node](https://github.com/bluesky-social/atproto/tree/main/packages/oauth/oauth-client-node) library. This is the arrangement we're aiming toward:
When the user logs in, the OAuth client will create a new session with their repo server and give us read/write access along with basic user info.
Our login page just asks the user for their "handle," which is the domain name associated with their account. For [Bluesky](https://bsky.app) users, these tend to look like `alice.bsky.social`, but they can be any kind of domain (eg `alice.com`).
```html
<!-- src/pages/login.ts -->
<form action="/login" method="post" class="login-form">
<input
type="text"
name="handle"
placeholder="Enter your handle (eg alice.bsky.social)"
required
/>
<button type="submit">Log in</button>
</form>
```
When they submit the form, we tell our OAuth client to initiate the authorization flow and then redirect the user to their server to complete the process.
```typescript
/** src/routes.ts **/
// Login handler
router.post(
'/login',
handler(async (req, res) => {
// Initiate the OAuth flow
const handle = req.body?.handle
const url = await oauthClient.authorize(handle, {
scope: 'atproto transition:generic',
})
return res.redirect(url.toString())
})
)
```
This is the same kind of SSO flow that Google or GitHub uses. The user will be asked for their password, then asked to confirm the session with your application.
When that finishes, the user will be sent back to `/oauth/callback` on our Web app. The OAuth client will store the access tokens for the user's server, and then we attach their account's [DID](https://atproto.com/specs/did) to the cookie-session.
```typescript
/** src/routes.ts **/
// OAuth callback to complete session creation
router.get(
'/oauth/callback',
handler(async (req, res) => {
// Store the credentials
const { session } = await oauthClient.callback(params)
// Attach the account DID to our user via a cookie
const cookieSession = await getIronSession(req, res)
cookieSession.did = session.did
await cookieSession.save()
// Send them back to the app
return res.redirect('/')
})
)
```
With that, we're in business! We now have a session with the user's repo server and can use that to access their data.
## Step 3. Fetching the user's profile
Why don't we learn something about our user? In [Bluesky](https://bsky.app), users publish a "profile" record which looks like this:
```typescript
interface ProfileRecord {
displayName?: string // a human friendly name
description?: string // a short bio
avatar?: BlobRef // small profile picture
banner?: BlobRef // banner image to put on profiles
createdAt?: string // declared time this profile data was added
// ...
}
```
You can examine this record directly using [atproto-browser.vercel.app](https://atproto-browser.vercel.app). For instance, [this is the profile record for @bsky.app](https://atproto-browser.vercel.app/at?u=at://did:plc:z72i7hdynmk6r22z27h6tvur/app.bsky.actor.profile/self).
We're going to use the [Agent](https://github.com/bluesky-social/atproto/tree/main/packages/api) associated with the user's OAuth session to fetch this record.
```typescript
await agent.com.atproto.repo.getRecord({
repo: agent.assertDid, // The user
collection: 'app.bsky.actor.profile', // The collection
rkey: 'self', // The record key
})
```
When asking for a record, we provide three pieces of information.
- **repo** The [DID](https://atproto.com/specs/did) which identifies the user,
- **collection** The collection name, and
- **rkey** The record key
We'll explain the collection name shortly. Record keys are strings with [some restrictions](https://atproto.com/specs/record-key#record-key-syntax) and a couple of common patterns. The `"self"` pattern is used when a collection is expected to only contain one record which describes the user.
Let's update our homepage to fetch this profile record:
```typescript
/** src/routes.ts **/
// Homepage
router.get(
'/',
handler(async (req, res) => {
// If the user is signed in, get an agent which communicates with their server
const agent = await getSessionAgent(req, res, ctx)
if (!agent) {
// Serve the logged-out view
return res.type('html').send(page(home()))
}
// Fetch additional information about the logged-in user
const { data: profileRecord } = await agent.com.atproto.repo.getRecord({
repo: agent.assertDid, // our user's repo
collection: 'app.bsky.actor.profile', // the bluesky profile record type
rkey: 'self', // the record's key
})
// Serve the logged-in view
return res
.type('html')
.send(page(home({ profile: profileRecord.value || {} })))
})
)
```
With that data, we can give a nice personalized welcome banner for our user:
```html
<!-- pages/home.ts -->
</form>`
: html`
</div>`}
</div>
```
## Step 4. Reading & writing records
You can think of the user repositories as collections of JSON records:
Let's look again at how we read the "profile" record:
```typescript
await agent.com.atproto.repo.getRecord({
repo: agent.assertDid, // The user
collection: 'app.bsky.actor.profile', // The collection
rkey: 'self', // The record key
})
```
We write records using a similar API. Since our goal is to write "status" records, let's look at how that will happen:
```typescript
// Generate a time-based key for our record
const rkey = TID.nextStr()
// Write the
await agent.com.atproto.repo.putRecord({
repo: agent.assertDid, // The user
collection: 'xyz.statusphere.status', // The collection
rkey, // The record key
record: { // The record value
status: "👍",
createdAt: new Date().toISOString()
}
})
```
Our `POST /status` route is going to use this API to publish the user's status to their repo.
```typescript
/** src/routes.ts **/
// "Set status" handler
router.post(
'/status',
handler(async (req, res) => {
// If the user is signed in, get an agent which communicates with their server
const agent = await getSessionAgent(req, res, ctx)
if (!agent) {
return res.status(401).type('html').send('<h1>Error: Session required</h1>')
}
// Construct their status record
const record = {
$type: 'xyz.statusphere.status',
status: req.body?.status,
createdAt: new Date().toISOString(),
}
try {
// Write the status record to the user's repository
await agent.com.atproto.putRecord({
repo: agent.assertDid,
collection: 'xyz.statusphere.status',
rkey: TID.nextStr(),
record,
})
} catch (err) {
logger.warn({ err }, 'failed to write record')
return res.status(500).type('html').send('<h1>Error: Failed to write record</h1>')
}
res.status(200).json({})
})
)
```
Now in our homepage we can list out the status buttons:
```html
<!-- src/pages/home.ts -->
<form action="/status" method="post" class="status-options">
${STATUS_OPTIONS.map(status => html`
<button class="status-option" name="status" value="${status}">
${status}
</button>
`)}
</form>
```
And here we are!
## Step 5. Creating a custom "status" schema
Repo collections are typed, meaning that they have a defined schema. The `app.bsky.actor.profile` type definition [can be found here](https://github.com/bluesky-social/atproto/blob/main/lexicons/app/bsky/actor/profile.json).
Anybody can create a new schema using the [Lexicon](https://atproto.com/specs/lexicon) language, which is very similar to [JSON-Schema](http://json-schema.org/). The schemas use [reverse-DNS IDs](https://atproto.com/specs/nsid) which indicate ownership. In this demo app we're going to use `xyz.statusphere` which we registered specifically for this project (aka statusphere.xyz).
> ### Why create a schema?
>
> Schemas help other applications understand the data your app is creating. By publishing your schemas, you make it easier for other application authors to publish data in a format your app will recognize and handle.
Let's create our schema in the `/lexicons` folder of our codebase. You can [read more about how to define schemas here](https://atproto.com/guides/lexicon).
```json
/** lexicons/status.json **/
{
"lexicon": 1,
"id": "xyz.statusphere.status",
"defs": {
"main": {
"type": "record",
"key": "tid",
"record": {
"type": "object",
"required": ["status", "createdAt"],
"properties": {
"status": {
"type": "string",
"minLength": 1,
"maxGraphemes": 1,
"maxLength": 32
},
"createdAt": {
"type": "string",
"format": "datetime"
}
}
}
}
}
}
```
Now let's run some code-generation using our schema:
```bash
./node_modules/.bin/lex gen-server ./src/lexicon ./lexicons/*
```
This will produce Typescript interfaces as well as runtime validation functions that we can use in our app. Here's what that generated code looks like:
```typescript
/** src/lexicon/types/xyz/statusphere/status.ts **/
export interface Record {
status: string
createdAt: string
[k: string]: unknown
}
export function isRecord(v: unknown): v is Record {
return (
isObj(v) &&
hasProp(v, '$type') &&
(v.$type === 'xyz.statusphere.status#main' || v.$type === 'xyz.statusphere.status')
)
}
export function validateRecord(v: unknown): ValidationResult {
return lexicons.validate('xyz.statusphere.status#main', v)
}
```
Let's use that code to improve the `POST /status` route:
```typescript
/** src/routes.ts **/
// ...
// "Set status" handler
router.post(
'/status',
handler(async (req, res) => {
// ...
// Construct & validate their status record
const record = {
$type: 'xyz.statusphere.status',
status: req.body?.status,
createdAt: new Date().toISOString(),
}
if (!Status.validateRecord(record).success) {
return res.status(400).json({ error: 'Invalid status' })
}
// ...
})
)
```
## Step 6. Listening to the firehose
So far, we have:
- Logged in via OAuth
- Created a custom schema
- Read & written records for the logged in user
Now we want to fetch the status records from other users.
Remember how we referred to our app as being like Google, crawling around the repos to get their records? One advantage we have in the AT Protocol is that each repo publishes an event log of their updates.
Using a [Relay service](https://docs.bsky.app/docs/advanced-guides/federation-architecture#relay) we can listen to an aggregated firehose of these events across all users in the network. In our case what we're looking for are valid `xyz.statusphere.status` records.
```typescript
/** src/ingester.ts **/
// ...
new Firehose({
filterCollections: ['xyz.statusphere.status'],
handleEvent: async (evt) => {
// Watch for write events
if (evt.event === 'create' || evt.event === 'update') {
const record = evt.record
// If the write is a valid status update
if (
evt.collection === 'xyz.statusphere.status' &&
Status.isRecord(record) &&
Status.validateRecord(record).success
) {
// Store the status
// TODO
}
}
},
})
```
Let's create a SQLite table to store these statuses:
```typescript
/** src/db.ts **/
// Create our statuses table
await db.schema
.createTable('status')
.addColumn('uri', 'varchar', (col) => col.primaryKey())
.addColumn('authorDid', 'varchar', (col) => col.notNull())
.addColumn('status', 'varchar', (col) => col.notNull())
.addColumn('createdAt', 'varchar', (col) => col.notNull())
.addColumn('indexedAt', 'varchar', (col) => col.notNull())
.execute()
```
Now we can write these statuses into our database as they arrive from the firehose:
```typescript
/** src/ingester.ts **/
// If the write is a valid status update
if (
evt.collection === 'xyz.statusphere.status' &&
Status.isRecord(record) &&
Status.validateRecord(record).success
) {
// Store the status in our SQLite
await db
.insertInto('status')
.values({
uri: evt.uri.toString(),
authorDid: evt.author,
status: record.status,
createdAt: record.createdAt,
indexedAt: new Date().toISOString(),
})
.onConflict((oc) =>
oc.column('uri').doUpdateSet({
status: record.status,
indexedAt: new Date().toISOString(),
})
)
.execute()
}
```
You can almost think of information flowing in a loop:
Applications write to the repo. The write events are then emitted on the firehose where they're caught by the apps and ingested into their databases.
Why sync from the event log like this? Because there are other apps in the network that will write the records we're interested in. By subscribing to the event log, we ensure that we catch all the data we're interested in &mdash; including data published by other apps!
## Step 7. Listing the latest statuses
Now that we have statuses populating our SQLite, we can produce a timeline of status updates by users. We also use a [DID](https://atproto.com/specs/did)-to-handle resolver so we can show a nice username with the statuses:
```typescript
/** src/routes.ts **/
// Homepage
router.get(
'/',
handler(async (req, res) => {
// ...
// Fetch data stored in our SQLite
const statuses = await db
.selectFrom('status')
.selectAll()
.orderBy('indexedAt', 'desc')
.limit(10)
.execute()
// Map user DIDs to their domain-name handles
const didHandleMap = await resolver.resolveDidsToHandles(
statuses.map((s) => s.authorDid)
)
// ...
})
)
```
Our HTML can now list these status records:
```html
<!-- src/pages/home.ts -->
${statuses.map((status, i) => {
const handle = didHandleMap[status.authorDid] || status.authorDid
return html`
</div>
</div>
`
})}
```
## Step 8. Optimistic updates
As a final optimization, let's introduce "optimistic updates."
Remember the information flow loop with the repo write and the event log?
Since we're updating our users' repos locally, we can short-circuit that flow to our own database:
This is an
To do this, we just update `POST /status` to include an additional write to our SQLite DB:
```typescript
/** src/routes.ts **/
// "Set status" handler
router.post(
'/status',
handler(async (req, res) => {
// ...
let uri
try {
// Write the status record to the user's repository
const res = await agent.com.atproto.repo.putRecord({
repo: agent.assertDid,
collection: 'xyz.statusphere.status',
rkey: TID.nextStr(),
record,
})
uri = res.uri
} catch (err) {
logger.warn({ err }, 'failed to write record')
return res.status(500).json({ error: 'Failed to write record' })
}
try {
// Optimistically update our SQLite <-- HERE!
await db
.insertInto('status')
.values({
uri,
authorDid: agent.assertDid,
status: record.status,
createdAt: record.createdAt,
indexedAt: new Date().toISOString(),
})
.execute()
} catch (err) {
logger.warn(
{ err },
'failed to update computed view; ignoring as it should be caught by the firehose'
)
}
res.status(200).json({})
})
)
```
You'll notice this code looks almost exactly like what we're doing in `ingester.ts`.
## Thinking in AT Proto
In this tutorial we've covered the key steps to building an atproto app. Data is published in its canonical form on users' `at://` repos and then aggregated into apps' databases to produce views of the network.
When building your app, think in these four key steps:
- Design the [Lexicon](#) schemas for the records you'll publish into the Atmosphere.
- Create a database for aggregating the records into useful views.
- Build your application to write the records on your users' repos.
- Listen to the firehose to aggregate data across the network.
Remember this flow of information throughout:
This is how every app in the Atmosphere works, including the [Bluesky social app](https://bsky.app).
## Next steps
If you want to practice what you've learned, here are some additional challenges you could try:
- Sync the profile records of all users so that you can show their display names instead of their handles.
- Count the number of each status used and display the total counts.
- Fetch the authed user's `app.bsky.graph.follow` follows and show statuses from them.
- Create a different kind of schema, like a way to post links to websites and rate them 1 through 4 stars.
--------------------------------------------------------------------------------
# guides > data-repos
# Data Repositories
A data repository is a collection of data published by a single user. Repositories are self-authenticating data structures, meaning each update is signed and can be verified by anyone.
They are described in more depth in the [Repository specification](/specs/repository).
## Data Layout
The content of a repository is laid out in a [Merkle Search Tree (MST)](https://hal.inria.fr/hal-02303490/document) which reduces the state to a single root hash. It can be visualized as the following layout:
```
┌────────────────┐
│ Commit │ (Signed Root)
└───────┬────────┘
↓
┌────────────────┐
│ Tree Nodes │
└───────┬────────┘
↓
┌────────────────┐
│ Record │
└────────────────┘
```
Every node is an [IPLD](https://ipld.io/) object ([dag-cbor](https://ipld.io/docs/codecs/known/dag-cbor/)) which is referenced by a [CID](https://github.com/multiformats/cid) hash. The arrows in the diagram above represent a CID reference.
This layout is reflected in the [AT URIs](/specs/at-uri-scheme):
```
Root | at://alice.com
Collection | at://alice.com/app.bsky.feed.post
Record | at://alice.com/app.bsky.feed.post/1234
```
A “commit” to a data repository is simply a keypair signature over a Root node’s CID. Each mutation to the repository produces a new Commit node.
## Identifier Types
Multiple types of identifiers are used within a Personal Data Repository.
<DescriptionList>
<Description title="DIDs"><a href="https://w3c.github.io/did-core/">Decentralized IDs (DIDs)</a> identify data repositories. They are broadly used as user IDs, but since every user has one data repository then a DID can be considered a reference to a data repository. The format of a DID varies by the “DID method” used but all DIDs ultimately resolve to a keypair and a list of service providers. This keypair can sign commits to the data repository.</Description>
<Description title="CIDs"><a href="https://github.com/multiformats/cid">Content IDs (CIDs)</a> identify content using a fingerprint hash. They are used throughout the repository to reference the objects (nodes) within it. When a node in the repository changes, its CID also changes. Parents which reference the node must then update their reference, which in turn changes the parent’s CID as well. This chains all the way to the Commit node, which is then signed.</Description>
<Description title="NSIDs"><a href="/specs/nsid">Namespaced Identifiers (NSIDs)</a> identify the Lexicon type for groups of records within a repository.</Description>
<Description title="rkey"><a href="/specs/record-key">Record Keys</a> ("rkeys") identify individual records within a collection in a given repository. The format is specified by the collection Lexicon, with some collections having only a single record with a key like "self", and other collections having many records, with keys using a base32-encoded timestamp called a Timestamp Identifier (TID).</Description>
</DescriptionList>
--------------------------------------------------------------------------------
# guides > faq
# FAQ
Frequently Asked Questions about the Authenticated Transfer Protocol (AT Proto). For FAQ about Bluesky, visit [here](https://bsky.social/about/faq).
## Is the AT Protocol a blockchain?
No. The AT Protocol is a [federated protocol](https://en.wikipedia.org/wiki/Federation_(information_technology)). It's not a blockchain nor does it use a blockchain.
## Why not use ActivityPub?
[ActivityPub](https://en.wikipedia.org/wiki/ActivityPub) is a federated social networking technology popularized by [Mastodon](https://joinmastodon.org/).
Account portability is a major reason why we chose to build a separate protocol. We consider portability to be crucial because it protects users from sudden bans, server shutdowns, and policy disagreements. Our solution for portability requires both [signed data repositories](/guides/data-repos) and [DIDs](/guides/identity), neither of which are easy to retrofit into ActivityPub. The migration tools for ActivityPub are comparatively limited; they require the original server to provide a redirect and cannot migrate the user's previous data.
Another major reason is scalability. ActivityPub depends heavily on delivering messages between a wide network of small-to-medium sized nodes, which can cause individual nodes to be flooded with traffic and generally struggles to provide global views of activity. The AT Protocol uses aggregating applications to merge activity from the users' hosts, reducing the overall traffic and dramatically reducing the load on individual hosts.
Other smaller differences include: a different viewpoint about how schemas should be handled, a preference for domain usernames over AP's double-@ email usernames, and the goal of having large scale search and algorithmic feeds.
## Why create Lexicon instead of using JSON-LD or RDF?
Atproto exchanges data and RPC commands across organizations. For the data and RPC to be useful, the software needs to correctly handle schemas created by separate teams. This is the purpose of [Lexicon](/guides/lexicon).
We want engineers to feel comfortable using and creating new schemas, and we want developers to enjoy the DX of the system. Lexicon helps us produce strongly typed APIs which are extremely familiar to developers and which provides a variety of runtime correctness checks.
[RDF](https://en.wikipedia.org/wiki/Resource_Description_Framework) is intended for extremely general cases in which the systems share very little infrastructure. It's conceptually elegant but difficult to use, often adding a lot of syntax which devs don't understand. JSON-LD simplifies the task of consuming RDF vocabularies, but it does so by hiding the underlying concepts, not by making RDF more legible.
We looked very closely at using RDF but just didn't love the developer experience (DX) or the tooling it offered.
## What is “XRPC,” and why not use ___?
[XRPC](/specs/xrpc) is HTTP with some added conventions. We're aiming to retire the term "XRPC" and just refer to it as the ATProto usage of HTTP.
XRPC uses [Lexicon](/guides/lexicon) to describe HTTP calls and maps them to `/xrpc/{methodId}`. For example, this API call:
```typescript
await api.com.atproto.repo.listRecords({
user: 'alice.com',
collection: 'app.bsky.feed.post'
})
```
...maps to:
```text
GET /xrpc/com.atproto.repo.listRecords
?user=alice.com
&collection=app.bsky.feed.post
```
Lexicon establishes a shared method id (`com.atproto.repo.listRecords`) and the expected query params, input body, and output body. By using Lexicon we get runtime checks on the inputs and outputs of the call, and can generate typed code like the API call example above.
--------------------------------------------------------------------------------
# guides > glossary
# Glossary of terms
The AT Protocol uses a lot of terms that may not be immediately familiar. This page gives a quick reference to these terms and includes some links to more information.
## Atmosphere
The "Atmosphere" is the term we use to describe the ecosystem around the [AT Protocol](#at-protocol).
## AT Protocol
The AT Protocol stands for "Authenticated Transfer Protocol," and is frequently shortened to "atproto." The name is in reference to the fact that all user-data is signed by the authoring users, which makes it possible to broadcast the data through many services and prove it's real without having to speak directly to the originating server.
The name is also a play on the "@" symbol, aka the "at" symbol, since atproto is designed for social systems.
## PDS (Personal Data Server)
A PDS, or Personal Data Server, is a server that hosts a user. A PDS will always store the user's [data repo](#data-repo) and signing keys. It may also assign the user a [handle](#handle) and a [DID](#did). Many PDSes will host multiple users.
A PDS communicates with [AppViews](#appview) to run applications. A PDS doesn't typically run any applications itself, though it will have general account management interfaces such as the OAuth login screen. PDSes actively sync their [data repos](#data-repo) with [Relays](#relay).
## AppView
An AppView is an application in the [Atmosphere](#atmosphere). It's called an "AppView" because it's just one view of the network. The canonical data lives in [data repos](#data-repo) which is hosted by [PDSes](#pds-personal-data-server), and that data can be viewed many different ways.
AppViews function a bit like search engines on the Web: they aggregate data from across the Atmosphere to produce their UIs. The difference is that AppViews also communicate with users' [PDSes](#pds) to publish information on their [repos](#data-repo), forming the full application model. This communication is established as a part of the OAuth login flow.
## Relay
A Relay is an aggregator of [data repos](#data-repo) from across the [Atmosphere](#atmosphere). They sync the repos from [PDSes](#pds) and produce a firehose of change events. [AppViews](#appview) use a Relay to fetch user data.
Relays are an optimization and are not strictly necessary. An [AppView](#appview) could communicate directly with [PDSes](#pds) (in fact, this is encouraged if needed). The Relay serves to reduce the number of connections that are needed in the network.
## Lexicon
Lexicon is a schema language. It's used in the [Atmosphere](#atmosphere) to describe [data records](#record) and HTTP APIs. Functionally it's very similar to [JSON-Schema](https://json-schema.org/) and [OpenAPI](https://www.openapis.org/).
Lexicon's sole purpose is to help developers build compatible software.
- [An introduction to Lexicon](/guides/lexicon)
- [Lexicon spec](/specs/lexicon)
## Data Repo
The "data repository" or "repo" is the public dataset which represents a user. It is comprised of [collections](#collection) of JSON [records](#record) and unstructured [blobs](#blob). Every repo is assigned a single permanent [DID](#did) which identifies it. Repos may also have any number of [domain handles](#handle) which act as human-readable names.
Data repositories are signed merkle trees. Their signatures can be verified against the key material published under the repo's [did](#did).
- [An introduction to data repos](/guides/data-repos)
- [Repository spec](/specs/repository)
## Collection
The "collection" is a bucket of JSON [records](#record) in a [data repository](#data-repo). They support ordered list operations. Every collection is identified by an [NSID](#nsid-namespaced-id) which is expected to map to a [Lexicon](#lexicon) schema.
## Record
A "record" is a JSON document inside a [repo](#data-repo) [collection](#collection). The type of a record is identified by the `$type` field, which is expected to map to a [Lexicon](#lexicon) schema. The type is also expected to match the [collection](#collection) which contains it.
- [Record key spec](/specs/record-key)
## Blob
Blobs are unstructured data stored inside a [repo](#data-repo). They are most commonly used to store media such as images and video.
## Label
Labels are metadata objects which are attached to accounts ([DIDs](#did-decentralized-id)) and [records](#record). They are typically referenced by their values, such as "nudity" or "graphic-media," which identify the meaning of the label. Labels are primarily used by applications for moderation, but they can be used for other purposes.
- [Labels spec](/specs/label)
## Handle
Handles are domain names which are used to identify [data repos](#data-repo). More than one handle may be assigned to a repo. Handles may be used in `at://` URIs in the domain segment.
- [Handle spec](/specs/handle)
- [URI Scheme spec](/specs/at-uri-scheme)
## DID (Decentralized ID)
DIDs, or Decentralized IDentifiers, are universally-unique identifiers which represent [data repos](#data-repo). They are permanent and non-human-readable. DIDs are a [W3C specification](https://www.w3.org/TR/did-core/). The AT Protocol currently supports `did:web` and `did:plc`, two different DID methods.
DIDs resolve to documents which contain metadata about a [repo](#data-repo), including the address of the repo's [PDS](#pds), the repo's [handles](#handle), and the public signing keys.
- [DID spec](/specs/did)
## NSID (Namespaced ID)
NSIDs, or Namespaced IDentifiers, are an identifier format used in the [Atmosphere](#atmosphere) to identify [Lexicon](#lexicon) schemas. They follow a reverse DNS format such as `app.bsky.feed.post`. They were chosen because they give clear schema governance via the domain ownership. The reverse-DNS format was chosen to avoid confusion with domains in URIs.
- [NSID spec](/specs/nsid)
## TID (Timestamp ID)
TIDs, or Timestamp IDentifiers, are an identifier format used for [record](#record) keys. They are derived from the current time and designed to avoid collisions, maintain a lexicographic sort, and efficiently balance the [data repository's](#data-repo) internal data structures.
- [Record keys spec](/specs/record-key)
## CID (Content ID)
CIDs, or Content Identifiers, are cryptographic hashes of [records](#record). They are used to track specific versions of records.
## DAG-CBOR
DAG-CBOR is a serialization format used by [atproto](#at-protocol). It was chosen because it provides a reliable canonical form, which is
- [Data model spec](/specs/data-model)
## XRPC
XRPC is a term we are deprecating, but it was historically used to describe [atproto's](#at-protocol) flavor of HTTP usage. It stood for "Cross-organizational Remote Procedure Calls" and we regret inventing it, because really we're just using HTTP.
- [HTTP API spec](/specs/xrpc)
--------------------------------------------------------------------------------
# guides > identity
# Identity
The atproto identity system has a number of requirements:
* **ID provision.** Users should be able to create global IDs which are stable across services. These IDs should never change, to ensure that links to their content are stable.
* **Public key distribution.** Distributed systems rely on cryptography to prove the authenticity of data. The identity system must publish their public keys with strong security.
* **Key rotation.** Users must be able to rotate their key material without disrupting their identity.
* **Service discovery.** Applications must be able to discover the services in use by a given user.
* **Usability.** Users should have human-readable and memorable names.
* **Portability.** Identities should be portable across services. Changing a provider should not cause a user to lose their identity, social graph, or content.
Using the atproto identity system gives applications the tools for end-to-end encryption, signed user data, service sign-in, and general interoperation.
## Identifiers
We use two interrelated forms of identifiers: _handles_ and _DIDs_. Handles are DNS names while DIDs are a [W3C standard](https://www.w3.org/TR/did-core/) with multiple implementations which provide secure & stable IDs. AT Protocol supports the DID PLC and DID Web variants.
The following are all valid user identifiers:
```
alice.host.com
at://alice.host.com
did:plc:bv6ggog3tya2z3vxsub7hnal
```
The relationship between them can be visualized as:
```
┌──────────────────┐ ┌───────────────┐
│ DNS name ├──resolves to──→ │ DID │
│ (alice.host.com) │ │ (did:plc:...) │
└──────────────────┘ └─────┬─────────┘
↑ │
│ resolves to
│ │
│ ↓
│ ┌───────────────┐
└───────────references───────┤ DID Document │
│ {"id":"..."} │
└───────────────┘
```
The DNS handle is a user-facing identifier — it should be displayed in user interfaces and promoted as a way to find users. Applications resolve handles to DIDs and then use the DID as the canonical identifier for accounts. Any DID can be rapidly resolved to a DID document which includes public keys and user services.
<DescriptionList>
<Description title="Handles">Handles are DNS names. They are resolved using DNS TXT records or an HTTP well-known endpoint, and must be confirmed by a matching entry in the DID document. Details in the <a href="/specs/handle">Handle specification</a>.</Description>
<Description title="DIDs">DIDs are a <a href="https://www.w3.org/TR/did-core/">W3C standard</a> for providing stable & secure IDs. They are used as stable, canonical IDs of users. Details of how they are used in AT Protocol in the <a href="/specs/did">DID specification</a>.</Description>
<Description title="DID Documents">
DID Documents are standardized JSON objects which are returned by the DID resolution process. They include the following information:
<ul>
<li>The handle associated with the DID.</li>
<li>The signing key.</li>
<li>The URL of the user’s PDS.</li>
</ul>
</Description>
</DescriptionList>
## DID Methods
The [DID standard](https://www.w3.org/TR/did-core/) describes a framework for different "methods" of publishing and resolving DIDs to the [DID Document](https://www.w3.org/TR/did-core/#core-properties), instead of specifying a single mechanism. A variety of existing methods [have been registered](https://w3c.github.io/did-spec-registries/#did-methods), with different features and properties. We established the following criteria for use with atproto:
- **Strong consistency.** For a given DID, a resolution query should produce only one valid document at any time. (In some networks, this may be subject to probabilistic transaction finality.)
- **High availability**. Resolution queries must succeed reliably.
- **Online API**. Clients must be able to publish new DID documents through a standard API.
- **Secure**. The network must protect against attacks from its operators, a Man-in-the-Middle, and other users.
- **Low cost**. Creating and updating DID documents must be affordable to services and users.
- **Key rotation**. Users must be able to rotate keypairs without losing their identity.
- **Decentralized governance**. The network should not be governed by a single stakeholder; it must be an open network or a consortium of providers.
When we started the project, none of the existing DID methods met all of these criteria. Therefore, we chose to support both the existing [did-web](https://w3c-ccg.github.io/did-method-web/) method (which is simple), and a novel method we created called [DID PLC](https://github.com/bluesky-social/did-method-plc).
## Handle Resolution
Handles in atproto are domain names which resolve to a DID, which in turn resolves to a DID Document containing the user's signing key and hosting service.
Handle resolution uses either a DNS TXT record, or an HTTPS well-known endpoint. Details can be found in the [Handle specification](/specs/handle).
### Example: Hosting service
Consider a scenario where a hosting service is using PLC and is providing the handle for the user as a subdomain:
- The handle: `alice.pds.com`
- The DID: `did:plc:12345`
- The hosting service: `https://pds.com`
At first, all we know is `alice.pds.com`, so we look up the DNS TXT record `_atproto.alice.pds.com`. This tells us the DID: `did:plc:12345`.
Next we query the PLC directory for the DID, so that we can learn the hosting service's endpoint and the user's key material.
```typescript
await didPlc.resolve('did:plc:12345') /* => {
id: 'did:plc:12345',
alsoKnownAs: `https://alice.pds.com`,
verificationMethod: [...],
service: [{serviceEndpoint: 'https://pds.com', ...}]
}*/
```
We can now communicate with `https://pds.com` to access Alice's data.
### Example: Self-hosted
Let's consider a self-hosting scenario. If it's using `did:plc`, it would look something like:
- The handle: `alice.com`
- The DID: `did:plc:12345`
- The hosting service: `https://alice.com`
However, **if the self-hoster is confident they will retain ownership of the domain name**, they can use `did:web` instead of `did:plc`:
- The handle: `alice.com`
- The DID: `did:web:alice.com`
- The hosting service: `https://alice.com`
We can resolve the handle the same way, resolving `_atproto.alice.com`, which returns the DID: `did:web:alice.com`
Which we then resolve:
```typescript
await didWeb.resolve('did:web:alice.com') /* => {
id: 'did:web:alice.com',
alsoKnownAs: `https://alice.com`,
verificationMethod: [...],
service: [{serviceEndpoint: 'https://alice.com', ...}]
}*/
```
--------------------------------------------------------------------------------
# guides > lexicon
# Intro to Lexicon
Lexicon is a schema system used to define RPC methods and record types. Every Lexicon schema is written in JSON, in a format similar to [JSON-Schema](https://json-schema.org/) for defining constraints.
The schemas are identified using [NSIDs](/specs/nsid) which are a reverse-DNS format. Here are some example API endpoints:
```
com.atproto.repo.getRecord
com.atproto.identity.resolveHandle
app.bsky.feed.getPostThread
app.bsky.notification.listNotifications
```
And here are some example record types:
```
app.bsky.feed.post
app.bsky.feed.like
app.bsky.actor.profile
app.bsky.graph.follow
```
The schema types, definition language, and validation constraints are described in the [Lexicon specification](/specs/lexicon), and representations in JSON and CBOR are described in the [Data Model specification](/specs/data-model).
## Why is Lexicon needed?
**Interoperability.** An open network like atproto needs a way to agree on behaviors and semantics. Lexicon solves this while making it relatively simple for developers to introduce new schemas.
**Lexicon is not RDF.** While RDF is effective at describing data, it is not ideal for enforcing schemas. Lexicon is easier to use because it doesn't need the generality that RDF provides. In fact, Lexicon's schemas enable code-generation with types and validation, which makes life much easier!
## HTTP API methods
The AT Protocol's API system, [XRPC](/specs/xrpc), is essentially a thin wrapper around HTTPS. For example, a call to:
```typescript
com.example.getProfile()
```
is actually just an HTTP request:
```text
GET /xrpc/com.example.getProfile
```
The schemas establish valid query parameters, request bodies, and response bodies.
```json
{
"lexicon": 1,
"id": "com.example.getProfile",
"defs": {
"main": {
"type": "query",
"parameters": {
"type": "params",
"required": ["user"],
"properties": {
"user": { "type": "string" }
},
},
"output": {
"encoding": "application/json",
"schema": {
"type": "object",
"required": ["did", "name"],
"properties": {
"did": {"type": "string"},
"name": {"type": "string"},
"displayName": {"type": "string", "maxLength": 64},
"description": {"type": "string", "maxLength": 256}
}
}
}
}
}
}
```
With code-generation, these schemas become very easy to use:
```typescript
await client.com.example.getProfile({user: 'bob.com'})
// => {name: 'bob.com', did: 'did:plc:1234', displayName: '...', ...}
```
## Record types
Schemas define the possible values of a record. Every record has a "type" which maps to a schema and also establishes the URL of a record.
For instance, this "follow" record:
```json
{
"$type": "com.example.follow",
"subject": "at://did:plc:12345",
"createdAt": "2022-10-09T17:51:55.043Z"
}
```
...would have a URL like:
```text
at://bob.com/com.example.follow/12345
```
...and a schema like:
```json
{
"lexicon": 1,
"id": "com.example.follow",
"defs": {
"main": {
"type": "record",
"description": "A social follow",
"record": {
"type": "object",
"required": ["subject", "createdAt"],
"properties": {
"subject": { "type": "string" },
"createdAt": {"type": "string", "format": "datetime"}
}
}
}
}
}
```
## Tokens
Tokens declare global identifiers which can be used in data.
Let's say a record schema wanted to specify three possible states for a traffic light: 'red', 'yellow', and 'green'.
```json
{
"lexicon": 1,
"id": "com.example.trafficLight",
"defs": {
"main": {
"type": "record",
"record": {
"type": "object",
"required": ["state"],
"properties": {
"state": { "type": "string", "enum": ["red", "yellow", "green"] },
}
}
}
}
}
```
This is perfectly acceptable, but it's not extensible. You could never add new states, like "flashing yellow" or "purple" (who knows, it could happen).
To add flexibility, you could remove the enum constraint and just document the possible values:
```json
{
"lexicon": 1,
"id": "com.example.trafficLight",
"defs": {
"main": {
"type": "record",
"record": {
"type": "object",
"required": ["state"],
"properties": {
"state": {
"type": "string",
"description": "Suggested values: red, yellow, green"
}
}
}
}
}
}
```
This isn't bad, but it lacks specificity. People inventing new values for state are likely to collide with each other, and there won't be clear documentation on each state.
Instead, you can define Lexicon tokens for the values you use:
```json
{
"lexicon": 1,
"id": "com.example.green",
"defs": {
"main": {
"type": "token",
"description": "Traffic light state representing 'Go!'.",
}
}
}
{
"lexicon": 1,
"id": "com.example.yellow",
"defs": {
"main": {
"type": "token",
"description": "Traffic light state representing 'Stop Soon!'.",
}
}
}
{
"lexicon": 1,
"id": "com.example.red",
"defs": {
"main": {
"type": "token",
"description": "Traffic light state representing 'Stop!'.",
}
}
}
```
This gives us unambiguous values to use in our trafficLight state. The final schema will still use flexible validation, but other teams will have more clarity on where the values originate from and how to add their own:
```json
{
"lexicon": 1,
"id": "com.example.trafficLight",
"defs": {
"main": {
"type": "record",
"record": {
"type": "object",
"required": ["state"],
"properties": {
"state": {
"type": "string",
"knownValues": [
"com.example.green",
"com.example.yellow",
"com.example.red"
]
}
}
}
}
}
}
```
## Versioning
Once a schema is published, it can never change its constraints. Loosening a constraint (adding possible values) will cause old software to fail validation for new data, and tightening a constraint (removing possible values) will cause new software to fail validation for old data. As a consequence, schemas may only add optional constraints to previously unconstrained fields.
If a schema must change a previously-published constraint, it should be published as a new schema under a new NSID.
## Schema distribution
Schemas are designed to be machine-readable and network-accessible. While it is not currently _required_ that a schema is available on the network, it is strongly advised to publish schemas so that a single canonical & authoritative representation is available to consumers of the method.
--------------------------------------------------------------------------------
# guides > overview
# Protocol Overview
The **Authenticated Transfer Protocol**, aka **atproto**, is a decentralized protocol for large-scale social web applications. This document will introduce you to the ideas behind the AT Protocol.
## Identity
Users in AT Protocol have permanent decentralized identifiers (DIDs) for their accounts. They also have a configurable domain name, which acts as a human-readable handle. Identities include a reference to the user's current hosting provider and cryptographic keys.
## Data Repositories
User data is exchanged in [signed data repositories](/guides/data-repos). These repositories are collections of records which include posts, comments, likes, follows, etc.
## Network Architecture
The AT Protocol has a federated network architecture, meaning that account data is stored on host servers, as opposed to a peer-to-peer model between end devices. Federation was chosen to ensure the network is convenient to use and reliably available. Repository data is synchronized between servers over standard web technologies ([HTTP](/specs/xrpc) and [WebSockets](/specs/event-stream)).
The three core services in our network are Personal Data Servers (PDS), Relays, and App Views. There are also supporting services such as feed generators and labelers.
The lower-level primitives that can get stacked together differently are the repositories, lexicons, and DIDs. We published an overview of our technical decisions around federation architecture [on our blog](https://bsky.social/about/blog/5-5-2023-federation-architecture).
## Interoperation
A global schemas network called [Lexicon](/specs/lexicon) is used to unify the names and behaviors of the calls across the servers. Servers implement "lexicons" to support featuresets, including the core `com.atproto.*` lexicons for syncing user repositories and the `app.bsky.*` lexicons to provide basic social behaviors.
While the Web exchanges documents, the AT Protocol exchanges schematic and semantic information, enabling the software from different organizations to understand each others' data. This gives atproto clients freedom to produce user interfaces independently of the servers, and removes the need to exchange rendering code (HTML/JS/CSS) while browsing content.
## Achieving Scale
Personal Data Servers are your home in the cloud. They host your data, distribute it, manage your identity, and orchestrate requests to other services to give you your views.
Relays collect data updates from many servers in to a single firehose.
App Views provide aggregated application data for the entire network. They support large-scale metrics (likes, reposts, followers), content discovery (algorithms), and user search.
This separation of roles is intended to provide users with choice between multiple interoperable providers, while also scaling to large network sizes.
## Algorithmic choice
As with Web search engines, users are free to select their aggregators. Feeds, labelers, and search indices can be provided by independent third parties, with requests routed by the PDS, based on client app configuration. Client apps may be tied to specific services, such as App Views or mandatory labelers.
## Account portability
We assume that a Personal Data Server may fail at any time, either by going offline in its entirety, or by ceasing service for specific users. The goal of the AT Protocol is to ensure that a user can migrate their account to a new PDS without the server's involvement.
User data is stored in [signed data repositories](/guides/data-repos) and authenticated by [DIDs](/guides/identity). Signed data repositories are like Git repos but for database records, and DIDs provide a directory of cryptographic keys, similar in some ways to the TLS certificate system. Identities are expected to be secure, reliable, and independent of the user's PDS.
Most DID documents publish two types of public keys: a signing key and rotation keys.
* **Signing key**: Validates the user's data repository. All DIDs include such a key.
* **Rotation keys**: Asserts changes to the DID Document itself. The PLC DID method includes this, while the DID Web method does not.
The signing key is entrusted to the PDS so that it can manage the user's data, but rotation keys can be controlled by the user, e.g. as a paper key. This makes it possible for the user to update their account to a new PDS without the original host's help.
A backup of the user’s data could be persistently synced to a user's own device as a backup (contingent on the disk space available), or mirrored by a third-party service. In the event their PDS disappears without notice, the user should be able to migrate to a new provider by updating their DID Document and uploading their data backup.
## Speech, reach, and moderation
AT Protocol's model is that _speech_ and _reach_ should be two separate layers, built to work with each other. The “speech” layer should remain permissive, distributing authority and designed to ensure everyone has a voice. The “reach” layer lives on top, built for flexibility and designed to scale.
The base layer of atproto (personal data repositories and federated networking) creates a common space for speech where everyone is free to participate, analogous to the Web where anyone can put up a website. The indexing services then enable reach by aggregating content from the network, analogous to a search engine.
## Specifications
Some of the primary specifications comprising the initial version of the AT Protocol are:
- [Authenticated Transfer Protocol](/specs/atp)
- [DIDs](/specs/did) and [Handles](/specs/handle)
- [Repository](/specs/repository) and [Data Model](/specs/data-model)
- [Lexicon](/specs/lexicon)
- [HTTP API (XRPC)](/specs/xrpc) and [Event Streams](/specs/event-stream)
--------------------------------------------------------------------------------
# guides > self-hosting
# PDS Self-hosting
Self-hosting a Bluesky PDS means running your own Personal Data Server that is capable of federating with the wider ATProto network.
## Table of Contents
* [Preparation for self-hosting PDS](#preparation-for-self-hosting-pds)
* [Open your cloud firewall for HTTP and HTTPS](#open-your-cloud-firewall-for-http-and-https)
* [Configure DNS for your domain](#configure-dns-for-your-domain)
* [Check that DNS is working as expected](#check-that-dns-is-working-as-expected)
* [Installer on Ubuntu 20.04/22.04 and Debian 11/12](#installer-on-ubuntu-20-04-22-04-and-debian-11-12)
* [Verifying that your PDS is online and accessible](#verifying-that-your-pds-is-online-and-accessible)
* [Creating an account using pdsadmin](#creating-an-account-using-pdsadmin)
* [Creating an account using an invite code](#creating-an-account-using-an-invite-code)
* [Using the Bluesky app with your PDS](#using-the-bluesky-app-with-your-pds)
* [Updating your PDS](#updating-your-pds)
* [Getting help](#getting-help)
## Preparation for self-hosting PDS
Launch a server on any cloud provider, [Digital Ocean](https://digitalocean.com/) and [Vultr](https://vultr.com/) are two popular choices.
Ensure that you can ssh to your server and have root access.
**Server Requirements**
* Public IPv4 address
* Public DNS name
* Public inbound internet access permitted on port 80/tcp and 443/tcp
**Server Recommendations**
| | |
| ---------------- | ------------ |
| Operating System | Ubuntu 22.04 |
| Memory (RAM) | 1 GB |
| CPU Cores | 1 |
| Storage | 20 GB SSD |
| Architectures | amd64, arm64 |
| Number of users | 1-20 |
**Note:** It is a good security practice to restrict inbound ssh access (port 22/tcp) to your own computer's public IP address. You can check your current public IP address using [ifconfig.me](https://ifconfig.me/).
## Open your cloud firewall for HTTP and HTTPS
One of the most common sources of misconfiguration is not opening firewall ports correctly. Please be sure to double check this step.
In your cloud provider's console, the following ports should be open to inbound access from the public internet.
* 80/tcp (Used only for TLS certification verification)
* 443/tcp (Used for all application requests)
**Note:** there is no need to set up TLS or redirect requests from port 80 to 443 because the Caddy web server, included in the Docker compose file, will handle this for you.
## Configure DNS for your domain
From your DNS provider's control panel, set up a domain with records pointing to your server.
| Name | Type | Value | TTL |
| --------------- | ---- | ------------- | --- |
| `example.com` | `A` | `12.34.56.78` | 600 |
| `*.example.com` | `A` | `12.34.56.78` | 600 |
**Note:**
* Replace `example.com` with your domain name.
* Replace `12.34.56.78` with your server's IP address.
* Some providers may use the `@` symbol to represent the root of your domain.
* The wildcard record is required when allowing users to create new accounts on your PDS.
* The TTL can be anything but 600 (10 minutes) is reasonable
## Check that DNS is working as expected
Use a service like [DNS Checker](https://dnschecker.org/) to verify that you can resolve domain names.
Examples to check (record type `A`):
* `example.com`
* `random.example.com`
* `test123.example.com`
These should all return your server's public IP.
## Installer on Ubuntu 20.04/22.04 and Debian 11/12
On your server via ssh, download the installer script using wget:
```bash
wget https://raw.githubusercontent.com/bluesky-social/pds/main/installer.sh
```
or download it using curl:
```bash
curl https://raw.githubusercontent.com/bluesky-social/pds/main/installer.sh >installer.sh
```
And then run the installer using bash:
```bash
sudo bash installer.sh
```
## Verifying that your PDS is online and accessible
> [!TIP]
> The most common problems with getting PDS content consumed in the live network are when folks substitute the provided Caddy configuration for nginx, apache, or similar reverse proxies. Getting TLS certificates, WebSockets, and virtual server names all correct can be tricky. We are not currently providing tech support for other configurations.
You can check if your server is online and healthy by requesting the healthcheck endpoint.
You can visit `https://example.com/xrpc/_health` in your browser. You should see a JSON response with a version, like:
```
{"version":"0.2.2-beta.2"}
```
You'll also need to check that WebSockets are working, for the rest of the network to pick up content from your PDS. You can test by installing a tool like `wsdump` and running a command like:
```bash
wsdump "wss://example.com/xrpc/com.atproto.sync.subscribeRepos?cursor=0"
```
Note that there will be no events output on the WebSocket until they are created in the PDS, so the above command may continue to run with no output if things are configured successfully.
## Creating an account using pdsadmin
Using ssh on your server, use `pdsadmin` to create an account if you haven't already.
```bash
sudo pdsadmin account create
```
## Creating an account using an invite code
Using ssh on your server, use `pdsadmin` to create an invite code.
```bash
sudo pdsadmin create-invite-code
```
When creating an account using the app, enter this invite code.
## Using the Bluesky app with your PDS
You can use the Bluesky app to connect to your PDS.
1. Get the Bluesky app
* [Bluesky for Web](https://bsky.app/)
* [Bluesky for iPhone](https://apps.apple.com/us/app/bluesky-social/id6444370199)
* [Bluesky for Android](https://play.google.com/store/apps/details?id=xyz.blueskyweb.app)
1. Enter the URL of your PDS (e.g. `https://example.com/`)
_Note: because the subdomain TLS certificate is created on-demand, it may take 10-30s for your handle to be accessible. If you aren't seeing your first post/profile, wait 30s and try to make another post._
## Updating your PDS
It is recommended that you keep your PDS up to date with new versions, otherwise things may break. You can use the `pdsadmin` tool to update your PDS.
```bash
sudo pdsadmin update
```
## Getting help
- [Visit the GitHub](https://github.com/bluesky-social/pds) for issues and discussions.
- [Join the AT Protocol PDS Admins Discord](https://discord.gg/e7hpHxRfBP) to chat with other folks hosting instances and get
--------------------------------------------------------------------------------
# specs > account
# Account Hosting
All users in the atproto network have an "identity", based around a unique and immutable DID. Active users also have an "account" on a Personal Data Server (PDS). The PDS provides a number of network services, including repository hosting, authorization and authentication, and blob storage. Accounts have a lifecycle (including deletions and takedowns). Identities may migrate between hosting providers. Downstream services, including relays and AppViews, may redistribute account content.
Each service which redistributes content must decide independently what accounts and content to host and redistribute. For example, they might focus on a subset of accounts in the network, define their own content policies, or have regional legal obligations. All services are expected to respect certain protocol-level account actions (described below), such as temporary account deactivation. Combined, each network service has a synthesized “hosting status” for each account they distribute public data for.
This document describes the various hosting states an account and identity can have in the network, what the expectations are for downstream services when states change, and how accounts can be migrated between hosting providers (PDS instances).
## Hosting Status
On any network service, a known account is either `active` or not (a boolean flag) at a given point in time. If an account is not active, the service should not redistribute content for that account (repositories, individual records, blobs, etc). If the account is unknown (never seen before), the state can be undefined.
Identity metadata (DID documents and handle status), and the account hosting status itself, are distinct from hosted content, and can be redistributed.
Network services are encouraged to implement API endpoints (such as `com.atproto.sync.getRepoStatus`) which describe current hosting status for individual accounts. If they expose an event stream, they should also emit `#account` events when their local account hosting status updates for an account.
In addition to the `active` boolean, an account might in a more specific state (all of which correspond with `active=false`):
- `deleted`: user or host has deleted the account, and content should be removed from the network. Implied permanent or long-term, though may be reverted (deleted accounts may reactivate on the same or another host).
- `deactivated`: user has temporarily paused their overall account. Content should not be displayed or redistributed, but does not need to be deleted from infrastructure. Implied time-limited. Also the initial state for an account after migrating to another PDS instance.
- `takendown`: host or service has takendown the account. Implied permanent or long-term, though may be reverted.
- `suspended`: host or service has temporarily paused the account. Implied time-limited.
New account states will be added in the future. To prevent broken expectations, relevant API endpoints (such as `com.atproto.sync.getRepoStatus`, and the `#account` event on the `com.atproto.sync.subscribeRepos` event stream) break out `active` as a boolean flag, and then clarify non-active accounts with a separate `status` string. Services should use the `active` flag to control overall account visibility (observable behavior) with the `status` string acting as clarification which might determine more specific infrastructure behaviors (such as data deletion).
The deactivation and suspension states are implied to be temporary, though they might have indefinite time periods. Deletion and takedowns are implied to be more final, but can still technically be reverted (and frequently are, in practice).
Downstream services might decide to delete content rapidly upon takedown (like deletion), or simply stop redistributing content until a later time.
Identities might also be tombstoned, which is also implied to be permanent and final, but can technically be reversed in certain conditions. If the account's identity is `tombstoned`, the account hosting status should be interpreted as `deleted`.
To summarize, when an account status is non-`active`, the content that hosts should not redistributed includes:
- repository exports (CAR files)
- repo records
- transformed records ("views", embeds, etc)
- blobs
- transformed blobs (thumbnails, etc)
### Hosting Status Propagation
Unlike other aspects of atproto, account status is not self-certifying, and there is not a method for authenticated transfer or redistributing of account status between arbitrary parties.
Most commonly, account status is propagated “hop by hop”, with each “downstream” service accepting and adopting the hosting status for accounts at their upstream. If an intermediate service (such as a relay) overrides the status of their upstream (for example, an infrastructure account takedown), they will broadcast and propagate that action downstream to subscribers.
By default, if an account’s current active PDS is not available, account status should remain unchanged, at least for some time period. This keeps the network resilient and accounts online during temporary infrastructure outages. Services may decide their own policies for accounts whose PDS hosts remain offline for long periods of time.
The general expectation is that downstream services will not list an account as `active` if it is inactive at an upstream service. However, the concept of “upstream” is informal, and services may need to define their own policies for some corner-cases. For example, an AppView might consume from multiple independent relay providers, who report differing statuses for the same account, due to policy differences. If the network connection between a relay and PDS is disrupted, they may report different statuses for the same account.
If there is doubt about about an account’s general status in the network, the account’s current active PDS host can be queried using the `com.atproto.sync.getRepoStatus` endpoint. If the account is not `active` there, it is generally expected to not be active elsewhere, especially in the case of user-initiated actions (deactivation or deletion).
### Content Deletion
Separate from overall account status, individual pieces of content may be removed. When accounts remove records from their public repositories, hosting services should remove public access to that content.
Applications which would benefit from explicit “tombstones” (indications that content previously existed) should explicitly design them in to Lexicon schemas. In the absence of explicit application-level tombstones, services are expected not to differentiate between content which has never existed and content which has been entirely deleted. Note that this is different from acknowledging account-level deletion, which is encouraged.
## PDS Account Migration
At a high level, the current PDS host for an account is indicated in the account’s DID document. If a DID document is updated to point at a new PDS host, the account status is `active` at that host, and there is a functioning repository (with valid signature and commit `rev`), then the account has effectively been migrated.
At a protocol level, this is how account migration works. And in some situations, when any previous PDS instance is unavailable or has inactive account status, getting in to the above state may be the only account recovery path available.
However, in the common case, when there is an active account on a functional current PDS, a more seamless account migration process is possible. It can summarized in a few high-level steps:
- creating an account on the new PDS (which may start with `active` false, such as `deactivated`)
- migrating data from the old PDS to the new PDS
- updating identity (DID document) to point to the new PDS and using a new atproto signing key. For PLC DIDs, usually involves updating PLC rotation keys as well.
- updating account status on both PDS instances, such that old is inactive, and new is active
- emitting a `#commit` event from the new PDS, signed by the new atproto signing key, with a higher commit `rev`.
## Usage and Implementation Guidelines
One possible account migration flow is described in detail in a [separate guide](/guides/account-migration). Note that the specific mechanisms described are not formally part of the protocol, and are not the only way to migrate accounts.
Guidelines for specific firehose event sequencing during different account events are described in an [Account Lifecycle Best Practices guide](/guides/account-lifecycle).
## Security Considerations
Account hosting status is not authenticated, and is specific to every individual network service. The current active PDS host is a good default for querying overall account state, though it may not represent broad network consensus (eg, intermediary takedowns), and it is technically possible for PDS hosts to misrepresent account activation state.
Processing account status changes may be resource intensive for downstream services. Rate-limits at the account (DID) level and upstream service provider (eg, PDS) level are recommended to prevent resource exhaustion attacks from bad actors. It may be appropriate for relevant accounts to be put in an inactive state (eg, `suspended`) in such a situation, to prevent the accounts from being "locked open".
Control of identity (DID and handle) is critical in the atproto authority model. Users should take special care to select and assess their PDS host when they delegate management of their identity to that host (eg, when using a `did:plc`).
## Future Work
Account migration between hosting providers is one of the core design goals for atproto, and it is expected that new protocol features will support account migration.
The migration process itself may also be improved and simplified.
--------------------------------------------------------------------------------
# specs > at-uri-scheme
# AT URI Scheme (at://)
The AT URI scheme (`at://`) makes it easy to reference individual records in a specific repository, identified by either DID or handle. AT URIs can also be used to reference a collection within a repository, or an entire repository (aka, an identity). {{ className: 'lead' }}
Both of these AT URIs reference the same record in the same repository; one uses the account's DID, and one uses the account's handle.
- `at://did:plc:44ybard66vv44zksje25o7dz/app.bsky.feed.post/3jwdwj2ctlk26`
- `at://bnewbold.bsky.team/app.bsky.feed.post/3jwdwj2ctlk26`
<Note>
**Caveats for Handle-based AT URIs**
AT URIs referencing handles are not durable.
If a user changes their handle, any AT URIs using that handle will become invalid and could potentially point to a record in another repo if the handle is reused.
AT URIs are not content-addressed, so the _contents_ of the record they refer to may also change over time.
</Note>
### Structure
The full, general structure of an AT URI is:
```text
"at://" AUTHORITY [ PATH ] [ "?" QUERY ] [ "#" FRAGMENT ]
```
The **authority** part of the URI can be either a handle or a DID, indicating the identity associated with the repository. Note that a handle can refer to different DIDs (and thus different repositories) over time. See discussion below about strong references, and in "Usage and Implementation".
In current atproto Lexicon use, the **query** and **fragment** parts are not yet supported, and only a fixed pattern of paths are allowed:
```text
"at://" AUTHORITY [ "/" COLLECTION [ "/" RKEY ] ]
```
The **authority** section is required, must be normalized, and if a DID must be one of the "blessed" DID methods. The optional **collection** part of the path must be a normalized [NSID](./nsid). The optional **rkey** part of the path must be a valid [Record Key](./record-key).
An AT URI pointing to a specific record in a repository is not a *strong* reference, in that it is not content-addressed. The record may change or be removed over time, or the DID itself may be deleted or unavailable. For `did:web`, control of the DID (and thus repository) may change over time. For AT URIs with a handle in the authority section, the handle-to-DID mapping can also change.
A major semantic difference between AT URIs and common URL formats like `https://`, `ftp://`, or `wss://` is that the "authority" part of an AT URI does not indicate a network location for the indicated resource. Even when a handle is in the authority part, the hostname is only used for identity lookup, and is often not the ultimate host for repository content (aka, the handle hostname is often not the PDS host).
### Generic URI Compliance
AT URIs meet the generic syntax for Universal Resource Identifiers, as defined in IETF [RFC-3986](https://www.rfc-editor.org/rfc/rfc3986). They utilize some generic URI features outlined in that document, though not all. As a summary of generic URI parts and features:
- Authority part, preceded by double slash: supported
- Empty authority part: not supported
- Userinfo: not currently supported, but reserved for future use. a lone `@` character preceding a handle is not valid (eg, `at://@handle.example.com` is not valid)
- Host and port separation: not supported. syntax conflicts with DID in authority part
- Path part: supported, optional
- Query: supported in general syntax, not currently used
- Fragment: supported in general syntax, not currently used
- Relative references: not yet supported
- Normalization rules: supported in general syntax, not currently used
AT URIs are not compliant with the WHATWG URL Standard ([https://url.spec.whatwg.org/](https://url.spec.whatwg.org/)). Un-encoded colon characters in DIDs in the authority part of the URI are disallowed by that standard. Note that it is possible to un-ambigiously differentiate a DID in the authority section from a `host:port` pair. DIDs always have at least two colons, always begin with `did:`, and the DID method can not contain digits.
### Full AT URI Syntax
The full syntax for AT URIs is flexible to a variety of future use cases, including future extensions to the path structure, query parameters, and a fragment part. The full syntax rules are:
- The overall URI is restricted to a subset of ASCII characters
- For reference below, the set of unreserved characters, as defined in [RFC-3986](https://www.rfc-editor.org/rfc/rfc3986), includes alphanumeric (`A-Za-z0-9`), period, hyphen, underscore, and tilde (`.-_~`)
- Maximum overall length is 8 kilobytes (which may be shortened in the future)
- Hex-encoding of characters is permitted (but in practice not necessary)
- The URI scheme is `at`, and an authority part preceded with double slashes is always required, so the URI always starts `at://`
- An authority section is required and must be non-empty. the authority can be either an atproto Handle, or a DID meeting the restrictions for use with atproto. note that the authority part can *not* be interpreted as a host:port pair, because of the use of colon characters (`:`) in DIDs. Colons and unreserved characters should not be escaped in DIDs, but other reserved characters (including `#`, `/`, `$`, `&`, `@`) must be escaped.
- Note that none of the current "blessed" DID methods for atproto allow these characters in DID identifiers
- An optional path section may follow the authority. The path may contain multiple segments separated by a single slash (`/`). Generic URI path normalization rules may be used.
- An optional query part is allowed, following generic URI syntax restrictions
- An optional fragment part is allowed, using JSON Path syntax
### Restricted AT URI Syntax
A restricted sub-set of valid AT URIs are currently used in Lexicons for the `at-uri` type. Query parameters and fragments are not currently used. Trailing slashes are not allowed, including a trailing slash after the authority with no other path. The URI should be in normalized form (see "Normalization" section), with all of the individual sub-identifiers also normalized.
```text
AT-URI = "at://" AUTHORITY [ "/" COLLECTION [ "/" RKEY ] ]
AUTHORITY = HANDLE | DID
COLLECTION = NSID
RKEY = RECORD-KEY
```
### Normalization
Particularly when included in atproto records, strict normalization should be followed to ensure that the representation is reproducible and can be used with simple string equality checks.
- No unnecessary hex-encoding in any part of the URI
- Any hex-encoding hex characters must be upper-case
- URI schema is lowercase
- Authority as handle: lowercased
- Authority as DID: in normalized form, and no duplicate hex-encoding. For example, if the DID is already hex-encoded, don't re-encode the percent signs.
- No trailing slashes in path part
- No duplicate slashes or "dot" sections in path part (`/./` or `/abc/../` for example)
- NSID in path: domain authority part lowercased
- Record Key is case-sensitive and not normalized
- Query and fragment parts should not be included when referencing repositories or records in Lexicon records
Refer to [RFC-3986](https://www.rfc-editor.org/rfc/rfc3986) for generic rules to normalize paths and remove `..` / `.` relative references.
### Examples
Valid AT URIs (both general and Lexicon syntax):
```text
at://foo.com/com.example.foo/123
```
Valid general AT URI syntax, invalid in current Lexicon:
```text
at://foo.com/example/123 // invalid NSID
at://computer // not a valid DID or handle
at://example.com:3000 // not a valid DID or handle
```
Invalid AT URI (in both contexts)
```text
at://foo.com/ // trailing slash
at://user:[email protected] // userinfo not currently supported
```
### Usage and Implementation Guidelines
Generic URI and URL parsing libraries can sometimes be used with AT URIs, but not always. A key requirement is the ability to work with the authority (or origin) part of the URI as a simple string, without being parsed in to userinfo, host, and port sub-parts. Specifically: the Python 3 `urllib` module (from the standard library) works; the Javascript `url-parse` package works; the Golang `net/url` package does not work; and most of the popular Rust URL parsing crates do not work.
When referencing records, especially from other repositories, best practice is to use a DID in the authority part, not a handle. For application display, a handle can be used as a more human-readable alternative. In HTML, it is permissible to *display* the handle version of an AT-URI and *link* (`href`) to the DID version.
When a *strong* reference to another record is required, best practice is to use a CID hash in addition to the AT URI.
In Lexicons (APIs, records, and other contexts), sometimes a specific variant of an AT URI is required, beyond the general purpose `at-uri` string format. For example, references to records from inside records usually require a DID in the authority section, and the URI must include the collection and rkey path segments. URIs not meeting these criteria will fail to validate.
Do not confuse the JSON Path fragment syntax with the Lexicon reference syntax. They both use `#`-based fragments to reference other fields in JSON documents, but, for example, JSON Path syntax starts with a slash (`#/key`).
### Possible Future Changes
The maximum length constraint may change.
Relative references may be supported in Lexicons in `at-uri` fields. For example, one record referencing other records in the same repository could use `../<collection>/<rkey>` relative path syntax.
--------------------------------------------------------------------------------
# specs > atp
# AT Protocol
The Authenticated Transfer Protocol (AT Protocol or atproto) is a generic federated protocol for building open social media applications. Some recurring themes and features are: {{ className: 'lead' }}
- Self-authenticating data and identity, allowing seamless account migrations and redistribution of content {{ className: 'lead' }}
- Design for "big world" use cases, scaling to billions of accounts {{ className: 'lead' }}
- Delegated authority over application-layer schemas and aggregation infrastructure {{ className: 'lead' }}
- Re-use of existing data models from the dweb protocol family and network primitives from the web platform {{ className: 'lead' }}
## Protocol Structure
**Identity:** account control is rooted in stable [DID](/specs/did) identifiers, which can be rapidly resolved to determine the current service provider location and [Cryptographic keys](/specs/cryptography) associated with the account. [Handles](/specs/handle) provide a more human-recognizable and mutable identifier for accounts.
**Data:** public content is stored in content-addressed and cryptographically verifiable [Repositories](/specs/repository). Data records and network messages all conform to a unified [Data Model](/specs/data-model) (with [CBOR](https://en.wikipedia.org/wiki/CBOR) and JSON representations). [Labels](/specs/label) are a separate lightweight form of metadata, individually signed and distributed outside repositories.
**Network:** HTTP client-server and server-server [APIs](/specs/xrpc) are described with Lexicons, as are WebSocket [Event Streams](/specs/event-stream). Individual records can be referenced across the network by [AT URI](/specs/at-uri-scheme). A [Personal Data Server (PDS)](/specs/account) acts as an account's trusted agent in the network, routes client network requests, and hosts repositories. A relay crawls many repositories and outputs a unified event [Firehose](/specs/sync).
**Application:** APIs and record schemas for applications built on atproto are specified in [Lexicons](/specs/lexicon), which are referenced by [Namespaced Identifiers](/specs/nsid) (NSIDs). Application-specific aggregations (such as search) are provided by an Application View (App View) service. Clients can include mobile apps, desktop software, or web interfaces.
The AT Protocol itself does not specify common social media conventions like follows or avatars, leaving these to application-level Lexicons. The `com.atproto.*` Lexicons provide common APIs for things like account signup and login. These could be considered part of AT Protocol itself, though they can also be extended or replaced over time as needed. Bluesky is a microblogging social app built on top of AT Protocol, with lexicons under the `app.bsky.*` namespace.
While atproto borrows several formats and specifications from the IPFS ecosystem (such as [IPLD](https://ipld.io/) and [CID](https://github.com/multiformats/cid)), atproto data does not need to be stored in the IPFS network, and the atproto reference implementation does not use the IPFS network at all.
## Protocol Extension and Applications
AT Protocol was designed from the beginning to balance stability and interoperation against flexibility for third-party application development.
The core protocol extension mechanism is development of new Lexicons under independent namespaces. Lexicons can declare new repository record schemas (stored in collections by NSID), new HTTP API endpoints, and new event stream endpoints and message types. It is also expected that new applications might require new network aggregation services ("AppViews") and client apps (eg, mobile apps or web interfaces).
It is expected that third parties will reuse Lexicons and record data across namespaces. For example, new applications are welcome to build on top of the social graph records specified in the `app.bsky.*` Lexicons, as long as they comply with the schemas controlled by the `bsky.app` authority.
Governance structures for individual Lexicon namespaces are flexible. They could be developed and maintained by volunteer communities, corporations, consortia, academic researchers, funded non-profits, etc.
## What Is Missing?
These specifications cover most details as implemented in Bluesky's reference implementation. A few
**Moderation Primitives:** The `com.atproto.admin.*` routes for handling moderation reports and doing infrastructure-level take-downs is specified in Lexicons but should also be described in more detail.
## Future Work
Smaller changes are described in individual specification documents, but a few large changes span the entire protocol.
**Non-Public Content:** mechanisms for private group and one-to-one communication will be an entire second phase of protocol development. This encompasses primitives like "private accounts", direct messages, encrypted data, and more. We recommend against simply "bolting on" encryption or private content using the existing protocol primitives.
**Protocol Governance and Formal Standards Process:** The current development focus is to demonstrate all the core protocol features via the reference implementation, including open federation. After that milestone, the intent is to stabilize the lower-level protocol and submit the specification for independent review and revision through a standards body such as the IETF or the W3C.
--------------------------------------------------------------------------------
# specs > blob
# Blobs
"Blobs" are media files stored alongside an account's repository. They include images, video, and audio, but could also include any other file format. Blobs are referenced by individual records by the `blob` lexicon datatype, which includes a content hash (CID) for the blob.
Blob files are uploaded and distributed separately from records. Blobs are authoritatively stored by the account's PDS instance, but views are commonly served by CDNs associated with individual applications ("AppViews"), to reduce traffic on the PDS. CDNs may serve transformed (resized, transcoded, etc) versions of the original blob.
While blobs are universally content addressed (by CID), they are always referenced and managed in the context of an individual account (DID).
The empty blob (zero bytes) is valid in general, though it may be disallowed by individual Lexicons/applications.
## Blob Metadata
Currently, the only "blessed" CID type for blobs is similar to that for repository records, but with the `raw` multicodec:
- CIDv1
- Multibase: binary serialization within DAG-CBOR (or `base32` for JSON mappings)
- Multicodec: `raw` (0x55)
- Multihash: `sha-256` with 256 bits (0x12)
An example blob CID, in base32 string encoding: `bafkreibjfgx2gprinfvicegelk5kosd6y2frmqpqzwqkg7usac74l3t2v4`
Blob metadata also includes the size of the blob (in bytes), and the MIME type. The size and CID are deterministic and must be valid and consistent. The MIME type is somewhat more subjective: it is possible for the same bytes to be valid for multiple MIME types.
## Blob Lifecycle
Blobs must be uploaded to the PDS before a record can be created referencing that blob. Note that the server does not know the intended Lexicon when receiving an upload, so can only apply generic blob limits and restrictions at initial upload time, and then enforce Lexicon-defined limits later when the record is created.
Clients use the `com.atproto.repo.uploadBlob` endpoint on their PDS, which will return verified metadata in the form of a Lexicon blob object. Clients "should" set the HTTP `Content-Type` header and "should" set the `Content-Length` headers on the upload request. Chunked transfer encoding may also be permitted for uploads. Servers may sniff the blob mimetype to validate against the declared `Content-Type` header, and either return a modified mimetype in the response, or reject the upload. See "Security Considerations" below. If the actual blob upload size differs from the `Content-Length` header, the server should reject the upload.
After a successful upload, blobs are placed in temporary storage. They are not accessible for download or distribution while in this state. Servers should "garbage collect" (delete) un-referenced temporary blobs after an appropriate time span (see implementation guidelines below). Blobs which are in temporary storage should not be included in the `listBlobs` output.
The upload blob can now be referenced from records by including the returned blob metadata in a record. When processing record creation, the server extracts the set of all referenced blobs, and checks that they are either already referenced, or are in temporary storage. Once the record creation succeeds, the server makes the blob publicly accessible.
The same blob can be referenced by multiple records in the same repository. Re-uploading a blob which has already been stored and referenced results in no change to the existing blobs or records.
When a record referencing blobs is deleted, the server checks if any other current records from the same repository reference the blob. If not, the blob is deleted along with the record.
When an account is deleted, all the hosted blobs are deleted, within some reasonable time frame. When an account is deactivated, takendown, or suspended, blobs should not be publicly accessible.
Servers may decide to make individual blobs inaccessible, separately from any account takedown or other account lifecycle events.
Creation of new individual records which reference a blob which does not exist should be rejected at the time of creation (or update). However, it is possible for servers to host repository records which reference blobs which are not available locally. For example, during a bulk repository
Original blobs can be fetched from the PDS using the `com.atproto.sync.getBlob` endpoint. The server should return appropriate `Content-Type` and `Content-Length` HTTP headers. It is not a recommended or required pattern to serve media directly from the PDS to end-user browsers, and servers do not need to support or facilitate this use case. See "Security Considerations" below for more.
## Usage and Implementation Guidelines
Servers may have their own generic limits and policies for blobs, separate from any Lexicon-defined constraints. They might implement account-wide quotas on data storage; maximum blob sizes; content policies; etc. Any of these restrictions might be enforced at the initial upload. Server operators should be aware that limits and other restrictions may impact functionality with existing and future applications. To maximize interoperability, operators are recommended to prefer limits on overall account resource consumption (eg, "total blob size" quota, not "per blob" size limits).
Some applications may have a long delay between blob upload and reference from a record. To maximize interoperability, server implementations and operators are recommended to allow several hours of grace time before "garbage collecting", with at least one hour a firm lower bound.
## Security Considerations
Serving arbitrary user-uploaded files from a web server raises many content security issues. For example, cross-site scripting (XSS) of scripts or SVG content form the same "origin" as other web pages. It is effectively mandatory to enable a Content Security Policy (LINK: https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP) for the `getBlob` endpoint. It is effectively not supported to dynamically serve assets directly out of blob storage (the `getBlob` endpoint) directly to browsers and web applications. Applications must proxy blobs, files, and assets through an independent CDN, proxy, or other web service before serving to browsers and web agents, and such services are expected to implement security precautions.
An example set of content security headers for this endpoint is:
```
Content-Security-Policy: default-src 'none'; sandbox
X-Content-Type-Options: nosniff
```
Some media types may contain sensitive metadata. For example, EXIF metadata in JPEG image files may contain GPS coordinates. Servers might take steps to prevent accidental leakage of such metadata, for example by blocking upload of blobs containing them. See note in "Future Changes" section.
Parsing of media files is a notorious source of memory safety bugs and security vulnerabilities. Even content type detection (or "sniffing") can be a source of exploits. Servers are strongly recommended against parsing media files (image, video, audio, or any other non-trivial formats) directly, without the use of strong sandboxing mechanisms. In particular, PDS instances themselves should not directly implement media resizing or transcoding.
Richer media types raise the stakes for abusive and illegal content. Services should implement appropriate mechanisms to takedown such content when it is detected and reported.
Servers may need to take measures to prevent malicious resource consumption. For example, intentional exhaustion of disk space, network congestion, bandwidth utilization, etc. Rate-limits, size limits, and quotas are recommended.
## Possible Future Changes
The allowed CID type is expected to evolve over time. There has been interest in `blake3` for larger file types.
More specific mitigation of metadata leakage (eg, EXIF metadata stripping) should be recommended and/or enabled via API changes. There is a tension between providing default safety, and always intervening to manipulate "original" uploaded user data. Additionally, parsing and manipulating uploaded media files raises other categories of security concerns.
--------------------------------------------------------------------------------
# specs > cryptography
# Cryptography
Two elliptic curves are currently supported throughout the protocol, and implementations are expected to fully support both: {{ className: 'lead' }}
- `p256` elliptic curve: aka "NIST P-256", aka `secp256r1` (note the `r`), aka `prime256v1`
- This curve *is* included in the WebCrypto API. It is commonly supported by personal device hardware (Trusted Platform Modules (TPMs) and mobile Secure Enclaves), and by cloud Hardware Security Modules (HSMs)
- `k256` elliptic curve: aka "NIST K-256", aka `secp256k1` (note the `k`)
- This curve *is not* included in the WebCrypto API. It is used in Bitcoin and other cryptocurrencies, and as a result is broadly supported by personal secret key management technologies. It is also supported by cloud HSMs.
Because of the subtle visual distinction when the full curve names are written out, we often refer to them as `p256` or `k256`.
The atproto reference implementation from Bluesky supports both curves in all contexts, and creates `k256` key pairs by default.
Key points for both systems have loss-less "compressed" representations, which are useful when sharing the public keys. This is usually supported natively for `k256`, but sometimes requires extra methods or jumping through hoops for `p256`. You can read more about this at: [02, 03 or 04? So What Are Compressed and Uncompressed Public Keys?](https://medium.com/asecuritysite-when-bob-met-alice/02-03-or-04-so-what-are-compressed-and-uncompressed-public-keys-6abcb57efeb6).
A common pattern when signing data in atproto is to encode the data in DAG-CBOR, hash the CBOR bytes with SHA-256, yielding raw bytes (not a hex-encoded string), and then sign the hash bytes.
## ECDSA Signature Malleability
Some ECDSA signatures can be transformed to yield a new distinct but still-valid signature. This does not require access to the private signing key or the data that was signed. The scope of attacks possible using this property is limited, but it is an unexpected property.
For `k256` specifically, the distinction is between "low-S" and "high-S" signatures, as discussed in [Bitcoin BIP-0062](https://github.com/bitcoin/bips/blob/master/bip-0062.mediawiki).
In atproto, use of the "low-S" signature variant is required for both `p256` and `k256` curves.
In atproto, signatures should always be verified using the verification routines provided by the cryptographic library, never by comparing signature values as raw bytes.
## Public Key Encoding
When encoding public keys as strings, the preferred representation uses multibase (with `base58btc` specifically) and a multicode prefix to indicate the specific key type. By embedding metadata about the type of key in the encoding itself, they can be parsed unambiguously. The process for encoding a public key in this format is:
- Encode the public key curve "point" as bytes. Be sure to use the smaller "compact" or "compressed" representation. This is usually easy for `k256`, but might require a special argument or configuration for `p256` keys
- Prepend the appropriate curve multicodec value, as varint-encoded bytes, in front of the key bytes:
- `p256` (compressed, 33 byte key length): `p256-pub`, code 0x1200, varint-encoded bytes: [0x80, 0x24]
- `k256` (compressed, 33 byte key length): `secp256k1-pub`, code 0xE7, varint bytes: [0xE7, 0x01]
- Encode the combined bytes with with `base58btc`, and prefix with a `z` character, yielding a multibase-encoded string
The decoding process is the same in reverse, using the identified curve type as context.
To encode a key as a `did:key` identifier, use the above multibase encoding, and add the ASCII prefix `did:key:`. This identifier is used as an internal implementation detail in the DID PLC method.
Note that there is a variant legacy multibase encoding described in the [atproto DID specification document](/specs/did), which does not include a multicodec type value, and uses uncompressed byte encoding of keys. This format is deprecated.
### Encoded Examples
A P-256 public key, encoded in multibase (with multicodec), and as `did:key`:
```
zDnaembgSGUhZULN2Caob4HLJPaxBh92N7rtH21TErzqf8HQo
did:key:zDnaembgSGUhZULN2Caob4HLJPaxBh92N7rtH21TErzqf8HQo
```
A K-256 public key, encoded in multibase (with multicodec), and as `did:key`:
```
zQ3shqwJEJyMBsBXCWyCBpUBMqxcon9oHB7mCvx4sSpMdLJwc
did:key:zQ3shqwJEJyMBsBXCWyCBpUBMqxcon9oHB7mCvx4sSpMdLJwc
```
## Usage and Implementation Guidelines
There is no specific recommended byte or string encoding for private keys across the atproto ecosystem. Sometimes simple hex encoding is used, sometimes multibase with or without multicodec type information.
## Possible Future Changes
The set of supported cryptographic systems is expected to evolve slowly. There are significant interoperability and implementation advantages to having as few systems as possible at any point in time.
--------------------------------------------------------------------------------
# specs > data-model
# Data Model
Records and messages in atproto are stored, transmitted, encoded, and authenticated in a consistent way. The core "data model" supports both binary (CBOR) and textual (JSON) representations. {{ className: 'lead' }}
When data needs to be authenticated (signed), referenced (linked by content hash), or stored efficiently, it is encoded in Concise Binary Object Representation (CBOR). CBOR is an IETF standard roughly based on JSON. The specific normalized subset of CBOR used in the atproto data model is called **DAG-CBOR**. All DAG-CBOR data is valid CBOR, and can be read with any CBOR library. Writing or strictly verifying DAG-CBOR with the correct normalization rules sometimes requires additional configuration or a special CBOR implementation. {{ className: 'lead' }}
The schema definition language for atproto is [Lexicon](/specs/lexicon). Other lower-level data structures, like [repository](/specs/repository) internals, are not specified with Lexicons, but use the same data model and encodings.
Distinct pieces of data are called **nodes,** and when encoded in binary (DAG-CBOR) result in a **block.** A node may have internal nested structure (maps or lists). Nodes may reference each other by string URLs or URIs, just like with regular JSON on the web. They can also reference each other strongly by hash, referred a **link.** A set of linked nodes can form higher-level data structures like [Merkle Trees](https://en.wikipedia.org/wiki/Merkle_tree) or [Directed Acyclical Graphs (DAG)](https://en.wikipedia.org/wiki/Directed_acyclic_graph). Links can also refer to arbitrary binary data (blobs).
Unlike URLs, hash references (links) do not encode a specific network location where the content can be found. The location and access mechanism must be inferred by protocol-level context. Hash references do have the property of being "self-certifying", meaning that returned data can be verified against the link hash. This makes it possible to redistribute content and trust copies even if coming from an untrusted party.
Links are encoded as [Content Identifiers](https://docs.ipfs.tech/concepts/content-addressing/#identifier-formats) (CIDs), which have both binary and string representations. CIDs include a metadata code which indicates whether it links to a node (DAG-CBOR) or arbitrary binary data. Some additional constraints on the use of CIDs in atproto are described below.
In atproto, object nodes often include a string field `$type` that specifies their Lexicon schema. Data is mostly self-describing and can be processed in schema-agnostic ways (including decoding and re-encoding), but can not be fully validated without the schema on-hand or known ahead of time.
## Relationship With IPLD
The data model is inspired by [Interplanetary Linked Data (IPLD)](https://ipld.io/docs/data-model/), a specification for hash-linked data structures from the IPFS ecosystem.
IPLD specifies a normalized JSON encoding called **DAG-JSON,** but atproto uses a different set of conventions when encoding JSON data. The atproto JSON encoding is not designed to be byte-determinisitic, and the CBOR representation is used when data needs to be cryptographically signed or hashed.
The IPLD Schema language is not used.
## Data Types
| Lexicon Type | IPLD Type | JSON | CBOR | Note |
| --- | --- | --- | --- | --- |
| `null` | null | Null | Special Value (major 7) | |
| `boolean` | boolean | Boolean | Special Value (major 7) | |
| `integer` | integer | Number | Integer (majors 0,1) | signed, 64-bit |
| `string` | string | String | UTF-8 String (major 3) | Unicode, UTF-8 |
| - | float | Number | Special (major 7) | not allowed in atproto |
| `bytes` | bytes | `$bytes` Object | Byte String (major 2) | |
| `cid-link` | link | `$link` Object | CID (tag 42) | CID |
| `array` | list | Array | Array (major 4) | |
| `object` | map | Object | Map (major 5) | keys are always strings |
| `blob` | - | `$type: blob` Object | `$type: blob` Map | |
`blob` is for references to files, such as images. It includes basic metadata like MIME Type and size (in bytes).
As a best practice to ensure Javascript compatibility with default types, `integer` should be limited to 53 bits of precision. Note that JSON numbers can have an arbitrary number of digits, but `integer` is limited to 64 bits even ignoring Javascript.
Lexicons can include additional validation constraints on individual fields. For example, integers can have maximum and minimum values. Data can not be validated against these additional constraints without access to the relevant Lexicon schema, but there is a concept of validating free-form JSON or CBOR against the atproto data model in an abstract sense. For example, a JSON object with a nested `$bytes` object with a boolean instead of a base64-encoded string might be valid JSON, but can never be valid under the atproto data model.
Lexicon string fields can have additional `format` type information associated with them for validation, but as with other validation constraints this information is not available without the Lexicon itself.
Data field names starting with `$` are reserved for use by the data model or protocol itself, in both JSON and CBOR representations. For example, the `$bytes` key name (used in CBOR and JSON), the `$link` key (used for JSON CID Links), or `$type` (used to indicate record type). Implementations should ignore unknown `$` fields (to allow protocol evolution). Applications, extensions, and integrations should not use or unilaterally define new `$` fields, to prevent conflicts as the protocol evolves.
### Nullable and False-y
In the atproto data model there is a semantic difference between explicitly setting an map field to `null` and not including the field at all. Both JSON and CBOR have the same distinction.
Null or missing fields are also distinct from "false-y" value like `false` (for booleans), `0` (for integers), empty lists, or empty objects.
### Why No Floats?
CBOR and JSON both natively support floating point numbers, so why does atproto go out of the way to disallow them?
The IPLD specification describes [some of the complexities and sharp edges](https://ipld.io/docs/data-model/kinds/#float-kind) when working with floats in a content-addressable world. In short, de-serializing in to machine-native format, then later re-encoding, is not always consistent. This is definitely true for special values and corner-cases, but can even be true with "normal" float values on less-common architectures.
It may be possible to come up with rules to ensure reliable round-trip encoding of floats in the future, but for now we disallow floats.
If you have a use-case where integers can not be substituted for floats, we recommend encoding the floats as strings or even bytes. This provides a safe default round-trip representation.
## `blob` Type
References to "blobs" (arbitrary files) have a consistent format in atproto, and can be detected and processed without access to any specific Lexicon. That is, it is possible to parse nodes and extract any blob references without knowing the schema.
Blob nodes are maps with following fields:
- `$type` (string, required): fixed value `blob`. Note that this is not a valid NSID.
- `ref` (link, required): CID reference to blob, with multicodec type `raw`. In JSON, encoded as a `$link` object as usual
- `mimeType` (string, required, not empty): content type of blob. `application/octet-stream` if not known
- `size` (integer, required, positive, non-zero): length of blob in bytes
There is also a deprecated legacy blob format, with some records in the wild still containing blob references in this format:
- `cid` (string, required): a CID in *string* format, not *link* format
- `mimeType` (string, required, not empty): same as `mimeType` above
Note that the legacy format has no `$type` and can only be parsed for known Lexicons. Implementations should not throw errors when encountering the old format, but should never write them, and it is acceptable to only partially support them.
## JSON Representation
atproto uses its own conventions for JSON, instead of using DAG-JSON directly. The main motivation was to have more idiomatic and human-readable representations for `link` and `bytes` in HTTP APIs. The DAG-JSON specification itself mentions that it is primarily oriented toward debugging and development environments, and we found that the use of `/` as a field key was confusing to developers.
Normalizations like key sorting are also not required or enforced when using JSON in atproto: only DAG-CBOR is used as a byte-reproducible representation.
The encoding for most of the core and compound types is straight forward, with only `link` and `bytes` needing special treatment.
### `link`
The JSON encoding for link is an object with the single key `$link` and the string-encoded CID as a value.
For example, a node with a single field `"exampleLink"` with type `link` would encode in JSON like:
```
{
"exampleLink": {
"$link": "bafyreidfayvfuwqa7qlnopdjiqrxzs6blmoeu4rujcjtnci5beludirz2a"
}
}
```
For comparison, this is very similar to the DAG-JSON encoding, but substitutes `$link` as the key name instead of `/` (single-character, forward slash).
### `bytes`
The JSON encoding for bytes is an object with the single key `$bytes` and string value with the base64-encoded bytes. The base64 scheme is the one specified in [RFC-4648, section 4](https://datatracker.ietf.org/doc/html/rfc4648#section-4), frequently referred to as simple "base64". This scheme is not URL-safe, and `=` padding is optional.
For example, a node with a single field `"exampleBytes"` with type `bytes` would be represented in JSON like:
```
{
"exampleBytes": {
"$bytes": "nFERjvLLiw9qm45JrqH9QTzyC2Lu1Xb4ne6+sBrCzI0"
}
}
```
For comparison, the DAG-JSON encoding has two nested objects, with outer key `/` (single-character, forward slash), inner key `bytes`, and the same base64 encoding.
## Link and CID Formats
The [IPFS CID specification](https://github.com/multiformats/cid) is very flexible. It supports a wide variety of hash types, a field indicating the "type" of content being linked to, and various string encoding options. These features are valuable to allow evolution over time, but to maximize interoperability among implementations, only a specific "blessed" set of CID types are allowed.
The blessed formats for CIDs in atproto are:
- CIDv1
- multibase: binary serialization within DAG-CBOR `link` fields, and `base32` for string encoding
- multicodec: `dag-cbor` (0x71) for links to data objects, and `raw` (0x55) for links to blobs
- multihash: `sha-256` with 256 bits (0x12) is preferred
The use of SHA-256 is a stable requirement in some contexts, such as the repository MST nodes. In other contexts, like referencing media blobs, there will likely be a set of "blessed" hash types which evolve over time. A balance needs to be struck between protocol flexibility on the one hand (to adopt improved hashes and remove weak ones), and ensuring broad and consistent interoperability throughout an ecosystem of protocol implementations.
There are several ways to include a CID hash reference in an atproto object:
- `link` field type (Lexicon type `cid-link`). In DAG-CBOR encodes as a binary CID (multibase type 0x00) in a bytestring with CBOR tag 42. In JSON, encodes as `$link` object (see above)
- `string` field type, with Lexicon string format `cid`. In DAG-CBOR and JSON, encodes as a simple string
- `string` field type, with Lexicon string format `uri`, with URI scheme `ipld://`
## Usage and Implementation Guidelines
When working with the deprecated/legacy "blob" format, it is recommend to store in the same internal representation as regular "blob" references, but to set the `size` to zero or a negative value. This field should be checked when re-serializing to ensure proper round-trip behavior and avoid ever encoding a zero or negative `size` value in the normal object format.
## Security and Privacy Considerations
There are a number of resource-consumption attacks possible when parsing untrusted CBOR content. It is recommended to use a library that automatically protects against huge allocations, deep nesting, invalid references, etc. This is particularly
## Possible Future Changes
Floats may be supported in one form or another.
The legacy "blob" format may be entirely removed, if all known records and repositories can be rewritten.
Additional hash types are likely to be included in the set of "blessed" CID configurations.
--------------------------------------------------------------------------------
# specs > did
# AT Protocol DIDs
The AT Protocol uses [Decentralized Identifiers](https://en.wikipedia.org/wiki/Decentralized_identifier) (DIDs) as persistent, long-term account identifiers. DID is a W3C standard, with many standardized and proposed DID method implementations. {{ className: 'lead' }}
## Blessed DID Methods
Currently, atproto supports two DID methods:
- `did:web`, which is a W3C standard based on HTTPS (and DNS). The identifier section is a hostname. This method is supported in atproto to provide an independent alternative to `did:plc`. The method is inherently tied to the domain name used, and does not provide a mechanism for migration or recovering from loss of control of the domain name. In the context of atproto, only hostname-level `did:web` DIDs are supported: path-based DIDs are not supported. The same restrictions on top-level domains that apply to handles (eg, no `.arpa`) also apply to `did:web` domains. The special `localhost` hostname is allowed, but only in testing and development environments. Port numbers (with separating colon hex-encoded) are only allowed for `localhost`, and only in testing and development.
- `did:plc`, which is a novel DID method developed by Bluesky. See the [did-method-plc](https://github.com/did-method-plc/did-method-plc) GitHub repository for details.
In the future, a small number of additional methods may be supported. It is not the intention to support all or even many DID methods, even with the existence of universal resolver software.
## AT Protocol DID Identifier Syntax
Lexicon string format type: `did`
The DID Core specification constraints on DID identifier syntax, regardless of the method used. A summary of those syntax constraints, which may be used to validate DID generically in atproto are:
- The entire URI is made up of a subset of ASCII, containing letters (`A-Z`, `a-z`), digits (`0-9`), period, underscore, colon, percent sign, or hyphen (`._:%-`)
- The URI is case-sensitive
- The URI starts with lowercase `did:`
- The method segment is one or more lowercase letters (`a-z`), followed by `:`
- The remainder of the URI (the identifier) may contain any of the above-allowed ASCII characters, except for percent-sign (`%`)
- The URI (and thus the remaining identifier) may not end in `:`.
- Percent-sign (`%`) is used for "percent encoding" in the identifier section, and must always be followed by two hex characters
- Query (`?`) and fragment (`#`) sections are allowed in DID URIs, but not in DID identifiers. In the context of atproto, the query and fragment parts are not allowed.
DID identifiers do not generally have a maximum length restriction, but in the context of atproto, there is an initial hard limit of 2 KB.
In the context of atproto, implementations do not need to validate percent encoding. The percent symbol is allowed in DID identifier segments, but the identifier should not end in a percent symbol. A DID containing invalid percent encoding *should* fail any attempt at registration, resolution, etc.
A reasonable starting-point regex for DIDs in the context of atproto is:
```
// NOTE: does not constrain overall length
/^did:[a-z]+:[a-zA-Z0-9._:%-]*[a-zA-Z0-9._-]$/
```
### Examples
Valid DIDs for use in atproto (correct syntax, and supported method):
```
did:plc:z72i7hdynmk6r22z27h6tvur
did:web:blueskyweb.xyz
```
Valid DID syntax (would pass Lexicon syntax validation), but unsupported DID method:
```
did:method:val:two
did:m:v
did:method::::val
did:method:-:_:.
did:key:zQ3shZc2QzApp2oymGvQbzP8eKheVshBHbU4ZYjeXqwSKEn6N
```
Invalid DID identifier syntax (regardless of DID method):
```
did:METHOD:val
did:m123:val
DID:method:val
did:method:
did:method:val/two
did:method:val?two
did:method:val#two
```
## DID Documents
After a DID document has been resolved, atproto-specific information needs to be extracted. This parsing process is agnostic to the DID method used to resolve the document.
The current **handle** for the DID is found in the `alsoKnownAs` array. Each element of this array is a URI. Handles will have the URI scheme `at://`, followed by the handle, with no path or other URI parts. The current primary handle is the first valid handle URI found in the ordered list. Any other handle URIs should be ignored.
It is crucial to validate the handle bidirectionally, by resolving the handle to a DID and checking that it matches the current DID document.
The DID is the primary account identifier, and an account whose DID document does not contain a valid and confirmed handle can still, in theory, participate in the atproto ecosystem. Software should be careful to either not display any handle for such account, or obviously indicate that any handle associated with it is invalid.
The public **signing key** for the account is found under the `verificationMethod` array, in an object with `id` ending `#atproto`, and the `controller` matching the DID itself. The first valid atproto signing key in the array should be used, and any others ignored. The `type` field will indicate the cryptographic curve type, and the `publicKeyMultibase` field will be the public key in multibase encoding. See below for details for parsing these fields.
A valid signing key is required for atproto functionality, and an account with no valid key in their DID document is broken.
The **PDS service network location** for the account is found under the `service` array, with `id` ending `#atproto_pds`, and `type` matching `AtprotoPersonalDataServer`. The first matching entry in the array should be used, and any others ignored. The `serviceEndpoint` field must contain an HTTPS URL of server. It should contain only the URI scheme (`http` or `https`), hostname, and optional port number, not any "userinfo", path prefix, or other components.
A working PDS is required for atproto account functionality, and an account with no valid PDS location in their DID document is broken.
Note that a valid URL doesn't mean the the PDS itself is currently functional or hosting content for the account. During account migrations or server downtime there may be windows when the PDS is not accessible, but this does not mean the account should immediately be considered broken or invalid.
## Representation of Public Keys
The atproto cryptographic systems are described in [Cryptography](/specs/cryptography), including details of byte and string encoding of public keys.
Public keys in DID documents under `verificationMethod`, including atproto signing keys, are represented as an object with the following fields:
- `id` (string, required): the DID followed by an identifying fragment. Use `#atproto` as the fragment for atproto signing keys
- `type` (string, required): the fixed string `Multikey`
- `controller` (string, required): DID controlling the key, which in the current version of atproto must match the account DID itself
- `publicKeyMultibase` (string, required): the public key itself, encoded in multibase format (with multicodec type indicator, and "compressed" key bytes)
The `publicKeyMultibase` format for `Multikey` is the same encoding scheme as used with `did:key`, but without the `did:key:` prefix. See [Cryptography](/specs/cryptography) for details.
Note that there is not yet a formal W3C standard for using P-256 public keys in DID `verificationMethod` sections, but that the `Multikey` standard does clarify what the encoding encoding should be for this key type.
### Legacy Representation
Some older DID documents, which may still appear in `did:web` docs, had slightly different key encodings and `verificationMethod` syntax. Implementations may support these older DID documents during a transition period, but the intentent is to require DID specification compliance going forward.
The older `verificationMethod` for atproto signing keys contained:
- `id` (string, required): the fixed string `#atproto`, without the full DID included
- `type` (string, required): a fixed name identifying the key's curve type
- `p256`: `EcdsaSecp256r1VerificationKey2019` (note the "r")
- `k256`: `EcdsaSecp256k1VerificationKey2019` (note the "k")
- `controller` (string, required): DID controlling the key, which in the current version of atproto must match the account DID itself
- `publicKeyMultibase` (string, required): the public key itself, encoded in multibase format (*without* multicodec, and *uncompressed* key bytes)
Note that the `EcdsaSecp256r1VerificationKey2019` type is not a final W3C standard.
The `EcdsaSecp256r1VerificationKey2019` `verificationMethod` is not a final W3C standard. We will move to whatever ends up standardized by W3C for representing P-256 public keys with `publicKeyMultibase`. This may mean a transition to `Multikey`, and we would transition K-256 representations to that `type` as well.
A summary of the multibase encoding in this context:
- Start with the full public key bytes. Do not use the "compressed" or "compact" representation (unlike for `did:key` or `Multikey` encoding)
- Do *not* prefix with a multicodec value indicating the key type
- Encode the key bytes with `base58btc`, yielding a string
- Add the character `z` as a prefix, to indicate the multibase, and include no other multicodec indicators
The decoding process is the same in reverse, using the curve type as context.
Here is an example of a single public key encoded in the legacy and current formats:
```
// legacy multibase encoding of K-256 public key
{
"id": ...,
"controller": ...,
"type": "EcdsaSecp256k1VerificationKey2019",
"publicKeyMultibase": "zQYEBzXeuTM9UR3rfvNag6L3RNAs5pQZyYPsomTsgQhsxLdEgCrPTLgFna8yqCnxPpNT7DBk6Ym3dgPKNu86vt9GR"
}
// preferred multibase encoding of same K-256 public key
{
"id": ...,
"controller": ...,
"type": "Multikey",
"publicKeyMultibase": "zQ3shXjHeiBuRCKmM36cuYnm7YEMzhGnCmCyW92sRJ9pribSF"
}
```
## Usage and Implementation Guidelines
Protocol implementations should be flexible to processing content containing DIDs based on unsupported DID methods. This is
While longer DIDs are supported in the protocol, a good best practice is to use relatively short DIDs, and to avoid DIDs longer than 64 characters.
DIDs are case-sensitive. While the currently-supported methods are *not* case sensitive, and could be safely lowercased, protocol implementations should reject DIDs with invalid casing. It is permissible to attempt case normalization when receiving user-controlled input, such as when parsing public URL path components, or text input fields.
## Possible Future Changes
The hard maximum DID length limit may be reduced, at the protocol syntax level. We are not aware of any DID methods that we would consider supporting which have identifiers longer than, say, 256 characters.
There is a good chance that the set of "blessed" DID methods will slowly expand over time.
--------------------------------------------------------------------------------
# specs > event-stream
# Event Stream
In addition to regular [HTTP API](/specs/xrpc) endpoints, atproto supports continuous event streams. Message schemas and endpoint names are transport-agnostic and defined in [Lexicons](/specs/lexicon). The initial encoding and transport scheme uses binary [DAG-CBOR](https://ipld.io/docs/codecs/known/dag-cbor/) encoding over [WebSockets](https://en.wikipedia.org/wiki/WebSocket). {{ className: 'lead' }}
The Lexicon type for streams is `subscription`. The schema includes an identifier (`id`) for the endpoint, a `message` schema (usually a union, allowing multiple message types), and a list of error types (`errors`).
Clients subscribe to a specific stream by initiating a connection at the indicated endpoint. Streams are currently one-way, with messages flowing from the server to the client. Clients may provide query parameters to configure the stream when opening the connection.
A **backfill window** mechanism allows clients to catch up with stream messages they may have missed. At a high level, this works by assigning monotonically increasing sequence numbers to stream events, and allowing clients to specify an initial sequence number when initiating a connection. The intent of this mechanism is to ensure reliable delivery of events following disruptions during a reasonable time window (eg, hours or days). It is not to enable clients to roll all the way back to the beginning of the stream.
All of the initial subscription Lexicons in the `com.atproto` namespace use the backfill mechanism. However, a backfill mechanism (and even cursors, which we define below) is not _required_ for streams. Subscription endpoints which do not require reliable delivery do not need to implement a backfill mechanism or use sequence numbers.
The initial subscription endpoints are also public and do not require authentication or prior permission to subscribe (though resource limits may be imposed on client). But subscription endpoints may require authentication at connection time, using the existing HTTP API (XRPC) authentication methods.
## Streaming Wire Protocol (v0)
To summarize, messages are encoded as DAG-CBOR and sent over a binary WebSocket. Clients connect to to a specific HTTP endpoint, with query parameters, then upgrade to WebSocket. Every WebSocket frame contains two DAG-CBOR objects, with bytes concatenated together: a header (indicating message type), and the actual message.
The WebSockets "living standard" is currently maintained by [WHATWG](https://en.wikipedia.org/wiki/WHATWG), and can be found in full at [https://websockets.spec.whatwg.org/](https://websockets.spec.whatwg.org/).
### Connection
Clients initialize stream subscriptions by opening an HTTP connection and upgrading to a WebSocket. HTTPS and "WebSocket Secure" (`wss://`) on the default port (443) should be used for all connections on the internet. HTTP, cleartext WebSocket (`ws://`), and non-standard ports should only be used for testing, development, and local connections (for example, behind a reverse proxy implementing SSL). From the client perspective, failure to upgrade connection to a WebSocket is an error.
Query parameters may be provided in the initial HTTP request to configure the stream in an application-specific way, as specified in the endpoint's Lexicon schema.
Errors are usually returned through the stream itself. Connection-time errors are sent as the first message on the stream, and then the server drops the connection. But some errors can not be handled through the stream, and are returned as HTTP errors:
- `405 Method Not Allowed`: Returned to client for non-GET HTTP requests to a stream endpoint.
- `426 Upgrade Required`: Returned to client if `Upgrade` header is not included in a request to a stream endpoint.
- `429 Too Many Requests`: Frequently used for rate-limiting. Client may try again after a delay. Support for the `Retry-After` header is encouraged.
- `500 Internal Server Error`: Client may try again after a delay
- `501 Not Implemented`: Service does not implement WebSockets or streams, at least for this endpoint. Client should not try again.
- `502 Bad Gateway`, `503 Service Unavailable`, `504 Gateway Timeout`: Client may try again after a delay
Servers *should* return HTTP bodies as JSON with the standard XRPC error message schema for these status codes. But clients also need to be robust to unexpected response body formats. A common situation is receiving a default load-balancer or reverse-proxy error page during scheduled or unplanned downtime.
Either the server or the client may decided to drop an open stream connection if there have been no messages for some time. It is also acceptable to leave connections open indefinitely.
### Framing
Each binary WebSocket frame contains two DAG-CBOR objects, concatenated. The first is a **header** and the second is the **payload.**
The header DAG-CBOR object has the following fields:
- `op` ("operation", integer, required): fixed values, indicating what this frame contains
- `1`: a regular message, with type indicated by `t`
- `-1`: an error message
- `t` ("type", string, optional): required if `op` is `1`, indicating the Lexicon sub-type for this message, in short form. Does not include the full Lexicon identifier, just a fragment. Eg: `#commit`. Should not be included in header if `op` is `-1`.
Clients should ignore frames with headers that have unknown `op` or `t` values. Unknown fields in both headers and payloads should be ignored. Invalid framing or invalid DAG-CBOR encoding are hard errors, and the client should drop the entire connection instead of skipping the frame. Servers should ignore any frames received from the client, not treat them as errors.
Error payloads all have the following fields:
- `error` (string, required): the error type name, with no namespace or `#` prefix
- `message` (string, optional): a description of the error
Streams should be closed immediately following transmitting or receiving an error frame.
Message payloads must always be objects. They should omit the `$type` field, as this information is already indicated in the header. There is no specific limit on the size of WebSocket frames in atproto, but they should be kept reasonably small (around a couple megabytes).
If a client can not keep up with the rate of messages, the server may send a "too slow" error and close the connection.
### Sequence Numbers
Streams can optionally make use of per-message sequence numbers to improve the reliability of transmission. Clients keep track of the last sequence number they received and successfully processed, and can specify that number after a re-connection to receive any missed messages, up to some roll-back window. Servers persist no client state across connections. The semantics are similar to [Apache Kafka](https://en.wikipedia.org/wiki/Apache_Kafka)'s consumer groups and other stream-processing protocols.
Subscription Lexicons must include a `seq` field (integer type), and a `cursor` query parameter (integer type). Not all message types need to include `seq`. Errors do not, and it is common to have an `#info` message type that is not persisted.
Sequence numbers are always positive integers (non-zero), and increase monotonically, but otherwise have flexible semantics. They may contain arbitrary gaps. For example, they might be timestamps.
To prevent confusion when working with Javascript (which by default represents all numbers as floating point), sequence numbers should be limited to the range of integers which can safely be represented by a 64-bit float. That is, the integer range `1` to `2^53` (not inclusive on the upper bound).
The connection-time rules for cursors and sequence numbers:
- no `cursor` is specified: the server starts transmitting from the current stream position
- `cursor` is higher than current `seq` ("in the future"): server sends an error message and closes connection
- `cursor` is in roll-back window: server sends any persisted messages with greater-or-equal `seq` number, then continues once "caught up" with current stream
- `cursor` is older than roll-back window: the first message in stream is an info indicating that `cursor` is too-old, then starts at the oldest available `seq` and sends the entire roll-back window, then continues with current stream
- `cursor` is `0`: server will start at the oldest available `seq`, send the entire roll-back window, then continue with current stream
The scope for sequence numbers is the combination of service provider (hostname) and endpoint (NSID). This roughly corresponds to the `wss://` URL used for connections. That is, sequence numbers may or may not be unique across different stream endpoints on the same service.
Services should ensure that sequence numbers are not re-used, usually by committing events (with sequence number) to robust persistent storage before transmitting them over streams.
In some catastrophic failure modes (or large changes to infrastructure), it is possible that a server would lose data from the backfill window, and need to reset the sequence number back to `1`. In this case, if a client re-connects with a higher number, the server would send back a `FutureCursor` error to the client. The client needs to decide what strategy to follow in these scenarios. We suggest that clients treat out-of-order or duplicate sequence numbers as an error, not process the message, and drop the connection. Most clients should not reset sequence state without human operator intervention, though this may be a reasonable behavior for some ephemeral clients not requiring reliable delivery of every event in the stream.
## Usage and Implementation Guidelines
The current stream transport is primarily designed for server-to-server data synchronization. It is also possible for web applications to connect directly from end-user browsers, but note that decoding binary frames and DAG-CBOR is non-trivial.
The combination of HTTP redirects and WebSocket upgrades is not consistently supported by WebSocket client libraries. Support is not specifically required or forbidden in atproto.
Supported versions of the WebSockets standard are not specified by atproto. The current stable WebSocket standard is version 13. Implementations should make reasonable efforts to support modern versions, with some window of backwards compatibility.
WebSockets have distinct resource rate-limiting and denial-of-service issues. Network bandwidth limits and throttling are recommended for both servers and clients. Servers should tune concurrent connection limits and buffer sizes to prevent resource exhaustion.
If services need to reset sequence state, it is recommended to chose a new initial sequence number with a healthy margin above any previous sequence number. For example, after persistent storage loss, or if clearing prior stream state.
URLs referencing a stream endpoint at a particular host should generally use `wss://` as the URI scheme (as opposed to `https://`).
## Security and Privacy Considerations
As mentioned in the "Connection" section, only `wss://` (SSL) should be used for stream connections over the internet. Public services should reject non-SSL connections.
Most HTTP XRPC endpoints work with content in JSON form, while stream endpoints work directly with DAG-CBOR objects as untrusted input. Precautions must be taken against hostile data encoding and data structure manipulation. Specific issues are discussed in the [Data Model](/specs/data-model) and [Repository](/specs/repository) specifications.
## Possible Future Changes
Event Streams are one of the newest components of the AT Protocol, and the details are more likely to be iterated on compared to other components.
The sequence number scheme may be tweaked to better support sharded streams. The motivation would be handle higher data throughputs over the public internet by splitting across multiple connections.
Additional transports (other than WebSocket) and encodings (other than DAG-CBOR) may be specified. For example, JSON payloads in text WebSocket frames would be simpler to decode in browsers.
Additional WebSocket features may be adopted:
- transport compression "extensions" like `permessage-deflate`
- definition of a sub-protocol
- bi-directional messaging
- 1000-class response codes
Ambiguities in this specification may be resolved, or left open. For example:
- HTTP redirects
- CORS and other issues for browser connections
- maximum message/frame size
Authentication schemes may be supported, similar to those for regular HTTP XRPC endpoints.
--------------------------------------------------------------------------------
# specs > handle
# Handle
DIDs are the long-term persistent identifiers for accounts in atproto, but they can be opaque and unfriendly for human use. Handles are a less-permanent identifier for accounts. The mechanism for verifying the link between an account handle and an account DID relies on DNS, and possibly connections to a network host, so every handle must be a valid network hostname. *Almost* every valid "hostname" is also a valid handle, though there are a small number of exceptions. {{ className: 'lead' }}
The definition "hostnames" (as a subset of all possible "DNS names") has evolved over time and across several RFCs. Some relevant documents are [RFC-1035](https://www.rfc-editor.org/rfc/rfc1035), [RFC-3696](https://www.rfc-editor.org/rfc/rfc3696) section 2, and [RFC-3986](https://www.rfc-editor.org/rfc/rfc3986) section 3. {{ className: 'lead' }}
## Handle Identifier Syntax
Lexicon string format type: `handle`
To synthesize other standards, and define "handle" syntax specifically:
- The overall handle must contain only ASCII characters, and can be at most 253 characters long (in practice, handles may be restricted to a slightly shorter length)
- The overall handle is split in to multiple segments (referred to as "labels" in standards documents), separated by ASCII periods (`.`)
- No proceeding or trailing ASCII periods are allowed, and there must be at least two segments. That is, "bare" top-level domains are not allowed as handles, even if valid "hostnames" and "DNS names." "Trailing dot" syntax for DNS names is not allowed for handles.
- Each segment must have at least 1 and at most 63 characters (not including the periods). The allowed characters are ASCII letters (`a-z`), digits (`0-9`), and hyphens (`-`).
- Segments can not start or end with a hyphen
- The last segment (the "top level domain") can not start with a numeric digit
- Handles are not case-sensitive, and should be normalized to lowercase (that is, normalize ASCII `A-Z` to `a-z`)
To be explicit (the above rules already specify this), no whitespace, null bytes, joining characters, or other ASCII control characters are allowed in the handle, including as prefix/suffix.
Modern "hostnames" (and thus handles) allow ASCII digits in most positions, with the exception that the last segment (top-level domain, TLD) cannot start with a digit.
IP addresses are not valid syntax: IPv4 addresses have a final segment starting with a digit, and IPv6 addresses are separated by colons (`:`).
A reference regular expression (regex) for the handle syntax is:
```
/^([a-zA-Z0-9]([a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?\.)+[a-zA-Z]([a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?$/
```
## Additional Non-Syntax Restrictions
"Reserved" top-level domains should not fail syntax validation (eg, in atproto Lexicon validation), but they must immediately fail any attempt at registration, resolution, etc. See also: [https://en.wikipedia.org/wiki/Top-level_domain#Reserved_domains](https://en.wikipedia.org/wiki/Top-level_domain#Reserved_domains)
`.local` hostnames (for mDNS on local networks) should not be used in atproto.
The `.onion` TLD is a special case for Tor protocol hidden services. Resolution of handles via Tor would require ecosystem-wide support, so they are currently disallowed.
To summarize the above, the initial list of disallowed TLDs includes:
- `.alt`
- `.arpa`
- `.example`
- `.internal`
- `.invalid`
- `.local`
- `.localhost`
- `.onion`
The `.test` TLD is intended for examples, testing, and development. It may be used in atproto development, but should fail in real-world environments.
The `.invalid` TLD should only be used for the special `handle.invalid` value (see below). This value is syntactically valid in the Lexicon schema language, but should not be accepted as a valid handle in most contexts.
## Identifier Examples
Syntactically valid handles (which may or may not have existing TLDs):
```
jay.bsky.social
8.cn
name.t--t // not a real TLD, but syntax ok
XX.LCS.MIT.EDU
a.co
xn--notarealidn.com
xn--fiqa61au8b7zsevnm8ak20mc4a87e.xn--fiqs8s
xn--ls8h.test
example.t // not a real TLD, but syntax ok
```
Invalid syntax:
```
[email protected]
💩.test
john..test
xn--bcher-.tld
john.0
cn.8
www.masełkowski.pl.com
org
name.org.
```
Valid syntax, but must always fail resolution due to other restrictions:
```
2gzyxa5ihm7nsggfxnu52rck2vv4rvmdlkiu3zzui5du4xyclen53wid.onion
laptop.local
blah.arpa
```
## Handle Resolution
Handles have a limited role in atproto, and need to be resolved to a DID in almost all situations. Resolution mechanisms must demonstrate a reasonable degree of authority over the domain name at a point in time, and need to be relatively efficient to look up. There are currently two supported resolution mechanisms, one using a TXT DNS record containing the DID, and another over HTTPS at a special `/.well-known/` URL.
Clients can rely on network services (eg, their PDS) to resolve handles for them, using the `com.atproto.identity.resolveHandle` endpoint, and don't usually need to implement resolution directly themselves.
The DNS TXT method is the recommended and preferred resolution method for individual handle configuration, but services should fully support both methods. The intended use-case for the HTTPS method is existing large-scale web services which may not have the infrastructure to automate the registration of thousands or millions of DNS TXT records.
Handles should not be trusted or considered valid until the DID is also resolved and the current DID document is confirmed to link back to the handle. The link between handle and DID must be confirmed bidirectionally, otherwise anybody could create handle aliases for third-party accounts.
### DNS TXT Method
For this resolution method, a DNS TXT record is registered for the `_atproto` sub-domain under the handle hostname. The record value should have the prefix `did=`, followed by the full domain. This method aligns with [RFC-1464](https://www.rfc-editor.org/rfc/rfc1464.html), "Using the Domain Name System To Store Arbitrary String Attributes".
For example, the handle `bsky.app` would have a TXT record on the name `_atproto.bsky.app`, and the value would look like `did=did:plc:z72i7hdynmk6r22z27h6tvur`.
Any TXT records with values not starting with `did=` should be ignored. Only a single valid record should exist at any point in time. If multiple valid records with different DIDs are present, resolution should fail. In this case resolution can be re-tried after a delay, or using a recursive resolver.
Note that very long handles can not be resolved using this method if the additional `_atproto.` name segment pushes the overall name over the 253 character maximum for DNS queries. The HTTPS method will work for such handles.
DNSSEC is not required.
### HTTPS well-known Method
For this resolution method, a web server at the handle domain implements a special well-known endpoint at the path `/.well-known/atproto-did`. A valid HTTP response will have an HTTP success status (2xx), `Content-Type` header set to `text/plain`, and include the DID as the HTTP body with no prefix or wrapper formatting.
For example, the handle `bsky.app` would be resolved by an GET request to `https://bsky.app/.well-known/atproto-did`, and a valid response would look like:
```
HTTP/1.1 200 OK
Content-Length: 33
Content-Type: text/plain
Date: Wed, 14 Jun 2023 00:47:21 GMT
did:plc:z72i7hdynmk6r22z27h6tvur
```
The response `Content-Type` header does not need to be strictly verified.
It is acceptable to strip any prefix and suffix whitespace from the response body before attempting to parse as a DID.
Secure HTTPS on the default port (443) is required for all real-world handle resolutions. HTTP should only be used for local development and testing.
HTTP redirects (eg, 301, 302) are allowed, up to a reasonable number of redirect hops.
### Invalid Handles
If the handle for a known DID is confirmed to no longer resolve, it should be marked as invalid. In API responses, the special handle value `handle.invalid` can be used to indicate that there is no bi-directionally valid handle for the given DID. This handle can not be used in most situations (search queries, API requests, etc).
### Resolution Best Practices
It is ok to attempt both resolution methods in parallel, and to use the first successful result available. If the two methods return conflicting results (aka, different DIDs), the DNS TXT result should be preferred, though it is also acceptable to record the result as ambiguous and try again later.
It is considered a best practice for services to cache handle resolution results internally, up to some lifetime, and re-resolve periodically. DNS TTL values provide a possible cache lifetime, but are probably too aggressive (aka, too short a lifetime) for the handle resolution use case.
Use of a recursive DNS resolver can help with propagation delays, which are
With both techniques, it is beneficial to initiate resolution requests from a relatively trusted network environment and configuration. Running resolution requests from multiple regions and environments can help mitigate (though not fully resolve) concerns about traffic manipulation or intentionally segmented responses.
## Usage and Implementation Guidelines
Handles may be prefixed with the "at" symbol (like `@jay.bsky.team`) in user interfaces, but this is not a valid syntax for a handle in records, APIs, and other back-end contexts.
Internationalized Domain Names ("IDN", or "punycode") are not directly relevant to the low-level handle syntax. In their encoded form, IDNs are already valid hostnames, and thus valid handles. Such handles must be stored and transmitted in encoded ASCII form. Handles that "look like" IDNs, but do not parse as valid IDNs, are valid handles, just as they are valid hostnames. Applications may, optionally, parse and display IDN handles as Unicode.
Handles are not case-sensitive, which means they can be safely normalized from user input to lower-case (ASCII) form. Only normalized (lowercase) handles should be stored in records or used in outbound API calls. Applications should not preserve user-provided case information and attempt to display handles in anything other than lower-case. For example, the handle input string `BlueskyWeb.xyz` should be normalized, stored, and displayed as `blueskyweb.xyz`. Long all-lowercase handles can be a readability and accessibility challenge. Sub-domain separation (periods), hyphenation, or use of "display names" in application protocols can all help.
Very long handles are known to present user interface challenges, but they are allowed in the protocol, and application developers are expected to support them.
Handles which look similar to a well-known domain present security and impersonation challenges. For example, handles like `paypa1.com` or `paypal.cc` being confused for `paypal.com`. Very long handles can result in similar issues when truncated at the start or end (`paypal.com…`).
Handles should generally not be truncated to local context. For example, the handle `@jay.bsky.social` should not be displayed as `@jay`, even in the local context of a `bsky.social` service.
Providers of handle "namespaces" (eg, as subdomains on a registered domain) may impose any additional limits on handles that they wish. It is recommended to constrain the allowed segment length to something reasonable, and to reserve a common set of segment strings like `www`, `admin`, `mail`, etc. There are multiple public lists of "commonly disallowed usernames" that can be used as a starting point.
From a practical standpoint, handles should be limited to at most 244 characters, fewer than the 253 allowed for DNS names. This is because DNS verification works with the prefix `_atproto.`, which adds 9 characters, and that overall name needs to be valid.
Handle hostnames are expected to be mainstream DNS domain names, registered through the mainstream DNS name system. Handles with non-standard TLDs, or using non-standard naming systems, will fail to interoperate with other network services and protocol implementations in the atproto ecosystem.
PDS implementations hosting an account *may* prevent repo mutation if the account's handle can no longer be verified (aka, `handle.invalid` situation). Other network services should generally continue to display the content (to prevent breakage), possibly with a contextual note or warning indicator.
## Possible Future Changes
The handle syntax is relatively stable.
It is conceivable that `.onion` handles would be allowed at some point in the future.
--------------------------------------------------------------------------------
# specs > label
# Labels
**Labels** are a form of metadata about any account or content in the atproto ecosystem. {{ className: 'lead' }}
They exist as free-standing, self-authenticated data objects, though they are also frequently distributed as part of API responses (in which context the signatures might not be included). Additionally, label "values" may be directly embedded in records themselves ("self-labels"). {{ className: 'lead' }}
Labels primarily consist of a source (DID), a subject (URI), and value. The value is a short string, similar to a tag or hashtag, which presumably has pre-defined semantics shared between the creator and consumer of the label. Additional metadata fields can give additional context, but at any point of time there should be only one coherent set of metadata for the combination of source, subject, and value. If there are multiple sets of metadata, the `created-at` timestamp is used to clarify which label is current. {{ className: 'lead' }}
The label concept and protocol primitive is flexible in scope, use cases, and data transport. One of the original design motivations was to enable some forms of composable moderation, with labels generated by moderation services. But labels are not moderation-exclusive, and may be reused freely for other purposes in atproto applications. {{ className: 'lead' }}
The core label schema is versioned, and this document describes labels version `1`. {{ className: 'lead' }}
## Schema and Data Model
Labels are protocol objects, similar to repository commits or MST nodes. They are canonically encoded as DAG-CBOR (a strict, normalized subset of CBOR) for for signing (see sections below). There is a Lexicon definition (`com.atproto.label.defs#label`) which represents labels, but it has slightly different field requirements than the core protocol object: the version field (`ver`) and signature (`sig`) are both optional in that version.
The fields on the label object are:
- `ver` (integer, required): label schema version. Current version is always `1`.
- `src` (string, DID format, required): the authority (account) which generated this label
- `uri` (string, URI format, required): the content that this label applies to. For a specific record, an `at://` URI. For an account, the `did:`.
- `cid` (string, CID format, optional): if provided, the label applies to a specific version of the subject `uri`
- `val` (string, 128 bytes max, required): the value of the label. Semantics and preferred syntax discussed below.
- `neg` (boolean, optional): if `true`, indicates that this label "negates" an earlier label with the same `src`, `uri`, and `val`.
- `cts` (string, datetime format, required): the timestamp when the label was created. Note that timestamps in a distributed system are not trustworthy or verified by default.
- `exp` (string, datetime format, optional): a timestamp at which this label expires (is not longer valid)
- `sig` (bytes, optional): cryptographic signature bytes. Uses the `bytes` type from the [Data Model](/specs/data-model), which encodes in JSON as a `$bytes` object with base64 encoding
When labels are being transferred as full objects between services, the `ver` and `sig` fields are required.
If the `neg` field is `false`, best practice is to simply not include the field at all.
The use of short three-character field names is different from most parts of atproto, and aligns more closely with JWTs. The motivation for this difference is to minimize the size of labels when many of them are included in network requests and responses.
## Value
The `val` field is core to the label. To keep the protocol flexible, and allow future development of ontologies, norms, and governance structures, very little about the semantics, behavior, and known values of this string are specified here.
The current expectation is that label value strings are "tokens" with fixed vocabulary. They are similar to hashtags.
At this time, we strongly recommend *against* the following patterns:
- packing additional structure in to value fields. for example, base64-encoded data, key/value syntax, lists or arrays of values, etc
- encoding arbitrary numerical values (eg, "scores" or "confidence")
- using punctuation characters (like `.`, `:`, `;`, `#`, `_`, `'`, `>`, or others) to structure the namespace of labels
- using URLs or URIs in values
- use of any whitespace
- use of non-ASCII characters, including emoji
These are all promising ideas, but we hope to coordinate and more formally specify this sort of syntax extension.
One current convention is to use a bang punctuation character (`!`) as a prefix for system-level labels which specify an expected behavior on a subject, but don't describe the content or indicate a reason for the behavior. For example, `!warn` as a behavior, as opposed to `scam` as a descriptive label which might result in the same warning behavior.
The behavior, definition, meaning, and policies around labels are generally communicated elsewhere. The value does not need to be entirely descriptive.
### Recommended String Syntax
The current recommended syntax for label strings is lower-case kebab-syntax (using `-` internally), using only ASCII letters. Specifically:
- lower-case alphabetical ASCII letters (`a` to `z`)
- dash (`-`) used for internal separation, but not as a first or last character
- no other punctuation or whitespace
- 128 bytes maximum length. Shorter is better (try to keep labels to a couple dozen characters at most), while still being somewhat descriptive.
## Label Lifecycle: Negation and Expiration
Labels are generally broadcast and persisted internally by receiving services. Some services may bulk re-broadcast or re-distribute labels to downstream services. They may ignore and drop any labels which are not relevant to their use-case. They may "hydrate" labels in to requests from clients.
When hydrating labels, services should generally only include "active" and relevant labels.
If the authoritative creator of a label wishes to retract or remove the label, they do so by publishing a new label with the same source, subject, and value, but with the negated field (`neg`) set to true, and a current timestamp (later than any previous timestamps). A negation label does not mean that the inverse of the label is “true”, only that the previous label has been retracted. For example, a label with value `spam` and `neg` true does not mean the subject is *not* spam, only that a previous `spam` label should be disregarded.
Receiving services that encounter a valid negation label may store the negation internally, and may re-broadcast the negation, but should not hydrate the negated label in API responses.
Likewise, may continue to persist expired labels (after the expiration timestamp), but should not continue to hydrate them in API responses.
## Signatures
Labels are signed using public-key [Cryptography](/specs/cryptography), similar to repository commit objects. Signatures should be validated when labels are transferred between services. It is assumed that most end-clients will not validate signatures themselves, and signatures may be removed from API responses sent to clients for network efficiency. Clients and other parties should have a mechanism to verify signatures, by querying individual signatures from labeling authorities, and receiving back the full label, including signature.
The process to sign or verify a signature is to construct a complete version of the label, using only the specified schema fields, and not including the `sig` field. This means including the `ver` field, but not any `$type` field or other un-specified fields which may have been included in a Lexicon representation of the label. This data object is then encoded in CBOR, following the deterministic IPLD/DAG-CBOR normalization rules. The CBOR bytes are hashed with SHA-256, and then the direct hash bytes (not a hex-encoded string) are signed (or verified) using the appropriate cryptographic key. The signature bytes are stored in the `sig` field as bytes (see [Data Model](/specs/data-model) for details representing bytes).
The key used for signing labels is found in the DID document for the issuing identity, and has fragment identifier `#atproto_label`. This key *may* have the same value as the `#atproto` signing key used for repository signatures. At this time, if an `#atproto_label` key is not found, implementation *should not* attempt to use other keys present in the DID document to verify the signature, they should simply consider the signature invalid and ignore the label.
### Signature Lifecycle
Signatures are verified at the time the label is received. They do not need to be re-verified before hydration in to API responses.
Signing key rotation can be difficult and disruptive for a large labeling service. The rough mechanism for doing a rotation is:
- labeling services should persist signatures alongside labels, and also persist an indicator of which key was used to sign the label
- when starting a rotation, pause creation and signing of new signatures
- update the DID document with the new key
- resume signing new labels with the new key
- when servicing queries for old labels, check which key was used for signing. if out of date, re-sign and persist the signatures for that batch of labels. the created-at timestamp should not be changed.
- older labels in the label event stream backfill period may have invalid signatures; this is acceptable
When encountering a label with an invalid signature, a good practice is to re-resolve the issuer identity (DID document) and check if there is an updated signing key. If there is, validation should be retried.
A downstream service can decide for themselves whether to bulk query and receive updated signatures when an upstream key rotation has occurred; or to fetch updated signatures on demand; or to consider old labels still valid even if the signature no longer validates against the current label signing key.
## Self-Labels in Records
One Lexicon design pattern is to include an array of label values inside a record. Downstream clients can interpret these as "self-labels", similarly to labels coming from external sources.
Note that the repository data storage mechanism provides context and lifecycle support similar to a full label object:
- the source of the label is the account controlling the repository
- the subject is the record itself, or possibly the overall repository account (depending on context)
- the CID is the current version of the record
- negation and expiration are not necessary, as the record can be deleted or updated to change the set of labels
- the created-at timestamp would be the same as the record itself (eg, via a `createdAt` field)
- authenticity (signature) is provided by the repository commit signing mechanism
## Labeler Service Identity
Labeler services each have a service identity, meaning a DID document. This is the DID that appears in label source (`src`) field.
The DID document will also have a key used for signing labels (with ID `#atproto_label`; see above for signature details), and a service endpoint (with ID `#atproto_labeler` and type `AtprotoLabeler`) which indicates the server URL (the URL includes method, hostname, and optional port, but no path segment at this time). Note that in real world use-cases, use of HTTPS on the default port (443) is strongly recommended and may be required by service operators.
Depending on the application, the identity may also have an atproto repository containing a “declaration record” which describes application-specific context about the labeler. This may be required for integration with a specific application, client, or AppView, but is not a requirement at the base atproto level.
## Label Distribution Endpoints
Two Lexicon endpoints are defined for labeler services to distribute labels:
`com.atproto.label.subscribeLabels`: an event stream (WebSocket) endpoint, which broadcasts new labels. Implements the `seq` backfill mechanism, similar to repository event stream, but with some small differences: the “backfill” period may extend to `cursor=0` (meaning that the full history of labels is available via the stream). Labels which have been redacted have the original label removed from the stream, but the negation remains.
`com.atproto.label.queryLabels`: a flexible query endpoint. Can be used to scroll over all labels (using a cursor parameter), or can filter to labels relating to a specific subject.
Note that unlike public repository content, labels are not *required* to be publicly enumerable. It is acceptable for labeler services to make all labels publicly available using these endpoints, or to require authorization and access control, or to not implement these endpoints at all (if they have another mechanism for distributing labels).
## Labeler HTTP Headers
Labels are often “hydrated” in to HTTP API responses by atproto services, such as AppViews. To give clients control over which label sources they want included, two special HTTP headers are used, which PDS implementations are expected to pass-through when proxying requests:
`atproto-accept-labelers`: used in requests. A list of labeler service DIDs, with optional per-DID flags.
`atproto-content-labelers`: used in responses. Same content, syntax, and semantics as the `accept` header, but indicates which labelers could actually be queried. Presence in this header doesn’t mean that any labels from a given DID were actually included, only that it they would have been if such labels existed.
The syntax of these headers follows IETF RFC-8941 (”Structured Field Values for HTTP”), section 3.1.2 (”Parameters”). Values are separated by comma (ASCII `,` character), and values from repeated declaration of the header should be merged in to a single list. One or more optional parameters may follow the item value (the DID), separated by a semicolon (ASCII `;` character). For boolean parameters, the full RFC syntax (just as `param=?0` for false) is not currently supported. Instead, the presence of the parameter indicates it is “true”, and the absence indicates “false”. No other parameter values types (such as integers or strings) are supported at this time.
The only currently supported parameter is the boolean parameter `redact`. This flag indicates that the service hydrating labels should handle the special protocol-level label values `!takedown` and `!suspend` by entirely redacting content from the API response, instead of simply labeling it. This may result in an application-specific tombstone entry, which might indicate the Labeler responsible for the redaction, or could result in the content being removed without a tombstone.
Complete example syntax for these headers:
```
# on a request
atproto-accept-labelers: did:web:mod.example.com;redact, did:plc:abc123, did:plc:xyz789
# on a response:
atproto-content-labelers: did:web:mod.example.com;redact, did:plc:abc123, did:plc:xyz789
```
If the syntax of the request header is invalid or can not be parsed, the service should return an error instead of ignoring the header.
If a labeler DID is repeated in the header, parameters should be combined from each instance. For example, if a DID is included once with `redact` and once without, the service should treat this the same as if the DID was included once, with `redact`. The `atproto-content-labelers` response header should represent how the request header was de-duplicated and interpreted.
If the request header is not supplied at all, the service may substitute a default. This is distinct from supplying the header with no value, in which case the service should not hydrate or apply any labels.
If any labeler DID is indicated with correct syntax, but the identity does not exist; does not include labeler service or key entries in the DID doc; has been taken down at the service level; or is otherwise inactive or non-functional; then that labeler should not be included in the `atproto-content-labelers` response header, but does not need to be treated as an error.
A service implementation may decide, as a policy matter, that specific conditions must be met or the request will error. For example, that a specific labeler DID must be included; or a minimum or maximum number of labelers can be included; or a minimum or maximum number of labelers with `redact` are included.
## Security Considerations
Note that there is no "domain differentiation" of the signature, meaning that there is potential security risk of signing a label which is also a valid object (and signature) in an entirely different context, like an authentication bearer token. This makes it
## Usage and Implementation Guidelines
It is strongly recommended to stick to the “recommended string syntax” for label values at this time.
## Possible Future Changes
More mature governance, namespacing, and style guide recommendations on label values.
--------------------------------------------------------------------------------
# specs > lexicon
# Lexicon
Lexicon is a schema definition language used to describe atproto records, HTTP endpoints (XRPC), and event stream messages. It builds on top of the atproto [Data Model](/specs/data-model). {{ className: 'lead' }}
The schema language is similar to [JSON Schema](http://json-schema.org/) and [OpenAPI](https://en.wikipedia.org/wiki/OpenAPI_Specification), but includes some atproto-specific features and semantics. {{ className: 'lead' }}
This specification describes version 1 of the Lexicon definition language. {{ className: 'lead' }}
## Overview of Types
| Lexicon Type | Data Model Type | Category |
| --- | --- | --- |
| `null` | Null | concrete |
| `boolean` | Boolean | concrete |
| `integer` | Integer | concrete |
| `string` | String | concrete |
| `bytes` | Bytes | concrete |
| `cid-link` | Link | concrete |
| `blob` | Blob | concrete |
| `array` | Array | container |
| `object` | Object | container |
| `params` | | container |
| `token` | | meta |
| `ref` | | meta |
| `union` | | meta |
| `unknown` | | meta |
| `record` | | primary |
| `query` | | primary |
| `procedure` | | primary |
| `subscription` | | primary |
## Lexicon Files
Lexicons are JSON files associated with a single NSID. A file contains one or more definitions, each with a distinct short name. A definition with the name `main` optionally describes the "primary" definition for the entire file. A Lexicon with zero definitions is invalid.
A Lexicon JSON file is an object with the following fields:
- `lexicon` (integer, required): indicates Lexicon language version. In this version, a fixed value of `1`
- `id` (string, required): the NSID of the Lexicon
- `description` (string, optional): short overview of the Lexicon, usually one or two sentences
- `defs` (map of strings-to-objects, required): set of definitions, each with a distinct name (key)
Schema definitions under `defs` all have a `type` field to distinguish their type. A file can have at most one definition with one of the "primary" types. Primary types should always have the name `main`. It is possible for `main` to describe a non-primary type.
References to specific definitions within a Lexicon use fragment syntax, like `com.example.defs#someView`. If a `main` definition exists, it can be referenced without a fragment, just using the NSID. For references in the `$type` fields in data objects themselves (eg, records or contents of a union), this is a "must" (use of a `#main` suffix is invalid). For example, `com.example.record` not `com.example.record#main`.
Related Lexicons are often grouped together in the NSID hierarchy. As a convention, any definitions used by multiple Lexicons are defined in a dedicated `*.defs` Lexicon (eg, `com.atproto.server.defs`) within the group. A `*.defs` Lexicon should generally not include a definition named `main`, though it is not strictly invalid to do so.
## Primary Type Definitions
The primary types are:
- `query`: describes an XRPC Query (HTTP GET)
- `procedure`: describes an XRPC Procedure (HTTP POST)
- `subscription`: Event Stream (WebSocket)
- `record`: describes an object that can be stored in a repository record
Each primary definition schema object includes these fields:
- `type` (string, required): the type value (eg, `record` for records)
- `description` (string, optional): short, usually only a sentence or two
### Record
Type-specific fields:
- `key` (string, required): specifies the [Record Key type](/specs/record-key)
- `record` (object, required): a schema definition with type `object`, which specifies this type of record
### Query and Procedure (HTTP API)
Type-specific fields:
- `parameters` (object, optional): a schema definition with type `params`, describing the HTTP query parameters for this endpoint
- `output` (object, optional): describes the HTTP response body
- `description` (string, optional): short description
- `encoding` (string, required): MIME type for body contents. Use `application/json` for JSON responses.
- `schema` (object, optional): schema definition, either an `object`, a `ref`, or a `union` of refs. Used to describe JSON encoded responses, though schema is optional even for JSON responses.
- `input` (object, optional, only for `procedure`): describes HTTP request body schema, with the same format as the `output` field
- `errors` (array of objects, optional): set of string error codes which might be returned
- `name` (string, required): short name for the error type, with no whitespace
- `description` (string, optional): short description, one or two sentences
### Subscription (Event Stream)
Type-specific fields:
- `parameters` (object, optional): same as Query and Procedure
- `message` (object, optional): specifies what messages can be
- `description` (string, optional): short description
- `schema` (object, required): schema definition, which must be a `union` of refs
- `errors` (array of objects, optional): same as Query and Procedure
Subscription schemas (referenced by the `schema` field under `message`) must be a `union` of refs, not an `object` type.
## Field Type Definitions
As with the primary definitions, every schema object includes these fields:
- `type` (string, required): fixed value for each type
- `description` (string, optional): short, usually only a sentence or two
### `null`
No additional fields.
### `boolean`
Type-specific fields:
- `default` (boolean, optional): a default value for this field
- `const` (boolean, optional): a fixed (constant) value for this field
When included as an HTTP query parameter, should be rendered as `true` or `false` (no quotes).
### `integer`
A signed integer number.
Type-specific fields:
- `minimum` (integer, optional): minimum acceptable value
- `maximum` (integer, optional): maximum acceptable value
- `enum` (array of integers, optional): a closed set of allowed values
- `default` (integer, optional): a default value for this field
- `const` (integer, optional): a fixed (constant) value for this field
### `string`
Type-specific fields:
- `format` (string, optional): string format restriction
- `maxLength` (integer, optional): maximum length of value, in UTF-8 bytes
- `minLength` (integer, optional): minimum length of value, in UTF-8 bytes
- `maxGraphemes` (integer, optional): maximum length of value, counted as Unicode Grapheme Clusters
- `minGraphemes` (integer, optional): minimum length of value, counted as Unicode Grapheme Clusters
- `knownValues` (array of strings, optional): a set of suggested or common values for this field. Values are not limited to this set (aka, not a closed enum).
- `enum` (array of strings, optional): a closed set of allowed values
- `default` (string, optional): a default value for this field
- `const` (string, optional): a fixed (constant) value for this field
Strings are Unicode. For non-Unicode encodings, use `bytes` instead. The basic `minLength`/`maxLength` validation constraints are counted as UTF-8 bytes. Note that Javascript stores strings with UTF-16 by default, and it is necessary to re-encode to count accurately. The `minGraphemes`/`maxGraphemes` validation constraints work with Grapheme Clusters, which have a complex technical and linguistic definition, but loosely correspond to "distinct visual characters" like Latin letters, CJK characters, punctuation, digits, or emoji (which might comprise multiple Unicode codepoints and many UTF-8 bytes).
`format` constrains the string format and provides additional semantic context. Refer to the Data Model specification for the available format types and their definitions.
`const` and `default` are mutually exclusive.
### `bytes`
Type-specific fields:
- `minLength` (integer, optional): minimum size of value, as raw bytes with no encoding
- `maxLength` (integer, optional): maximum size of value, as raw bytes with no encoding
### `cid-link`
No type-specific fields.
See [Data Model spec](/specs/data-model) for CID restrictions.
### `array`
Type-specific fields:
- `items` (object, required): describes the schema elements of this array
- `minLength` (integer, optional): minimum count of elements in array
- `maxLength` (integer, optional): maximum count of elements in array
In theory arrays have homogeneous types (meaning every element as the same type). However, with union types this restriction is meaningless, so implementations can not assume that all the elements have the same type.
### `object`
A generic object schema which can be nested inside other definitions by reference.
Type-specific fields:
- `properties` (map of strings-to-objects, required): defines the properties (fields) by name, each with their own schema
- `required` (array of strings, optional): indicates which properties are required
- `nullable` (array of strings, optional): indicates which properties can have `null` as a value
As described in the data model specification, there is a semantic difference in data between omitting a field; including the field with the value `null`; and including the field with a "false-y" value (`false`, `0`, empty array, etc).
### `blob`
Type-specific fields:
- `accept` (array of strings, optional): list of acceptable MIME types. Each may end in `*` as a glob pattern (eg, `image/*`). Use `*/*` to indicate that any MIME type is accepted.
- `maxSize` (integer, optional): maximum size in bytes
### `params`
This is a limited-scope type which is only ever used for the `parameters` field on `query`, `procedure`, and `subscription` primary types. These map to HTTP query parameters.
Type-specific fields:
- `required` (array of strings, optional): same semantics as field on `object`
- `properties`: similar to properties under `object`, but can only include the types `boolean`, `integer`, `string`, and `unknown`; or an `array` of one of these types
Note that unlike `object`, there is no `nullable` field on `params`.
### `token`
Tokens are empty data values which exist only to be referenced by name. They are used to define a set of values with specific meanings. The `description` field should clarify the meaning of the token. Tokens encode as string data, with the string being the fully-qualified reference to the token itself (NSID followed by an optional fragment).
Tokens are similar to the concept of a "symbol" in some programming languages, distinct from strings, variables, built-in keywords, or other identifiers.
For example, tokens could be defined to represent the state of an entity (in a state machine), or to enumerate a list of categories.
No type-specific fields.
### `ref`
Type-specific fields:
- `ref` (string, required): reference to another schema definition
Refs are a mechanism for re-using a schema definition in multiple places. The `ref` string can be a global reference to a Lexicon type definition (an NSID, optionally with a `#`-delimited name indicating a definition other than `main`), or can indicate a local definition within the same Lexicon file (a `#` followed by a name).
### `union`
Type-specific fields:
- `refs` (array of strings, required): references to schema definitions
- `closed` (boolean, optional): indicates if a union is "open" or "closed". defaults to `false` (open union)
Unions represent that multiple possible types could be present at this location in the schema. The references follow the same syntax as `ref`, allowing references to both global or local schema definitions. Actual data will validate against a single specific type: the union does not *combine* fields from multiple schemas, or define a new *hybrid* data type. The different types are referred to as **variants**.
By default unions are "open", meaning that future revisions of the schema could add more types to the list of refs (though can not remove types). This means that implementations should be permissive when validating, in case they do not have the most recent version of the Lexicon. The `closed` flag (boolean) can indicate that the set of types is fixed and can not be extended in the future.
A `union` schema definition with no `refs` is allowed and similar to `unknown`, as long as the `closed` flag is false (the default). The main difference is that the data would be required to have the `$type` field. An empty refs list with `closed` set to true is an invalid schema.
The schema definitions pointed to by a `union` are objects or types with a clear mapping to an object, like a `record`. All the variants must be represented by a CBOR map (or JSON Object) and must include a `$type` field indicating the variant type. Because the data must be an object, unions can not reference `token` (which would correspond to string data).
### `unknown`
Indicates than any data object could appear at this location, with no specific validation. The top-level data must be an object (not a string, boolean, etc). As with all other data types, the value `null` is not allowed unless the field is specifically marked as `nullable`.
The data object may contain a `$type` field indicating the schema of the data, but this is not currently required. The top-level data object must not have the structure of a compound data type, like blob (`$type: blob`) or CID link (`$link`).
The (nested) contents of the data object must still be valid under the atproto data model. For example, it should not contain floats. Nested compound types like blobs and CID links should be validated and transformed as expected.
Lexicon designers are strongly recommended to not use `unknown` fields in `record` objects for now.
No type-specific fields.
## String Formats
Strings can optionally be constrained to one of the following `format` types:
- `at-identifier`: either a [Handle](/specs/handle) or a [DID](/specs/did), details described below
- `at-uri`: [AT-URI](/specs/at-uri-scheme)
- `cid`: CID in string format, details specified in [Data Model](/specs/data-model)
- `datetime`: timestamp, details specified below
- `did`: generic [DID Identifier](/specs/did)
- `handle`: [Handle Identifier](/specs/handle)
- `nsid`: [Namespaced Identifier](/specs/nsid)
- `tid`: [Timestamp Identifier (TID)](/specs/tid)
- `record-key`: [Record Key](/specs/record-key), matching the general syntax ("any")
- `uri`: generic URI, details specified below
- `language`: language code, details specified below
For the various identifier formats, when doing Lexicon schema validation the most expansive identifier syntax format should be permitted. Problems with identifiers which do pass basic syntax validation should be reported as application errors, not lexicon data validation errors. For example, data with any kind of DID in a `did` format string field should pass Lexicon validation, with unsupported DID methods being raised separately as an application error.
### `at-identifier`
A string type which is either a DID (type: did) or a handle (handle). Mostly used in XRPC query parameters. It is unambiguous whether an at-identifier is a handle or a DID because a DID always starts with did:, and the colon character (:) is not allowed in handles.
### `datetime`
Full-precision date and time, with timezone information.
This format is intended for use with computer-generated timestamps in the modern computing era (eg, after the UNIX epoch). If you need to represent historical or ancient events, ambiguity, or far-future times, a different format is probably more appropriate. Datetimes before the Current Era (year zero) as specifically disallowed.
Datetime format standards are notoriously flexible and overlapping. Datetime strings in atproto should meet the [intersecting](https://ijmacd.github.io/rfc3339-iso8601/) requirements of the [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339), [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601), and [WHATWG HTML](https://html.spec.whatwg.org/#dates-and-times) datetime standards.
The character separating "date" and "time" parts must be an upper-case `T`.
Timezone specification is required. It is *strongly* preferred to use the UTC timezone, and to represent the timezone with a simple capital `Z` suffix (lower-case is not allowed). While hour/minute suffix syntax (like `+01:00` or `-10:30`) is supported, "negative zero" (`-00:00`) is specifically disallowed (by ISO 8601).
Whole seconds precision is required, and arbitrary fractional precision digits are allowed. Best practice is to use at least millisecond precision, and to pad with zeros to the generated precision (eg, trailing `:12.340Z` instead of `:12.34Z`). Not all datetime formatting libraries support trailing zero formatting. Both millisecond and microsecond precision have reasonable cross-language support; nanosecond precision does not.
Implementations should be aware when round-tripping records containing datetimes of two ambiguities: loss-of-precision, and ambiguity with trailing fractional second zeros. If de-serializing Lexicon records into native types, and then re-serializing, the string representation may not be the same, which could result in broken hash references, sanity check failures, or repository update churn. A safer thing to do is to deserialize the datetime as a simple string, which ensures round-trip re-serialization.
Implementations "should" validate that the semantics of the datetime are valid. For example, a month or day `00` is invalid.
Valid examples:
```text
# preferred
1985-04-12T23:20:50.123Z
1985-04-12T23:20:50.123456Z
1985-04-12T23:20:50.120Z
1985-04-12T23:20:50.120000Z
# supported
1985-04-12T23:20:50.12345678912345Z
1985-04-12T23:20:50Z
1985-04-12T23:20:50.0Z
1985-04-12T23:20:50.123+00:00
1985-04-12T23:20:50.123-07:00
```
Invalid examples:
```text
1985-04-12
1985-04-12T23:20Z
1985-04-12T23:20:5Z
1985-04-12T23:20:50.123
+001985-04-12T23:20:50.123Z
23:20:50.123Z
-1985-04-12T23:20:50.123Z
1985-4-12T23:20:50.123Z
01985-04-12T23:20:50.123Z
1985-04-12T23:20:50.123+00
1985-04-12T23:20:50.123+0000
# ISO-8601 strict capitalization
1985-04-12t23:20:50.123Z
1985-04-12T23:20:50.123z
# RFC-3339, but not ISO-8601
1985-04-12T23:20:50.123-00:00
1985-04-12 23:20:50.123Z
# timezone is required
1985-04-12T23:20:50.123
# syntax looks ok, but datetime is not valid
1985-04-12T23:99:50.123Z
1985-00-12T23:20:50.123Z
```
### `uri`
Flexible to any URI schema, following the generic RFC-3986 on URIs. This includes, but isn’t limited to: `did`, `https`, `wss`, `ipfs` (for CIDs), `dns`, and of course `at`.
Maximum length in Lexicons is 8 KBytes.
### `language`
An [IETF Language Tag](https://en.wikipedia.org/wiki/IETF_language_tag) string, compliant with [BCP 47](https://www.rfc-editor.org/info/bcp47), defined in [RFC 5646](https://www.rfc-editor.org/rfc/rfc5646.txt) ("Tags for Identifying Languages"). This is the same standard used to identify languages in HTTP, HTML, and other web standards. The Lexicon string must validate as a "well-formed" language tag, as defined in the RFC. Clients should ignore language strings which are "well-formed" but not "valid" according to the RFC.
As specified in the RFC, ISO 639 two-character and three-character language codes can be used on their own, lower-cased, such as `ja` (Japanese) or `ban` (Balinese). Regional sub-tags can be added, like `pt-BR` (Brazilian Portuguese). Additional subtags can also be added, such as `hy-Latn-IT-arevela`.
Language codes generally need to be parsed, normalized, and matched semantically, not simply string-compared. For example, a search engine might simplify language tags to ISO 639 codes for indexing and filtering, while a client application (user agent) would retain the full language code for presentation (text rendering) locally.
## When to use `$type`
Data objects sometimes include a `$type` field which indicates their Lexicon type. The general principle is that this field needs to be included any time there could be ambiguity about the content type when validating data.
The specific rules are:
- `record` objects must always include `$type`. While the type is often known from context (eg, the collection part of the path for records stored in a repository), record objects can also be passed around outside of repositories and need to be self-describing
- `union` variants must always include `$type`, except at the top level of `subscription` messages
Note that `blob` objects always include `$type`, which allows generic processing.
As a reminder, `main` types must be referenced in `$type` fields as just the NSID, not including a `#main` suffix.
## Lexicon Evolution
Lexicons are allowed to change over time, within some bounds to ensure both forwards and backwards compatibility. The basic principle is that all old data must still be valid under the updated Lexicon, and new data must be valid under the old Lexicon.
- Any new fields must be optional
- Non-optional fields can not be removed. A best practice is to retain all fields in the Lexicon and mark them as deprecated if they are no longer used.
- Types can not change
- Fields can not be renamed
If larger breaking changes are necessary, a new Lexicon name must be used.
It can be ambiguous when a Lexicon has been published and becomes "set in stone". At a minimum, public adoption and implementation by a third party, even without explicit permission, indicates that the Lexicon has been released and should not break compatibility. A best practice is to clearly indicate in the Lexicon type name any experimental or development status. Eg, `com.corp.experimental.newRecord`.
## Authority and Control
The authority for a Lexicon is determined by the NSID, and rooted in DNS control of the domain authority. That authority has ultimate control over the Lexicon definition, and responsibility for maintenance and distribution of Lexicon schema definitions.
In a crisis, such as unintentional loss of DNS control to a bad actor, the protocol ecosystem could decide to disregard this chain of authority. This should only be done in exceptional circumstances, and not as a mechanism to subvert an active authority. The primary mechanism for resolving protocol disputes is to fork Lexicons in to a new namespace.
Protocol implementations should generally consider data which fails to validate against the Lexicon to be entirely invalid, and should not try to repair or do partial processing on the individual piece of data.
Unexpected fields in data which otherwise conforms to the Lexicon should be ignored. When doing schema validation, they should be treated at worst as warnings. This is necessary to allow evolution of the schema by the controlling authority, and to be robust in the case of out-of-date Lexicons.
Third parties can technically insert any additional fields they want into data. This is not the recommended way to extend applications, but it is not specifically disallowed. One danger with this is that the Lexicon may be updated to include fields with the same field names but different types, which would make existing data invalid.
## Lexicon Publication and Resolution
Lexicon schemas are published publicly as records in atproto repositories, using the `com.atproto.lexicon.schema` type. The domain name authority for [NSIDs](/specs/nsid) to specific atproto repositories (identified by [DID](/specs/did) is linked by a DNS TXT record (`_lexicon`), similar to but distinct from the [handle resolution](/specs/handle) system.
The `com.atproto.lexicon.schema` Lexicon itself is very minimal: it only requires the `lexicon` integer field, which must be `1` for this version of the Lexicon language. In practice, same fields as [Lexicon Files](#lexicon-files) should be included, along with `$type`. The record key is the NSID of the schema.
A summary of record fields:
- `$type`: must be `com.atproto.lexicon.schema` (as with all atproto records)
- `lexicon`: integer, indicates the overall version of the Lexicon (currently `1`)
- `id`: the NSID of this Lexicon. Must be a simple NSID (no fragment), and must match the record key
- `defs`: the schema definitions themselves, as a map-of-objects. Names should not include a `#` prefix.
- `description`: optional description of the overall schema; though descriptions are best included on individual defs, not the overall schema.
The `com.atproto.lexicon.schema` meta-schema is somewhat unlike other Lexicons, in that it is defined and governed as part of the protocol. Future versions of the language and protocol might not follow the evolution rules. It is an intentional decision to not express the Lexicon schema language itself recursively, using the schema language.
Authority for NSID namespaces is done at the "group" level, meaning that all NSIDs which differ only by the final "name" part are all published in the same repository. Lexicon resolution of NSIDs is not hierarchical: DNS TXT records must be created for each authority section, and resolvers should not recurse up or down the DNS hierarchy looking for TXT records.
As an example, the NSID `edu.university.dept.lab.blogging.getBlogPost` has a "name" `getBlogPost`. Removing the name and reversing the rest of the NSID gives an "authority domain name" of `blogging.lab.dept.university.edu`. To link the authority to a specific DID (say `did:plc:ewvi7nxzyoun6zhxrhs64oiz`), a DNS TXT record with the name `_lexicon.blogging.lab.dept.university.edu` and value `did=did:plc:ewvi7nxzyoun6zhxrhs64oiz` (note the `did=` prefix) would be created. Then a record with collection `com.atproto.lexicon.schema` and record-key `edu.university.dept.lab.blogging.getBlogPost` would be created in that account's repository.
A resolving service would start with the NSID (`edu.university.dept.lab.blogging.getBlogPost`) and do a DNS TXT resolution for `_lexicon.blogging.lab.dept.university.edu`. Finding the DID, it would proceed with atproto DID resolution, look for a PDS, and then fetch the relevant record. The overall AT-URI for the record would be `at://did:plc:ewvi7nxzyoun6zhxrhs64oiz/com.atproto.lexicon.schema/edu.university.dept.lab.blogging.getBlogPost`.
If the DNS TXT resolution for `_lexicon.blogging.lab.dept.university.edu` failed, the resolving service would *NOT* try `_lexicon.lab.dept.university.edu` or `_lexicon.getBlogPost.blogging.lab.dept.university.edu` or `_lexicon.university.edu`, or any other domain name. The Lexicon resolution would simply fail.
If another NSID `edu.university.dept.lab.blogging.getBlogComments` was created, it would have the same authority name, and must be published in the same atproto repository (with a different record key). If a Lexicon for `edu.university.dept.lab.gallery.photo` was published, a new DNS TXT record would be required (`_lexicon.gallery.lab.dept.university.edu`; it could point at the same repository (DID), or a different repository.
As a simpler example, an NSID `app.toy.record` would resolve via `_lexicon.toy.app`.
A single repository can host Lexicons for multiple authority domains, possibly across multiple registered domains and TLDs. Resolution DNS records can change over time, moving schema resolution to different repositories, though it may take time for DNS and cache changes to propagate.
Note that Lexicon record operations are broadcast over repository event streams ("firehose"), but that DNS resolution changes do not (unlike handle changes). Resolving services should not cache DNS resolution results for long time periods.
## Usage and Implementation Guidelines
It should be possible to translate Lexicon schemas to JSON Schema or OpenAPI and use tools and libraries from those ecosystems to work with atproto data in JSON format.
Implementations which serialize and deserialize data from JSON or CBOR into structures derived from specific Lexicons should be aware of the risk of "clobbering" unexpected fields. For example, if a Lexicon is updated to add a new (optional) field, old implementations would not be aware of that field, and might accidentally strip the data when de-serializing and then re-serializing. Depending on the context, one way to avoid this problem is to retain any "extra" fields, or to pass-through the original data object instead of re-serializing it.
## Possible Future Changes
The validation rules for unexpected additional fields may change. For example, a mechanism for Lexicons to indicate that the schema is "closed" and unexpected fields are not allowed, or a convention around field name prefixes (`x-`) to indicate unofficial extension.
--------------------------------------------------------------------------------
# specs > nsid
# Namespaced Identifiers (NSIDs)
Namespaced Identifiers (NSIDs) are used to reference Lexicon schemas for records, XRPC endpoints, and more. {{ className: 'lead' }}
The basic structure and semantics of an NSID are a fully-qualified hostname in Reverse Domain-Name Order, followed by a simple name. The hostname part is the **domain authority,** and the final segment is the **name**. {{ className: 'lead' }}
### NSID Syntax
Lexicon string type: `nsid`
The domain authority part of an NSID must be a valid handle with the order of segments reversed. That is followed by a name segment which must be an ASCII camel-case string.
For example, `com.example.fooBar` is a syntactically valid NSID, where `com.example` is the domain authority, and `fooBar` is the name segment.
The comprehensive list of syntax rules is:
- Overall NSID:
- must contain only ASCII characters
- separate the domain authority and the name by an ASCII period character (`.`)
- must have at least 3 segments
- can have a maximum total length of 317 characters
- Domain authority:
- made of segments separated by periods (`.`)
- at most 253 characters (including periods), and must contain at least two segments
- each segment must have at least 1 and at most 63 characters (not including any periods)
- the allowed characters are ASCII letters (`a-z`), digits (`0-9`), and hyphens (`-`)
- segments can not start or end with a hyphen
- the first segment (the top-level domain) can not start with a numeric digit
- the domain authority is not case-sensitive, and should be normalized to lowercase (that is, normalize ASCII `A-Z` to `a-z`)
- Name:
- must have at least 1 and at most 63 characters
- the allowed characters are ASCII letters and digits only (`A-Z`, `a-z`, `0-9`)
- hyphens are not allowed
- the first character can not be a digit
- case-sensitive and should not be normalized
A reference regex for NSID is:
```
/^[a-zA-Z]([a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(\.[a-zA-Z0-9]([a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)+(\.[a-zA-Z]([a-zA-Z0-9]{0,62})?)$/
```
### NSID Syntax Variations
A **fragment** may be appended to an NSID in some contexts to refer to a specific sub-field within the schema. The fragment is separated from the NSID by an ASCII hash character (`#`). The fragment identifier string (after the `#`) has the same syntax restrictions as the final segment of an NSID: ASCII alphabetic, one or more characters, length restricted, etc.
When referring to a group or pattern of NSIDs, a trailing ASCII star character (`*`) can be used as a "glob" character. For example, `com.atproto.*` would refer to any NSIDs under the `atproto.com` domain authority, including nested sub-domains (sub-authorities). A free-standing `*` would match all NSIDs from all authorities. Currently, there may be only a single start character; it must be the last character; and it must be at a segment boundary (no partial matching of segment names). This means the start character must be proceeded by a period, or be a bare star matching all NSIDs.
### Examples
Syntactically valid NSIDs:
```
com.example.fooBar
net.users.bob.ping
a-0.b-1.c
a.b.c
com.example.fooBarV2
cn.8.lex.stuff
```
Invalid NSIDs:
```
com.exa💩ple.thing
com.example
com.example.3
```
### Usage and Implementation Guidelines
A **strongly-encouraged** best practice is to use authority domains with only ASCII alphabetic characters (that is, no digits or hyphens). This makes it significantly easier to generate client libraries in most programming languages.
The overall NSID is case-sensitive for display, storage, and validation. However, having multiple NSIDs that differ only by casing is not allowed. Namespace authorities are responsible for preventing duplication and confusion. Implementations should not force-lowercase NSIDs.
It is common to use "subdomains" as part of the "domain authority" to organize related NSIDs. For example, the NSID `com.atproto.sync.getHead` uses the `sync` segment. Note that this requires control of the full domain `sync.atproto.com`, in addition to the domain `atproto.com`.
Lexicon language documentation will provide style guidelines on choosing and organizing NSIDs for both record types and XRPC methods. In short, records are usually single nouns, not pluralized. XRPC methods are usually in "verbNoun" form.
### Possible Future Changes
It is conceivable that NSID syntax would be relaxed to allow Unicode characters in the final segment.
The "glob" syntax variation may be modified to extended to make the distinction between single-level and nested matching more explicit.
The "fragment" syntax variation may be relaxed in the future to allow nested references.
No automated mechanism for verifying control of a "domain authority" currently exists. Also, not automated mechanism exists for fetching a lexicon schema for a given NSID, or for enumerating all NSIDs for a base domain.
--------------------------------------------------------------------------------
# specs > oauth
# OAuth
<Note>
The OAuth profile for atproto is new and may be revised based on feedback from the development community and ongoing standards work. Read more about the rollout in the [OAuth Roadmap](https://github.com/bluesky-social/atproto/discussions/2656).
</Note>
<Note>
This specification is authoritative, but is not an implementation guide and does not provide much background or context around design decisions. The earlier [design proposal](https://github.com/bluesky-social/proposals/tree/main/0004-oauth) is not authoritative but provides more context and examples. SDK documentation and the [client implementation guide](https://docs.bsky.app/docs/advanced-guides/oauth-client) are more approachable for developers.
</Note>
OAuth is the primary mechanism in atproto for clients to make authorized requests to PDS instances. Most user-facing software is expected to use OAuth, including "front-end" clients like mobile apps, rich browser apps, or native desktop apps, as well as "back-end" clients like web services.
See the [HTTP API specification](./xrpc) for other forms of auth in atproto, including legacy HTTP client sessions/tokens, and inter-service auth.
OAuth is a constantly evolving framework of standards and best practices, standardized by the IETF. atproto uses a specific "profile" of OAuth which mandates a particular combination of OAuth standards, as described in this document.
At a high level, we start with the "OAuth 2.1" ([`draft-ietf-oauth-v2-1`](https://datatracker.ietf.org/doc/draft-ietf-oauth-v2-1/)) evolution of OAuth 2.0, which means:
- only the "authorization code" OAuth 2.0 grant type is supported, not "implicit" or other grant types
- mandatory Proof Key for Code Exchange (PKCE, [RFC 7636](https://datatracker.ietf.org/doc/html/rfc7636))
- security best practices ([`draft-ietf-oauth-security-topics`](https://datatracker.ietf.org/doc/html/draft-ietf-oauth-security-topics) and [`draft-ietf-oauth-browser-based-apps`](https://datatracker.ietf.org/doc/html/draft-ietf-oauth-browser-based-apps)) are required
Unlike a centralized app platform, in atproto there are many independent server implementations, so server discovery and client registration are automated using a combination of public auth server metadata and public client metadata. The `client_id` is a fully-qualified web URL pointing to the public client metadata (JSON document). There is no `client_secret` shared between servers and clients. When initiating a login with a handle or DID, an atproto-specific identity resolution step is required to discover the account’s PDS network location.
In OAuth terminology, an atproto Personal Data Server (PDS) is a "Resource Server" to which authorized HTTP requests are made using access tokens. Sometimes the PDS is also the "Authorization Server" - which services OAuth authorization flows and token requests - while in other situations a separate "entryway" service acts as the Authorization Server for multiple PDS instances. Clients from a metadata file from the PDS to discover the Authorization Server network location.
DPoP (with mandatory server issued nonces) is required to bind auth tokens to specific client software instances (eg, end devices or browser sessions). Pushed Authentication Requests (PAR) are used to streamline the authorization request flow. "Confidential" clients use JWTs signed with a secret key to authenticate the client software to Authorization Servers when making authorization requests.
Automated client registration using client metadata is one of the more novel aspects of OAuth in atproto. As of August 2024, client metadata is still an Internet Draft ([`draft-parecki-oauth-client-id-metadata-document`](https://datatracker.ietf.org/doc/draft-parecki-oauth-client-id-metadata-document/)); it should not be confused with the existing "Dynamic Client Registration" standard ([RFC 7591](https://datatracker.ietf.org/doc/html/rfc7591)). We are hopeful other open protocols will adopt similar automated registration flows in the future, but there may not be general OAuth ecosystem support for some time.
OAuth 2.0 is traditionally an authorization (`authz`) system, not an authentication (`authn`) system, meaning that it is not always a solution for pure account authentication use cases, such as "Signup/Login with XYZ" identity integrations. OpenID Connect (OIDC), which builds on top of OAuth 2.0, is usually the recommended standard for identity authentication. Unfortunately, the current version of OIDC does not enable authentication of atproto identities in a secure and generic way. The atproto profile of OAuth includes a (mandatory) mechanism for account authentication during the authorization flow and can be used for atproto identity authentication use cases.
## Clients
This section describes requirements for OAuth clients, which are enforced by Authorization Servers.
OAuth client software is identified by a globally unique `client_id`. Distinct variants of client software may have distinct `client_id` values; for example the browser app and Android (mobile OS) variants of the same software might have different `client_id` values. As required by the [`draft-parecki-oauth-client-id-metadata-document`](https://datatracker.ietf.org/doc/draft-parecki-oauth-client-id-metadata-document) specification draft, the `client_id` must be a fully-qualified web URL from which the client-metadata JSON document can be fetched. For example, `https://app.example.com/client-metadata.json`. Some more about the `client_id`:
- it must be a well-formed URL, following the W3C URL specification
- the schema must be `https://`, and there must not be a port number included. Note that there is a special exception for `http://localhost` `client_id` values for development, see details below
- the path does not need to include `client-metadata.json`, but it is helpful convention
Authorization Servers which support both the atproto OAuth profile and other forms of OAuth should take care to prevent `client_id` value collisions. For example, `client_id` values for clients which are not auto-registered should never have the prefix `https://` or `http://`.
### Types of Clients
All atproto OAuth clients need to meet a core set of standards and requirements, but there are a few variations in capabilities (such as session lifetime) depending on the security properties of the client itself.
As described in the OAuth 2.0 specification ([RFC 6749](https://datatracker.ietf.org/doc/html/rfc6749)), every client is one of two broad types:
- **confidential clients** are clients which can authenticate themselves to Authorization Servers using a cryptographic signing key. This allows refresh tokens to be bound to the specific client. Note that this form of client authentication is distinct from DPoP: the client authentication key is common to all client sessions (although it can be rotated). This usually means that there is a web service controlled by the client which holds the key. Because they are authenticated and can revoke tokens in a security incident, confidential clients may be trusted with longer session and token lifetimes.
- **public clients** do not authenticate using a client signing key, either because they don’t have a server-side component (the client software all runs on end-user devices), or they simply chose not to implement it.
It is acceptable for a web service to act as a public client, and conversely it is possible for mobile apps and browser apps to coordinate with a token-mediating backend service and for the combination to form a confidential client. Mobile apps and browser apps can also adopt a "backend-for-frontend" (BFF) architecture with a web service backend acting as the OAuth client. This document will use the "public" vs "confidential" client terminology for clarity.
The environment a client runs in also impacts the type of redirect (callback) URLs it uses during the Authorization Flow:
- **web clients** include web services and browser apps. Redirect URLs are regular web URLs which open in a browser.
- **native clients** include some mobile and desktop native clients. Redirect URLs may use platform-specific app callback schemes to open in the app itself.
Authorization Servers may maintain a set of "trusted" clients, identified by `client_id`. Because any client could use unverified client metadata to impersonate a better-known app or brand, Authorization Servers should not display such metadata to end users in the Authorization Interface by default. Trusted clients can have additional metadata shown, such as a readable name (`client_name`), project URI (`client_uri`, which may have a different domain/origin than `client_id`) and logo (`logo_uri`). See the "Security Considerations" section for more details.
Clients which are only using atproto OAuth for account authentication (without authorization to access PDS resources) should request minimal scopes (see "Scopes" section), but still need to implement most of the authorization flow. In particular, it is critical that they check the `sub` field in a token response to verify the account identity (this is an atproto-specific detail).
### Client ID Metadata Document
<Note>
The Client ID Metadata Document specification ([`draft-parecki-oauth-client-id-metadata-document`](https://datatracker.ietf.org/doc/draft-parecki-oauth-client-id-metadata-document/) is still a draft and may evolve over time. Our intention is to evolve and align with subsequent drafts and any final standard, while minimizing disruption and breakage with existing implementations.
</Note>
Clients must publish a "client metadata" JSON file on the public web. This will be fetched dynamically by Authorization Servers as part of the authorization request (PAR) and at other times during the session lifecycle. The response HTTP status must be 200 (not another 2xx or a redirect), with a JSON object body with the correct `Content-Type` (`application/json`).
Authorization Servers need to fetch client metadata documents from the public web. They should use a hardened HTTP client for these requests (see "OAuth Security Considerations"). Servers may cache client metadata responses, optionally respecting HTTP caching headers (within limits). Minimum and maximum cache TTLs are not currently specified, but should be chosen to ensure that auth token requests predicated on stale confidential client authentication keys (`jwks` or `jwks_uris`) are rejected in a timely manner.
The following fields are relevant for all client types:
- `client_id` (string, required): the `client_id`. Must exactly match the full URL used to fetch the client metadata file itself
- `application_type` (string, optional): must be one of `web` or `native`, with `web` as the default if not specified. Note that this is field specified by OpenID/OIDC, which we are borrowing. Used by the Authorization Server to enforce the relevant "best current practices".
- `grant_types` (array of strings, required): `authorization_code` must always be included. `refresh_token` is optional, but must be included if the client will make token refresh requests.
- `scope` (string, sub-strings space-separated, required): all scope values which *might* be requested by this client are declared here. The `atproto` scope is required, so must be included here. See "Scopes" section.
- `response_types` (array of strings, required): `code` must be included.
- `redirect_uris` (array of strings, required): at least one redirect URI is required. See Authorization Request Fields section for rules about redirect URIs, which also apply here.
- `token_endpoint_auth_method` (string, optional): confidential clients must set this to `private_key_jwt`.
- `token_endpoint_auth_signing_alg` (string, optional): `none` is never allowed here. The current recommended and most-supported algorithm is `ES256`, but this may evolve over time. Authorization Servers will compare this against their supported algorithms.
- `dpop_bound_access_tokens` (boolean, required): DPoP is mandatory for all clients, so this must be present and `true`
- `jwks` (object with array of JWKs, optional): confidential clients must supply at least one public key in JWK format for use with JWT client authentication. Either this field or the `jwks_uri` field must be provided for confidential clients, but not both.
- `jwks_uri` (string, optional): URL pointing to a JWKS JSON object. See `jwks` above for details.
These fields are optional but recommended:
- `client_name` (string, optional): human-readable name of the client
- `client_uri` (string, optional): not to be confused with `client_id`, this is a homepage URL for the client. If provided, the `client_uri` must have the same hostname as `client_id`.
- `logo_uri` (string, optional): URL to client logo. Only `https:` URIs are allowed.
- `tos_uri` (string, optional): URL to human-readable terms of service (ToS) for the client. Only `https:` URIs are allowed.
- `policy_uri` (string, optional): URL to human-readable privacy policy for the client. Only `https:` URIs are allowed.
See "OAuth Security Considerations" below for when `client_name`, `client_uri`, and `logo_uri` will or will not be displayed to end users.
Additional optional client metadata fields are enumerated with [IANA](https://www.iana.org/assignments/oauth-parameters/oauth-parameters.xhtml#client-metadata). Note that these are shared with the "Dynamic Client Registration" standard, which is not used directly by the atproto OAuth profile.
### Localhost Client Development
When working with a developent environment (Authorization Server and Client), it may be difficult for developers to publish in-progress client metadata at a public URL so that authorization servers can access it. This may even be true for development environments using a containerized Authorization Server and local DNS, because of SSRF protections against local IP ranges.
To make development workflows easier, a special exception is made for clients with `client_id` having origin `http://localhost` (with no port number specified). Authorization Servers are encouraged to support this exception - including in production environments - but it is optional.
In a localhost `client_id` scenario, the Authorization Server should verify that the scheme is `http`, and that the hostname is exactly `localhost` with no port specified. IP addresses (`127.0.0.1`, etc) are not supported. The path parameter must be empty (`/`).
In the Authorization Request, the `redirect_uri` must match one of those supplied (or a default). Path components must match, but port numbers are not matched.
Some metadata fields can be configured via query parameter in the `client_id` URL (with appropriate urlencoding):
- `redirect_uri` (string, multiple query parameters allowed, optional): allows declaring a local redirect/callback URL, with path component matched but port numbers ignored. The default values (if none are supplied) are `http://127.0.0.1/` and `http://[::1]/`.
- `scope` (string with space-separated values, single query parameter allowed, optional): the set of scopes which might be requested by the client. Default is `atproto`.
The other parameters in the virtual client metadata document will be:
- `client_id` (string): the exact `client_id` (URL) used to generate the virtual document
- `client_name` (string): a value chosen by the Authorization Server (e.g. "Development client")
- `response_types` (array of strings): must include `code`
- `grant_types` (array of strings): `authorization_code` and `refresh_token`
- `token_endpoint_auth_method`: `none`
- `application_type`: `native`
- `dpop_bound_access_tokens`: `true`
Note that this works as a public client, not a confidential client.
## Identity Authentication
As mentioned in the introduction, OAuth 2.0 generally provides only Authorization (`authz`), and additional standards like OpenID/OIDC are used for Authentication (`authn`). The atproto profile of OAuth requires authentication of account identity and supports the use case of simple identity authentication without additional resource access authorization.
In atproto, account identity is anchored in the account DID, which is the permanent, globally unique, publicly resolvable identifier for the account. The DID resolves to a DID document which indicates the current PDS host location for the account. That PDS (combined with an optional entryway) is the authorization authority and the OAuth Authorization Server for the account. When speaking to any Authorization Server, it is critical (mandatory) for clients to confirm that it is actually the authoritative server for the account in question, which means independently resolving the account identity (by DID) and confirming that the Authorization Server matches. It is also critical (mandatory) to confirm at the end of an authorization flow that the Authorization Server actually authorized the expected account. The reason this is necessary is to confirm that the Authorization Server is authoritative for the account in question. Otherwise a malicious server could authenticate arbitrary accounts (DIDs) to the client.
Clients can start an auth flow in one of two ways:
- starting with a public account identifier, provided by the user: handle or DID
- starting with a server hostname, provided by the user: PDS or entryway, mapping to either Resource Server and/or Authorization Server
One use case for starting with a server instead of an account identifier is when the user does not remember their full account handle or only knows their account email. Another is for authentication when a user’s handle is broken. The user will still need to know their hosting provider in these situation.
When starting with an account identifier, the client must resolve the atproto identity to a DID document. If starting with a handle, it is critical (mandatory) to bidirectionally verify the handle by checking that the DID document claims the handle (see atproto Handle specification). All handle resolution techniques and all atproto-blessed DID methods must be supported to ensure interoperability with all accounts.
In some client environments, it may be difficult to resolve all identity types. For example, handle resolution may involve DNS TXT queries, which are not directly supported from browser apps. Client implementations might use alternative techniques (such as DNS-over-HTTP) or could make use of a supporting web service to resolve identities.
Because authorization flows are security-critical, any caching of identity resolution should choose cache lifetimes carefully. Cache lifetimes of less than 10 minutes are recommended for auth flows specifically.
The resolved DID should be bound to the overall auth session and should be used as the primary account identifier within client app code. Handles (when verified) are acceptable to display in user interfaces, but may change over time and need to be re-verified periodically. When passing an account identifier through to the Authorization Server as part of the Authorization Request in the `login_hint`, it is recommended to use the exact account identifier supplied by the user (handle or DID) to ensure any sign-in flow is consistent (users might not recognize their own account DID).
At the end of the auth flow, when the client does an initial token fetch, the Authorization Server must return the account DID in the `sub` field of the JSON response body. If the entire auth flow started with an account identifier, it is critical for the client to verify that this DID matches the expected DID bound to the session earlier; the linkage from account to Authorization Server will already have been verified in this situation.
If the auth flow instead starts with a server (hostname or URL), the client will first attempt to fetch Resource Server metadata (and resolve to Authorization Server if found) and then attempt to fetch Authorization Server metadata. See "Authorization Server" section for server metadata fetching. If either is successful, the client will end up with an identified Authorization Server. The Authorization Request flow will proceed without a `login_hint` or account identifier being bound to the session, but the Authorization Server `issuer` will be bound to the session.
After the auth flow continues and an initial token request succeeds, the client will parse the account identifier from the `sub` field in the token response. At this point, the client still cannot trust that it has actually authenticated the indicated account. It is critical for the client to resolve the identity (DID document), extract the declared PDS host, confirm that the PDS (Resource Server) resolves to the Authorization Server bound to the session by fetching the Resource Server metadata, and fetch the Authorization Server metadata to confirm that the `issuer` field matches the Authorization Server origin (see [`draft-ietf-oauth-v2-1` section 7.3.1](https://datatracker.ietf.org/doc/html/draft-ietf-oauth-v2-1-11#section-7.13.1) regarding this last point).
To reiterate, it is critical for all clients - including those only interested in atproto Identity Authentication - to go through the entire Authorization flow and to verify that the account identifier (DID) in the `sub` field of the token response is consistent with the Authorization Server hostname/origin (`issuer`).
## Authorization Scopes
OAuth scopes allow more granular control over the resources and actions a client is granted access to.
The special `atproto` scope is required for all atproto OAuth sessions. The semantics are somewhat similar to the `openid` scope: inclusion of it confirms that the client is using the atproto profile of OAuth and will comply with all the requirements laid out in this specification. No access to any atproto-specific PDS resources will be granted without this scope included.
Authorization Servers may support other profiles of OAuth if client does not include the `atproto` scope. For example, an Authorization Server might function as both an atproto PDS/entryway, and support other protocols/standards at the same time.
Use of the atproto OAuth profile, as indicated by the `atproto` scope, means that the Authorization Server will return the atproto account DID as an account identifier in the `sub` field of token requests. Authorization Servers must return `atproto` in `scopes_supported` in their metadata document, so that clients know they support the atproto OAuth profile. A client may include only the `atproto` scope if they only need account authentication - for example a "Login with atproto" use case. Unlike OpenID, profile metadata in atproto is generally public, so an additional authorization scope for fetching profile metadata is not needed.
The OAuth 2.0 specification does not require Authorization Servers to return the granted scopes in the token responses unless the scope that was granted is different from what the client requested. In the atproto OAuth profile, servers must always return the granted scopes in the token response. Clients should reject token responses if they don't contain a `scope` field, or if the `scope` field does not contain `atproto`.
The intention is to support flexible scopes based on Lexicon namespaces (NSIDs) so that clients can be given access only to the specific content and API endpoints they need access to. Until the design of that scope system is ready, the atproto profile of OAuth defines two transitional scopes which align with the permissions granted under the original "session token" auth system:
- `transition:generic`: broad PDS account permissions, equivalent to the previous "App Password" authorization level.
- write (create/update/delete) any repository record type
- upload blobs (media files)
- read and write any personal preferences
- API endpoints and service proxying for most Lexicon endpoints, to any service provider (identified by DID)
- ability to generate service auth tokens for the specific API endpoints the client has access to
- no account management actions: change handle, change email, delete or deactivate account, migrate account
- no access to DMs (the `chat.bsky.*` Lexicons), specifically
- `transition:chat.bsky`: equivalent to adding the "DM Access" toggle for "App Passwords"
- API endpoints and service proxying for the `chat.bsky` Lexicons specifically
- ability to generate service auth tokens for the `chat.bsky` Lexicons
- this scope depends on and does not function without the `transition:generic` scope
## Authorization Requests
This section details standards and requirements specific to Authorization Requests.
PKCE and PAR are required for all client types and Authorization Servers. Confidential clients authenticate themselves using JWT client assertions.
### Request Fields
A summary of fields relevant to authorization requests with the atproto OAuth profile:
- `client_id` (string, required): identifies the client software. See "Clients" section above for details.
- `response_type` (string, required): must be `code`
- `code_challenge` (string, required): the PKCE challenge value. See "PKCE" section.
- `code_challenge_method` (string, required): which code challenge method is used, for example `S256`. See "PKCE" section.
- `state` (string, required): random token used to verify the authorization request against the response. See below.
- `redirect_uri` (string, required): must match against URIs declared in client metadata and have a format consistent with the `application_type` declared in the client metadata. See below.
- `scope` (string with space-separated values, required): must be a subset of the scopes declared in client metadata. Must include `atproto`. See "Scopes" section.
- `client_assertion_type` (string, optional): used by confidential clients to describe the client authentication mechanism. See "Confidential Client" section.
- `client_assertion` (string, optional): only used for confidential clients, for client authentication. See "Confidential Client" section.
- `login_hint` (string, optional): account identifier to be used for login. See "Authorization Interface" section.
The `client_secret` value, used in many other OAuth profiles, should not be included.
The `state` parameter in client authorization requests is mandatory. Clients should use randomly-generated tokens for this parameter and not have collisions or reuse tokens across any combination of device, account, or session. Authorization Servers should reject duplicate state parameters, but are not currently required to track state values across accounts or sessions. The `state` parameter is effectively used to verify the `issuer` later, and it is
For web clients, the `redirect_uri` is a HTTPS URL which will be redirected in the browser to return users to the application at the end of the Authorization flow. The URL may include a port number, but not if it is the default port number. The `redirect_uri` must match one of the URIs declared in the client metadata and the Authorization Server must verify this condition. The URL origin must match that of the `client_id`.
There is a special exception for the localhost development workflow to use `http://127.0.0.1` or `http://[::1]` URLs, with matching rules described in the "Localhost Client Development" section. These clients use web URLs, but have `application_type` set to `native` in the generated client metadata.
For native clients, the `redirect_uri` may use a custom URI scheme to have the operating system redirect the user back to the app, instead of a web browser. Native clients are also allowed to use an HTTPS URL. Any custom scheme must match the `client_id` hostname in reverse-domain order. The URI scheme must be followed by a single colon (`:`) then a single forward slash (`/`) and then a URI path component. For example, an app with `client_id` [`https://app.example.com/client-metadata.json`](https://app.example.com/client-metadata.json) could have a `redirect_uri` of `com.example.app:/callback`.
Native clients are also allowed to use an HTTPS URL. In this case, the URL origin must be the same as the `client_id`. One example use-case is "Apple Universal Links".
Clients may include additional optional authorization request parameters - and servers may process them - but they are not required to. Refer to other OAuth standards and the [IANA OAuth parameter registry](https://www.iana.org/assignments/oauth-parameters/oauth-parameters.xhtml).
### Proof Key for Code Exchange (PKCE)
PKCE is mandatory for all Authorization Requests. Clients must generate new, unique, random challenges for every authorization request. Authorization Servers must prevent reuse of `code_challenge` values across sessions (at least within some reasonable time frame, such as a 24 hour period).
The `S256` challenge method must be supported by all clients and Authorization Servers; see [RFC 7636](https://datatracker.ietf.org/doc/html/rfc7636) for details. The `plain` method is not allowed. Additional methods may in theory be supported if both client and server support them.
Authorization Servers should reject reuse of a `code` value, and revoke any outstanding sessions and tokens associated with the earlier use of the `code` value.
### Pushed Authorization Requests (PAR)
Authorization Servers must support PAR and clients of all types must use PAR for Authorization Requests.
Authorization Servers must set `require_pushed_authorization_requests` to `true` in their server metadata document and include a valid URL in `pushed_authorization_request_endpoint`. See [RFC 9207](https://datatracker.ietf.org/doc/html/rfc9207) for requirements on this URL.
Clients make an HTTPS POST request to the `pushed_authorization_request_endpoint` URL, with the request parameters in the form-encoded request body. They receive a `request_uri` (not to be confused with `redirect_uri`) in the JSON response object. When they redirect the user to the authorization endpoint (`authorization_endpoint`), they omit most of the request parameters they already sent and include this `redirect_uri` along with `client_id` as query parameters instead.
<Note>
PAR is a relatively new and less-supported standard, and the requirement to use PAR may be relaxed if it is found to be too onerous a requirement for client implementations. In that case, Authorization Servers would be required to support both PAR and non-PAR requests with PAR being optional for clients.
</Note>
### Confidential Client Authentication
Confidential clients authenticate themselves during the Authorization Request using a JWT client assertion. Authorization Servers may grant confidential clients longer token/session lifetimes. See "Tokens" section for more context.
The client assertion type to use is `urn:ietf:params:oauth:client-assertion-type:jwt-bearer`, as described in "JSON Web Token (JWT) Profile for OAuth 2.0 Client Authentication and Authorization Grants" ([RFC 7523](https://datatracker.ietf.org/doc/html/rfc7523)). Clients and Authorization Servers currently must support the `ES256` cryptographic system. The set of recommended systems/algorithms is expected to evolve over time.
Additional requirements:
- confidential clients must publish one or more client authentication keys (public key) in the client metadata. This can be either direct JWK format as JSON in the `jwks` field, or as a separate JWKS JSON object on the web linked by a `jwks_uri` URL. A `jwks_uri` URL must be a valid fully qualified URL with `https://` scheme.
- confidential clients should periodically rotate client keys, adding new keys to the JWKS set and using then for new sessions, then removing old keys once they are no longer associated with any active auth sessions
- confidential clients must include `token_endpoint_auth_method` as `private_key_jwt` in their client metadata document
- confidential clients are expected to immediately remove client authentication keys from their client metadata if the key has been leaked or compromised
- Authorization Servers must bind active auth sessions for confidential clients to the client authentication key used at the start of the session. The server should revoke the session and reject further token refreshes if the client authentication key becomes absent from the client metadata. This means the Authorization Server is expected to periodically re-fetch client metadata.
## Tokens and Session Lifetime
Access tokens are used to authorize client requests to the account's PDS ("Resource Server"). From the standpoint of the client they are opaque, but they are often signed JWTs including an expiration time. Depending on the PDS implementation, it may or may not be possible to revoke individual access tokens in the event of a compromise, so they must be restricted to a relatively short lifetime.
Refresh tokens are used to request new tokens (of both types) from the Authorization Server (PDS or entryway). They are also opaque from the standpoint of clients. Auth sessions can be revoked - invalidating the refresh tokens - so they may have a longer lifetime. In the atproto OAuth profile, refresh tokens are generally single-use, with the "new" refresh token replacing that used in the token request. This means client implementations may need locking primitives to prevent concurrent token refresh requests.
To request refresh tokens, the client must declare `refresh_token` as a grant type in their client metadata.
Tokens are always bound to a unique session DPoP key. Tokens must not be shared or reused across client devices. They must also be uniquely bound to the client software (`client_id`). The overall session ends when the access and refresh tokens can no longer be used.
The specific lifetime of sessions, access tokens, and refresh tokens is up to the Authorization Server implementation and may depend on security assessments of client type and reputation.
Some guidelines and requirements:
- access token lifetimes should be less than 30 minutes in all situations. If the server cannot revoke individual access tokens then the maximum is 15 minutes, and 5 minutes is recommended.
- for "untrusted" public clients, overall session lifetime should be limited to 7 days, and the lifetime of individual refresh tokens should be limited to 24 hours
- for confidential clients, the overall session lifetime may be unlimited. Individual refresh tokens should have a lifetime limited to 180 days
- confidential clients must use the same client authentication key and assertion method for refresh token requests that they did for the initial authentication request
## Demonstrating Proof of Possession (DPoP)
The atproto OAuth profile mandates use of DPoP for all client types when making auth token requests to the Authorization Server and when making authorized requests to the Resource Server. See [RFC 9449](https://datatracker.ietf.org/doc/html/rfc9449) for details.
Clients must initiate DPoP in the initial authorization request (PAR).
Server-provided DPoP nonces are mandatory. The Resource Server and Authorization Server may share nonces (especially if they are the same server) or they may have separate nonces. Clients should track the DPoP nonce per account session and per server. Servers must rotate nonces periodically, with a maximum lifetime of 5 minutes. Servers may use the same nonce across all client sessions and across multiple requests at any point in time. Servers should accept recently-stale (old) nonces to make rotation smoother for clients with multiple concurrent request in-flight. Clients should be resilient to unexpected nonce updates in the form of HTTP 400 errors and should retry those failed requests. Clients must reject responses missing a `DPoP-Nonce` header (case insensitive), if the request included DPoP.
Clients must generate and sign a unique DPoP token (JWT) for every request. Each DPoP request JWT must have a unique (randomly generated) `jti` nonce. Servers should prevent token replays by tracking `jti` nonces and rejecting re-use. They can restrict their client-generated `jti` nonce history to the server-generated DPoP nonce so that they do not need to track an endlessly growing set of nonces.
The `ES256` (NIST "P-256") cryptographic algorithm must be supported by all clients and servers for DPoP JWT signing. The set of algorithms recommended for use is expected to evolve over time. Clients and Servers may implement additional algorithms and declare them in metadata documents to facilitate cryptographic evolution and negotiation.
## Authorization Servers
To enable browser apps, Authorization Servers must support HTTP CORS requests/headers on relevant endpoints, including server metadata, auth requests (PAR), and token requests.
### Server Metadata
Both Resource Servers (PDS instances) and Authorization Servers (PDS or entryway) need to publish metadata files at well-known HTTPS endpoints.
Resource Server (PDS) metadata must comply with the "OAuth 2.0 Protected Resource Metadata" ([`draft-ietf-oauth-resource-metadata`](https://datatracker.ietf.org/doc/draft-ietf-oauth-resource-metadata/)) draft specification. A summary of requirements:
- the URL path is `/.well-known/oauth-protected-resource`
- response must be an HTTP 200 (not 2xx or redirect), and must be a valid JSON object with content type `application/json`
- must contain an `authorization_servers` array of strings, with a single element, which is a fully-qualified URL
The Authorization Server URL may be the same origin as the Resource Server (PDS), or might point to a separate server (e.g. entryway). The URL must be a simple origin URL: `https` scheme, no credentials (user:password), no path, no query string or fragment. A port number is allowed, but a default port (443 for HTTPS) must not be included.
The Authorization Server also publishes metadata, complying with the "OAuth 2.0 Authorization Server Metadata" ([RFC 8414](https://datatracker.ietf.org/doc/html/rfc8414)) standard. A summary of requirements:
- the URL path is `/.well-known/oauth-authorization-server`
- response must be an HTTP 200 (not 2xx or redirect), and must be a valid JSON object with content type `application/json`
- `issuer` (string, required): the "origin" URL of the Authorization Server. Must be a valid URL, with `https` scheme. A port number is allowed (if that matches the origin), but the default port (443 for HTTPS) must not be specified. There must be no path segments. Must match the origin of the URL used to fetch the metadata document itself.
- `authorization_endpoint` (string, required): endpoint URL for authorization redirects
- `token_endpoint` (string, required): endpoint URL for token requests
- `response_types_supported` (array of strings, required): must include `code`
- `grant_types_supported` (array of strings, required): must include `authorization_code` and `refresh_token` (refresh tokens must be supported)
- `code_challenge_methods_supported` (array of strings, required): must include `S256` (see "PKCE" section)
- `token_endpoint_auth_methods_supported` (array of strings, required): must include both `none` (public clients) and `private_key_jwt` (confidential clients)
- `token_endpoint_auth_signing_alg_values_supported` (array of strings, required): must not include `none`. Must include `ES256` for now. Cryptographic algorithm suites are expected to evolve over time.
- `scopes_supported` (array of strings, required): must include `atproto`. If supporting the transitional grants, they should be included here as well. See "Scopes" section.
- `authorization_response_iss_parameter_supported` (boolean): must be `true`
- `require_pushed_authorization_requests` (boolean): must be `true`. See "PAR" section.
- `pushed_authorization_request_endpoint` (string, required): must be the PAR endpoint URL. See "PAR" section.
- `dpop_signing_alg_values_supported` (array of strings, required): currently must include `ES256`. See "DPoP" section.
- `require_request_uri_registration` (boolean, optional): default is `true`; does not need to be set explicitly, but must not be `false`
- `client_id_metadata_document_supported` (boolean, required): must be `true`. See "Client ID Metadata" section.
The `issuer` ("origin") is the overall identifier for the Authorization Server.
### Authorization Interface
The Authorization Server (PDS/entryway) must implement a web interface for users to authenticate with the server, approve (or reject) Authorization Requests from clients, and manage active sessions. This is called the "Authorization Interface".
Server implementations can chose their own technology for user authentication and account recovery: secure cookies, email, various two-factor authentication, passkeys, external identity providers (including upstream OpenID/OIDC), etc. Servers may also support multiple concurrent auth sessions with users.
When a client redirects to the Authorization Server’s authorization URL (the declared `authorization_endpoint`), the server first needs to authenticate the user. If there is no active auth session, the user may be prompted to log in. If a `login_hint` was provided in the Authorization Request, that can be used to pre-populate the login form. If there are multiple active auth sessions, the user could be prompted to select one from a list, or the `login_hint` could be used to auto-select. If there is a single active session, the interface can move to the approval view, possibly with the option to login as a different account. If a `login_hint` was supplied, the Authorization Server should only allow the user to authenticate with that account. Otherwise the overall authorization flow will fail when the client verifies the account identity (`sub` field).
The authorization approval prompt should identify the client app and describe the scope of authorization that has been requested.
The amount of client metadata that should be displayed may depend on whether the client is "trusted" by the Authorization Server; see the "Client" and "Security Concerns" sections. The full `client_id` URL should be displayed by default.
See the "Scopes" section for a description of scope options.
If a client is a confidential client and the user has already approved the same scopes for the same client in the past, the Authorization Server may allow "silent sign-in" by auto-approving the request. Authorization Servers can set their own policies for this flow: it may require explicit user configuration, or the client may be required to be "trusted".
Authorization Servers should separately implement a web interface which allows authenticated users to view active OAuth sessions and delete them.
## Summary of Authorization Flow
This is a high-level description of what an atproto OAuth authorization flow looks like. It assumes the user already has an atproto account.
The client starts by asking for the user’s account identifier (handle or DID), or for a PDS/entryway hostname. See "Identity Authentication" section for details.
For an account identifier, the client resolves the identity to a DID document, extracts the declared PDS URL, then fetches the Resource Server and Authorization Server locations. If starting with a server hostname, the client resolves that hostname to an Authorization Server. In either case, Authorization Server metadata is fetched and verified against requirements for atproto OAuth (see "Authorization Server" section).
The client next makes a Pushed Authorization Request via HTTP POST request. See "Authorization Request" section; some notable details include:
- a randomly generated `state` token is required, and will be used to reference this authorization request with the subsequent response callback
- PKCE is required, so a secret value is generated and stored, and a derived challenge is included in the request
- `scope` values are requested here, and must include `atproto`
- for confidential clients, a `client_assertion` is included, with type `jwt-bearer`, signed using the secret client authentication key
- the client generates a new DPoP key for the user/device/session and uses it starting with the PAR request
- if the auth flow started with an account identifier, the client should pass that starting identifier via the `login_hint` field
- atproto uses PAR, so the request will be sent as an HTTP POST request to the Authorization Server
The Authorization Server will receive the PAR request and use the `client_id` URL to resolve the client metadata document. The server validates the request and client metadata, then stores information about the session, including binding a DPoP key to the session. The server returns a `request_uri` token to the client, including a DPoP nonce via HTTP header.
The client receives the `request_uri` and prepares to redirect the user. At this point, the client usually needs to persist information about the session to some type of secure storage, so it can be read back after the redirect returns. This might be a database (for a web service backend) or web platform storage like IndexedDB (for a browser app). The client then redirects the user via browser to the Authorization Server’s auth endpoint, including the `request_uri` as a URL parameter. In any case, clients must not store session data directly in the `state` request param.
The Authorization Server uses the `request_uri` to look up the earlier Authorization Request parameters, authenticates the user (which might include sign-in or account selection), and prompts the user with the Authorization Interface. The user might refine any granular requested scopes, then approves or rejects the request. The Authorization Server redirects the user back to the `redirect_uri`, which might be a web callback URL, or a native app URI (for native clients).
The client uses URL query parameters (`state` and `iss`) to look up and verify session information. Using the `code` query parameter, the client then makes an initial token request to the Authorization Server’s token endpoint. The client completes the PKCE flow by including the earlier value in the `code_verifier` field. Confidential clients need to include a client assertion JWT in the token request; see the "Confidential Client" section. The Authorization Server validates the request and returns a set of tokens, as well as a `sub` field indicating the account identifier (DID) for this session, and the `scope` that is covered by the issued access token.
At this point it is critical (mandatory) for all clients to verify that the account identified by the `sub` field is consistent with the Authorization Server "issuer" (present in the `iss` query string), either by validating against the originally-supplied account DID, or by resolving the accounts DID to confirm the PDS is consistent with the Authorization Server. See "Identity Authentication" section. The Authorization always returns the scopes approved for the session in the `scopes` field (even if they are the same as the request, as an atproto OAuth profile requirement), which may reflect partial authorization by the user. Clients must reject the session if the response does not include `atproto` in the returned scopes.
Authentication-only clients can end the flow here.
Using the access token, clients are now able to make authorized requests to the PDS ("Resource Server"). They must use DPoP for all such requests, along with the access token. Tokens (both access and refresh) will need to be periodically "refreshed" by subsequent request to the Authorization Server token endpoint. These also require DPoP. See "Tokens and Session Lifetime" section for details.
## Security Considerations
There are a number of situations where HTTP URLs provided by external parties are fetched by both clients and providers (servers). Care must be taken to prevent harmful fetches due to maliciously crafted URLs, including hostnames which resolve to private or internal IP addresses. The general term for this class of security issue is Server-Side Request Forgery (SSRF). There is also a class of denial-of-service attacks involving HTTP requests to malicious servers, such as huge response bodies, TCP-level slow-loris attacks, etc. We strongly recommend using "hardened" HTTP client implementations/configurations to mitigate these attacks.
Any party can create a client and client metadata file with any contents at any time. Even the hostname in the `client_id` cannot be entirely trusted to represent the overall client: an untrusted user may have been able to upload the client metadata file to an arbitrary URL on the host. In particular, the `client_uri`, `client_name`, and `logo_uri` fields are not verified and could be used by a malicious actor to impersonate a legitimate client. It is strongly recommended for Authorization Servers to not display these fields to end users during the auth flow for unknown clients. Service operators may maintain a list of "trusted" `client_id` values and display the extra metadata for those apps only.
## Possible Future Changes
Client metadata requests (by the authorization server) might fail for any number of reasons: transient network disruptions, the client server being down for regular maintenance, etc. It seems brittle for the Authorization Server to immediately revoke access to active client sessions in this scenario. Maybe there should be an explicit grace period?
The requirement that resource server metadata only have a single URL reference to an authorization server might be relaxed.
The details around session and token lifetimes might change with further security review.
--------------------------------------------------------------------------------
# specs > record-key
# Record Keys
A **Record Key** (sometimes shortened to `rkey`) is used to name and reference an individual record within the same collection of an atproto repository. It ends up as a segment in AT URIs, and in the repo MST path. {{ className: 'lead' }}
A few different Record Key naming schemes are supported. Every record Lexicon schema will indicate which of the record key types should be used, depending on the needs and semantics of the record collection. {{ className: 'lead' }}
### Record Key Type: `tid`
This is the most common record naming scheme, using [TID ("timestamp identifier") syntax](/specs/tid), such as `3jzfcijpj2z2a`. TIDs are usually generated from a local clock at the time of record creation, with some additional mechanisms to ensure they always increment and are not reused or duplicated with the same collection in a given repository.
The original creation timestamp of a record can be inferred if it has a TID record key, but remember that these keys can be specified by the end user and could have any value, so they should not be trusted.
The same TID may be used for records in different collections in the same repository. This usually indicates some relationship between the two records (eg, a "sidecar" extension record).
An early motivation for the TID scheme was to provide a loose temporal ordering of records, which improves storage efficiency of the repository data structure (MST).
### Record Key Type: `nsid`
For cases where the record key must be a valid [NSID](/specs/nsid).
### Record Key Type: `literal:<value>`
This key type is used when there should be only a single record in the collection, with a fixed, well-known Record Key.
The most common value is `self`, specified as `literal:self` in a Lexicon schema.
### Record Key Type: `any`
Any string meeting the overall Record Key schema requirements (see below) is allowed. This is the most flexible type of Record Key.
This may be used to encode semantics in the name, for example, a domain name, integer, or (transformed) AT URI. This enables de-duplication and known-URI lookups.
### Record Key Syntax
Lexicon string type: `record-key`
Regardless of the type, Record Keys must fulfill some baseline syntax constraints:
- restricted to a subset of ASCII characters — the allowed characters are alphanumeric (`A-Za-z0-9`), period, dash, underscore, colon, or tilde (`.-_:~`)
- must have at least 1 and at most 512 characters
- the specific record key values `.` and `..` are not allowed
- must be a permissible part of repository MST path string (the above constraints satisfy this condition)
- must be permissible to include in a path component of a URI (following RFC-3986, section 3.3). The above constraints satisfy this condition, by matching the "unreserved" characters allowed in generic URI paths.
Record Keys are case-sensitive.
### Examples
Valid Record Keys:
```
3jui7kd54zh2y
self
example.com
~1.2-3_
dHJ1ZQ
pre:fix
_
```
Invalid Record Keys:
```
alpha/beta
.
..
#extra
@handle
any space
any+space
number[3]
number(3)
"quote"
dHJ1ZQ==
```
### Usage and Implementation Guidelines
Implementations should not rely on global uniqueness of TIDs, and should not trust TID timestamps as actual record creation timestamps. Record Keys are "user-controlled data" and may be arbitrarily selected by hostile accounts.
Most software processing repositories and records should be agnostic to the Record Key type and values, and usually treat them as simple strings. For example, relying on TID keys to decode as `base32` into a unique `uint64` makes it tempting to rely on this for use as a database key, but doing so is not resilient to key format changes and is discouraged.
Note that in the context of a repository, the same Record Key value may be used under multiple collections. The tuple of `(did, rkey)` is not unique; the tuple `(did, collection, rkey)` is unique.
As a best practice, keep key paths to under 80 characters in virtually all situations.
The colon character was de-facto allowed in the reference implementation in 2023, and the spec was updated to allow this character in February 2024.
Note that "most" DIDs work as record keys, but that the full DID W3C specification allows DIDs with additional characters. This means that DIDs could be used as record keys at time of writing, and this will work with current "blessed" DID methods (note that `did:web` is restricted in atproto to only full domains, not paths or port specification), but that this could break in the future with different DID methods or DID features (like `did:web` paths) allowed.
While Record Keys are case-sensitive, it is a recommended practice to use all-lower-case Record Keys to avoid confusion and maximize possible re-use in case-insensitive contexts.
### Possible Future Changes
The constraints on Record Key syntax may be relaxed in the future to allow non-ASCII Unicode characters. Record keys will always be valid Unicode, never relaxed to allow arbitrary byte-strings.
Additional Record Key types may be defined.
The maximum length may be tweaked.
The `%` character is reserved for possible use with URL encoding, but note that such encoding is not currently supported.
Additional constraints on the generic syntax may be added. For example, requiring at least one alphanumeric character.
--------------------------------------------------------------------------------
# specs > repository
# Repository
*See the [Data Repositories Guide](../guides/data-repos) for a higher-level introduction.*
Public atproto content (**records**) is stored in per-account repositories (frequently shortened to **repo**). All currently active records are stored in the repository, and current repository contents are publicly available, but both content deletions and account deletions are fully supported. {{ className: 'lead' }}
The repository data structure is content-addressed (a [Merkle-tree](https://en.wikipedia.org/wiki/Merkle_tree)), and every mutation of repository contents (eg, addition, removal, and updates to records) results in a new commit `data` hash value (CID). Commits are cryptographically signed, with rotatable signing keys, which allows recursive validation of content as a whole or in part. {{ className: 'lead' }}
Repositories and their contents are canonically stored in binary [DAG-CBOR](https://ipld.io/docs/codecs/known/dag-cbor/) format, as a graph of data objects referencing each other by content hash (CID Links). Large binary blobs are not stored directly in repositories, though they are referenced by hash ([CID](https://github.com/multiformats/cid)). This includes images and other media objects. Repositories can be exported as [CAR](https://ipld.io/specs/transport/car/carv1/) files for offline backup, account migration, or other purposes. {{ className: 'lead' }}
In the atproto federation architecture, the authoritative location of an account's repository is the associated Personal Data Server (PDS). An account's current PDS location is authoritatively indicated in the DID Document. {{ className: 'lead' }}
In real-world use, it is expected that individual repositories will contain anywhere from dozens to millions of records. {{ className: 'lead' }}
## Repo Data Structure (v3)
This describes version `3` of the repository binary format.
Version `2` had a slightly different commit object schema, but is mostly compatible with `3`.
Version `1` had a different MST fanout configuration, and an incompatible schema for commits and repository metadata. Version `1` is deprecated, no repositories in this format exist in the network, and implementations do not need to support it.
At a high level, a repository is a key/value mapping where the keys are path names (as strings) and the values are records (DAG-CBOR objects).
A **Merkle Search Tree** (MST) is used to store this mapping. This content-addressed deterministic data structure stores data in key-sorted order. It is reasonably efficient for key lookups, key range scans, and appends (assuming sorted record paths). The properties of MSTs in general are described in this academic publication:
> Alex Auvolat, François Taïani. Merkle Search Trees: Efficient State-Based CRDTs in Open Networks. SRDS 2019 - 38th IEEE International Symposium on Reliable Distributed Systems, Oct 2019, Lyon, France. pp.1-10, ff10.1109/SRDS.2019.00032 ([pdf](https://inria.hal.science/hal-02303490/document))
The specific details of the MST as used in atproto repositories are described below.
Repo paths are strings, while MST keys are byte arrays. Neither may be empty (zero-length). While repo path strings are currently limited to a subset of ASCII (making encoding a no-op), the encoding is specified as UTF-8.
Repo paths currently have a fixed structure of `<collection>/<record-key>`. This means a valid, normalized [NSID](./nsid), followed by a `/`, followed by a valid [Record Key](./record-key). The path should not start with a leading `/`, and should always have exactly two path segments. The ASCII characters allowed in the entire path string are currently: letters (`A-Za-z`), digits (`0-9`), slash (`/`), period (`.`), hyphen (`-`), underscore (`_`), and tilde (`~`). The specific path segments `.` and `..` are not valid NSIDs or Record Keys, and will always be disallowed in repo paths.
Note that repo paths for all records in the same collection are sorted together in the MST, making enumeration (via key scan) and export efficient. Additionally, the TID Record Key scheme was intentionally selected to provide chronological sorting of MST keys within the scope of a collection. Appends are more efficient than random insertions/mutations within the tree, and when enumerating records within a collection they will be in chronological order (assuming that TID generation was done correctly, which cannot be relied on in general).
### Commit Objects
The top-level data object in a repository is a signed commit. The data fields are:
- `did` (string, required): the account DID associated with the repo, in strictly normalized form (eg, lowercase as appropriate)
- `version` (integer, required): fixed value of `3` for this repo format version
- `data` (CID link, required): pointer to the top of the repo contents tree structure (MST)
- `rev` (string, TID format, required): revision of the repo, used as a logical clock. Must increase monotonically. Recommend using current timestamp as TID; `rev` values in the "future" (beyond a fudge factor) should be ignored and not processed.
- `prev` (CID link, nullable): pointer (by hash) to a previous commit object for this repository. Could be used to create a chain of history, but largely unused (included for v2 backwards compatibility). In version `3` repos, this field must exist in the CBOR object, but is virtually always `null`. NOTE: previously specified as nullable and optional, but this caused interoperability issues.
- `sig` (byte array, required): cryptographic signature of this commit, as raw bytes
An UnsignedCommit data object has all the same fields except for `sig`. The process for signing a commit is to populate all the data fields, and then serialize the UnsignedCommit with DAG-CBOR. The output bytes are then hashed with SHA-256, and the binary hash output (without hex encoding) is then signed using the current "signing key" for the account. The signature is then stored as raw bytes in a commit object, along with all the other data fields.
The CID for a commit overall is generated by serializing a *signed* commit object as DAG-CBOR. See notes on the "blessed" CID format below, and in particular be sure to use the `dag-cbor` multicodec for CIDs linking to commit objects.
Note that neither the signature itself nor the signed commit indicate either the type of key used (curve type), or the specific public key used. That information must be fetched from the account's DID document. With key rotation, verification of older commit signatures can become ambiguous. The most recent commit should always be verifiable using the current DID document. This implies that a new repository commit should be created every time the signing key is rotated. Such a commit does not need to update the `data` CID link.
### MST Structure
At a high level, the repository MST is a key/value mapping where the keys are non-empty byte arrays, and the values are CID links to records. The MST data structure should be fully reproducible from such a mapping of bytestrings-to-CIDs, with exactly reproducible root CID hash (aka, the `data` field in commit object).
Every node in the tree structure contains a set of key/CID mappings, as well as links to other sub-tree nodes. The entries and links are in key-sorted order, with all of the keys of a linked sub-tree (recursively) falling in the range corresponding to the link location. The sort order is from **left** (lexically first) to **right** (lexically latter). Each key has a **depth** derived from the key itself, which determines which sub-tree it ends up in. The top node in the tree contains all of the keys with the highest depth value (which for a small tree may be all depth zero, so a single node). Links to the left or right of the entire node, or between any two keys in the node, point to a sub-tree node containing keys that fall in the corresponding key range.
An empty repository with no records is represented as a single MST node with an empty array of entries. This is the only situation in which a tree may contain an empty leaf node which does not either contain keys ("entries") or point to a sub-tree containing entries. The top of the tree must not be a an empty node which only points to a sub-tree. Empty intermediate nodes are allowed, as long as they point to a sub-tree which does contain entries. In other words, empty nodes must be pruned from the top and bottom of the tree, but empty intermediate nodes must be kept, such that sub-tree links do not skip a level of depth. The overall structure and shape of the MST is deterministic based on the current key/value content, regardless of the history of insertions and deletions that lead to the current contents.
For the atproto MST implementation, the hash algorithm used is SHA-256 (binary output), counting "prefix zeros" in 2-bit chunks, giving a fanout of 4. To compute the depth of a key:
- hash the key (a byte array) with SHA-256, with binary output
- count the number of leading binary zeros in the hash, and divide by two, rounding down
- the resulting positive integer is the depth of the key
Some examples, with the given ASCII strings mapping to byte arrays:
- `2653ae71`: depth "0"
- `blue`: depth "1"
- `app.bsky.feed.post/454397e440ec`: depth "4"
- `app.bsky.feed.post/9adeb165882c`: depth "8"
There are many MST nodes in repositories, so it is
The node data schema fields are:
- `l` ("left", CID link, nullable): link to sub-tree Node on a lower level and with all keys sorting before keys at this node
- `e` ("entries", array of objects, required): ordered list of TreeEntry objects
- `p` ("prefixlen", integer, required): count of bytes shared with previous TreeEntry in this Node (if any)
- `k` ("keysuffix", byte array, required): remainder of key for this TreeEntry, after "prefixlen" have been removed
- `v` ("value", CID Link, required): link to the record data (CBOR) for this entry
- `t` ("tree", CID Link, nullable): link to a sub-tree Node at a lower level which has keys sorting after this TreeEntry's key (to the "right"), but before the next TreeEntry's key in this Node (if any)
When parsing MST data structures, the depth and sort order of keys should be verified. This is particularly true for untrusted inputs, but is simplest to just verify every time. Additional checks on node size and other parameters of the tree structure also need to be limited; see the "Security Considerations" section of this document.
### CID Formats
The IPFS CID specification is very flexible, and supports a wide variety of hash types, a field indicating the type of content being linked to, and various string encoding options. These features are valuable to allow evolution of the repo format over time, but to maximize interoperability among implementations, only a specific "blessed" set of CID types are allowed.
The blessed format for commit objects and MST node objects, when linking to commit objects, MST nodes (aka, `data`, or MST internal links), or records (aka, MST leaf nodes to records), is:
- CIDv1
- Multibase: binary serialization within DAG-CBOR (or `base32` for JSON mappings)
- Multicodec: `dag-cbor` (0x71)
- Multihash: `sha-256` with 256 bits (0x12)
In the context of repositories, it is also desirable for the overall data structure to be reproducible given the contents, so the allowed CID types are strictly constrained and enforced. Commit objects with non-compliant `prev` or `data` links are considered invalid. MST Node objects with non-compliant links to other MST Node objects are considered invalid, and the entire MST data structure invalid.
More flexibility is allowed in processing the "leaf" links from MST to records, and implementations should retain the exact CID links used for these mappings, instead of normalizing. Implementations should strictly follow the CID blessed format when generating new CID Links to records.
## CAR File Serialization
The standard file format for storing data objects is Content Addressable aRchives (CAR). The standard repository export format for atproto repositories is [CAR v1](https://ipld.io/specs/transport/car/carv1/), which have file suffix `.car` and mimetype `application/vnd.ipld.car`.
The CARv1 format is very simple. It contains a small metadata header (which can indicate one or more "root" CID links), and then a series of binary "blocks", each of which is a data object. In the context of atproto repositories:
- The first element of the CAR `roots` metadata array must be the CID of the most relevant Commit object. for a generic export, this is the current (most recent) commit. additional CIDs may also be present in the `roots` array, with (for now) undefined meaning or order
- For full exports, the full repo structure must be included for the indicated commit, which includes all records and all MST nodes
- The order of blocks within the CAR file is not currently defined or restricted. implementations may have a "preferred" ordering, but should be tolerant of unexpected ordering
- Additional blocks, including records, may or may not be included in the CAR file
When
The CARv1 specification is agnostic about the same block appearing multiple times in the same file ("Duplicate Blocks)". Implementations should be robust to both duplication and de-duplication of blocks, and should also ignore any unnecessary or unlinked blocks.
## Repository Diffs
A concept which supports efficient synchronization of data between independent services is "diffs" of repository trees between different revisions. The basic principle is that a repository diff contains all the data (commit object, MST nodes, and records) that have changed between an older revision and the current revision of a repo. The diff can be "applied" to the older mirror of the repository, and the result will be the complete MST tree at the current (newer) commit revision.
Repo diffs can be serialized as CAR files, sometimes referred to as "CAR slices". Some details about diff CAR slices:
- same format, version, and atproto-specific constraints as full repo export CAR files
- blocks "should" be de-duplicated by CID (only one copy included), though receiving implementations must be resilient to duplication
- the root CID indicated in the CAR header (the first element of `roots`) should point to the commit block (which must be included)
- any required blocks must be included even if they have appeared in the history of the repository previously. eg, if a record is created in rev C, deleted in rev F, and re-created in rev N, the diff "since F" must include the record block
- all "created" records must be included
- any records which have been "deleted" and do not exist in the current repo should not be included
- any records which have been "updated" should include the final version, and should not include the previous version
- all MST nodes in the current repo which didn't exist in the previous repo version must be included
- with the exception of removed record data, the diff may include additional blocks, which receivers should ignore.
- however, diffs which intentionally contain a large amount of irrelevant block data to consume network or compute resources are considered a form of network abuse.
The diff is a partial Merkle tree, including a signed commit, and can be partially verified. This means that an observer which has successfully resolved the identity of the relevant account (including cryptographic public keys) can verify certain aspects of the data. The diff is a reliable "proof chain" for creation and updates of records: an observer can verify that the new or updated records have the specific record values in the overall repo as of the commit revision. If the observer knows of specific records (by repo path, or by full AT-URI) that have been deleted, they can verify that those records no longer exist in the repo as of the final commit revision.
However, an observer which does not know the full state of the repository at the "older" revision *can not* reliably enumerate all of the records that have been removed from the repository. Such an observer also can not see the previous values of deleted or updated records, either as full values or by CID. Note that the later is an intentional design goal for the diff concept: it is desired that content deletion happen rapidly and not "draw attention" to the content which has been deleted. It is technically possible for "archival" observers to track deletion events and lookup the previous content value, but this requires additional resources and effort.
Sometimes repo diffs are generated automatically. For example, every commit to a repo can result in a diff against the immediately preceding commit. In other contexts, diffs are generated on demand: a diff can be requested "since" an arbitrary previous revision. It is not expected that repo hosts support generating diffs between two arbitrary revisions, only "from" an arbitrary older revision and the current revision. Repo hosts are not required to maintain a complete history of prior commits/revisions, and in some cases (such as account migration) may never have had prior repo history. Some details about how to interpret and service requests for diffs "since" a prior revision:
- it is helpful to track internally the commit revision when a block (record or MST node) was created or re-created. This enables querying blocks "since" a point in time
- "since" revisions are not expected to be an exact match
- for example, if a repo had a sequence of commits "333", "666", "999", and a "since" value of "444" was requested, the changes in "666" and "999" should be included, as if the "since" parameter was "666"..
- a host is allowed to include additional history, but is encouraged to return the minimal or most granular requested data
- for example, a host may have "compacted" repo rev history to a smaller number of commits. If a repo had commit history "288", "300", "320", "340", "400", and got a request "since" 340, it might return all changes since 300. Hosts are encouraged to return the smallest diff when possible (eg, “since” 340), but clients should be resilient.
- if a host receives a “since” request earlier than the oldest available revision for a repository, it should return the full repository. This may happen if the host does not have the complete history of the repository.
- for example, if a repository had revisions "140", "150", and "160", then migrated to a new PDS and revisions continued "161" and "170", if the new PDS is asked for a diff "since" 150, the new PDS would probably need to return the full repository, because the earliest revision it would be aware of was "160" or "161" (depending on how migration was implemented).
In the specific case of chained commit-to-commit diffs which appear on the firehose, diffs should be "minimal": they should not contain additional records or additional history.
## Security Considerations
Repositories are untrusted input: accounts have full control over repository contents, and PDS instances have full control over binary encoding. It is
Generic precautions should be followed with CBOR decoding: a maximum serialized object size, a maximum recursion depth for nested fields, maximum memory budget for deserialized data, etc. Some CBOR libraries include these precautions by default, but others do not.
The efficiency of the MST data structure depends on key hashes being relatively randomly dispersed. Because accounts have control over Record Keys, they can mine for sets of record keys with particular depths and sorting order, which result in inefficient tree shapes, which can cause both large storage overhead, and network amplification in the context of federation streams. To protect against these attacks, implementations should limit the number of TreeEntries per Node to a statistically unlikely maximum length. It may also be necessary to limit the overall depth of the repo, or other parameters, to prevent more sophisticated key mining attacks.
When
## Possible Future Changes
An optional in-repo mechanism for storing multiple versions of the same record (by path) may be implemented. Eg, adding additional path field to indicate the version by CID, timestamp, or monotonically increasing version integer.
Mechanisms for storing metadata associated with each record are being considered, for example, generic label, re-use rights, or hashtag metadata. This would allow mutating the metadata without mutating the record itself, and make some metadata generic across lexicons.
Repo path restrictions may be relaxed in other ways, including fewer or additional path segments, more allowed characters (including non-ASCII), etc. Paths will always be valid Unicode strings, mapped to MST keys (byte arrays) by UTF-8 encoding.
At the overall atproto specification level, additional "blessed" cryptographic algorithms may be added over time. Likewise, additional CID formats to reference records and blobs may be added. Internal CID format changes would require a repo format version bump.
Repository CAR exports may include linked "blobs" (larger binary files). This might become the default, or a configurable option, or some another mechanism for blob export might be chosen (eg, `.tar` or `.zip` export).
Record content could conceivably be something other than DAG-CBOR some day. This would probably be a repo format version bump. Note that it is possible to efficiently wrap other data formats in a DAG-CBOR wrapper (via a byte array field), or to have a small DAG-CBOR record type that links to a blob in arbitrary format.
Repository CAR exports may end up with a preferred block ordering scheme specified.
The CARv2 file format, which includes optimizations for some use cases, may be adopted in some form.
Adding optional fields to commit and MST node objects may or may not result in a repo format version change. Changing the MST fanout, or any changes to the current MST fields, would be a full repo version change.
--------------------------------------------------------------------------------
# specs > sync
# Data Synchronization
One of the main design goals of atproto (the "Authenticated Transfer Protocol") is to reliably distribute public content between independent network services. This data transfer should be trustworthy (cryptographicly authenticated) and relatively low-latency even at large scale. It is also
This section describes the major data synchronization features in atproto. The primary real-time data synchronization mechanism is repository event streams, commonly referred to as "firehoses". The primary batch data transfer mechanism is repository exports as CAR files. These two mechanisms can be combined in a "bootstrapping" process which result in a live-synchronized copy of the network.
## Synchronization Primitives
As described in the repository spec, each commit to a repository has a *revision* number, in TID syntax. The revision number must always increase between commits for the same account, even if the account is migrated between hosts or has a period of inactivity in the network. Revision numbers can be used as a logical clock to aid synchronization of individual accounts. To keep this simple, it is recommended to use the current time as a TID for each commit, including the initial commit when creating a new account. Services should reject or ignore revision numbers corresponding to future timestamps (beyond a short fuzzy time drift window). Network services can track the commit revision for every account they have seen, and use this to verify synchronization progress. Services which synchronize data can include the most-recently-processed revision in HTTP responses to API requests from the account in question, in the `Atproto-Repo-Rev` HTTP response header. This allows clients (and users) to detect if the response is up-to-date with the actual repository, and detect any problems with synchronization.
## Firehose
The repository event stream (`com.atproto.sync.subscribeRepos`, also called the "firehose") is an [Event Stream](/specs/event-stream) which broadcasts updates to repositories (`#commit` events), handles and DID documents (`#identity`), and account hosting status (`#account`). PDS hosts provide a single stream with updates for all locally-hosted accounts. "Relays" are network services which subscribe to one or more repo streams (eg, multiple PDS instances) and aggregate them in to a single combined repo stream. The combined stream has the same structure and event types. A Relay which aggregates nearly all accounts from nearly all PDS instances in the network (possibly through intermediate relays) outputs a “full-network” firehose. Relays often mirror and can re-distribute the repository contents, though their core functionality is to verify content and output a unified firehose.
In most cases the repository data synchronized over the firehose is self-certifying (contains verifiable signatures), and consumers can verify content without making additional requests directly to account PDS instances. It is possible for services to redact events from the firehose, such that downstream services would not be aware of new content.
Identity and account information is *not* self-certifying, and services may need need to verify independently. This usually means independent DID and [handle resolution](/specs/handle). Account hosting status might also be checked at account PDS hosts, to disambiguate hosting status at different pieces of infrastructure.
The event message types are declared in the `com.atproto.sync.subscribeRepos` Lexicon schema, and are summarized below. A few fields are the same for all event types (except for `repo` vs `did` for `#commit` events):
- `seq` (integer, required): used to ensure reliable consumption, as described in Event Streams
- `did` / `repo`(string with DID syntax, required): the account/identity associated with the event
- `time` (string with datetime syntax, required): an informal and non-authoritative estimate of when event was received. Intermediate services may decide to pass this field through as-is, or update to the current time
### `#identity` Events
Indicates that there *may* have been a change to the indicated identity (meaning the DID document or handle), and optionally what the current handle is. Does not indicate what changed, or reliably indicate what the current state of the identity is.
Event fields:
- `seq` (integer, required): same for all event types
- `did` (string with DID syntax, required): same for all event types
- `time` (string with datetime syntax, required): same for all event types
- `handle` (string with handle syntax, optional): the current handle for this identity. May be `handle.invalid` if the handle does not currently resolve correctly.
Presence or absence of the `handle` field does not indicate that it is the handle which has changed.
The semantics and expected behavior are that downstream services should update any cached identity metadata (including DID document and handle) for the indicated DID. They might mark caches as stale, immediately purge cached data, or attempt to re-resolve metadata.
Identity events are emitted on a "best-effort" basis. It is possible for the DID document or handle resolution status to change without any atproto service detecting the change, in which case an event would not be emitted. It is also possible for the event to be emitted redundantly, when nothing has actually changed.
Intermediate services (eg, relays) may chose to modify or pass through identity events:
- they may replace the handle with the result of their own resolution; or always remove the handle field; or always pass it through unaltered
- they may filter out identity events if they observe that identity has not actually changed
- they may emit identity events based on changes they became aware of independently (eg, via periodic re-validation of handles)
### `#account` Events
Indicates that there may have been a change in [Account Hosting status](/specs/account) at the service which emits the event, and what the new status is. For example, it could be the result of creation, deletion, or temporary suspension of an account. The event describes the current hosting status, not what changed.
Event Fields:
- `seq` (integer, required): same for all event types
- `did` (string with DID syntax, required): same for all event types
- `time` (string with datetime syntax, required): same for all event types
- `active` (boolean, required): whether the repository is currently available and can be redistributed
- `status` (string, optional): string status code which describes the account state in more detail. Known values include:
- `takendown`: indefinite removal of the repository by a service provider, due to a terms or policy violation
- `suspended`: temporary or time-limited variant of `takedown`
- `deleted`: account has been deactivated, possibly permanently.
- `deactivated`: temporary or indefinite removal of all public data by the account themselves.
When coming from any service which redistributes account data, the event describes what the new status is *at that service*, and is authoritative in that context. In other words, the event is hop-by-hop for repository hosts and mirrors.
See the Account Hosting specification for more details.
### `#commit` Events
This event indicates that there has been a new repository commit for the indicated account. The event usually contains the "diff" of repository data, in the form of a CAR slice. See the [Repository specification](/specs/repository) for details on "diffs" and the CAR file format.
See the Repository specification for more details around repo diffs.
Event Fields:
- `seq` (integer, required): same for all event types
- `repo` (string with DID syntax, required): the same as `did` for all other event types
- `time` (string with datetime syntax, required): same for all event types
- `rev` (string with TID syntax, required): the revision of the commit. Must match the `rev` in the commit block itself.
- `since` (string with TID syntax, nullable): indicates the `rev` of a preceding commit, which the the repo diff contains differences from
- `commit` (cid-link, required): CID of the commit object (in `blocks`)
- `tooBig` (boolean, required): if true, indicates that the repo diff was too large, and that `blocks`, `ops`, and complete `blobs` are not all included
- `blocks` (bytes, required): CAR "slice" for the corresponding repo diff. The commit object must always be included.
- `ops` (array of objects, required): list of record-level operations in this commit: specific records created, updated, deleted
- `blobs` (array of cid-link, required): set of new blobs (by CID) referenced by records in this commit
Commit events are broadcast when the account repository changes. Commits can be "empty", meaning no actual record content changed, and only the `rev` was incremented. They can contain a single record update, or multiple updates. Only the commit object, record blocks, and MST tree nodes are authenticated (signed): the `since`, `ops`, `blobs`, and `tooBig` fields are not self-certifying, and could in theory be manipulated, or otherwise be incorrect or incomplete.
If `since` is not included, the commit should include the full repo tree, or set the `tooBig` flag.
If the `tooBig` flag is set, then the amount of updated data was too much to be serialized in a single stream event message. Downstream services which want to maintain complete synchronized copies for the repo need to fetch the diff separately, as discussed below.
### Firehose Validation Best Practices
A service which does full validation of upstream events has a number of properties to track and check. For example, Relay instances should fully validate content from PDS instances before re-broadcasting.
Here is a summary of validation rules and behaviors:
- services should independently resolve identity data for each DID. They should ignore `#commit` events for accounts which do not have a functioning atproto identity (eg, lacking a signing key, or lacking a PDS service entry, or for which the DID has been tombstoned)
- services which subscribe directly to PDS instances should keep track of which PDS is authoritative for each DID. They should remember the host each subscription (WebSocket) is connected to, and reject `#commit` events for accounts if they come from a stream which does not correspond to the current account for that DID
- services should track account hosting status for each DID, and ignore `#commit` events for events which are not `active`
- services should verify commit signatures for each `#commit` event, using the current identity data. If the signature initially fails to verify, the service should refresh the identity metadata in case it had recently changed. Events with conclusively invalid signatures should be rejected.
- services should reject any event messages which exceed reasonable limits. A reasonable upper bound for producers is 5 MBytes (for any event stream message type). The `subscribeRepos` Lexicon also limits `blocks` to one million bytes, and `ops` to 200 entries. Commits with too much data must use the `tooBig` mechanism, though such commits should generally be avoided in the first place by breaking them up in to multiple smaller commits.
- services should verify that repository data structures are valid against the specification. Missing fields, incorrect MST structure, or other protocol-layer violations should result in events being rejected.
- services may apply rate-limits to identity, account, and commit events, and suspend accounts or upstream services which violate those limits. Rate limits might also be applied to recovery modes such as invalid signatures resulting in an identity refresh, `tooBig` events, missing or out-of-order commits, etc.
- services should ignore commit events with a `rev` lower or equal to the most recent processed `rev` for that DID, and should reject commit events with a `rev` corresponding to a future timestamp (beyond a clock drift window of a few minutes)
- services should check the `since` value in commit events, and if it is not consistent with the previous seen `rev` for that DID (see discussion in "Reliable Synchronization"), mark the repo as out-of-sync (similar to a `tooBig` commit event)
- data limits on records specifically should be verified. Events containing corrupt or entirely invalid records may be rejected. for example, a record not being CBOR at all, or exceeding normal data size limits.
- more subtle data validation of records may be enforced, or may be ignored, depending on the service. For example, unsupported CID hash types embedded in records should probably be ignored by Relays (even if they violate the atproto data model), but may result in the record or commit event being rejected by an AppView
- mirroring services, which retain a full copy of repository data, should verify that commit diffs leave the MST tree in a complete and valid state (eg, no missing records, no invalid MST nodes, commit CID would be reproducible if the MST structure was re-generated from scratch)
- Relays (specifically) should not validate records against Lexicons
## Reliable Synchronization
This section describes some details on how to reliably subscribe to the firehose and maintain an existing synchronized mirror of the network.
Services should generally maintain a few pieces of state for all accounts they are tracking data from:
- track the most recent commit `rev` they have successfully processed
- keep cached identity data, and use cache expiration to ensure periodic re-validation of that data
- track account status
Identity caches should be purged any time an `#identity` event is received. Additionally, identity resolution should be refreshed if a commit signature fails to verify, in case the signing key was updated but the identity cache has not been updated yet.
When `tooBig` events are emitted on the firehose, downstream services will need to fetch the diff out-of-band. This usually means an API request to the `com.atproto.sync.getRepo` endpoint on the current PDS host for the account, with the `since` field included. The `since` value should be the most recently processed `rev` value for the account, which may or may not match the `since` field in the commit event message.
If a `#commit` is received with a `since` that does not match the most recently processed `rev` for the account, and is “later” (higher value) than the most recent commit `rev` the service has processed for that account, the service may need to do the same kind of out-of-band fetch as for a `tooBig` event.
Services should keep track of the `seq` number of their upstream subscriptions. This should be stored separately per-upstream, even if there is only a single Relay connection, in case a different Relay is subscribed to in the future (which will have different `seq` numbers).
Events can be processed concurrently, but they should be processed sequentially in-order for any given account. This can be accomplished by partitioning multiple workers using the repo DID as a partitioning key.
Services can confirm that they are consuming content reliably by fetching a snapshot of repository DIDs and `rev` numbers from other services, including PDS hosts and Relay instances. After a short delay, these can be compared against the current state of the service to identify any accounts which have lower than expected `rev` numbers. These repos can then be updated out-of-band.
## Bootstrapping a Live Mirror
The firehose can be used to follow new data updates, and repo exports can be used for snapshots. Actually combining the two to bootstrap a complete live-updating mirror can be a bit tricky. One approach is described below.
Keep a sync status table for all accounts (DIDs) encountered. The status can be:
- `dirty`: there is either no local repo data for this account, or it has gotten out of sync
- `in-process`: the repo is "dirty", but there is a background task in process to update it
- `synchronized`: a complete copy of the repository has been processed
Start by subscribing to the full firehose. If there is no existing repository data for the account, mark the account as "dirty". When new events come in for a repo, the behavior depends on the repo state. If it is "dirty", the event is ignored. If the state is "synchronized", the event is immediately processed as an update to the repo. If the state is "in-process", the event is enqueued locally.
Have a set of background workers start processing "dirty" repos. First they mark the status as `in-process`, so that new events are enqueued locally. Then the full repo export (CAR file) is fetched from the PDS and processed in full. The commit `rev` of the repo export is noted. When the full repo
After some time, most of the known accounts will be marked as `synchronized`, though this will only represent the most recently active accounts in the network. Next a more complete set of repositories in the network can be fetched, for example using an API query against an existing large service. Any new identified accounts can be marked as `dirty` in the service, and the background workers can start processing them.
When all of the accounts are `synchronized`, the process is complete. At large scale it may be hard to get perfect synchronization: PDS instances may be down at various times, identities may fail to resolve, or invalid events, data, or signatures may end up in the network.
## Usage and Implementation Guidelines
Guidelines for specific firehose event sequencing during different account events are described in an [Account Lifecycle Best Practices guide](/guides/account-lifecycle).
## Security Concerns
General mitigations for resource exhaustion attacks are recommended: event rate-limits, data quotas per account, limits on data object sizes and deserialized data complexity, etc.
Care should always be taken when making network requests to unknown or untrusted hosts, especially when the network locators for those host from from untrusted input. This includes validating URLs to not connect to local or internal hosts (including via HTTP redirects), avoiding SSRF in browser contexts, etc.
To prevent traffic amplification attacks, outbound network requests should be rate-limited by host. For example, identity resolution requests when consuming from the firehose, including DNS TXT traffic volume and DID resolution requests.
## Future Work
The `subscribeRepos` Lexicon is likely to be tweaked, with deprecated fields removed, even if this breaks Lexicon evolution rules.
The event stream sequence/cursor scheme may be iterated on to support sharding, timestamp-based resumption, and easier failover between independent instances.
Alternatives to the full authenticated firehose may be added to the protocol. For example, simple JSON serialization, filtering by record collection type, omitting MST nodes, and other changes which would simplify development and reduce resource consumption for use-cases where full authentication is not necessary or desired.
--------------------------------------------------------------------------------
# specs > tid
# Timestamp Identifiers (TIDs)
A TID ("timestamp identifier") is a compact string identifier based on an integer timestamp. They are sortable, appropriate for use in web URLs, and useful as a "logical clock" in networked systems. TIDs are currently used in atproto as record keys and for repository commit "revision" numbers.
Note: There are similarities to ["snowflake identifiers"](https://en.wikipedia.org/wiki/Snowflake_ID). In the decentralized context of atproto, the global uniqueness of TIDs can not be guaranteed, and an antagonistic repo controller could trivially create records re-using known TIDs.
## TID Structure
The high-level semantics of a TID are:
- 64-bit integer
- big-endian byte ordering
- encoded as `base32-sortable`. That is, encoded with characters `234567abcdefghijklmnopqrstuvwxyz`
- no special padding characters (like `=`) are used, but all digits are always encoded, so length is always 13 ASCII characters. The TID corresponding to integer zero is `2222222222222`.
The layout of the 64-bit integer is:
- The top bit is always 0
- The next 53 bits represent microseconds since the UNIX epoch. 53 bits is chosen as the maximum safe integer precision in a 64-bit floating point number, as used by Javascript.
- The final 10 bits are a random "clock identifier."
TID generators should generate a random clock identifier number, chosen to avoid collisions as much as possible (for example, between multiple worker instances of a PDS service cluster). A local clock can be used to generate the timestamp itself. Care should be taken to ensure the TID output stream is monotonically increasing and never repeats, even if multiple TIDs are generated in the same microsecond, or during "clock smear" or clock synchronization incidents. If the local clock has only millisecond precision, the timestamp should be padded. (You can do this by multiplying by 1000.)
## TID Syntax
Lexicon string type: `tid`
TID string syntax parsing rules:
- length is always 13 ASCII characters
- uses base32-sortable character set, meaning `234567abcdefghijklmnopqrstuvwxyz`
- the first character must be one of `234567abcdefghij`
Early versions of the TID syntax allowed hyphens, but they are no longer allowed and should be rejected when parsing.
A reference regex for TID is:
```
/^[234567abcdefghij][234567abcdefghijklmnopqrstuvwxyz]{12}$/
```
### Examples
Syntactically valid TIDs:
```
3jzfcijpj2z2a
7777777777777
3zzzzzzzzzzzz
2222222222222
```
Invalid TIDs:
```
# not base32
3jzfcijpj2z21
0000000000000
# case-sensitive
3JZFCIJPJ2Z2A
# too long/short
3jzfcijpj2z2aa
3jzfcijpj2z2
222
# legacy dash syntax *not* supported (TTTT-TTT-TTTT-CC)
3jzf-cij-pj2z-2a
# high bit can't be set
zzzzzzzzzzzzz
kjzfcijpj2z2a
```
--------------------------------------------------------------------------------
# specs > xrpc
# HTTP API (XRPC)
HTTP APIs for client-server and server-server requests in atproto use a set of common conventions called XRPC. Endpoint path names include an NSID indicating the [Lexicon](/specs/lexicon) specifying the request and response schemas, usually with JSON content type. {{ className: 'lead' }}
## Lexicon HTTP Endpoints
The HTTP request path starts with `/xrpc/`, followed by an NSID. Paths must always be top-level, not below a prefix. The NSID maps to the `id` field in the associated Lexicon.
The two requests types that can be expressed in Lexicons are "query" (HTTP GET) and "procedure" (HTTP POST). Following HTTP REST semantics, queries (GET) are cacheable and should not mutate resource state. Procedures are not cacheable and may mutate state.
Lexicon `params` (under the `parameters` field) map to HTTP URL query parameters. Only certain Lexicon types can be included in params, as specified by the `params` type. Multiple query parameters with the same name can be used to represent an array of parameters. When encoding `boolean` parameters, the strings `true` and `false` should be used. Strings should not be quoted. If a `default` value is included in the schema, it should be included in every request to ensure consistent caching behavior.
Request and response body content types can be specified in Lexicon. The schema can require an exact MIME type, or a blob pattern indicating a range of acceptable types (eg, `image/*`).
JSON body schemas are specified in Lexicon using the usual atproto data model. Full validation by the server or client requires knowledge of the Lexicon, but partial validation against the abstract data model is always possible.
CORS support is encouraged but not required.
### Error Responses
All unsuccessful responses should follow a standard error response schema. The `Content-Type` should be `application/json`, and the payload should be a JSON object with the following fields:
- `error` (string, required): type name of the error (generic ASCII constant, no whitespace)
- `message` (string, optional): description of the error, appropriate for display to humans
The error type should map to an error name defined in the endpoint's Lexicon schema. This enables more specific error-handling by client software. This is particularly encouraged on `400`, `500`, and `502` status codes, where further information will be useful.
### Blob Upload and Download
Blobs are something of a special case because they can have any MIME type and are not stored directly in repositories, and thus are not directly associated with an NSID or Lexicon (though they do end up referenced from Lexicons).
The convention for working with blobs is for clients to upload them via the `com.atproto.repo.uploadBlob` endpoint, which returns a `blob` JSON object containing a CID and basic metadata about the blob. Client can then include this `blob` data in future requests (eg, include in new records). Constraints like MIME type and file size are only validated at this second step. The server may implement content type sniffing at the upload step and return a MIME type different from any `Content-Type` header provided, but a `Content-Type` header is still expected on the upload HTTP request.
Blobs for a specific account can be listed and downloaded using endpoints in the `com.atproto.sync.*` NSID space. These endpoints give access to the complete original blob, as uploaded. A common pattern is for applications to mirror both the original blob and any downsized thumbnail or preview versions via separate URLs (eg, on a CDN), instead of deep-linking to the `getBlob` endpoint on the original PDS.
### Cursors and Pagination
A common pattern in Lexicon design is to include a `cursor` parameter for pagination. The client should not include the `cursor` parameter in the first request, and should keep all other parameters fixed between requests. If a cursor is included in a response, the next batch of responses can be fetched by including that value in a follow-on, continuing until the cursor is not included any longer, indicating the end of the result set has been reached.
## Authentication
The primary client/server authentication and authorization scheme for XRPC is OAuth, described in the [Auth Specification](./oauth).
Not all endpoints require authentication, but there is not yet a consistent way to enumerate which endpoints do or do not.
There is also a legacy authentication scheme using HTTP Bearer auth with JWT tokens, including refresh tokens, described here. Initial login uses the `com.atproto.server.createSession` endpoint, including the password and an account identifier (eg, handle or registered email address). This returns a `refreshJwt` (as well as an initial `accessJwt`).
Most requests should be authenticated using an access JWT, but the validity lifetime for these tokens is short. Every couple minutes, a new access JWT can be requested by hitting the `com.atproto.server.refreshSession` endpoint, using the refresh JWT instead of an access JWT.
Clients should treat the tokens as opaque string tokens: the JWT fields and semantics are not a stable part of the specification.
Servers (eg, PDS implementations) which generate and valiate auth JWTs should implement domain separation between access and refresh tokens, using the `typ` header field: access tokens should use `at+jwt`, and refresh tokens should use `refresh+jwt`. Note that `at+jwt` (defined in [RFC 9068](https://www.rfc-editor.org/rfc/rfc9068.html)) is short for "access token", and is not a reference to the "at" in atproto.
### App Passwords
App Passwords are a mechanism to reduce security risks when logging in to third-party clients and web applications. Accounts can create and revoke app passwords separate from their primary password. They are used to log in the same way as the primary password, but grant slightly restricted permissions to the client application, preventing destructive actions like account or changes to authentication settings (including app passwords themselves).
Clients and apps themselves do not need to do anything special to use app passwords. It is a best practice for most clients and apps to include a reminder to use an app password when logging in. App passwords usually have the form `xxxx-xxxx-xxxx-xxxx`, and clients can check against this format to prevent accidental logins with primary passwords (unless the primary password itself has this format).
### Admin Token (Temporary Specification)
Some administrative XRPC endpoints require authentication with admin privileges. The current scheme for this is to use HTTP Basic authentication with user "admin" and a fixed token in the password field, instead of HTTP Bearer auth with a JWT. This means that admin requests do not have a link to the account or identity of the client beyond "admin".
As a reminder, HTTP Basic authentication works by joining the username and password together with a colon (`:`), and encoding the resulting string using `base64` ("standard" version). The encoded string is included in the `Authorization` header, prefixed with `Basic ` (with separating space).
As an example, if the admin token was `secret-token`, the header would look like:
```
Authorization: Basic YWRtaW46c2VjcmV0LXRva2Vu
```
The set of endpoints requiring admin auth is likely to get out of date in this specification, but currently includes:
- `com.atproto.admin.*`
- `com.atproto.server.createInviteCode`
- `com.atproto.server.createInviteCodes`
### Inter-Service Authentication (JWT)
This section describes a mechanism for authentication between services using signed JWTs.
The current mechanism is to use short-lived JWTs signed by the account's atproto signing key. The receiving service can validate the signature by checking this key against the account's DID document.
The JWT parameters are:
- `alg` header field (string, required): indicates the signing key type (see [Cryptography](/specs/cryptography))
- use `ES256K` for `k256` keys
- use `ES256` for `p256` keys
- `typ` header field (string, required): currently `JWT`, but intend to update to a more specific value.
- `iss` body field (string, required): account DID that the request is being sent on behalf of. This may include a suffix service identifier; see below
- `aud` body field (string, required): service DID associated with the service that the request is being sent to
- `exp` body field (number, required): token expiration time, as a UNIX timestamp with seconds precision. Should be a short time window, as revocation is not implemented. 60 seconds is a good token lifetime.
- `iat` body field (number, required): token creation time, as a UNIX timestamp with seconds precision.
- `lxm` body field (string, optional): short for "lexicon method". NSID syntax. Indicates the endpoint that this token authorizes. Servers must always validate this field if included, and should require it for security-sensitive operations. May become required in the future.
- `jti` body field (string, required): unique random string nonce. May be used by receiving services to prevent reuse of token and replay attacks.
- JWT signature (string, required): base64url-encoded signature using the account DID's signing key.
When the token is generated in the context of a specific service in the issuer's DID document, the issuer field may have the corresponding *service* identifier in the `iss` field, separated by a `#` character. For example, `did:web:label.example.com#atproto_labeler` for a labeler service. When this is included the appropriate signing key is determined based on a fixed mapping of service identifiers to key identifiers:
- service identifier `atproto_labeler` maps to key identifier `atproto_label`
If the service identifier is not included, the scope is general purpose and the `atproto` key identifier should be used.
The receiving service may require or prohibit specific service identifiers for access to specific resources or endpoints.
The signature is computed using the regular JWT process, using the account's signing key (the same used to sign repo commits). As Typescript pseudo-code, this looks like:
```
const headerPayload = utf8ToBase64Url(jsonStringify(header)) + '.' + utf8ToBase64Url(jsonString(body))
const signature = hashAndSign(accountSigningKey, utf8Bytes(headerPayload))
const jwt = headerPayload + '.' + bytesToBase64Url(signature)
```
## Service Proxying
The PDS acts as a generic proxy between clients and other atproto services. Clients can use an HTTP header to specify which service in the network they want the request forwarded to (eg, a specific AppView or Labeler service). The PDS will do some safety checks, then forward the request on with an inter-service authentication token (JWT, described above) issued and signed by the user's identity.
The HTTP header is `atproto-proxy`, and the value is a DID (identifying a service), followed by a service endpoint identifier, joined by a `#` character. The PDS resolves the service DID, extracts a service endpoint URL from the DID document, and proxies the request on to the identified server.
An example request header, to proxy to a labeling service, is:
```
atproto-proxy: did:web:labeler.example.com#atproto_labeler
```
A few requirements must be met for proxying to happen. These conditions may be extended in the future to address network abuse concerns.
- the target service must have a resolvable DID, a well-formed DID document, and a corresponding service entry with a matching identifier
- only atproto endpoint paths are supported. This means an `/xrpc/` prefix, followed by a valid NSID and endpoint name. Note that the `/xrpc/` prefix may become configurable in the future
- the request must be from an authenticated user with an active account on the PDS
- rate-limits at the PDS still apply
## Summary of HTTP Headers
Clients can use the following request headers:
`Content-Type`: If a request body is present, this header should be included and indicate the content type.
`Authorization`: Contains auth information. See "Authentication" section of this specification for details.
`atproto-proxy`: used for proxying to other atproto services. See "Service Proxying" section of this document for details.
`atproto-accept-labelers`: used by clients to request labels from specific labelers to be included and applied in the response. See [Label](/specs/label) specification for details.
## Summary of HTTP Status Codes
`200 OK`: The request was successful. If there is a response body (optional), there should be a `Content-Type` header.
`400 Bad Request`: Request was invalid, and was not processed
`401 Unauthorized`: Authentication is required for this endpoint. There should be a `WWW-Authenticate` header.
`403 Forbidden`: The client lacks permission for this endpoint
`404 Not Found`: Can indicate a missing resource. This can also indicate that the server does not support atproto, or does not support this endpoint. See response error message (or lack thereof) to clairfy.
`413 Payload Too Large`: Request body was too large. If possible, split in to multiple smaller requests.
`429 Too Many Requests`: A resource limit has been exceeded, client should back off. There may be a `Retry-After` header indicating a specific back-off time period.
`500 Internal Server Error`: Generic internal service error. Client may retry after a delay.
`501 Not Implemented`: The specified endpoint is known, but not implemented. Client should *not* retry. In particular, returned when WebSockets are requested by not implemented by server.
`502 Bad Gateway`, `503 Service Unavailable`, or `504 Gateway Timeout`: These all usually indicate temporary or permanent service downtime. Clients may retry after a delay.
## Usage and Implementation Guidelines
Clients are encouraged to implement timeouts, limited retries, and randomized exponential backoff. This increases robustness in the inevitable case of sporadic downtime, while minimizing load on struggling servers.
Servers *should* implement custom JSON error responses for all requests with an `/xrpc/` path prefix, but realistically, many services will return generic load-balancer or reverse-proxy HTML error pages. Clients should be robust to non-JSON error responses.
HTTP servers and client libraries usually limit the overall size of URLs, including query parameters, and these limits constrain the use of parameters in XRPC.
PDS implementations are free to restrict blob uploads as they see fit. For example, they may have a global maximum size or restricted set of allowed MIME types. These should be a superset of blob constraints for all supported Lexicons.
## Security and Privacy Considerations
Only HTTPS should be used over the open internet.
Care should be taken with personally identifiable information in blobs, such as EXIF metadata. It is currently the *client's* responsibility to strip any sensitive EXIF metadata from blobs before uploading. It would be reasonable for a PDS to help prevent accidental metadata leakage as well; see future changes section below.
## Possible Future Changes
The auth system is likely to be entirely overhauled.
Lexicons should be able to indicate whether auth is required.
The role of the PDS as a generic gateway may be formalized and extended. A generic mechanism for proxying specific XRPC endpoints on to other network services may be added. Generic caching of queries and blobs may be specified. Mutation of third-party responses by the PDS might be explicitly allowed.
An explicit decision about whether HTTP redirects are supported.
Cursor pagination behavior should be clarified when a cursor is returned but the result list is empty, and when a cursor value is repeated.
To help prevent accidental publishing of sensitive metadata embedded in media blobs, a query parameter may be added to the upload blob endpoint to opt-out of metadata stripping, and default to either blocking upload or auto-striping such metadata for all blobs.
The `lxm` JWT field for inter-service auth may become required.
--------------------------------------------------------------------------------
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment