Writing out the math used in Sonic MoE: https://arxiv.org/abs/2512.14080. Sec. 3.2 and Appendix C.
Goal: compute the backwards pass without needing to cache large tensors which would blow up with increased sparsification.
Tensors:
X_{ted}: input tensorsW^1_{edn}: up projection
Parallelism doesn't beat Amdahl's Law. This is the reason simulations requiring a supercomputer never make their way into laptop CAD applications.
Bootstrapping is a sequence of phases, where you use the speed & parallelism of the current phase to fabricate systems that eliminate the current biggest bottleneck.
We need to perform sequential chemical steps:
| use rusoto_core::ByteStream; | |
| use rusoto_s3::{S3Client, S3}; | |
| use futures::{future, stream, Stream, StreamExt, TryStreamExt}; | |
| pub async fn upload_stream( | |
| client: S3Client, | |
| bucket: String, | |
| key: String, | |
| data_stream: impl Stream<Item = String>, |
| def top_k_top_p_filtering(logits, top_k=0, top_p=0.0, filter_value=-float('Inf')): | |
| """ Filter a distribution of logits using top-k and/or nucleus (top-p) filtering | |
| Args: | |
| logits: logits distribution shape (vocabulary size) | |
| top_k >0: keep only top k tokens with highest probability (top-k filtering). | |
| top_p >0.0: keep the top tokens with cumulative probability >= top_p (nucleus filtering). | |
| Nucleus filtering is described in Holtzman et al. (http://arxiv.org/abs/1904.09751) | |
| """ | |
| assert logits.dim() == 1 # batch size 1 for now - could be updated for more but the code would be less clear | |
| top_k = min(top_k, logits.size(-1)) # Safety check |
