4
Petals – Decentralized platform for running 100B+ language models
source link: https://petals.ml/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
Petals
Run 100B+ language models at home, BitTorrent‑style
- Run large language models like BLOOM-176B collaboratively — you load a small part of the model, then team up with people serving the other parts to run inference or fine-tuning.
- Single-batch inference runs at ≈ 1 sec per step (token) — up to 10x faster than offloading, enough for chatbots and other interactive apps. Parallel inference reaches hundreds of tokens/sec.
- Beyond classic language model APIs — you can employ any fine-tuning and sampling methods, execute custom paths through the model, or see its hidden states. You get the comforts of an API with the flexibility of PyTorch.
Join our Discord
or subscribe via email
to follow Petals development:
Featured on:
This project is a part of the BigScience research workshop.
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK