Deploy Dolly v2.0 to SageMaker

deploy_dolly_v2.ipynb

Did you have any timeout issues when using this deployment? My deployed endpoint answered one question successfully and most other requests ended up timing out. But thank you, prior to finding the additions to requirements.txt and running in 8bit my previous deployments were running out of memory

Yep, I've had similar problems for certain prompts. I plan on testing the 7B version to see if it can respond to more complex prompts fast enough to avoid SageMaker's timeout limit. I think it should be as simple as changing the 12b's to 7b's in the notebook, and it probably doesn't need to be loaded in 8bit either.

Thanks for the example. We have deployed many conversational models on Sagemaker. The challenge is that this way the endpoint does not stream the response and a lot of times for longer responses it times out.

Thanks for the example. We have deployed many conversational models on Sagemaker. The challenge is that this way the endpoint does not stream the response and a lot of times for longer responses it times out.

You can try another conversational pattern for your server/client like a websocket

Has anyone used an inference config for the code as seen above so that the model can handle embeddings ?

Thanks for sharing, this is helping me a lot in trying to figure this topic out.
One question - why is there a mismatch between the transformers version in the requirements.txt file and in the Sagmaker model creation command? What is the difference, and how does it make sense that they will be different?