“`html
A Reddit user is reporting issues with the performance of their LLaMA model. They are using a combination of options including setting parallel to 1, and adjusting draft-related parameters like –spec-draft-n-max to 0.75. The current observed issue is that these adjustments seem to be making the model run slower rather than faster.
The user has noticed their throughput dropping from around 100 tokens per second (tok/s) when using these settings, down to approximately 80 tok/s. They are running this inside a Docker container on an RTX 5090 GPU and have tried various combinations of parameters without seeing improvements.
- The user is seeking advice on what might be wrong with their current setup or if there’s another way to optimize the model for better performance.
- It highlights potential issues in parameter tuning, especially when dealing with specific options like draft-related settings and parallel processing.
- This post could provide valuable insights into common pitfalls of fine-tuning LLaMA models, particularly those related to throughput optimization.
“`
Source Read original →
