[Bug]: DeepSeek-R1-Distill-Qwen-1.5B & DeepSeek-R1-Distill-Qwen-7B models Fails with "Max Token Reached" Error When Using Larger Chunk Size and Higher k-NN Value

[shailajafissionqa]

FloTorch Version

V2.0.3

Bug Description

The inferencing of the DeepSeek-R1-Distill-Qwen-1.5B & DeepSeek-R1-Distill-Qwen-7B models fails with a "Max Token Reached" error when a larger chunk size is used in combination with a higher k-NN value (greater than 3). This issue prevents successful execution and impacts model performance with larger configurations.

Steps to Reproduce

1. Login to FloTorch.ai
2. Click on Get Started button
3. click on Create Project
4. Enter the Project Name, select the region
5. Upload the knowledge base pdf file
6. Upload the Ground Truth json file
7. Click on Next
8. Select the Chunk size as Hierarchical, child chuck size as 512 and parent chunk size as 2048
9. Select the chuck overlap percentage
10. Select the Embedding model and Vector dimension as 1024
11. Select the indexing algorithm
12. Select the N shot prompt and attach the N shot prompt file
13. Select the KNN value greater than 3 (select 5)
14. Select the inferencing models as "DeepSeek-R1-Distill-Qwen-1.5B" & "DeepSeek-R1-Distill-Qwen-7B"
15. Select the inferencing model temperature
16. Click on Next button
17. Select the Ragas evaluation, embedding model and Inferencing model
18. Click on Submit

Expected Result: DeepSeek-R1-Distill-Qwen-1.5B & DeepSeek-R1-Distill-Qwen-7B models should be executed successfully and the user should be able to see the generated answers for all the questions
Actual Result: DeepSeek-R1-Distill-Qwen-1.5B & DeepSeek-R1-Distill-Qwen-7B models are getting Failed and user unable see the generated answers for all the questions.

Browser and Version

Chrome Version 133.0.6943.54

Runtime Version

python 3.12

Error Messages

Additional Context

No response