OllamaLLMUnit model running improperly (ie, too big model, too small gpu) should error gracefully with helpful error message

[atljoseph]

Minor Issue:

No matter what the user chooses for the dropdown of model selection, the first message always uses the default model...Even when the user specifically chose the non-default one. After that, it respects the user's choice.

---

MAIN ISSUE:

OllamaLLMUnit fails to converse, but Sonnet just fine.

• The error is totally unrecoverable.

INFO - executing ollama llm request
agent-server-1  | ERROR - Attempted to call a sync iterator on an async stream.
agent-server-1  | Traceback (most recent call last):
agent-server-1  |   File "/usr/local/lib/python3.11/site-packages/ollama/_client.py", line 364, in inner
agent-server-1  |     r.raise_for_status()
agent-server-1  |   File "/usr/local/lib/python3.11/site-packages/httpx/_models.py", line 763, in raise_for_status
agent-server-1  |     raise HTTPStatusError(message, request=request, response=self)
agent-server-1  | httpx.HTTPStatusError: Client error '404 Not Found' for url 'http://192.168.50.44:11434/api/chat'
agent-server-1  | For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404
agent-server-1  | 
agent-server-1  | During handling of the above exception, another exception occurred:
agent-server-1  | 
agent-server-1  | Traceback (most recent call last):
agent-server-1  |   File "/usr/local/lib/python3.11/site-packages/eidolon_ai_sdk/apu/conversational_apu.py", line 148, in schedule_request
agent-server-1  |     async for event in self._llm_execution_cycle(call_context, output_format, conversation):
agent-server-1  |   File "/usr/local/lib/python3.11/site-packages/eidolon_ai_sdk/apu/conversational_apu.py", line 199, in _llm_execution_cycle
agent-server-1  |     async for event in stream_collector:
agent-server-1  |   File "/usr/local/lib/python3.11/site-packages/eidolon_ai_sdk/util/stream_collector.py", line 56, in __anext__
agent-server-1  |     event = await self.stream.__anext__()
agent-server-1  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
agent-server-1  |   File "/usr/local/lib/python3.11/site-packages/eidolon_ai_sdk/apu/llm/ollama_llm_unit.py", line 84, in execute_llm
agent-server-1  |     async for m_chunk in llm_request(**request):
agent-server-1  |   File "/usr/local/lib/python3.11/site-packages/eidolon_ai_sdk/util/replay.py", line 93, in wrapper_async_gen
agent-server-1  |     async for e in fn(*args, **kwargs):
agent-server-1  |   File "/usr/local/lib/python3.11/site-packages/eidolon_ai_sdk/apu/llm/ollama_llm_unit.py", line 143, in fn
agent-server-1  |     async for e in await client.chat(**kwargs, stream=True):
agent-server-1  |   File "/usr/local/lib/python3.11/site-packages/ollama/_client.py", line 366, in inner
agent-server-1  |     e.response.read()
agent-server-1  |   File "/usr/local/lib/python3.11/site-packages/httpx/_models.py", line 815, in read
agent-server-1  |     self._content = b"".join(self.iter_bytes())
agent-server-1  |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
agent-server-1  |   File "/usr/local/lib/python3.11/site-packages/httpx/_models.py", line 831, in iter_bytes
agent-server-1  |     for raw_bytes in self.iter_raw():
agent-server-1  |   File "/usr/local/lib/python3.11/site-packages/httpx/_models.py", line 878, in iter_raw
agent-server-1  |     raise RuntimeError("Attempted to call a sync iterator on an async stream.")
agent-server-1  | RuntimeError: Attempted to call a sync iterator on an async stream.
agent-server-1  | ERROR - Unhandled Error APU Error: RuntimeError while processing request
agent-server-1  | Traceback (most recent call last):
agent-server-1  |   File "/usr/local/lib/python3.11/site-packages/ollama/_client.py", line 364, in inner
agent-server-1  |     r.raise_for_status()
agent-server-1  |   File "/usr/local/lib/python3.11/site-packages/httpx/_models.py", line 763, in raise_for_status
agent-server-1  |     raise HTTPStatusError(message, request=request, response=self)
agent-server-1  | httpx.HTTPStatusError: Client error '404 Not Found' for url 'http://192.168.50.44:11434/api/chat'
agent-server-1  | For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/404
agent-server-1  | 
agent-server-1  | During handling of the above exception, another exception occurred:
agent-server-1  | 
agent-server-1  | Traceback (most recent call last):
agent-server-1  |   File "/usr/local/lib/python3.11/site-packages/eidolon_ai_sdk/apu/conversational_apu.py", line 148, in schedule_request
agent-server-1  |     async for event in self._llm_execution_cycle(call_context, output_format, conversation):
agent-server-1  |   File "/usr/local/lib/python3.11/site-packages/eidolon_ai_sdk/apu/conversational_apu.py", line 199, in _llm_execution_cycle
agent-server-1  |     async for event in stream_collector:
agent-server-1  |   File "/usr/local/lib/python3.11/site-packages/eidolon_ai_sdk/util/stream_collector.py", line 56, in __anext__
agent-server-1  |     event = await self.stream.__anext__()
agent-server-1  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
agent-server-1  |   File "/usr/local/lib/python3.11/site-packages/eidolon_ai_sdk/apu/llm/ollama_llm_unit.py", line 84, in execute_llm
agent-server-1  |     async for m_chunk in llm_request(**request):
agent-server-1  |   File "/usr/local/lib/python3.11/site-packages/eidolon_ai_sdk/util/replay.py", line 93, in wrapper_async_gen
agent-server-1  |     async for e in fn(*args, **kwargs):
agent-server-1  |   File "/usr/local/lib/python3.11/site-packages/eidolon_ai_sdk/apu/llm/ollama_llm_unit.py", line 143, in fn
agent-server-1  |     async for e in await client.chat(**kwargs, stream=True):
agent-server-1  |   File "/usr/local/lib/python3.11/site-packages/ollama/_client.py", line 366, in inner
agent-server-1  |     e.response.read()
agent-server-1  |   File "/usr/local/lib/python3.11/site-packages/httpx/_models.py", line 815, in read
agent-server-1  |     self._content = b"".join(self.iter_bytes())
agent-server-1  |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
agent-server-1  |   File "/usr/local/lib/python3.11/site-packages/httpx/_models.py", line 831, in iter_bytes
agent-server-1  |     for raw_bytes in self.iter_raw():
agent-server-1  |   File "/usr/local/lib/python3.11/site-packages/httpx/_models.py", line 878, in iter_raw
agent-server-1  |     raise RuntimeError("Attempted to call a sync iterator on an async stream.")
agent-server-1  | RuntimeError: Attempted to call a sync iterator on an async stream.
agent-server-1  | 
agent-server-1  | The above exception was the direct cause of the following exception:
agent-server-1  | 
agent-server-1  | Traceback (most recent call last):
agent-server-1  |   File "/usr/local/lib/python3.11/site-packages/eidolon_ai_sdk/system/agent_controller.py", line 352, in stream_agent_iterator
agent-server-1  |     async for event in stream:
agent-server-1  |   File "/usr/local/lib/python3.11/site-packages/eidolon_ai_sdk/agent/simple_agent.py", line 242, in fn
agent-server-1  |     async for e in action_fn(self, action, process_id, **kwargs):
agent-server-1  |   File "/usr/local/lib/python3.11/site-packages/eidolon_ai_sdk/agent/simple_agent.py", line 300, in _act
agent-server-1  |     async for event in response:
agent-server-1  |   File "/usr/local/lib/python3.11/site-packages/eidolon_ai_sdk/apu/conversational_apu.py", line 156, in schedule_request
agent-server-1  |     raise APUException(f"{e.__class__.__name__} while processing request") from e
agent-server-1  | eidolon_ai_sdk.apu.apu.APUException: APU Error: RuntimeError while processing request
webui-1         | getting process with id 6723273c6651c5713b32c55a from machine http://agent-server:8080
webui-1         | [auth][warn][env-url-basepath-redundant] Read more: https://warnings.authjs.dev#env-url-basepath-redundant
agent-server-1  | INFO - Request: GET http://agent-server:8080/processes/6723273c6651c5713b32c55a
agent-server-1  | INFO - Response: 200
agent-server-1  | INFO - Response: 200
webui-1         | [auth][warn][env-url-basepath-redundant] Read more: https://warnings.authjs.dev#env-url-basepath-redundant
agent-server-1  | INFO - Request: GET http://agent-server:8080/processes?skip=0&limit=100
agent-server-1  | INFO - Response: 200

---

Why won't this go to my local sdk?

[tool.poetry.dependencies]
python = ">=3.10,<3.13"
# eidolon-ai-sdk = "^0.1.153"
eidolon-ai-sdk = {path = "../eidolon/sdk"}
aiofiles = "^24.1.0"

I changed one of the async for event in stream instances, but nothing in the errors change, even after poetry update and poetry install.

---

.env

OLLAMA_HOST=http://192.168.50.44:11434

---

apu_ollama_gemma...

apiVersion: server.eidolonai.com/v1alpha1
kind: Reference
metadata:
  name: apu_ollama_gemma
  annotations:
    - title: "Gemma 2"
spec:
  implementation: ConversationalAPU
  llm_unit:
    implementation: OllamaLLMUnit
    model:
      implementation: llama3-8b
      name: gemma2:9b

---

chatbot...

apus:
    - apu: apu_claude
      title: "Claude Sonnet 3.5"
      default: True
    - apu: apu_ollama_gemma
      title: "Gemma 2"
      default: False

@LukeLalor

[atljoseph]

Ok. Got it to work on a smaller model. Turns out I had a vscode extension loading up one gpu with FIM completion model, and the model eidolon was attempting to use was too big for both gpus (with context size and all). So the cause of this is:

• Ollama server error about can’t load model isn’t handled properly.
• When this happens, the conversation is permanently, unrecoverably hosed , unless ya copy paste everything over.

[LukeLalor]

This error sucks, note for grooming I think even if this was a config error, it isn't presented well enough to easily self-diagnose. This should turn into a task to improve how we pass ollama errors to the user (or likely llm errors more generally)

Separate product question on what we want to do about conversation recovery more generally. Simple agent allows this to be configurable, but that is definitely not obvious, so we should reconsider the default behavior on that agent.

[LukeLalor]

Ok, more technical notes looking at the stack a bit more

This error is coming from within the ollama client library as it tries to parse the bad http status response. This appears to be a bug in the ollama client library that prevents it from producing the real error and instead gives us Attempted to call a sync iterator on an async stream

We need to reproduce this, see if upgrading ollama client fixes it, or catch this as a workaround locally + contribute the ollama client if it is still a bug on their side

[atljoseph]

It’s probably that out of cuda or system memory exception. I might be able to upgrade ollama tonight to see. Interesting that it is in the vendor lib yeah. Smells like old version and new server. The primary goal is to say a voice command and call the Machine agent in the background.

…

On Thu, Oct 31, 2024 at 3:33 PM Luke Lalor ***@***.***> wrote: Ok, more technical notes looking at the stack a bit more This error is coming from within the ollama client library as it tries to parse the bad http status response. This appears to be a bug in the ollama client library that prevents it from producing the real error and instead gives us Attempted to call a sync iterator on an async stream We need to reproduce this, see if upgrading ollama client fixes it, or patch + contribute the ollama client if it is still a bug on their side — Reply to this email directly, view it on GitHub <#12 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AF632FMLG6OTJ5FSIXGGHVLZ6KAZNAVCNFSM6AAAAABQ5VIGLSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINJQGY3TCMZQHA> . You are receiving this because you authored the thread.Message ID: ***@***.***>