dillenberg/astra-vision
1
0
Fork 0
mirror of https://github.com/Jeuners/astra-vision.git synced 2026-09-09 15:02:35 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions

feat: image generation via ComfyUI and PDF/image upload

Browse source 
Zwei neue, sauber getrennte Fähigkeiten, jede in ihrem eigenen Modul:

- astra/comfyui.py: async HTTP-Client für einen lokalen ComfyUI-Server
  (z-image-turbo-Workflow), kennt nichts von der Pipeline.
- astra/documents.py: PDF-Textextraktion via pypdf, keine Netzwerkzugriffe.
- astra/tools.py: verdrahtet generate_image als natives Ollama-Tool-Call —
  Qwen 3.5 unterstützt Tools und Vision bereits nativ laut `ollama show`.

Dafür wurde astra/services.py so erweitert, dass NativeOllamaService
Ollamas native tool_calls im Streaming-Response erkennt und als
ChatCompletionChunk-Deltas an Pipecats bereits vorhandene, generische
Function-Calling-Maschinerie (_process_context/run_function_calls)
weiterreicht — die musste dafür nicht angefasst werden. trim_messages in
core.py bewahrt jetzt Tool-Roundtrips und Bild-Anhänge vollständig statt
sie auf role/content zu reduzieren.

Neuer Upload-Button im UI (Bild oder PDF, während eines laufenden
Gesprächs): PDFs gehen als Text, Bilder als Base64 über Qwens Vision in
den Gesprächskontext ein. Generierte Bilder werden über /api/media/<id>
ausgeliefert und per Datenkanal im Transkript angezeigt.

Kompletter Function-Calling-Roundtrip end-to-end gegen echtes Ollama und
echtes ComfyUI verifiziert (Modell ruft generate_image korrekt auf,
Bild wird erzeugt und im media_store abgelegt).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LVgSHNHdRx3UNTBodFmhRA
This commit is contained in:
Jeuner 2026-09-07 15:56:18 +02:00
parent 991f0d1eb5
commit 16786fd3fd
18 changed files with 2602 additions and 21 deletions
Download patch file Download diff file Expand all files Collapse all files

22
README.md
View file

@ -23,6 +23,27 @@ Die Stimme lässt sich im UI per Dropdown wählen (`/api/voices` listet alle
26, Auswahl wird im Browser gemerkt). Jede Stimme wird beim ersten Gebrauch
lazy geladen und danach für die Laufzeit des Prozesses gecacht.
## Bilder erzeugen und Dokumente lesen
Zwei zusätzliche, sauber getrennte Fähigkeiten, unabhängig von STT/LLM/TTS:
- **`astra/comfyui.py`** — reiner async HTTP-Client für einen lokalen
[ComfyUI](https://github.com/comfyanonymous/ComfyUI)-Server
(`z-image-turbo`-Workflow). Kennt nichts von Pipecat.
- **`astra/documents.py`** — PDF-Textextraktion (`pypdf`), keine
Netzwerkzugriffe.
- **`astra/tools.py`** — verdrahtet `generate_image` als natives
Ollama-Tool. Qwen 3.5 entscheidet selbst, wann es aufgerufen wird
(`ollama show qwen3.5` listet `tools` als unterstützte Fähigkeit); das
generierte Bild landet im laufenden Gespräch als `/api/media/<id>` und
wird per WebRTC-Datenkanal ans UI gemeldet.
Bilder (PNG/JPEG/WebP) und PDFs lassen sich während eines laufenden
Gesprächs über den Button „Bild oder PDF hinzufügen“ hochladen
(`POST /api/upload`). Ein PDF wird als Text in den Gesprächskontext
eingefügt, ein Bild als Base64 mit Qwens nativer Vision-Fähigkeit — beides
nur für die Dauer der Session, nichts wird auf Disk geschrieben.
## Starten
```bash
@ -47,6 +68,7 @@ gezogen sein. Die Seite öffnen, Mikrofon erlauben, sprechen.
| `ASTRA_VOICE` | `alba` |
| `ASTRA_PORT` | `7860` |
| `ASTRA_TAILNET_HOST` | *(leer)* — z. B. `minim4-1.tail0f2cb2.ts.net` |
| `ASTRA_COMFYUI_URL` | `http://100.125.107.123:8000` |
## Tests

65
astra/comfyui.py Normal file
View file

@ -0,0 +1,65 @@
"""Async client for a local ComfyUI server. Knows nothing about the voice pipeline."""
import asyncio
import json
import random
import time
from pathlib import Path
import httpx
WORKFLOW_PATH = Path(__file__).resolve().parent / "comfyui_workflow.json"
SAVE_NODE = "9"
PROMPT_NODE = "57:27"
SIZE_NODE = "57:13"
SEED_NODE = "57:3"
class ComfyUIError(Exception):
"""Raised when ComfyUI is unreachable or fails to produce an image."""
async def generate_image(
endpoint: str,
prompt: str,
*,
width: int = 1024,
height: int = 1024,
timeout: float = 120.0,
) -> bytes:
"""Generate one PNG via the z-image-turbo workflow and return its raw bytes."""
workflow = json.loads(WORKFLOW_PATH.read_text())
workflow[PROMPT_NODE]["inputs"]["text"] = prompt
workflow[SIZE_NODE]["inputs"]["width"] = width
workflow[SIZE_NODE]["inputs"]["height"] = height
workflow[SEED_NODE]["inputs"]["seed"] = random.randint(0, 2**32 - 1)
async with httpx.AsyncClient(timeout=timeout) as client:
try:
response = await client.post(f"{endpoint}/prompt", json={"prompt": workflow})
response.raise_for_status()
except httpx.HTTPError as exc:
raise ComfyUIError(f"ComfyUI ist nicht erreichbar: {exc}") from exc
prompt_id = response.json()["prompt_id"]
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
history = await client.get(f"{endpoint}/history/{prompt_id}")
history.raise_for_status()
data = history.json().get(prompt_id)
if data:
images = data.get("outputs", {}).get(SAVE_NODE, {}).get("images", [])
if images:
image = images[0]
view = await client.get(
f"{endpoint}/view",
params={
"filename": image["filename"],
"subfolder": image["subfolder"],
"type": image["type"],
},
)
view.raise_for_status()
return view.content
await asyncio.sleep(1)
raise ComfyUIError("Zeitüberschreitung beim Warten auf das generierte Bild.")

59
astra/comfyui_workflow.json Normal file
View file

@ -0,0 +1,59 @@
{
"9": {
"inputs": { "filename_prefix": "z+krea2", "images": ["57:8", 0] },
"class_type": "SaveImage"
},
"57:30": {
"inputs": { "clip_name": "qwen_3_4b.safetensors", "type": "lumina2", "device": "default" },
"class_type": "CLIPLoader"
},
"57:29": {
"inputs": { "vae_name": "ae.safetensors" },
"class_type": "VAELoader"
},
"57:33": {
"inputs": { "conditioning": ["57:27", 0] },
"class_type": "ConditioningZeroOut"
},
"57:8": {
"inputs": { "samples": ["57:3", 0], "vae": ["57:29", 0] },
"class_type": "VAEDecode"
},
"57:28": {
"inputs": { "unet_name": "z_image_turbo_bf16.safetensors", "weight_dtype": "default" },
"class_type": "UNETLoader"
},
"57:51": {
"inputs": {
"model": ["57:28", 0],
"clip": ["57:30", 0],
"lora_name": "krea2_turbo_lora_rank_64_bf16.safetensors",
"strength_model": 0.85,
"strength_clip": 0.85
},
"class_type": "LoraLoader"
},
"57:27": {
"inputs": { "text": "__PROMPT__", "clip": ["57:51", 1] },
"class_type": "CLIPTextEncode"
},
"57:13": {
"inputs": { "width": 1024, "height": 1024, "batch_size": 1 },
"class_type": "EmptySD3LatentImage"
},
"57:11": {
"inputs": { "shift": 3, "model": ["57:51", 0] },
"class_type": "ModelSamplingAuraFlow"
},
"57:3": {
"inputs": {
"seed": 0, "steps": 8, "cfg": 1,
"sampler_name": "res_multistep", "scheduler": "simple", "denoise": 1,
"model": ["57:11", 0],
"positive": ["57:27", 0],
"negative": ["57:33", 0],
"latent_image": ["57:13", 0]
},
"class_type": "KSampler"
}
}

30
astra/core.py
View file

@ -14,6 +14,7 @@ class Settings:
port: int = 7860
context_tokens: int = 4096
tailnet_host: str | None = None
comfyui_url: str = "http://100.125.107.123:8000"
@classmethod
def from_env(cls):
@ -25,6 +26,7 @@ class Settings:
voice=os.getenv("ASTRA_VOICE", cls.voice),
port=int(os.getenv("ASTRA_PORT", cls.port)),
tailnet_host=os.getenv("ASTRA_TAILNET_HOST", cls.tailnet_host),
comfyui_url=os.getenv("ASTRA_COMFYUI_URL", cls.comfyui_url).rstrip("/"),
)
@ -64,30 +66,40 @@ SYSTEM_PROMPT = (
"Antworte natürlich und knapp, normalerweise in ein bis drei kurzen Sätzen. "
"Deine Antwort wird vorgelesen: kein Markdown, keine Sternchen, keine Listen. "
"Sprich Zahlen und Abkürzungen verständlich aus. Stelle bei Bedarf eine kurze Rückfrage. "
"Du hast keine Werkzeuge, keinen Internetzugang und keinen Zugriff auf Dateien oder Apps. "
"Behaupte nicht, Aktionen ausgeführt zu haben."
"Du hast keinen Internetzugang und keinen Zugriff auf Dateien oder Apps, außer den dir "
"explizit gegebenen Werkzeugen. Behaupte nicht, andere Aktionen ausgeführt zu haben. "
"Wenn der Nutzer ein Bild, eine Grafik oder eine Illustration möchte, rufe sofort das "
"Werkzeug generate_image auf, statt das Bild nur in Worten zu beschreiben. "
"Wenn der Nutzer ein Bild, ein Dokument oder ein PDF hochlädt, geht dessen Inhalt oder "
"eine Textzusammenfassung als Nachricht in dieses Gespräch ein."
)
def trim_messages(messages: list[dict], max_chars: int = 10000) -> list[dict]:
"""Retain recent whole turns within a conservative context character budget."""
"""Retain recent whole turns within a conservative context character budget.
Preserves tool round-trips (assistant tool_calls + matching tool result)
and vision attachments (an "images" field) intact instead of reducing
every kept message down to a bare role/content pair.
"""
system = [dict(m) for m in messages if m["role"] == "system"][:1]
if system:
system[0]["content"] = system[0]["content"][: max_chars // 2]
budget = max_chars - sum(len(m["content"]) for m in system)
turns = []
for message in reversed(messages):
if message["role"] not in ("user", "assistant"):
if message["role"] not in ("user", "assistant", "tool"):
continue
content = message.get("content")
if not isinstance(content, str) or not content:
text = content if isinstance(content, str) else ""
if not text and not message.get("tool_calls") and not message.get("images"):
continue
if len(content) > budget:
if len(text) > budget:
if not turns:
turns.append({"role": message["role"], "content": content[-budget:]})
turns.append({**message, "content": text[-budget:]})
break
turns.append({"role": message["role"], "content": content})
budget -= len(content)
turns.append(message)
budget -= len(text)
turns.reverse()
while turns and turns[0]["role"] != "user":
turns.pop(0)

13
astra/documents.py Normal file
View file

@ -0,0 +1,13 @@
"""Local document ingestion: PDF text extraction. No network access, no persistence."""
from io import BytesIO
from pypdf import PdfReader
def extract_pdf_text(data: bytes, max_chars: int = 8000) -> str:
"""Extract and concatenate text from every page of a PDF, capped to max_chars."""
reader = PdfReader(BytesIO(data))
pages = (page.extract_text() or "" for page in reader.pages)
text = "\n\n".join(page.strip() for page in pages if page.strip())
return text[:max_chars]

68
astra/server.py
View file

@ -1,6 +1,7 @@
"""Loopback-only WebRTC voice app. One live session shares the warm models."""
import asyncio
import base64
import contextlib
import os
import sys
@ -11,8 +12,8 @@ from typing import Literal
import httpx
import uvicorn
from fastapi import FastAPI, HTTPException, Request
from fastapi.responses import FileResponse, JSONResponse
from fastapi import FastAPI, File, Form, HTTPException, Request, UploadFile
from fastapi.responses import FileResponse, JSONResponse, Response
from fastapi.staticfiles import StaticFiles
from loguru import logger
from pydantic import BaseModel, Field, field_validator
@ -25,9 +26,11 @@ from astra.core import (
build_request,
local_origin_allowed,
)
from astra.documents import extract_pdf_text
from astra.inference import Models, on_executor
ROOT = Path(__file__).resolve().parent.parent
MAX_UPLOAD_BYTES = 15 * 1024 * 1024
class Offer(BaseModel):
@ -47,7 +50,7 @@ class Disconnect(BaseModel):
pc_id: str = Field(max_length=100)
async def run_voice(connection, models, config, voice_state, voice_name):
async def run_voice(connection, models, config, voice_state, voice_name, context):
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.audio.vad.vad_analyzer import VADParams
from pipecat.frames.frames import (
@ -60,7 +63,6 @@ async def run_voice(connection, models, config, voice_state, voice_name):
from pipecat.observers.base_observer import BaseObserver
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.worker import PipelineParams, PipelineWorker
from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.processors.aggregators.llm_response_universal import (
LLMContextAggregatorPair,
LLMUserAggregatorParams,
@ -106,7 +108,6 @@ async def run_voice(connection, models, config, voice_state, voice_name):
audio_out_sample_rate=24000,
),
)
context = LLMContext([{"role": "system", "content": SYSTEM_PROMPT}])
aggregators = LLMContextAggregatorPair(
context,
user_params=LLMUserAggregatorParams(
@ -181,6 +182,7 @@ def create_app(config=None, *, load_models=True):
models = Models(config)
status = {"ready": False, "stage": "starting", "error": None}
sessions = {}
media_store: dict[str, bytes] = {}
lock = asyncio.Lock()
async def warmup():
@ -241,7 +243,7 @@ def create_app(config=None, *, load_models=True):
warming.cancel()
with contextlib.suppress(asyncio.CancelledError):
await warming
for connection, task in list(sessions.values()):
for connection, task, _ in list(sessions.values()):
await connection.disconnect()
task.cancel()
with contextlib.suppress(asyncio.CancelledError):
@ -295,8 +297,11 @@ def create_app(config=None, *, load_models=True):
)
voice_name = body.voice or config.voice
voice_state = await on_executor(models.tts_executor, models.get_voice, voice_name)
from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.transports.smallwebrtc.connection import SmallWebRTCConnection
from astra.tools import build_tools
connection = SmallWebRTCConnection(ice_servers=[], connection_timeout_secs=20)
try:
await connection.initialize(body.sdp, body.type)
@ -304,9 +309,15 @@ def create_app(config=None, *, load_models=True):
await connection.disconnect()
raise
notify = connection.send_app_message
context = LLMContext(
[{"role": "system", "content": SYSTEM_PROMPT}],
tools=build_tools(config, notify, media_store),
)
async def session():
try:
await run_voice(connection, models, config, voice_state, voice_name)
await run_voice(connection, models, config, voice_state, voice_name, context)
except asyncio.CancelledError:
raise
except Exception as exc:
@ -317,20 +328,59 @@ def create_app(config=None, *, load_models=True):
sessions.pop(connection.pc_id, None)
task = asyncio.create_task(session())
sessions[connection.pc_id] = (connection, task)
sessions[connection.pc_id] = (connection, task, context)
return connection.get_answer()
@app.post("/api/disconnect")
async def disconnect(body: Disconnect):
pair = sessions.get(body.pc_id)
if pair:
connection, task = pair
connection, task, _ = pair
await connection.disconnect()
task.cancel()
with contextlib.suppress(asyncio.CancelledError):
await task
return {"ok": True}
@app.get("/api/media/{image_id}")
async def media(image_id: str):
image = media_store.get(image_id)
if image is None:
raise HTTPException(404, "Bild nicht gefunden")
return Response(content=image, media_type="image/png")
@app.post("/api/upload")
async def upload(pc_id: str = Form(...), file: UploadFile = File(...)): # noqa: B008
pair = sessions.get(pc_id)
if not pair:
raise HTTPException(404, "Keine aktive Sitzung")
connection, _, context = pair
data = await file.read()
if len(data) > MAX_UPLOAD_BYTES:
raise HTTPException(413, "Datei zu groß (max. 15 MB)")
if file.content_type == "application/pdf":
text = extract_pdf_text(data)
if not text:
raise HTTPException(422, "Im PDF wurde kein Text gefunden")
context.add_message(
{
"role": "user",
"content": f'[Hochgeladenes Dokument "{file.filename}"]\n\n{text}',
}
)
elif file.content_type in {"image/png", "image/jpeg", "image/webp"}:
context.add_message(
{
"role": "user",
"content": f'[Hochgeladenes Bild "{file.filename}"]',
"images": [base64.b64encode(data).decode()],
}
)
else:
raise HTTPException(415, "Nur PDF, PNG, JPEG oder WebP werden unterstützt")
connection.send_app_message({"type": "upload", "filename": file.filename})
return {"ok": True}
app.mount("/static", StaticFiles(directory=ROOT / "web"), name="static")
return app

37
astra/services.py
View file

@ -4,6 +4,7 @@ import asyncio
import json
import time
from contextlib import aclosing
from uuid import uuid4
import httpx
from loguru import logger
@ -20,12 +21,14 @@ from pipecat.frames.frames import (
VADUserStartedSpeakingFrame,
VADUserStoppedSpeakingFrame,
)
from pipecat.processors.aggregators.llm_context import LLMContext
from pipecat.processors.frame_processor import FrameDirection
from pipecat.services.openai.llm import OpenAILLMService
from pipecat.services.settings import STTSettings, TTSSettings
from pipecat.services.stt_service import STTService
from pipecat.services.tts_service import TTSService
from pipecat.utils.time import time_now_iso8601
from pipecat.utils.types import is_given
from astra.core import Settings, build_request
from astra.inference import Models, Recognizer, drain_stream, on_executor
@ -45,9 +48,12 @@ class NativeOllamaService(OpenAILLMService):
self.config = config
self.notify = notify
async def get_chat_completions(self, context):
async def get_chat_completions(self, context: LLMContext):
payload = build_request(self.config, context.get_messages())
context.set_messages(payload["messages"])
tools = self.get_llm_adapter().from_standard_tools(context.tools)
if is_given(tools) and tools:
payload["tools"] = list(tools)
async def chunks():
start = time.monotonic()
@ -67,6 +73,35 @@ class NativeOllamaService(OpenAILLMService):
message = event.get("message", {})
if message.get("thinking"):
raise RuntimeError("Ollama liefert Thinking trotz think=false.")
for tool_index, call in enumerate(message.get("tool_calls") or []):
function = call.get("function", {})
yield ChatCompletionChunk(
id="local",
object="chat.completion.chunk",
created=int(time.time()),
model=self.config.model,
choices=[
{
"index": 0,
"delta": {
"tool_calls": [
{
"index": tool_index,
"id": f"call_{uuid4().hex}",
"type": "function",
"function": {
"name": function.get("name", ""),
"arguments": json.dumps(
function.get("arguments") or {}
),
},
}
]
},
"finish_reason": None,
}
],
)
content = message.get("content", "")
if content:
if first:

52
astra/tools.py Normal file
View file

@ -0,0 +1,52 @@
"""LLM-callable tools. Each tool owns its handler; server.py only wires in
per-session dependencies (notify, media_store) and hands the result to pipecat.
"""
import uuid
from collections.abc import Callable
from pipecat.adapters.schemas.function_schema import FunctionSchema
from pipecat.adapters.schemas.tools_schema import ToolsSchema
from astra.comfyui import ComfyUIError, generate_image
from astra.core import Settings
def build_tools(config: Settings, notify: Callable[[dict], None], media_store: dict[str, bytes]) -> ToolsSchema:
"""Assemble the tool set available to the LLM for one session."""
async def handle_generate_image(params):
prompt = params.arguments.get("prompt", "")
try:
image = await generate_image(config.comfyui_url, prompt)
except ComfyUIError as exc:
await params.result_callback({"error": str(exc)})
return
image_id = uuid.uuid4().hex
media_store[image_id] = image
notify({"type": "image", "url": f"/api/media/{image_id}"})
await params.result_callback(
{"status": "ok", "message": "Bild wurde erzeugt und dem Nutzer angezeigt."}
)
generate_image_schema = FunctionSchema(
name="generate_image",
description=(
"Erzeuge ein Bild anhand einer Beschreibung und zeige es dem Nutzer an. "
"Nutze dieses Werkzeug, wenn der Nutzer explizit ein Bild, eine Grafik "
"oder eine Illustration wünscht."
),
properties={
"prompt": {
"type": "string",
"description": (
"Konkrete, szenenorientierte Bildbeschreibung auf Englisch, "
"mit Hinweisen zu Licht und Kamera, z.B. "
"'close-up, cinematic light, warm tones'."
),
},
},
required=["prompt"],
handler=handle_generate_image,
)
return ToolsSchema(standard_tools=[generate_image_schema])

2
pyproject.toml
View file

@ -9,6 +9,8 @@ dependencies = [
"fastapi",
"uvicorn",
"httpx",
"pypdf",
"python-multipart",
]
[dependency-groups]

69
tests/test_comfyui.py Normal file
View file

@ -0,0 +1,69 @@
import httpx
import pytest
from astra.comfyui import ComfyUIError, generate_image
def _patched_client(monkeypatch, handler):
real_async_client = httpx.AsyncClient
def factory(*args, **kwargs):
kwargs["transport"] = httpx.MockTransport(handler)
return real_async_client(**kwargs)
monkeypatch.setattr(httpx, "AsyncClient", factory)
@pytest.mark.asyncio
async def test_generate_image_returns_bytes_from_completed_job(monkeypatch):
png_bytes = b"\x89PNG\r\n\x1a\nfake-image-bytes"
def handler(request: httpx.Request) -> httpx.Response:
if request.url.path == "/prompt":
return httpx.Response(200, json={"prompt_id": "abc123"})
if request.url.path == "/history/abc123":
return httpx.Response(
200,
json={
"abc123": {
"outputs": {
"9": {
"images": [
{"filename": "out.png", "subfolder": "", "type": "output"}
]
}
}
}
},
)
if request.url.path == "/view":
return httpx.Response(200, content=png_bytes)
raise AssertionError(f"unexpected request: {request.url}")
_patched_client(monkeypatch, handler)
result = await generate_image("http://fake-comfyui", "a cat")
assert result == png_bytes
@pytest.mark.asyncio
async def test_generate_image_raises_when_comfyui_unreachable(monkeypatch):
def handler(request: httpx.Request) -> httpx.Response:
raise httpx.ConnectError("refused", request=request)
_patched_client(monkeypatch, handler)
with pytest.raises(ComfyUIError):
await generate_image("http://fake-comfyui", "a cat")
@pytest.mark.asyncio
async def test_generate_image_times_out_if_job_never_completes(monkeypatch):
def handler(request: httpx.Request) -> httpx.Response:
if request.url.path == "/prompt":
return httpx.Response(200, json={"prompt_id": "abc123"})
if request.url.path == "/history/abc123":
return httpx.Response(200, json={})
raise AssertionError(f"unexpected request: {request.url}")
_patched_client(monkeypatch, handler)
with pytest.raises(ComfyUIError, match="Zeitüberschreitung"):
await generate_image("http://fake-comfyui", "a cat", timeout=0.2)

30
tests/test_core.py
View file

@ -46,6 +46,36 @@ class CoreTests(unittest.TestCase):
)
self.assertLessEqual(sum(len(m["content"]) for m in trimmed), 5000)
def test_tool_round_trip_survives_trimming(self):
messages = [
{"role": "system", "content": "Deutsch"},
{"role": "user", "content": "Zeig mir ein Bild von einer Katze."},
{
"role": "assistant",
"content": "",
"tool_calls": [{"function": {"name": "generate_image", "arguments": {}}}],
},
{"role": "tool", "content": '{"status": "ok"}', "tool_call_id": "call_1"},
{"role": "assistant", "content": "Fertig, schau mal!"},
]
trimmed = trim_messages(messages, max_chars=5000)
roles = [m["role"] for m in trimmed]
self.assertEqual(roles, ["system", "user", "assistant", "tool", "assistant"])
self.assertEqual(trimmed[2]["tool_calls"][0]["function"]["name"], "generate_image")
self.assertEqual(trimmed[3]["tool_call_id"], "call_1")
def test_image_attachment_survives_trimming(self):
messages = [
{"role": "system", "content": "Deutsch"},
{
"role": "user",
"content": '[Hochgeladenes Bild "katze.png"]',
"images": ["base64data"],
},
]
trimmed = trim_messages(messages, max_chars=5000)
self.assertEqual(trimmed[1]["images"], ["base64data"])
if __name__ == "__main__":
unittest.main()

27
tests/test_documents.py Normal file
View file

@ -0,0 +1,27 @@
from astra.documents import extract_pdf_text
# Minimal hand-built single-page PDF with a text-showing operator. pypdf
# recovers it via its xref-scanning fallback despite the bogus startxref
# offset, so no external PDF-writing dependency is needed for the fixture.
MINIMAL_PDF = b"""%PDF-1.1
1 0 obj<</Type/Catalog/Pages 2 0 R>>endobj
2 0 obj<</Type/Pages/Kids[3 0 R]/Count 1>>endobj
3 0 obj<</Type/Page/Parent 2 0 R/Resources<</Font<</F1 4 0 R>>>>/MediaBox[0 0 200 200]/Contents 5 0 R>>endobj
4 0 obj<</Type/Font/Subtype/Type1/BaseFont/Helvetica>>endobj
5 0 obj<</Length 44>>stream
BT /F1 24 Tf 20 100 Td (Hallo Astra) Tj ET
endstream
endobj
trailer<</Size 6/Root 1 0 R>>
startxref
0
%%EOF"""
def test_extract_pdf_text_reads_page_content():
assert extract_pdf_text(MINIMAL_PDF) == "Hallo Astra"
def test_extract_pdf_text_respects_max_chars():
text = extract_pdf_text(MINIMAL_PDF, max_chars=5)
assert text == "Hallo"

17
tests/test_server.py
View file

@ -55,6 +55,23 @@ def test_offer_rejects_unknown_voice():
assert response.status_code == 422
def test_media_endpoint_returns_404_for_unknown_image():
with TestClient(create_app(load_models=False), base_url="http://localhost:7860") as client:
response = client.get("/api/media/does-not-exist")
assert response.status_code == 404
def test_upload_requires_an_active_session():
with TestClient(create_app(load_models=False), base_url="http://localhost:7860") as client:
response = client.post(
"/api/upload",
headers={"Origin": "http://localhost:7860"},
data={"pc_id": "no-such-session"},
files={"file": ("test.pdf", b"%PDF-1.1", "application/pdf")},
)
assert response.status_code == 404
def test_disconnect_is_idempotent_and_host_is_checked():
with TestClient(create_app(load_models=False), base_url="http://localhost:7860") as client:
response = client.post(

60
tests/test_tools.py Normal file
View file

@ -0,0 +1,60 @@
import pytest
import astra.tools as tools_module
from astra.comfyui import ComfyUIError
from astra.core import Settings
class FakeFunctionCallParams:
"""Stand-in for pipecat's FunctionCallParams: only what the handler touches."""
def __init__(self, arguments):
self.arguments = arguments
self.results = []
async def result_callback(self, result, *, properties=None):
self.results.append(result)
@pytest.mark.asyncio
async def test_generate_image_tool_stores_media_and_notifies(monkeypatch):
async def fake_generate_image(endpoint, prompt, **kwargs):
assert endpoint == "http://fake-comfyui"
assert prompt == "a cat"
return b"png-bytes"
monkeypatch.setattr(tools_module, "generate_image", fake_generate_image)
notifications = []
media_store: dict[str, bytes] = {}
schema = tools_module.build_tools(
Settings(comfyui_url="http://fake-comfyui"), notifications.append, media_store
).standard_tools[0]
params = FakeFunctionCallParams({"prompt": "a cat"})
await schema.handler(params)
assert len(media_store) == 1
image_id, image_bytes = next(iter(media_store.items()))
assert image_bytes == b"png-bytes"
assert notifications == [{"type": "image", "url": f"/api/media/{image_id}"}]
assert params.results[0]["status"] == "ok"
@pytest.mark.asyncio
async def test_generate_image_tool_reports_comfyui_errors_without_storing_media(monkeypatch):
async def failing_generate_image(endpoint, prompt, **kwargs):
raise ComfyUIError("ComfyUI ist nicht erreichbar.")
monkeypatch.setattr(tools_module, "generate_image", failing_generate_image)
media_store: dict[str, bytes] = {}
schema = tools_module.build_tools(
Settings(comfyui_url="http://fake-comfyui"), lambda _: None, media_store
).standard_tools[0]
params = FakeFunctionCallParams({"prompt": "a cat"})
await schema.handler(params)
assert media_store == {}
assert "error" in params.results[0]

2017
uv.lock generated Normal file
View file

File diff suppressed because it is too large Load diff

51
web/app.js
View file

@ -33,17 +33,60 @@ function addMessage(event) {
while ($("messages").children.length > 80) $("messages").firstElementChild.remove();
$("messages").scrollTop = $("messages").scrollHeight;
}
function addImage(url) {
$("messages").querySelector(".empty")?.remove();
const article = document.createElement("article");
article.className = "message assistant";
const speaker = document.createElement("span");
speaker.className = "speaker";
speaker.textContent = "Astra";
const img = document.createElement("img");
img.className = "generated-image";
img.src = url;
img.alt = "Von Astra erzeugtes Bild";
article.append(speaker, img);
$("messages").append(article);
$("messages").scrollTop = $("messages").scrollHeight;
}
function addUploadNote(filename) {
$("messages").querySelector(".empty")?.remove();
const article = document.createElement("article");
article.className = "message user";
const speaker = document.createElement("span");
speaker.className = "speaker";
speaker.textContent = "Du";
const text = document.createElement("p");
text.textContent = `Hochgeladen: ${filename}`;
article.append(speaker, text);
$("messages").append(article);
$("messages").scrollTop = $("messages").scrollHeight;
}
function receive(event) {
let message;
try { message = JSON.parse(event.data); } catch { return; }
if (message.type === "state") state(message.state);
if (message.type === "partial") $("partial").textContent = message.text;
if (message.type === "transcript") addMessage(message);
if (message.type === "image") addImage(message.url);
if (message.type === "upload") addUploadNote(message.filename);
if (message.type === "error") showError(message.message);
if (message.type === "metric" && message.name === "llm_ms") {
$("latency").textContent = `Erstes Antwortwort · ${(message.value / 1000).toFixed(2)} s`;
}
}
async function uploadFile(file) {
if (!pcId) return;
const body = new FormData();
body.append("pc_id", pcId);
body.append("file", file);
try {
const response = await fetch("/api/upload", {method: "POST", body, signal: AbortSignal.timeout(30000)});
const data = await response.json();
if (!response.ok) throw new Error(typeof data.detail === "string" ? data.detail : "Upload fehlgeschlagen.");
} catch (error) {
showError(error.message || "Upload fehlgeschlagen.");
}
}
async function request(path, body, timeout = 20000) {
const response = await fetch(path, {method: "POST", headers: {"Content-Type": "application/json"},
body: JSON.stringify(body), signal: AbortSignal.timeout(timeout)});
@ -121,6 +164,7 @@ async function connect() {
await pc.setRemoteDescription({sdp: answer.sdp, type: answer.type});
$("connect").textContent = "Gespräch beenden";
$("mute").hidden = false;
$("attach").hidden = false;
$("clear").textContent = "Neues Gespräch";
$("hint").textContent = "Sprich frei. Beim Dazwischenreden hält Astra an.";
} catch (error) {
@ -155,6 +199,7 @@ async function disconnect() {
$("mute").hidden = true;
$("mute").setAttribute("aria-pressed", "false");
$("mute").textContent = "Mikrofon pausieren";
$("attach").hidden = true;
$("connect").textContent = "Gespräch starten";
$("clear").textContent = "Verlauf leeren";
$("partial").textContent = "";
@ -172,6 +217,12 @@ $("mute").addEventListener("click", () => {
$("mute").textContent = muted ? "Mikrofon aktivieren" : "Mikrofon pausieren";
state("listening");
});
$("attach").addEventListener("click", () => $("file-input").click());
$("file-input").addEventListener("change", async () => {
const file = $("file-input").files[0];
$("file-input").value = "";
if (file) await uploadFile(file);
});
$("clear").addEventListener("click", async () => {
const reconnect = Boolean(peer);
if (reconnect) await disconnect();

2
web/index.html
View file

@ -22,7 +22,7 @@
<label for="voice">Stimme</label>
<select id="voice" disabled></select>
</div>
<div class="controls"><button id="connect" class="primary" disabled><span aria-hidden="true">◉</span> Gespräch starten</button><button id="mute" class="secondary" hidden aria-pressed="false">Mikrofon pausieren</button></div>
<div class="controls"><button id="connect" class="primary" disabled><span aria-hidden="true">◉</span> Gespräch starten</button><button id="mute" class="secondary" hidden aria-pressed="false">Mikrofon pausieren</button><button id="attach" class="secondary" hidden>Bild oder PDF hinzufügen</button><input id="file-input" type="file" accept="application/pdf,image/png,image/jpeg,image/webp" hidden></div>
<p class="hint" id="hint">Mikrofon wird erst nach dem Start freigegeben.</p>
</section>
<section class="transcript" aria-labelledby="transcript-title">

2
web/style.css
View file

File diff suppressed because one or more lines are too long