commit 7c633a02ed71fa19eab1256eb18e224cf1d75d19 Author: ilia Date: Mon Jan 5 20:09:44 2026 -0500 Initialize project structure with essential files and documentation - Added .cursorrules for project guidelines and context - Created README.md for project overview and goals - Established ARCHITECTURE.md for architectural documentation - Set up tickets directory with initial ticket management files - Included .gitignore to manage ignored files and directories This commit lays the foundation for the Atlas project, ensuring a clear structure for development and collaboration. diff --git a/.cursorrules b/.cursorrules new file mode 100644 index 0000000..cb32b94 --- /dev/null +++ b/.cursorrules @@ -0,0 +1,126 @@ +# Cursor Rules for Atlas Project + +## Project Context + +Atlas is a local, privacy-focused home voice agent system with separate work and family agents running on dedicated hardware (RTX 4080 and RTX 1050). + +This project uses a kanban-based workflow (Vibe Kanban) for task management. All tickets are organized in the `tickets/` directory structure with 5 parallel tracks: +- Voice I/O (wake-word, ASR, TTS) +- LLM Infrastructure (4080 work agent, 1050 family agent) +- Tools/MCP (Model Context Protocol tools) +- Clients/UI (Phone PWA, web dashboard) +- Safety/Memory (boundaries, confirmations, long-term memory) + +## Code Style & Standards + +### General Principles +- Write clean, maintainable, and well-documented code +- Follow language-specific best practices and conventions +- Prioritize readability and maintainability over cleverness +- Use meaningful variable and function names +- Keep functions small and focused on a single responsibility + +### Code Organization +- Organize code into logical modules and components +- Separate concerns (business logic, data access, presentation) +- Use consistent file and folder naming conventions +- Group related functionality together + +### Documentation +- Document complex logic and algorithms +- Include docstrings/comments for public APIs +- Keep README and architecture docs up to date +- Document design decisions and trade-offs + +### Testing +- Write tests for critical functionality +- Aim for good test coverage +- Keep tests simple and focused +- Test edge cases and error conditions + +### Git & Version Control +- Write clear, descriptive commit messages +- Use conventional commit format when appropriate +- Keep commits focused and atomic +- Reference ticket numbers in commits when applicable + +## Development Workflow + +### Before Starting Work +- Check existing tickets in `tickets/` directory (46 tickets available) +- Review `tickets/TICKETS_SUMMARY.md` for overview +- Read `tickets/QUICK_START.md` for recommended starting order +- Review architecture documentation in `ARCHITECTURE.md` +- Check ticket dependencies before starting +- Understand the context and requirements + +### During Development +- Follow the established architecture patterns +- Update documentation as you make changes +- Write tests alongside implementation +- Keep code changes focused and incremental + +### Code Review Considerations +- Is the code readable and maintainable? +- Are edge cases handled? +- Is error handling appropriate? +- Are there any security concerns? +- Does it follow project conventions? + +## AI Assistant Guidelines + +### When Making Changes +- Always read existing code before modifying +- Understand the full context of changes +- Preserve existing patterns and conventions +- Update related documentation +- Consider backward compatibility + +### When Creating New Features +- Follow the established architecture +- Create appropriate tests +- Update relevant documentation +- Consider integration with existing systems + +### When Debugging +- Understand the problem fully before proposing solutions +- Check logs and error messages carefully +- Consider edge cases and race conditions +- Test fixes thoroughly + +## Project-Specific Rules + +### Ticket Management +- Always reference ticket IDs (e.g., TICKET-002) when working on features +- Check ticket dependencies before starting work +- Update ticket status by moving files: `backlog/` → `todo/` → `in-progress/` → `review/` → `done/` +- Include ticket ID in commit messages: `TICKET-002: Define repo structure` +- Keep tickets in sync with Vibe Kanban board + +### Architecture Principles +- **Privacy First**: No external APIs for core ASR/LLM (weather is exception) +- **Separation**: Strict boundaries between work and family agents +- **Local Operation**: All core processing runs locally +- **Safety**: Confirmation flows for high-risk actions +- **Modularity**: Services can be developed in parallel + +### Technology Stack +- **Backend**: Python (services), TypeScript/JavaScript (clients) +- **LLM**: Ollama/vLLM/llama.cpp on 4080 (work) and 1050 (family) +- **ASR**: faster-whisper or Whisper.cpp +- **TTS**: Piper, Mimic 3, or Coqui TTS +- **Wake-Word**: openWakeWord or Porcupine +- **Protocols**: MCP (Model Context Protocol), WebSocket, HTTP/gRPC +- **Storage**: SQLite (memory/sessions), Markdown files (tasks/notes) + +### Repository Structure +- This repo (`atlas/`): Planning, tickets, documentation +- `home-voice-agent/`: Main mono-repo (to be created) +- `family-agent-config/`: Separate config repo (to be created) + +### Security & Safety +- Enforce path whitelists for file operations +- Require confirmation tokens for high-risk actions +- Separate credentials/config for family vs work agents +- Network-level separation (containers, firewall rules) +- Static policy checks in CI/CD diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..dd67ef9 --- /dev/null +++ b/.gitignore @@ -0,0 +1,55 @@ +# Dependencies +node_modules/ +vendor/ +__pycache__/ +*.pyc +*.pyo +*.pyd +.Python + +# Environment variables +.env +.env.local +.env.*.local + +# IDE +.vscode/ +.idea/ +*.swp +*.swo +*~ + +# OS +.DS_Store +Thumbs.db + +# Build outputs +dist/ +build/ +*.egg-info/ +*.so +*.dylib + +# Logs +*.log +logs/ + +# Temporary files +tmp/ +temp/ +*.tmp + +# Coverage reports +coverage/ +*.coverage +.nyc_output/ + +# Database +*.db +*.sqlite +*.sqlite3 + +# Compiled files +*.class +*.jar +*.war diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md new file mode 100644 index 0000000..5e3babb --- /dev/null +++ b/ARCHITECTURE.md @@ -0,0 +1,424 @@ +# Architecture Documentation + +## Overview + +This document describes the architecture, design patterns, and technical decisions for the Atlas home voice agent project. + +Atlas is a local, privacy-focused voice agent system with separate work and family agents, running on dedicated hardware (RTX 4080 for work agent, RTX 1050 for family agent). + +## System Architecture + +### High-Level Design + +The system consists of 5 parallel tracks: + +1. **Voice I/O**: Wake-word detection, ASR (Automatic Speech Recognition), TTS (Text-to-Speech) +2. **LLM Infrastructure**: Two separate LLM servers (4080 for work, 1050 for family) +3. **Tools/MCP**: Model Context Protocol (MCP) tool servers for weather, tasks, notes, etc. +4. **Clients/UI**: Phone PWA and web LAN dashboard +5. **Safety/Memory**: Long-term memory, conversation management, safety controls + +### Component Architecture + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Clients Layer │ +│ ┌──────────────┐ ┌──────────────┐ │ +│ │ Phone PWA │ │ Web Dashboard│ │ +│ └──────┬───────┘ └──────┬───────┘ │ +└─────────┼──────────────────────────────┼──────────────────┘ + │ │ + │ WebSocket/HTTP │ + │ │ +┌─────────┼──────────────────────────────┼──────────────────┐ +│ │ Voice Stack │ │ +│ ┌──────▼──────┐ ┌──────────┐ ┌─────▼──────┐ │ +│ │ Wake-Word │ │ ASR │ │ TTS │ │ +│ │ Node │─▶│ Service │ │ Service │ │ +│ └─────────────┘ └────┬─────┘ └────────────┘ │ +└────────────────────────┼────────────────────────────────┘ + │ +┌────────────────────────┼────────────────────────────────┐ +│ │ LLM Infrastructure │ +│ ┌──────▼──────┐ ┌──────────────┐ │ +│ │ 4080 Server │ │ 1050 Server │ │ +│ │ (Work Agent)│ │(Family Agent)│ │ +│ └──────┬──────┘ └──────┬───────┘ │ +│ │ │ │ +│ └────────────┬───────────────┘ │ +│ │ │ +│ ┌───────▼────────┐ │ +│ │ Routing Layer │ │ +│ └───────┬────────┘ │ +└──────────────────────┼─────────────────────────────────┘ + │ +┌──────────────────────┼─────────────────────────────────┐ +│ │ MCP Tools Layer │ +│ ┌──────▼──────────┐ │ +│ │ MCP Server │ │ +│ │ ┌───────────┐ │ │ +│ │ │ Weather │ │ │ +│ │ │ Tasks │ │ │ +│ │ │ Timers │ │ │ +│ │ │ Notes │ │ │ +│ │ └───────────┘ │ │ +│ └──────────────────┘ │ +└────────────────────────────────────────────────────────┘ +┌─────────────────────────────────────────────────────────┐ +│ │ Safety & Memory │ +│ ┌──────────────┐ ┌──────────────┐ │ +│ │ Memory │ │ Boundaries │ │ +│ │ Store │ │ Enforcement │ │ +│ └──────────────┘ └──────────────┘ │ +└─────────────────────────────────────────────────────────┘ +``` + +### Technology Stack + +- **Languages**: Python (backend services), TypeScript/JavaScript (clients) +- **LLM Servers**: Ollama, vLLM, or llama.cpp +- **ASR**: faster-whisper or Whisper.cpp +- **TTS**: Piper, Mimic 3, or Coqui TTS +- **Wake-Word**: openWakeWord or Porcupine +- **Protocols**: MCP (Model Context Protocol), WebSocket, HTTP/gRPC +- **Storage**: SQLite (memory, sessions), Markdown files (tasks, notes) +- **Infrastructure**: Docker, systemd, Linux + +## Design Patterns + +### Core Patterns + +- **Microservices Architecture**: Separate services for wake-word, ASR, TTS, LLM servers, MCP tools +- **Event-Driven**: Wake-word events trigger ASR capture, tool calls trigger actions +- **API Gateway Pattern**: Routing layer directs requests to appropriate LLM server +- **Repository Pattern**: Separate config repo for family agent (no work content) +- **Tool Pattern**: MCP tools as independent, composable services + +### Architectural Patterns + +- **Separation of Concerns**: Clear boundaries between work and family agents +- **Layered Architecture**: Clients → Voice Stack → LLM Infrastructure → MCP Tools → Safety/Memory +- **Service-Oriented**: Each component is an independent service with defined APIs +- **Privacy by Design**: Local processing, minimal external dependencies + +## Project Structure + +### Repository Structure + +``` +home-voice-agent/ # Main mono-repo +├── llm-servers/ +│ ├── 4080/ # Work agent server +│ └── 1050/ # Family agent server +├── mcp-server/ # MCP tool server +│ └── tools/ # Individual tool implementations +├── wake-word/ # Wake-word detection node +├── asr/ # ASR service +├── tts/ # TTS service +├── clients/ +│ ├── phone/ # Phone PWA +│ └── web-dashboard/ # Web dashboard +├── routing/ # LLM routing layer +├── conversation/ # Conversation management +├── memory/ # Long-term memory +├── safety/ # Safety and boundary enforcement +└── admin/ # Admin tools + +family-agent-config/ # Separate config repo +├── prompts/ # System prompts +├── tools/ # Tool configurations +├── secrets/ # Credentials (no work stuff) +└── tasks/ # Home Kanban board + └── home/ # Home tasks only +``` + +### Atlas Project (This Repo) + +``` +atlas/ +├── tickets/ # Kanban tickets +│ ├── backlog/ # Future work +│ ├── todo/ # Ready to work on +│ ├── in-progress/ # Active work +│ ├── review/ # Awaiting review +│ └── done/ # Completed +├── docs/ # Documentation +├── ARCHITECTURE.md # This file +└── README.md # Project overview +``` + +## Data Models + +### Memory Schema + +Long-term memory stores personal facts, preferences, and routines: + +```python +MemoryEntry: + - id: str + - category: str # personal, family, preferences, routines + - content: str + - timestamp: datetime + - confidence: float + - source: str # conversation, explicit, inferred +``` + +### Conversation Session + +```python +Session: + - session_id: str + - agent_type: str # "work" or "family" + - messages: List[Message] + - created_at: datetime + - last_activity: datetime + - summary: str # after summarization +``` + +### Task Model (Markdown Kanban) + +```yaml +--- +id: TICKET-XXX +title: Task title +status: backlog|todo|in-progress|review|done +priority: high|medium|low +created: YYYY-MM-DD +updated: YYYY-MM-DD +assignee: name +tags: [tag1, tag2] +--- + +Task description... +``` + +### MCP Tool Definition + +```json +{ + "name": "tool_name", + "description": "Tool description", + "inputSchema": { + "type": "object", + "properties": {...} + } +} +``` + +## API Design + +### LLM Server API + +**Endpoint**: `POST /v1/chat/completions` + +```json +{ + "model": "work-agent" | "family-agent", + "messages": [...], + "tools": [...], + "temperature": 0.7 +} +``` + +### ASR Service API + +**Endpoint**: `WebSocket /asr/stream` + +- Input: Audio stream (PCM, 16kHz, mono) +- Output: Text segments with timestamps + +```json +{ + "text": "transcribed text", + "timestamp": 1234.56, + "confidence": 0.95, + "is_final": false +} +``` + +### TTS Service API + +**Endpoint**: `POST /tts/synthesize` + +```json +{ + "text": "Text to speak", + "voice": "family-voice", + "stream": true +} +``` + +Response: Audio stream (WAV or MP3) + +### MCP Server API + +**Protocol**: JSON-RPC 2.0 + +**Methods**: +- `tools/list`: List available tools +- `tools/call`: Execute a tool + +```json +{ + "jsonrpc": "2.0", + "method": "tools/call", + "params": { + "name": "weather", + "arguments": {...} + }, + "id": 1 +} +``` + +## Security Considerations + +### Privacy Policy + +- **Core Principle**: No external APIs for ASR/LLM processing +- **Exception**: Weather API (documented exception) +- **Local Processing**: All voice and LLM processing runs locally +- **Data Retention**: Configurable retention policies for conversations + +### Boundary Enforcement + +- **Repository Separation**: Family agent config in separate repo, no work content +- **Path Whitelists**: Tools can only access whitelisted directories +- **Network Isolation**: Containers/namespaces prevent cross-access +- **Firewall Rules**: Block family agent from accessing work repo paths +- **Static Analysis**: CI/CD checks reject code that would grant cross-access + +### Confirmation Flows + +- **High-Risk Actions**: Email send, calendar changes, file edits outside safe areas +- **Confirmation Tokens**: Signed tokens required from client, not just model intent +- **User Approval**: Explicit "Yes/No" confirmation for sensitive operations +- **Audit Logging**: All confirmations and high-risk actions logged + +### Authentication & Authorization + +- **Token-Based**: Separate tokens for work vs family agents +- **Revocation**: System to disable compromised tokens/devices +- **Admin Controls**: Kill switches for services, tools, or entire agent + +## Performance Considerations + +### Latency Targets + +- **Wake-Word Detection**: < 200ms +- **ASR Processing**: < 2s end-to-end (audio in → text out) +- **LLM Response**: < 3s for family agent (1050), < 5s for work agent (4080) +- **TTS Synthesis**: < 500ms first chunk, streaming thereafter +- **Tool Execution**: < 1s for simple tools (weather, time) + +### Resource Allocation + +- **4080 (Work Agent)**: + - Model: 8-14B or 30B quantized (Q4-Q6) + - Context: 8K-16K tokens + - Concurrency: 2-4 requests + +- **1050 (Family Agent)**: + - Model: 1B-3B quantized (Q4-Q5) + - Context: 4K-8K tokens + - Concurrency: 1-2 requests (always-on, low latency) + +### Optimization Strategies + +- **Model Quantization**: Q4-Q6 for 4080, Q4-Q5 for 1050 +- **Context Management**: Summarization and pruning for long conversations +- **Caching**: Weather API responses, tool results +- **Streaming**: ASR and TTS use streaming for lower perceived latency +- **Batching**: LLM requests where possible (work agent) + +## Deployment + +### Hardware Requirements + +- **RTX 4080 Server**: Work agent LLM, ASR (optional) +- **RTX 1050 Server**: Family agent LLM (always-on) +- **Wake-Word Node**: Raspberry Pi 4+, NUC, or SFF PC +- **Microphones**: USB mics or array mic for living room/office +- **Storage**: SSD for logs, HDD for archives + +### Service Deployment + +- **LLM Servers**: Systemd services or Docker containers +- **MCP Server**: Systemd service with auto-restart +- **Voice Services**: ASR and TTS as systemd services +- **Wake-Word Node**: Standalone service on dedicated hardware +- **Clients**: PWA served via web server, web dashboard on LAN + +### Configuration Management + +- **Family Agent Config**: Separate `family-agent-config/` repo +- **Secrets**: Environment variables, separate .env files +- **Prompts**: Version-controlled in config repo +- **Tool Configs**: YAML/JSON files in config repo + +### Monitoring & Logging + +- **Structured Logging**: JSON logs for all services +- **Metrics**: GPU usage, latency, error rates +- **Admin Dashboard**: Web UI for logs, metrics, controls +- **Alerting**: System notifications for errors or high resource usage + +## Development Workflow + +### Ticket-Based Development + +1. **Select Ticket**: Choose from `tickets/backlog/` or `tickets/todo/` +2. **Check Dependencies**: Review ticket dependencies before starting +3. **Move to In-Progress**: Move ticket to `tickets/in-progress/` +4. **Implement**: Follow architecture patterns and conventions +5. **Test**: Write and run tests +6. **Document**: Update relevant documentation +7. **Move to Review**: Move ticket to `tickets/review/` when complete +8. **Move to Done**: Move to `tickets/done/` after approval + +### Parallel Development + +Many tickets can be worked on simultaneously: +- **Voice I/O**: Independent of LLM and MCP +- **LLM Infrastructure**: Can proceed after model selection +- **MCP Tools**: Can start with minimal server, add tools incrementally +- **Clients/UI**: Can mock APIs early, integrate later + +### Milestone Progression + +- **Milestone 1**: Foundation and surveys (TICKET-002 through TICKET-029) +- **Milestone 2**: MVP with voice chat, weather, tasks (core functionality) +- **Milestone 3**: Memory, reminders, safety features +- **Milestone 4**: Optional integrations (email, calendar, smart home) + +## Future Considerations + +### Planned Enhancements + +- **Semantic Search**: Add embeddings for note search (beyond ripgrep) +- **Routine Learning**: Automatically learn and suggest routines from memory +- **Multi-Device**: Support multiple wake-word nodes and clients +- **Offline Mode**: Enhanced offline capabilities for clients +- **Voice Cloning**: Custom voice profiles for family members + +### Technical Debt + +- Start with basic implementations, optimize later +- Initial memory system: simple schema, enhance with better retrieval +- Tool permissions: Start with whitelists, add more granular control later +- Logging: Start with files, migrate to time-series DB if needed + +### Scalability + +- Current design supports single household +- Future: Multi-user support with user-specific memory and preferences +- Future: Distributed deployment across multiple nodes + +## Related Documentation + +- **Tickets**: See `tickets/TICKETS_SUMMARY.md` for all 46 tickets +- **Quick Start**: See `tickets/QUICK_START.md` for recommended starting order +- **Ticket Template**: See `tickets/TICKET_TEMPLATE.md` for creating new tickets + +--- + +**Note**: Update this document as the architecture evolves. diff --git a/README.md b/README.md new file mode 100644 index 0000000..9d87ac2 --- /dev/null +++ b/README.md @@ -0,0 +1,110 @@ +# Atlas Project + +## Overview + +Atlas is a local, privacy-focused home voice agent system. This repository contains the project planning, architecture documentation, and kanban tickets for building the system. + +The project uses a kanban-based workflow (Vibe Kanban) for task management, with tickets organized in the `tickets/` directory structure. + +## Project Goals + +Build a complete voice agent system with: +- **Separate work and family agents** running on dedicated hardware (RTX 4080 and RTX 1050) +- **Local, private operation** - no external APIs for core ASR/LLM (weather is exception) +- **Voice I/O** - wake-word detection, ASR, and TTS +- **MCP tools** - weather, tasks, timers, reminders, notes +- **Safety controls** - boundary enforcement, confirmation flows, admin tools +- **Long-term memory** - personal facts, preferences, routines + +## Project Structure + +``` +atlas/ +├── .cursorrules # Cursor IDE configuration and rules +├── README.md # This file +├── ARCHITECTURE.md # Architecture documentation +├── tickets/ # Kanban tickets and task management +│ ├── backlog/ # Backlog items +│ ├── todo/ # Items ready to work on +│ ├── in-progress/ # Currently active work +│ ├── review/ # Items awaiting review +│ └── done/ # Completed items +└── [project code] # Your project code will go here +``` + +## Getting Started + +1. Review the [ARCHITECTURE.md](./ARCHITECTURE.md) for project design and patterns +2. Check [tickets/TICKETS_SUMMARY.md](./tickets/TICKETS_SUMMARY.md) for an overview of all 46 tickets +3. Read [tickets/QUICK_START.md](./tickets/QUICK_START.md) for recommended starting order +4. Follow the guidelines in [.cursorrules](./.cursorrules) for development + +## Project Tracks + +The project is organized into 5 parallel tracks: + +1. **Voice I/O** (12 tickets): Wake-word detection, ASR (Automatic Speech Recognition), TTS (Text-to-Speech) +2. **LLM Infrastructure** (11 tickets): Model selection, servers for 4080 (work) and 1050 (family), routing, prompts +3. **Tools/MCP** (11 tickets): Model Context Protocol server, weather, tasks, timers, reminders, notes, optional integrations +4. **Clients/UI** (2 tickets): Phone PWA and web LAN dashboard +5. **Safety/Memory** (6 tickets): Long-term memory, conversation management, boundary enforcement, confirmations + +Plus **Project Setup** (4 tickets) for foundation work. + +## Kanban Workflow + +This project uses a kanban board (Vibe Kanban) for task management. **46 tickets** are organized in the `tickets/` directory: + +- **backlog/**: 46 tickets ready to start +- **todo/**: Ready to work on (dependencies met) +- **in-progress/**: Currently being worked on +- **review/**: Awaiting review or testing +- **done/**: Completed items + +### Working with Tickets + +1. **Start with Milestone 1**: Foundation tickets (TICKET-002, TICKET-003, TICKET-004) +2. **Move tickets as you work**: `backlog/` → `todo/` → `in-progress/` → `review/` → `done/` +3. **Reference ticket IDs in commits**: `git commit -m "TICKET-002: Define repo structure"` +4. **Sync with Vibe Kanban**: Import tickets using ticket IDs, tag by track and milestone +5. **Use ticket template**: See `tickets/TICKET_TEMPLATE.md` for new tickets + +### Milestones + +- **Milestone 1 - Survey & Architecture**: Foundation, technology evaluations +- **Milestone 2 - Voice Chat + Weather + Tasks MVP**: Core functionality working +- **Milestone 3 - Memory, Reminders, Safety**: Advanced features and safety +- **Milestone 4 - Optional Integrations**: Email, calendar, smart home + +## Development Guidelines + +- Follow the rules defined in `.cursorrules` +- Maintain code quality and documentation +- Write tests for new features +- Update architecture docs for significant changes +- Reference ticket numbers in all commits +- Keep tickets in sync with Vibe Kanban board + +## Repository Structure + +This repository (`atlas/`) contains: +- **Planning and documentation**: Architecture, tickets, project management +- **Kanban tickets**: 46 tickets organized by status and track + +The actual implementation will be in separate repositories: +- `home-voice-agent/`: Main mono-repo with all services and clients +- `family-agent-config/`: Separate config repo for prompts, tools, secrets (no work content) + +See [ARCHITECTURE.md](./ARCHITECTURE.md) for detailed repository structure. + +## Contributing + +1. Check existing tickets in `tickets/` before starting work +2. Create a ticket for new features or bugs using the template +3. Follow the established code style and patterns +4. Update documentation as needed +5. Move tickets through the workflow as you progress + +## License + +[Add your license here] diff --git a/tickets/.gitkeep b/tickets/.gitkeep new file mode 100644 index 0000000..c125a96 --- /dev/null +++ b/tickets/.gitkeep @@ -0,0 +1,3 @@ +# Tickets Directory + +This directory contains kanban tickets organized by status. diff --git a/tickets/QUICK_START.md b/tickets/QUICK_START.md new file mode 100644 index 0000000..81c7d21 --- /dev/null +++ b/tickets/QUICK_START.md @@ -0,0 +1,93 @@ +# Quick Start Guide + +## Overview + +This project has **47 tickets** organized across 5 parallel tracks. All tickets are currently in `backlog/` and ready to be moved to `todo/` as you start working on them. + +## Getting Started + +1. **Review the tickets summary**: See `TICKETS_SUMMARY.md` for an overview +2. **Start with Milestone 1**: Foundation and surveys +3. **Move tickets as you work**: `backlog/` → `todo/` → `in-progress/` → `review/` → `done/` + +## Recommended Starting Order + +### Phase 1: Foundation (Week 1) +Move these to `todo/` first: +- TICKET-002: Define Project Repos and Structure +- TICKET-003: Document Privacy Policy and Safety Constraints +- TICKET-004: High-Level Architecture Document + +### Phase 2: Parallel Surveys (Week 1-2) +These can be done in parallel: +- TICKET-005: Evaluate Wake-Word Engine +- TICKET-009: Select ASR Engine +- TICKET-013: Evaluate TTS Options +- TICKET-017: Survey LLM Models +- TICKET-018: LLM Capacity Assessment +- TICKET-028: Learn MCP Concepts + +### Phase 3: Model Selection (Week 2) +- TICKET-019: Select Work Agent Model (4080) +- TICKET-020: Select Family Agent Model (1050) + +### Phase 4: Implementation (Week 3+) +Start implementing based on selected technologies. + +## Ticket Status Workflow + +``` +backlog/ → todo/ → in-progress/ → review/ → done/ +``` + +1. **backlog/**: Future work, not yet started +2. **todo/**: Ready to work on (dependencies met) +3. **in-progress/**: Currently being worked on +4. **review/**: Complete, awaiting review/testing +5. **done/**: Approved and merged + +## Using with Vibe Kanban + +1. Import tickets into Vibe Kanban using ticket IDs (TICKET-002, etc.) +2. Tag tickets by track: Voice I/O, LLM Infra, Tools/MCP, Clients/UI, Safety/Memory +3. Tag tickets by milestone: M1, M2, M3, M4 +4. Keep ticket files in sync with kanban board status +5. Reference ticket IDs in commits: `git commit -m "TICKET-002: Define repo structure"` + +## Parallel Work Opportunities + +Many tickets can be worked on simultaneously: + +- **Voice I/O track**: Independent of LLM and MCP +- **LLM Infra track**: Can proceed after model selection +- **MCP Tools track**: Can start with minimal server, add tools incrementally +- **Clients/UI track**: Can mock APIs early, integrate later +- **Safety/Memory track**: Can design early, implement after MVP + +## Critical Path for MVP + +For fastest path to working MVP: + +1. Foundation (TICKET-002, TICKET-003, TICKET-004) +2. Technology selection (TICKET-005, TICKET-009, TICKET-013, TICKET-019, TICKET-020) +3. MCP foundation (TICKET-028, TICKET-029) +4. Voice stack (TICKET-006, TICKET-010, TICKET-014) +5. LLM servers (TICKET-021, TICKET-022) +6. MCP integration (TICKET-030) +7. Core tools (TICKET-031, TICKET-034) +8. Phone client (TICKET-039) +9. End-to-end testing + +## Tips + +- **Read dependencies**: Each ticket lists its dependencies +- **Start small**: Begin with evaluations and prototypes +- **Iterate**: Don't wait for perfection - get MVP working first +- **Document decisions**: Update ARCHITECTURE.md as you make choices +- **Test early**: Test each component as you build it + +## Questions? + +- See `TICKETS_SUMMARY.md` for detailed breakdown +- See `TICKET_TEMPLATE.md` for ticket format +- See `README.md` in tickets/ for workflow details diff --git a/tickets/README.md b/tickets/README.md new file mode 100644 index 0000000..fb5885c --- /dev/null +++ b/tickets/README.md @@ -0,0 +1,38 @@ +# Kanban Tickets + +This directory contains all project tickets organized by their kanban status. + +## Directory Structure + +- **backlog/**: Future work items that are planned but not yet ready to start +- **todo/**: Items that are ready to be worked on +- **in-progress/**: Items currently being actively developed +- **review/**: Items that are complete but awaiting review, testing, or approval +- **done/**: Completed items (archived for reference) + +## Ticket Naming Convention + +Tickets should be named using the format: +``` +[TICKET-ID]_[short-description].md +``` + +Example: `TICKET-001_setup-project-structure.md` + +## Workflow + +1. **Create**: New tickets start in `backlog/` or `todo/` depending on readiness +2. **Start Work**: Move ticket from `todo/` to `in-progress/` when starting +3. **Complete**: Move ticket from `in-progress/` to `review/` when implementation is done +4. **Approve**: Move ticket from `review/` to `done/` when approved/merged + +## Using with Vibe Kanban + +1. Create tickets in this directory structure +2. Import or reference these tickets in your Vibe Kanban board +3. Keep ticket files in sync with kanban board status +4. Use ticket IDs to reference in commits and PRs + +## Template + +Use `TICKET_TEMPLATE.md` when creating new tickets to ensure consistency. diff --git a/tickets/TICKETS_SUMMARY.md b/tickets/TICKETS_SUMMARY.md new file mode 100644 index 0000000..7263267 --- /dev/null +++ b/tickets/TICKETS_SUMMARY.md @@ -0,0 +1,166 @@ +# Tickets Summary + +This document provides an overview of all tickets organized by track and milestone. + +## Ticket Organization + +Tickets are organized in the `tickets/` directory by status: +- `backlog/` - Future work items +- `todo/` - Ready to work on +- `in-progress/` - Currently active +- `review/` - Awaiting review/testing +- `done/` - Completed + +## Tickets by Track + +### Project Setup (4 tickets) +- TICKET-002: Define Project Repos and Structure +- TICKET-003: Document Privacy Policy and Safety Constraints +- TICKET-004: High-Level Architecture Document +- TICKET-047: Hardware & Purchases + +### Voice I/O (12 tickets) +- TICKET-005: Evaluate and Select Wake-Word Engine +- TICKET-006: Prototype Local Wake-Word Node +- TICKET-007: Train or Configure Custom Wake-Word +- TICKET-008: Wake-Word False Positive/Negative Tuning +- TICKET-009: Select ASR Engine and Target Hardware +- TICKET-010: Implement Streaming Audio Capture → ASR Service +- TICKET-011: Define ASR API Contract +- TICKET-012: Benchmark ASR Latency and Quality +- TICKET-013: Evaluate TTS Options +- TICKET-014: Build TTS Service +- TICKET-015: Voice Consistency and Volume Leveling +- TICKET-016: Integrate TTS with Clients + +### LLM Infra (11 tickets) +- TICKET-017: Survey Candidate Open-Weight Models +- TICKET-018: LLM Capacity Assessment +- TICKET-019: Select Work Agent Model (4080) +- TICKET-020: Select Family Agent Model (1050) +- TICKET-021: Stand Up 4080 LLM Service +- TICKET-022: Stand Up 1050 LLM Service +- TICKET-023: Implement LLM Routing Layer +- TICKET-024: LLM Logging & Metrics +- TICKET-025: System Prompts for Family vs Work Agents +- TICKET-026: Tool-Calling Policy +- TICKET-027: Multi-Turn Conversation Handling + +### Tools/MCP (11 tickets) +- TICKET-028: Learn and Encode MCP Concepts +- TICKET-029: Implement Minimal MCP Server +- TICKET-030: Integrate MCP with LLM Host +- TICKET-031: Weather Tool +- TICKET-032: Time / Date / World-Clock Tools +- TICKET-033: Timers and Reminders +- TICKET-034: Home Tasks/Chores (Markdown Kanban) +- TICKET-035: Notes & Files (Markdown, PDFs) +- TICKET-036: Email Integration (Optional) +- TICKET-037: Calendar Integration (Optional) +- TICKET-038: Smart Home Integration (Optional) + +### Clients/UI (2 tickets) +- TICKET-039: Phone-Friendly Client (PWA or Native) +- TICKET-040: Web LAN Dashboard + +### Safety/Memory (6 tickets) +- TICKET-041: Long-Term Memory Design +- TICKET-042: Long-Term Memory Implementation +- TICKET-043: Conversation Summarization & Pruning +- TICKET-044: Boundary Enforcement +- TICKET-045: Confirmation Flows +- TICKET-046: Admin Tools + +## Tickets by Milestone + +### Milestone 1 - Survey & Architecture +- TICKET-002: Define Project Repos and Structure +- TICKET-003: Document Privacy Policy and Safety Constraints +- TICKET-004: High-Level Architecture Document +- TICKET-005: Evaluate and Select Wake-Word Engine +- TICKET-009: Select ASR Engine and Target Hardware +- TICKET-013: Evaluate TTS Options +- TICKET-017: Survey Candidate Open-Weight Models +- TICKET-018: LLM Capacity Assessment +- TICKET-019: Select Work Agent Model (4080) +- TICKET-020: Select Family Agent Model (1050) +- TICKET-028: Learn and Encode MCP Concepts +- TICKET-029: Implement Minimal MCP Server + +### Milestone 2 - Voice Chat + Weather + Tasks MVP +- TICKET-006: Prototype Local Wake-Word Node +- TICKET-007: Train or Configure Custom Wake-Word +- TICKET-008: Wake-Word False Positive/Negative Tuning +- TICKET-010: Implement Streaming Audio Capture → ASR Service +- TICKET-011: Define ASR API Contract +- TICKET-014: Build TTS Service +- TICKET-015: Voice Consistency and Volume Leveling +- TICKET-016: Integrate TTS with Clients +- TICKET-021: Stand Up 4080 LLM Service +- TICKET-022: Stand Up 1050 LLM Service +- TICKET-023: Implement LLM Routing Layer +- TICKET-025: System Prompts for Family vs Work Agents +- TICKET-027: Multi-Turn Conversation Handling +- TICKET-030: Integrate MCP with LLM Host +- TICKET-031: Weather Tool +- TICKET-032: Time / Date / World-Clock Tools +- TICKET-034: Home Tasks/Chores (Markdown Kanban) +- TICKET-039: Phone-Friendly Client (PWA or Native) +- TICKET-040: Web LAN Dashboard + +### Milestone 3 - Memory, Reminders, Safety +- TICKET-012: Benchmark ASR Latency and Quality +- TICKET-024: LLM Logging & Metrics +- TICKET-026: Tool-Calling Policy +- TICKET-033: Timers and Reminders +- TICKET-035: Notes & Files (Markdown, PDFs) +- TICKET-041: Long-Term Memory Design +- TICKET-042: Long-Term Memory Implementation +- TICKET-043: Conversation Summarization & Pruning +- TICKET-044: Boundary Enforcement +- TICKET-045: Confirmation Flows +- TICKET-046: Admin Tools + +### Milestone 4 - Optional Integrations & Smart Features +- TICKET-036: Email Integration (Optional) +- TICKET-037: Calendar Integration (Optional) +- TICKET-038: Smart Home Integration (Optional) + +## Dependency Graph + +### Foundation (No Dependencies) +- TICKET-002: Define Project Repos and Structure +- TICKET-003: Document Privacy Policy and Safety Constraints +- TICKET-004: High-Level Architecture Document (depends on TICKET-002, TICKET-003) + +### Parallel Tracks (Can Start After Foundation) +- **Voice I/O**: TICKET-005, TICKET-009, TICKET-013 (evaluations can start) +- **LLM Infra**: TICKET-017, TICKET-018 (surveys can start) +- **MCP**: TICKET-028 (MCP concepts can start) + +### Critical Path for MVP +1. TICKET-002, TICKET-003, TICKET-004 (Foundation) +2. TICKET-005, TICKET-009, TICKET-013 (Voice evaluations) +3. TICKET-017, TICKET-018, TICKET-019, TICKET-020 (Model selection) +4. TICKET-028, TICKET-029 (MCP foundation) +5. TICKET-006, TICKET-010, TICKET-014 (Voice implementation) +6. TICKET-021, TICKET-022 (LLM servers) +7. TICKET-030 (MCP-LLM integration) +8. TICKET-031, TICKET-034 (Core tools) +9. TICKET-039 (Phone client) +10. End-to-end testing + +## Quick Start + +1. Start with **Milestone 1** tickets (surveys and architecture) +2. Move tickets from `backlog/` to `todo/` as you're ready to work on them +3. Move to `in-progress/` when actively working +4. Move to `review/` when complete and ready for testing +5. Move to `done/` when approved/merged + +## Notes + +- Many tickets can be worked on in parallel (e.g., Voice I/O, LLM Infra, MCP) +- Some tickets have explicit dependencies listed in their descriptions +- Optional tickets (TICKET-036, TICKET-037, TICKET-038) should be done after MVP +- Hardware purchases (TICKET-047) can be done in parallel with software development diff --git a/tickets/TICKET_TEMPLATE.md b/tickets/TICKET_TEMPLATE.md new file mode 100644 index 0000000..61a637d --- /dev/null +++ b/tickets/TICKET_TEMPLATE.md @@ -0,0 +1,43 @@ +# Ticket Template + +Use this template when creating new tickets. + +## Ticket Information + +- **ID**: [e.g., TICKET-001] +- **Title**: [Brief, descriptive title] +- **Type**: [Feature / Bug / Enhancement / Documentation / Refactor] +- **Priority**: [High / Medium / Low] +- **Status**: [Backlog / Todo / In Progress / Review / Done] +- **Created**: [Date] +- **Assigned**: [Name/Team] + +## Description + +[Detailed description of what needs to be done] + +## Acceptance Criteria + +- [ ] Criterion 1 +- [ ] Criterion 2 +- [ ] Criterion 3 + +## Technical Details + +[Technical implementation details, architecture considerations, etc.] + +## Dependencies + +[Other tickets or work items this depends on] + +## Related Files + +[Links to relevant code files, documentation, etc.] + +## Notes + +[Additional notes, questions, or considerations] + +## Progress Log + +- [Date] - [Action taken or update] diff --git a/tickets/backlog/.gitkeep b/tickets/backlog/.gitkeep new file mode 100644 index 0000000..8df7505 --- /dev/null +++ b/tickets/backlog/.gitkeep @@ -0,0 +1,3 @@ +# Backlog + +Items planned for future work. diff --git a/tickets/backlog/TICKET-002_define-repos-structure.md b/tickets/backlog/TICKET-002_define-repos-structure.md new file mode 100644 index 0000000..90bb07c --- /dev/null +++ b/tickets/backlog/TICKET-002_define-repos-structure.md @@ -0,0 +1,55 @@ +# Ticket: Define Project Repos and Structure + +## Ticket Information + +- **ID**: TICKET-002 +- **Title**: Define Project Repos and Structure +- **Type**: Setup +- **Priority**: High +- **Status**: Backlog +- **Track**: Project Setup +- **Milestone**: Milestone 1 - Survey & Architecture +- **Created**: 2024-01-XX + +## Description + +Create the repository structure for the home voice agent project: +- Create `home-voice-agent` mono-repo containing: + - LLM servers (4080 and 1050) + - MCP tool servers + - Clients (phone PWA, web dashboard) + - Infrastructure code +- Create separate `family-agent-config` repo for: + - Prompts and system instructions + - Tool configurations + - Secrets and credentials (no work-related content) + +## Acceptance Criteria + +- [ ] `home-voice-agent` mono-repo structure defined and documented +- [ ] `family-agent-config` repo structure defined +- [ ] Clear separation between work and family agent code/config +- [ ] Directory structure documented in ARCHITECTURE.md +- [ ] Initial repos created (or structure documented for creation) + +## Technical Details + +Mono-repo structure should support: +- Separate services for LLM servers (4080 vs 1050) +- MCP tool servers as independent services +- Client applications (PWA, web dashboard) +- Shared libraries and utilities +- Clear boundaries for deployment and security + +## Dependencies + +None - this is a foundational task. + +## Related Files + +- `ARCHITECTURE.md` +- `README.md` + +## Notes + +This is a prerequisite for all other development work. Should be done first. diff --git a/tickets/backlog/TICKET-003_privacy-safety-constraints.md b/tickets/backlog/TICKET-003_privacy-safety-constraints.md new file mode 100644 index 0000000..e156be6 --- /dev/null +++ b/tickets/backlog/TICKET-003_privacy-safety-constraints.md @@ -0,0 +1,50 @@ +# Ticket: Document Privacy Policy and Safety Constraints + +## Ticket Information + +- **ID**: TICKET-003 +- **Title**: Document Privacy Policy and Safety Constraints +- **Type**: Documentation +- **Priority**: High +- **Status**: Backlog +- **Track**: Safety/Memory +- **Milestone**: Milestone 1 - Survey & Architecture +- **Created**: 2024-01-XX + +## Description + +Define and document privacy policy and safety constraints for the family voice agent: +- Privacy policy: No external APIs for core ASR/LLM (define exceptions like weather) +- List "forbidden actions" for family agent (e.g., touching work repos) +- Document allowed vs restricted tool access +- Define data retention and deletion policies + +## Acceptance Criteria + +- [ ] Privacy policy document created +- [ ] List of forbidden actions documented +- [ ] Exception list for external APIs defined (e.g., weather) +- [ ] Safety constraints documented in project docs +- [ ] Constraints integrated into architecture decisions + +## Technical Details + +This will inform: +- Tool permission design +- Network separation requirements +- Confirmation flow design +- Memory and data retention policies + +## Dependencies + +None - can be done in parallel with TICKET-002. + +## Related Files + +- `docs/PRIVACY_POLICY.md` (to be created) +- `docs/SAFETY_CONSTRAINTS.md` (to be created) +- `ARCHITECTURE.md` + +## Notes + +These constraints will be referenced throughout development, especially for safety and boundary enforcement tasks. diff --git a/tickets/backlog/TICKET-004_high-level-architecture.md b/tickets/backlog/TICKET-004_high-level-architecture.md new file mode 100644 index 0000000..b13ed17 --- /dev/null +++ b/tickets/backlog/TICKET-004_high-level-architecture.md @@ -0,0 +1,57 @@ +# Ticket: High-Level Architecture Document + +## Ticket Information + +- **ID**: TICKET-004 +- **Title**: High-Level Architecture Document +- **Type**: Documentation +- **Priority**: High +- **Status**: Backlog +- **Track**: Project Setup +- **Milestone**: Milestone 1 - Survey & Architecture +- **Created**: 2024-01-XX + +## Description + +Create comprehensive architecture documentation including: +- Text version of system architecture showing: + - 4080 LLM server (work agent) + - 1050 LLM server (family agent) + - MCP tool servers + - Wake-word nodes + - Clients (phone PWA, web dashboard) + - Storage systems + - Logging infrastructure +- Component interactions and data flows +- Network topology and separation + +## Acceptance Criteria + +- [ ] Architecture diagram created (text or visual) +- [ ] All major components documented +- [ ] Data flows between components described +- [ ] Network separation documented +- [ ] Architecture document updated in ARCHITECTURE.md +- [ ] Prerequisite for detailed design tasks + +## Technical Details + +Should cover: +- Hardware allocation (4080 vs 1050) +- Service boundaries and APIs +- Communication protocols (WebSocket, HTTP, gRPC) +- Storage locations and formats +- Security boundaries + +## Dependencies + +- TICKET-002 (repo structure) +- TICKET-003 (privacy/safety constraints) + +## Related Files + +- `ARCHITECTURE.md` + +## Notes + +This becomes a prerequisite for detailed Voice/LLM/MCP design tasks. Should be completed early. diff --git a/tickets/backlog/TICKET-005_wake-word-evaluation.md b/tickets/backlog/TICKET-005_wake-word-evaluation.md new file mode 100644 index 0000000..61518cf --- /dev/null +++ b/tickets/backlog/TICKET-005_wake-word-evaluation.md @@ -0,0 +1,57 @@ +# Ticket: Evaluate and Select Wake-Word Engine + +## Ticket Information + +- **ID**: TICKET-005 +- **Title**: Evaluate and Select Wake-Word Engine +- **Type**: Research +- **Priority**: High +- **Status**: Backlog +- **Track**: Voice I/O +- **Milestone**: Milestone 1 - Survey & Architecture +- **Created**: 2024-01-XX + +## Description + +Evaluate wake-word detection options and select one: +- Compare openWakeWord and Porcupine for: + - Hardware compatibility (Linux box/Pi/NUC) + - Licensing requirements + - Ability to train custom "Hey Atlas" wake-word + - Performance and resource usage + - False positive/negative characteristics + +## Acceptance Criteria + +- [ ] Comparison matrix of wake-word options +- [ ] Selected engine documented with rationale +- [ ] Hardware requirements documented +- [ ] Licensing considerations documented +- [ ] Decision recorded in architecture docs + +## Technical Details + +Options to evaluate: +- openWakeWord (open source, trainable) +- Porcupine (Picovoice, commercial) +- Other open-source alternatives + +Considerations: +- Custom wake-word training capability +- Resource usage on target hardware +- Latency requirements +- Integration complexity + +## Dependencies + +- TICKET-004 (architecture) - helpful but not required +- Hardware availability for testing + +## Related Files + +- `docs/WAKE_WORD_EVALUATION.md` (to be created) +- `ARCHITECTURE.md` + +## Notes + +Can start in parallel with LLM infra and MCP design. Needs hardware + microphone availability for testing. diff --git a/tickets/backlog/TICKET-006_prototype-wake-word-node.md b/tickets/backlog/TICKET-006_prototype-wake-word-node.md new file mode 100644 index 0000000..f539264 --- /dev/null +++ b/tickets/backlog/TICKET-006_prototype-wake-word-node.md @@ -0,0 +1,52 @@ +# Ticket: Prototype Local Wake-Word Node + +## Ticket Information + +- **ID**: TICKET-006 +- **Title**: Prototype Local Wake-Word Node +- **Type**: Feature +- **Priority**: High +- **Status**: Backlog +- **Track**: Voice I/O +- **Milestone**: Milestone 2 - Voice Chat MVP +- **Created**: 2024-01-XX + +## Description + +Build a minimal wake-word detection node: +- Script running on Linux box/Pi/NUC +- Listens to microphone input +- Detects wake-word ("Hey Atlas") +- Emits events via WebSocket/MQTT/HTTP when detected +- Handles audio capture and processing + +## Acceptance Criteria + +- [ ] Wake-word node script implemented +- [ ] Microphone input working +- [ ] Wake-word detection functional +- [ ] Event emission mechanism working (WebSocket/MQTT/HTTP) +- [ ] Basic error handling and logging +- [ ] Can run continuously on target hardware + +## Technical Details + +Implementation approach: +- Use selected wake-word engine from TICKET-005 +- Audio capture via PyAudio or similar +- Event emission to trigger ASR capture +- Minimal resource usage for always-on operation +- Logging for debugging false positives/negatives + +## Dependencies + +- TICKET-005 (wake-word engine selection) +- Hardware with microphone access + +## Related Files + +- `home-voice-agent/wake-word/` (to be created) + +## Notes + +Does not depend on ASR/LLM - just needs to send "start capture" messages. Can run in parallel with other voice work. diff --git a/tickets/backlog/TICKET-007_train-custom-wake-word.md b/tickets/backlog/TICKET-007_train-custom-wake-word.md new file mode 100644 index 0000000..f436336 --- /dev/null +++ b/tickets/backlog/TICKET-007_train-custom-wake-word.md @@ -0,0 +1,49 @@ +# Ticket: Train or Configure Custom Wake-Word + +## Ticket Information + +- **ID**: TICKET-007 +- **Title**: Train or Configure Custom Wake-Word "Hey Atlas" +- **Type**: Feature +- **Priority**: Medium +- **Status**: Backlog +- **Track**: Voice I/O +- **Milestone**: Milestone 2 - Voice Chat MVP +- **Created**: 2024-01-XX + +## Description + +Train or configure the custom "Hey Atlas" wake-word: +- Collect audio samples for "Hey Atlas" +- Train model if using openWakeWord's training pipeline +- Configure wake-word if using pre-trained engine +- Test detection accuracy + +## Acceptance Criteria + +- [ ] Audio samples collected for "Hey Atlas" +- [ ] Model trained or configured +- [ ] Wake-word detection working with custom phrase +- [ ] Accuracy tested in quiet and noisy environments +- [ ] Model file saved and versioned + +## Technical Details + +Process depends on selected engine: +- openWakeWord: Training pipeline with collected samples +- Porcupine: Custom wake-word creation via their platform +- Other: Follow engine-specific process + +## Dependencies + +- TICKET-005 (engine selection) +- TICKET-006 (prototype node) - for testing + +## Related Files + +- `home-voice-agent/wake-word/models/` (to be created) +- `family-agent-config/wake-word-samples/` (to be created) + +## Notes + +May require multiple iterations to achieve good accuracy. False positive tuning comes next. diff --git a/tickets/backlog/TICKET-008_wake-word-tuning.md b/tickets/backlog/TICKET-008_wake-word-tuning.md new file mode 100644 index 0000000..77d0b59 --- /dev/null +++ b/tickets/backlog/TICKET-008_wake-word-tuning.md @@ -0,0 +1,51 @@ +# Ticket: Wake-Word False Positive/Negative Tuning + +## Ticket Information + +- **ID**: TICKET-008 +- **Title**: Wake-Word False Positive/Negative Tuning +- **Type**: Enhancement +- **Priority**: Medium +- **Status**: Backlog +- **Track**: Voice I/O +- **Milestone**: Milestone 2 - Voice Chat MVP +- **Created**: 2024-01-XX + +## Description + +Tune wake-word detection for production use: +- Adjust sensitivity and thresholds +- Log all detection events (true and false) +- Test in noisy home environment +- Optimize balance between false positives and false negatives +- Document optimal settings + +## Acceptance Criteria + +- [ ] Sensitivity/threshold parameters configurable +- [ ] Event logging implemented +- [ ] Tested in various home noise conditions +- [ ] False positive rate acceptable (< 1 per hour) +- [ ] False negative rate acceptable (< 5%) +- [ ] Optimal settings documented + +## Technical Details + +Tuning approach: +- Parameterize sensitivity settings +- Log all events with timestamps and audio snippets +- A/B test different threshold values +- Monitor over extended period in real environment + +## Dependencies + +- TICKET-006 (prototype node) +- TICKET-007 (custom wake-word) + +## Related Files + +- `home-voice-agent/wake-word/config/` (to be created) + +## Notes + +This is an iterative process that may continue after MVP. Initial tuning should achieve reasonable performance. diff --git a/tickets/backlog/TICKET-009_asr-engine-selection.md b/tickets/backlog/TICKET-009_asr-engine-selection.md new file mode 100644 index 0000000..fa4a8ca --- /dev/null +++ b/tickets/backlog/TICKET-009_asr-engine-selection.md @@ -0,0 +1,49 @@ +# Ticket: Select ASR Engine and Target Hardware + +## Ticket Information + +- **ID**: TICKET-009 +- **Title**: Select ASR Engine and Target Hardware +- **Type**: Research +- **Priority**: High +- **Status**: Backlog +- **Track**: Voice I/O +- **Milestone**: Milestone 1 - Survey & Architecture +- **Created**: 2024-01-XX + +## Description + +Decide on ASR (Automatic Speech Recognition) engine and deployment: +- Evaluate options: faster-whisper, Whisper.cpp, etc. +- Decide deployment: faster-whisper on 4080, CPU-only on small box, or shared +- Consider model size vs latency trade-offs +- Document hardware requirements + +## Acceptance Criteria + +- [ ] ASR engine selected (faster-whisper recommended) +- [ ] Target hardware decided (4080 vs CPU box) +- [ ] Model size selected (medium vs small) +- [ ] Latency requirements documented +- [ ] Decision recorded in architecture docs + +## Technical Details + +Considerations: +- faster-whisper on 4080: Lower latency, higher quality +- CPU-only on small box: Lower cost, higher latency +- Shared deployment: Resource contention considerations +- Model sizes: tiny/small/medium/base for latency/quality trade-off + +## Dependencies + +- TICKET-004 (architecture) - helpful context + +## Related Files + +- `docs/ASR_EVALUATION.md` (to be created) +- `ARCHITECTURE.md` + +## Notes + +Can run in parallel with TTS and LLM work. Needs wake-word event flow defined for when to start/stop capture. diff --git a/tickets/backlog/TICKET-010_streaming-asr-service.md b/tickets/backlog/TICKET-010_streaming-asr-service.md new file mode 100644 index 0000000..8238c44 --- /dev/null +++ b/tickets/backlog/TICKET-010_streaming-asr-service.md @@ -0,0 +1,51 @@ +# Ticket: Implement Streaming Audio Capture → ASR Service + +## Ticket Information + +- **ID**: TICKET-010 +- **Title**: Implement Streaming Audio Capture → ASR Service +- **Type**: Feature +- **Priority**: High +- **Status**: Backlog +- **Track**: Voice I/O +- **Milestone**: Milestone 2 - Voice Chat MVP +- **Created**: 2024-01-XX + +## Description + +Build streaming ASR service: +- Implement audio capture (GStreamer/ffmpeg or browser getUserMedia) +- Create WebSocket endpoint for audio streaming +- Integrate selected ASR engine +- Stream audio chunks to ASR and return text segments +- Handle start/stop based on wake-word events + +## Acceptance Criteria + +- [ ] Audio capture working (from mic or WebSocket) +- [ ] WebSocket endpoint for audio streaming +- [ ] ASR engine integrated +- [ ] Text segments returned with timestamps +- [ ] Handles wake-word start/stop events +- [ ] Streaming latency acceptable (< 2s end-to-end) + +## Technical Details + +Implementation: +- Audio capture: PyAudio, GStreamer, or browser MediaRecorder +- WebSocket server for real-time streaming +- ASR processing: faster-whisper or selected engine +- Return format: JSON with text, timestamps, confidence + +## Dependencies + +- TICKET-009 (ASR engine selection) +- Wake-word event flow defined (from TICKET-006) + +## Related Files + +- `home-voice-agent/asr/` (to be created) + +## Notes + +Can run in parallel with TTS and LLM work. Needs wake-word integration for start/stop. diff --git a/tickets/backlog/TICKET-011_asr-api-contract.md b/tickets/backlog/TICKET-011_asr-api-contract.md new file mode 100644 index 0000000..19ff323 --- /dev/null +++ b/tickets/backlog/TICKET-011_asr-api-contract.md @@ -0,0 +1,51 @@ +# Ticket: Define ASR API Contract + +## Ticket Information + +- **ID**: TICKET-011 +- **Title**: Define ASR API Contract +- **Type**: Documentation +- **Priority**: Medium +- **Status**: Backlog +- **Track**: Voice I/O +- **Milestone**: Milestone 2 - Voice Chat MVP +- **Created**: 2024-01-XX + +## Description + +Define the ASR service API contract: +- gRPC or HTTP endpoint specification +- Request/response formats +- Text segments and timestamp format +- Error handling +- Streaming vs batch modes + +## Acceptance Criteria + +- [ ] API contract documented (OpenAPI/gRPC proto) +- [ ] Request/response schemas defined +- [ ] Error codes and handling documented +- [ ] Streaming protocol defined +- [ ] Client integration examples provided + +## Technical Details + +API should support: +- Streaming audio input +- Real-time text segments +- Timestamps for each segment +- Confidence scores +- End-of-speech detection + +## Dependencies + +- TICKET-010 (ASR service implementation) + +## Related Files + +- `docs/API_ASR.md` (to be created) +- `home-voice-agent/asr/api/` (to be created) + +## Notes + +Should be defined early to enable parallel client development. diff --git a/tickets/backlog/TICKET-012_asr-benchmarking.md b/tickets/backlog/TICKET-012_asr-benchmarking.md new file mode 100644 index 0000000..1e21667 --- /dev/null +++ b/tickets/backlog/TICKET-012_asr-benchmarking.md @@ -0,0 +1,49 @@ +# Ticket: Benchmark ASR Latency and Quality + +## Ticket Information + +- **ID**: TICKET-012 +- **Title**: Benchmark ASR Latency and Quality +- **Type**: Testing +- **Priority**: Medium +- **Status**: Backlog +- **Track**: Voice I/O +- **Milestone**: Milestone 2 - Voice Chat MVP +- **Created**: 2024-01-XX + +## Description + +Benchmark ASR performance: +- Compare medium vs small models +- Quantify latency on 1050 vs 4080 (if applicable) +- Measure accuracy on test audio samples +- Document performance characteristics + +## Acceptance Criteria + +- [ ] Benchmark suite created +- [ ] Latency measured for different model sizes +- [ ] Accuracy measured on test set +- [ ] Performance documented +- [ ] Optimal configuration selected + +## Technical Details + +Benchmarks should measure: +- End-to-end latency (audio in → text out) +- Word error rate (WER) +- Resource usage (CPU/GPU/memory) +- Throughput (concurrent requests) + +## Dependencies + +- TICKET-010 (ASR service) +- TICKET-011 (API contract) + +## Related Files + +- `home-voice-agent/asr/benchmarks/` (to be created) + +## Notes + +Can be done after initial implementation. Helps optimize for production. diff --git a/tickets/backlog/TICKET-013_tts-evaluation.md b/tickets/backlog/TICKET-013_tts-evaluation.md new file mode 100644 index 0000000..0c453db --- /dev/null +++ b/tickets/backlog/TICKET-013_tts-evaluation.md @@ -0,0 +1,55 @@ +# Ticket: Evaluate TTS Options + +## Ticket Information + +- **ID**: TICKET-013 +- **Title**: Evaluate TTS Options +- **Type**: Research +- **Priority**: High +- **Status**: Backlog +- **Track**: Voice I/O +- **Milestone**: Milestone 1 - Survey & Architecture +- **Created**: 2024-01-XX + +## Description + +Evaluate text-to-speech options: +- Compare open source options (Piper, Mimic 3, etc.) +- Evaluate local neural TTS solutions +- Select 1-2 voices for family agent +- Consider latency, quality, and resource usage + +## Acceptance Criteria + +- [ ] TTS options compared +- [ ] Selected TTS engine documented +- [ ] Voice samples selected +- [ ] Resource requirements documented +- [ ] Decision recorded in architecture docs + +## Technical Details + +Options to evaluate: +- Piper (lightweight, fast) +- Mimic 3 (high quality) +- Coqui TTS (neural, customizable) +- Other open-source solutions + +Considerations: +- Latency for interactive use +- Voice quality and naturalness +- Resource usage +- Customization options + +## Dependencies + +- TICKET-004 (architecture) - helpful context + +## Related Files + +- `docs/TTS_EVALUATION.md` (to be created) +- `ARCHITECTURE.md` + +## Notes + +Independent of LLM logic. Can be developed in parallel with other voice work. diff --git a/tickets/backlog/TICKET-014_tts-service.md b/tickets/backlog/TICKET-014_tts-service.md new file mode 100644 index 0000000..9df1db6 --- /dev/null +++ b/tickets/backlog/TICKET-014_tts-service.md @@ -0,0 +1,50 @@ +# Ticket: Build TTS Service + +## Ticket Information + +- **ID**: TICKET-014 +- **Title**: Build TTS Service +- **Type**: Feature +- **Priority**: High +- **Status**: Backlog +- **Track**: Voice I/O +- **Milestone**: Milestone 2 - Voice Chat MVP +- **Created**: 2024-01-XX + +## Description + +Build TTS service: +- HTTP/gRPC endpoint: text in → audio out +- Low-latency settings for interactive use +- Streaming audio output support +- Voice selection and configuration + +## Acceptance Criteria + +- [ ] TTS service implemented +- [ ] HTTP/gRPC endpoint working +- [ ] Text input → audio output functional +- [ ] Low-latency configuration (< 500ms) +- [ ] Streaming support for long text +- [ ] Voice selection working + +## Technical Details + +Implementation: +- Use selected TTS engine from TICKET-013 +- HTTP endpoint for simple requests +- Streaming for real-time playback +- Audio format: WAV or MP3 +- Sample rate: 22050 or 44100 Hz + +## Dependencies + +- TICKET-013 (TTS evaluation) + +## Related Files + +- `home-voice-agent/tts/` (to be created) + +## Notes + +Needs defined API between agent and TTS (simple JSON with text + voice profile). diff --git a/tickets/backlog/TICKET-015_tts-voice-consistency.md b/tickets/backlog/TICKET-015_tts-voice-consistency.md new file mode 100644 index 0000000..bd05f28 --- /dev/null +++ b/tickets/backlog/TICKET-015_tts-voice-consistency.md @@ -0,0 +1,48 @@ +# Ticket: Voice Consistency and Volume Leveling + +## Ticket Information + +- **ID**: TICKET-015 +- **Title**: Voice Consistency and Volume Leveling +- **Type**: Enhancement +- **Priority**: Medium +- **Status**: Backlog +- **Track**: Voice I/O +- **Milestone**: Milestone 2 - Voice Chat MVP +- **Created**: 2024-01-XX + +## Description + +Ensure consistent voice experience: +- Choose a "family voice" and standardize +- Normalize volume levels +- Configure speed and pause behavior +- Test across different devices + +## Acceptance Criteria + +- [ ] Single "family voice" selected and configured +- [ ] Volume normalization implemented +- [ ] Speed and pause settings tuned +- [ ] Consistent across different playback devices +- [ ] Settings documented + +## Technical Details + +Configuration: +- Voice profile saved in config +- Volume normalization (LUFS or peak) +- Speech rate: ~150-160 words/min +- Pause duration between sentences + +## Dependencies + +- TICKET-014 (TTS service) + +## Related Files + +- `family-agent-config/tts/voice-profile.yaml` (to be created) + +## Notes + +Important for user experience. Should be done before MVP launch. diff --git a/tickets/backlog/TICKET-016_tts-client-integration.md b/tickets/backlog/TICKET-016_tts-client-integration.md new file mode 100644 index 0000000..70a4e48 --- /dev/null +++ b/tickets/backlog/TICKET-016_tts-client-integration.md @@ -0,0 +1,49 @@ +# Ticket: Integrate TTS with Clients + +## Ticket Information + +- **ID**: TICKET-016 +- **Title**: Integrate TTS with Clients +- **Type**: Feature +- **Priority**: High +- **Status**: Backlog +- **Track**: Voice I/O, Clients/UI +- **Milestone**: Milestone 2 - Voice Chat MVP +- **Created**: 2024-01-XX + +## Description + +Integrate TTS service with clients: +- Streaming audio back to browser or mobile client +- Simple audio player with barge-in (stop speaking if interrupted) +- Handle audio playback errors +- Visual feedback during speech + +## Acceptance Criteria + +- [ ] TTS streaming to browser client working +- [ ] TTS streaming to mobile client working +- [ ] Barge-in functionality (interrupt speech) +- [ ] Audio playback error handling +- [ ] Visual feedback during speech + +## Technical Details + +Implementation: +- WebSocket or HTTP streaming for audio +- Audio element with controls +- Interrupt detection (wake-word or button) +- Graceful stop and cleanup + +## Dependencies + +- TICKET-014 (TTS service) +- TICKET-021 (phone client) or TICKET-022 (web dashboard) + +## Related Files + +- `home-voice-agent/clients/` (to be created) + +## Notes + +Requires client implementation. Can be done in parallel with client development. diff --git a/tickets/backlog/TICKET-017_llm-model-survey.md b/tickets/backlog/TICKET-017_llm-model-survey.md new file mode 100644 index 0000000..2e73d51 --- /dev/null +++ b/tickets/backlog/TICKET-017_llm-model-survey.md @@ -0,0 +1,49 @@ +# Ticket: Survey Candidate Open-Weight Models + +## Ticket Information + +- **ID**: TICKET-017 +- **Title**: Survey Candidate Open-Weight Models +- **Type**: Research +- **Priority**: High +- **Status**: Backlog +- **Track**: LLM Infra +- **Milestone**: Milestone 1 - Survey & Architecture +- **Created**: 2024-01-XX + +## Description + +Survey and evaluate open-weight LLM models: +- 8-14B and 30B quantized options for RTX 4080 (Q4-Q6 variants) +- Small models for RTX 1050 (family agent) +- Evaluate coding/research capabilities for work agent +- Evaluate instruction-following for family agent + +## Acceptance Criteria + +- [ ] Model comparison matrix created +- [ ] 4080 model candidates identified (8-14B, 30B quantized) +- [ ] 1050 model candidates identified (small, efficient) +- [ ] Evaluation criteria documented +- [ ] Recommendations documented + +## Technical Details + +Models to evaluate: +- 4080: Llama 3 8B/70B, Mistral 7B, Qwen, etc. +- 1050: TinyLlama, Phi-2, smaller quantized models +- Quantization: Q4, Q5, Q6, Q8 +- Function calling support required + +## Dependencies + +- TICKET-004 (architecture) - helpful context + +## Related Files + +- `docs/LLM_MODEL_SURVEY.md` (to be created) +- `ARCHITECTURE.md` + +## Notes + +Can start in parallel with wake-word and clients. Depends on high-level architecture doc. diff --git a/tickets/backlog/TICKET-018_llm-capacity-assessment.md b/tickets/backlog/TICKET-018_llm-capacity-assessment.md new file mode 100644 index 0000000..87e7816 --- /dev/null +++ b/tickets/backlog/TICKET-018_llm-capacity-assessment.md @@ -0,0 +1,49 @@ +# Ticket: LLM Capacity Assessment + +## Ticket Information + +- **ID**: TICKET-018 +- **Title**: LLM Capacity Assessment +- **Type**: Research +- **Priority**: High +- **Status**: Backlog +- **Track**: LLM Infra +- **Milestone**: Milestone 1 - Survey & Architecture +- **Created**: 2024-01-XX + +## Description + +Determine maximum context and parameter size: +- Assess 16GB VRAM capacity (13B-24B comfortable with quantization) +- Determine max context window for 4080 +- Assess 1050 capacity (smaller models, limited context) +- Document memory requirements + +## Acceptance Criteria + +- [ ] VRAM capacity documented for 4080 +- [ ] VRAM capacity documented for 1050 +- [ ] Max context window determined +- [ ] Model size limits documented +- [ ] Memory requirements in architecture docs + +## Technical Details + +Assessment should cover: +- 4080: 16GB VRAM, Q4/Q5 quantization +- 1050: 4GB VRAM, very small models +- Context window: 4K, 8K, 16K, 32K options +- Batch size and concurrency limits + +## Dependencies + +- TICKET-017 (model survey) + +## Related Files + +- `docs/LLM_CAPACITY.md` (to be created) +- `ARCHITECTURE.md` + +## Notes + +Critical for model selection. Should be done early. diff --git a/tickets/backlog/TICKET-019_select-work-agent-model.md b/tickets/backlog/TICKET-019_select-work-agent-model.md new file mode 100644 index 0000000..8c40c30 --- /dev/null +++ b/tickets/backlog/TICKET-019_select-work-agent-model.md @@ -0,0 +1,50 @@ +# Ticket: Select Work Agent Model (4080) + +## Ticket Information + +- **ID**: TICKET-019 +- **Title**: Select Work Agent Model for 4080 +- **Type**: Research +- **Priority**: High +- **Status**: Backlog +- **Track**: LLM Infra +- **Milestone**: Milestone 1 - Survey & Architecture +- **Created**: 2024-01-XX + +## Description + +Select the LLM model for work agent on 4080: +- Coding/research-optimized model +- Not used by family agent +- Suitable for 16GB VRAM with quantization +- Good function-calling support + +## Acceptance Criteria + +- [ ] Work agent model selected +- [ ] Quantization level chosen +- [ ] Rationale documented +- [ ] Model file location specified +- [ ] Performance characteristics documented + +## Technical Details + +Selection criteria: +- Coding capabilities (CodeLlama, DeepSeek Coder, etc.) +- Research/analysis capabilities +- Function calling support +- Context window size +- Quantization: Q4-Q6 for 16GB VRAM + +## Dependencies + +- TICKET-017 (model survey) +- TICKET-018 (capacity assessment) + +## Related Files + +- `docs/MODEL_SELECTION.md` (to be created) + +## Notes + +Separate from family agent model. Can be selected independently. diff --git a/tickets/backlog/TICKET-020_select-family-agent-model.md b/tickets/backlog/TICKET-020_select-family-agent-model.md new file mode 100644 index 0000000..79e2527 --- /dev/null +++ b/tickets/backlog/TICKET-020_select-family-agent-model.md @@ -0,0 +1,50 @@ +# Ticket: Select Family Agent Model (1050) + +## Ticket Information + +- **ID**: TICKET-020 +- **Title**: Select Family Agent Model for 1050 +- **Type**: Research +- **Priority**: High +- **Status**: Backlog +- **Track**: LLM Infra +- **Milestone**: Milestone 1 - Survey & Architecture +- **Created**: 2024-01-XX + +## Description + +Select the LLM model for family agent on 1050: +- Small, instruction-tuned model +- Latency-optimized for 24/7 operation +- Suitable for 4GB VRAM +- Good instruction-following + +## Acceptance Criteria + +- [ ] Family agent model selected +- [ ] Quantization level chosen +- [ ] Rationale documented +- [ ] Model file location specified +- [ ] Latency characteristics documented + +## Technical Details + +Selection criteria: +- Small model size (1B-3B parameters) +- Instruction-tuned +- Low latency (< 1s response time) +- Function calling support +- Quantization: Q4 or Q5 for 4GB VRAM + +## Dependencies + +- TICKET-017 (model survey) +- TICKET-018 (capacity assessment) + +## Related Files + +- `docs/MODEL_SELECTION.md` (to be created) + +## Notes + +Optimized for always-on, low-latency family interactions. Separate from work agent. diff --git a/tickets/backlog/TICKET-021_setup-4080-llm-server.md b/tickets/backlog/TICKET-021_setup-4080-llm-server.md new file mode 100644 index 0000000..38b80e9 --- /dev/null +++ b/tickets/backlog/TICKET-021_setup-4080-llm-server.md @@ -0,0 +1,55 @@ +# Ticket: Stand Up 4080 LLM Service + +## Ticket Information + +- **ID**: TICKET-021 +- **Title**: Stand Up 4080 LLM Service +- **Type**: Feature +- **Priority**: High +- **Status**: Backlog +- **Track**: LLM Infra +- **Milestone**: Milestone 2 - Voice Chat MVP +- **Created**: 2024-01-XX + +## Description + +Set up LLM service on 4080: +- Use Ollama/vLLM/llama.cpp-based server +- Expose HTTP/gRPC API +- Support function-calling/tool use +- Load selected work agent model +- Configure for optimal performance + +## Acceptance Criteria + +- [ ] LLM server running on 4080 +- [ ] HTTP/gRPC endpoint exposed +- [ ] Work agent model loaded +- [ ] Function-calling support working +- [ ] Basic health check endpoint +- [ ] Performance acceptable + +## Technical Details + +Server options: +- Ollama: Easy setup, good tool support +- vLLM: High throughput, batching +- llama.cpp: Lightweight, efficient + +Requirements: +- HTTP API for simple requests +- gRPC for streaming (optional) +- Function calling format (OpenAI-compatible) + +## Dependencies + +- TICKET-019 (work agent model selection) +- TICKET-004 (architecture) + +## Related Files + +- `home-voice-agent/llm-servers/4080/` (to be created) + +## Notes + +Independent of MCP/tool design - just needs common API. Can proceed after model selection. diff --git a/tickets/backlog/TICKET-022_setup-1050-llm-server.md b/tickets/backlog/TICKET-022_setup-1050-llm-server.md new file mode 100644 index 0000000..c2778b5 --- /dev/null +++ b/tickets/backlog/TICKET-022_setup-1050-llm-server.md @@ -0,0 +1,52 @@ +# Ticket: Stand Up 1050 LLM Service + +## Ticket Information + +- **ID**: TICKET-022 +- **Title**: Stand Up 1050 LLM Service +- **Type**: Feature +- **Priority**: High +- **Status**: Backlog +- **Track**: LLM Infra +- **Milestone**: Milestone 2 - Voice Chat MVP +- **Created**: 2024-01-XX + +## Description + +Set up LLM service on 1050: +- Smaller model, lower concurrency +- Persistent process managed via systemd/docker +- Expose HTTP/gRPC API +- Support function-calling/tool use +- Load selected family agent model + +## Acceptance Criteria + +- [ ] LLM server running on 1050 +- [ ] HTTP/gRPC endpoint exposed +- [ ] Family agent model loaded +- [ ] Function-calling support working +- [ ] Systemd/docker service configured +- [ ] Auto-restart on failure + +## Technical Details + +Server setup: +- Use llama.cpp or Ollama (lightweight) +- Systemd service for auto-start +- Docker option for isolation +- Lower concurrency (1-2 requests) +- Optimized for latency + +## Dependencies + +- TICKET-020 (family agent model selection) +- TICKET-004 (architecture) + +## Related Files + +- `home-voice-agent/llm-servers/1050/` (to be created) + +## Notes + +Optimized for always-on family agent. Lower resource usage than 4080 server. diff --git a/tickets/backlog/TICKET-023_llm-routing-layer.md b/tickets/backlog/TICKET-023_llm-routing-layer.md new file mode 100644 index 0000000..9a347a7 --- /dev/null +++ b/tickets/backlog/TICKET-023_llm-routing-layer.md @@ -0,0 +1,50 @@ +# Ticket: Implement LLM Routing Layer + +## Ticket Information + +- **ID**: TICKET-023 +- **Title**: Implement LLM Routing Layer +- **Type**: Feature +- **Priority**: High +- **Status**: Backlog +- **Track**: LLM Infra +- **Milestone**: Milestone 2 - Voice Chat MVP +- **Created**: 2024-01-XX + +## Description + +Build routing layer for LLM requests: +- Simple gateway deciding family vs work agent +- Route based on origin/identity +- Load balancing if needed +- Request logging and metrics + +## Acceptance Criteria + +- [ ] Routing gateway implemented +- [ ] Family vs work agent routing working +- [ ] Identity-based routing functional +- [ ] Request logging implemented +- [ ] Basic metrics collection + +## Technical Details + +Routing logic: +- Check request origin (client, IP, token) +- Route to 1050 for family agent +- Route to 4080 for work agent +- Fallback and error handling +- Rate limiting per agent + +## Dependencies + +- TICKET-021 (4080 server) +- TICKET-022 (1050 server) + +## Related Files + +- `home-voice-agent/routing/` (to be created) + +## Notes + +Simple gateway initially. Can be enhanced later with more sophisticated routing. diff --git a/tickets/backlog/TICKET-024_llm-logging-metrics.md b/tickets/backlog/TICKET-024_llm-logging-metrics.md new file mode 100644 index 0000000..b1ebaab --- /dev/null +++ b/tickets/backlog/TICKET-024_llm-logging-metrics.md @@ -0,0 +1,49 @@ +# Ticket: LLM Logging & Metrics + +## Ticket Information + +- **ID**: TICKET-024 +- **Title**: LLM Logging & Metrics +- **Type**: Feature +- **Priority**: Medium +- **Status**: Backlog +- **Track**: LLM Infra +- **Milestone**: Milestone 2 - Voice Chat MVP +- **Created**: 2024-01-XX + +## Description + +Implement logging and metrics for LLM services: +- Structured logs (prompt, tools called, latency, cost estimates) +- GPU usage dashboards +- Request/response logging +- Error tracking + +## Acceptance Criteria + +- [ ] Structured logging implemented +- [ ] Log format includes: prompt, tools, latency +- [ ] GPU usage monitoring +- [ ] Basic metrics dashboard +- [ ] Error logging and alerting + +## Technical Details + +Logging format: +- JSON structured logs +- Fields: timestamp, agent, prompt, tools, latency, tokens, errors +- GPU metrics: utilization, memory, temperature +- Storage: local files or time-series DB + +## Dependencies + +- TICKET-021 (4080 server) +- TICKET-022 (1050 server) + +## Related Files + +- `home-voice-agent/monitoring/` (to be created) + +## Notes + +Important for debugging and optimization. Can be enhanced over time. diff --git a/tickets/backlog/TICKET-025_system-prompts.md b/tickets/backlog/TICKET-025_system-prompts.md new file mode 100644 index 0000000..a9703b2 --- /dev/null +++ b/tickets/backlog/TICKET-025_system-prompts.md @@ -0,0 +1,52 @@ +# Ticket: System Prompts for Family vs Work Agents + +## Ticket Information + +- **ID**: TICKET-025 +- **Title**: System Prompts for Family vs Work Agents +- **Type**: Feature +- **Priority**: High +- **Status**: Backlog +- **Track**: LLM Infra +- **Milestone**: Milestone 2 - Voice Chat MVP +- **Created**: 2024-01-XX + +## Description + +Create system prompts for both agents: +- Separate prompt files for family vs work agents +- Explicit boundaries and allowed tools +- Personality and behavior guidelines +- Safety constraints integrated + +## Acceptance Criteria + +- [ ] Family agent system prompt created +- [ ] Work agent system prompt created +- [ ] Boundaries clearly defined +- [ ] Allowed tools listed +- [ ] Prompts stored in config repo +- [ ] Version controlled + +## Technical Details + +Prompt structure: +- Role and personality +- Allowed actions and tools +- Forbidden actions +- Response style guidelines +- Safety reminders + +## Dependencies + +- TICKET-003 (privacy/safety constraints) +- TICKET-004 (architecture) + +## Related Files + +- `family-agent-config/prompts/family-agent.md` (to be created) +- `family-agent-config/prompts/work-agent.md` (to be created) + +## Notes + +Critical for agent behavior. Should reference safety constraints from TICKET-003. diff --git a/tickets/backlog/TICKET-026_tool-calling-policy.md b/tickets/backlog/TICKET-026_tool-calling-policy.md new file mode 100644 index 0000000..58b72a6 --- /dev/null +++ b/tickets/backlog/TICKET-026_tool-calling-policy.md @@ -0,0 +1,50 @@ +# Ticket: Tool-Calling Policy + +## Ticket Information + +- **ID**: TICKET-026 +- **Title**: Tool-Calling Policy +- **Type**: Documentation +- **Priority**: High +- **Status**: Backlog +- **Track**: LLM Infra, Safety/Memory +- **Milestone**: Milestone 2 - Voice Chat MVP +- **Created**: 2024-01-XX + +## Description + +Define tool-calling policy: +- When model is encouraged/forbidden to call tools +- High-impact actions need confirmation +- Tool permission matrix +- Escalation rules + +## Acceptance Criteria + +- [ ] Tool-calling policy documented +- [ ] High-impact actions identified +- [ ] Confirmation requirements defined +- [ ] Permission matrix created +- [ ] Policy integrated into prompts + +## Technical Details + +Policy should cover: +- Low-risk tools: always allowed (weather, time) +- Medium-risk: require confirmation (tasks, notes) +- High-risk: explicit user approval (email, calendar, smart home) +- Tool categories and permissions + +## Dependencies + +- TICKET-003 (safety constraints) +- TICKET-025 (system prompts) +- MCP tool definitions (from MCP tickets) + +## Related Files + +- `docs/TOOL_CALLING_POLICY.md` (to be created) + +## Notes + +Ties into confirmation flows (TICKET-045). Should be defined before tool implementation. diff --git a/tickets/backlog/TICKET-027_multi-turn-conversation.md b/tickets/backlog/TICKET-027_multi-turn-conversation.md new file mode 100644 index 0000000..dc3341e --- /dev/null +++ b/tickets/backlog/TICKET-027_multi-turn-conversation.md @@ -0,0 +1,50 @@ +# Ticket: Multi-Turn Conversation Handling + +## Ticket Information + +- **ID**: TICKET-027 +- **Title**: Multi-Turn Conversation Handling +- **Type**: Feature +- **Priority**: High +- **Status**: Backlog +- **Track**: LLM Infra +- **Milestone**: Milestone 2 - Voice Chat MVP +- **Created**: 2024-01-XX + +## Description + +Implement multi-turn conversation handling: +- Per-session context management +- Summarized memory integration +- Define how many turns to keep in context +- Session storage and retrieval + +## Acceptance Criteria + +- [ ] Session context management implemented +- [ ] Turn history stored per session +- [ ] Context window management (N turns) +- [ ] Memory integration working +- [ ] Session cleanup/expiration + +## Technical Details + +Implementation: +- Session ID tracking +- Context buffer (last N messages) +- Memory retrieval and injection +- Summarization for old context +- Storage: in-memory or DB + +## Dependencies + +- TICKET-021 or TICKET-022 (LLM server) +- TICKET-040 (memory design) - helpful but not required initially + +## Related Files + +- `home-voice-agent/conversation/` (to be created) + +## Notes + +Needs at least a stub MCP/tool interface. Can start with simple in-memory sessions. diff --git a/tickets/backlog/TICKET-028_mcp-foundation.md b/tickets/backlog/TICKET-028_mcp-foundation.md new file mode 100644 index 0000000..5dbecaf --- /dev/null +++ b/tickets/backlog/TICKET-028_mcp-foundation.md @@ -0,0 +1,49 @@ +# Ticket: Learn and Encode MCP Concepts + +## Ticket Information + +- **ID**: TICKET-028 +- **Title**: Learn and Encode MCP Concepts into Architecture +- **Type**: Research +- **Priority**: High +- **Status**: Backlog +- **Track**: Tools/MCP +- **Milestone**: Milestone 1 - Survey & Architecture +- **Created**: 2024-01-XX + +## Description + +Learn Model Context Protocol (MCP) and integrate into architecture: +- Understand MCP concepts: hosts, clients, servers +- Tools/call and tools/list endpoints +- JSON-RPC protocol +- Architecture integration + +## Acceptance Criteria + +- [ ] MCP concepts understood and documented +- [ ] Architecture updated with MCP components +- [ ] Protocol details documented +- [ ] Integration points identified + +## Technical Details + +MCP concepts: +- Hosts: LLM servers +- Clients: Applications using LLM +- Servers: Tool providers +- Tools: call and list operations +- JSON-RPC 2.0 protocol + +## Dependencies + +- TICKET-004 (architecture) + +## Related Files + +- `docs/MCP_ARCHITECTURE.md` (to be created) +- `ARCHITECTURE.md` + +## Notes + +Foundation for all MCP tool work. Should be done early. diff --git a/tickets/backlog/TICKET-029_minimal-mcp-server.md b/tickets/backlog/TICKET-029_minimal-mcp-server.md new file mode 100644 index 0000000..77adc2c --- /dev/null +++ b/tickets/backlog/TICKET-029_minimal-mcp-server.md @@ -0,0 +1,53 @@ +# Ticket: Implement Minimal MCP Server + +## Ticket Information + +- **ID**: TICKET-029 +- **Title**: Implement Minimal MCP Server +- **Type**: Feature +- **Priority**: High +- **Status**: Backlog +- **Track**: Tools/MCP +- **Milestone**: Milestone 1 - Survey & Architecture +- **Created**: 2024-01-XX + +## Description + +Build a minimal MCP server: +- One service exposing a few tools (e.g., weather, echo) +- JSON-RPC protocol implementation +- Tools/call and tools/list endpoints +- Basic error handling + +## Acceptance Criteria + +- [ ] MCP server implemented +- [ ] JSON-RPC protocol working +- [ ] Tools/list endpoint functional +- [ ] Tools/call endpoint functional +- [ ] At least 2 example tools (weather, echo) +- [ ] Error handling implemented + +## Technical Details + +Implementation: +- JSON-RPC 2.0 server +- Tool registration system +- Request/response handling +- Error codes and messages + +## Dependencies + +- TICKET-028 (MCP concepts) + +## Related Files + +- `home-voice-agent/mcp-server/` (to be created) + +## Related Files + +- `home-voice-agent/mcp-server/` (to be created) + +## Notes + +Independent of specific tools - start with stubs. Can be tested with dummy models. diff --git a/tickets/backlog/TICKET-030_mcp-llm-integration.md b/tickets/backlog/TICKET-030_mcp-llm-integration.md new file mode 100644 index 0000000..511037d --- /dev/null +++ b/tickets/backlog/TICKET-030_mcp-llm-integration.md @@ -0,0 +1,49 @@ +# Ticket: Integrate MCP with LLM Host + +## Ticket Information + +- **ID**: TICKET-030 +- **Title**: Integrate MCP with Chosen LLM Host +- **Type**: Feature +- **Priority**: High +- **Status**: Backlog +- **Track**: Tools/MCP, LLM Infra +- **Milestone**: Milestone 2 - Voice Chat MVP +- **Created**: 2024-01-XX + +## Description + +Integrate MCP server with LLM: +- Write adapter converting model tool-use outputs into MCP calls +- Convert MCP responses back to LLM format +- Handle tool discovery and registration +- Error handling and retries + +## Acceptance Criteria + +- [ ] MCP-LLM adapter implemented +- [ ] Tool-use outputs → MCP calls working +- [ ] MCP responses → LLM format working +- [ ] Tool discovery automatic +- [ ] Error handling robust + +## Technical Details + +Adapter should: +- Parse LLM function calls +- Map to MCP tool calls +- Handle responses and errors +- Support streaming if needed + +## Dependencies + +- TICKET-029 (MCP server) +- TICKET-021 or TICKET-022 (LLM server with function-calling) + +## Related Files + +- `home-voice-agent/mcp-adapter/` (to be created) + +## Notes + +Needs LLM server with function-calling support. Critical for tool integration. diff --git a/tickets/backlog/TICKET-031_weather-tool.md b/tickets/backlog/TICKET-031_weather-tool.md new file mode 100644 index 0000000..b3d441d --- /dev/null +++ b/tickets/backlog/TICKET-031_weather-tool.md @@ -0,0 +1,55 @@ +# Ticket: Weather Tool + +## Ticket Information + +- **ID**: TICKET-031 +- **Title**: Weather Tool +- **Type**: Feature +- **Priority**: High +- **Status**: Backlog +- **Track**: Tools/MCP +- **Milestone**: Milestone 2 - Voice Chat MVP +- **Created**: 2024-01-XX + +## Description + +Implement weather MCP tool: +- Choose API (or local weather source) +- Implement MCP tool with location handling +- Rate limiting to prevent API abuse +- Error handling for API failures + +## Acceptance Criteria + +- [ ] Weather tool implemented as MCP tool +- [ ] Location handling (address, coordinates) +- [ ] Rate limiting implemented +- [ ] Error handling robust +- [ ] Tool registered in MCP server +- [ ] Tested with LLM + +## Technical Details + +Options: +- OpenWeatherMap API (external - exception to privacy policy) +- Local weather station data +- National Weather Service API + +Implementation: +- Location parsing and geocoding +- API key management +- Rate limiting (requests per hour) +- Caching for recent queries + +## Dependencies + +- TICKET-029 (MCP server) +- TICKET-003 (privacy policy - weather is exception) + +## Related Files + +- `home-voice-agent/mcp-server/tools/weather/` (to be created) + +## Notes + +Weather is an exception to "no external APIs" policy. Document in privacy policy. diff --git a/tickets/backlog/TICKET-032_time-date-tools.md b/tickets/backlog/TICKET-032_time-date-tools.md new file mode 100644 index 0000000..ee894e5 --- /dev/null +++ b/tickets/backlog/TICKET-032_time-date-tools.md @@ -0,0 +1,48 @@ +# Ticket: Time / Date / World-Clock Tools + +## Ticket Information + +- **ID**: TICKET-032 +- **Title**: Time / Date / World-Clock Tools +- **Type**: Feature +- **Priority**: Medium +- **Status**: Backlog +- **Track**: Tools/MCP +- **Milestone**: Milestone 2 - Voice Chat MVP +- **Created**: 2024-01-XX + +## Description + +Implement time and date tools: +- Simple tools returning local time +- Date information +- Daylight saving time info +- World clock for different timezones + +## Acceptance Criteria + +- [ ] Local time tool implemented +- [ ] Date tool implemented +- [ ] Timezone support +- [ ] Daylight saving time handling +- [ ] Tools registered in MCP server + +## Technical Details + +Tools to implement: +- `get_current_time`: Local time with timezone +- `get_date`: Current date +- `get_timezone_info`: DST, offset, etc. +- `convert_timezone`: Convert between timezones + +## Dependencies + +- TICKET-029 (MCP server) + +## Related Files + +- `home-voice-agent/mcp-server/tools/time/` (to be created) + +## Notes + +Simple tools, no external dependencies. Can be developed in parallel with other tools. diff --git a/tickets/backlog/TICKET-033_timers-reminders.md b/tickets/backlog/TICKET-033_timers-reminders.md new file mode 100644 index 0000000..c62e7e1 --- /dev/null +++ b/tickets/backlog/TICKET-033_timers-reminders.md @@ -0,0 +1,49 @@ +# Ticket: Timers and Reminders + +## Ticket Information + +- **ID**: TICKET-033 +- **Title**: Timers and Reminders +- **Type**: Feature +- **Priority**: High +- **Status**: Backlog +- **Track**: Tools/MCP +- **Milestone**: Milestone 2 - Voice Chat MVP +- **Created**: 2024-01-XX + +## Description + +Implement timers and reminders: +- Local service storing timers/reminders +- Trigger callbacks/notifications +- MCP tools: create_timer, create_reminder, list_timers +- Persistent storage + +## Acceptance Criteria + +- [ ] Timer service implemented +- [ ] Reminder service implemented +- [ ] MCP tools for timers/reminders +- [ ] Persistent storage (DB or files) +- [ ] Notification system working +- [ ] Tools registered in MCP server + +## Technical Details + +Implementation: +- Storage: SQLite or JSON files +- Scheduling: cron-like or event loop +- Notifications: WebSocket, HTTP callback, or system notification +- Tools: create, list, cancel, update + +## Dependencies + +- TICKET-029 (MCP server) + +## Related Files + +- `home-voice-agent/mcp-server/tools/timers/` (to be created) + +## Notes + +Can be developed in parallel with notes/tasks tools. Important for family agent use cases. diff --git a/tickets/backlog/TICKET-034_home-tasks-kanban.md b/tickets/backlog/TICKET-034_home-tasks-kanban.md new file mode 100644 index 0000000..393e337 --- /dev/null +++ b/tickets/backlog/TICKET-034_home-tasks-kanban.md @@ -0,0 +1,59 @@ +# Ticket: Home Tasks/Chores (Markdown Kanban) + +## Ticket Information + +- **ID**: TICKET-034 +- **Title**: Home Tasks/Chores (Markdown Kanban) +- **Type**: Feature +- **Priority**: High +- **Status**: Backlog +- **Track**: Tools/MCP +- **Milestone**: Milestone 2 - Voice Chat MVP +- **Created**: 2024-01-XX + +## Description + +Implement home task management: +- Decide home task representation (Markdown-only repo for Vibe Kanban) +- YAML frontmatter or similar structure +- MCP tools: add_task, update_task_status, list_tasks +- Permission and separation logic (only touch home repo, not work repos) + +## Acceptance Criteria + +- [ ] Task representation decided (Markdown + YAML) +- [ ] Home Kanban repo structure defined +- [ ] MCP tools implemented (add, update, list) +- [ ] Path whitelist enforcement (home repo only) +- [ ] Tools tested with dummy models +- [ ] Tools registered in MCP server + +## Technical Details + +Task format: +- Markdown files with YAML frontmatter +- Status: backlog, todo, in-progress, done +- Metadata: created, updated, assignee, etc. + +Tools: +- `add_task`: Create new task file +- `update_task_status`: Move between columns +- `list_tasks`: Query tasks by status/filter + +Safety: +- Path whitelist: only `family-agent-config/tasks/home/` +- Reject any paths containing "work" or work repo paths + +## Dependencies + +- TICKET-029 (MCP server) +- TICKET-002 (repo structure) + +## Related Files + +- `home-voice-agent/mcp-server/tools/tasks/` (to be created) +- `family-agent-config/tasks/home/` (to be created) + +## Notes + +Critical safety requirement: ensure tools only touch home repo. Test thoroughly. diff --git a/tickets/backlog/TICKET-035_notes-files-tools.md b/tickets/backlog/TICKET-035_notes-files-tools.md new file mode 100644 index 0000000..a81b057 --- /dev/null +++ b/tickets/backlog/TICKET-035_notes-files-tools.md @@ -0,0 +1,60 @@ +# Ticket: Notes & Files (Markdown, PDFs) + +## Ticket Information + +- **ID**: TICKET-035 +- **Title**: Notes & Files (Markdown, PDFs) +- **Type**: Feature +- **Priority**: Medium +- **Status**: Backlog +- **Track**: Tools/MCP +- **Milestone**: Milestone 3 - Memory, Reminders, Safety +- **Created**: 2024-01-XX + +## Description + +Implement notes and file tools: +- File indexing/search strategy (ripgrep + embeddings later) +- Start with basic full-text search + metadata +- MCP tools: search_notes, read_note, append_to_note, create_note +- PDF handling with lightweight text extractor + +## Acceptance Criteria + +- [ ] File indexing implemented +- [ ] Full-text search working +- [ ] MCP tools for notes (search, read, append, create) +- [ ] PDF text extraction working +- [ ] get_pdf_text tool implemented +- [ ] Tools registered in MCP server + +## Technical Details + +Search strategy: +- Phase 1: ripgrep for full-text search +- Phase 2: Add embeddings for semantic search +- Index: Markdown files, text files, PDFs + +PDF extraction: +- Use PyPDF2, pdfplumber, or similar +- Extract text, preserve structure if possible + +Tools: +- `search_notes`: Full-text search +- `read_note`: Read file content +- `append_to_note`: Add to existing note +- `create_note`: Create new note +- `get_pdf_text`: Extract PDF text + +## Dependencies + +- TICKET-029 (MCP server) +- Stable directory layout and backup strategy + +## Related Files + +- `home-voice-agent/mcp-server/tools/notes/` (to be created) + +## Notes + +Needs stable directory layout. Can be enhanced with embeddings later. diff --git a/tickets/backlog/TICKET-036_email-integration.md b/tickets/backlog/TICKET-036_email-integration.md new file mode 100644 index 0000000..8801643 --- /dev/null +++ b/tickets/backlog/TICKET-036_email-integration.md @@ -0,0 +1,57 @@ +# Ticket: Email Integration (Optional) + +## Ticket Information + +- **ID**: TICKET-036 +- **Title**: Email Integration (Optional) +- **Type**: Feature +- **Priority**: Low +- **Status**: Backlog +- **Track**: Tools/MCP +- **Milestone**: Milestone 4 - Optional Integrations +- **Created**: 2024-01-XX + +## Description + +Implement email tools (optional, after MVP): +- Tools: draft_email, send_email, list_recent_emails +- Confirmation gate before send +- IMAP/SMTP integration +- Security and authentication + +## Acceptance Criteria + +- [ ] Email tools implemented +- [ ] draft_email tool working +- [ ] send_email with confirmation +- [ ] list_recent_emails working +- [ ] Confirmation flow enforced +- [ ] Tools registered in MCP server + +## Technical Details + +Implementation: +- IMAP for reading emails +- SMTP for sending +- OAuth or app passwords for auth +- Confirmation: require explicit approval before send + +Tools: +- `draft_email`: Create email draft +- `send_email`: Send (with confirmation) +- `list_recent_emails`: List inbox +- `read_email`: Read specific email + +## Dependencies + +- TICKET-045 (confirmation flows) +- TICKET-029 (MCP server) +- Strong safety design + +## Related Files + +- `home-voice-agent/mcp-server/tools/email/` (to be created) + +## Notes + +High-risk tool. Requires strong confirmation flow. Do after MVP. diff --git a/tickets/backlog/TICKET-037_calendar-integration.md b/tickets/backlog/TICKET-037_calendar-integration.md new file mode 100644 index 0000000..92074a0 --- /dev/null +++ b/tickets/backlog/TICKET-037_calendar-integration.md @@ -0,0 +1,56 @@ +# Ticket: Calendar Integration (Optional) + +## Ticket Information + +- **ID**: TICKET-037 +- **Title**: Calendar Integration (Optional) +- **Type**: Feature +- **Priority**: Low +- **Status**: Backlog +- **Track**: Tools/MCP +- **Milestone**: Milestone 4 - Optional Integrations +- **Created**: 2024-01-XX + +## Description + +Implement calendar tools (optional, after MVP): +- Tools: list_events, create_event, modify_event +- Explicit confirmation required +- Calendar API integration (Google Calendar, CalDAV, etc.) +- Read-only and write operations + +## Acceptance Criteria + +- [ ] Calendar tools implemented +- [ ] list_events working +- [ ] create_event with confirmation +- [ ] modify_event with confirmation +- [ ] Confirmation flow enforced +- [ ] Tools registered in MCP server + +## Technical Details + +Implementation: +- Google Calendar API or CalDAV +- OAuth authentication +- Confirmation: require explicit approval for create/modify + +Tools: +- `list_events`: List upcoming events +- `create_event`: Create (with confirmation) +- `modify_event`: Update (with confirmation) +- `delete_event`: Delete (with confirmation) + +## Dependencies + +- TICKET-045 (confirmation flows) +- TICKET-029 (MCP server) +- Strong safety design + +## Related Files + +- `home-voice-agent/mcp-server/tools/calendar/` (to be created) + +## Notes + +High-risk tool. Requires strong confirmation flow. Do after MVP. diff --git a/tickets/backlog/TICKET-038_smart-home-integration.md b/tickets/backlog/TICKET-038_smart-home-integration.md new file mode 100644 index 0000000..96b7964 --- /dev/null +++ b/tickets/backlog/TICKET-038_smart-home-integration.md @@ -0,0 +1,58 @@ +# Ticket: Smart Home Integration (Optional) + +## Ticket Information + +- **ID**: TICKET-038 +- **Title**: Smart Home Integration (Optional) +- **Type**: Feature +- **Priority**: Low +- **Status**: Backlog +- **Track**: Tools/MCP +- **Milestone**: Milestone 4 - Optional Integrations +- **Created**: 2024-01-XX + +## Description + +Implement smart home tools (optional, after MVP): +- Abstract actions: set_scene, toggle_device, adjust_thermostat +- Map to Home Assistant or similar +- Confirmation for high-impact actions +- Device discovery and status + +## Acceptance Criteria + +- [ ] Smart home tools implemented +- [ ] set_scene tool working +- [ ] toggle_device tool working +- [ ] adjust_thermostat tool working +- [ ] Home Assistant integration +- [ ] Confirmation for high-impact actions +- [ ] Tools registered in MCP server + +## Technical Details + +Implementation: +- Home Assistant API integration +- Device abstraction layer +- Confirmation: require approval for scene changes, thermostat + +Tools: +- `set_scene`: Activate scene +- `toggle_device`: Turn on/off device +- `adjust_thermostat`: Change temperature +- `list_devices`: List available devices +- `get_device_status`: Get device state + +## Dependencies + +- TICKET-045 (confirmation flows) +- TICKET-029 (MCP server) +- Home Assistant or similar setup + +## Related Files + +- `home-voice-agent/mcp-server/tools/smart-home/` (to be created) + +## Notes + +High-risk tool. Requires strong confirmation flow. Do after MVP. diff --git a/tickets/backlog/TICKET-039_phone-client-pwa.md b/tickets/backlog/TICKET-039_phone-client-pwa.md new file mode 100644 index 0000000..0022cc9 --- /dev/null +++ b/tickets/backlog/TICKET-039_phone-client-pwa.md @@ -0,0 +1,59 @@ +# Ticket: Phone-Friendly Client (PWA or Native) + +## Ticket Information + +- **ID**: TICKET-039 +- **Title**: Phone-Friendly Client (PWA or Native) +- **Type**: Feature +- **Priority**: High +- **Status**: Backlog +- **Track**: Clients/UI +- **Milestone**: Milestone 2 - Voice Chat MVP +- **Created**: 2024-01-XX + +## Description + +Build phone-friendly client: +- Decide PWA vs native (PWA likely: microphone access, push notifications, WebSocket) +- Voice capture UI (tap-to-talk + optional wake-word) +- Stream audio to ASR endpoint +- Conversation view (history, agent responses, tasks created) +- Text + play/pause TTS + +## Acceptance Criteria + +- [ ] PWA or native app decision made +- [ ] Voice capture UI implemented +- [ ] Audio streaming to ASR working +- [ ] Conversation view implemented +- [ ] TTS playback working +- [ ] Task display working + +## Technical Details + +PWA approach: +- Service worker for offline support +- WebSocket for real-time communication +- getUserMedia for microphone access +- Push notifications for reminders/timers + +Features: +- Tap-to-talk button +- Wake-word option (if browser supports) +- Conversation history +- Audio playback controls +- Task list view + +## Dependencies + +- TICKET-010 (ASR endpoint) +- TICKET-014 (TTS service) +- Can be mocked early for UI development + +## Related Files + +- `home-voice-agent/clients/phone/` (to be created) + +## Notes + +Independent of MCP tools - only needs chat endpoint to start. Can begin with mocks. diff --git a/tickets/backlog/TICKET-040_web-dashboard.md b/tickets/backlog/TICKET-040_web-dashboard.md new file mode 100644 index 0000000..38c2602 --- /dev/null +++ b/tickets/backlog/TICKET-040_web-dashboard.md @@ -0,0 +1,62 @@ +# Ticket: Web LAN Dashboard + +## Ticket Information + +- **ID**: TICKET-040 +- **Title**: Web LAN Dashboard +- **Type**: Feature +- **Priority**: Medium +- **Status**: Backlog +- **Track**: Clients/UI +- **Milestone**: Milestone 2 - Voice Chat MVP +- **Created**: 2024-01-XX + +## Description + +Build web LAN dashboard: +- Simple web interface showing: + - Current conversation + - Recent tasks + - Reminders +- Admin panel: + - Pause family agent + - View logs + - Revoke tokens + - Kill MCP servers +- Home project board view (read-only Kanban) + +## Acceptance Criteria + +- [ ] Web dashboard implemented +- [ ] Conversation view working +- [ ] Task list view working +- [ ] Reminder display working +- [ ] Admin panel functional +- [ ] Home Kanban board view (read-only) + +## Technical Details + +Implementation: +- Simple web framework (Flask, FastAPI, or static + API) +- WebSocket for real-time updates +- Admin authentication +- Read-only access to Kanban board + +Features: +- Dashboard: conversation, tasks, reminders +- Admin: controls, logs, token management +- Board view: Kanban visualization + +## Dependencies + +- TICKET-034 (home tasks tools) +- TICKET-033 (reminders) +- TICKET-024 (logging) - for admin panel + +## Related Files + +- `home-voice-agent/clients/web-dashboard/` (to be created) + +## Notes + +Needs at least basic MCP tools for tasks and reminders. Admin panel requires log format. diff --git a/tickets/backlog/TICKET-041_long-term-memory-design.md b/tickets/backlog/TICKET-041_long-term-memory-design.md new file mode 100644 index 0000000..911e4a9 --- /dev/null +++ b/tickets/backlog/TICKET-041_long-term-memory-design.md @@ -0,0 +1,54 @@ +# Ticket: Long-Term Memory Design + +## Ticket Information + +- **ID**: TICKET-041 +- **Title**: Long-Term Memory Design +- **Type**: Design +- **Priority**: High +- **Status**: Backlog +- **Track**: Safety/Memory +- **Milestone**: Milestone 3 - Memory, Reminders, Safety +- **Created**: 2024-01-XX + +## Description + +Design long-term memory system: +- Schema for personal memory (facts, preferences, routines) +- Timestamps and confidence scores +- Memory write policy (when agent can persist) +- Retrieval strategy (light retrieval over structured store) + +## Acceptance Criteria + +- [ ] Memory schema designed +- [ ] Data model documented +- [ ] Write policy defined +- [ ] Retrieval strategy documented +- [ ] Integration points identified + +## Technical Details + +Memory schema: +- Facts: names, preferences, routines +- Metadata: timestamp, confidence, source +- Categories: personal, family, preferences, routines + +Storage: +- SQLite or lightweight DB +- Indexed for fast retrieval +- Backup strategy + +## Dependencies + +- TICKET-025 (system prompts) - for integration +- TICKET-004 (architecture) + +## Related Files + +- `docs/MEMORY_DESIGN.md` (to be created) +- `home-voice-agent/memory/` (to be created) + +## Notes + +Needs LLM prompting framework. Independent from ASR/TTS and client work. diff --git a/tickets/backlog/TICKET-042_memory-implementation.md b/tickets/backlog/TICKET-042_memory-implementation.md new file mode 100644 index 0000000..3b576ce --- /dev/null +++ b/tickets/backlog/TICKET-042_memory-implementation.md @@ -0,0 +1,51 @@ +# Ticket: Long-Term Memory Implementation + +## Ticket Information + +- **ID**: TICKET-042 +- **Title**: Long-Term Memory Implementation +- **Type**: Feature +- **Priority**: High +- **Status**: Backlog +- **Track**: Safety/Memory +- **Milestone**: Milestone 3 - Memory, Reminders, Safety +- **Created**: 2024-01-XX + +## Description + +Implement long-term memory system: +- Create memory storage (DB schema) +- Implement write operations (with policy enforcement) +- Implement retrieval operations +- Integrate into prompts as separate section + +## Acceptance Criteria + +- [ ] Memory storage implemented +- [ ] Write operations working (with policy) +- [ ] Retrieval operations working +- [ ] Integration into prompts working +- [ ] Memory injection into LLM context +- [ ] Basic testing completed + +## Technical Details + +Implementation: +- SQLite database with schema +- API for read/write operations +- Policy enforcement (confirmation, heuristics) +- Prompt integration: retrieve relevant memories, inject into system prompt + +## Dependencies + +- TICKET-041 (memory design) +- TICKET-025 (system prompts) +- TICKET-021 or TICKET-022 (LLM server) + +## Related Files + +- `home-voice-agent/memory/` (to be created) + +## Notes + +Can be implemented after first working chat. Integrates with conversation handling. diff --git a/tickets/backlog/TICKET-043_conversation-summarization.md b/tickets/backlog/TICKET-043_conversation-summarization.md new file mode 100644 index 0000000..b62f79b --- /dev/null +++ b/tickets/backlog/TICKET-043_conversation-summarization.md @@ -0,0 +1,50 @@ +# Ticket: Conversation Summarization & Pruning + +## Ticket Information + +- **ID**: TICKET-043 +- **Title**: Conversation Summarization & Pruning +- **Type**: Feature +- **Priority**: Medium +- **Status**: Backlog +- **Track**: Safety/Memory +- **Milestone**: Milestone 3 - Memory, Reminders, Safety +- **Created**: 2024-01-XX + +## Description + +Implement conversation management: +- Session store design (local DB/files, retention duration) +- Summarization routines (after N turns or size threshold) +- Use work model or family model for summarization +- Privacy and deletion tools + +## Acceptance Criteria + +- [ ] Session store implemented +- [ ] Summarization working (scheduled or threshold-based) +- [ ] Context pruning functional +- [ ] Privacy deletion tools implemented +- [ ] Retention policy enforced + +## Technical Details + +Implementation: +- Store transcripts in DB or files +- Summarization: use LLM to condense old messages +- Pruning: remove old messages, keep summary +- Deletion: UI and admin tools to delete sessions + +## Dependencies + +- TICKET-027 (multi-turn conversation) +- TICKET-021 or TICKET-022 (LLM services) +- TICKET-024 (logging) + +## Related Files + +- `home-voice-agent/conversation/summarization/` (to be created) + +## Notes + +Can be implemented after first working chat. Important for long conversations. diff --git a/tickets/backlog/TICKET-044_boundary-enforcement.md b/tickets/backlog/TICKET-044_boundary-enforcement.md new file mode 100644 index 0000000..2344ad8 --- /dev/null +++ b/tickets/backlog/TICKET-044_boundary-enforcement.md @@ -0,0 +1,52 @@ +# Ticket: Boundary Enforcement + +## Ticket Information + +- **ID**: TICKET-044 +- **Title**: Boundary Enforcement +- **Type**: Feature +- **Priority**: High +- **Status**: Backlog +- **Track**: Safety/Memory +- **Milestone**: Milestone 3 - Memory, Reminders, Safety +- **Created**: 2024-01-XX + +## Description + +Implement boundary enforcement: +- Separate credentials and config (different .env files, service accounts, key stores) +- Network-level separation (dedicated containers/namespaces, firewall rules) +- Prevent family agent from reaching work repos +- Static policy checks (lint/CI rules rejecting cross-access merges) + +## Acceptance Criteria + +- [ ] Separate credentials/config for family vs work +- [ ] Network separation implemented +- [ ] Firewall rules preventing cross-access +- [ ] Static policy checks (lint/CI) +- [ ] Family agent cannot access work repos +- [ ] Policy violations caught automatically + +## Technical Details + +Separation strategies: +- Config: separate .env files, key stores +- Network: containers, namespaces, VLANs +- Firewall: block family agent from work repo paths +- CI: lint rules checking for cross-access code + +## Dependencies + +- TICKET-002 (repo structure) +- TICKET-003 (safety constraints) +- TICKET-004 (architecture) + +## Related Files + +- `home-voice-agent/safety/boundaries/` (to be created) +- `.github/workflows/policy-check.yml` (to be created) + +## Notes + +Can proceed in parallel with most tool work. Critical for safety. diff --git a/tickets/backlog/TICKET-045_confirmation-flows.md b/tickets/backlog/TICKET-045_confirmation-flows.md new file mode 100644 index 0000000..ff3f2b3 --- /dev/null +++ b/tickets/backlog/TICKET-045_confirmation-flows.md @@ -0,0 +1,60 @@ +# Ticket: Confirmation Flows + +## Ticket Information + +- **ID**: TICKET-045 +- **Title**: Confirmation Flows +- **Type**: Feature +- **Priority**: High +- **Status**: Backlog +- **Track**: Safety/Memory +- **Milestone**: Milestone 3 - Memory, Reminders, Safety +- **Created**: 2024-01-XX + +## Description + +Implement confirmation flows for high-risk actions: +- Risk classification (high-impact actions list) +- Confirmation UX (text and voice prompts) +- Agent explains action, asks for "Yes/No" +- Log decisions +- Enforcement at tool level (signed confirmation token from client) + +## Acceptance Criteria + +- [ ] Risk classification system implemented +- [ ] High-impact actions identified and documented +- [ ] Confirmation UX designed (text + voice) +- [ ] Confirmation flow working +- [ ] Decision logging implemented +- [ ] Tool-level enforcement (confirmation token) + +## Technical Details + +High-impact actions: +- Send email +- Calendar changes +- File edits outside "safe" areas +- Smart-home actions +- High-value task changes + +Confirmation flow: +1. Agent proposes action +2. User confirms (voice or UI) +3. Signed token generated +4. Tool validates token before execution +5. Action logged + +## Dependencies + +- TICKET-026 (tool-calling policy) +- TICKET-029 (MCP server) +- TICKET-039 (phone client) or TICKET-040 (web dashboard) + +## Related Files + +- `home-voice-agent/safety/confirmations/` (to be created) + +## Notes + +Ties into clients for presenting confirmations. Critical for safety. diff --git a/tickets/backlog/TICKET-046_admin-tools.md b/tickets/backlog/TICKET-046_admin-tools.md new file mode 100644 index 0000000..2829b76 --- /dev/null +++ b/tickets/backlog/TICKET-046_admin-tools.md @@ -0,0 +1,50 @@ +# Ticket: Admin Tools + +## Ticket Information + +- **ID**: TICKET-046 +- **Title**: Admin Tools +- **Type**: Feature +- **Priority**: Medium +- **Status**: Backlog +- **Track**: Safety/Memory +- **Milestone**: Milestone 3 - Memory, Reminders, Safety +- **Created**: 2024-01-XX + +## Description + +Build admin tools: +- Log browser (web UI for searching requests, tool calls, errors) +- Kill switches (buttons/CLI commands to shut down: family agent, specific tools, or whole MCP server) +- Access revocation (system to disable compromised tokens/devices) + +## Acceptance Criteria + +- [ ] Log browser implemented +- [ ] Search functionality working +- [ ] Kill switches functional (family agent, tools, MCP server) +- [ ] Access revocation working +- [ ] Admin authentication implemented +- [ ] UI integrated into web dashboard + +## Technical Details + +Implementation: +- Log browser: search interface, filters, export +- Kill switches: API endpoints + UI buttons +- Access revocation: token blacklist, device management + +## Dependencies + +- TICKET-024 (logging) +- TICKET-040 (web dashboard) +- Authentication system + +## Related Files + +- `home-voice-agent/admin/` (to be created) +- `home-voice-agent/clients/web-dashboard/admin/` (to be created) + +## Notes + +Can be done after basic MVP. Important for operational control. diff --git a/tickets/backlog/TICKET-047_hardware-purchases.md b/tickets/backlog/TICKET-047_hardware-purchases.md new file mode 100644 index 0000000..3ead1af --- /dev/null +++ b/tickets/backlog/TICKET-047_hardware-purchases.md @@ -0,0 +1,56 @@ +# Ticket: Hardware & Purchases + +## Ticket Information + +- **ID**: TICKET-047 +- **Title**: Hardware & Purchases +- **Type**: Planning +- **Priority**: Medium +- **Status**: Backlog +- **Track**: Project Setup +- **Milestone**: Various +- **Created**: 2024-01-XX + +## Description + +Plan and purchase required hardware: + +**Must-have / likely:** +- 1-2 high-quality USB microphones or small array mic (living room/office) +- Small always-on node (Pi/NUC/old SFF PC) if not reusing existing hardware +- Storage (additional SSD/HDD for logs, transcripts, note archives) +- Network gear (extra Ethernet runs or cheap PoE switch) if needed +- Headset or dedicated mic for desk usage + +**Nice-to-have:** +- Dedicated low-power box for 1050 "family agent" +- UPS for servers (4080/1050) to avoid abrupt shutdowns +- Small tablet or wall-mounted screen for LAN dashboard + +## Acceptance Criteria + +- [ ] Hardware requirements documented +- [ ] Purchase list created +- [ ] Must-have items acquired +- [ ] Hardware tested and integrated +- [ ] Nice-to-have items prioritized + +## Technical Details + +Hardware specifications: +- Microphones: USB, array mic, or headset +- Always-on node: Raspberry Pi 4+, NUC, or SFF PC +- Storage: SSD for logs (500GB+), HDD for archives +- Network: PoE switch if needed for mic nodes + +## Dependencies + +None - can be done in parallel with software development. + +## Related Files + +- `docs/HARDWARE.md` (to be created) + +## Notes + +Some hardware can be acquired as needed. Microphones and always-on node are critical for MVP. diff --git a/tickets/done/.gitkeep b/tickets/done/.gitkeep new file mode 100644 index 0000000..b384c3d --- /dev/null +++ b/tickets/done/.gitkeep @@ -0,0 +1,3 @@ +# Done + +Completed items. diff --git a/tickets/in-progress/.gitkeep b/tickets/in-progress/.gitkeep new file mode 100644 index 0000000..29f5497 --- /dev/null +++ b/tickets/in-progress/.gitkeep @@ -0,0 +1,3 @@ +# In Progress + +Items currently being worked on. diff --git a/tickets/review/.gitkeep b/tickets/review/.gitkeep new file mode 100644 index 0000000..83364f9 --- /dev/null +++ b/tickets/review/.gitkeep @@ -0,0 +1,3 @@ +# Review + +Items awaiting review or testing. diff --git a/tickets/todo/.gitkeep b/tickets/todo/.gitkeep new file mode 100644 index 0000000..a63d4b9 --- /dev/null +++ b/tickets/todo/.gitkeep @@ -0,0 +1,3 @@ +# Todo + +Items ready to be worked on. diff --git a/tickets/todo/TICKET-001_project-setup.md b/tickets/todo/TICKET-001_project-setup.md new file mode 100644 index 0000000..bac9ac7 --- /dev/null +++ b/tickets/todo/TICKET-001_project-setup.md @@ -0,0 +1,57 @@ +# Ticket: Project Setup + +## Ticket Information + +- **ID**: TICKET-001 +- **Title**: Initial Project Setup and Structure +- **Type**: Setup +- **Priority**: High +- **Status**: Done +- **Created**: 2024-01-XX +- **Assigned**: [Your Name] + +## Description + +Set up the initial project structure including: +- Cursor rules configuration +- Architecture documentation +- README with project overview +- Kanban ticket system with folder structure +- Git ignore file + +## Acceptance Criteria + +- [x] `.cursorrules` file created with development guidelines +- [x] `README.md` with project overview and workflow +- [x] `ARCHITECTURE.md` template created +- [x] `tickets/` directory structure with all status folders +- [x] Ticket template created +- [x] `.gitignore` file configured +- [x] Documentation explains kanban workflow + +## Technical Details + +Created a comprehensive project structure that supports: +- Kanban-based workflow with Vibe Kanban integration +- Organized ticket management in folder structure +- Clear documentation for onboarding and development + +## Dependencies + +None - this is the initial setup. + +## Related Files + +- `.cursorrules` +- `README.md` +- `ARCHITECTURE.md` +- `tickets/README.md` +- `tickets/TICKET_TEMPLATE.md` + +## Notes + +This ticket serves as an example of the ticket format. Future tickets should follow the same structure. + +## Progress Log + +- 2024-01-XX - Project structure created and documented