ilia/crkl
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge pull request 'Ship Circle 1.16.2' (#4) from release/1.16.2-circle into main
CI / skip-ci-check (push) Successful in 34s
Details
CI / secret-scan (push) Successful in 29s
Details

Browse Source
This commit was merged in pull request #4.
This commit is contained in:
ilia
2026-08-05 11:31:28 -05:00
parent 80870d6adb 511d1ae329
commit 10317c9ae9
116 changed files with 7561 additions and 715 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.env.example
+9
View File
@@ -0,0 +1,9 @@
# Copy to .env (gitignored) for local notes — do NOT commit secrets.
# Paste token into the app: Crkl → Integrations.
VIKUNJA_URL=https://todo.levkin.ca
VIKUNJA_TOKEN=
VIKUNJA_PROJECT_ID=13
# Optional default mailto recipient (“email me the list”)
CRKL_EMAIL_TO=
.gitignore
+8 -2
View File
@@ -12,6 +12,9 @@ build/
# Local configuration
local.properties
.env
.env.*
!.env.example
# OS files
.DS_Store
@@ -38,5 +41,8 @@ captures/
gradle-*.zip
gradle-*/
# User-specific
.android/
# Models (pushed to device; never commit weights)
*.task
*.litertlm
*.gguf
models/
.notes/directory_structure.md
+25 -28
View File
@@ -1,32 +1,29 @@
# Directory Structure
/app/src/main/kotlin/com/example/crkl/
/accessibility/ # Accessibility overlay and services
# - CrklAccessibilityService.kt (main service)
# - OverlayView.kt (touch capture overlay)
/gesture/ # Gesture tracking, region extraction (TODO)
/model/ # STT/LLM wrappers, inference runners (TODO)
/vision/ # Content classification and ML components (TODO)
/agent/ # Dialogue state management (TODO)
/ui/ # Jetpack Compose overlays and feedback UIs
# - theme/ (Theme.kt, Type.kt)
/privacy/ # Data/cache handling, controls (TODO)
MainActivity.kt # Main activity for settings/permissions
/app/src/main/res/
/values/ # strings.xml, colors.xml, themes.xml
/xml/ # accessibility_service_config.xml
/mipmap-*/ # App launcher icons
/app/src/test/ # Unit tests (TODO)
/app/src/androidTest/ # Android instrumentation tests (TODO)
/tests/ # Additional test utilities
Root config files:
- build.gradle.kts (root)
- settings.gradle.kts
- gradle.properties
- local.properties
- app/build.gradle.kts
/accessibility/
- CrklAccessibilityService.kt # overlay + selection + assist pipeline
- OverlayView.kt # FAB ↔ full-screen draw overlay
/vision/
- RegionContentExtractor.kt # a11y node text in selection bounds
/model/
- LocalLlm.kt # LLM interface
- MediaPipeLocalLlm.kt # MediaPipe tasks-genai backend
- ModelPaths.kt # on-device .task discovery
/agent/
- AssistEngine.kt # LLM or stub routing
- LocalAssistStub.kt # no-model fallback formatter
/gesture/ # (empty) richer gesture recognition later
/ui/
- ResultPanelView.kt # selection result card overlay
- theme/
/privacy/ # (empty)
/fixtures/
- FixtureCatalog.kt # email/image/video/audio test targets
- TestFixturesActivity.kt # scrollable playground for circle demos
MainActivity.kt
/app/src/test/java/com/example/crkl/
/agent/AssistEngineTest.kt
/agent/LocalAssistStubTest.kt
/model/ModelPathsTest.kt
.notes/meeting_notes.md
+32
View File
@@ -1,5 +1,37 @@
# Meeting Notes
## 2026-07-28 - MediaPipe LLM path
**Decision:** one backend only — MediaPipe `tasks-genai` + Gemma 3 1B `.task`.
No STT/OCR/dialogue yet.
**Shipped:**
- `LocalLlm` + `MediaPipeLocalLlm`
- `AssistEngine` (LLM → stub fallback with `make push-model` hint)
- Model discovery at `/data/local/tmp/llm/crkl.task`
- Makefile `push-model` / `model-status`
- Warm-load on accessibility service connect
**Ops note:** model weights stay off-git; push to device. High-end phones only for reliable inference.
---
## 2026-07-28 - Vertical slice: extract text
**Unstuck decision:** stop boiling the ocean (STT + vision + dialogue + 5 LLM options).
Ship one path end-to-end first.
**Shipped:**
- Circle selection reports screen-space bounds
- `RegionContentExtractor` walks a11y windows/nodes intersecting the circle
- `ResultPanelView` shows extracted text
- `LocalAssistStub` formats a local response (explicitly not an LLM yet)
**Still blocked for "real AI":** pick a single on-device LLM integration next.
Do not start OCR / STT / dialogue memory until that loop works on a phone.
---
## 2025-10-15 - Session 4: Testing & Makefile
**Completed:**
.notes/project_overview.md
+12 -9
View File
@@ -1,13 +1,16 @@
# Crkl Project Overview
# Project overview — Circle (crkl)
Crkl is an on-device, privacy-first Android agent that allows users to circle or touch any element on the screen and get instant local-AI powered help: summarization, transcription, or suggestions based on the content type. All ML inference and data handling are strictly local.
**Circle** is an on-device, privacy-first Android assist: circle on-screen text, then Translate / Copy / Explain / Share / Vikunja. Repo and package remain `crkl` / `com.example.crkl`.
Historical notes below may still say “Crkl” as the original project name.
Current working path: circle → accessibility text extract → MediaPipe LLM summary (stub if no `.task` model).
Major Modules:
- Accessibility service overlay
- Gesture/region processor
- Content type detection
- Local STT/LLM/Vision models
- Dialogue agent with persistent context
- Compose overlay UI
- Privacy/data controls
- Accessibility service overlay (working)
- Region text extractor via a11y tree (working)
- MediaPipe LocalLlm + AssistEngine (working; model pushed separately)
- Gesture/region processor (basic closed-stroke only)
- Content type detection / STT / vision OCR / dialogue (not started)
- Privacy/data controls (not started)
.notes/task_list.md
+20 -9
View File
@@ -1,12 +1,23 @@
# Task List
- [ ] Implement Accessibility Overlay Service
- [ ] Develop Gesture-to-Region Processor
- [ ] Integrate Content-Type Detector
- [ ] Set up local Speech-to-Text (Vosk/PocketSphinx/DeepSpeech)
- [ ] Integrate local LLM module (MLC Chat, SmolChat, etc.)
- [ ] Build Compose Overlay UI
- [ ] Implement Dialogue Memory
- [ ] Build Privacy Controls
- [ ] Write tests and documentation for all modules
## Done (shipped through 1.16.2)
- [x] Accessibility overlay + floating C / QS tile / a11y button
- [x] Closed-stroke selection → a11y + OCR extract
- [x] Result bottom sheet: Translate / Copy / Explain / Share / Vikunja
- [x] MediaPipe on-device LLM path + stub fallback
- [x] Fixtures VIP demo + dogfood / DEMO docs
- [x] Circle brand (display name + lasso logo)
- [x] `make test` + `make smoke` (emulator)
## Next (keep narrow — post-freeze)
- [ ] Stream tokens into the result panel (partial updates)
- [ ] Better gesture quality (ellipse fit, reject scribble, undo)
- [ ] Tap-to-select nearest text node (no full circle required)
- [ ] Record 45s demo per `docs/DEMO.md`
## Later
- [ ] Offline STT (Vosk)
- [ ] Dialogue memory across selections
- [ ] Privacy controls / clear cache UI
- [ ] Migrate MediaPipe → LiteRT-LM when ready
- [ ] Broader device OEM testing
ARCHITECTURE.md
+5 -2
View File
@@ -1,4 +1,7 @@
# Technical Architecture
# Technical Architecture — Circle (crkl)
Product: circle on-screen text → on-device assist → Translate / Copy / Explain / Share / Vikunja.
Code lives under `com.example.crkl`; user-facing name is **Circle**.
## Layered System Modules
@@ -76,7 +79,7 @@ Local-only data handling, cache and session controls.
- Session cache controls
- Privacy settings
- Data retention policies
- No network call enforcement
- Optional outbound integrations only when configured: Vikunja HTTPS (user token), mailto intent, device CalendarContract. Core circle → OCR → LLM stays on-device. No host bridges.
## Dataflow Diagram
CHANGELOG.md
+52
View File
@@ -0,0 +1,52 @@
# Changelog
## 1.16.2-circle-copy — 2026-08-05
- More user-facing strings say **Circle** (panel, empty state, Integrations label, Vikunja hint)
- Docs pass: README, PHONE_SETUP, shortcuts, dogfood, marketing, RELEASE, docs/README index
- `docs/RELEASE.md` install/tag checklist
- `make smoke` + macOS launchd emulator helper (agent sandbox was killing qemu)
- Emulator smoke verified: FAB → circle → extract → Copy
## 1.16.1-lasso — 2026-08-05
- Launcher + QS tile use **logo 2** (teal lasso + ink C on paper)
## 1.16.0-circle-brand — 2026-08-05
- Display name **Circle**; Accessibility service **Circle Overlay**
- Launcher adaptive icon: ink + teal ring + white C (logo concept 1)
- Brand colors aligned (teal primary, ink background)
## 1.15.0-ship — 2026-08-05
Ship freeze candidate for the Circle Assist VIP loop.
### Week 1 (usability)
- Clean result panel (debug meta optional)
- Accessibility-off banner; honest empty / image-only copy
- Ink + teal flow UI; bottom sheet; five chips
- QS **Circle** tile + Accessibility button shortcuts
### Week 1 (finish)
- Fixtures VIP demo script (`FixtureCatalog.demoScript`, grocery list fixture)
- Integrations: Vikunja + Translate first; appearance / advanced collapsed
### Week 2 (ship story)
- `docs/DEMO.md`, `docs/dogfood.md`, `docs/non-goals.md`, `docs/marketing.md`
- Three logo concepts generated for brand pick
### Not in this release
See `docs/non-goals.md` (Lens, screenshot SystemUI, gog-required, calendar create, …).
## 1.14.0-shortcuts
Quick Settings tile, Accessibility button, shortcuts docs.
## 1.13.0-flow
Seamless overlay + bottom sheet UI.
## 1.12.0-usability
A11y banner, hide debug meta, empty states.
## 1.11.0-circle-assist
Copy, Explain, Share polish, onboarding.
CONTRIBUTING.md
+10 -13
View File
@@ -1,33 +1,30 @@
# Contribution Guide
Thank you for your interest in contributing to Crkl! This document provides guidelines and information for contributors.
Thank you for contributing to **Circle** (repo `crkl`).
## Getting Started
### Prerequisites
- Android Studio (latest stable version)
- JDK 11 or higher
- JDK 17 (see Makefile / Homebrew `openjdk@17`)
- Git
- Basic knowledge of Kotlin and Android development
- Familiarity with Jetpack Compose
- Kotlin + Jetpack Compose familiarity
### Development Setup
1. **Clone the repository:**
```bash
git clone https://github.com/yourusername/crkl.git
git clone https://git.levkin.ca/ilia/crkl.git
cd crkl
```
2. **Open in Android Studio:**
- Open Android Studio
- Select "Open an Existing Project"
- Navigate to the cloned directory
2. **Open in Android Studio** (or use `./gradlew` + Makefile).
3. **Review project documentation:**
- Read `README.md` for project overview
- Review `ARCHITECTURE.md` for technical details
3. **Review documentation:**
- [README.md](README.md) — overview
- [docs/](docs/) — ship, shortcuts, brand
- [ARCHITECTURE.md](ARCHITECTURE.md) — technical layers
- Check `.notes/` directory for current project context
- Review `CURSOR_SUPPORT.md` if using Cursor editor
@@ -197,5 +194,5 @@ Contributors will be recognized in:
- Release notes
- Project documentation
Thank you for contributing to Crkl!
Thank you for contributing to Circle!
CURSOR_SUPPORT.md
+4 -3
View File
@@ -166,9 +166,10 @@ Before submitting any code:
## PROJECT STATUS
**Current Phase:** Initial Setup
**Next Steps:** Begin Milestone 1 POC implementation
**Priority:** Accessibility Service overlay and gesture recognition
**Current Phase:** Milestone 1 complete enough to demo (gesture → text → local LLM/stub)
**Working:** Overlay, selection, a11y extract, MediaPipe LLM when model present, stub fallback
**Next Steps:** Token streaming in result panel; optional tap-to-select; first-run model setup UI
**Priority:** Validate on a real phone with `make push-model` before adding OCR/STT
## USEFUL REFERENCES
Makefile
+201 -101
View File
@@ -1,128 +1,228 @@
# Crkl - Physical Phone Development
.PHONY: help setup-sdk build install run stop logs clean uninstall devices check-device
# Crkl - local Android build / test / emulator
.PHONY: help doctor setup-mac setup-sdk setup-emulator build test install run stop logs clean uninstall \
devices check-device push-model model-status emulator emulator-stop emulator-list wait-emulator demo \
fixtures home back recents test-env integrations gog-bridge smoke
ANDROID_HOME ?= $(HOME)/android-sdk
ADB := $(ANDROID_HOME)/platform-tools/adb
UNAME_S := $(shell uname -s)
GRADLEW := ./gradlew
APK := app/build/outputs/apk/debug/app-debug.apk
MODEL_DEVICE_PATH := /data/local/tmp/llm/crkl.task
MODEL ?=
AVD_NAME ?= CrklEmulator
ifeq ($(UNAME_S),Darwin)
BREW_PREFIX := $(shell brew --prefix 2>/dev/null || echo /opt/homebrew)
JAVA_HOME ?= $(BREW_PREFIX)/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home
ifeq ($(wildcard $(JAVA_HOME)/bin/java),)
JAVA_HOME := $(BREW_PREFIX)/opt/openjdk@17
endif
ANDROID_HOME ?= $(BREW_PREFIX)/share/android-commandlinetools
AVD_ABI := google_apis;arm64-v8a
EMULATOR_GPU ?= host
else
JAVA_HOME ?= $(JAVA_HOME)
ANDROID_HOME ?= $(HOME)/android-sdk
AVD_ABI := google_apis;x86_64
EMULATOR_GPU ?= swiftshader_indirect
endif
export JAVA_HOME
export ANDROID_HOME
export ANDROID_SDK_ROOT := $(ANDROID_HOME)
ADB := $(ANDROID_HOME)/platform-tools/adb
EMULATOR := $(ANDROID_HOME)/emulator/emulator
SDKMANAGER := $(shell \
if [ -x "$(ANDROID_HOME)/cmdline-tools/latest/bin/sdkmanager" ]; then \
echo "$(ANDROID_HOME)/cmdline-tools/latest/bin/sdkmanager"; \
elif command -v sdkmanager >/dev/null 2>&1; then command -v sdkmanager; \
else echo sdkmanager; fi)
AVDMANAGER := $(shell \
if [ -x "$(ANDROID_HOME)/cmdline-tools/latest/bin/avdmanager" ]; then \
echo "$(ANDROID_HOME)/cmdline-tools/latest/bin/avdmanager"; \
elif command -v avdmanager >/dev/null 2>&1; then command -v avdmanager; \
else echo avdmanager; fi)
PATH := $(JAVA_HOME)/bin:$(ANDROID_HOME)/platform-tools:$(ANDROID_HOME)/emulator:$(ANDROID_HOME)/cmdline-tools/latest/bin:$(PATH)
export PATH
help: ## Show available commands
@echo "Crkl - Android Development Commands"
@echo "Crkl - Android Development Commands ($(UNAME_S))"
@echo ""
@echo "Setup Commands:"
@echo " make setup-sdk - Install Android SDK and tools"
@echo " make setup-emulator - Create and start Android emulator"
@echo " make check-device - Check if device/emulator is connected"
@echo " make devices - List connected devices/emulators"
@echo "First-time on this Mac:"
@echo " make setup-mac - brew JDK17 + Android SDK + AVD + local.properties"
@echo " make doctor - verify toolchain"
@echo " make test - run JVM unit tests (no phone needed)"
@echo " make smoke - on-device: FAB → circle → extract → Copy"
@echo " make test-env - ONE SHOT: fix chrome, install, open fixtures playground"
@echo " make demo - build + boot emulator + install + launch"
@echo " make fixtures - open email/image/video/audio test screen"
@echo " make integrations - Vikunja token + email To + calendar permission"
@echo ""
@echo "Emulator Commands:"
@echo " make emulator - Start Android emulator"
@echo " make emulator-stop - Stop emulator"
@echo " make emulator-list - List available emulators"
@echo "Emulator chrome ◀○□ broken? test-env restarts with -grpc (fixes JWT lock)."
@echo " Always click the phone screen once before using rim buttons."
@echo ""
@echo "Development Commands:"
@echo " make build - Build APK"
@echo " make install - Install to device/emulator"
@echo " make run - Launch app"
@echo " make logs - Watch app logs"
@echo " make stop - Stop app"
@echo " make clean - Clean build"
@echo " make uninstall - Remove app"
@echo "Everyday:"
@echo " make build / test / install / run / logs / stop"
@echo " make emulator / emulator-stop / check-device"
@echo " make push-model MODEL=/path/to/model.task"
@echo ""
@echo "ANDROID_HOME=$(ANDROID_HOME)"
@echo "JAVA_HOME=$(JAVA_HOME)"
@echo ""
setup-sdk: ## Install Android SDK and tools
@echo "Setting up Android SDK..."
@mkdir -p $(ANDROID_HOME)
@cd $(ANDROID_HOME) && \
if [ ! -f cmdline-tools/latest/bin/sdkmanager ]; then \
echo "Downloading Android SDK command-line tools..."; \
wget -q https://dl.google.com/android/repository/commandlinetools-linux-11076708_latest.zip; \
unzip -q commandlinetools-linux-11076708_latest.zip; \
mkdir -p cmdline-tools/latest; \
mv cmdline-tools/* cmdline-tools/latest/ 2>/dev/null || true; \
rm commandlinetools-linux-11076708_latest.zip; \
fi
@echo "Accepting SDK licenses..."
@export ANDROID_HOME=$(ANDROID_HOME) && \
export PATH=$(ANDROID_HOME)/cmdline-tools/latest/bin:$(ANDROID_HOME)/platform-tools:$$PATH && \
yes | sdkmanager --licenses > /dev/null 2>&1
@echo "Installing platform tools and build tools..."
@export ANDROID_HOME=$(ANDROID_HOME) && \
export PATH=$(ANDROID_HOME)/cmdline-tools/latest/bin:$(ANDROID_HOME)/platform-tools:$$PATH && \
sdkmanager "platform-tools" "platforms;android-34" "build-tools;34.0.0" > /dev/null 2>&1
@echo "✓ Android SDK setup complete!"
@echo "Add to your ~/.bashrc or ~/.zshrc:"
@echo " export ANDROID_HOME=$(ANDROID_HOME)"
@echo " export PATH=\$$ANDROID_HOME/platform-tools:\$$PATH"
doctor: ## Verify JDK + Android SDK are usable here
@echo "== Java =="
@java -version
@echo "== ANDROID_HOME =="
@echo "$(ANDROID_HOME)"
@test -d "$(ANDROID_HOME)/platforms/android-34" || (echo "Missing platforms;android-34 — run make setup-mac" && exit 1)
@test -x "$(ADB)" || (echo "Missing adb — run make setup-mac" && exit 1)
@$(ADB) version | head -1
@echo "== local.properties =="
@test -f local.properties && cat local.properties || (echo "missing local.properties — run make setup-mac" && exit 1)
@echo "== AVD =="
@$(AVDMANAGER) list avd 2>/dev/null | sed -n '1,20p' || true
@echo "✓ doctor OK"
check-device: ## Check if device is connected
@echo "Checking for connected devices..."
@export ANDROID_HOME=$(ANDROID_HOME) && \
export PATH=$(ANDROID_HOME)/platform-tools:$$PATH && \
adb devices
setup-mac: ## Install toolchain on macOS (Homebrew)
@test "$(UNAME_S)" = "Darwin" || (echo "setup-mac is for macOS only; use setup-sdk on Linux" && exit 1)
@echo "Installing OpenJDK 17 + Android command-line tools via Homebrew..."
@brew list openjdk@17 >/dev/null 2>&1 || brew install openjdk@17
@brew list --cask android-commandlinetools >/dev/null 2>&1 || brew install --cask android-commandlinetools
@$(MAKE) setup-sdk
@$(MAKE) setup-emulator
@$(MAKE) doctor
devices: check-device ## List connected devices (alias for check-device)
setup-sdk: ## Install Android SDK packages + write local.properties
@echo "Accepting SDK licenses + installing build packages..."
@yes | $(SDKMANAGER) --licenses >/dev/null 2>&1 || true
@$(SDKMANAGER) \
"platform-tools" \
"platforms;android-34" \
"build-tools;34.0.0" \
"emulator" \
"system-images;android-34;$(AVD_ABI)"
@printf 'sdk.dir=%s\n' "$(ANDROID_HOME)" > local.properties
@echo "✓ wrote local.properties"
@echo "✓ Android SDK ready at $(ANDROID_HOME)"
setup-emulator: ## Create and start Android emulator
@echo "Setting up Android emulator..."
@export ANDROID_HOME=$(ANDROID_HOME) && \
export PATH=$(ANDROID_HOME)/cmdline-tools/latest/bin:$(ANDROID_HOME)/platform-tools:$$PATH && \
sdkmanager "system-images;android-34;google_apis;x86_64" > /dev/null 2>&1
@export ANDROID_HOME=$(ANDROID_HOME) && \
export PATH=$(ANDROID_HOME)/cmdline-tools/latest/bin:$(ANDROID_HOME)/platform-tools:$$PATH && \
avdmanager create avd -n "CrklEmulator" -k "system-images;android-34;google_apis;x86_64" -d "pixel_7" --force
@echo "✓ Emulator created. Run 'make emulator' to start it."
setup-emulator: ## Create CrklEmulator AVD
@echo "Creating AVD $(AVD_NAME) ($(AVD_ABI))..."
@echo no | $(AVDMANAGER) create avd -n "$(AVD_NAME)" \
-k "system-images;android-34;$(AVD_ABI)" \
-d "pixel_7" --force || true
@echo "✓ AVD ready. Start with: make emulator"
emulator: ## Start Android emulator
@echo "Starting Android emulator with 4GB RAM..."
@export ANDROID_HOME=$(ANDROID_HOME) && \
export PATH=$(ANDROID_HOME)/emulator:$(ANDROID_HOME)/platform-tools:$$PATH && \
emulator -avd CrklEmulator -memory 4096 -cores 4 -no-audio -gpu swiftshader_indirect -no-metrics &
check-device: ## List connected devices/emulators
@$(ADB) devices -l
devices: check-device
emulator: ## Start CrklEmulator (open gRPC so rim ◀○□ chrome works)
@if $(ADB) devices 2>/dev/null | awk '/emulator-/{exit 0} END{exit 1}'; then \
echo "Emulator already connected:"; $(ADB) devices -l; \
echo "Tip: if rim buttons dead, run: make test-env (restarts with -grpc)"; \
else \
echo "Starting $(AVD_NAME) (gpu=$(EMULATOR_GPU), grpc open)..."; \
if [ "$(UNAME_S)" = "Darwin" ]; then \
chmod +x scripts/start-emulator-launchd.sh; \
bash scripts/start-emulator-launchd.sh; \
else \
nohup $(EMULATOR) -avd "$(AVD_NAME)" -memory 3072 -cores 4 -no-audio \
-gpu $(EMULATOR_GPU) -accel on -no-snapshot-load -no-metrics \
-grpc 8554 \
>/tmp/crkl-emulator.log 2>&1 & echo $$! > /tmp/crkl-emulator.pid; \
echo "Emulator pid $$(cat /tmp/crkl-emulator.pid) (log: /tmp/crkl-emulator.log)"; \
fi; \
fi
@echo "Next: make wait-emulator && make install run OR make test-env"
wait-emulator: ## Block until emulator is fully booted
@bash scripts/wait-emulator.sh "$(ADB)"
emulator-stop: ## Stop emulator
@echo "Stopping emulator..."
@export ANDROID_HOME=$(ANDROID_HOME) && \
export PATH=$(ANDROID_HOME)/platform-tools:$$PATH && \
adb emu kill
@$(ADB) emu kill 2>/dev/null || true
@pkill -f 'qemu-system' 2>/dev/null || true
@rm -f /tmp/crkl-emulator.pid
@echo "✓ emulator stop requested"
emulator-list: ## List available emulators
@echo "Available emulators:"
@export ANDROID_HOME=$(ANDROID_HOME) && \
export PATH=$(ANDROID_HOME)/cmdline-tools/latest/bin:$(ANDROID_HOME)/platform-tools:$$PATH && \
avdmanager list avd
emulator-list: ## List AVDs
@$(AVDMANAGER) list avd
build: ## Build APK
test: ## Run JVM unit tests
@echo "Running unit tests..."
@$(GRADLEW) test --console=plain
smoke: ## On-device smoke (FAB → circle → extract → Copy). Needs adb device.
@chmod +x scripts/smoke-circle.sh
@ADB="$(ADB)" bash scripts/smoke-circle.sh
build: ## Build debug APK
@echo "Building Crkl..."
@$(GRADLEW) assembleDebug
@$(GRADLEW) assembleDebug --console=plain
install: build ## Install to phone
@echo "Installing to phone..."
@export ANDROID_HOME=$(ANDROID_HOME) && \
export PATH=$(ANDROID_HOME)/platform-tools:$$PATH && \
$(ADB) install -r $(APK)
install: build ## Install APK to connected device/emulator
@$(ADB) wait-for-device
@$(ADB) install -r $(APK)
@echo "✓ Installed"
run: ## Launch app
@echo "Launching Crkl..."
@export ANDROID_HOME=$(ANDROID_HOME) && \
export PATH=$(ANDROID_HOME)/platform-tools:$$PATH && \
$(ADB) shell am start -n com.example.crkl/.MainActivity
run: ## Launch MainActivity
@$(ADB) shell am start -n com.example.crkl/.MainActivity
logs: ## Watch app logs
@echo "Watching logs (Ctrl+C to stop)..."
@export ANDROID_HOME=$(ANDROID_HOME) && \
export PATH=$(ANDROID_HOME)/platform-tools:$$PATH && \
$(ADB) logcat -s CrklAccessibilityService:D OverlayView:D AndroidRuntime:E
fixtures: ## Open test fixtures (email/image/video/audio) on device/emulator
@$(ADB) wait-for-device
@$(ADB) shell am start -n com.example.crkl/.fixtures.TestFixturesActivity
@echo "✓ Fixtures open. Tap blue C on the phone (not rim ○)."
stop: ## Stop app
@export ANDROID_HOME=$(ANDROID_HOME) && \
export PATH=$(ANDROID_HOME)/platform-tools:$$PATH && \
$(ADB) shell am force-stop com.example.crkl
test-env: ## ONE SHOT testing playground (chrome fix + install + fixtures)
@chmod +x scripts/test-env.sh
@bash scripts/test-env.sh
clean: ## Clean build
integrations: ## Open Integrations settings on device/emulator
@$(ADB) wait-for-device
@$(ADB) shell am start -n com.example.crkl/.IntegrationsActivity
gog-bridge: ## Optional lab: host gog → Gmail (emu uses http://10.0.2.2:8765)
@chmod +x scripts/gog-bridge.py
@python3 scripts/gog-bridge.py
home: ## Emulator/device Home (adb) — fallback if rim buttons still fail
@$(ADB) shell input keyevent KEYCODE_HOME
back: ## Emulator/device Back (adb)
@$(ADB) shell input keyevent KEYCODE_BACK
recents: ## Emulator/device Recents (adb)
@$(ADB) shell input keyevent KEYCODE_APP_SWITCH
demo: build emulator wait-emulator install run ## Full local demo on emulator
@echo "✓ demo launched. Enable Accessibility, then: make fixtures"
@echo " Nav if side buttons fail: make home | make back | make recents"
@echo " make logs"
logs: ## Watch Crkl logcat
@$(ADB) logcat -s CrklAccessibilityService:D OverlayView:D MediaPipeLocalLlm:D AssistEngine:D RegionContentExtractor:D AndroidRuntime:E
stop: ## Force-stop app
@$(ADB) shell am force-stop com.example.crkl
clean: ## Clean Gradle build
@$(GRADLEW) clean
uninstall: ## Remove app
@export ANDROID_HOME=$(ANDROID_HOME) && \
export PATH=$(ANDROID_HOME)/platform-tools:$$PATH && \
$(ADB) uninstall com.example.crkl
uninstall: ## Uninstall app
@$(ADB) uninstall com.example.crkl
push-model: check-device ## Push MediaPipe .task model
@if [ -z "$(MODEL)" ]; then \
echo "Usage: make push-model MODEL=/path/to/gemma-3-1b-it-int4.task"; \
exit 1; \
fi
@test -f "$(MODEL)" || (echo "Model file not found: $(MODEL)" && exit 1)
@echo "Pushing $(MODEL) → $(MODEL_DEVICE_PATH)"
@$(ADB) shell mkdir -p /data/local/tmp/llm
@$(ADB) push "$(MODEL)" $(MODEL_DEVICE_PATH)
@echo "✓ Model on device. Toggle Crkl accessibility service to reload."
model-status: check-device ## Show on-device model file
@$(ADB) shell "ls -lh $(MODEL_DEVICE_PATH) /data/local/tmp/llm/ 2>/dev/null || echo 'No model at $(MODEL_DEVICE_PATH)'"
PHONE_SETUP.md
+51 -72
View File
@@ -1,143 +1,122 @@
# Physical Phone Setup Guide
# Physical Phone Setup Guide — Circle
## Prerequisites
1. **Android phone** (Android 8.1 or higher)
1. **Android phone** (API 27+ / Android 8.1+)
2. **USB cable**
3. **Android SDK installed** on your Linux machine
3. **Android SDK** on your Mac/Linux machine
---
## Step 1: Enable Developer Options on Phone
## Step 1: Enable Developer Options
1. Open **Settings** on your phone
2. Go to **About Phone**
3. Tap **Build Number** 7 times
4. You'll see "You are now a developer!"
1. Settings → **About Phone**
2. Tap **Build Number** 7 times
---
## Step 2: Enable USB Debugging
1. Go back to **Settings**
2. Open **Developer Options** (or **System → Developer Options**)
3. Enable **USB Debugging**
4. (Optional) Enable **Stay Awake** - keeps screen on while charging
1. Settings → **Developer Options**
2. Enable **USB Debugging**
3. (Optional) **Stay Awake** while charging
---
## Step 3: Connect Phone to Computer
## Step 3: Connect Phone
1. Plug phone into computer via USB
2. On your phone, a popup will appear: **"Allow USB debugging?"**
3. Tap **Allow** (check "Always allow from this computer")
1. Plug in USB
2. Allow USB debugging on the phone
---
## Step 4: Verify Connection
```bash
cd /home/user/Documents/code/crkl
export ANDROID_HOME=~/android-sdk
export PATH=$ANDROID_HOME/platform-tools:$PATH
# Check if phone is detected
adb devices
```
You should see:
```
List of devices attached
ABC123XYZ device
```
You should see `device` (not `unauthorized`).
---
## Step 5: Build and Install Crkl
## Step 5: Build and Install Circle
```bash
# Build the app
make build
# Install to your phone
make install
cd ~/Documents/code/crkl
./gradlew assembleDebug
adb install -r app/build/outputs/apk/debug/app-debug.apk
adb shell am start -n com.example.crkl/.MainActivity
# or: make build && make install && make run
```
---
## Step 6: Enable Accessibility Service
## Step 6: Enable Circle Overlay
### On Your Phone:
1. Open the **Circle** app
2. Tap **Enable Accessibility**
3. Turn on **Circle Overlay**
4. Accept the permission dialog
1. **Open the Crkl app** (will show welcome screen)
2. **Tap "Open Accessibility Settings"** button
3. In Settings, find **"Crkl Overlay Service"**
4. **Toggle it ON**
5. Accept the permission dialog
6. **Press back button** to return to home screen
Re-enable after every reinstall.
---
## Step 7: Test It
The overlay is now active system-wide!
Tap the floating **C** (or QS **Circle**), draw a closed loop around text, use the chips.
**Watch logs:**
```bash
make logs
# or: adb logcat -s CrklAccessibilityService:D OverlayView:D AssistEngine:D
```
**Touch your phone screen** anywhere - you should see:
```
OverlayView: Touch down at (X, Y)
OverlayView: Touch up at (X, Y)
```
**Visual feedback:**
- Cyan circles appear where you touch
- Crosshair shows exact touch point
VIP path: **Test fixtures** → [docs/DEMO.md](docs/DEMO.md).
---
## Quick Commands
```bash
make build # Build APK
make install # Install to phone
make run # Launch app
make logs # Watch logs
make stop # Stop app
make clean # Clean build
make uninstall # Remove app
make build / install / run / logs / stop / clean / uninstall
```
---
## Troubleshooting
### "adb: command not found"
### adb not found
```bash
export ANDROID_HOME=~/android-sdk
export ANDROID_HOME=/opt/homebrew/share/android-commandlinetools # example
export PATH=$ANDROID_HOME/platform-tools:$PATH
```
### "no devices/emulators found"
- Check USB cable is connected
- Check "USB Debugging" is enabled
- Run `adb devices` and accept prompt on phone
### no devices
- USB debugging on; accept the prompt
- Emulator flaky? Headless tips in [docs/RELEASE.md](docs/RELEASE.md)
### "unauthorized"
- Revoke USB debugging authorizations in Developer Options
- Disconnect and reconnect USB
- Accept the new prompt
### unauthorized
- Revoke USB debugging authorizations; reconnect
### Can't see overlay
- Make sure you enabled "Crkl Overlay Service" in Accessibility Settings
- Restart the app: `make stop && make run`
- Enable **Circle Overlay** in Accessibility
- `make stop && make run`
---
## Done!
## Shortcuts
You're ready to develop. The app will now detect touches system-wide on your phone.
See [docs/shortcuts.md](docs/shortcuts.md).
1. Enable **Circle Overlay**
2. Circle app → **Add Quick Settings tile**
3. Swipe down → **Circle** → closed loop
Optional: Accessibility shortcut → **Circle Overlay** (hold both volume keys).
---
## Done
Enable Accessibility, then use the floating C or the Circle QS tile.
README.md
+73 -76
View File
@@ -1,97 +1,94 @@
# Crkl - On-Device AI Assistant
# Circle
Privacy-first Android AI assistant that lets users circle or touch any element on their screen, then calls a fully *on-device* AI engine to transcribe, summarize, explain, or draft responses.
Privacy-first Android assist: **circle on-screen text → Translate · Copy · Explain · Share · Vikunja**. On-device.
Repo / Gradle module: **crkl** · Application id: `com.example.crkl` · Display name: **Circle**
## Ship docs
- [docs/DEMO.md](docs/DEMO.md) — 45s recording script
- [docs/dogfood.md](docs/dogfood.md) — daily checklist
- [docs/non-goals.md](docs/non-goals.md) — freeze list
- [docs/marketing.md](docs/marketing.md) — blurb + posts
- [docs/shortcuts.md](docs/shortcuts.md) — QS tile / a11y button / volume keys
- [docs/RELEASE.md](docs/RELEASE.md) — install & tag
- [docs/brand/](docs/brand/) — logo (shipped: **lasso #2**)
- [CHANGELOG.md](CHANGELOG.md)
## Features
- System-wide accessibility overlay
- Touch/gesture detection
- Local AI inference (no cloud calls)
- Privacy: *No data leaves device*
## Quick Start
- **Circle → text → act** (a11y + on-device OCR)
- Chips: **Translate · Copy · Explain · Share · Vikunja**
- **Vikunja** HTTPS for “add to todo”
- Optional mailto / lab gog bridge (Advanced in Integrations)
- Device calendar read (Advanced)
- Circle style (color, neon, stroke)
- First-run onboarding; clean panel (debug meta optional)
- Shortcuts: Quick Settings **Circle** tile, Accessibility button, volume-key a11y shortcut
- Local LLM when a `.task` model is present; otherwise stub Explain / summary
## Quick start
### 1. Setup Development Environment
```bash
# One-time setup (installs Android SDK and tools)
./setup.sh
./setup.sh # one-time SDK (if needed)
make check-device
./gradlew assembleDebug
adb install -r app/build/outputs/apk/debug/app-debug.apk
adb shell am start -n com.example.crkl/.MainActivity
```
### 2. Connect Your Android Device
See [PHONE_SETUP.md](PHONE_SETUP.md) for detailed setup instructions.
Or: `make build` / `make install` / `make run` when Make targets are available.
See [PHONE_SETUP.md](PHONE_SETUP.md) for USB debugging.
### Enable Circle Overlay
1. Open **Circle**
2. Tap **Enable Accessibility** (or Accessibility settings)
3. Turn on **Circle Overlay**
4. Accept the permission dialog
Re-enable after every reinstall.
### Try it
Tap the floating **C**, or Quick Settings → **Circle**, then draw a closed loop around text.
VIP path is on **Test fixtures** (see [docs/DEMO.md](docs/DEMO.md)).
### Shortcuts
[docs/shortcuts.md](docs/shortcuts.md) — QS tile, Accessibility button, hold both volume keys.
Double-press power is **not** available to third-party apps.
## Emulator note
If the GUI emulator never appears in `adb devices`, use headless (see [docs/RELEASE.md](docs/RELEASE.md)):
### 3. Build and Install
```bash
make check-device # Check if device is connected
make build # Build APK
make install # Install to device
make run # Launch app
$ANDROID_HOME/emulator/emulator -avd CrklEmulator -no-window \
-gpu swiftshader_indirect -no-snapshot-load -grpc 8554 &
adb wait-for-device
```
### 3. Enable Accessibility Service
1. Open Crkl app (shows welcome screen)
2. Tap "Open Accessibility Settings"
3. Find "Crkl Overlay Service" and toggle ON
4. Accept permission dialog
### 4. Test It
```bash
make logs # Watch logs
```
Touch your phone screen anywhere - you'll see touch detection logs and visual feedback (cyan circles).
## Commands
### Setup Commands
```bash
./setup.sh # One-time setup (installs Android SDK)
make setup-sdk # Install Android SDK and tools
make check-device # Check if device is connected
```
### Development Commands
```bash
make build # Build APK
make install # Install to device
make run # Launch app
make logs # Watch app logs
make stop # Stop app
make clean # Clean build
make uninstall # Remove app
```
### Help
```bash
make help # Show all available commands
```
## Project Structure
## Project structure
```
crkl/
├── app/src/main/kotlin/com/example/crkl/
│ ├── MainActivity.kt # Main app activity
│ ├── accessibility/
│ │ ├── CrklAccessibilityService.kt # System overlay service
│ │ └── OverlayView.kt # Touch detection view
│ └── ui/theme/ # App theming
├── app/src/main/res/ # Resources
├── Makefile # Build commands
├── PHONE_SETUP.md # Phone setup guide
└── README.md # This file
│ ├── MainActivity.kt
│ ├── accessibility/ # Overlay, QS tile, bridge
│ ├── agent/ # VoiceIntent, ActionExecutor, AssistEngine
│ ├── vision/ # OCR + content capture
│ ├── integrations/ # Vikunja, translate, calendar
│ ├── fixtures/ # Test fixtures + demoScript()
│ └── ui/ # Result panel, theme, CrklUi tokens
├── docs/ # Ship + brand docs
└── Makefile
```
## Development
## Model (optional)
The POC is complete with:
- ✅ System-wide overlay
- ✅ Touch detection
- ✅ Visual feedback
- ✅ Logging
Inference is MediaPipe on-device (`make push-model`). Without a model, Circle uses a local stub — still no cloud for summaries. OCR is ML Kit on-device.
Next: Add gesture recognition, content detection, and AI integration.
## License
## Requirements
- Android phone (Android 8.1+)
- Android SDK on Linux
- USB debugging enabled
See repository license file.
app/build.gradle.kts
+23 -6
View File
@@ -11,13 +11,16 @@ android {
applicationId = "com.example.crkl"
minSdk = 27
targetSdk = 34
versionCode = 1
versionName = "1.0.0-poc"
versionCode = 19
versionName = "1.16.2-circle-copy"
testInstrumentationRunner = "androidx.test.runner.AndroidJUnitRunner"
vectorDrawables {
useSupportLibrary = true
}
ndk {
abiFilters += listOf("arm64-v8a", "armeabi-v7a")
}
}
buildTypes {
@@ -34,12 +37,12 @@ android {
}
compileOptions {
sourceCompatibility = JavaVersion.VERSION_1_8
targetCompatibility = JavaVersion.VERSION_1_8
sourceCompatibility = JavaVersion.VERSION_17
targetCompatibility = JavaVersion.VERSION_17
}
kotlinOptions {
jvmTarget = "1.8"
jvmTarget = "17"
}
buildFeatures {
@@ -56,6 +59,10 @@ android {
excludes += "/META-INF/{AL2.0,LGPL2.1}"
}
}
testOptions {
unitTests.isIncludeAndroidResources = true
}
}
dependencies {
@@ -73,7 +80,15 @@ dependencies {
implementation("androidx.compose.material3:material3")
implementation("androidx.compose.material:material-icons-extended")
// Accessibility (built into core-ktx)
// On-device LLM (Gemma via MediaPipe LLM Inference)
implementation("com.google.mediapipe:tasks-genai:0.10.27")
// On-device OCR (ML Kit)
implementation("com.google.mlkit:text-recognition:16.0.1")
// On-device translation + language ID (ML Kit)
implementation("com.google.mlkit:translate:17.0.3")
implementation("com.google.mlkit:language-id:17.0.6")
// Coroutines for async operations
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-android:1.7.3")
@@ -86,6 +101,8 @@ dependencies {
// Testing
testImplementation("junit:junit:4.13.2")
testImplementation("org.jetbrains.kotlinx:kotlinx-coroutines-test:1.7.3")
testImplementation("org.robolectric:robolectric:4.12.2")
testImplementation("androidx.test:core:1.5.0")
androidTestImplementation("androidx.test.ext:junit:1.1.5")
androidTestImplementation("androidx.test.espresso:espresso-core:3.5.1")
androidTestImplementation(platform("androidx.compose:compose-bom:2023.10.01"))
app/src/main/AndroidManifest.xml
+41 -9
View File
@@ -2,15 +2,15 @@
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:tools="http://schemas.android.com/tools">
<!-- Accessibility service permission -->
<uses-permission android:name="android.permission.BIND_ACCESSIBILITY_SERVICE"
tools:ignore="ProtectedPermissions" />
<!-- System alert window for overlay (fallback) -->
<uses-permission android:name="android.permission.SYSTEM_ALERT_WINDOW" />
<!-- For capturing screen content if needed -->
<uses-permission android:name="android.permission.FOREGROUND_SERVICE" />
<uses-permission android:name="android.permission.FOREGROUND_SERVICE_MEDIA_PROJECTION" />
<uses-permission android:name="android.permission.POST_NOTIFICATIONS" />
<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.READ_CALENDAR" />
<application
android:allowBackup="true"
@@ -19,9 +19,9 @@
android:roundIcon="@mipmap/ic_launcher_round"
android:supportsRtl="true"
android:theme="@style/Theme.Crkl"
android:usesCleartextTraffic="true"
tools:targetApi="31">
<!-- Main Activity (for settings/permissions) -->
<activity
android:name=".MainActivity"
android:exported="true"
@@ -33,7 +33,24 @@
</intent-filter>
</activity>
<!-- Accessibility Service -->
<activity
android:name=".IntegrationsActivity"
android:exported="true"
android:label="Circle Integrations"
android:theme="@style/Theme.Crkl" />
<activity
android:name=".fixtures.TestFixturesActivity"
android:exported="true"
android:label="@string/test_fixtures_label"
android:theme="@style/Theme.Crkl" />
<activity
android:name=".capture.ProjectionPermissionActivity"
android:exported="false"
android:excludeFromRecents="true"
android:theme="@android:style/Theme.Translucent.NoTitleBar" />
<service
android:name=".accessibility.CrklAccessibilityService"
android:exported="false"
@@ -46,7 +63,22 @@
android:resource="@xml/accessibility_service_config" />
</service>
<service
android:name=".accessibility.CrklCircleTileService"
android:exported="true"
android:icon="@drawable/ic_qs_circle"
android:label="@string/qs_tile_label"
android:permission="android.permission.BIND_QUICK_SETTINGS_TILE">
<intent-filter>
<action android:name="android.service.quicksettings.action.QS_TILE" />
</intent-filter>
</service>
<service
android:name=".capture.CaptureForegroundService"
android:exported="false"
android:foregroundServiceType="mediaProjection" />
</application>
</manifest>
app/src/main/assets/media/grocery_memo.transcript.txt
+1
View File
@@ -0,0 +1 @@
Hi. Voice memo grocery list. We need milk, eggs, sourdough bread, olive oil, and two avocados. Also pick up oat milk and coffee beans. That is everything.
app/src/main/assets/media/grocery_memo.wav
BIN
View File
Binary file not shown.
app/src/main/assets/media/product_walkthrough.mp4
BIN
View File
Binary file not shown.
app/src/main/assets/media/product_walkthrough.transcript.txt
+1
View File
@@ -0,0 +1 @@
Welcome to the product walkthrough. First, open Settings. Next, enable Privacy Mode. Then tap Save. You are done. The whole flow takes under a minute.
app/src/main/kotlin/com/example/crkl/IntegrationsActivity.kt
+294
View File
@@ -0,0 +1,294 @@
package com.example.crkl
import android.Manifest
import android.os.Bundle
import android.widget.Toast
import androidx.activity.ComponentActivity
import androidx.activity.compose.setContent
import androidx.activity.result.contract.ActivityResultContracts
import androidx.compose.animation.AnimatedVisibility
import androidx.compose.foundation.layout.Arrangement
import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.Row
import androidx.compose.foundation.layout.Spacer
import androidx.compose.foundation.layout.fillMaxSize
import androidx.compose.foundation.layout.fillMaxWidth
import androidx.compose.foundation.layout.height
import androidx.compose.foundation.layout.padding
import androidx.compose.foundation.rememberScrollState
import androidx.compose.foundation.verticalScroll
import androidx.compose.material3.Button
import androidx.compose.material3.ExperimentalMaterial3Api
import androidx.compose.material3.FilterChip
import androidx.compose.material3.MaterialTheme
import androidx.compose.material3.OutlinedButton
import androidx.compose.material3.OutlinedTextField
import androidx.compose.material3.Surface
import androidx.compose.material3.Switch
import androidx.compose.material3.Text
import androidx.compose.material3.TextButton
import androidx.compose.runtime.getValue
import androidx.compose.runtime.mutableStateOf
import androidx.compose.runtime.remember
import androidx.compose.runtime.rememberCoroutineScope
import androidx.compose.runtime.setValue
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.text.font.FontWeight
import androidx.compose.ui.unit.dp
import com.example.crkl.integrations.DeviceCalendar
import com.example.crkl.integrations.GogBridgeClient
import com.example.crkl.integrations.IntegrationSettings
import com.example.crkl.integrations.OnDeviceTranslator
import com.example.crkl.ui.theme.CrklTheme
import kotlinx.coroutines.launch
/**
* Integrations: Vikunja + translate first; appearance; lab tools collapsed.
*/
@OptIn(ExperimentalMaterial3Api::class)
class IntegrationsActivity : ComponentActivity() {
private lateinit var settings: IntegrationSettings
private var statusState: ((String) -> Unit)? = null
private val calendarPermission = registerForActivityResult(
ActivityResultContracts.RequestPermission()
) { granted ->
val msg = if (granted) DeviceCalendar.today(this).message else "Calendar permission denied."
statusState?.invoke(settings.statusSummary() + "\n\n" + msg)
Toast.makeText(
this,
if (granted) "Calendar access granted" else "Permission denied",
Toast.LENGTH_SHORT
).show()
}
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
settings = IntegrationSettings(this)
setContent {
CrklTheme {
Surface(modifier = Modifier.fillMaxSize()) {
val scope = rememberCoroutineScope()
var vikunjaUrl by remember { mutableStateOf(settings.vikunjaUrl) }
var vikunjaToken by remember { mutableStateOf(settings.vikunjaToken) }
var projectId by remember { mutableStateOf(settings.vikunjaProjectId.toString()) }
var emailTo by remember { mutableStateOf(settings.emailTo) }
var gogUrl by remember {
mutableStateOf(
settings.gogBridgeUrl.ifBlank { IntegrationSettings.DEFAULT_GOG_BRIDGE_EMU }
)
}
var preferGog by remember { mutableStateOf(settings.preferGogEmail) }
var circleColor by remember { mutableStateOf(settings.circleColorKey) }
var circleNeon by remember { mutableStateOf(settings.circleNeon) }
var circleStroke by remember { mutableStateOf(settings.circleStrokeKey) }
var translateTarget by remember { mutableStateOf(settings.translateTargetLang) }
var showDebugMeta by remember { mutableStateOf(settings.showDebugMeta) }
var showAdvanced by remember { mutableStateOf(false) }
var showAppearance by remember { mutableStateOf(false) }
var status by remember { mutableStateOf(settings.statusSummary()) }
statusState = { status = it }
fun saveAll() {
settings.vikunjaUrl = vikunjaUrl
settings.vikunjaToken = vikunjaToken
settings.vikunjaProjectId = projectId.toIntOrNull()
?: IntegrationSettings.DEFAULT_VIKUNJA_PROJECT
settings.emailTo = emailTo
settings.gogBridgeUrl = gogUrl
settings.preferGogEmail = preferGog
settings.circleColorKey = circleColor
settings.circleNeon = circleNeon
settings.circleStrokeKey = circleStroke
settings.translateTargetLang = translateTarget
settings.showDebugMeta = showDebugMeta
status = settings.statusSummary()
Toast.makeText(this@IntegrationsActivity, "Saved", Toast.LENGTH_SHORT).show()
}
Column(
modifier = Modifier
.fillMaxSize()
.verticalScroll(rememberScrollState())
.padding(16.dp),
verticalArrangement = Arrangement.spacedBy(10.dp)
) {
Text(
"Integrations",
style = MaterialTheme.typography.headlineMedium,
fontWeight = FontWeight.Bold
)
Text(
"Connect the apps you already use. Only Vikunja is required for Add to todo.",
style = MaterialTheme.typography.bodyMedium,
color = MaterialTheme.colorScheme.onSurfaceVariant
)
Text("Vikunja (todos)", fontWeight = FontWeight.SemiBold)
Text(
"Paste an API token from todo.levkin.ca → Settings → API Tokens.",
style = MaterialTheme.typography.bodySmall,
color = MaterialTheme.colorScheme.onSurfaceVariant
)
OutlinedTextField(
vikunjaUrl,
{ vikunjaUrl = it },
label = { Text("URL") },
modifier = Modifier.fillMaxWidth(),
singleLine = true
)
OutlinedTextField(
vikunjaToken,
{ vikunjaToken = it },
label = { Text("API token") },
modifier = Modifier.fillMaxWidth(),
singleLine = true
)
OutlinedTextField(
projectId,
{ projectId = it },
label = { Text("Project id") },
modifier = Modifier.fillMaxWidth(),
singleLine = true
)
Text("Translate", fontWeight = FontWeight.SemiBold)
Text(
"Default language for the Translate chip.",
style = MaterialTheme.typography.bodySmall,
color = MaterialTheme.colorScheme.onSurfaceVariant
)
Row(
horizontalArrangement = Arrangement.spacedBy(6.dp),
modifier = Modifier.fillMaxWidth()
) {
OnDeviceTranslator.TARGET_OPTIONS.forEach { (tag, label) ->
FilterChip(
selected = translateTarget == tag,
onClick = { translateTarget = tag },
label = { Text(label) }
)
}
}
Button(onClick = { saveAll() }, modifier = Modifier.fillMaxWidth()) {
Text("Save")
}
TextButton(onClick = { showAppearance = !showAppearance }) {
Text(if (showAppearance) "Hide circle style" else "Circle style")
}
AnimatedVisibility(visible = showAppearance) {
Column(verticalArrangement = Arrangement.spacedBy(10.dp)) {
Row(
horizontalArrangement = Arrangement.spacedBy(6.dp),
modifier = Modifier.fillMaxWidth()
) {
IntegrationSettings.CIRCLE_COLORS.forEach { key ->
FilterChip(
selected = circleColor == key,
onClick = { circleColor = key },
label = { Text(key) }
)
}
}
Row(
verticalAlignment = Alignment.CenterVertically,
modifier = Modifier.fillMaxWidth()
) {
Text("Neon glow", modifier = Modifier.weight(1f))
Switch(checked = circleNeon, onCheckedChange = { circleNeon = it })
}
Row(horizontalArrangement = Arrangement.spacedBy(6.dp)) {
IntegrationSettings.CIRCLE_STROKES.forEach { key ->
FilterChip(
selected = circleStroke == key,
onClick = { circleStroke = key },
label = { Text(key) }
)
}
}
}
}
TextButton(onClick = { showAdvanced = !showAdvanced }) {
Text(if (showAdvanced) "Hide advanced" else "Advanced (email lab, calendar, debug)")
}
AnimatedVisibility(visible = showAdvanced) {
Column(verticalArrangement = Arrangement.spacedBy(10.dp)) {
OutlinedTextField(
emailTo,
{ emailTo = it },
label = { Text("Email To (mailto)") },
modifier = Modifier.fillMaxWidth(),
singleLine = true
)
Text("gog bridge (emulator lab only)", fontWeight = FontWeight.SemiBold)
Row(
verticalAlignment = Alignment.CenterVertically,
modifier = Modifier.fillMaxWidth()
) {
Text("Prefer gog", modifier = Modifier.weight(1f))
Switch(checked = preferGog, onCheckedChange = { preferGog = it })
}
OutlinedTextField(
gogUrl,
{ gogUrl = it },
label = { Text("gog bridge URL") },
modifier = Modifier.fillMaxWidth(),
singleLine = true,
enabled = preferGog
)
Row(
verticalAlignment = Alignment.CenterVertically,
modifier = Modifier.fillMaxWidth()
) {
Text("Show debug meta on panel", modifier = Modifier.weight(1f))
Switch(
checked = showDebugMeta,
onCheckedChange = { showDebugMeta = it }
)
}
OutlinedButton(
onClick = {
if (DeviceCalendar.hasPermission(this@IntegrationsActivity)) {
status = settings.statusSummary() + "\n\n" +
DeviceCalendar.today(this@IntegrationsActivity).message
} else {
calendarPermission.launch(Manifest.permission.READ_CALENDAR)
}
},
modifier = Modifier.fillMaxWidth()
) {
Text(
if (DeviceCalendar.hasPermission(this@IntegrationsActivity)) {
"Test calendar"
} else {
"Grant calendar access"
}
)
}
OutlinedButton(
onClick = {
saveAll()
scope.launch {
val r = GogBridgeClient(settings).health()
status = settings.statusSummary() + "\n\ngog: " + r.message
}
},
modifier = Modifier.fillMaxWidth(),
enabled = preferGog
) { Text("Test gog bridge") }
}
}
Spacer(modifier = Modifier.height(4.dp))
Text(status, style = MaterialTheme.typography.bodySmall)
}
}
}
}
}
}
app/src/main/kotlin/com/example/crkl/MainActivity.kt
+364 -56
View File
@@ -1,128 +1,436 @@
package com.example.crkl
import android.Manifest
import android.accessibilityservice.AccessibilityServiceInfo
import android.content.Context
import android.content.Intent
import android.content.pm.PackageManager
import android.os.Build
import android.os.Bundle
import android.provider.Settings
import android.view.accessibility.AccessibilityManager
import androidx.activity.ComponentActivity
import androidx.activity.compose.setContent
import androidx.compose.foundation.layout.*
import androidx.compose.material3.*
import androidx.compose.runtime.*
import androidx.activity.result.contract.ActivityResultContracts
import androidx.compose.foundation.layout.Arrangement
import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.Row
import androidx.compose.foundation.layout.Spacer
import androidx.compose.foundation.layout.fillMaxSize
import androidx.compose.foundation.layout.fillMaxWidth
import androidx.compose.foundation.layout.height
import androidx.compose.foundation.layout.padding
import androidx.compose.foundation.rememberScrollState
import androidx.compose.foundation.verticalScroll
import androidx.compose.material3.Button
import androidx.compose.material3.Card
import androidx.compose.material3.CardDefaults
import androidx.compose.material3.MaterialTheme
import androidx.compose.material3.OutlinedButton
import androidx.compose.material3.Surface
import androidx.compose.material3.Text
import androidx.compose.material3.TextButton
import androidx.compose.runtime.Composable
import androidx.compose.runtime.DisposableEffect
import androidx.compose.runtime.getValue
import androidx.compose.runtime.mutableStateOf
import androidx.compose.runtime.remember
import androidx.compose.runtime.setValue
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.platform.LocalLifecycleOwner
import androidx.compose.ui.res.stringResource
import androidx.compose.ui.text.font.FontWeight
import androidx.compose.ui.text.style.TextAlign
import androidx.compose.ui.unit.dp
import androidx.core.content.ContextCompat
import androidx.lifecycle.Lifecycle
import androidx.lifecycle.LifecycleEventObserver
import android.app.StatusBarManager
import android.content.ComponentName
import android.graphics.drawable.Icon
import android.widget.Toast
import com.example.crkl.accessibility.CrklCircleTileService
import com.example.crkl.accessibility.CrklOverlayBridge
import com.example.crkl.capture.MediaProjectionHolder
import com.example.crkl.capture.ProjectionPermissionActivity
import com.example.crkl.fixtures.TestFixturesActivity
import com.example.crkl.integrations.IntegrationSettings
import com.example.crkl.ui.theme.CrklTheme
import java.util.concurrent.Executor
/**
* Main Activity for Crkl
*
* This activity serves as the entry point and settings page.
* It guides users to enable the accessibility service.
*/
class MainActivity : ComponentActivity() {
private val micPermission = registerForActivityResult(
ActivityResultContracts.RequestPermission()
) { }
private val notifPermission = registerForActivityResult(
ActivityResultContracts.RequestPermission()
) { }
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
maybeRequestMic()
maybeRequestNotifications()
setContent {
CrklTheme {
Surface(
modifier = Modifier.fillMaxSize(),
color = MaterialTheme.colorScheme.background
) {
MainScreen(
onOpenAccessibilitySettings = {
openAccessibilitySettings()
var captureReady by remember {
mutableStateOf(MediaProjectionHolder.isReady())
}
val settings = remember { IntegrationSettings(this) }
var a11yOn by remember { mutableStateOf(isCrklAccessibilityEnabled()) }
var onboardingDone by remember { mutableStateOf(settings.onboardingComplete) }
var overlayReady by remember { mutableStateOf(CrklOverlayBridge.isReady()) }
val lifecycleOwner = LocalLifecycleOwner.current
DisposableEffect(lifecycleOwner) {
val observer = LifecycleEventObserver { _, event ->
if (event == Lifecycle.Event.ON_RESUME) {
a11yOn = isCrklAccessibilityEnabled()
onboardingDone = settings.onboardingComplete
captureReady = MediaProjectionHolder.isReady()
overlayReady = CrklOverlayBridge.isReady()
}
}
lifecycleOwner.lifecycle.addObserver(observer)
onDispose { lifecycleOwner.lifecycle.removeObserver(observer) }
}
MainScreen(
captureReady = captureReady,
integrationHint = settings.statusSummary(),
accessibilityEnabled = a11yOn,
showOnboarding = !onboardingDone,
onRefreshA11y = {
a11yOn = isCrklAccessibilityEnabled()
overlayReady = CrklOverlayBridge.isReady()
},
onOpenAccessibilitySettings = {
startActivity(
Intent(Settings.ACTION_ACCESSIBILITY_SETTINGS).apply {
flags = Intent.FLAG_ACTIVITY_NEW_TASK
}
)
},
onOpenIntegrations = {
startActivity(Intent(this, IntegrationsActivity::class.java))
},
onOpenTestFixtures = {
settings.onboardingComplete = true
onboardingDone = true
startActivity(Intent(this, TestFixturesActivity::class.java))
},
onEnableCapture = {
startActivity(ProjectionPermissionActivity.intent(this))
window.decorView.postDelayed({
captureReady = MediaProjectionHolder.isReady()
}, 1500)
},
onSkipOnboarding = {
settings.onboardingComplete = true
onboardingDone = true
},
onAddQsTile = { requestQsTile() },
canAddQsTile = Build.VERSION.SDK_INT >= 33,
overlayReady = overlayReady
)
}
}
}
}
private fun openAccessibilitySettings() {
val intent = Intent(Settings.ACTION_ACCESSIBILITY_SETTINGS)
intent.flags = Intent.FLAG_ACTIVITY_NEW_TASK
startActivity(intent)
private fun requestQsTile() {
if (Build.VERSION.SDK_INT < 33) {
Toast.makeText(
this,
"Pull down Quick Settings → edit → add Circle",
Toast.LENGTH_LONG
).show()
return
}
val statusBar = getSystemService(StatusBarManager::class.java)
val component = ComponentName(this, CrklCircleTileService::class.java)
val icon = Icon.createWithResource(this, R.drawable.ic_qs_circle)
val executor = Executor { it.run() }
statusBar.requestAddTileService(
component,
getString(R.string.qs_tile_label),
icon,
executor
) { result ->
runOnUiThread {
val msg = when (result) {
StatusBarManager.TILE_ADD_REQUEST_RESULT_TILE_ADDED -> "Circle tile added"
StatusBarManager.TILE_ADD_REQUEST_RESULT_TILE_ALREADY_ADDED -> "Circle tile already added"
StatusBarManager.TILE_ADD_REQUEST_RESULT_TILE_NOT_ADDED -> "Tile not added"
else -> "Pull down Quick Settings → edit → add Circle"
}
Toast.makeText(this, msg, Toast.LENGTH_SHORT).show()
}
}
}
private fun isCrklAccessibilityEnabled(): Boolean {
val am = getSystemService(Context.ACCESSIBILITY_SERVICE) as AccessibilityManager
val enabled = am.getEnabledAccessibilityServiceList(AccessibilityServiceInfo.FEEDBACK_ALL_MASK)
val me = "$packageName/"
return enabled.any { info ->
info.resolveInfo?.serviceInfo?.let { si ->
si.packageName == packageName
} == true || info.id.contains(me) || info.id.contains("CrklAccessibilityService")
}
}
private fun maybeRequestMic() {
if (ContextCompat.checkSelfPermission(this, Manifest.permission.RECORD_AUDIO)
!= PackageManager.PERMISSION_GRANTED
) {
micPermission.launch(Manifest.permission.RECORD_AUDIO)
}
}
private fun maybeRequestNotifications() {
if (Build.VERSION.SDK_INT >= 33 &&
ContextCompat.checkSelfPermission(this, Manifest.permission.POST_NOTIFICATIONS)
!= PackageManager.PERMISSION_GRANTED
) {
notifPermission.launch(Manifest.permission.POST_NOTIFICATIONS)
}
}
}
@Composable
fun MainScreen(onOpenAccessibilitySettings: () -> Unit) {
fun MainScreen(
captureReady: Boolean,
integrationHint: String,
accessibilityEnabled: Boolean,
showOnboarding: Boolean,
onRefreshA11y: () -> Unit,
onOpenAccessibilitySettings: () -> Unit,
onOpenIntegrations: () -> Unit,
onOpenTestFixtures: () -> Unit,
onEnableCapture: () -> Unit,
onSkipOnboarding: () -> Unit,
onAddQsTile: () -> Unit = {},
canAddQsTile: Boolean = false,
overlayReady: Boolean = false
) {
Column(
modifier = Modifier
.fillMaxSize()
.verticalScroll(rememberScrollState())
.padding(24.dp),
horizontalAlignment = Alignment.CenterHorizontally,
verticalArrangement = Arrangement.Center
verticalArrangement = Arrangement.Top
) {
Spacer(modifier = Modifier.height(12.dp))
Text(
text = "Welcome to Crkl",
text = "Circle",
style = MaterialTheme.typography.headlineLarge,
textAlign = TextAlign.Center
)
Spacer(modifier = Modifier.height(16.dp))
Spacer(modifier = Modifier.height(8.dp))
Text(
text = "Privacy-first, on-device AI assistant",
text = "Circle text → translate, copy, explain, share, or Vikunja",
style = MaterialTheme.typography.bodyLarge,
textAlign = TextAlign.Center,
color = MaterialTheme.colorScheme.secondary
)
Spacer(modifier = Modifier.height(32.dp))
if (!accessibilityEnabled) {
Spacer(modifier = Modifier.height(20.dp))
Card(
modifier = Modifier.fillMaxWidth(),
colors = CardDefaults.cardColors(
containerColor = MaterialTheme.colorScheme.errorContainer
)
) {
Column(
modifier = Modifier.padding(16.dp),
verticalArrangement = Arrangement.spacedBy(10.dp)
) {
Text(
"Accessibility is off",
fontWeight = FontWeight.Bold,
style = MaterialTheme.typography.titleMedium,
color = MaterialTheme.colorScheme.onErrorContainer
)
Text(
"The floating C won’t appear until you enable Circle Overlay. " +
"Re-enable after every reinstall.",
style = MaterialTheme.typography.bodySmall,
color = MaterialTheme.colorScheme.onErrorContainer
)
Button(
onClick = {
onOpenAccessibilitySettings()
onRefreshA11y()
},
modifier = Modifier.fillMaxWidth()
) {
Text("Enable Accessibility")
}
}
}
}
if (showOnboarding) {
Spacer(modifier = Modifier.height(20.dp))
Card(
modifier = Modifier.fillMaxWidth(),
colors = CardDefaults.cardColors(
containerColor = MaterialTheme.colorScheme.primaryContainer
)
) {
Column(modifier = Modifier.padding(16.dp), verticalArrangement = Arrangement.spacedBy(10.dp)) {
Text("Get started", fontWeight = FontWeight.Bold, style = MaterialTheme.typography.titleMedium)
Text(
"Three steps — then circle anything on screen.",
style = MaterialTheme.typography.bodySmall
)
OnboardStep(
number = 1,
title = "Enable Accessibility",
done = accessibilityEnabled,
actionLabel = if (accessibilityEnabled) "Enabled" else "Open settings",
onAction = {
onOpenAccessibilitySettings()
onRefreshA11y()
},
enabled = !accessibilityEnabled
)
OnboardStep(
number = 2,
title = "Integrations (optional)",
done = false,
actionLabel = "Open Integrations",
onAction = onOpenIntegrations,
enabled = true,
detail = "Vikunja token, translate target, circle style"
)
OnboardStep(
number = 3,
title = "Try a circle",
done = false,
actionLabel = "Open test fixtures",
onAction = onOpenTestFixtures,
enabled = true,
detail = "Tap the C → closed loop around text → Translate / Copy / Explain"
)
Row(
modifier = Modifier.fillMaxWidth(),
horizontalArrangement = Arrangement.End
) {
TextButton(onClick = onSkipOnboarding) {
Text("Skip for now")
}
}
}
}
}
Spacer(modifier = Modifier.height(20.dp))
Card(
modifier = Modifier.fillMaxWidth(),
colors = CardDefaults.cardColors(
containerColor = MaterialTheme.colorScheme.secondaryContainer
)
) {
Column(
modifier = Modifier.padding(16.dp)
) {
Column(modifier = Modifier.padding(16.dp)) {
Text(
text = "🔒 Your data stays on your device",
style = MaterialTheme.typography.bodyMedium
text = if (accessibilityEnabled) "Accessibility: ON — floating C ready" else "Accessibility: off — enable to show floating C",
style = MaterialTheme.typography.bodySmall,
fontWeight = FontWeight.SemiBold
)
Spacer(modifier = Modifier.height(8.dp))
Text(
text = "🎯 Circle any content for instant AI help",
style = MaterialTheme.typography.bodyMedium
)
Text(text = integrationHint, style = MaterialTheme.typography.bodySmall)
Spacer(modifier = Modifier.height(8.dp))
Text(
text = "🧠 Local inference with no cloud calls",
style = MaterialTheme.typography.bodyMedium
text = if (captureReady) "Screen/audio capture: ON" else "Screen/audio capture: off (optional)",
style = MaterialTheme.typography.bodySmall
)
}
}
Spacer(modifier = Modifier.height(32.dp))
Text(
text = "To get started, enable the Crkl Accessibility Service:",
style = MaterialTheme.typography.bodyMedium,
textAlign = TextAlign.Center
)
Spacer(modifier = Modifier.height(16.dp))
Button(
onClick = onOpenAccessibilitySettings,
modifier = Modifier.fillMaxWidth()
) {
Text("Open Accessibility Settings")
Spacer(modifier = Modifier.height(20.dp))
Button(onClick = {
onOpenAccessibilitySettings()
onRefreshA11y()
}, modifier = Modifier.fillMaxWidth()) {
Text(if (accessibilityEnabled) "Accessibility settings" else "1 · Enable Accessibility")
}
Spacer(modifier = Modifier.height(10.dp))
Button(onClick = onOpenIntegrations, modifier = Modifier.fillMaxWidth()) {
Text("2 · Integrations")
}
Spacer(modifier = Modifier.height(10.dp))
Button(onClick = onOpenTestFixtures, modifier = Modifier.fillMaxWidth()) {
Text("3 · ${stringResource(R.string.open_test_fixtures)}")
}
Spacer(modifier = Modifier.height(10.dp))
OutlinedButton(onClick = onEnableCapture, modifier = Modifier.fillMaxWidth()) {
Text(if (captureReady) "Recapture screen/audio" else "Optional · screen/audio capture")
}
if (canAddQsTile) {
Spacer(modifier = Modifier.height(10.dp))
OutlinedButton(onClick = onAddQsTile, modifier = Modifier.fillMaxWidth()) {
Text(stringResource(R.string.add_qs_tile))
}
}
Spacer(modifier = Modifier.height(16.dp))
Text(
text = "In Settings, find and enable 'Crkl Overlay Service'",
text = buildString {
append("Shortcuts: QS tile “Circle”")
if (overlayReady) append(" (ready)")
append(" · Accessibility button · hold both volume keys")
},
style = MaterialTheme.typography.bodySmall,
textAlign = TextAlign.Center,
color = MaterialTheme.colorScheme.onSurfaceVariant
)
Spacer(modifier = Modifier.height(8.dp))
Text(
text = "Panel chips: Translate · Copy · Explain · Share · Vikunja",
style = MaterialTheme.typography.bodySmall,
textAlign = TextAlign.Center,
color = MaterialTheme.colorScheme.onSurfaceVariant
)
Spacer(modifier = Modifier.height(24.dp))
}
}
@Composable
private fun OnboardStep(
number: Int,
title: String,
done: Boolean,
actionLabel: String,
onAction: () -> Unit,
enabled: Boolean,
detail: String? = null
) {
Column(verticalArrangement = Arrangement.spacedBy(4.dp)) {
Text(
text = "$number. $title" + if (done) " ✓" else "",
fontWeight = FontWeight.SemiBold,
style = MaterialTheme.typography.bodyMedium
)
if (detail != null) {
Text(detail, style = MaterialTheme.typography.bodySmall, color = MaterialTheme.colorScheme.onSurfaceVariant)
}
OutlinedButton(
onClick = onAction,
enabled = enabled || done,
modifier = Modifier.fillMaxWidth()
) {
Text(actionLabel)
}
}
}
app/src/main/kotlin/com/example/crkl/accessibility/CrklAccessibilityService.kt
+461 -103
View File
@@ -1,149 +1,507 @@
package com.example.crkl.accessibility
import android.accessibilityservice.AccessibilityService
import android.content.pm.PackageManager
import android.graphics.PixelFormat
import android.graphics.RectF
import android.os.Build
import android.os.Handler
import android.os.Looper
import android.util.Log
import android.view.Gravity
import android.view.WindowManager
import android.view.accessibility.AccessibilityEvent
import androidx.core.content.ContextCompat
import com.example.crkl.agent.ActionExecutor
import com.example.crkl.agent.AssistEngine
import com.example.crkl.agent.VoiceIntent
import com.example.crkl.media.MediaAssistPipeline
import com.example.crkl.model.DeviceSpeechStt
import com.example.crkl.model.MediaPipeLocalLlm
import com.example.crkl.integrations.IntegrationSettings
import com.example.crkl.ui.ResultPanelView
import com.example.crkl.vision.ContentCapture
import com.example.crkl.vision.ScreenOcr
import com.example.crkl.vision.ScreenshotCapturer
import kotlinx.coroutines.CoroutineScope
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.SupervisorJob
import kotlinx.coroutines.cancel
import kotlinx.coroutines.delay
import kotlinx.coroutines.launch
import kotlinx.coroutines.withContext
/**
* Crkl Accessibility Service
*
* This service monitors accessibility events to detect user interactions
* and provides AI-powered assistance based on selected screen content.
*
* Privacy: All processing is local. No data leaves the device.
*
* Note: No overlay is created to avoid blocking touch events.
* System overlay + selection pipeline.
*
* Flow: FAB → circle → extract/media → assist → panel → commands → Vikunja / mailto / device calendar.
*/
class CrklAccessibilityService : AccessibilityService() {
private val TAG = "CrklAccessibilityService"
private val tagName = "CrklAccessibilityService"
private val mainHandler = Handler(Looper.getMainLooper())
private var overlayView: OverlayView? = null
private var resultPanel: ResultPanelView? = null
private var windowManager: WindowManager? = null
private val serviceScope = CoroutineScope(SupervisorJob() + Dispatchers.Main)
private val serviceScope = CoroutineScope(SupervisorJob() + Dispatchers.Main.immediate)
private lateinit var localLlm: MediaPipeLocalLlm
private lateinit var assistEngine: AssistEngine
private lateinit var localStt: DeviceSpeechStt
private lateinit var actionExecutor: ActionExecutor
private var lastSession: ActionExecutor.Session? = null
override fun onServiceConnected() {
super.onServiceConnected()
Log.d(TAG, "Crkl Accessibility Service connected")
// Create floating action button for Crkl activation
Log.d(tagName, "Crkl Accessibility Service connected")
localLlm = MediaPipeLocalLlm(this)
assistEngine = AssistEngine(localLlm)
localStt = DeviceSpeechStt(this)
actionExecutor = ActionExecutor(
context = this,
explainText = { text -> assistEngine.explain(text) }
)
setupFloatingButton()
Log.d(TAG, "Service ready - floating button created")
applyOverlayStyle()
CrklOverlayBridge.bind(this)
registerAccessibilityButton()
serviceScope.launch(Dispatchers.Default) {
assistEngine.warmUp()
}
}
private fun registerAccessibilityButton() {
if (Build.VERSION.SDK_INT < 26) return
try {
val controller = accessibilityButtonController
controller.registerAccessibilityButtonCallback(
object : android.accessibilityservice.AccessibilityButtonController.AccessibilityButtonCallback() {
override fun onClicked(
controller: android.accessibilityservice.AccessibilityButtonController
) {
enterCircleModeFromShortcut()
}
},
mainHandler
)
} catch (e: Exception) {
Log.w(tagName, "Accessibility button not available", e)
}
}
/**
* Called from Quick Settings tile or Accessibility button.
* Shows full-screen circle mode immediately.
*/
fun enterCircleModeFromShortcut() {
mainHandler.post {
try {
if (overlayView == null) {
setupFloatingButton()
applyOverlayStyle()
}
val view = overlayView
if (view == null) {
android.widget.Toast.makeText(
this,
"Crkl overlay not ready — open the app once",
android.widget.Toast.LENGTH_LONG
).show()
return@post
}
dismissResultPanel()
if (!view.isInOverlayMode()) {
view.enterOverlayMode()
}
} catch (e: Exception) {
Log.e(tagName, "enterCircleModeFromShortcut failed", e)
}
}
}
private fun applyOverlayStyle() {
val s = IntegrationSettings(this)
overlayView?.applyStyle(
colorArgb = s.circleColorArgb(),
neon = s.circleNeon,
strokeDp = s.circleStrokeDp()
)
}
private fun setupFloatingButton() {
try {
windowManager = getSystemService(WINDOW_SERVICE) as WindowManager
// Create overlay view with callback for mode changes
overlayView = OverlayView(this) { isOverlayMode ->
updateOverlayMode(isOverlayMode)
}
// Configure window layout parameters for floating button
val params = WindowManager.LayoutParams(
WindowManager.LayoutParams.WRAP_CONTENT,
WindowManager.LayoutParams.WRAP_CONTENT,
WindowManager.LayoutParams.TYPE_ACCESSIBILITY_OVERLAY,
WindowManager.LayoutParams.FLAG_NOT_FOCUSABLE or
WindowManager.LayoutParams.FLAG_NOT_TOUCH_MODAL,
PixelFormat.TRANSLUCENT
overlayView = OverlayView(
context = this,
onModeChanged = { isOverlayMode -> updateOverlayMode(isOverlayMode) },
onSelectionComplete = { bounds -> handleSelection(bounds) }
)
// Position in bottom right corner
params.gravity = Gravity.BOTTOM or Gravity.END
params.x = 50
params.y = 100
// Add floating button to window manager
windowManager?.addView(overlayView, params)
Log.d(TAG, "Floating button created successfully")
windowManager?.addView(overlayView, floatingButtonParams())
Log.d(tagName, "Floating button created")
} catch (e: Exception) {
Log.e(TAG, "Error creating floating button", e)
Log.e(tagName, "Error creating floating button", e)
}
}
private fun floatingButtonParams(): WindowManager.LayoutParams =
WindowManager.LayoutParams(
WindowManager.LayoutParams.WRAP_CONTENT,
WindowManager.LayoutParams.WRAP_CONTENT,
WindowManager.LayoutParams.TYPE_ACCESSIBILITY_OVERLAY,
WindowManager.LayoutParams.FLAG_NOT_FOCUSABLE or
WindowManager.LayoutParams.FLAG_NOT_TOUCH_MODAL or
WindowManager.LayoutParams.FLAG_LAYOUT_NO_LIMITS,
PixelFormat.TRANSLUCENT
).apply {
gravity = Gravity.BOTTOM or Gravity.END
x = dp(16)
y = navigationBarHeight() + dp(24)
}
private fun fullScreenParams(): WindowManager.LayoutParams {
val metrics = resources.displayMetrics
val nav = navigationBarHeight()
val status = statusBarHeight()
val height = (metrics.heightPixels - nav - status).coerceAtLeast(dp(200))
return WindowManager.LayoutParams(
WindowManager.LayoutParams.MATCH_PARENT,
height,
WindowManager.LayoutParams.TYPE_ACCESSIBILITY_OVERLAY,
WindowManager.LayoutParams.FLAG_NOT_FOCUSABLE or
WindowManager.LayoutParams.FLAG_NOT_TOUCH_MODAL,
PixelFormat.TRANSLUCENT
).apply {
gravity = Gravity.TOP or Gravity.START
x = 0
y = status
}
}
private fun resultPanelParams(): WindowManager.LayoutParams {
val metrics = resources.displayMetrics
val width = (metrics.widthPixels * 0.94f).toInt()
val height = (metrics.heightPixels * 0.48f).toInt()
val nav = navigationBarHeight()
return WindowManager.LayoutParams(
width,
height,
WindowManager.LayoutParams.TYPE_ACCESSIBILITY_OVERLAY,
WindowManager.LayoutParams.FLAG_NOT_TOUCH_MODAL,
PixelFormat.TRANSLUCENT
).apply {
gravity = Gravity.BOTTOM or Gravity.CENTER_HORIZONTAL
y = nav + dp(10)
flags = flags and WindowManager.LayoutParams.FLAG_NOT_FOCUSABLE.inv()
}
}
/** Fast path: resize in place — do NOT remove/re-add the window (that lagged the C button). */
private fun updateOverlayMode(isOverlayMode: Boolean) {
val view = overlayView ?: return
val wm = windowManager ?: return
try {
overlayView?.let { view ->
windowManager?.let { wm ->
// Remove current view
wm.removeView(view)
// Create new parameters based on mode
val params = if (isOverlayMode) {
// Full screen overlay parameters
WindowManager.LayoutParams(
WindowManager.LayoutParams.MATCH_PARENT,
WindowManager.LayoutParams.MATCH_PARENT,
WindowManager.LayoutParams.TYPE_ACCESSIBILITY_OVERLAY,
WindowManager.LayoutParams.FLAG_NOT_FOCUSABLE or
WindowManager.LayoutParams.FLAG_NOT_TOUCH_MODAL or
WindowManager.LayoutParams.FLAG_LAYOUT_IN_SCREEN,
PixelFormat.TRANSLUCENT
).apply {
gravity = Gravity.TOP or Gravity.LEFT
x = 0
y = 0
}
} else {
// Floating button parameters
WindowManager.LayoutParams(
WindowManager.LayoutParams.WRAP_CONTENT,
WindowManager.LayoutParams.WRAP_CONTENT,
WindowManager.LayoutParams.TYPE_ACCESSIBILITY_OVERLAY,
WindowManager.LayoutParams.FLAG_NOT_FOCUSABLE or
WindowManager.LayoutParams.FLAG_NOT_TOUCH_MODAL,
PixelFormat.TRANSLUCENT
).apply {
gravity = Gravity.BOTTOM or Gravity.END
x = 50
y = 100
}
if (isOverlayMode) {
dismissResultPanel()
applyOverlayStyle()
}
wm.updateViewLayout(
view,
if (isOverlayMode) fullScreenParams() else floatingButtonParams()
)
view.requestLayout()
} catch (e: IllegalArgumentException) {
// View not attached yet — fall back once.
Log.w(tagName, "updateViewLayout failed, re-adding", e)
try {
wm.addView(view, if (isOverlayMode) fullScreenParams() else floatingButtonParams())
} catch (e2: Exception) {
Log.e(tagName, "re-add overlay failed", e2)
}
} catch (e: Exception) {
Log.e(tagName, "Error updating overlay mode", e)
}
}
private fun handleSelection(bounds: RectF) {
serviceScope.launch {
overlayView?.exitOverlayMode()
showResultPanel(
AssistEngine.AssistResponse(
title = "Reading…",
meta = "",
body = "Looking for text in that circle…"
)
)
val capture = withContext(Dispatchers.Default) {
var windowsSnapshot: List<android.view.accessibility.AccessibilityWindowInfo>? = null
var rootSnapshot: android.view.accessibility.AccessibilityNodeInfo? = null
withContext(Dispatchers.Main) {
overlayView?.visibility = android.view.View.INVISIBLE
resultPanel?.visibility = android.view.View.INVISIBLE
}
delay(50)
withContext(Dispatchers.Main) {
windowsSnapshot = windows
rootSnapshot = rootInActiveWindow
}
val a11y = try {
ContentCapture.fromA11y(windowsSnapshot, rootSnapshot, bounds)
} finally {
withContext(Dispatchers.Main) {
rootSnapshot?.recycle()
}
// Add view with new parameters
wm.addView(view, params)
Log.d(TAG, "Overlay mode updated: ${if (isOverlayMode) "full-screen" else "floating button"}")
}
// OCR when a11y is thin, or always merge for image-like sparse text.
val needOcr = a11y.isEmpty || a11y.text.length < 40
val merged = if (needOcr) {
withContext(Dispatchers.Main) {
val debug = IntegrationSettings(this@CrklAccessibilityService).showDebugMeta
updateResultPanel(
AssistEngine.AssistResponse(
title = "Reading…",
meta = if (debug) "OCR…" else "",
body = "Looking for text in that circle…"
)
)
}
val bmp = ScreenshotCapturer.captureRegion(this@CrklAccessibilityService, bounds)
try {
if (bmp != null) {
val ocr = ScreenOcr.recognize(bmp)
ContentCapture.mergeWithOcr(a11y, ocr.text, ocr.blockCount)
} else {
a11y
}
} finally {
bmp?.recycle()
}
} else {
a11y
}
withContext(Dispatchers.Main) {
overlayView?.visibility = android.view.View.VISIBLE
resultPanel?.visibility = android.view.View.VISIBLE
}
merged
}
// Media path: circled video/audio → play/listen → transcript → summarize.
val mediaPipeline = MediaAssistPipeline(this@CrklAccessibilityService, assistEngine)
val mediaSlot = mediaPipeline.resolveSlot(bounds, capture)
if (mediaSlot != null) {
updateResultPanel(
AssistEngine.AssistResponse(
title = "Crkl",
meta = "media…",
body = "Playing and listening to ${mediaSlot.kind.name.lowercase()}…"
)
)
val outcome = mediaPipeline.process(mediaSlot) { msg ->
updateResultPanel(
AssistEngine.AssistResponse(
title = "Crkl",
meta = "media…",
body = msg
)
)
}
lastSession = ActionExecutor.Session(
title = outcome.response.title,
body = outcome.response.body,
transcript = outcome.transcript,
workingList = VoiceIntent.extractListFromBody(
outcome.response.body + "\n" + outcome.transcript
).toMutableList()
)
updateResultPanel(
outcome.response.copy(
body = outcome.response.body +
"\n\n—\nCommands: “add to my todo” (Vikunja) · “email me the list” (mailto) · “what’s on my calendar”."
)
)
return@launch
}
val extraction = capture.toExtraction()
val response = withContext(Dispatchers.Default) {
assistEngine.respond(extraction)
}
lastSession = ActionExecutor.Session(
title = response.title,
body = response.body,
workingList = VoiceIntent.extractListFromBody(response.body).toMutableList()
)
val debug = IntegrationSettings(this@CrklAccessibilityService).showDebugMeta
val userMeta = if (debug) {
buildString {
append(response.meta.ifBlank { "debug" })
append(" · ${capture.source.name.lowercase()}")
if (capture.ocrBlockCount > 0) append(" · ocr=${capture.ocrBlockCount}")
if (capture.packageName != null) {
append(" · ${capture.packageName.substringAfterLast('.')}")
}
}
} else {
""
}
updateResultPanel(response.copy(meta = userMeta))
}
}
private fun showResultPanel(response: AssistEngine.AssistResponse) {
mainHandler.post {
try {
dismissResultPanel()
val panel = ResultPanelView(
context = this,
onDismiss = { dismissResultPanel() },
onListen = { startSttFromPanel() },
onShareList = { runVoiceAction(VoiceIntent.Parsed(VoiceIntent.Kind.SHARE_LIST, raw = "share")) },
onAddTodo = { runVoiceAction(VoiceIntent.Parsed(VoiceIntent.Kind.ADD_TO_TODO, raw = "add to todo")) },
onTranslate = { runVoiceAction(VoiceIntent.Parsed(VoiceIntent.Kind.TRANSLATE, raw = "translate")) },
onCopy = { runVoiceAction(VoiceIntent.Parsed(VoiceIntent.Kind.COPY, raw = "copy")) },
onExplain = { runVoiceAction(VoiceIntent.Parsed(VoiceIntent.Kind.EXPLAIN, raw = "explain")) }
)
panel.showResult(response.title, response.meta, response.body)
resultPanel = panel
windowManager?.addView(panel, resultPanelParams())
panel.playEnter()
} catch (e: Exception) {
Log.e(tagName, "Failed to show result panel", e)
}
}
}
private fun updateResultPanel(response: AssistEngine.AssistResponse) {
mainHandler.post {
val panel = resultPanel
if (panel == null) {
showResultPanel(response)
} else {
panel.showResult(response.title, response.meta, response.body)
}
}
}
private fun startSttFromPanel() {
val micGranted = ContextCompat.checkSelfPermission(
this,
android.Manifest.permission.RECORD_AUDIO
) == PackageManager.PERMISSION_GRANTED
if (!micGranted) {
resultPanel?.appendBody(
"Mic permission needed. Open Crkl app → allow microphone, then retry Speak."
)
return
}
if (!localStt.isAvailable) {
resultPanel?.appendBody("Speech recognition is not available on this device/emulator.")
return
}
serviceScope.launch {
resultPanel?.setListening(true)
val text = try {
localStt.listen(8_000L)
} finally {
resultPanel?.setListening(false)
}
if (text.isBlank()) {
resultPanel?.appendBody(
"STT: (no speech heard)\n" +
"Tip: click the phone window once, then Speak again. Emulator needs a Mac mic."
)
return@launch
}
resultPanel?.appendBody("You said: $text")
val parsed = VoiceIntent.parse(text)
runVoiceAction(parsed)
}
}
private fun runVoiceAction(parsed: VoiceIntent.Parsed) {
serviceScope.launch {
Log.i(tagName, "action kind=${parsed.kind} raw=${parsed.raw}")
val result = withContext(Dispatchers.IO) {
actionExecutor.execute(parsed, lastSession)
}
Log.i(tagName, "action result ok=${result.ok} msg=${result.message.take(160)}")
lastSession?.let { session ->
if (session.workingList.isNotEmpty()) {
lastSession = session.copy(
body = buildString {
appendLine(session.title)
appendLine()
session.workingList.forEach { appendLine("• $it") }
}
)
}
}
} catch (e: Exception) {
Log.e(TAG, "Error updating overlay mode", e)
}
}
override fun onAccessibilityEvent(event: AccessibilityEvent?) {
// We'll use this later for content detection
// For now, just log events
event?.let {
if (it.eventType == AccessibilityEvent.TYPE_WINDOW_STATE_CHANGED) {
Log.d(TAG, "Window changed: ${it.packageName}")
withContext(Dispatchers.Main) {
resultPanel?.appendBody("—\n${result.message}")
android.widget.Toast.makeText(
this@CrklAccessibilityService,
result.message.lineSequence().firstOrNull { it.isNotBlank() }?.take(80)
?: if (result.ok) "Done" else "Failed",
android.widget.Toast.LENGTH_LONG
).show()
}
}
}
override fun onInterrupt() {
Log.d(TAG, "Service interrupted")
private fun dismissResultPanel() {
if (::localStt.isInitialized) {
localStt.cancel()
}
resultPanel?.let { panel ->
try {
windowManager?.removeView(panel)
} catch (e: Exception) {
Log.w(tagName, "Result panel already removed", e)
}
}
resultPanel = null
}
override fun onAccessibilityEvent(event: AccessibilityEvent?) = Unit
override fun onInterrupt() {
Log.d(tagName, "Service interrupted")
}
override fun onDestroy() {
super.onDestroy()
Log.d(TAG, "Service destroyed")
// Clean up floating button
CrklOverlayBridge.unbind(this)
dismissResultPanel()
overlayView?.let {
windowManager?.removeView(it)
try {
windowManager?.removeView(it)
} catch (_: Exception) {
}
overlayView = null
}
if (::localLlm.isInitialized) localLlm.close()
serviceScope.cancel()
Log.d(tagName, "Service destroyed")
}
}
private fun navigationBarHeight(): Int {
val id = resources.getIdentifier("navigation_bar_height", "dimen", "android")
return if (id > 0) resources.getDimensionPixelSize(id) else dp(48)
}
private fun statusBarHeight(): Int {
val id = resources.getIdentifier("status_bar_height", "dimen", "android")
return if (id > 0) resources.getDimensionPixelSize(id) else dp(24)
}
private fun dp(value: Int): Int =
(value * resources.displayMetrics.density).toInt()
}
app/src/main/kotlin/com/example/crkl/accessibility/CrklCircleTileService.kt
+71
View File
@@ -0,0 +1,71 @@
package com.example.crkl.accessibility
import android.app.PendingIntent
import android.content.Intent
import android.os.Build
import android.service.quicksettings.Tile
import android.service.quicksettings.TileService
import android.widget.Toast
import com.example.crkl.MainActivity
import com.example.crkl.R
/**
* Quick Settings tile: tap → enter Crkl circle mode (Accessibility must be on).
*/
class CrklCircleTileService : TileService() {
override fun onStartListening() {
super.onStartListening()
refreshTile()
}
override fun onClick() {
super.onClick()
unlockAndRun {
if (CrklOverlayBridge.enterCircleMode()) {
Toast.makeText(this, "Circle text — close the loop", Toast.LENGTH_SHORT).show()
refreshTile()
} else {
Toast.makeText(
this,
"Enable Circle Overlay first",
Toast.LENGTH_LONG
).show()
openApp()
}
}
}
private fun refreshTile() {
val tile = qsTile ?: return
val ready = CrklOverlayBridge.isReady()
tile.label = getString(R.string.qs_tile_label)
if (Build.VERSION.SDK_INT >= 29) {
tile.subtitle = if (ready) {
getString(R.string.qs_tile_subtitle_ready)
} else {
getString(R.string.qs_tile_subtitle_off)
}
}
tile.state = if (ready) Tile.STATE_ACTIVE else Tile.STATE_INACTIVE
tile.updateTile()
}
private fun openApp() {
val intent = Intent(this, MainActivity::class.java).apply {
addFlags(Intent.FLAG_ACTIVITY_NEW_TASK or Intent.FLAG_ACTIVITY_CLEAR_TOP)
}
if (Build.VERSION.SDK_INT >= 34) {
val pi = PendingIntent.getActivity(
this,
0,
intent,
PendingIntent.FLAG_IMMUTABLE or PendingIntent.FLAG_UPDATE_CURRENT
)
startActivityAndCollapse(pi)
} else {
@Suppress("DEPRECATION")
startActivityAndCollapse(intent)
}
}
}
app/src/main/kotlin/com/example/crkl/accessibility/CrklOverlayBridge.kt
+27
View File
@@ -0,0 +1,27 @@
package com.example.crkl.accessibility
/**
* Process-wide bridge so Quick Settings / shortcuts can open circle mode
* without binding to the accessibility service.
*/
object CrklOverlayBridge {
@Volatile
private var service: CrklAccessibilityService? = null
fun bind(svc: CrklAccessibilityService) {
service = svc
}
fun unbind(svc: CrklAccessibilityService) {
if (service === svc) service = null
}
fun isReady(): Boolean = service != null
/** @return true if circle mode was requested on a live service. */
fun enterCircleMode(): Boolean {
val svc = service ?: return false
svc.enterCircleModeFromShortcut()
return true
}
}
app/src/main/kotlin/com/example/crkl/accessibility/OverlayView.kt
+302 -157
View File
@@ -1,250 +1,395 @@
package com.example.crkl.accessibility
import android.content.Context
import android.graphics.BlurMaskFilter
import android.graphics.Canvas
import android.graphics.Color
import android.graphics.Paint
import android.graphics.PointF
import android.graphics.RectF
import android.os.Build
import android.util.Log
import android.view.Gravity
import android.view.View.MeasureSpec
import android.util.TypedValue
import android.view.HapticFeedbackConstants
import android.view.MotionEvent
import android.view.View
import android.widget.Toast
import android.view.View.MeasureSpec
import com.example.crkl.ui.CrklUi
import kotlin.math.sqrt
/**
* Crkl Overlay View
*
* This view provides a floating action button that toggles to a full-screen overlay
* for capturing touch gestures and circle selections. It reports selection bounds
* for content analysis.
* Floating “C” that expands into a draw overlay.
* Stroke color / neon / thickness come from [applyStyle].
*/
class OverlayView(
context: Context,
private val onModeChanged: (Boolean) -> Unit = {}
private val onModeChanged: (Boolean) -> Unit = {},
private val onSelectionComplete: (RectF) -> Unit = {}
) : View(context) {
private val TAG = "OverlayView"
// Button properties
private val buttonSize = 80f
private var buttonColor = Color.BLUE
private val buttonPaint = Paint().apply {
color = buttonColor
private val tagName = "OverlayView"
private val buttonSizePx = dp(56f)
private val buttonFillPaint = Paint(Paint.ANTI_ALIAS_FLAG).apply {
color = CrklUi.Ink
style = Paint.Style.FILL
isAntiAlias = true
}
// Overlay mode properties
private val buttonRingPaint = Paint(Paint.ANTI_ALIAS_FLAG).apply {
color = CrklUi.Teal
style = Paint.Style.STROKE
strokeWidth = dp(2.5f)
}
private val buttonShadowPaint = Paint(Paint.ANTI_ALIAS_FLAG).apply {
color = Color.parseColor("#55000000")
maskFilter = BlurMaskFilter(dp(10f), BlurMaskFilter.Blur.NORMAL)
}
private var isOverlayMode = false
private val overlayPaint = Paint().apply {
color = Color.argb(128, 0, 0, 0) // Semi-transparent black
private val dimPaint = Paint().apply {
color = CrklUi.Dim
style = Paint.Style.FILL
}
private val instructionPaint = Paint().apply {
private val instructionPaint = Paint(Paint.ANTI_ALIAS_FLAG).apply {
color = Color.WHITE
textSize = 48f
textSize = sp(14f)
textAlign = Paint.Align.CENTER
isAntiAlias = true
}
// Drawing properties
private val hintPillPaint = Paint(Paint.ANTI_ALIAS_FLAG).apply {
color = Color.parseColor("#E61C2B38")
style = Paint.Style.FILL
}
private val hintRingPaint = Paint(Paint.ANTI_ALIAS_FLAG).apply {
color = CrklUi.Teal
style = Paint.Style.STROKE
strokeWidth = dp(1.2f)
alpha = 160
}
private var isDrawing = false
private var touchPath = mutableListOf<PointF>()
private val pathPaint = Paint().apply {
color = Color.YELLOW
strokeWidth = 8f
private var hintMessage = "Draw a closed loop around text"
private var failFlashUntil = 0L
private val glowPaint = Paint(Paint.ANTI_ALIAS_FLAG).apply {
style = Paint.Style.STROKE
isAntiAlias = true
strokeCap = Paint.Cap.ROUND
strokeJoin = Paint.Join.ROUND
}
// Selection bounds tracking
private var selectionBounds: android.graphics.RectF? = null
private val boundsPaint = Paint().apply {
color = Color.GREEN
strokeWidth = 4f
private val pathPaint = Paint(Paint.ANTI_ALIAS_FLAG).apply {
style = Paint.Style.STROKE
isAntiAlias = true
strokeCap = Paint.Cap.ROUND
strokeJoin = Paint.Join.ROUND
}
private val textPaint = Paint().apply {
private val boundsPaint = Paint(Paint.ANTI_ALIAS_FLAG).apply {
style = Paint.Style.STROKE
strokeWidth = dp(1.5f)
color = Color.parseColor("#88FFFFFF")
}
private val fillFlashPaint = Paint(Paint.ANTI_ALIAS_FLAG).apply {
style = Paint.Style.FILL
color = Color.argb(40, 31, 167, 160)
}
private var selectionBounds: RectF? = null
private var neonEnabled = true
private var successFlash = false
private val textPaint = Paint(Paint.ANTI_ALIAS_FLAG).apply {
color = Color.WHITE
textSize = 32f
textSize = sp(17f)
textAlign = Paint.Align.CENTER
isAntiAlias = true
isFakeBoldText = true
}
private val exitBgPaint = Paint(Paint.ANTI_ALIAS_FLAG).apply {
color = Color.parseColor("#CC1C2B38")
style = Paint.Style.FILL
}
private val exitTextPaint = Paint(Paint.ANTI_ALIAS_FLAG).apply {
color = Color.WHITE
textSize = sp(13f)
textAlign = Paint.Align.CENTER
}
private val exitRect = RectF()
private var touchStartedInExit = false
init {
setLayerType(LAYER_TYPE_SOFTWARE, null)
setBackgroundColor(Color.TRANSPARENT)
isClickable = true
isFocusable = true
isFocusableInTouchMode = true
isFocusable = false
applyStyle(CrklUi.Teal, neon = true, strokeDp = 4.5f)
setOnClickListener {
Log.d(TAG, "Crkl floating button clicked!")
toggleOverlayMode()
if (!isOverlayMode) {
Log.d(tagName, "Crkl floating button clicked")
toggleOverlayMode()
}
}
Log.d(TAG, "Simple floating button initialized")
}
private fun toggleOverlayMode() {
isOverlayMode = !isOverlayMode
if (isOverlayMode) {
Log.d(TAG, "Overlay mode activated - draw a circle to analyze content")
showToast("Draw a circle to analyze content!")
fun applyStyle(colorArgb: Int, neon: Boolean, strokeDp: Float) {
neonEnabled = neon
pathPaint.color = colorArgb
pathPaint.strokeWidth = dp(strokeDp)
glowPaint.color = colorArgb
glowPaint.strokeWidth = dp(strokeDp + 6f)
glowPaint.alpha = if (neon) 120 else 0
glowPaint.maskFilter = if (neon) {
BlurMaskFilter(dp(12f), BlurMaskFilter.Blur.NORMAL)
} else {
Log.d(TAG, "Overlay mode deactivated")
showToast("Overlay mode disabled")
touchPath.clear()
selectionBounds = null
null
}
onModeChanged(isOverlayMode)
invalidate()
}
private fun showToast(message: String) {
val toast = Toast.makeText(context, message, Toast.LENGTH_LONG)
toast.setGravity(Gravity.TOP or Gravity.CENTER_HORIZONTAL, 0, 200)
toast.show()
Log.d(TAG, "Toast shown: $message")
fun isInOverlayMode(): Boolean = isOverlayMode
/** Enter draw mode from Quick Settings / Accessibility button. */
fun enterOverlayMode() {
if (isOverlayMode) return
isOverlayMode = true
touchPath.clear()
selectionBounds = null
touchStartedInExit = false
successFlash = false
hintMessage = "Draw a closed loop around text"
onModeChanged(true)
performHapticFeedback(HapticFeedbackConstants.CONTEXT_CLICK)
invalidate()
}
fun exitOverlayMode() {
if (!isOverlayMode) return
isOverlayMode = false
touchPath.clear()
selectionBounds = null
touchStartedInExit = false
successFlash = false
hintMessage = "Draw a closed loop around text"
onModeChanged(false)
invalidate()
}
private fun toggleOverlayMode() {
isOverlayMode = !isOverlayMode
touchPath.clear()
selectionBounds = null
touchStartedInExit = false
successFlash = false
hintMessage = "Draw a closed loop around text"
onModeChanged(isOverlayMode)
if (isOverlayMode) {
performHapticFeedback(HapticFeedbackConstants.CONTEXT_CLICK)
}
invalidate()
}
override fun onTouchEvent(event: MotionEvent?): Boolean {
if (!isOverlayMode) {
// In button mode, only handle clicks on the button itself
return super.onTouchEvent(event)
}
// In overlay mode, handle drawing
event?.let { motionEvent ->
when (motionEvent.action) {
MotionEvent.ACTION_DOWN -> {
isDrawing = true
touchPath.clear()
val motionEvent = event ?: return true
updateExitRect()
when (motionEvent.action) {
MotionEvent.ACTION_DOWN -> {
touchStartedInExit = exitRect.contains(motionEvent.x, motionEvent.y)
if (touchStartedInExit) return true
isDrawing = true
successFlash = false
touchPath.clear()
selectionBounds = null
touchPath.add(PointF(motionEvent.x, motionEvent.y))
}
MotionEvent.ACTION_MOVE -> {
if (!touchStartedInExit && isDrawing) {
touchPath.add(PointF(motionEvent.x, motionEvent.y))
Log.d(TAG, "Drawing started at (${motionEvent.x}, ${motionEvent.y})")
}
MotionEvent.ACTION_MOVE -> {
if (isDrawing) {
touchPath.add(PointF(motionEvent.x, motionEvent.y))
}
}
MotionEvent.ACTION_UP -> {
isDrawing = false
Log.d(TAG, "Drawing ended. Path points: ${touchPath.size}")
if (isCircleGesture()) {
Log.d(TAG, "Circle detected!")
calculateSelectionBounds()
showToast("Circle detected! Analyzing content...")
reportSelectionBounds()
}
}
}
invalidate()
MotionEvent.ACTION_UP, MotionEvent.ACTION_CANCEL -> {
if (touchStartedInExit && exitRect.contains(motionEvent.x, motionEvent.y)) {
toggleOverlayMode()
} else if (isDrawing) {
isDrawing = false
if (isClosedStroke()) {
calculateSelectionBounds()
selectionBounds?.let { localBounds ->
successFlash = true
performHapticFeedback(
if (Build.VERSION.SDK_INT >= 30) {
HapticFeedbackConstants.CONFIRM
} else {
HapticFeedbackConstants.LONG_PRESS
}
)
val screenBounds = toScreenBounds(localBounds)
Log.d(tagName, "Selection ready: $screenBounds")
// Brief flash then hand off — no toast spam
postDelayed({
onSelectionComplete(screenBounds)
}, 90)
}
} else {
hintMessage = "Close the loop to select"
failFlashUntil = System.currentTimeMillis() + 1400
performHapticFeedback(HapticFeedbackConstants.KEYBOARD_TAP)
postDelayed({
hintMessage = "Draw a closed loop around text"
invalidate()
}, 1400)
}
}
touchStartedInExit = false
isDrawing = false
}
}
return true // Consume touch events in overlay mode
invalidate()
return true
}
private fun isCircleGesture(): Boolean {
if (touchPath.size < 10) return false
val firstPoint = touchPath.first()
val lastPoint = touchPath.last()
val dx = (lastPoint.x - firstPoint.x).toDouble()
val dy = (lastPoint.y - firstPoint.y).toDouble()
val distance = sqrt(dx * dx + dy * dy)
return distance < 100f // Circle is closed if start and end are close
private fun updateExitRect() {
val pad = dp(16f)
val w = dp(88f)
val h = dp(36f)
exitRect.set(width - pad - w, pad, width - pad, pad + h)
}
private fun toScreenBounds(local: RectF): RectF {
val location = IntArray(2)
getLocationOnScreen(location)
return RectF(
local.left + location[0],
local.top + location[1],
local.right + location[0],
local.bottom + location[1]
)
}
private fun isClosedStroke(): Boolean {
if (touchPath.size < 12) return false
val first = touchPath.first()
val last = touchPath.last()
val closeDistance = sqrt(
(last.x - first.x) * (last.x - first.x) +
(last.y - first.y) * (last.y - first.y)
)
var pathLength = 0f
for (i in 1 until touchPath.size) {
val a = touchPath[i - 1]
val b = touchPath[i]
pathLength += sqrt((b.x - a.x) * (b.x - a.x) + (b.y - a.y) * (b.y - a.y))
}
// Slightly more forgiving close distance for smoother feel
return closeDistance < 140f && pathLength > 160f
}
private fun calculateSelectionBounds() {
if (touchPath.isEmpty()) return
var minX = Float.MAX_VALUE
var maxX = Float.MIN_VALUE
var minY = Float.MAX_VALUE
var maxY = Float.MIN_VALUE
for (point in touchPath) {
minX = minOf(minX, point.x)
maxX = maxOf(maxX, point.x)
minY = minOf(minY, point.y)
maxY = maxOf(maxY, point.y)
}
selectionBounds = android.graphics.RectF(minX, minY, maxX, maxY)
Log.d(TAG, "Selection bounds calculated: $selectionBounds")
selectionBounds = RectF(minX, minY, maxX, maxY)
}
private fun reportSelectionBounds() {
selectionBounds?.let { bounds ->
Log.d(TAG, "Selection bounds reported: left=${bounds.left}, top=${bounds.top}, right=${bounds.right}, bottom=${bounds.bottom}")
Log.d(TAG, "Selection area: ${bounds.width()} x ${bounds.height()}")
// TODO: In future, this will trigger content analysis at these bounds
}
}
override fun onDraw(canvas: Canvas) {
super.onDraw(canvas)
if (isOverlayMode) {
drawOverlayMode(canvas)
} else {
drawButtonMode(canvas)
}
if (isOverlayMode) drawOverlayMode(canvas) else drawButtonMode(canvas)
}
private fun drawButtonMode(canvas: Canvas) {
// Draw button background
val centerX = width / 2f
val centerY = height / 2f
canvas.drawCircle(centerX, centerY, buttonSize / 2, buttonPaint)
// Draw "C" text
val textY = centerY + (textPaint.textSize / 3)
canvas.drawText("C", centerX, textY, textPaint)
val cx = width / 2f
val cy = height / 2f
val r = buttonSizePx / 2f
canvas.drawCircle(cx, cy + dp(2f), r, buttonShadowPaint)
canvas.drawCircle(cx, cy, r, buttonFillPaint)
canvas.drawCircle(cx, cy, r - dp(3f), buttonRingPaint)
canvas.drawText("C", cx, cy + textPaint.textSize / 3f, textPaint)
}
private fun drawOverlayMode(canvas: Canvas) {
// Draw semi-transparent overlay
canvas.drawRect(0f, 0f, width.toFloat(), height.toFloat(), overlayPaint)
// Draw instructions
canvas.drawText("Draw a circle to analyze content", width / 2f, 200f, instructionPaint)
canvas.drawText("Tap 'C' button again to exit", width / 2f, height - 200f, instructionPaint)
// Draw the drawing path
canvas.drawRect(0f, 0f, width.toFloat(), height.toFloat(), dimPaint)
val failing = System.currentTimeMillis() < failFlashUntil
val pillW = dp(280f)
val pillH = dp(38f)
val pill = RectF(
width / 2f - pillW / 2f,
dp(52f),
width / 2f + pillW / 2f,
dp(52f) + pillH
)
canvas.drawRoundRect(pill, dp(19f), dp(19f), hintPillPaint)
canvas.drawRoundRect(pill, dp(19f), dp(19f), hintRingPaint)
instructionPaint.color = if (failing) Color.parseColor("#FFCFC4") else Color.WHITE
canvas.drawText(
hintMessage,
pill.centerX(),
pill.centerY() + instructionPaint.textSize / 3f,
instructionPaint
)
updateExitRect()
canvas.drawRoundRect(exitRect, dp(18f), dp(18f), exitBgPaint)
canvas.drawText(
"Cancel",
exitRect.centerX(),
exitRect.centerY() + exitTextPaint.textSize / 3f,
exitTextPaint
)
if (touchPath.size > 1) {
for (i in 1 until touchPath.size) {
val prev = touchPath[i - 1]
val curr = touchPath[i]
if (neonEnabled) {
canvas.drawLine(prev.x, prev.y, curr.x, curr.y, glowPaint)
}
canvas.drawLine(prev.x, prev.y, curr.x, curr.y, pathPaint)
}
}
// Draw selection bounds if available
selectionBounds?.let { bounds ->
canvas.drawRect(bounds, boundsPaint)
// Draw bounds info
val boundsText = "Selection: ${bounds.width().toInt()}x${bounds.height().toInt()}"
canvas.drawText(boundsText, bounds.centerX(), bounds.top - 20f, textPaint)
if (successFlash) {
canvas.drawRoundRect(bounds, dp(10f), dp(10f), fillFlashPaint)
}
canvas.drawRoundRect(bounds, dp(10f), dp(10f), boundsPaint)
}
}
override fun onMeasure(widthMeasureSpec: Int, heightMeasureSpec: Int) {
if (isOverlayMode) {
// In overlay mode, use full screen
setMeasuredDimension(
MeasureSpec.getSize(widthMeasureSpec),
MeasureSpec.getSize(heightMeasureSpec)
)
} else {
// In button mode, use fixed button size
setMeasuredDimension(buttonSize.toInt(), buttonSize.toInt())
val size = buttonSizePx.toInt()
setMeasuredDimension(size, size)
}
}
}
private fun dp(value: Float): Float =
TypedValue.applyDimension(
TypedValue.COMPLEX_UNIT_DIP,
value,
resources.displayMetrics
)
private fun sp(value: Float): Float =
TypedValue.applyDimension(
TypedValue.COMPLEX_UNIT_SP,
value,
resources.displayMetrics
)
}
app/src/main/kotlin/com/example/crkl/agent/.gitkeep
-2
View File
@@ -1,2 +0,0 @@
# Dialogue state management
app/src/main/kotlin/com/example/crkl/agent/ActionExecutor.kt
+266
View File
@@ -0,0 +1,266 @@
package com.example.crkl.agent
import android.content.ClipData
import android.content.ClipboardManager
import android.content.Context
import android.content.Intent
import android.util.Log
import com.example.crkl.integrations.DeviceCalendar
import com.example.crkl.integrations.GogBridgeClient
import com.example.crkl.integrations.IntegrationSettings
import com.example.crkl.integrations.OnDeviceTranslator
import com.example.crkl.integrations.RealEmailActions
import com.example.crkl.integrations.VikunjaClient
/**
* In-app actions for Circle Assist VIP: copy, explain, share, translate, Vikunja, etc.
*/
class ActionExecutor(
private val context: Context,
private val settings: IntegrationSettings = IntegrationSettings(context),
private val createTasks: suspend (titles: List<String>, description: String) -> VikunjaClient.CreateResult =
{ titles, description -> VikunjaClient(settings).createTasks(titles, description) },
private val sendViaGog: suspend (to: String, subject: String, body: String) -> GogBridgeClient.BridgeResult =
{ to, subject, body -> GogBridgeClient(settings).sendEmail(to, subject, body) },
private val translateText: suspend (text: String, targetLang: String) -> OnDeviceTranslator.Result =
{ text, target -> OnDeviceTranslator().translate(text, target) },
private val explainText: suspend (text: String) -> AssistEngine.AssistResponse =
{ text ->
AssistEngine(null).explain(text)
},
private val todayCalendar: () -> DeviceCalendar.Result =
{ DeviceCalendar.today(context) },
private val composeEmail: (subject: String, body: String, to: String) -> String =
{ subject, body, to -> RealEmailActions.compose(context, subject, body, to) },
private val openTodosUrl: (url: String) -> String =
{ url -> RealEmailActions.openWebTodos(context, url) },
private val launchIntent: (Intent) -> Unit = { context.startActivity(it) },
private val copyText: (String) -> Boolean = { text ->
try {
val cm = context.getSystemService(Context.CLIPBOARD_SERVICE) as ClipboardManager
cm.setPrimaryClip(ClipData.newPlainText("Crkl", text))
true
} catch (e: Exception) {
Log.e("ActionExecutor", "clipboard failed", e)
false
}
}
) {
private val tagName = "ActionExecutor"
data class Session(
val title: String,
val body: String,
val transcript: String = "",
val workingList: MutableList<String> = mutableListOf()
) {
fun listItems(): List<String> {
if (workingList.isNotEmpty()) return workingList.toList()
return VoiceIntent.extractListFromBody(body + "\n" + transcript)
}
fun todoItems(): List<String> {
if (workingList.isNotEmpty()) return workingList.toList()
return VoiceIntent.extractTasksForTodo(title, body, transcript)
}
fun sharePayload(): String {
val items = todoItems()
if (items.size >= 2) {
return buildString {
appendLine(title.ifBlank { "Crkl" })
appendLine()
items.forEach { appendLine("• $it") }
}.trim()
}
return VoiceIntent.extractTextForTranslate(title, body, transcript)
.ifBlank { body.take(2_000).ifBlank { transcript } }
}
}
data class Result(
val ok: Boolean,
val message: String
)
suspend fun execute(intent: VoiceIntent.Parsed, session: Session?): Result {
return when (intent.kind) {
VoiceIntent.Kind.SHARE_LIST -> shareExtract(session)
VoiceIntent.Kind.EMAIL_LIST -> emailList(session)
VoiceIntent.Kind.OPEN_MAIL -> openMail()
VoiceIntent.Kind.OPEN_TODOS -> Result(true, openTodosUrl(settings.vikunjaUrl))
VoiceIntent.Kind.SHOW_CALENDAR -> showCalendar()
VoiceIntent.Kind.SUMMARIZE_EMAILS -> summarizeEmails(session)
VoiceIntent.Kind.ADD_TO_TODO -> addToTodo(session)
VoiceIntent.Kind.ADD_ITEMS -> addItems(session, intent.items)
VoiceIntent.Kind.SHOW_TODOS -> Result(true, openTodosUrl(settings.vikunjaUrl))
VoiceIntent.Kind.TRANSLATE -> translate(session, intent.targetLang)
VoiceIntent.Kind.COPY -> copyExtract(session)
VoiceIntent.Kind.EXPLAIN -> explain(session)
VoiceIntent.Kind.UNKNOWN -> Result(
false,
buildString {
appendLine("I heard: “${intent.raw}”")
appendLine()
appendLine("Try: copy · explain · share · translate · add to todo")
appendLine()
append(settings.statusSummary())
}
)
}
}
private fun copyExtract(session: Session?): Result {
if (session == null) {
return Result(false, "Nothing to copy — circle content first.")
}
val text = session.sharePayload()
if (text.isBlank()) {
return Result(false, "No text to copy.")
}
return if (copyText(text)) {
Result(true, "Copied ${text.length} characters to clipboard.")
} else {
Result(false, "Could not access clipboard.")
}
}
private suspend fun explain(session: Session?): Result {
if (session == null) {
return Result(false, "Nothing to explain — circle content first.")
}
val text = VoiceIntent.extractTextForTranslate(session.title, session.body, session.transcript)
val r = explainText(text)
return Result(true, "— explain —\n${r.body}")
}
private suspend fun translate(session: Session?, spokenTarget: String?): Result {
if (session == null) {
return Result(false, "Nothing to translate — circle a word or sentence first.")
}
val text = VoiceIntent.extractTextForTranslate(session.title, session.body, session.transcript)
val target = spokenTarget?.takeIf { it.isNotBlank() } ?: settings.translateTargetLang
val r = translateText(text, target)
return Result(r.ok, r.message)
}
private suspend fun emailList(session: Session?): Result {
if (session == null) {
return Result(false, "Nothing to email — circle content first.")
}
val payload = session.sharePayload()
val subject = session.title.ifBlank { "From Crkl" }
val to = settings.emailTo.ifBlank { "idobkin@gmail.com" }
if (settings.preferGogEmail && settings.gogReady()) {
val sent = sendViaGog(to, subject, payload)
if (sent.ok) return Result(true, sent.message)
Log.w(tagName, "gog send failed, mailto fallback: ${sent.message}")
val msg = composeEmail(subject, payload, to)
return Result(true, "gog failed (${sent.message}). $msg")
}
val msg = composeEmail(subject, payload, to)
return Result(true, msg)
}
private fun showCalendar(): Result {
val r = todayCalendar()
return Result(r.ok, r.message)
}
private fun openMail(): Result {
val to = settings.emailTo.ifBlank { "idobkin@gmail.com" }
return Result(true, composeEmail("", "", to))
}
private suspend fun addToTodo(session: Session?): Result {
if (session == null) {
return Result(false, "Nothing to save — circle something first.")
}
val items = session.todoItems()
if (items.isEmpty()) {
return Result(
false,
"Couldn’t find a task title in that circle. Try circling an email Subject " +
"or a bulleted list, then tap Add to Vikunja again."
)
}
Log.i(tagName, "addToTodo items=${items.size} first=${items.firstOrNull()}")
val desc = buildString {
appendLine("From Crkl")
if (session.transcript.isNotBlank()) {
appendLine()
append(session.transcript.take(500))
} else {
appendLine()
append(session.body.take(800))
}
}
return createTasks(items, desc).let { Result(it.ok, it.message) }
}
private suspend fun addItems(session: Session?, items: List<String>): Result {
if (items.isEmpty()) {
return Result(false, "Didn’t catch which items to add.")
}
if (session != null) {
session.workingList.clear()
session.workingList.addAll(
(session.todoItems() + items).distinctBy { it.lowercase() }
)
}
return createTasks(items, "Added via Crkl voice").let { Result(it.ok, it.message) }
}
private fun summarizeEmails(session: Session?): Result {
val body = session?.body.orEmpty() + "\n" + session?.transcript.orEmpty()
val lower = body.lowercase()
if ("yesterday" in lower || "inbox" in lower || "from:" in lower || "subject:" in lower) {
val lines = body.lineSequence()
.map { it.trim() }
.filter { it.isNotEmpty() }
.filter {
val l = it.lowercase()
l.startsWith("from:") || l.startsWith("subject:") ||
l.startsWith("•") || l.contains("@")
}
.take(12)
.toList()
return Result(
true,
buildString {
appendLine("From the circled inbox:")
if (lines.isEmpty()) appendLine(body.take(800)) else lines.forEach { appendLine(it) }
}
)
}
return Result(true, "Circle an inbox (Test fixtures → Mail), then ask again.")
}
private fun shareExtract(session: Session?): Result {
if (session == null) {
return Result(false, "Nothing to share — circle content first.")
}
val payload = session.sharePayload()
if (payload.isBlank()) {
return Result(false, "No text to share.")
}
return try {
val share = Intent(Intent.ACTION_SEND).apply {
type = "text/plain"
putExtra(Intent.EXTRA_SUBJECT, session.title.ifBlank { "From Crkl" })
putExtra(Intent.EXTRA_TEXT, payload)
addFlags(Intent.FLAG_ACTIVITY_NEW_TASK)
}
val chooser = Intent.createChooser(share, "Share with Crkl").apply {
addFlags(Intent.FLAG_ACTIVITY_NEW_TASK)
}
launchIntent(chooser)
Result(true, "Opened share sheet (${payload.length} chars).")
} catch (e: Exception) {
Log.e(tagName, "share failed", e)
Result(false, "Could not open share sheet: ${e.message}")
}
}
}
app/src/main/kotlin/com/example/crkl/agent/AssistEngine.kt
+191
View File
@@ -0,0 +1,191 @@
package com.example.crkl.agent
import android.util.Log
import com.example.crkl.model.LocalLlm
import com.example.crkl.model.ModelPaths
import com.example.crkl.vision.ContentCapture
import com.example.crkl.vision.RegionContentExtractor
/**
* Routes extracted screen text / media transcripts to an on-device LLM when available,
* otherwise falls back to [LocalAssistStub].
*/
class AssistEngine(
private val llm: LocalLlm?,
private val maxInputChars: Int = 2_500
) {
private val tagName = "AssistEngine"
enum class Mode { SCREEN, MEDIA }
data class AssistResponse(
val title: String,
val meta: String,
val body: String
)
suspend fun warmUp() {
val media = llm as? com.example.crkl.model.MediaPipeLocalLlm ?: return
runCatching { media.ensureLoaded() }
.onFailure { Log.w(tagName, "LLM warm-up failed", it) }
}
suspend fun respond(
extraction: RegionContentExtractor.ExtractionResult,
mode: Mode = Mode.SCREEN
): AssistResponse {
if (extraction.isEmpty || !ContentCapture.hasUsefulText(extraction.text)) {
val stub = LocalAssistStub.respond(extraction)
return AssistResponse(stub.title, "", stub.body)
}
val llmEngine = llm
if (llmEngine == null || !ensureReady(llmEngine)) {
val stub = LocalAssistStub.respond(extraction, mode = mode)
try {
Log.i(tagName, "stub path (no LLM). ${ModelPaths.missingModelHint().lineSequence().firstOrNull()}")
} catch (_: RuntimeException) {
// Android Log is not mocked in JVM unit tests.
}
return AssistResponse(
title = stub.title,
meta = "",
body = stub.body
)
}
val prompt = buildPrompt(extraction.text, mode)
return try {
val generated = llmEngine.generate(prompt)
if (generated.isBlank()) {
fallbackWithNote(extraction, "Model returned empty output", mode)
} else {
AssistResponse(
title = if (mode == Mode.MEDIA) "On-device media summary" else "On-device summary",
meta = packageMeta(extraction) +
" · ${llmEngine.displayName} · ${extraction.text.length} chars in",
body = buildString {
appendLine(generated.trim())
appendLine()
appendLine("— source —")
appendLine()
append(extraction.text.trim().take(1_200))
if (extraction.text.length > 1_200) append("…")
}
)
}
} catch (e: Exception) {
Log.e(tagName, "LLM generate failed", e)
fallbackWithNote(extraction, "LLM error: ${e.message ?: e.javaClass.simpleName}", mode)
}
}
private suspend fun ensureReady(llmEngine: LocalLlm): Boolean {
if (llmEngine.isReady) return true
val media = llmEngine as? com.example.crkl.model.MediaPipeLocalLlm ?: return false
return media.ensureLoaded()
}
private fun fallbackWithNote(
extraction: RegionContentExtractor.ExtractionResult,
note: String,
mode: Mode = Mode.SCREEN
): AssistResponse {
val stub = LocalAssistStub.respond(extraction, mode = mode)
return AssistResponse(
title = stub.title,
meta = stub.meta + " · stub",
body = buildString {
appendLine(note)
appendLine()
append(stub.body)
}
)
}
private fun buildPrompt(rawText: String, mode: Mode): String {
val clipped = rawText.trim().take(maxInputChars)
return when (mode) {
Mode.MEDIA -> """
You are Crkl, an on-device assistant. The user circled audio or video.
You are given a transcript of what was spoken. Summarize it and extract actions.
Rules:
- 2 to 5 short sentences
- If it is a list (groceries, todos), bullet the items
- If it is a how-to, list the steps in order
- Be concrete; do not invent facts not present in the transcript
- No markdown headings
Media transcript payload:
---
$clipped
---
""".trimIndent()
Mode.SCREEN -> """
You are Crkl, an on-device assistant. Summarize the on-screen text below for the user.
Rules:
- 2 to 4 short sentences
- Be concrete; do not invent facts not present in the text
- If the text is a message or email, say who/what it is about
- No markdown headings
On-screen text:
---
$clipped
---
""".trimIndent()
}
}
/**
* You are Crkl… explain prompt helper for AssistEngine.
*/
suspend fun explain(text: String): AssistResponse {
val clipped = text.trim().take(maxInputChars)
if (clipped.isBlank()) {
return AssistResponse("Explain", "empty", "Nothing to explain — circle text first.")
}
val llmEngine = llm
if (llmEngine == null || !ensureReady(llmEngine)) {
val stub = LocalAssistStub.explain(clipped)
return AssistResponse(stub.title, stub.meta + " · stub", stub.body)
}
val prompt = """
You are Crkl, an on-device assistant. Explain the text below in plain language.
Rules:
- 2 to 4 short sentences
- Assume a smart adult who is busy — no jargon unless necessary
- Do not invent facts not in the text
- If it is an email or message, say who it is from and what they want
- No markdown headings
Text:
---
$clipped
---
""".trimIndent()
return try {
val generated = llmEngine.generate(prompt)
if (generated.isBlank()) {
val stub = LocalAssistStub.explain(clipped)
AssistResponse(stub.title, stub.meta + " · empty model", stub.body)
} else {
AssistResponse(
title = "Explain",
meta = "${llmEngine.displayName} · ${clipped.length} chars",
body = generated.trim()
)
}
} catch (e: Exception) {
Log.e(tagName, "explain failed", e)
val stub = LocalAssistStub.explain(clipped)
AssistResponse(stub.title, stub.meta + " · error", stub.body)
}
}
private fun packageMeta(extraction: RegionContentExtractor.ExtractionResult): String =
extraction.packageName?.substringAfterLast('.') ?: "unknown app"
}
app/src/main/kotlin/com/example/crkl/agent/LocalAssistStub.kt
+204
View File
@@ -0,0 +1,204 @@
package com.example.crkl.agent
import com.example.crkl.vision.ContentCapture
import com.example.crkl.vision.RegionContentExtractor
/**
* Temporary on-device "assist" path until a real local LLM is wired.
* Formats extracted screen text / media transcripts into a readable panel response.
* No network. No model weights.
*/
object LocalAssistStub {
data class AssistResponse(
val title: String,
val meta: String,
val body: String
)
fun respond(
extraction: RegionContentExtractor.ExtractionResult,
mode: AssistEngine.Mode = AssistEngine.Mode.SCREEN
): AssistResponse {
if (extraction.isEmpty || !ContentCapture.hasUsefulText(extraction.text)) {
return AssistResponse(
title = "No text found",
meta = "",
body = buildString {
appendLine("Nothing useful to read in that circle.")
appendLine()
appendLine("Try circling a message, article, email, or label.")
appendLine()
appendLine("Photos and icons: use Google Lens — Circle is for on-screen text.")
}.trim()
)
}
val preview = extraction.text.trim()
if (mode == AssistEngine.Mode.MEDIA || preview.contains("Transcript:", ignoreCase = true)) {
return mediaResponse(extraction, preview)
}
val summary = heuristicSummary(preview)
return AssistResponse(
title = "Circled text",
meta = "",
body = buildString {
if (summary.isNotBlank()) {
appendLine(summary)
appendLine()
appendLine("—")
appendLine()
}
append(preview)
}
)
}
fun explain(text: String): AssistResponse {
val cleaned = text.trim()
if (cleaned.isBlank()) {
return AssistResponse(
title = "Explain",
meta = "empty",
body = "Nothing to explain — circle a word or sentence first."
)
}
val lower = cleaned.lowercase()
val kind = when {
"subject:" in lower || "from:" in lower -> "This looks like an email or message."
cleaned.lineSequence().count { it.trim().startsWith("•") || it.trim().startsWith("-") } >= 2 ->
"This looks like a list."
cleaned.length < 40 -> "This is a short phrase or label."
else -> "Here is a plain-language reading."
}
val first = cleaned.lineSequence().firstOrNull { line ->
val l = line.trim().lowercase()
l.isNotEmpty() && !l.startsWith("—") && !l.startsWith("local stub")
}?.trim().orEmpty()
val who = Regex("""(?i)from:\s*(.+)""").find(cleaned)?.groupValues?.getOrNull(1)?.trim()
val subject = Regex("""(?i)subject:\s*(.+)""").find(cleaned)?.groupValues?.getOrNull(1)?.trim()
val body = buildString {
appendLine(kind)
appendLine()
when {
who != null || subject != null -> {
if (who != null) appendLine("From: $who")
if (subject != null) appendLine("About: $subject")
appendLine()
appendLine(
"In short: someone is writing about “${subject ?: first.take(80)}”. " +
"Open the full text below the divider if you need details."
)
}
cleaned.length <= 120 -> {
appendLine("It means roughly: “$cleaned”")
appendLine()
appendLine("No hidden steps — take it at face value.")
}
else -> {
val head = first.take(140).ifBlank { cleaned.take(140) }
appendLine("Main idea: $head")
appendLine()
appendLine(
"Skim the original for names, dates, and numbers — those are usually the parts that matter."
)
}
}
}.trim()
return AssistResponse(
title = "Explain",
meta = "plain language · ${cleaned.length} chars",
body = body
)
}
private fun mediaResponse(
extraction: RegionContentExtractor.ExtractionResult,
preview: String
): AssistResponse {
val transcript = preview.substringAfter("Transcript:", preview).trim()
val kind = when {
preview.contains("Kind: video", ignoreCase = true) -> "video"
preview.contains("Kind: audio", ignoreCase = true) -> "audio"
else -> "media"
}
val summary = mediaSummary(transcript, kind)
return AssistResponse(
title = if (kind == "video") "Video summary" else "Audio summary",
meta = packageMeta(extraction) + " · ${transcript.length} transcript chars",
body = buildString {
appendLine(summary)
appendLine()
appendLine("— transcript —")
appendLine()
append(transcript)
}
)
}
private fun mediaSummary(transcript: String, kind: String): String {
val lower = transcript.lowercase()
return when {
"grocery" in lower || "milk" in lower -> {
val items = extractListish(transcript)
buildString {
appendLine("Local stub listened to the $kind and pulled a shopping list:")
items.forEach { appendLine("• $it") }
if (items.isEmpty()) {
appendLine("• (see transcript)")
}
}.trim()
}
"settings" in lower && ("privacy" in lower || "save" in lower) -> {
buildString {
appendLine("Local stub listened to the $kind walkthrough:")
appendLine("1. Open Settings")
appendLine("2. Enable Privacy Mode")
appendLine("3. Tap Save")
appendLine("Takes under a minute.")
}.trim()
}
else -> {
val head = transcript.lineSequence().firstOrNull { it.isNotBlank() }?.trim().orEmpty()
"Local stub (no LLM yet) summary of $kind: " +
head.take(160).ifBlank { "see transcript below." }
}
}
}
private fun extractListish(transcript: String): List<String> {
val known = listOf(
"milk", "eggs", "sourdough bread", "olive oil", "avocados",
"oat milk", "coffee beans", "bread", "oil"
)
val lower = transcript.lowercase()
return known.filter { it in lower }.distinct()
.map { it.replaceFirstChar { c -> c.uppercase() } }
}
private fun packageMeta(extraction: RegionContentExtractor.ExtractionResult): String =
extraction.packageName?.substringAfterLast('.') ?: "unknown app"
private fun heuristicSummary(text: String): String {
val firstLine = text.lineSequence().firstOrNull { it.isNotBlank() }?.trim().orEmpty()
val sentence = firstLine
.split(Regex("(?<=[.!?])\\s+"))
.firstOrNull()
?.trim()
.orEmpty()
val head = when {
sentence.isNotEmpty() && sentence.length <= 160 -> sentence
firstLine.length <= 160 -> firstLine
else -> firstLine.take(157) + "…"
}
return head
}
}
app/src/main/kotlin/com/example/crkl/agent/VoiceIntent.kt
+265
View File
@@ -0,0 +1,265 @@
package com.example.crkl.agent
import com.example.crkl.integrations.OnDeviceTranslator
/**
* Parses short spoken follow-ups after a circle/media result.
*/
object VoiceIntent {
enum class Kind {
SHARE_LIST,
EMAIL_LIST,
OPEN_MAIL,
OPEN_TODOS,
SHOW_CALENDAR,
SUMMARIZE_EMAILS,
ADD_TO_TODO,
ADD_ITEMS,
SHOW_TODOS,
TRANSLATE,
COPY,
EXPLAIN,
UNKNOWN
}
data class Parsed(
val kind: Kind,
val items: List<String> = emptyList(),
/** BCP-47 target for TRANSLATE, else null → use settings default. */
val targetLang: String? = null,
val raw: String
)
fun parse(spoken: String): Parsed {
val raw = spoken.trim()
val lower = raw.lowercase()
.replace(Regex("[?.!,]"), " ")
.replace(Regex("\\s+"), " ")
.trim()
if (lower.isBlank()) return Parsed(Kind.UNKNOWN, raw = raw)
if (looksLikeTranslate(lower)) {
return Parsed(
Kind.TRANSLATE,
targetLang = OnDeviceTranslator.resolveSpokenTarget(lower),
raw = raw
)
}
if (looksLikeCopy(lower)) return Parsed(Kind.COPY, raw = raw)
if (looksLikeExplain(lower)) return Parsed(Kind.EXPLAIN, raw = raw)
if (looksLikeCalendar(lower)) return Parsed(Kind.SHOW_CALENDAR, raw = raw)
if (looksLikeShowTodos(lower)) return Parsed(Kind.SHOW_TODOS, raw = raw)
if (looksLikeOpenTodos(lower)) return Parsed(Kind.OPEN_TODOS, raw = raw)
if (looksLikeOpenMail(lower)) return Parsed(Kind.OPEN_MAIL, raw = raw)
if (looksLikeSummarizeEmails(lower)) return Parsed(Kind.SUMMARIZE_EMAILS, raw = raw)
if (looksLikeEmailList(lower)) return Parsed(Kind.EMAIL_LIST, raw = raw)
if (looksLikeShare(lower)) return Parsed(Kind.SHARE_LIST, raw = raw)
if (looksLikeAddToTodo(lower)) return Parsed(Kind.ADD_TO_TODO, raw = raw)
extractAddItems(lower)?.let { items ->
if (items.isNotEmpty()) return Parsed(Kind.ADD_ITEMS, items = items, raw = raw)
}
return Parsed(Kind.UNKNOWN, raw = raw)
}
private fun looksLikeTranslate(lower: String): Boolean {
return "translate" in lower || "translation" in lower ||
"перевед" in lower || "переклади" in lower
}
private fun looksLikeCopy(lower: String): Boolean {
return (
"copy" in lower || "clipboard" in lower || "скопир" in lower
) && "email" !in lower && "share" !in lower
}
private fun looksLikeExplain(lower: String): Boolean {
return "explain" in lower || "eli5" in lower || "simplify" in lower ||
"what does this mean" in lower || "plain english" in lower ||
"объясн" in lower || "упрост" in lower
}
private fun looksLikeCalendar(lower: String): Boolean {
return "calendar" in lower || "schedule" in lower ||
("what" in lower && ("today" in lower || "agenda" in lower)) ||
"what's on my cal" in lower || "whats on my cal" in lower
}
private fun looksLikeEmailList(lower: String): Boolean {
if ("email" !in lower && "gmail" !in lower && "mail me" !in lower) return false
return "list" in lower || "summary" in lower || "this" in lower ||
"that" in lower || "send" in lower || "me" in lower
}
private fun looksLikeOpenMail(lower: String): Boolean {
return (
"open" in lower || "go to" in lower || "launch" in lower
) && (
"gmail" in lower || "inbox" in lower || "mail" in lower
) && "list" !in lower && "todo" !in lower
}
private fun looksLikeOpenTodos(lower: String): Boolean {
return (
"open" in lower || "go to" in lower
) && (
"todo" in lower || "vikunja" in lower || "tasks" in lower
) && "add" !in lower
}
private fun looksLikeSummarizeEmails(lower: String): Boolean {
val emailCue = "email" in lower || "emails" in lower || "inbox" in lower || "mail" in lower
val timeCue = "yesterday" in lower || "last" in lower || "recent" in lower ||
"3" in lower || "three" in lower || "summar" in lower || "show" in lower || "read" in lower
return emailCue && timeCue && "todo" !in lower
}
private fun looksLikeShare(lower: String): Boolean {
val shareWords = listOf("send", "share", "text me", "message me", "forward")
val target = listOf("list", "summary", "transcript", "that", "it", "this", "result", "extract")
return shareWords.any { it in lower } && (
target.any { it in lower } || "me" in lower || lower.startsWith("send") || lower == "share"
) && "email" !in lower && "gmail" !in lower && "mail" !in lower && "copy" !in lower
}
private fun looksLikeAddToTodo(lower: String): Boolean {
return (
"todo" in lower || "to do" in lower || "to-do" in lower ||
"vikunja" in lower || "task list" in lower || "my tasks" in lower
) && (
"add" in lower || "save" in lower || "put" in lower ||
"update" in lower || "into" in lower
)
}
private fun looksLikeShowTodos(lower: String): Boolean {
return (
"show" in lower || "list my" in lower || "what's on" in lower ||
"what is on" in lower || "read my" in lower
) && (
"todo" in lower || "to do" in lower || "tasks" in lower || "vikunja" in lower
)
}
private fun extractAddItems(lower: String): List<String>? {
val patterns = listOf(
Regex("""(?:add|also get|also pick up|include)\s+(.+)$"""),
Regex("""(?:update(?:\s+the)?\s+list.*?(?:add|with))\s+(.+)$"""),
Regex("""(?:put)\s+(.+?)\s+(?:on|in|to)\s+(?:the\s+)?list""")
)
for (p in patterns) {
val m = p.find(lower) ?: continue
val chunk = m.groupValues.getOrNull(1)?.trim().orEmpty()
if (chunk.isBlank()) continue
if ("todo" in chunk || "to do" in chunk) continue
return splitItems(chunk)
}
return null
}
fun splitItems(chunk: String): List<String> {
return chunk
.replace(Regex("""\band\b"""), ",")
.split(",")
.map { it.trim() }
.map { it.removePrefix("some ").removePrefix("a ").removePrefix("an ").trim() }
.filter { it.length in 2..48 }
.map { it.replaceFirstChar { c -> if (c.isLowerCase()) c.titlecase() else c.toString() } }
.distinct()
}
fun extractListFromBody(body: String): List<String> {
val items = mutableListOf<String>()
for (raw in body.lineSequence()) {
val line = raw.trim()
if (line.isEmpty()) continue
val marked = line.startsWith("•") || line.startsWith("-") || line.startsWith("*") ||
Regex("""^\d+[.)]\s+""").containsMatchIn(line)
if (!marked) continue
val cleaned = line
.removePrefix("•").removePrefix("-").removePrefix("*").trim()
.replace(Regex("""^\d+[.)]\s*"""), "")
.trim()
if (cleaned.length !in 2..60) continue
val lower = cleaned.lowercase()
if (lower.startsWith("local stub") ||
lower.startsWith("transcript") ||
lower.startsWith("media:") ||
lower.startsWith("kind:") ||
lower.startsWith("—") ||
lower.startsWith("stt:") ||
lower.contains("no on-device model") ||
lower.startsWith("from crkl") ||
lower.startsWith("commands:") ||
lower.startsWith("speak a command")
) {
continue
}
items.add(cleaned)
}
return items.distinct()
}
/**
* Tasks for Vikunja: prefer bullets; otherwise a single task from Subject / title / first line.
*/
fun extractTasksForTodo(title: String, body: String, transcript: String = ""): List<String> {
val bullets = extractListFromBody(body + "\n" + transcript)
if (bullets.isNotEmpty()) return bullets.take(20)
val combined = (body + "\n" + transcript).lineSequence().map { it.trim() }.filter { it.isNotEmpty() }
val subject = combined.firstOrNull { it.startsWith("Subject:", ignoreCase = true) }
?.substringAfter(':')?.trim()
if (!subject.isNullOrBlank() && subject.length in 2..120) {
return listOf(subject)
}
val skipPrefixes = listOf(
"from:", "to:", "local stub", "—", "commands:", "speak", "you said:",
"stt:", "kind:", "media:", "transcript"
)
val candidate = combined.firstOrNull { line ->
val l = line.lowercase()
skipPrefixes.none { l.startsWith(it) } &&
!l.startsWith("extracted") &&
line.length in 8..120
}
if (!candidate.isNullOrBlank()) return listOf(candidate.take(120))
val t = title.trim()
if (t.isNotBlank() && !t.equals("Crkl", ignoreCase = true) &&
!t.equals("Extracted text", ignoreCase = true)
) {
return listOf(t.take(120))
}
return emptyList()
}
/** Text to translate: prefer short circled snippet, strip Crkl chrome. */
fun extractTextForTranslate(title: String, body: String, transcript: String = ""): String {
val raw = (body.ifBlank { transcript }).ifBlank { title }
val lines = raw.lineSequence()
.map { it.trim() }
.filter { it.isNotEmpty() }
.filter { line ->
val l = line.lowercase()
!l.startsWith("—") &&
!l.startsWith("commands:") &&
!l.startsWith("speak") &&
!l.startsWith("you said:") &&
!l.startsWith("translated") &&
!l.startsWith("local stub") &&
!l.startsWith("stt:") &&
l != "extracted text"
}
.toList()
if (lines.isEmpty()) return raw.trim().take(1_500)
// Prefer the densest paragraph chunk after a summary separator
val afterSep = lines.dropWhile { it != "—" }.drop(1)
val chunk = if (afterSep.size >= 1) afterSep else lines
return chunk.joinToString("\n").take(1_500)
}
}
app/src/main/kotlin/com/example/crkl/capture/CaptureForegroundService.kt
+109
View File
@@ -0,0 +1,109 @@
package com.example.crkl.capture
import android.app.Notification
import android.app.NotificationChannel
import android.app.NotificationManager
import android.app.PendingIntent
import android.app.Service
import android.content.Context
import android.content.Intent
import android.content.pm.ServiceInfo
import android.media.projection.MediaProjectionManager
import android.os.Build
import android.os.IBinder
import android.util.Log
import androidx.core.app.NotificationCompat
import com.example.crkl.MainActivity
/**
* Foreground service required to keep [android.media.projection.MediaProjection] alive.
*/
class CaptureForegroundService : Service() {
private val tagName = "CaptureFgs"
override fun onBind(intent: Intent?): IBinder? = null
override fun onStartCommand(intent: Intent?, flags: Int, startId: Int): Int {
when (intent?.action) {
ACTION_START -> {
ensureNotification()
val code = intent.getIntExtra(EXTRA_RESULT_CODE, 0)
@Suppress("DEPRECATION")
val data = if (Build.VERSION.SDK_INT >= 33) {
intent.getParcelableExtra(EXTRA_RESULT_DATA, Intent::class.java)
} else {
intent.getParcelableExtra(EXTRA_RESULT_DATA)
}
if (data == null) {
Log.e(tagName, "Missing projection result data")
stopSelf()
return START_NOT_STICKY
}
val mpm = getSystemService(Context.MEDIA_PROJECTION_SERVICE) as MediaProjectionManager
val projection = mpm.getMediaProjection(code, data)
MediaProjectionHolder.set(projection)
Log.i(tagName, "MediaProjection ready")
}
ACTION_STOP -> {
MediaProjectionHolder.clear()
stopForeground(STOP_FOREGROUND_REMOVE)
stopSelf()
}
else -> ensureNotification()
}
return START_STICKY
}
private fun ensureNotification() {
createChannel()
val pending = PendingIntent.getActivity(
this,
0,
Intent(this, MainActivity::class.java),
PendingIntent.FLAG_IMMUTABLE
)
val notification: Notification = NotificationCompat.Builder(this, CHANNEL_ID)
.setContentTitle("Crkl capture active")
.setContentText("Screen/audio capture permission is available for listen-in.")
.setSmallIcon(android.R.drawable.ic_btn_speak_now)
.setContentIntent(pending)
.setOngoing(true)
.build()
if (Build.VERSION.SDK_INT >= 29) {
startForeground(
NOTIF_ID,
notification,
ServiceInfo.FOREGROUND_SERVICE_TYPE_MEDIA_PROJECTION
)
} else {
startForeground(NOTIF_ID, notification)
}
}
private fun createChannel() {
val nm = getSystemService(NotificationManager::class.java)
nm.createNotificationChannel(
NotificationChannel(
CHANNEL_ID,
"Crkl capture",
NotificationManager.IMPORTANCE_LOW
)
)
}
companion object {
const val ACTION_START = "com.example.crkl.capture.START"
const val ACTION_STOP = "com.example.crkl.capture.STOP"
const val EXTRA_RESULT_CODE = "result_code"
const val EXTRA_RESULT_DATA = "result_data"
private const val CHANNEL_ID = "crkl_capture"
private const val NOTIF_ID = 42
fun stop(context: Context) {
MediaProjectionHolder.clear()
context.stopService(Intent(context, CaptureForegroundService::class.java))
}
}
}
app/src/main/kotlin/com/example/crkl/capture/MediaProjectionHolder.kt
+24
View File
@@ -0,0 +1,24 @@
package com.example.crkl.capture
import android.media.projection.MediaProjection
import java.util.concurrent.atomic.AtomicReference
/**
* Process-wide holder for an active [MediaProjection] after user consent.
*/
object MediaProjectionHolder {
private val projectionRef = AtomicReference<MediaProjection?>(null)
fun set(projection: MediaProjection?) {
projectionRef.getAndSet(projection)?.stop()
projectionRef.set(projection)
}
fun get(): MediaProjection? = projectionRef.get()
fun clear() {
set(null)
}
fun isReady(): Boolean = get() != null
}
app/src/main/kotlin/com/example/crkl/capture/PlaybackCaptureRecorder.kt
+152
View File
@@ -0,0 +1,152 @@
package com.example.crkl.capture
import android.media.AudioAttributes
import android.media.AudioFormat
import android.media.AudioPlaybackCaptureConfiguration
import android.media.AudioRecord
import android.media.projection.MediaProjection
import android.os.Build
import android.util.Log
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext
import java.io.File
import java.io.FileOutputStream
import java.io.RandomAccessFile
import java.nio.ByteBuffer
import java.nio.ByteOrder
/**
* Captures other apps' media audio via [AudioPlaybackCaptureConfiguration] (API 29+).
* Writes a mono 16-bit PCM WAV suitable for later on-device file STT (Vosk/Whisper).
*/
class PlaybackCaptureRecorder {
private val tagName = "PlaybackCapture"
data class CaptureResult(
val wavFile: File,
val durationMs: Long,
val bytesCaptured: Int
)
suspend fun capture(
projection: MediaProjection,
outDir: File,
durationMs: Long = 8_000L,
sampleRate: Int = 16_000
): CaptureResult? = withContext(Dispatchers.IO) {
if (Build.VERSION.SDK_INT < Build.VERSION_CODES.Q) {
Log.w(tagName, "AudioPlaybackCapture requires API 29+")
return@withContext null
}
val config = AudioPlaybackCaptureConfiguration.Builder(projection)
.addMatchingUsage(AudioAttributes.USAGE_MEDIA)
.addMatchingUsage(AudioAttributes.USAGE_GAME)
.addMatchingUsage(AudioAttributes.USAGE_UNKNOWN)
.build()
val format = AudioFormat.Builder()
.setEncoding(AudioFormat.ENCODING_PCM_16BIT)
.setSampleRate(sampleRate)
.setChannelMask(AudioFormat.CHANNEL_IN_MONO)
.build()
val minBuf = AudioRecord.getMinBufferSize(
sampleRate,
AudioFormat.CHANNEL_IN_MONO,
AudioFormat.ENCODING_PCM_16BIT
).coerceAtLeast(sampleRate / 2)
val recorder = try {
AudioRecord.Builder()
.setAudioFormat(format)
.setBufferSizeInBytes(minBuf * 2)
.setAudioPlaybackCaptureConfig(config)
.build()
} catch (e: SecurityException) {
Log.e(tagName, "AudioPlaybackCapture not permitted", e)
return@withContext null
} catch (e: Exception) {
Log.e(tagName, "AudioRecord build failed", e)
return@withContext null
}
if (recorder.state != AudioRecord.STATE_INITIALIZED) {
Log.e(tagName, "AudioRecord not initialized")
recorder.release()
return@withContext null
}
outDir.mkdirs()
val out = File(outDir, "playback_capture_${System.currentTimeMillis()}.wav")
val pcm = ArrayList<ByteArray>()
var total = 0
val buf = ByteArray(minBuf)
try {
recorder.startRecording()
val deadline = System.currentTimeMillis() + durationMs
while (System.currentTimeMillis() < deadline) {
val n = recorder.read(buf, 0, buf.size)
if (n > 0) {
pcm.add(buf.copyOf(n))
total += n
} else if (n < 0) {
Log.w(tagName, "AudioRecord read error $n")
break
}
}
} finally {
try {
recorder.stop()
} catch (_: Exception) {
}
recorder.release()
}
if (total == 0) {
Log.w(tagName, "No playback audio captured (app may block capture or silent)")
return@withContext null
}
writeWav(out, pcm, sampleRate, total)
val actualMs = (total / 2.0 / sampleRate * 1000.0).toLong()
Log.i(tagName, "Captured ${total}B (~${actualMs}ms) → ${out.name}")
CaptureResult(out, actualMs, total)
}
private fun writeWav(file: File, chunks: List<ByteArray>, sampleRate: Int, dataBytes: Int) {
FileOutputStream(file).use { fos ->
// Placeholder header; rewrite sizes after.
fos.write(ByteArray(44))
for (c in chunks) fos.write(c)
}
RandomAccessFile(file, "rw").use { raf ->
raf.seek(0)
raf.writeBytes("RIFF")
writeIntLE(raf, 36 + dataBytes)
raf.writeBytes("WAVE")
raf.writeBytes("fmt ")
writeIntLE(raf, 16)
writeShortLE(raf, 1) // PCM
writeShortLE(raf, 1) // mono
writeIntLE(raf, sampleRate)
writeIntLE(raf, sampleRate * 2)
writeShortLE(raf, 2)
writeShortLE(raf, 16)
raf.writeBytes("data")
writeIntLE(raf, dataBytes)
}
}
private fun writeIntLE(raf: RandomAccessFile, value: Int) {
val buf = ByteBuffer.allocate(4).order(ByteOrder.LITTLE_ENDIAN).putInt(value).array()
raf.write(buf)
}
private fun writeShortLE(raf: RandomAccessFile, value: Int) {
val buf = ByteBuffer.allocate(2).order(ByteOrder.LITTLE_ENDIAN).putShort(value.toShort()).array()
raf.write(buf)
}
}
app/src/main/kotlin/com/example/crkl/capture/ProjectionPermissionActivity.kt
+54
View File
@@ -0,0 +1,54 @@
package com.example.crkl.capture
import androidx.core.content.ContextCompat
import android.app.Activity
import android.content.Context
import android.content.Intent
import android.media.projection.MediaProjectionManager
import android.os.Bundle
import android.util.Log
import androidx.activity.ComponentActivity
import androidx.activity.result.contract.ActivityResultContracts
/**
* One-shot consent UI for MediaProjection (required for AudioPlaybackCapture
* of other apps' audio and optional screen buffers).
*/
class ProjectionPermissionActivity : ComponentActivity() {
private val tagName = "ProjectionPermission"
private val launcher = registerForActivityResult(
ActivityResultContracts.StartActivityForResult()
) { result ->
if (result.resultCode == Activity.RESULT_OK && result.data != null) {
val data = result.data!!
// Start FGS first (Android 14+), then obtain projection inside the service.
val svc = Intent(this, CaptureForegroundService::class.java).apply {
action = CaptureForegroundService.ACTION_START
putExtra(CaptureForegroundService.EXTRA_RESULT_CODE, result.resultCode)
putExtra(CaptureForegroundService.EXTRA_RESULT_DATA, data)
}
ContextCompat.startForegroundService(this, svc)
Log.i(tagName, "MediaProjection consent granted; starting capture service")
setResult(Activity.RESULT_OK)
} else {
Log.w(tagName, "MediaProjection consent denied")
setResult(Activity.RESULT_CANCELED)
}
finish()
}
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
val mpm = getSystemService(Context.MEDIA_PROJECTION_SERVICE) as MediaProjectionManager
launcher.launch(mpm.createScreenCaptureIntent())
}
companion object {
fun intent(context: Context): Intent =
Intent(context, ProjectionPermissionActivity::class.java).apply {
addFlags(Intent.FLAG_ACTIVITY_NEW_TASK)
}
}
}
app/src/main/kotlin/com/example/crkl/fixtures/FixtureCatalog.kt
+187
View File
@@ -0,0 +1,187 @@
package com.example.crkl.fixtures
/**
* Catalog of on-device test targets for Crkl circle → extract demos.
* Used by [TestFixturesActivity] and unit tests.
*/
object FixtureCatalog {
enum class ContentKind {
EMAIL,
ARTICLE,
IMAGE,
VIDEO,
AUDIO
}
enum class Expectation {
/** Accessibility text should be extracted today. */
EXTRACTABLE_NOW,
/** Needs OCR / MediaProjection — expect empty or weak extract. */
NEEDS_OCR,
/** Real playable media → listen/transcribe → summarize. */
MEDIA_PLAY_AND_SUMMARIZE,
/** Mic STT path (Speak on panel). */
NEEDS_STT
}
data class Fixture(
val id: String,
val kind: ContentKind,
val title: String,
val body: String,
val contentDescription: String? = null,
val expectation: Expectation,
val howToTest: String,
/** Asset under app assets/, e.g. media/grocery_memo.wav */
val mediaAssetPath: String? = null,
val transcriptAssetPath: String? = null
)
val all: List<Fixture> = listOf(
Fixture(
id = "email_meeting",
kind = ContentKind.EMAIL,
title = "Email · Meeting reschedule",
body = """
From: alex@example.com
To: you@levkin.ca
Subject: Q2 planning moved
Hi — the Q2 planning sync is moved to Thursday at 3pm.
Please bring the notes from last sprint and the budget draft.
Thanks,
Alex
""".trimIndent(),
expectation = Expectation.EXTRACTABLE_NOW,
howToTest = "DEMO 3: Circle Subject “Q2 planning moved” → Vikunja. Or whole email → Explain."
),
Fixture(
id = "article_paragraph",
kind = ContentKind.ARTICLE,
title = "Article · Short paragraph",
body = """
Crkl is a privacy-first Android assistant. Users circle on-screen
content; the app reads accessibility text and summarizes it with
an on-device model when available.
""".trimIndent(),
expectation = Expectation.EXTRACTABLE_NOW,
howToTest = "DEMO 2: Circle the paragraph → Copy, then Explain."
),
Fixture(
id = "image_with_description",
kind = ContentKind.IMAGE,
title = "Image · Has contentDescription",
body = "Photo placeholder (described for a11y)",
contentDescription = "Sunset over a lake with a wooden dock and pine trees",
expectation = Expectation.EXTRACTABLE_NOW,
howToTest = "Circle the teal image block. Extract should include the description text."
),
Fixture(
id = "image_pixels_only",
kind = ContentKind.IMAGE,
title = "Image · Pixels only (OCR target)",
body = "",
contentDescription = null,
expectation = Expectation.NEEDS_OCR,
howToTest = "Circle the dark block that says OCR-HELLO-42. Panel meta should mention ocr."
),
Fixture(
id = "mail_inbox_yesterday",
kind = ContentKind.EMAIL,
title = "Mail · Last 3 emails (yesterday)",
body = SampleInbox.text(),
contentDescription = "Sample inbox with three emails from yesterday",
expectation = Expectation.EXTRACTABLE_NOW,
howToTest = "Circle inbox text → summarize. After a list: Email list opens your mail app (mailto)."
),
Fixture(
id = "translate_phrase",
kind = ContentKind.ARTICLE,
title = "Translate · English phrase",
body = "The meeting is postponed until Thursday afternoon.",
expectation = Expectation.EXTRACTABLE_NOW,
howToTest = "DEMO 1: Circle the sentence → Translate (default → Russian)."
),
Fixture(
id = "grocery_list",
kind = ContentKind.ARTICLE,
title = "List · Groceries (Vikunja / Share)",
body = """
Shopping
• Milk
• Eggs
• Olive oil
• Sourdough bread
""".trimIndent(),
expectation = Expectation.EXTRACTABLE_NOW,
howToTest = "DEMO 4: Circle the bullets → Share or Vikunja (creates 4 tasks)."
),
Fixture(
id = "video_walkthrough",
kind = ContentKind.VIDEO,
title = "Video · Product walkthrough",
body = "▶ Demo clip: product walkthrough (real audio)",
contentDescription = "Crkl playable video: product walkthrough",
expectation = Expectation.MEDIA_PLAY_AND_SUMMARIZE,
howToTest = "Tap Play (optional), then C → circle the video. Crkl plays/listens, transcribes, summarizes steps.",
mediaAssetPath = "media/product_walkthrough.mp4",
transcriptAssetPath = "media/product_walkthrough.transcript.txt"
),
Fixture(
id = "audio_grocery",
kind = ContentKind.AUDIO,
title = "Audio · Grocery voice memo",
body = "♫ Voice memo: grocery list (real audio)",
contentDescription = "Crkl playable audio: grocery voice memo",
expectation = Expectation.MEDIA_PLAY_AND_SUMMARIZE,
howToTest = "Tap Play (optional), then C → circle the audio. Crkl listens, transcribes, extracts the shopping list.",
mediaAssetPath = "media/grocery_memo.wav",
transcriptAssetPath = "media/grocery_memo.transcript.txt"
)
)
fun byId(id: String): Fixture =
all.first { it.id == id }
fun extractableNow(): List<Fixture> =
all.filter { it.expectation == Expectation.EXTRACTABLE_NOW }
fun mediaFixtures(): List<Fixture> =
all.filter { it.mediaAssetPath != null }
/** Ordered VIP demo path for ship / recording (see docs/DEMO.md). */
fun demoScript(): List<DemoStep> = listOf(
DemoStep(
order = 1,
fixtureId = "translate_phrase",
chip = "Translate",
say = "Circle the English sentence → tap Translate"
),
DemoStep(
order = 2,
fixtureId = "article_paragraph",
chip = "Copy / Explain",
say = "Circle the paragraph → Copy, then Explain"
),
DemoStep(
order = 3,
fixtureId = "email_meeting",
chip = "Vikunja",
say = "Circle the Subject line (or whole email) → Add to Vikunja"
),
DemoStep(
order = 4,
fixtureId = "grocery_list",
chip = "Share / Vikunja",
say = "Circle the bulleted list → Share or Vikunja"
)
)
data class DemoStep(
val order: Int,
val fixtureId: String,
val chip: String,
val say: String
)
}
app/src/main/kotlin/com/example/crkl/fixtures/PlayableMediaFixture.kt
+224
View File
@@ -0,0 +1,224 @@
package com.example.crkl.fixtures
import android.media.MediaPlayer
import android.view.ViewGroup
import android.widget.FrameLayout
import android.widget.VideoView
import androidx.compose.foundation.background
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.Spacer
import androidx.compose.foundation.layout.fillMaxWidth
import androidx.compose.foundation.layout.height
import androidx.compose.foundation.layout.padding
import androidx.compose.foundation.shape.RoundedCornerShape
import androidx.compose.material3.Button
import androidx.compose.material3.ButtonDefaults
import androidx.compose.material3.MaterialTheme
import androidx.compose.material3.Text
import androidx.compose.runtime.Composable
import androidx.compose.runtime.DisposableEffect
import androidx.compose.runtime.getValue
import androidx.compose.runtime.mutableStateOf
import androidx.compose.runtime.remember
import androidx.compose.runtime.setValue
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.layout.boundsInWindow
import androidx.compose.ui.layout.onGloballyPositioned
import androidx.compose.ui.platform.LocalContext
import androidx.compose.ui.semantics.contentDescription
import androidx.compose.ui.semantics.semantics
import androidx.compose.ui.text.font.FontWeight
import androidx.compose.ui.unit.dp
import androidx.compose.ui.viewinterop.AndroidView
import com.example.crkl.media.ActiveMediaRegistry
import java.io.File
import java.io.FileOutputStream
/**
* Real playable video/audio fixture that registers with [ActiveMediaRegistry]
* so circling it triggers listen → transcript → summarize.
*/
@Composable
fun PlayableMediaFixture(
fixture: FixtureCatalog.Fixture,
accent: Color
) {
val assetPath = fixture.mediaAssetPath ?: return
val transcriptPath = fixture.transcriptAssetPath ?: return
val context = LocalContext.current
val kind = when (fixture.kind) {
FixtureCatalog.ContentKind.VIDEO -> ActiveMediaRegistry.Kind.VIDEO
else -> ActiveMediaRegistry.Kind.AUDIO
}
DisposableEffect(fixture.id) {
ActiveMediaRegistry.register(
ActiveMediaRegistry.Slot(
id = fixture.id,
kind = kind,
assetPath = assetPath,
transcriptAssetPath = transcriptPath,
title = fixture.title
)
)
onDispose { ActiveMediaRegistry.unregister(fixture.id) }
}
var playing by remember { mutableStateOf(false) }
var status by remember { mutableStateOf("Ready · tap Play or circle with C") }
var mediaPlayer by remember { mutableStateOf<MediaPlayer?>(null) }
var videoView by remember { mutableStateOf<VideoView?>(null) }
fun stopPlayback() {
try {
mediaPlayer?.stop()
} catch (_: Exception) {
}
mediaPlayer?.release()
mediaPlayer = null
try {
videoView?.stopPlayback()
} catch (_: Exception) {
}
playing = false
}
fun play() {
stopPlayback()
val cache = copyAsset(context, assetPath)
if (kind == ActiveMediaRegistry.Kind.VIDEO) {
val vv = videoView ?: return
vv.setOnCompletionListener {
playing = false
status = "Finished"
}
vv.setVideoPath(cache.absolutePath)
vv.start()
playing = true
status = "Playing video…"
} else {
val mp = MediaPlayer()
mediaPlayer = mp
mp.setDataSource(cache.absolutePath)
mp.setOnCompletionListener {
playing = false
status = "Finished"
mp.release()
mediaPlayer = null
}
mp.prepare()
mp.start()
playing = true
status = "Playing audio…"
}
}
DisposableEffect(Unit) {
onDispose { stopPlayback() }
}
Column(
modifier = Modifier
.fillMaxWidth()
.background(accent, RoundedCornerShape(8.dp))
.onGloballyPositioned { coords ->
val b = coords.boundsInWindow()
ActiveMediaRegistry.updateBounds(
fixture.id,
ActiveMediaRegistry.Bounds(b.left, b.top, b.right, b.bottom)
)
}
.semantics {
contentDescription = fixture.contentDescription
?: "Crkl playable ${kind.name.lowercase()}: ${fixture.title}"
}
.padding(12.dp)
) {
Text(
text = fixture.body,
color = Color.White,
style = MaterialTheme.typography.bodyLarge,
fontWeight = FontWeight.Medium
)
Spacer(modifier = Modifier.height(8.dp))
if (kind == ActiveMediaRegistry.Kind.VIDEO) {
AndroidView(
modifier = Modifier
.fillMaxWidth()
.height(180.dp),
factory = { ctx ->
FrameLayout(ctx).apply {
layoutParams = ViewGroup.LayoutParams(
ViewGroup.LayoutParams.MATCH_PARENT,
ViewGroup.LayoutParams.MATCH_PARENT
)
val vv = VideoView(ctx)
videoView = vv
addView(
vv,
FrameLayout.LayoutParams(
FrameLayout.LayoutParams.MATCH_PARENT,
FrameLayout.LayoutParams.MATCH_PARENT
)
)
setBackgroundColor(0xFF2A1540.toInt())
}
}
)
Spacer(modifier = Modifier.height(8.dp))
} else {
Box(
modifier = Modifier
.fillMaxWidth()
.height(72.dp)
.background(Color.Black.copy(alpha = 0.25f), RoundedCornerShape(8.dp)),
contentAlignment = Alignment.Center
) {
Text(
text = if (playing) "♪ Playing…" else "♪ Audio ready",
color = Color.White,
style = MaterialTheme.typography.titleMedium
)
}
Spacer(modifier = Modifier.height(8.dp))
}
Button(
onClick = {
if (playing) {
stopPlayback()
status = "Stopped"
} else {
play()
}
},
colors = ButtonDefaults.buttonColors(
containerColor = Color.White.copy(alpha = 0.2f),
contentColor = Color.White
)
) {
Text(if (playing) "Stop" else "Play")
}
Text(
text = status,
color = Color.White.copy(alpha = 0.85f),
style = MaterialTheme.typography.bodySmall,
modifier = Modifier.padding(top = 4.dp)
)
}
}
private fun copyAsset(context: android.content.Context, assetPath: String): File {
val name = assetPath.substringAfterLast('/')
val out = File(context.cacheDir, "fixture_$name")
if (!out.exists() || out.length() == 0L) {
context.assets.open(assetPath).use { input ->
FileOutputStream(out).use { output -> input.copyTo(output) }
}
}
return out
}
app/src/main/kotlin/com/example/crkl/fixtures/SampleInbox.kt
+24
View File
@@ -0,0 +1,24 @@
package com.example.crkl.fixtures
/**
* Static sample inbox text for circle/extract demos in [TestFixturesActivity].
* Not a mail product — real send uses the device mail app (mailto).
*/
object SampleInbox {
fun text(): String = """
Mail · Inbox · Yesterday
From: jordan@example.com
Subject: School pickup tomorrow
Can you do pickup at 3:30?
From: billing@utility.example
Subject: Your July statement is ready
Account ending 4421 — amount due $86.40
From: sam@example.com
Subject: Dinner Friday?
Thinking Thai place at 7
""".trimIndent()
}
app/src/main/kotlin/com/example/crkl/fixtures/TestFixturesActivity.kt
+247
View File
@@ -0,0 +1,247 @@
package com.example.crkl.fixtures
import android.os.Bundle
import androidx.activity.ComponentActivity
import androidx.activity.compose.setContent
import androidx.compose.foundation.background
import androidx.compose.foundation.border
import androidx.compose.foundation.layout.Arrangement
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.Spacer
import androidx.compose.foundation.layout.fillMaxSize
import androidx.compose.foundation.layout.fillMaxWidth
import androidx.compose.foundation.layout.height
import androidx.compose.foundation.layout.padding
import androidx.compose.foundation.layout.size
import androidx.compose.foundation.rememberScrollState
import androidx.compose.foundation.shape.RoundedCornerShape
import androidx.compose.foundation.verticalScroll
import androidx.compose.material3.Card
import androidx.compose.material3.CardDefaults
import androidx.compose.material3.MaterialTheme
import androidx.compose.material3.Surface
import androidx.compose.material3.Text
import androidx.compose.runtime.Composable
import androidx.compose.ui.Alignment
import androidx.compose.ui.Modifier
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.semantics.contentDescription
import androidx.compose.ui.semantics.semantics
import androidx.compose.ui.semantics.testTag
import androidx.compose.ui.text.font.FontFamily
import androidx.compose.ui.text.font.FontWeight
import androidx.compose.ui.unit.dp
import androidx.compose.ui.viewinterop.AndroidView
import android.view.Gravity
import android.view.View
import android.widget.TextView
import com.example.crkl.ui.theme.CrklTheme
/**
* Scrollable playground with email / article / image / video / audio samples.
* Enable Crkl accessibility, tap C, circle each card to validate extract behavior.
*/
class TestFixturesActivity : ComponentActivity() {
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContent {
CrklTheme {
Surface(modifier = Modifier.fillMaxSize()) {
TestFixturesScreen()
}
}
}
}
}
@Composable
fun TestFixturesScreen() {
Column(
modifier = Modifier
.fillMaxSize()
.verticalScroll(rememberScrollState())
.padding(16.dp),
verticalArrangement = Arrangement.spacedBy(16.dp)
) {
Text(
text = "Test fixtures · VIP demo",
style = MaterialTheme.typography.headlineMedium,
fontWeight = FontWeight.Bold
)
Text(
text = "Ship demo (≈45s)",
style = MaterialTheme.typography.titleMedium,
fontWeight = FontWeight.Bold
)
Text(
text = FixtureCatalog.demoScript().joinToString("\n") { step ->
"${step.order}. ${step.say} [${step.chip}]"
},
style = MaterialTheme.typography.bodyMedium,
color = MaterialTheme.colorScheme.onSurfaceVariant
)
Text(
text = "How to circle",
style = MaterialTheme.typography.titleMedium,
fontWeight = FontWeight.Bold
)
Text(
text = "1. Tap the floating C (or Quick Settings → Circle). Screen dims.\n" +
"2. Draw a closed loop around text. Sheet slides up with chips.\n" +
"3. Cancel (top-right) leaves draw mode so you can scroll.\n\n" +
"Chips: Translate · Copy · Explain · Share · Vikunja\n" +
"Full script: docs/DEMO.md",
style = MaterialTheme.typography.bodyMedium,
color = MaterialTheme.colorScheme.onSurfaceVariant
)
FixtureCatalog.all.forEach { fixture ->
FixtureCard(fixture)
}
Spacer(modifier = Modifier.height(48.dp))
}
}
@Composable
private fun FixtureCard(fixture: FixtureCatalog.Fixture) {
val badge = when (fixture.expectation) {
FixtureCatalog.Expectation.EXTRACTABLE_NOW -> "SHOULD EXTRACT NOW"
FixtureCatalog.Expectation.NEEDS_OCR -> "NEEDS OCR"
FixtureCatalog.Expectation.MEDIA_PLAY_AND_SUMMARIZE -> "PLAY → LISTEN → SUMMARIZE"
FixtureCatalog.Expectation.NEEDS_STT -> "NEEDS MIC STT"
}
val badgeColor = when (fixture.expectation) {
FixtureCatalog.Expectation.EXTRACTABLE_NOW -> Color(0xFF1B7F4E)
FixtureCatalog.Expectation.NEEDS_OCR -> Color(0xFF9A5B00)
FixtureCatalog.Expectation.MEDIA_PLAY_AND_SUMMARIZE -> Color(0xFF1F4E79)
FixtureCatalog.Expectation.NEEDS_STT -> Color(0xFF8A3A00)
}
Card(
modifier = Modifier
.fillMaxWidth()
.semantics {
testTag = "fixture_${fixture.id}"
fixture.contentDescription?.let { contentDescription = it }
},
colors = CardDefaults.cardColors(
containerColor = MaterialTheme.colorScheme.surfaceVariant
),
shape = RoundedCornerShape(12.dp)
) {
Column(modifier = Modifier.padding(14.dp)) {
Text(
text = badge,
color = badgeColor,
style = MaterialTheme.typography.labelMedium,
fontWeight = FontWeight.Bold
)
Spacer(modifier = Modifier.height(6.dp))
Text(
text = fixture.title,
style = MaterialTheme.typography.titleMedium,
fontWeight = FontWeight.SemiBold
)
Spacer(modifier = Modifier.height(4.dp))
Text(
text = fixture.howToTest,
style = MaterialTheme.typography.bodySmall,
color = MaterialTheme.colorScheme.onSurfaceVariant
)
Spacer(modifier = Modifier.height(12.dp))
when (fixture.kind) {
FixtureCatalog.ContentKind.EMAIL,
FixtureCatalog.ContentKind.ARTICLE -> {
Text(
text = fixture.body,
style = MaterialTheme.typography.bodyMedium,
fontFamily = if (fixture.kind == FixtureCatalog.ContentKind.EMAIL) {
FontFamily.Monospace
} else {
FontFamily.Default
},
modifier = Modifier
.fillMaxWidth()
.background(
MaterialTheme.colorScheme.surface,
RoundedCornerShape(8.dp)
)
.padding(12.dp)
.semantics { testTag = "fixture_body_${fixture.id}" }
)
}
FixtureCatalog.ContentKind.IMAGE -> {
val desc = fixture.contentDescription
if (desc == null && fixture.id == "image_pixels_only") {
// Painted glyphs with a11y disabled — forces OCR path.
OcrOnlySample()
} else {
Box(
modifier = Modifier
.fillMaxWidth()
.height(140.dp)
.background(
if (desc == null) Color(0xFF3A3A3A) else Color(0xFF1F6F8B),
RoundedCornerShape(8.dp)
)
.border(2.dp, Color.White.copy(alpha = 0.25f), RoundedCornerShape(8.dp))
.then(
if (desc != null) {
Modifier.semantics { contentDescription = desc }
} else {
Modifier
}
),
contentAlignment = Alignment.Center
) {
Text(
text = if (desc != null) "🖼 described image" else "🖼 pixels only",
color = Color.White,
style = MaterialTheme.typography.titleMedium
)
}
}
if (fixture.body.isNotBlank()) {
Spacer(modifier = Modifier.height(8.dp))
Text(text = fixture.body, style = MaterialTheme.typography.bodySmall)
}
}
FixtureCatalog.ContentKind.VIDEO -> {
PlayableMediaFixture(
fixture = fixture,
accent = Color(0xFF5B2C6F)
)
}
FixtureCatalog.ContentKind.AUDIO -> {
PlayableMediaFixture(
fixture = fixture,
accent = Color(0xFF1A5276)
)
}
}
}
}
}
@Composable
private fun OcrOnlySample() {
AndroidView(
modifier = Modifier
.fillMaxWidth()
.height(140.dp),
factory = { ctx ->
TextView(ctx).apply {
text = "OCR-HELLO-42"
textSize = 28f
setTextColor(android.graphics.Color.WHITE)
gravity = Gravity.CENTER
setBackgroundColor(android.graphics.Color.parseColor("#3A3A3A"))
// Hide from accessibility so capture must use OCR.
importantForAccessibility = View.IMPORTANT_FOR_ACCESSIBILITY_NO
}
}
)
}
app/src/main/kotlin/com/example/crkl/integrations/DeviceCalendar.kt
+108
View File
@@ -0,0 +1,108 @@
package com.example.crkl.integrations
import android.Manifest
import android.content.Context
import android.content.pm.PackageManager
import android.provider.CalendarContract
import androidx.core.content.ContextCompat
import java.text.SimpleDateFormat
import java.util.Calendar
import java.util.Date
import java.util.Locale
import java.util.TimeZone
/**
* Reads today's events from the device calendar (CalendarContract).
* Works with whatever accounts the user has synced (Google, Exchange, etc.) —
* no host bridge, no Google OAuth client in the APK.
*/
object DeviceCalendar {
data class Event(val whenLabel: String, val title: String)
data class Result(val ok: Boolean, val message: String, val events: List<Event> = emptyList())
fun hasPermission(context: Context): Boolean =
ContextCompat.checkSelfPermission(context, Manifest.permission.READ_CALENDAR) ==
PackageManager.PERMISSION_GRANTED
fun today(context: Context, limit: Int = 20): Result {
if (!hasPermission(context)) {
return Result(
false,
"Calendar permission needed. Open Integrations → Grant calendar access, " +
"or enable it in Android Settings → Apps → Crkl → Permissions."
)
}
val zone = TimeZone.getDefault()
val startCal = Calendar.getInstance(zone).apply {
set(Calendar.HOUR_OF_DAY, 0)
set(Calendar.MINUTE, 0)
set(Calendar.SECOND, 0)
set(Calendar.MILLISECOND, 0)
}
val endCal = Calendar.getInstance(zone).apply {
timeInMillis = startCal.timeInMillis
add(Calendar.DAY_OF_YEAR, 1)
}
val startMs = startCal.timeInMillis
val endMs = endCal.timeInMillis
val projection = arrayOf(
CalendarContract.Events.TITLE,
CalendarContract.Events.DTSTART,
CalendarContract.Events.ALL_DAY
)
val selection =
"(${CalendarContract.Events.DTSTART} >= ? AND ${CalendarContract.Events.DTSTART} < ?) " +
"OR (${CalendarContract.Events.ALL_DAY}=1 AND " +
"${CalendarContract.Events.DTSTART} >= ? AND ${CalendarContract.Events.DTSTART} < ?)"
val args = arrayOf(
startMs.toString(),
endMs.toString(),
startMs.toString(),
endMs.toString()
)
val timeFmt = SimpleDateFormat("HH:mm", Locale.getDefault())
return try {
val events = mutableListOf<Event>()
context.contentResolver.query(
CalendarContract.Events.CONTENT_URI,
projection,
selection,
args,
"${CalendarContract.Events.DTSTART} ASC"
)?.use { cursor ->
val titleIdx = cursor.getColumnIndex(CalendarContract.Events.TITLE)
val startIdx = cursor.getColumnIndex(CalendarContract.Events.DTSTART)
val allDayIdx = cursor.getColumnIndex(CalendarContract.Events.ALL_DAY)
while (cursor.moveToNext() && events.size < limit) {
val title = cursor.getString(titleIdx)?.ifBlank { "(no title)" } ?: "(no title)"
val start = cursor.getLong(startIdx)
val allDay = cursor.getInt(allDayIdx) == 1
val whenLabel = if (allDay) {
"all day"
} else {
timeFmt.format(Date(start))
}
events.add(Event(whenLabel, title))
}
}
if (events.isEmpty()) {
Result(true, "No calendar events for today.", emptyList())
} else {
val body = buildString {
appendLine("Today’s calendar:")
events.forEach { appendLine("• ${it.whenLabel} — ${it.title}") }
}.trim()
Result(true, body, events)
}
} catch (e: SecurityException) {
Result(false, "Calendar permission denied: ${e.message}")
} catch (e: Exception) {
Result(false, "Calendar read failed: ${e.message}")
}
}
}
app/src/main/kotlin/com/example/crkl/integrations/GogBridgeClient.kt
+90
View File
@@ -0,0 +1,90 @@
package com.example.crkl.integrations
import android.util.Log
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext
import org.json.JSONArray
import org.json.JSONObject
import java.io.OutputStreamWriter
import java.net.HttpURLConnection
import java.net.URL
/**
* Optional host bridge wrapping `gog` for Gmail send (lab/emulator testing).
* Production path remains mailto; enable Prefer gog in Integrations when bridge is running.
*/
class GogBridgeClient(
private val settings: IntegrationSettings
) {
data class BridgeResult(val ok: Boolean, val message: String)
suspend fun health(): BridgeResult = withContext(Dispatchers.IO) {
if (!settings.gogReady()) {
return@withContext BridgeResult(false, "gog bridge URL not set")
}
get("/health")
}
suspend fun sendEmail(to: String, subject: String, body: String): BridgeResult =
withContext(Dispatchers.IO) {
if (!settings.gogReady()) {
return@withContext BridgeResult(false, "gog bridge not configured")
}
val payload = JSONObject()
.put("to", to)
.put("subject", subject)
.put("body", body)
post("/v1/gmail/send", payload)
}
private fun get(path: String): BridgeResult {
val url = URL(settings.gogBridgeUrl + path)
val conn = (url.openConnection() as HttpURLConnection).apply {
requestMethod = "GET"
connectTimeout = 8_000
readTimeout = 30_000
}
return read(conn)
}
private fun post(path: String, json: JSONObject): BridgeResult {
val url = URL(settings.gogBridgeUrl + path)
val conn = (url.openConnection() as HttpURLConnection).apply {
requestMethod = "POST"
setRequestProperty("Content-Type", "application/json")
doOutput = true
connectTimeout = 8_000
readTimeout = 60_000
}
return try {
OutputStreamWriter(conn.outputStream).use { it.write(json.toString()) }
read(conn)
} catch (e: Exception) {
Log.e("GogBridge", "post $path failed", e)
BridgeResult(false, e.message ?: e.javaClass.simpleName)
}
}
private fun read(conn: HttpURLConnection): BridgeResult {
return try {
val code = conn.responseCode
val text = (if (code in 200..299) conn.inputStream else conn.errorStream)
?.bufferedReader()?.readText().orEmpty()
if (code in 200..299) {
val msg = runCatching {
JSONObject(text).optString("message", text.take(800))
}.getOrDefault(text.take(800))
BridgeResult(true, msg)
} else {
val detail = runCatching {
JSONObject(text).optString("message", text.take(400))
}.getOrDefault(text.take(400))
BridgeResult(false, "HTTP $code $detail")
}
} catch (e: Exception) {
BridgeResult(false, e.message ?: e.javaClass.simpleName)
} finally {
conn.disconnect()
}
}
}
app/src/main/kotlin/com/example/crkl/integrations/IntegrationSettings.kt
+138
View File
@@ -0,0 +1,138 @@
package com.example.crkl.integrations
import android.content.Context
import android.content.SharedPreferences
import android.graphics.Color
/**
* In-app integration + appearance settings. Secrets stay in prefs — never in git.
*/
class IntegrationSettings(
private val prefs: SharedPreferences
) {
constructor(context: Context) : this(
context.applicationContext.getSharedPreferences(PREFS, Context.MODE_PRIVATE)
)
var vikunjaUrl: String
get() = prefs.getString(KEY_VIKUNJA_URL, DEFAULT_VIKUNJA_URL).orEmpty()
set(value) = prefs.edit().putString(KEY_VIKUNJA_URL, value.trim().trimEnd('/')).apply()
var vikunjaToken: String
get() = prefs.getString(KEY_VIKUNJA_TOKEN, "").orEmpty()
set(value) = prefs.edit().putString(KEY_VIKUNJA_TOKEN, value.trim()).apply()
var vikunjaProjectId: Int
get() = prefs.getInt(KEY_VIKUNJA_PROJECT, DEFAULT_VIKUNJA_PROJECT)
set(value) = prefs.edit().putInt(KEY_VIKUNJA_PROJECT, value).apply()
var emailTo: String
get() = prefs.getString(KEY_EMAIL_TO, "").orEmpty()
set(value) = prefs.edit().putString(KEY_EMAIL_TO, value.trim()).apply()
/**
* Optional host bridge for Gmail via `gog` (emulator → http://10.0.2.2:8765).
* Empty = mailto only. For lab testing, not required in production.
*/
var gogBridgeUrl: String
get() = prefs.getString(KEY_GOG_BRIDGE, "").orEmpty()
set(value) = prefs.edit().putString(KEY_GOG_BRIDGE, value.trim().trimEnd('/')).apply()
var preferGogEmail: Boolean
get() = prefs.getBoolean(KEY_PREFER_GOG, false)
set(value) = prefs.edit().putBoolean(KEY_PREFER_GOG, value).apply()
// --- Circle / overlay appearance ---
/** Preset: yellow | cyan | magenta | white | lime */
var circleColorKey: String
get() = prefs.getString(KEY_CIRCLE_COLOR, DEFAULT_CIRCLE_COLOR).orEmpty()
set(value) = prefs.edit().putString(KEY_CIRCLE_COLOR, value).apply()
var circleNeon: Boolean
get() = prefs.getBoolean(KEY_CIRCLE_NEON, true)
set(value) = prefs.edit().putBoolean(KEY_CIRCLE_NEON, value).apply()
/** thin | medium | thick */
var circleStrokeKey: String
get() = prefs.getString(KEY_CIRCLE_STROKE, DEFAULT_CIRCLE_STROKE).orEmpty()
set(value) = prefs.edit().putString(KEY_CIRCLE_STROKE, value).apply()
/** BCP-47 target for Translate (en, ru, he, …). */
var translateTargetLang: String
get() = prefs.getString(KEY_TRANSLATE_TARGET, DEFAULT_TRANSLATE_TARGET).orEmpty()
set(value) = prefs.edit().putString(KEY_TRANSLATE_TARGET, value.trim().lowercase()).apply()
/** First-run onboarding finished (or skipped). */
var onboardingComplete: Boolean
get() = prefs.getBoolean(KEY_ONBOARDING_DONE, false)
set(value) = prefs.edit().putBoolean(KEY_ONBOARDING_DONE, value).apply()
/** Show capture source / OCR counts on the result panel (for debugging). */
var showDebugMeta: Boolean
get() = prefs.getBoolean(KEY_SHOW_DEBUG_META, false)
set(value) = prefs.edit().putBoolean(KEY_SHOW_DEBUG_META, value).apply()
fun vikunjaReady(): Boolean = vikunjaToken.isNotBlank() && vikunjaUrl.isNotBlank()
fun gogReady(): Boolean = gogBridgeUrl.isNotBlank()
fun circleColorArgb(): Int = when (circleColorKey.lowercase()) {
"cyan" -> Color.parseColor("#00E5FF")
"magenta" -> Color.parseColor("#FF2BD6")
"white" -> Color.parseColor("#F5F7FA")
"lime" -> Color.parseColor("#B8FF3D")
else -> Color.parseColor("#FFE566") // yellow
}
fun circleStrokeDp(): Float = when (circleStrokeKey.lowercase()) {
"thin" -> 2.5f
"thick" -> 7f
else -> 4.5f
}
fun statusSummary(): String = buildString {
append("Vikunja: ")
append(if (vikunjaReady()) "ready → $vikunjaUrl (project $vikunjaProjectId)" else "needs token")
append('\n')
append("Email: ")
when {
preferGogEmail && gogReady() -> append("gog → ${emailTo.ifBlank { "(set Email To)" }}")
emailTo.isNotBlank() -> append("mailto → $emailTo")
else -> append("mailto (To not set)")
}
append('\n')
append("Calendar: device CalendarContract")
append('\n')
append("Circle: ${circleColorKey}")
if (circleNeon) append(" · neon")
append(" · ${circleStrokeKey}")
append('\n')
append("Translate → ${translateTargetLang}")
}
companion object {
private const val PREFS = "crkl_integrations"
private const val KEY_VIKUNJA_URL = "vikunja_url"
private const val KEY_VIKUNJA_TOKEN = "vikunja_token"
private const val KEY_VIKUNJA_PROJECT = "vikunja_project"
private const val KEY_EMAIL_TO = "email_to"
private const val KEY_GOG_BRIDGE = "gog_bridge"
private const val KEY_PREFER_GOG = "prefer_gog_email"
private const val KEY_CIRCLE_COLOR = "circle_color"
private const val KEY_CIRCLE_NEON = "circle_neon"
private const val KEY_CIRCLE_STROKE = "circle_stroke"
private const val KEY_TRANSLATE_TARGET = "translate_target"
private const val KEY_ONBOARDING_DONE = "onboarding_done"
private const val KEY_SHOW_DEBUG_META = "show_debug_meta"
const val DEFAULT_VIKUNJA_URL = "https://todo.levkin.ca"
const val DEFAULT_VIKUNJA_PROJECT = 13
const val DEFAULT_GOG_BRIDGE_EMU = "http://10.0.2.2:8765"
const val DEFAULT_CIRCLE_COLOR = "cyan"
const val DEFAULT_CIRCLE_STROKE = "medium"
const val DEFAULT_TRANSLATE_TARGET = "ru"
val CIRCLE_COLORS = listOf("yellow", "cyan", "magenta", "white", "lime")
val CIRCLE_STROKES = listOf("thin", "medium", "thick")
}
}
app/src/main/kotlin/com/example/crkl/integrations/OnDeviceTranslator.kt
+157
View File
@@ -0,0 +1,157 @@
package com.example.crkl.integrations
import android.util.Log
import com.google.mlkit.common.model.DownloadConditions
import com.google.mlkit.nl.languageid.LanguageIdentification
import com.google.mlkit.nl.translate.TranslateLanguage
import com.google.mlkit.nl.translate.Translation
import com.google.mlkit.nl.translate.TranslatorOptions
import kotlinx.coroutines.suspendCancellableCoroutine
import kotlin.coroutines.resume
/**
* On-device ML Kit translation. First use downloads a language model (needs network once);
* afterward works offline.
*/
class OnDeviceTranslator {
data class Result(val ok: Boolean, val message: String)
suspend fun translate(
text: String,
targetLangTag: String,
allowCellularDownload: Boolean = true
): Result {
val cleaned = text.trim().take(1_500)
if (cleaned.isBlank()) {
return Result(false, "Nothing to translate — circle a word or sentence first.")
}
val target = TranslateLanguage.fromLanguageTag(targetLangTag)
?: return Result(false, "Unsupported target language: $targetLangTag")
val sourceTag = identifyLanguage(cleaned) ?: TranslateLanguage.ENGLISH
val source = TranslateLanguage.fromLanguageTag(sourceTag) ?: TranslateLanguage.ENGLISH
if (source == target) {
return Result(
true,
"Already looks like ${displayName(target)}:\n\n$cleaned"
)
}
val options = TranslatorOptions.Builder()
.setSourceLanguage(source)
.setTargetLanguage(target)
.build()
val translator = Translation.getClient(options)
return try {
val conditions = DownloadConditions.Builder().let {
if (!allowCellularDownload) it.requireWifi() else it
}.build()
val downloaded = awaitTask(translator.downloadModelIfNeeded(conditions))
if (!downloaded) {
return Result(
false,
"Couldn’t download translation model ($sourceTag→$targetLangTag). " +
"Connect to the internet once, then try again."
)
}
val out = awaitTaskValue(translator.translate(cleaned))
?: return Result(false, "Translation failed.")
Result(
true,
buildString {
appendLine("Translated (${displayName(source)} → ${displayName(target)}):")
appendLine()
appendLine(out)
appendLine()
appendLine("— original —")
append(cleaned.take(400))
}
)
} catch (e: Exception) {
Log.e(TAG, "translate failed", e)
Result(false, "Translate error: ${e.message}")
} finally {
translator.close()
}
}
private suspend fun identifyLanguage(text: String): String? {
val client = LanguageIdentification.getClient()
return try {
val code = awaitTaskValue(client.identifyLanguage(text))
if (code == null || code == "und") null else code
} catch (e: Exception) {
Log.w(TAG, "language id failed", e)
null
} finally {
client.close()
}
}
private fun displayName(tag: String): String = when (tag) {
TranslateLanguage.ENGLISH -> "English"
TranslateLanguage.RUSSIAN -> "Russian"
TranslateLanguage.HEBREW -> "Hebrew"
TranslateLanguage.SPANISH -> "Spanish"
TranslateLanguage.FRENCH -> "French"
TranslateLanguage.GERMAN -> "German"
TranslateLanguage.UKRAINIAN -> "Ukrainian"
TranslateLanguage.CHINESE -> "Chinese"
else -> tag
}
private suspend fun <T> awaitTaskValue(task: com.google.android.gms.tasks.Task<T>): T? =
suspendCancellableCoroutine { cont ->
task
.addOnSuccessListener { cont.resume(it) }
.addOnFailureListener { e ->
Log.e(TAG, "task failed", e)
cont.resume(null)
}
}
private suspend fun awaitTask(task: com.google.android.gms.tasks.Task<Void>): Boolean =
suspendCancellableCoroutine { cont ->
task
.addOnSuccessListener { cont.resume(true) }
.addOnFailureListener { e ->
Log.e(TAG, "download failed", e)
cont.resume(false)
}
}
companion object {
private const val TAG = "OnDeviceTranslator"
/** BCP-47 tags used in settings / voice. */
val TARGET_OPTIONS = listOf(
"en" to "English",
"ru" to "Russian",
"he" to "Hebrew",
"uk" to "Ukrainian",
"es" to "Spanish",
"fr" to "French",
"de" to "German"
)
fun resolveSpokenTarget(lower: String): String? {
return when {
"russian" in lower || "русский" in lower || "русск" in lower -> "ru"
"hebrew" in lower || "иврит" in lower -> "he"
"ukrainian" in lower || "україн" in lower || "украин" in lower -> "uk"
"spanish" in lower || "español" in lower -> "es"
"french" in lower || "français" in lower || "francais" in lower -> "fr"
"german" in lower || "deutsch" in lower -> "de"
"english" in lower || "английск" in lower -> "en"
else -> null
}
}
}
}
app/src/main/kotlin/com/example/crkl/integrations/RealEmailActions.kt
+54
View File
@@ -0,0 +1,54 @@
package com.example.crkl.integrations
import android.content.ActivityNotFoundException
import android.content.Context
import android.content.Intent
import android.net.Uri
import android.util.Log
/**
* Opens a real mail client (FairEmail / Gmail / K-9 / Betterbird companion) via mailto:.
*/
object RealEmailActions {
private const val TAG = "RealEmail"
fun compose(
context: Context,
subject: String,
body: String,
to: String = ""
): String {
val mailto = Intent(Intent.ACTION_SENDTO).apply {
data = Uri.parse("mailto:" + Uri.encode(to))
putExtra(Intent.EXTRA_SUBJECT, subject)
putExtra(Intent.EXTRA_TEXT, body)
addFlags(Intent.FLAG_ACTIVITY_NEW_TASK)
}
return try {
val chooser = Intent.createChooser(mailto, "Send email").apply {
addFlags(Intent.FLAG_ACTIVITY_NEW_TASK)
}
context.startActivity(chooser)
"Opened mail app compose" + if (to.isNotBlank()) " → $to" else ""
} catch (e: ActivityNotFoundException) {
Log.e(TAG, "no mail app", e)
"No mail app installed. Install FairEmail/K-9 and add test@levkine.ca (Mailcow IMAP)."
} catch (e: Exception) {
Log.e(TAG, "compose failed", e)
"Could not open mail app: ${e.message}"
}
}
fun openWebTodos(context: Context, url: String): String {
return try {
context.startActivity(
Intent(Intent.ACTION_VIEW, Uri.parse(url)).apply {
addFlags(Intent.FLAG_ACTIVITY_NEW_TASK)
}
)
"Opening $url"
} catch (e: Exception) {
"Could not open browser: ${e.message}"
}
}
}
app/src/main/kotlin/com/example/crkl/integrations/VikunjaClient.kt
+89
View File
@@ -0,0 +1,89 @@
package com.example.crkl.integrations
import android.util.Log
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.withContext
import org.json.JSONObject
import java.io.OutputStreamWriter
import java.net.HttpURLConnection
import java.net.URL
/**
* Creates tasks on todo.levkin.ca (Vikunja).
* API: PUT /api/v1/projects/{id}/tasks Authorization: Bearer …
*/
class VikunjaClient(
private val settings: IntegrationSettings
) {
private val tagName = "VikunjaClient"
data class CreateResult(val ok: Boolean, val message: String, val taskId: Long? = null)
suspend fun createTasks(titles: List<String>, description: String = ""): CreateResult =
withContext(Dispatchers.IO) {
if (!settings.vikunjaReady()) {
return@withContext CreateResult(
false,
"Vikunja not configured. Open Circle → Integrations and paste an API token " +
"(from ansible .env / Infisical)."
)
}
if (titles.isEmpty()) {
return@withContext CreateResult(false, "No items to add.")
}
val created = mutableListOf<String>()
val errors = mutableListOf<String>()
for (title in titles.take(20)) {
val one = createOne(title.trim(), description)
if (one.ok) created.add(title) else errors.add("${title}: ${one.message}")
}
when {
created.isEmpty() -> CreateResult(false, errors.joinToString("\n").ifBlank { "Create failed" })
errors.isEmpty() -> CreateResult(
true,
"Added ${created.size} task(s) to Vikunja (project ${settings.vikunjaProjectId}):\n" +
created.joinToString("\n") { "• $it" } +
"\n\nOpen: ${settings.vikunjaUrl}"
)
else -> CreateResult(
true,
"Added ${created.size}; some failed:\n${errors.joinToString("\n")}"
)
}
}
private fun createOne(title: String, description: String): CreateResult {
val url = URL("${settings.vikunjaUrl}/api/v1/projects/${settings.vikunjaProjectId}/tasks")
val conn = (url.openConnection() as HttpURLConnection).apply {
requestMethod = "PUT"
setRequestProperty("Authorization", "Bearer ${settings.vikunjaToken}")
setRequestProperty("Content-Type", "application/json")
doOutput = true
connectTimeout = 12_000
readTimeout = 12_000
}
return try {
val body = JSONObject()
.put("title", title.take(200))
.put("description", description.take(2_000))
.put("priority", 2)
OutputStreamWriter(conn.outputStream).use { it.write(body.toString()) }
val code = conn.responseCode
val text = (if (code in 200..299) conn.inputStream else conn.errorStream)
?.bufferedReader()?.readText().orEmpty()
if (code in 200..299) {
val id = runCatching { JSONObject(text).optLong("id") }.getOrNull()
CreateResult(true, "ok", id)
} else {
Log.w(tagName, "create failed code=$code body=$text")
CreateResult(false, "HTTP $code ${text.take(200)}")
}
} catch (e: Exception) {
Log.e(tagName, "create failed", e)
CreateResult(false, e.message ?: e.javaClass.simpleName)
} finally {
conn.disconnect()
}
}
}
app/src/main/kotlin/com/example/crkl/media/ActiveMediaRegistry.kt
+94
View File
@@ -0,0 +1,94 @@
package com.example.crkl.media
import android.graphics.RectF
import java.util.concurrent.CopyOnWriteArrayList
/**
* Tracks on-screen playable media so a circle selection can resolve to a real asset.
* Fixtures (and later real players) register slots with screen bounds.
*/
object ActiveMediaRegistry {
enum class Kind { VIDEO, AUDIO }
/** Plain bounds so JVM unit tests do not need mocked android.graphics.RectF methods. */
data class Bounds(
val left: Float,
val top: Float,
val right: Float,
val bottom: Float
) {
fun isValid(): Boolean = right > left && bottom > top
fun toRectF(): RectF = RectF(left, top, right, bottom)
companion object {
fun from(rect: RectF): Bounds =
Bounds(rect.left, rect.top, rect.right, rect.bottom)
}
}
data class Slot(
val id: String,
val kind: Kind,
/** Path under assets/, e.g. media/grocery_memo.wav */
val assetPath: String,
/** Matching transcript under assets/ (fixture / on-device STT stand-in). */
val transcriptAssetPath: String,
val title: String,
@Volatile var screenBounds: Bounds = Bounds(0f, 0f, 0f, 0f)
)
private val slots = CopyOnWriteArrayList<Slot>()
fun register(slot: Slot) {
slots.removeAll { it.id == slot.id }
slots.add(slot)
}
fun unregister(id: String) {
slots.removeAll { it.id == id }
}
fun updateBounds(id: String, bounds: RectF) {
slots.find { it.id == id }?.screenBounds = Bounds.from(bounds)
}
fun updateBounds(id: String, bounds: Bounds) {
slots.find { it.id == id }?.screenBounds = bounds
}
fun clear() {
slots.clear()
}
fun findOverlapping(selection: RectF): Slot? =
findOverlapping(Bounds.from(selection))
fun findOverlapping(selection: Bounds): Slot? {
var best: Slot? = null
var bestArea = 0f
for (slot in slots) {
val b = slot.screenBounds
if (!b.isValid()) continue
val overlap = overlapArea(selection, b)
if (overlap > bestArea && overlap > 2_000f) {
bestArea = overlap
best = slot
}
}
return best
}
fun all(): List<Slot> = slots.toList()
private fun overlapArea(a: Bounds, b: Bounds): Float {
val left = maxOf(a.left, b.left)
val top = maxOf(a.top, b.top)
val right = minOf(a.right, b.right)
val bottom = minOf(a.bottom, b.bottom)
val w = right - left
val h = bottom - top
return if (w > 0f && h > 0f) w * h else 0f
}
}
app/src/main/kotlin/com/example/crkl/media/AssetMediaTranscriber.kt
+142
View File
@@ -0,0 +1,142 @@
package com.example.crkl.media
import android.content.Context
import android.media.MediaMetadataRetriever
import android.media.MediaPlayer
import android.util.Log
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.delay
import kotlinx.coroutines.suspendCancellableCoroutine
import kotlinx.coroutines.withContext
import kotlinx.coroutines.withTimeoutOrNull
import java.io.File
import java.io.FileOutputStream
import kotlin.coroutines.resume
/**
* Plays an asset media file (so the user hears/sees it) and returns a transcript.
*
* Fixture assets ship matching `*.transcript.txt` sidecars generated from the same
* spoken script. That unblocks end-to-end circle → listen → summarize on-device
* without cloud STT. Swap [loadTranscript] for Vosk/Whisper file STT later.
*/
class AssetMediaTranscriber(
private val context: Context
) {
private val tagName = "AssetMediaTranscriber"
data class Result(
val transcript: String,
val durationMs: Long,
val played: Boolean,
val source: String
)
suspend fun transcribe(
assetPath: String,
transcriptAssetPath: String,
playAudio: Boolean = true,
maxPlayMs: Long = 12_000L
): Result = withContext(Dispatchers.IO) {
val transcript = loadTranscript(transcriptAssetPath)
val cacheFile = copyAssetToCache(assetPath)
val durationMs = probeDurationMs(cacheFile) ?: estimateFromTranscript(transcript)
var played = false
if (playAudio) {
played = withContext(Dispatchers.Main) {
playFile(cacheFile, maxPlayMs.coerceAtMost(durationMs + 500L))
}
}
Result(
transcript = transcript,
durationMs = durationMs,
played = played,
source = "asset-transcript"
)
}
private fun loadTranscript(transcriptAssetPath: String): String {
return try {
context.assets.open(transcriptAssetPath).bufferedReader().use { it.readText() }.trim()
} catch (e: Exception) {
Log.w(tagName, "Missing transcript $transcriptAssetPath", e)
""
}
}
private fun copyAssetToCache(assetPath: String): File {
val name = assetPath.substringAfterLast('/')
val out = File(context.cacheDir, "crkl_media_$name")
if (!out.exists() || out.length() == 0L) {
context.assets.open(assetPath).use { input ->
FileOutputStream(out).use { output -> input.copyTo(output) }
}
}
return out
}
private fun probeDurationMs(file: File): Long? {
val retriever = MediaMetadataRetriever()
return try {
retriever.setDataSource(file.absolutePath)
retriever.extractMetadata(MediaMetadataRetriever.METADATA_KEY_DURATION)
?.toLongOrNull()
} catch (e: Exception) {
Log.w(tagName, "duration probe failed", e)
null
} finally {
try {
retriever.release()
} catch (_: Exception) {
}
}
}
private fun estimateFromTranscript(text: String): Long {
val words = text.split(Regex("\\s+")).count { it.isNotBlank() }
return (words * 420L).coerceIn(2_000L, 20_000L)
}
private suspend fun playFile(file: File, maxMs: Long): Boolean {
return withTimeoutOrNull(maxMs + 1_500L) {
suspendCancellableCoroutine { cont ->
val player = MediaPlayer()
try {
player.setDataSource(file.absolutePath)
player.setOnCompletionListener {
player.release()
if (cont.isActive) cont.resume(true)
}
player.setOnErrorListener { _, what, extra ->
Log.w(tagName, "MediaPlayer error what=$what extra=$extra")
player.release()
if (cont.isActive) cont.resume(false)
true
}
cont.invokeOnCancellation {
try {
player.stop()
} catch (_: Exception) {
}
player.release()
}
player.prepare()
player.start()
} catch (e: Exception) {
Log.e(tagName, "play failed", e)
try {
player.release()
} catch (_: Exception) {
}
if (cont.isActive) cont.resume(false)
}
}
} ?: run {
// Timed out — still count as attempted listen.
delay(50)
true
}
}
}
app/src/main/kotlin/com/example/crkl/media/MediaAssistPipeline.kt
+122
View File
@@ -0,0 +1,122 @@
package com.example.crkl.media
import android.content.Context
import android.graphics.RectF
import android.util.Log
import com.example.crkl.agent.AssistEngine
import com.example.crkl.capture.MediaProjectionHolder
import com.example.crkl.capture.PlaybackCaptureRecorder
import com.example.crkl.vision.ContentCapture
import com.example.crkl.vision.RegionContentExtractor
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.async
import kotlinx.coroutines.coroutineScope
import kotlinx.coroutines.withContext
/**
* Circle media → play/listen → transcript → [AssistEngine] summarize.
* When MediaProjection is granted, also runs [AudioPlaybackCapture] in parallel.
*/
class MediaAssistPipeline(
private val context: Context,
private val assistEngine: AssistEngine,
private val transcriber: AssetMediaTranscriber = AssetMediaTranscriber(context),
private val playbackCapture: PlaybackCaptureRecorder = PlaybackCaptureRecorder()
) {
private val tagName = "MediaAssistPipeline"
data class Outcome(
val response: AssistEngine.AssistResponse,
val slot: ActiveMediaRegistry.Slot,
val transcript: String
)
fun resolveSlot(selection: RectF, capture: ContentCapture.CaptureResult): ActiveMediaRegistry.Slot? {
ActiveMediaRegistry.findOverlapping(selection)?.let { return it }
val hay = capture.text.lowercase()
return ActiveMediaRegistry.all().firstOrNull { slot ->
hay.contains(slot.title.lowercase()) ||
hay.contains(slot.id.lowercase()) ||
(slot.kind == ActiveMediaRegistry.Kind.VIDEO && "video" in hay && "walkthrough" in hay) ||
(slot.kind == ActiveMediaRegistry.Kind.AUDIO && ("grocery" in hay || "voice memo" in hay))
}
}
suspend fun process(
slot: ActiveMediaRegistry.Slot,
onProgress: (String) -> Unit = {}
): Outcome = coroutineScope {
onProgress("Playing ${slot.kind.name.lowercase()}…")
Log.d(tagName, "process id=${slot.id} asset=${slot.assetPath}")
val projection = MediaProjectionHolder.get()
val captureJob = if (projection != null) {
onProgress("Playing + AudioPlaybackCapture…")
async(Dispatchers.IO) {
playbackCapture.capture(
projection = projection,
outDir = context.cacheDir,
durationMs = 9_000L
)
}
} else {
null
}
val media = withContext(Dispatchers.Default) {
transcriber.transcribe(
assetPath = slot.assetPath,
transcriptAssetPath = slot.transcriptAssetPath,
playAudio = true,
maxPlayMs = 3_500L
)
}
val capture = captureJob?.await()
val captureNote = when {
projection == null -> "capture=off (enable in Crkl home)"
capture == null -> "capture=empty"
else -> "capture=${capture.durationMs}ms wav"
}
onProgress("Transcribed · summarizing…")
val bodyForAssist = buildString {
appendLine("Media: ${slot.title}")
appendLine("Kind: ${slot.kind.name.lowercase()}")
appendLine("Duration: ${media.durationMs} ms")
appendLine("Transcript source: ${media.source}")
appendLine("PlaybackCapture: $captureNote")
if (capture != null) {
appendLine("Captured file: ${capture.wavFile.name} (${capture.bytesCaptured} bytes)")
}
appendLine()
appendLine("Transcript:")
append(media.transcript.ifBlank { "(empty transcript)" })
}
val extraction = RegionContentExtractor.ExtractionResult(
text = bodyForAssist,
nodeCount = 1,
bounds = slot.screenBounds.toRectF(),
packageName = context.packageName
)
val response = withContext(Dispatchers.Default) {
assistEngine.respond(extraction, mode = AssistEngine.Mode.MEDIA)
}
val playedNote = if (media.played) "played" else "play-skipped"
Outcome(
response = response.copy(
title = when (slot.kind) {
ActiveMediaRegistry.Kind.VIDEO -> "Video summary"
ActiveMediaRegistry.Kind.AUDIO -> "Audio summary"
},
meta = response.meta + " · media · $playedNote · ${media.source} · $captureNote"
),
slot = slot,
transcript = media.transcript
)
}
}
app/src/main/kotlin/com/example/crkl/model/.gitkeep
-2
View File
@@ -1,2 +0,0 @@
# STT/LLM wrappers, inference runners
app/src/main/kotlin/com/example/crkl/model/DeviceSpeechStt.kt
+95
View File
@@ -0,0 +1,95 @@
package com.example.crkl.model
import android.content.Context
import android.content.Intent
import android.os.Bundle
import android.speech.RecognitionListener
import android.speech.RecognizerIntent
import android.speech.SpeechRecognizer
import android.util.Log
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.suspendCancellableCoroutine
import kotlinx.coroutines.withContext
import kotlinx.coroutines.withTimeoutOrNull
import java.util.Locale
import kotlin.coroutines.resume
/**
* Device speech recognizer. Requests offline when the OEM supports it.
* Falls back to on-device/network engine provided by the system.
*/
class DeviceSpeechStt(
context: Context
) : LocalStt {
private val appContext = context.applicationContext
private val tagName = "DeviceSpeechStt"
private var recognizer: SpeechRecognizer? = null
override val isAvailable: Boolean
get() = SpeechRecognizer.isRecognitionAvailable(appContext)
override val displayName: String = "Device STT"
override suspend fun listen(timeoutMs: Long): String = withContext(Dispatchers.Main) {
if (!isAvailable) return@withContext ""
withTimeoutOrNull(timeoutMs + 1_500L) {
suspendCancellableCoroutine { cont ->
val sr = SpeechRecognizer.createSpeechRecognizer(appContext)
recognizer = sr
sr.setRecognitionListener(object : RecognitionListener {
override fun onReadyForSpeech(params: Bundle?) {
Log.d(tagName, "ready for speech")
}
override fun onBeginningOfSpeech() = Unit
override fun onRmsChanged(rmsdB: Float) = Unit
override fun onBufferReceived(buffer: ByteArray?) = Unit
override fun onEndOfSpeech() = Unit
override fun onPartialResults(partialResults: Bundle?) = Unit
override fun onEvent(eventType: Int, params: Bundle?) = Unit
override fun onError(error: Int) {
Log.w(tagName, "STT error=$error")
cleanup()
if (cont.isActive) cont.resume("")
}
override fun onResults(results: Bundle?) {
val texts = results
?.getStringArrayList(SpeechRecognizer.RESULTS_RECOGNITION)
.orEmpty()
cleanup()
if (cont.isActive) cont.resume(texts.firstOrNull().orEmpty())
}
})
cont.invokeOnCancellation { cleanup() }
val intent = Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH).apply {
putExtra(
RecognizerIntent.EXTRA_LANGUAGE_MODEL,
RecognizerIntent.LANGUAGE_MODEL_FREE_FORM
)
putExtra(RecognizerIntent.EXTRA_LANGUAGE, Locale.getDefault())
putExtra(RecognizerIntent.EXTRA_MAX_RESULTS, 1)
putExtra(RecognizerIntent.EXTRA_PARTIAL_RESULTS, false)
// Prefer on-device when the recognizer supports it (API 33+ constant mirrored).
putExtra("android.speech.extra.PREFER_OFFLINE", true)
}
sr.startListening(intent)
}
}.orEmpty()
}
override fun cancel() = cleanup()
private fun cleanup() {
try {
recognizer?.cancel()
recognizer?.destroy()
} catch (_: Exception) {
}
recognizer = null
}
}
app/src/main/kotlin/com/example/crkl/model/LocalLlm.kt
+16
View File
@@ -0,0 +1,16 @@
package com.example.crkl.model
/**
* On-device text generation. Implementations must not call the network.
*/
interface LocalLlm {
val isReady: Boolean
val displayName: String
/**
* Blocking/suspend generation. Call from a background dispatcher.
*/
suspend fun generate(prompt: String): String
fun close()
}
app/src/main/kotlin/com/example/crkl/model/LocalStt.kt
+16
View File
@@ -0,0 +1,16 @@
package com.example.crkl.model
/**
* Short-form speech-to-text. Implementations should prefer on-device engines.
*/
interface LocalStt {
val isAvailable: Boolean
val displayName: String
/**
* Listen for up to [timeoutMs] and return transcribed text (may be blank).
*/
suspend fun listen(timeoutMs: Long = 6_000L): String
fun cancel()
}
app/src/main/kotlin/com/example/crkl/model/MediaPipeLocalLlm.kt
+89
View File
@@ -0,0 +1,89 @@
package com.example.crkl.model
import android.content.Context
import android.util.Log
import com.google.mediapipe.tasks.genai.llminference.LlmInference
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.sync.Mutex
import kotlinx.coroutines.sync.withLock
import kotlinx.coroutines.withContext
import java.io.File
import java.util.concurrent.atomic.AtomicReference
/**
* MediaPipe LLM Inference backend (Gemma 3 .task / .litertlm).
*
* Loads lazily on first [ensureLoaded] / [generate]. Safe to call when no model
* file is present — [isReady] stays false and callers should fall back.
*/
class MediaPipeLocalLlm(
context: Context
) : LocalLlm {
private val appContext = context.applicationContext
private val tagName = "MediaPipeLocalLlm"
private val mutex = Mutex()
private val engine = AtomicReference<LlmInference?>(null)
private var modelFile: File? = null
@Volatile
override var isReady: Boolean = false
private set
@Volatile
override var displayName: String = "MediaPipe (unloaded)"
private set
suspend fun ensureLoaded(): Boolean = withContext(Dispatchers.Default) {
mutex.withLock {
if (engine.get() != null) return@withLock true
val file = ModelPaths.findModel(appContext.filesDir)
if (file == null) {
Log.i(tagName, "No model file found; LLM disabled")
isReady = false
displayName = "MediaPipe (no model)"
return@withLock false
}
try {
Log.i(tagName, "Loading model from ${file.absolutePath} (${file.length()} bytes)")
val options = LlmInference.LlmInferenceOptions.builder()
.setModelPath(file.absolutePath)
.setMaxTokens(1024)
.setMaxTopK(40)
.build()
val created = LlmInference.createFromOptions(appContext, options)
engine.set(created)
modelFile = file
isReady = true
displayName = "MediaPipe · ${file.name}"
Log.i(tagName, "Model ready: $displayName")
true
} catch (e: Exception) {
Log.e(tagName, "Failed to load model ${file.absolutePath}", e)
engine.set(null)
isReady = false
displayName = "MediaPipe (load failed)"
false
}
}
}
override suspend fun generate(prompt: String): String = withContext(Dispatchers.Default) {
if (!ensureLoaded()) {
error("On-device model is not loaded")
}
mutex.withLock {
val llm = engine.get() ?: error("On-device model is not loaded")
llm.generateResponse(prompt).orEmpty().trim()
}
}
override fun close() {
engine.getAndSet(null)?.close()
isReady = false
displayName = "MediaPipe (closed)"
modelFile = null
}
}
app/src/main/kotlin/com/example/crkl/model/ModelPaths.kt
+44
View File
@@ -0,0 +1,44 @@
package com.example.crkl.model
import java.io.File
/**
* Resolves on-device MediaPipe `.task` model paths.
* Models are NOT bundled in the APK — push with `make push-model`.
*/
object ModelPaths {
const val ADB_MODEL_PATH = "/data/local/tmp/llm/crkl.task"
private const val LEGACY_ADB_MODEL_PATH = "/data/local/tmp/llm/model.task"
fun candidateFiles(filesDir: File): List<File> {
val appModelsDir = File(filesDir, "models")
val appDirMatches = appModelsDir
.listFiles { f -> f.isFile && (f.name.endsWith(".task") || f.name.endsWith(".litertlm")) }
?.sortedBy { it.name }
.orEmpty()
return listOf(
File(ADB_MODEL_PATH),
File(LEGACY_ADB_MODEL_PATH),
File(appModelsDir, "gemma-3-1b-it-int4.task"),
File(appModelsDir, "model.task")
) + appDirMatches
}
fun findModel(filesDir: File): File? =
candidateFiles(filesDir).firstOrNull { it.isFile && it.canRead() && it.length() > 0L }
fun missingModelHint(): String = buildString {
appendLine("No on-device model found.")
appendLine()
appendLine("Download Gemma 3 1B INT4 (.task) from Hugging Face")
appendLine("(litert-community/Gemma3-1B-IT), then:")
appendLine()
appendLine(" make push-model MODEL=/path/to/model.task")
appendLine()
appendLine("Expected path: $ADB_MODEL_PATH")
appendLine()
appendLine("Until then, Crkl shows extracted text only.")
}
}
app/src/main/kotlin/com/example/crkl/ui/.gitkeep
-2
View File
@@ -1,2 +0,0 @@
# Jetpack Compose overlays and feedback UIs
app/src/main/kotlin/com/example/crkl/ui/CrklUi.kt
+23
View File
@@ -0,0 +1,23 @@
package com.example.crkl.ui
import android.graphics.Color
/**
* Shared overlay palette — ink paper + teal ring (not Material purple).
* Used by FAB, circle stroke defaults, and the result sheet.
*/
object CrklUi {
val Ink = Color.parseColor("#0E1A24")
val InkSoft = Color.parseColor("#1C2B38")
val Paper = Color.parseColor("#F3F5F2")
val PaperHi = Color.parseColor("#FFFFFF")
val Teal = Color.parseColor("#1FA7A0")
val TealDeep = Color.parseColor("#157F7A")
val Mist = Color.parseColor("#6B7A88")
val MistLight = Color.parseColor("#9AA6B2")
val Hairline = Color.parseColor("#1A0E1A24")
val Dim = Color.argb(140, 8, 14, 20)
val Danger = Color.parseColor("#B85A4A")
val Ok = Color.parseColor("#2A7A4B")
val ChipFg = Color.parseColor("#FFFFFF")
}
app/src/main/kotlin/com/example/crkl/ui/ResultPanelView.kt
+256
View File
@@ -0,0 +1,256 @@
package com.example.crkl.ui
import android.animation.AnimatorSet
import android.animation.ObjectAnimator
import android.content.Context
import android.graphics.Typeface
import android.graphics.drawable.GradientDrawable
import android.util.TypedValue
import android.view.Gravity
import android.view.View
import android.view.animation.DecelerateInterpolator
import android.widget.HorizontalScrollView
import android.widget.LinearLayout
import android.widget.ScrollView
import android.widget.TextView
/**
* Bottom-sheet style result card: title, text, five VIP chips, quiet status.
*/
class ResultPanelView(
context: Context,
private val onDismiss: () -> Unit,
private val onListen: (() -> Unit)? = null,
private val onShareList: (() -> Unit)? = null,
private val onAddTodo: (() -> Unit)? = null,
private val onTranslate: (() -> Unit)? = null,
private val onCopy: (() -> Unit)? = null,
private val onExplain: (() -> Unit)? = null
) : LinearLayout(context) {
private val titleView: TextView
private val metaView: TextView
private val bodyView: TextView
private val listenView: TextView
private val statusView: TextView
private var entered = false
init {
orientation = VERTICAL
setPadding(dp(18), dp(12), dp(18), dp(14))
background = GradientDrawable().apply {
setColor(CrklUi.PaperHi)
cornerRadii = floatArrayOf(
dp(22).toFloat(), dp(22).toFloat(),
dp(22).toFloat(), dp(22).toFloat(),
dp(10).toFloat(), dp(10).toFloat(),
dp(10).toFloat(), dp(10).toFloat()
)
setStroke(dp(1), CrklUi.Hairline)
}
elevation = dp(16).toFloat()
clipToOutline = false
// Drag handle
val handle = View(context).apply {
background = GradientDrawable().apply {
setColor(CrklUi.MistLight)
cornerRadius = dp(2).toFloat()
}
layoutParams = LayoutParams(dp(36), dp(4)).apply {
gravity = Gravity.CENTER_HORIZONTAL
bottomMargin = dp(12)
}
}
val header = LinearLayout(context).apply {
orientation = HORIZONTAL
gravity = Gravity.CENTER_VERTICAL
}
titleView = TextView(context).apply {
text = "Circle"
setTextColor(CrklUi.Ink)
setTextSize(TypedValue.COMPLEX_UNIT_SP, 17f)
typeface = Typeface.create(Typeface.SANS_SERIF, Typeface.BOLD)
layoutParams = LayoutParams(0, LayoutParams.WRAP_CONTENT, 1f)
}
val close = TextView(context).apply {
text = "Close"
setTextColor(CrklUi.Mist)
setTextSize(TypedValue.COMPLEX_UNIT_SP, 13f)
setPadding(dp(10), dp(6), dp(2), dp(6))
setOnClickListener { onDismiss() }
}
header.addView(titleView)
header.addView(close)
// Teal accent rule under header
val rule = View(context).apply {
background = GradientDrawable().apply {
setColor(CrklUi.Teal)
alpha = 180
}
layoutParams = LayoutParams(LayoutParams.MATCH_PARENT, dp(2)).apply {
topMargin = dp(10)
bottomMargin = dp(10)
}
}
metaView = TextView(context).apply {
setTextColor(CrklUi.Mist)
setTextSize(TypedValue.COMPLEX_UNIT_SP, 11f)
setPadding(0, 0, 0, dp(6))
visibility = GONE
}
bodyView = TextView(context).apply {
setTextColor(CrklUi.InkSoft)
setTextSize(TypedValue.COMPLEX_UNIT_SP, 14.5f)
setLineSpacing(dp(2).toFloat(), 1.2f)
}
val scroll = ScrollView(context).apply {
layoutParams = LayoutParams(LayoutParams.MATCH_PARENT, 0, 1f)
isFillViewport = true
isVerticalScrollBarEnabled = false
addView(bodyView, LayoutParams(LayoutParams.MATCH_PARENT, LayoutParams.WRAP_CONTENT))
}
statusView = TextView(context).apply {
setTextColor(CrklUi.Ok)
setTextSize(TypedValue.COMPLEX_UNIT_SP, 12.5f)
setPadding(0, dp(6), 0, 0)
visibility = GONE
}
val actionsRow = LinearLayout(context).apply {
orientation = HORIZONTAL
gravity = Gravity.CENTER_VERTICAL
}
fun solidChip(label: String, bg: Int, onClick: () -> Unit): TextView =
TextView(context).apply {
text = label
setTextColor(CrklUi.ChipFg)
setTextSize(TypedValue.COMPLEX_UNIT_SP, 13f)
typeface = Typeface.DEFAULT_BOLD
setPadding(dp(14), dp(11), dp(14), dp(11))
background = GradientDrawable().apply {
setColor(bg)
cornerRadius = dp(20).toFloat()
}
val lp = LayoutParams(LayoutParams.WRAP_CONTENT, LayoutParams.WRAP_CONTENT)
lp.marginEnd = dp(8)
layoutParams = lp
isClickable = true
isFocusable = true
setOnClickListener {
statusView.visibility = VISIBLE
statusView.setTextColor(CrklUi.Mist)
statusView.text = "$label…"
onClick()
}
}
// VIP only — Email / Calendar stay on voice
if (onTranslate != null) {
actionsRow.addView(solidChip("Translate", CrklUi.TealDeep, onTranslate))
}
if (onCopy != null) {
actionsRow.addView(solidChip("Copy", CrklUi.InkSoft, onCopy))
}
if (onExplain != null) {
actionsRow.addView(solidChip("Explain", CrklUi.Ink, onExplain))
}
if (onShareList != null) {
actionsRow.addView(solidChip("Share", CrklUi.Mist, onShareList))
}
if (onAddTodo != null) {
actionsRow.addView(solidChip("Vikunja", CrklUi.Ok, onAddTodo))
}
val actionsScroll = HorizontalScrollView(context).apply {
isHorizontalScrollBarEnabled = false
overScrollMode = OVER_SCROLL_NEVER
setPadding(0, dp(10), 0, dp(2))
addView(actionsRow)
}
listenView = TextView(context).apply {
text = "Or speak a command"
gravity = Gravity.CENTER
setTextColor(CrklUi.TealDeep)
setTextSize(TypedValue.COMPLEX_UNIT_SP, 13f)
setPadding(0, dp(10), 0, dp(2))
setOnClickListener { onListen?.invoke() }
visibility = if (onListen != null) VISIBLE else GONE
}
addView(handle)
addView(header, LayoutParams(LayoutParams.MATCH_PARENT, LayoutParams.WRAP_CONTENT))
addView(rule)
addView(metaView, LayoutParams(LayoutParams.MATCH_PARENT, LayoutParams.WRAP_CONTENT))
addView(scroll)
if (actionsRow.childCount > 0) {
addView(actionsScroll, LayoutParams(LayoutParams.MATCH_PARENT, LayoutParams.WRAP_CONTENT))
}
addView(listenView, LayoutParams(LayoutParams.MATCH_PARENT, LayoutParams.WRAP_CONTENT))
addView(statusView, LayoutParams(LayoutParams.MATCH_PARENT, LayoutParams.WRAP_CONTENT))
}
fun playEnter() {
if (entered) return
entered = true
alpha = 0f
translationY = dp(48).toFloat()
val fade = ObjectAnimator.ofFloat(this, View.ALPHA, 0f, 1f)
val slide = ObjectAnimator.ofFloat(this, View.TRANSLATION_Y, dp(48).toFloat(), 0f)
AnimatorSet().apply {
playTogether(fade, slide)
duration = 240
interpolator = DecelerateInterpolator()
start()
}
}
fun showResult(title: String, meta: String, body: String) {
titleView.text = title
if (meta.isBlank()) {
metaView.visibility = GONE
metaView.text = ""
} else {
metaView.visibility = VISIBLE
metaView.text = meta
}
bodyView.text = body
if (!entered) playEnter()
}
fun appendBody(extra: String) {
val current = bodyView.text?.toString().orEmpty()
bodyView.text = if (current.isBlank()) extra else "$current\n\n$extra"
statusView.visibility = VISIBLE
statusView.text = extra.lineSequence().firstOrNull { it.isNotBlank() }?.take(100)
?: "Done"
val bad = extra.contains("fail", ignoreCase = true) ||
extra.contains("Couldn’t", ignoreCase = true) ||
extra.contains("Couldn't", ignoreCase = true) ||
extra.contains("not configured", ignoreCase = true) ||
extra.contains("Nothing", ignoreCase = true)
statusView.setTextColor(if (bad) CrklUi.Danger else CrklUi.Ok)
}
fun setListening(listening: Boolean) {
listenView.text = if (listening) "Listening…" else "Or speak a command"
}
private fun dp(value: Int): Int =
TypedValue.applyDimension(
TypedValue.COMPLEX_UNIT_DIP,
value.toFloat(),
resources.displayMetrics
).toInt()
}
app/src/main/kotlin/com/example/crkl/ui/theme/Theme.kt
+38 -17
View File
@@ -1,22 +1,49 @@
package com.example.crkl.ui.theme
import androidx.compose.foundation.isSystemInDarkTheme
import androidx.compose.material3.*
import androidx.compose.material3.MaterialTheme
import androidx.compose.material3.darkColorScheme
import androidx.compose.material3.lightColorScheme
import androidx.compose.runtime.Composable
import androidx.compose.ui.graphics.Color
private val Ink = Color(0xFF0E1A24)
private val InkSoft = Color(0xFF1C2B38)
private val Paper = Color(0xFFF3F5F2)
private val Teal = Color(0xFF1FA7A0)
private val TealDeep = Color(0xFF157F7A)
private val Mist = Color(0xFF6B7A88)
private val Danger = Color(0xFFB85A4A)
private val DarkColorScheme = darkColorScheme(
primary = Color(0xFF6200EE),
secondary = Color(0xFF03DAC6),
tertiary = Color(0xFF018786)
primary = Teal,
onPrimary = Color.White,
secondary = TealDeep,
tertiary = Mist,
background = Ink,
surface = InkSoft,
onBackground = Paper,
onSurface = Paper,
error = Danger
)
private val LightColorScheme = lightColorScheme(
primary = Color(0xFF6200EE),
secondary = Color(0xFF03DAC6),
tertiary = Color(0xFF018786),
background = Color(0xFFFFFBFE),
surface = Color(0xFFFFFBFE),
primary = TealDeep,
onPrimary = Color.White,
secondary = InkSoft,
tertiary = Mist,
background = Paper,
surface = Color.White,
onBackground = Ink,
onSurface = Ink,
onSurfaceVariant = Mist,
primaryContainer = Color(0xFFD7EFEC),
onPrimaryContainer = Ink,
secondaryContainer = Color(0xFFE8ECF0),
onSecondaryContainer = InkSoft,
errorContainer = Color(0xFFF5DED9),
onErrorContainer = Color(0xFF5C2A22),
error = Danger
)
@Composable
@@ -24,15 +51,9 @@ fun CrklTheme(
darkTheme: Boolean = isSystemInDarkTheme(),
content: @Composable () -> Unit
) {
val colorScheme = when {
darkTheme -> DarkColorScheme
else -> LightColorScheme
}
MaterialTheme(
colorScheme = colorScheme,
typography = Typography(),
colorScheme = if (darkTheme) DarkColorScheme else LightColorScheme,
typography = Typography,
content = content
)
}
app/src/main/kotlin/com/example/crkl/vision/.gitkeep
-2
View File
@@ -1,2 +0,0 @@
# Content classification and ML components
app/src/main/kotlin/com/example/crkl/vision/ContentCapture.kt
+87
View File
@@ -0,0 +1,87 @@
package com.example.crkl.vision
import android.graphics.RectF
import android.util.Log
import android.view.accessibility.AccessibilityNodeInfo
import android.view.accessibility.AccessibilityWindowInfo
/**
* Merges accessibility text + optional OCR into one extraction payload.
*/
object ContentCapture {
private const val TAG = "ContentCapture"
enum class Source { A11Y, OCR, MERGED, EMPTY }
data class CaptureResult(
val text: String,
val nodeCount: Int,
val bounds: RectF,
val packageName: String?,
val source: Source,
val ocrBlockCount: Int = 0
) {
val isEmpty: Boolean get() = text.isBlank()
fun toExtraction(): RegionContentExtractor.ExtractionResult =
RegionContentExtractor.ExtractionResult(
text = text,
nodeCount = nodeCount,
bounds = bounds,
packageName = packageName
)
}
fun fromA11y(
windows: List<AccessibilityWindowInfo>?,
rootFallback: AccessibilityNodeInfo?,
selection: RectF
): CaptureResult {
val base = RegionContentExtractor.extract(windows, rootFallback, selection)
return CaptureResult(
text = base.text,
nodeCount = base.nodeCount,
bounds = base.bounds,
packageName = base.packageName,
source = if (base.isEmpty) Source.EMPTY else Source.A11Y
)
}
fun mergeWithOcr(a11y: CaptureResult, ocrText: String, ocrBlocks: Int): CaptureResult {
val a11yText = a11y.text.trim()
val ocr = ocrText.trim()
val (merged, source) = when {
a11yText.isBlank() && ocr.isBlank() -> "" to Source.EMPTY
a11yText.isBlank() -> ocr to Source.OCR
ocr.isBlank() -> a11yText to Source.A11Y
ocr.contains(a11yText) || a11yText.contains(ocr) -> {
// Prefer the longer unique reading.
(if (ocr.length >= a11yText.length) ocr else a11yText) to Source.MERGED
}
else -> "$a11yText\n\n$ocr" to Source.MERGED
}
try {
Log.d(TAG, "merge source=$source a11y=${a11yText.length} ocr=${ocr.length}")
} catch (_: RuntimeException) {
// Android Log is not mocked in JVM unit tests.
}
return a11y.copy(
text = merged.take(4_000),
source = source,
ocrBlockCount = ocrBlocks
)
}
/** Enough letter characters to show as circled text (not OCR junk / icons). */
fun hasUsefulText(text: String): Boolean {
return text.count { it.isLetter() } >= 6
}
/** Heuristic: selection looks like media chrome (audio/video). */
fun looksLikeMedia(text: String, packageName: String?): Boolean {
val hay = (text + " " + (packageName ?: "")).lowercase()
return listOf("audio", "voice", "video", "player", "♫", "▶", "memo", "podcast")
.any { it in hay }
}
}
app/src/main/kotlin/com/example/crkl/vision/RegionContentExtractor.kt
+124
View File
@@ -0,0 +1,124 @@
package com.example.crkl.vision
import android.graphics.Rect
import android.graphics.RectF
import android.util.Log
import android.view.accessibility.AccessibilityNodeInfo
import android.view.accessibility.AccessibilityWindowInfo
/**
* Pulls readable text from accessibility nodes that intersect a screen-space selection.
*
* Prefers the accessibility node tree over MediaProjection so the first vertical
* slice stays permission-light and fully on-device.
*/
object RegionContentExtractor {
private const val TAG = "RegionContentExtractor"
private const val MAX_NODES = 80
private const val MAX_CHARS = 4_000
data class ExtractionResult(
val text: String,
val nodeCount: Int,
val bounds: RectF,
val packageName: String?
) {
val isEmpty: Boolean get() = text.isBlank()
}
fun extract(
windows: List<AccessibilityWindowInfo>?,
rootFallback: AccessibilityNodeInfo?,
selection: RectF
): ExtractionResult {
val selectionRect = Rect().also { selection.round(it) }
val snippets = linkedSetOf<String>()
var nodesHit = 0
var packageName: String? = null
val roots = mutableListOf<AccessibilityNodeInfo>()
windows
?.filter { it.type == AccessibilityWindowInfo.TYPE_APPLICATION }
?.sortedByDescending { it.layer }
?.forEach { window ->
window.root?.let { roots.add(it) }
}
if (roots.isEmpty()) {
rootFallback?.let { roots.add(it) }
}
for (root in roots) {
if (packageName == null) {
packageName = root.packageName?.toString()
}
try {
nodesHit += collectIntersectingText(root, selectionRect, snippets)
} finally {
// Only recycle roots we obtained from windows; caller owns rootFallback.
if (root !== rootFallback) {
root.recycle()
}
}
if (snippets.isNotEmpty() || nodesHit >= MAX_NODES) break
}
val joined = snippets
.joinToString(separator = "\n")
.trim()
.take(MAX_CHARS)
Log.d(
TAG,
"extract: nodes=$nodesHit snippets=${snippets.size} chars=${joined.length} pkg=$packageName"
)
return ExtractionResult(
text = joined,
nodeCount = nodesHit,
bounds = RectF(selection),
packageName = packageName
)
}
private fun collectIntersectingText(
node: AccessibilityNodeInfo,
selection: Rect,
out: MutableSet<String>,
depth: Int = 0
): Int {
if (depth > 40 || out.size >= MAX_NODES) return 0
var hit = 0
val bounds = Rect()
node.getBoundsInScreen(bounds)
if (Rect.intersects(bounds, selection)) {
readableText(node)?.let { out.add(it) }
hit = 1
}
for (i in 0 until node.childCount) {
val child = node.getChild(i) ?: continue
try {
hit += collectIntersectingText(child, selection, out, depth + 1)
} finally {
child.recycle()
}
if (out.size >= MAX_NODES) break
}
return hit
}
private fun readableText(node: AccessibilityNodeInfo): String? {
val candidates = listOfNotNull(
node.text?.toString(),
node.contentDescription?.toString(),
node.hintText?.toString()
)
.map { it.trim() }
.filter { it.isNotEmpty() && it.length < 2_000 }
return candidates.firstOrNull()
}
}
app/src/main/kotlin/com/example/crkl/vision/ScreenOcr.kt
+55
View File
@@ -0,0 +1,55 @@
package com.example.crkl.vision
import android.graphics.Bitmap
import android.util.Log
import com.google.mlkit.vision.common.InputImage
import com.google.mlkit.vision.text.TextRecognition
import com.google.mlkit.vision.text.latin.TextRecognizerOptions
import kotlinx.coroutines.suspendCancellableCoroutine
import kotlin.coroutines.resume
/**
* On-device OCR via ML Kit (latin script). No network.
*/
object ScreenOcr {
private const val TAG = "ScreenOcr"
private val recognizer by lazy {
TextRecognition.getClient(TextRecognizerOptions.DEFAULT_OPTIONS)
}
data class OcrResult(
val text: String,
val blockCount: Int
) {
val isEmpty: Boolean get() = text.isBlank()
}
suspend fun recognize(bitmap: Bitmap): OcrResult {
return try {
val image = InputImage.fromBitmap(bitmap, 0)
suspendCancellableCoroutine { cont ->
recognizer.process(image)
.addOnSuccessListener { result ->
val text = result.text.trim()
Log.d(TAG, "OCR blocks=${result.textBlocks.size} chars=${text.length}")
if (cont.isActive) {
cont.resume(
OcrResult(
text = text.take(4_000),
blockCount = result.textBlocks.size
)
)
}
}
.addOnFailureListener { e ->
Log.e(TAG, "OCR failed", e)
if (cont.isActive) cont.resume(OcrResult("", 0))
}
}
} catch (e: Exception) {
Log.e(TAG, "OCR failed", e)
OcrResult(text = "", blockCount = 0)
}
}
}
app/src/main/kotlin/com/example/crkl/vision/ScreenshotCapturer.kt
+83
View File
@@ -0,0 +1,83 @@
package com.example.crkl.vision
import android.accessibilityservice.AccessibilityService
import android.graphics.Bitmap
import android.graphics.Rect
import android.graphics.RectF
import android.os.Build
import android.util.Log
import android.view.Display
import androidx.annotation.RequiresApi
import kotlinx.coroutines.suspendCancellableCoroutine
import java.util.concurrent.Executors
import kotlin.coroutines.resume
/**
* Captures a screenshot via AccessibilityService (API 30+) and crops to [bounds].
*/
object ScreenshotCapturer {
private const val TAG = "ScreenshotCapturer"
private val executor = Executors.newSingleThreadExecutor()
suspend fun captureRegion(
service: AccessibilityService,
bounds: RectF
): Bitmap? {
if (Build.VERSION.SDK_INT < Build.VERSION_CODES.R) {
Log.w(TAG, "takeScreenshot requires API 30+")
return null
}
val full = captureFull(service) ?: return null
return try {
crop(full, bounds)
} finally {
if (!full.isRecycled) full.recycle()
}
}
@RequiresApi(Build.VERSION_CODES.R)
private suspend fun captureFull(service: AccessibilityService): Bitmap? =
suspendCancellableCoroutine { cont ->
service.takeScreenshot(
Display.DEFAULT_DISPLAY,
executor,
object : AccessibilityService.TakeScreenshotCallback {
override fun onSuccess(screenshot: AccessibilityService.ScreenshotResult) {
try {
val hw = screenshot.hardwareBuffer
val bitmap = Bitmap.wrapHardwareBuffer(hw, screenshot.colorSpace)
?.copy(Bitmap.Config.ARGB_8888, false)
hw.close()
if (cont.isActive) cont.resume(bitmap)
} catch (e: Exception) {
Log.e(TAG, "Screenshot wrap failed", e)
if (cont.isActive) cont.resume(null)
}
}
override fun onFailure(errorCode: Int) {
Log.e(TAG, "takeScreenshot failed code=$errorCode")
if (cont.isActive) cont.resume(null)
}
}
)
}
private fun crop(source: Bitmap, bounds: RectF): Bitmap? {
val rect = Rect().also { bounds.round(it) }
val left = rect.left.coerceIn(0, source.width - 1)
val top = rect.top.coerceIn(0, source.height - 1)
val right = rect.right.coerceIn(left + 1, source.width)
val bottom = rect.bottom.coerceIn(top + 1, source.height)
val w = right - left
val h = bottom - top
if (w < 8 || h < 8) return null
return try {
Bitmap.createBitmap(source, left, top, w, h)
} catch (e: Exception) {
Log.e(TAG, "crop failed $rect on ${source.width}x${source.height}", e)
null
}
}
}
app/src/main/res/drawable-hdpi/ic_launcher_foreground.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 32 KiB

app/src/main/res/drawable-mdpi/ic_launcher_foreground.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 15 KiB

app/src/main/res/drawable-xhdpi/ic_launcher_foreground.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 57 KiB

app/src/main/res/drawable-xhdpi/ic_qs_circle.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.2 KiB

app/src/main/res/drawable-xxhdpi/ic_launcher_foreground.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 132 KiB

app/src/main/res/drawable-xxhdpi/ic_qs_circle.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 3.6 KiB

app/src/main/res/drawable-xxxhdpi/ic_launcher_foreground.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 237 KiB

app/src/main/res/drawable-xxxhdpi/ic_qs_circle.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 7.0 KiB

app/src/main/res/drawable/ic_launcher_foreground.xml
-22
View File
@@ -1,22 +0,0 @@
<?xml version="1.0" encoding="utf-8"?>
<vector xmlns:android="http://schemas.android.com/apk/res/android"
android:width="108dp"
android:height="108dp"
android:viewportWidth="108"
android:viewportHeight="108">
<group
android:scaleX="0.5"
android:scaleY="0.5"
android:translateX="27"
android:translateY="27">
<!-- Circle shape representing the "circle to select" gesture -->
<path
android:fillColor="#FFFFFF"
android:pathData="M54,10 A44,44 0 1,1 54,98 A44,44 0 1,1 54,10 Z M54,20 A34,34 0 1,0 54,88 A34,34 0 1,0 54,20 Z"/>
<!-- AI sparkle/star in center -->
<path
android:fillColor="#03DAC6"
android:pathData="M54,40 L58,50 L68,54 L58,58 L54,68 L50,58 L40,54 L50,50 Z"/>
</group>
</vector>
app/src/main/res/mipmap-anydpi-v26/ic_launcher.xml
+2 -3
View File
@@ -1,6 +1,5 @@
<?xml version="1.0" encoding="utf-8"?>
<adaptive-icon xmlns:android="http://schemas.android.com/apk/res/android">
<background android:drawable="@color/primary"/>
<foreground android:drawable="@mipmap/ic_launcher_foreground"/>
<background android:drawable="@color/ic_launcher_background" />
<foreground android:drawable="@drawable/ic_launcher_foreground" />
</adaptive-icon>
app/src/main/res/mipmap-anydpi-v26/ic_launcher_round.xml
+2 -3
View File
@@ -1,6 +1,5 @@
<?xml version="1.0" encoding="utf-8"?>
<adaptive-icon xmlns:android="http://schemas.android.com/apk/res/android">
<background android:drawable="@color/primary"/>
<foreground android:drawable="@mipmap/ic_launcher_foreground"/>
<background android:drawable="@color/ic_launcher_background" />
<foreground android:drawable="@drawable/ic_launcher_foreground" />
</adaptive-icon>
app/src/main/res/values/colors.xml
+8 -7
View File
@@ -1,11 +1,12 @@
<?xml version="1.0" encoding="utf-8"?>
<resources>
<color name="primary">#6200EE</color>
<color name="primary_variant">#3700B3</color>
<color name="secondary">#03DAC6</color>
<color name="secondary_variant">#018786</color>
<!-- Brand: ink + teal (matches CrklUi / Circle mark) -->
<color name="ic_launcher_background">#F3F5F2</color>
<color name="primary">#157F7A</color>
<color name="primary_variant">#0E1A24</color>
<color name="secondary">#1FA7A0</color>
<color name="secondary_variant">#157F7A</color>
<color name="on_primary">#FFFFFF</color>
<color name="on_secondary">#000000</color>
<color name="overlay_highlight">#8003DAC6</color>
<color name="on_secondary">#0E1A24</color>
<color name="overlay_highlight">#801FA7A0</color>
</resources>
app/src/main/res/values/strings.xml
+13 -8
View File
@@ -1,14 +1,19 @@
<?xml version="1.0" encoding="utf-8"?>
<resources>
<string name="app_name">Crkl</string>
<string name="accessibility_service_name">Crkl Overlay Service</string>
<string name="accessibility_service_description">Allows Crkl to create overlays and capture screen content for local AI assistance. All processing is done on-device with no data leaving your phone.</string>
<string name="enable_service">Enable Crkl Accessibility Service</string>
<string name="app_name">Circle</string>
<string name="accessibility_service_name">Circle Overlay</string>
<string name="accessibility_service_description">Lets Circle draw over other apps and read on-screen text for on-device assist. Nothing leaves your phone unless you use an integration you configure.</string>
<string name="enable_service">Enable Circle Overlay</string>
<string name="service_enabled">Service Enabled</string>
<string name="service_disabled">Service Disabled</string>
<string name="open_settings">Open Accessibility Settings</string>
<string name="welcome_message">Welcome to Crkl</string>
<string name="privacy_notice">Privacy-first, on-device AI assistant. No data leaves your device.</string>
<string name="setup_instructions">To get started, enable the Crkl Accessibility Service in Settings.</string>
<string name="welcome_message">Welcome to Circle</string>
<string name="privacy_notice">Privacy-first, on-device assist. Circle text → act.</string>
<string name="setup_instructions">Enable Circle Overlay in Accessibility Settings to show the floating C.</string>
<string name="test_fixtures_label">Circle Test Fixtures</string>
<string name="open_test_fixtures">Open test fixtures</string>
<string name="qs_tile_label">Circle</string>
<string name="qs_tile_subtitle_ready">Draw on screen</string>
<string name="qs_tile_subtitle_off">Enable Accessibility</string>
<string name="add_qs_tile">Add Quick Settings tile</string>
</resources>
app/src/main/res/xml/accessibility_service_config.xml
+3 -1
View File
@@ -2,9 +2,11 @@
<accessibility-service xmlns:android="http://schemas.android.com/apk/res/android"
android:description="@string/accessibility_service_description"
android:accessibilityEventTypes="typeAllMask"
android:accessibilityFlags="flagDefault|flagRetrieveInteractiveWindows"
android:accessibilityFlags="flagDefault|flagRetrieveInteractiveWindows|flagReportViewIds|flagIncludeNotImportantViews|flagRequestAccessibilityButton"
android:accessibilityFeedbackType="feedbackGeneric"
android:canRetrieveWindowContent="true"
android:canTakeScreenshot="true"
android:canRequestTouchExplorationMode="false"
android:notificationTimeout="100" />
app/src/test/java/com/example/crkl/accessibility/CrklOverlayBridgeTest.kt
+22
View File
@@ -0,0 +1,22 @@
package com.example.crkl.accessibility
import org.junit.Assert.assertFalse
import org.junit.Assert.assertTrue
import org.junit.Test
class CrklOverlayBridgeTest {
@Test
fun enterCircleMode_withoutService_returnsFalse() {
// Ensure unbound (no live a11y service in JVM unit tests).
assertFalse(CrklOverlayBridge.isReady())
assertFalse(CrklOverlayBridge.enterCircleMode())
}
@Test
fun unbind_clearsOnlyMatchingInstance() {
// Smoke: API stays callable; readiness stays false without a real service.
assertFalse(CrklOverlayBridge.isReady())
assertTrue(!CrklOverlayBridge.enterCircleMode())
}
}
app/src/test/java/com/example/crkl/agent/ActionExecutorTest.kt
+213
View File
@@ -0,0 +1,213 @@
package com.example.crkl.agent
import android.content.Context
import android.content.Intent
import com.example.crkl.integrations.DeviceCalendar
import com.example.crkl.integrations.GogBridgeClient
import com.example.crkl.integrations.IntegrationSettings
import com.example.crkl.integrations.OnDeviceTranslator
import com.example.crkl.integrations.VikunjaClient
import com.example.crkl.testutil.MemorySharedPreferences
import kotlinx.coroutines.test.runTest
import org.junit.Assert.assertEquals
import org.junit.Assert.assertFalse
import org.junit.Assert.assertTrue
import org.junit.Test
import org.junit.runner.RunWith
import org.robolectric.RobolectricTestRunner
import org.robolectric.RuntimeEnvironment
@RunWith(RobolectricTestRunner::class)
class ActionExecutorTest {
private val context: Context get() = RuntimeEnvironment.getApplication()
private fun settings(
token: String = "tok",
emailTo: String = "idobkin@gmail.com",
preferGog: Boolean = false,
gogUrl: String = ""
): IntegrationSettings {
val s = IntegrationSettings(MemorySharedPreferences())
s.vikunjaToken = token
s.emailTo = emailTo
s.preferGogEmail = preferGog
s.gogBridgeUrl = gogUrl
return s
}
private fun session(body: String = "• Milk\n• Eggs") =
ActionExecutor.Session(title = "Groceries", body = body, transcript = "")
@Test
fun addToTodo_callsVikunjaWithListItems() = runTest {
var captured: List<String> = emptyList()
val exec = ActionExecutor(
context = context,
settings = settings(),
createTasks = { titles, _ ->
captured = titles
VikunjaClient.CreateResult(true, "ok")
}
)
val r = exec.execute(VoiceIntent.parse("add this to my todo"), session())
assertTrue(r.ok)
assertEquals(listOf("Milk", "Eggs"), captured)
}
@Test
fun addToTodo_usesEmailSubjectWhenNoBullets() = runTest {
var captured: List<String> = emptyList()
val exec = ActionExecutor(
context = context,
settings = settings(),
createTasks = { titles, _ ->
captured = titles
VikunjaClient.CreateResult(true, "Added 1")
}
)
val emailSession = ActionExecutor.Session(
title = "Extracted text",
body = """
From: alex@example.com
Subject: Q2 planning moved
Hi — the sync is Thursday.
""".trimIndent()
)
val r = exec.execute(VoiceIntent.parse("add this to my todo"), emailSession)
assertTrue(r.ok)
assertEquals(listOf("Q2 planning moved"), captured)
}
@Test
fun emailList_usesGogWhenPreferred() = runTest {
var sentTo: String? = null
val exec = ActionExecutor(
context = context,
settings = settings(preferGog = true, gogUrl = "http://10.0.2.2:8765"),
sendViaGog = { to, _, _ ->
sentTo = to
GogBridgeClient.BridgeResult(true, "Sent via gog")
},
composeEmail = { _, _, _ -> error("mailto should not run") }
)
val r = exec.execute(VoiceIntent.parse("email me the list"), session())
assertTrue(r.ok)
assertEquals("idobkin@gmail.com", sentTo)
}
@Test
fun showCalendar_returnsDeviceCalendarMessage() = runTest {
val exec = ActionExecutor(
context = context,
settings = settings(),
todayCalendar = {
DeviceCalendar.Result(true, "Today’s calendar:\n• 18:00 — Interview")
}
)
val r = exec.execute(VoiceIntent.parse("what's on my calendar"), null)
assertTrue(r.ok)
assertTrue(r.message.contains("Interview"))
}
@Test
fun openTodos_opensVikunjaUrl() = runTest {
var opened: String? = null
val exec = ActionExecutor(
context = context,
settings = settings(),
openTodosUrl = {
opened = it
"Opening $it"
}
)
val r = exec.execute(VoiceIntent.parse("open my todos"), null)
assertTrue(r.ok)
assertEquals(IntegrationSettings.DEFAULT_VIKUNJA_URL, opened)
}
@Test
fun translate_usesCircledTextAndTarget() = runTest {
var gotText: String? = null
var gotTarget: String? = null
val exec = ActionExecutor(
context = context,
settings = settings().also { it.translateTargetLang = "ru" },
translateText = { text, target ->
gotText = text
gotTarget = target
OnDeviceTranslator.Result(true, "Translated:\nпривет")
}
)
val session = ActionExecutor.Session(
title = "Extracted text",
body = "Hello world\n\n—\nCommands: ignore"
)
val r = exec.execute(VoiceIntent.parse("translate"), session)
assertTrue(r.ok)
assertTrue(gotText!!.contains("Hello"))
assertEquals("ru", gotTarget)
}
@Test
fun copy_putsSharePayloadOnClipboard() = runTest {
var copied: String? = null
val exec = ActionExecutor(
context = context,
settings = settings(),
copyText = { text ->
copied = text
true
}
)
val r = exec.execute(VoiceIntent.parse("copy"), session())
assertTrue(r.ok)
assertTrue(r.message.contains("Copied"))
assertTrue(copied!!.contains("Milk"))
assertTrue(copied!!.contains("Eggs"))
}
@Test
fun explain_usesInjectedExplainer() = runTest {
var seen: String? = null
val exec = ActionExecutor(
context = context,
settings = settings(),
explainText = { text ->
seen = text
AssistEngine.AssistResponse("Explain", "test", "Plain: meeting moved.")
}
)
val emailSession = ActionExecutor.Session(
title = "Extracted text",
body = "From: alex@example.com\nSubject: Q2 planning moved\nHi — Thursday."
)
val r = exec.execute(VoiceIntent.parse("explain"), emailSession)
assertTrue(r.ok)
assertTrue(r.message.contains("Plain: meeting moved"))
assertTrue(seen!!.contains("Subject:"))
}
@Test
fun share_alwaysOpensChooserWithPayload() = runTest {
var launched: Intent? = null
val exec = ActionExecutor(
context = context,
settings = settings(),
launchIntent = { launched = it }
)
val prose = ActionExecutor.Session(
title = "Note",
body = "The sync is Thursday afternoon at 3pm."
)
val r = exec.execute(VoiceIntent.parse("share"), prose)
assertTrue(r.ok)
assertTrue(r.message.contains("share sheet"))
assertEquals(Intent.ACTION_CHOOSER, launched!!.action)
@Suppress("DEPRECATION")
val share = launched!!.getParcelableExtra<Intent>(Intent.EXTRA_INTENT)
requireNotNull(share)
assertEquals(Intent.ACTION_SEND, share.action)
assertTrue(share.getStringExtra(Intent.EXTRA_TEXT)!!.contains("Thursday"))
}
}
app/src/test/java/com/example/crkl/agent/AssistEngineTest.kt
+57
View File
@@ -0,0 +1,57 @@
package com.example.crkl.agent
import com.example.crkl.model.LocalLlm
import com.example.crkl.vision.RegionContentExtractor
import kotlinx.coroutines.runBlocking
import org.junit.Assert.assertFalse
import org.junit.Assert.assertTrue
import org.junit.Test
class AssistEngineTest {
private val sampleExtraction = RegionContentExtractor.ExtractionResult(
text = "Meeting moved to Thursday at 3pm. Bring the Q2 notes.",
nodeCount = 2,
bounds = android.graphics.RectF(0f, 0f, 100f, 40f),
packageName = "com.google.android.apps.messaging"
)
@Test
fun respond_withoutLlm_fallsBackToStubClean() = runBlocking {
val engine = AssistEngine(llm = null)
val response = engine.respond(sampleExtraction)
assertTrue(response.body.contains("Thursday") || response.body.contains("Meeting"))
assertFalse(response.body.contains("push-model"))
assertFalse(response.body.contains("No on-device model"))
assertTrue(response.meta.isEmpty())
}
@Test
fun respond_withReadyLlm_usesGeneratedText() = runBlocking {
val fake = object : LocalLlm {
override val isReady = true
override val displayName = "fake-llm"
override suspend fun generate(prompt: String): String =
"The message reschedules a meeting to Thursday 3pm and asks for Q2 notes."
override fun close() = Unit
}
val engine = AssistEngine(llm = fake)
val response = engine.respond(sampleExtraction)
assertTrue(response.title.contains("On-device", ignoreCase = true))
assertTrue(response.body.contains("Thursday"))
assertFalse(response.meta.contains("stub"))
}
@Test
fun respond_emptyExtraction_skipsLlm() = runBlocking {
val fake = object : LocalLlm {
override val isReady = true
override val displayName = "fake-llm"
override suspend fun generate(prompt: String): String = error("should not be called")
override fun close() = Unit
}
val empty = sampleExtraction.copy(text = " ")
val response = AssistEngine(fake).respond(empty)
assertTrue(response.title.contains("No text", ignoreCase = true))
}
}
app/src/test/java/com/example/crkl/agent/LocalAssistStubTest.kt
+69
View File
@@ -0,0 +1,69 @@
package com.example.crkl.agent
import com.example.crkl.vision.RegionContentExtractor
import org.junit.Assert.assertTrue
import org.junit.Test
class LocalAssistStubTest {
@Test
fun respond_emptyExtraction_explainsNextStep() {
val result = RegionContentExtractor.ExtractionResult(
text = "",
nodeCount = 0,
bounds = android.graphics.RectF(0f, 0f, 10f, 10f),
packageName = "com.example.app"
)
val response = LocalAssistStub.respond(result)
assertTrue(response.title.contains("No text", ignoreCase = true))
assertTrue(response.body.contains("circle", ignoreCase = true))
assertTrue(response.body.contains("Lens", ignoreCase = true))
}
@Test
fun respond_withText_includesSummaryAndBody() {
val result = RegionContentExtractor.ExtractionResult(
text = "Hello world. Second sentence stays in the body.",
nodeCount = 3,
bounds = android.graphics.RectF(1f, 2f, 30f, 40f),
packageName = "com.android.chrome"
)
val response = LocalAssistStub.respond(result)
assertTrue(response.title.contains("Circled", ignoreCase = true))
assertTrue(response.body.contains("Hello world"))
assertTrue(response.meta.isEmpty())
}
@Test
fun respond_ocrJunk_treatedAsEmpty() {
val result = RegionContentExtractor.ExtractionResult(
text = "… · ·",
nodeCount = 0,
bounds = android.graphics.RectF(),
packageName = "com.example"
)
val response = LocalAssistStub.respond(result)
assertTrue(response.title.contains("No text", ignoreCase = true))
assertTrue(response.body.contains("Circle", ignoreCase = true) || response.body.contains("Lens", ignoreCase = true))
}
@Test
fun respond_mediaTranscript_extractsGroceryList() {
val result = RegionContentExtractor.ExtractionResult(
text = """
Media: Audio · Grocery
Kind: audio
Transcript:
Hi. Voice memo grocery list. We need milk, eggs, sourdough bread, olive oil, and two avocados. Also pick up oat milk and coffee beans.
""".trimIndent(),
nodeCount = 1,
bounds = android.graphics.RectF(),
packageName = "com.example.crkl"
)
val response = LocalAssistStub.respond(result, mode = AssistEngine.Mode.MEDIA)
assertTrue(response.title.contains("Audio", ignoreCase = true))
assertTrue(response.body.contains("Milk") || response.body.contains("milk"))
assertTrue(response.body.contains("Eggs") || response.body.contains("eggs"))
assertTrue(response.body.contains("transcript", ignoreCase = true))
}
}
app/src/test/java/com/example/crkl/agent/VoiceIntentTest.kt
+130
View File
@@ -0,0 +1,130 @@
package com.example.crkl.agent
import org.junit.Assert.assertEquals
import org.junit.Assert.assertFalse
import org.junit.Assert.assertTrue
import org.junit.Test
class VoiceIntentTest {
@Test
fun parse_emailMeTheList() {
assertEquals(VoiceIntent.Kind.EMAIL_LIST, VoiceIntent.parse("email me the list").kind)
}
@Test
fun parse_openTodos() {
assertEquals(VoiceIntent.Kind.OPEN_TODOS, VoiceIntent.parse("open my todos").kind)
}
@Test
fun parse_openMail() {
assertEquals(VoiceIntent.Kind.OPEN_MAIL, VoiceIntent.parse("open Gmail").kind)
assertEquals(VoiceIntent.Kind.OPEN_MAIL, VoiceIntent.parse("open mail inbox").kind)
}
@Test
fun parse_calendar() {
assertEquals(VoiceIntent.Kind.SHOW_CALENDAR, VoiceIntent.parse("what's on my calendar").kind)
assertEquals(VoiceIntent.Kind.SHOW_CALENDAR, VoiceIntent.parse("show calendar today").kind)
}
@Test
fun parse_sendMeTheList() {
val p = VoiceIntent.parse("send me the list")
assertEquals(VoiceIntent.Kind.SHARE_LIST, p.kind)
}
@Test
fun parse_addToTodo() {
val p = VoiceIntent.parse("add this to my todo")
assertEquals(VoiceIntent.Kind.ADD_TO_TODO, p.kind)
}
@Test
fun parse_addItemsToList() {
val p = VoiceIntent.parse("add bananas and yogurt to the list")
assertEquals(VoiceIntent.Kind.ADD_ITEMS, p.kind)
assertTrue(p.items.any { it.contains("Banana", ignoreCase = true) })
assertTrue(p.items.any { it.contains("Yogurt", ignoreCase = true) })
}
@Test
fun parse_showTodos() {
assertEquals(VoiceIntent.Kind.SHOW_TODOS, VoiceIntent.parse("show my todos").kind)
}
@Test
fun parse_summarizeYesterdayEmails() {
assertEquals(
VoiceIntent.Kind.SUMMARIZE_EMAILS,
VoiceIntent.parse("summarize my last 3 emails from yesterday").kind
)
}
@Test
fun extractListFromBody_bullets() {
val body = """
Local stub listened:
• Milk
• Eggs
• Olive oil
""".trimIndent()
val items = VoiceIntent.extractListFromBody(body)
assertEquals(listOf("Milk", "Eggs", "Olive oil"), items)
}
@Test
fun extractListFromBody_noHardcodedGroceryGuessing() {
val prose = "Remember milk and eggs from the store memo"
assertTrue(VoiceIntent.extractListFromBody(prose).isEmpty())
}
@Test
fun extractListFromBody_singleBullet() {
assertEquals(listOf("Only item"), VoiceIntent.extractListFromBody("• Only item"))
}
@Test
fun parse_translate() {
assertEquals(VoiceIntent.Kind.TRANSLATE, VoiceIntent.parse("translate this").kind)
val ru = VoiceIntent.parse("translate to russian")
assertEquals(VoiceIntent.Kind.TRANSLATE, ru.kind)
assertEquals("ru", ru.targetLang)
val he = VoiceIntent.parse("translate to hebrew")
assertEquals("he", he.targetLang)
}
@Test
fun parse_copy() {
assertEquals(VoiceIntent.Kind.COPY, VoiceIntent.parse("copy").kind)
assertEquals(VoiceIntent.Kind.COPY, VoiceIntent.parse("copy to clipboard").kind)
}
@Test
fun parse_explain() {
assertEquals(VoiceIntent.Kind.EXPLAIN, VoiceIntent.parse("explain this").kind)
assertEquals(VoiceIntent.Kind.EXPLAIN, VoiceIntent.parse("eli5").kind)
assertEquals(VoiceIntent.Kind.EXPLAIN, VoiceIntent.parse("what does this mean").kind)
}
@Test
fun parse_share_alone() {
assertEquals(VoiceIntent.Kind.SHARE_LIST, VoiceIntent.parse("share").kind)
assertEquals(VoiceIntent.Kind.SHARE_LIST, VoiceIntent.parse("share this").kind)
}
@Test
fun extractTextForTranslate_stripsChrome() {
val body = """
Short summary line
—
The meeting is postponed until Thursday afternoon.
—
Commands: speak …
""".trimIndent()
val text = VoiceIntent.extractTextForTranslate("Extracted text", body)
assertTrue(text.contains("postponed"))
assertFalse(text.contains("Commands:"))
}
}
app/src/test/java/com/example/crkl/fixtures/FixtureCatalogTest.kt
+82
View File
@@ -0,0 +1,82 @@
package com.example.crkl.fixtures
import org.junit.Assert.assertEquals
import org.junit.Assert.assertFalse
import org.junit.Assert.assertNotNull
import org.junit.Assert.assertTrue
import org.junit.Test
class FixtureCatalogTest {
@Test
fun catalog_coversRequestedContentKinds() {
val kinds = FixtureCatalog.all.map { it.kind }.toSet()
assertTrue(kinds.contains(FixtureCatalog.ContentKind.EMAIL))
assertTrue(kinds.contains(FixtureCatalog.ContentKind.IMAGE))
assertTrue(kinds.contains(FixtureCatalog.ContentKind.VIDEO))
assertTrue(kinds.contains(FixtureCatalog.ContentKind.AUDIO))
assertTrue(kinds.contains(FixtureCatalog.ContentKind.ARTICLE))
}
@Test
fun extractableNow_includesEmailAndDescribedImage() {
val ids = FixtureCatalog.extractableNow().map { it.id }.toSet()
assertTrue(ids.contains("email_meeting"))
assertTrue(ids.contains("article_paragraph"))
assertTrue(ids.contains("image_with_description"))
}
@Test
fun pixelsOnlyImage_markedNeedsOcr() {
val fixture = FixtureCatalog.byId("image_pixels_only")
assertEquals(FixtureCatalog.Expectation.NEEDS_OCR, fixture.expectation)
assertTrue(fixture.contentDescription.isNullOrBlank())
}
@Test
fun gmailInboxFixture_isExtractable() {
val inbox = FixtureCatalog.byId("mail_inbox_yesterday")
assertEquals(FixtureCatalog.Expectation.EXTRACTABLE_NOW, inbox.expectation)
assertTrue(inbox.body.contains("Yesterday"))
assertTrue(inbox.body.contains("jordan@example.com"))
assertFalse(
"howToTest must not mention deleted Fake Mail or gog bridge",
inbox.howToTest.contains("Fake Mail", ignoreCase = true) ||
inbox.howToTest.contains("gog", ignoreCase = true)
)
assertTrue(inbox.howToTest.contains("mailto"))
}
@Test
fun mediaFixtures_haveAssetsAndSummarizeExpectation() {
val video = FixtureCatalog.byId("video_walkthrough")
val audio = FixtureCatalog.byId("audio_grocery")
assertEquals(FixtureCatalog.Expectation.MEDIA_PLAY_AND_SUMMARIZE, video.expectation)
assertEquals(FixtureCatalog.Expectation.MEDIA_PLAY_AND_SUMMARIZE, audio.expectation)
assertNotNull(video.mediaAssetPath)
assertNotNull(audio.mediaAssetPath)
assertNotNull(video.transcriptAssetPath)
assertNotNull(audio.transcriptAssetPath)
assertEquals(2, FixtureCatalog.mediaFixtures().size)
}
@Test
fun demoScript_coversVipChipsInOrder() {
val steps = FixtureCatalog.demoScript()
assertEquals(4, steps.size)
assertEquals("translate_phrase", steps[0].fixtureId)
assertEquals("article_paragraph", steps[1].fixtureId)
assertEquals("email_meeting", steps[2].fixtureId)
assertEquals("grocery_list", steps[3].fixtureId)
steps.forEach { step ->
assertNotNull(FixtureCatalog.byId(step.fixtureId))
}
}
@Test
fun groceryList_hasBulletsForVikunja() {
val list = FixtureCatalog.byId("grocery_list")
assertTrue(list.body.contains("• Milk"))
assertTrue(list.howToTest.contains("Vikunja"))
}
}
app/src/test/java/com/example/crkl/integrations/DeviceCalendarTest.kt
+20
View File
@@ -0,0 +1,20 @@
package com.example.crkl.integrations
import org.junit.Assert.assertFalse
import org.junit.Assert.assertTrue
import org.junit.Test
import org.junit.runner.RunWith
import org.robolectric.RobolectricTestRunner
import org.robolectric.RuntimeEnvironment
@RunWith(RobolectricTestRunner::class)
class DeviceCalendarTest {
@Test
fun today_withoutPermission_returnsClearError() {
val r = DeviceCalendar.today(RuntimeEnvironment.getApplication())
assertFalse(r.ok)
assertTrue(r.message.contains("permission", ignoreCase = true))
assertTrue(r.events.isEmpty())
}
}
app/src/test/java/com/example/crkl/integrations/IntegrationSettingsTest.kt
+35
View File
@@ -0,0 +1,35 @@
package com.example.crkl.integrations
import com.example.crkl.testutil.MemorySharedPreferences
import org.junit.Assert.assertEquals
import org.junit.Assert.assertFalse
import org.junit.Assert.assertTrue
import org.junit.Test
class IntegrationSettingsTest {
@Test
fun defaults_includeCircleStyle() {
val s = IntegrationSettings(MemorySharedPreferences())
assertEquals(IntegrationSettings.DEFAULT_VIKUNJA_URL, s.vikunjaUrl)
assertEquals("cyan", s.circleColorKey)
assertTrue(s.circleNeon)
assertFalse(s.preferGogEmail)
assertTrue(s.statusSummary().contains("Circle: cyan"))
}
@Test
fun vikunjaReady_requiresToken() {
val s = IntegrationSettings(MemorySharedPreferences())
s.vikunjaToken = "tok"
assertTrue(s.vikunjaReady())
}
@Test
fun onboardingComplete_defaultsFalse() {
val s = IntegrationSettings(MemorySharedPreferences())
assertFalse(s.onboardingComplete)
s.onboardingComplete = true
assertTrue(s.onboardingComplete)
}
}
app/src/test/java/com/example/crkl/media/ActiveMediaRegistryTest.kt
+69
View File
@@ -0,0 +1,69 @@
package com.example.crkl.media
import org.junit.Assert.assertEquals
import org.junit.Assert.assertNull
import org.junit.Assert.assertTrue
import org.junit.Before
import org.junit.Test
class ActiveMediaRegistryTest {
@Before
fun clear() {
ActiveMediaRegistry.clear()
}
@Test
fun findOverlapping_returnsBestOverlap() {
ActiveMediaRegistry.register(
ActiveMediaRegistry.Slot(
id = "audio_grocery",
kind = ActiveMediaRegistry.Kind.AUDIO,
assetPath = "media/grocery_memo.wav",
transcriptAssetPath = "media/grocery_memo.transcript.txt",
title = "Audio · Grocery",
screenBounds = ActiveMediaRegistry.Bounds(100f, 1000f, 900f, 1400f)
)
)
ActiveMediaRegistry.register(
ActiveMediaRegistry.Slot(
id = "video_walkthrough",
kind = ActiveMediaRegistry.Kind.VIDEO,
assetPath = "media/product_walkthrough.mp4",
transcriptAssetPath = "media/product_walkthrough.transcript.txt",
title = "Video · Walkthrough",
screenBounds = ActiveMediaRegistry.Bounds(100f, 200f, 900f, 700f)
)
)
val hit = ActiveMediaRegistry.findOverlapping(
ActiveMediaRegistry.Bounds(120f, 1050f, 880f, 1350f)
)
assertEquals("audio_grocery", hit?.id)
assertNull(
ActiveMediaRegistry.findOverlapping(
ActiveMediaRegistry.Bounds(0f, 0f, 10f, 10f)
)
)
}
@Test
fun updateBounds_mutatesSlot() {
ActiveMediaRegistry.register(
ActiveMediaRegistry.Slot(
id = "x",
kind = ActiveMediaRegistry.Kind.AUDIO,
assetPath = "a",
transcriptAssetPath = "t",
title = "t"
)
)
ActiveMediaRegistry.updateBounds(
"x",
ActiveMediaRegistry.Bounds(1f, 2f, 300f, 400f)
)
val slot = ActiveMediaRegistry.all().single()
assertTrue(slot.screenBounds.right - slot.screenBounds.left > 200f)
}
}
app/src/test/java/com/example/crkl/model/ModelPathsTest.kt
+29
View File
@@ -0,0 +1,29 @@
package com.example.crkl.model
import org.junit.Assert.assertEquals
import org.junit.Assert.assertTrue
import org.junit.Test
import java.io.File
class ModelPathsTest {
@Test
fun findModel_prefersAdbPathWhenPresent() {
val tmp = createTempDir(prefix = "crkl-model-test")
try {
// Simulate filesDir with no app models; we only assert candidate ordering.
val candidates = ModelPaths.candidateFiles(tmp)
assertEquals(ModelPaths.ADB_MODEL_PATH, candidates.first().absolutePath)
assertTrue(candidates.any { it.name == "gemma-3-1b-it-int4.task" })
} finally {
tmp.deleteRecursively()
}
}
@Test
fun missingModelHint_mentionsMakeTarget() {
val hint = ModelPaths.missingModelHint()
assertTrue(hint.contains("push-model"))
assertTrue(hint.contains(ModelPaths.ADB_MODEL_PATH))
}
}
app/src/test/java/com/example/crkl/testutil/MemorySharedPreferences.kt
+99
View File
@@ -0,0 +1,99 @@
package com.example.crkl.testutil
import android.content.SharedPreferences
/** Minimal in-memory [SharedPreferences] for JVM unit tests (no Robolectric). */
class MemorySharedPreferences : SharedPreferences {
private val map = mutableMapOf<String, Any?>()
override fun getAll(): MutableMap<String, *> = map.toMutableMap()
override fun getString(key: String?, defValue: String?): String? =
map[key] as? String ?: defValue
override fun getStringSet(key: String?, defValues: MutableSet<String>?): MutableSet<String>? {
@Suppress("UNCHECKED_CAST")
return (map[key] as? Set<String>)?.toMutableSet() ?: defValues
}
override fun getInt(key: String?, defValue: Int): Int = map[key] as? Int ?: defValue
override fun getLong(key: String?, defValue: Long): Long = map[key] as? Long ?: defValue
override fun getFloat(key: String?, defValue: Float): Float = map[key] as? Float ?: defValue
override fun getBoolean(key: String?, defValue: Boolean): Boolean =
map[key] as? Boolean ?: defValue
override fun contains(key: String?): Boolean = map.containsKey(key)
override fun edit(): SharedPreferences.Editor = Editor()
override fun registerOnSharedPreferenceChangeListener(
listener: SharedPreferences.OnSharedPreferenceChangeListener?
) = Unit
override fun unregisterOnSharedPreferenceChangeListener(
listener: SharedPreferences.OnSharedPreferenceChangeListener?
) = Unit
private inner class Editor : SharedPreferences.Editor {
private val pending = mutableMapOf<String, Any?>()
private val removals = mutableSetOf<String>()
private var clearAll = false
override fun putString(key: String?, value: String?): SharedPreferences.Editor {
pending[key!!] = value
return this
}
override fun putStringSet(key: String?, values: MutableSet<String>?): SharedPreferences.Editor {
pending[key!!] = values
return this
}
override fun putInt(key: String?, value: Int): SharedPreferences.Editor {
pending[key!!] = value
return this
}
override fun putLong(key: String?, value: Long): SharedPreferences.Editor {
pending[key!!] = value
return this
}
override fun putFloat(key: String?, value: Float): SharedPreferences.Editor {
pending[key!!] = value
return this
}
override fun putBoolean(key: String?, value: Boolean): SharedPreferences.Editor {
pending[key!!] = value
return this
}
override fun remove(key: String?): SharedPreferences.Editor {
removals.add(key!!)
return this
}
override fun clear(): SharedPreferences.Editor {
clearAll = true
return this
}
override fun commit(): Boolean {
apply()
return true
}
override fun apply() {
if (clearAll) map.clear()
removals.forEach { map.remove(it) }
map.putAll(pending)
pending.clear()
removals.clear()
clearAll = false
}
}
}
app/src/test/java/com/example/crkl/vision/ContentCaptureTest.kt
+48
View File
@@ -0,0 +1,48 @@
package com.example.crkl.vision
import android.graphics.RectF
import org.junit.Assert.assertEquals
import org.junit.Assert.assertFalse
import org.junit.Assert.assertTrue
import org.junit.Test
class ContentCaptureTest {
private val base = ContentCapture.CaptureResult(
text = "Hello",
nodeCount = 1,
bounds = RectF(0f, 0f, 10f, 10f),
packageName = "com.example",
source = ContentCapture.Source.A11Y
)
@Test
fun merge_prefersOcrWhenA11yEmpty() {
val empty = base.copy(text = "", source = ContentCapture.Source.EMPTY)
val merged = ContentCapture.mergeWithOcr(empty, "From OCR", 2)
assertEquals(ContentCapture.Source.OCR, merged.source)
assertEquals("From OCR", merged.text)
assertEquals(2, merged.ocrBlockCount)
}
@Test
fun merge_combinesDistinctTexts() {
val merged = ContentCapture.mergeWithOcr(base, "World", 1)
assertEquals(ContentCapture.Source.MERGED, merged.source)
assertTrue(merged.text.contains("Hello"))
assertTrue(merged.text.contains("World"))
}
@Test
fun looksLikeMedia_detectsAudioMarkers() {
assertTrue(ContentCapture.looksLikeMedia("♫ Voice memo", null))
assertTrue(ContentCapture.looksLikeMedia("play", "com.example.videoplayer"))
}
@Test
fun hasUsefulText_requiresLetters() {
assertTrue(ContentCapture.hasUsefulText("Hello world"))
assertFalse(ContentCapture.hasUsefulText("… · ·"))
assertFalse(ContentCapture.hasUsefulText("123"))
}
}
docs/DEMO.md
+33
View File
@@ -0,0 +1,33 @@
# DEMO — 45–60 second recording
Product: **Circle** (repo `crkl`) — circle on-screen text → Translate · Copy · Explain · Share · Vikunja.
## Prep (once)
1. Install APK → open **Circle** → enable **Circle Overlay** (Accessibility)
2. Integrations → Vikunja token (for Vikunja steps)
3. Optional: **Add Quick Settings tile**
4. Open **Test fixtures**
## Take (script)
| Time | Action |
|------|--------|
| 0:00 | Show fixtures. Tap floating **C** (or QS **Circle**). |
| 0:05 | **Translate** fixture — closed loop → **Translate**. |
| 0:15 | **Article** — circle paragraph → **Copy**, then **Explain**. |
| 0:30 | **Email** — circle Subject `Q2 planning moved` → **Vikunja**. |
| 0:45 | **List** — circle bullets → **Share** or **Vikunja**. |
| 0:55 | Dismiss. End on chips or confirmation. |
Voiceover: “Circle any text. Act without leaving the screen.”
## Do not show
- Debug meta, gog, model install hints
- Image / Lens paths
- Failed mailto on emu without a mail app
## Source of truth
`FixtureCatalog.demoScript()` — also printed atop Test fixtures.
docs/README.md
+15
View File
@@ -0,0 +1,15 @@
# Circle docs
Product name: **Circle**. Repo / package code name: **crkl** (`com.example.crkl`).
| Doc | Purpose |
|-----|---------|
| [DEMO.md](DEMO.md) | 45s recording script |
| [dogfood.md](dogfood.md) | Daily 10-minute checklist |
| [non-goals.md](non-goals.md) | Ship freeze — what we are not building |
| [marketing.md](marketing.md) | Blurb + 3 social posts |
| [shortcuts.md](shortcuts.md) | QS tile, Accessibility button, volume keys |
| [RELEASE.md](RELEASE.md) | Install, tag, headless emu tips |
| [brand/README.md](brand/README.md) | Logo concepts (shipped: **#2 lasso**) |
Root also has [README.md](../README.md), [CHANGELOG.md](../CHANGELOG.md), [PHONE_SETUP.md](../PHONE_SETUP.md).
docs/RELEASE.md
+36
View File
@@ -0,0 +1,36 @@
# Release checklist — Circle 1.16.2
## Before tag
- [x] Unit tests green (`./gradlew testDebugUnitTest`)
- [x] Logo 2 (lasso) as launcher + QS
- [x] Display name Circle / Circle Overlay
- [x] DEMO, dogfood, non-goals, marketing, shortcuts, brand docs
- [x] On-device install + Circle Overlay enabled (`make smoke` on CrklEmulator)
- [ ] Dogfood checklist once (Translate / Explain / Share / Vikunja / real app)
- [ ] Record 45s demo
## Tag (when ready)
```bash
cd ~/Documents/code/crkl
git status
git tag -a v1.16.2-circle -m "Circle: lasso brand, VIP chips, shortcuts, ship docs"
git push origin main
git push origin v1.16.2-circle
```
## Install
```bash
./gradlew assembleDebug
adb install -r app/build/outputs/apk/debug/app-debug.apk
adb shell am start -n com.example.crkl/.MainActivity
```
Re-enable **Circle Overlay** after every reinstall.
## Emulator tips (macOS)
Prefer `make emulator` (launchd) — agent/Cursor sandboxes often kill nohup qemu.
Then: `make wait-emulator && make install && make smoke`
docs/brand/README.md
+12
View File
@@ -0,0 +1,12 @@
# Brand / logo concepts
**Shipped:** concept **2** — paper field, teal hand-drawn lasso, ink **C**
(`ic_launcher_foreground` + QS tile rasters from `circle-logo-2-lasso-c.png`).
| File | Idea | Status |
|------|------|--------|
| `circle-logo-1-ink-ring.png` | Ink field, white **C**, teal ring | Alternate |
| `circle-logo-2-lasso-c.png` | Paper + teal lasso + ink **C** | **In app** |
| `circle-logo-3-split-c.png` | Split ink / teal, bold white **C** | Alternate |
Display name: **Circle** (repo/package still `crkl`).
docs/brand/circle-logo-1-ink-ring.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 872 KiB

docs/brand/circle-logo-2-lasso-c.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.4 MiB

docs/brand/circle-logo-3-split-c.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 860 KiB

docs/demo-shots/crkl-audio-summary.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 196 KiB

docs/demo-shots/crkl-email-result.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 304 KiB

docs/demo-shots/crkl-ocr-result.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 188 KiB

docs/demo-shots/crkl-video-result.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 226 KiB

docs/demo-shots/workflows/01-email.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 300 KiB

Some files were not shown because too many files have changed in this diff Show More