micke/amity
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Lets commit to the new version #1

Merged
micke merged 44 commits from amity-ng into main 2026-01-12 09:49:02 +01:00
Conversation 0 Commits 44 Files changed 442 +50533 -61680
micke commented 2026-01-12 09:48:28 +01:00
Owner
Copy link
No description provided. 
Major architectural overhaul introducing a plugin-based system with
Lua scripting support. This is a complete rewrite of the codebase.

Key changes:
- Add plugin system with Lua scripting (mlua)
- Add amity-sign tool for plugin signing and verification
- Switch from sqlx to SeaORM for database abstraction
- Switch from Askama to Tera for runtime templates
- Add WebAuthn passwordless authentication
- Upgrade MLS to openmls 0.7 for E2EE chat
- Add ActivityPub message ID tracking for federated chat
- Add theming support via plugins

Infrastructure:
- Remove Docker examples and test infrastructure
- Add debian packaging for v2.0.0
- Update rust-version requirement to 1.82+

Breaking changes:
- Database schema incompatible with v0.1.x
- Configuration format updated for plugin paths
- Template syntax changed (Askama → Tera)
Rename the plugin signing tool and add scaffolding functionality:

- Rename binary from amity-sign to amity-plugin
- Add 'init' subcommand to create plugin directory structure
- Move signing commands under 'sign' subcommand group
- Split debian packaging into two binary packages:
  - amity: main server
  - amity-plugin: standalone plugin development tool

New command structure:
  amity-plugin init <name>      - Create plugin scaffold
  amity-plugin sign keygen      - Generate signing keypair
  amity-plugin sign archive     - Sign plugin archive
  amity-plugin sign verify      - Verify signature
  amity-plugin sign show-public - Show public key
- Update README with release build instructions and expanded API docs
- Add .luarc.json and types/ for Lua Language Server support
- Theming plugin: namespace KV keys, serve CSS via API instead of files
- Plugin scaffolding: generate LSP config and type definitions
Authentication & User Management:
- Implement password change with wrapping key re-encryption
- Add password reset flow using vault password (multi-step UI)
- Create personal MLS group for new users on registration
- Admin password reset returns 403 with explanation (by design)

Plugin System:
- Wire up schedule callback execution via channel-based executor
- Fix WebSocket broadcast with shared registry and route handler
- Add WebSocket route at /ws/{*path} for plugin websockets

OCM Federation:
- Add shares table migration for OCM share tokens
- Implement share entity with SHA-256 token hashing
- Add OCM share token authentication for WebDAV (Basic/Bearer)
- Add Share variant to SFTP AuthResult for consistency

Other:
- Update TODO.md with known issues tracking
- Minor theming plugin updates
Per the OCM IETF-RFC specification, SSH/SCP protocol uses public key
authentication rather than token-based auth. Remote servers provide
their SSH public key when accepting a share, which is used to
authenticate subsequent SFTP connections.

Changes:
- Add migration for ssh_public_key and ssh_key_fingerprint columns
- Update share entity with SSH key fields and helper methods
- Implement try_share_auth() in SFTP handler for key-based auth
- Add path scoping checks to all SFTP file operations
- Enforce read/write permissions based on share permissions

Security:
- Path scoping prevents access outside share boundaries
- Write operations require share write permission
- Expired shares are rejected during authentication
Security improvements based on schema audit:
- Hash session tokens with SHA-256 (prevent hijacking if DB compromised)
- Add session binding: IP address, user agent, last activity tracking
- Enforce plugin KV encryption (three-tier: user/group/server, no plaintext)
- Add login_attempts table for brute-force protection
- Add audit_logs table for security event tracking
- Add soft deletes for users (compliance/audit trail)
- Add CHECK constraints for enum columns
- Add encrypted settings support (is_sensitive, value_encrypted)
- Fix SQL injection in signing key migration (parameterized queries)
- Add missing indexes on expires_at columns

SQLite compatibility: migrations use table recreation for schema changes
that SQLite doesn't support (DROP COLUMN, MODIFY COLUMN).
Replace 24 incremental migrations with a single comprehensive migration
that creates all 27 tables with the complete security-hardened schema.

Changes:
- Single migration: m20260101_000001_initial_schema.rs
- Includes all security features: hashed tokens, encryption tiers,
  soft deletes, audit logging, login attempts tracking
- Seeds core signing key for plugin verification
- Add ConnectInfo extension for session IP binding

This simplifies fresh deployments and removes SQLite compatibility
issues with incremental ALTER TABLE operations.

BREAKING: Existing databases must be recreated (no production deployments yet)
Update ARCHITECTURE.md section 5 with:
- UML-style ASCII diagram of all 27 tables
- Column types, primary/foreign keys, indexes
- Encryption indicators (🔐) for encrypted columns
- Encryption details table with methods and key sources
- Plugin KV three-tier encryption explanation
- Security features summary
The core signing key is now seeded in the database migration and
embedded as a constant in src/plugin/core_plugins.rs. The separate
public key files are no longer needed.

Private keys were never tracked by git and have been moved to
~/.config/amity/keys/ for plugin signing operations.
- Add derive_share_key() using HKDF-SHA256 for key derivation
- Store encrypted group key on share records (encrypted_group_key column)
- Update WebDAV auth to decrypt group key from bearer token (sharedSecret)
- Update SFTP auth to use SSH username as sharedSecret for key derivation
- Add ShareService for share lifecycle management
- Add Lua plugin API for share creation and management (amity.share.*)
- Remove silent fallback in encrypted storage - operations now fail
  loudly when encryption keys are unavailable

Security: Server cannot decrypt share data without the sharedSecret,
enabling true end-to-end encryption for shared content.
Extends the plugin API to match more Nextcloud capabilities:
- User Management API (amity.users) with capability-based access control
- Rate limiting support in HTTP routes
- Profile actions and sections for plugin UI integration
- GroupMembers server capability for listing group members

Adds automatic Lua type generation:
- build.rs generates types/amity.lua on every build
- generate-lua-types binary for standalone generation
- amity-plugin update-types command to refresh plugin type definitions
Replaces hardcoded type generation with syn-based parsing of
src/plugin/api/schema.rs. The schema file contains:
- Struct definitions that become Lua @class types
- Trait definitions that define the Lua module APIs

build.rs now:
- Parses schema.rs with syn
- Converts Rust types to Lua type annotations
- Generates types/amity.lua automatically on every build

Also adds:
- Makefile for release builds and Debian packaging
- debian/rules now reuses pre-built binaries when available
- Document all entity structs, enums, fields, and relations in db/entity/
- Document error types and variants in error.rs
- Document web forms and fields in admin.rs and routes.rs
- Document plugin API types in manifest.rs, http.rs, navigation.rs, schedule.rs
- Document service types in notify/service.rs and storage/service.rs
- Fix lifetime elision warning in mls/service.rs by adding explicit '_
- Fix unused mut warning in plugin/registry.rs
- Add KV group-tier functions: get_group, set_group, del_group, list_group
- Add log.query() function with database storage for plugin logs
- Implement MLS async invitation system:
  - invite_member() creates pending invitation with encrypted group key
  - accept_invitation() allows invitee to join using server-mediated key transfer
  - pending_invitations() lists invitations for current user
- Add group_invitations table migration and entity
- Update schema.rs with all API function signatures
- Sync ARCHITECTURE.md with actual implementation
- Regenerate types/amity.lua with complete API definitions
- Fix nav API: use admin_menu() instead of non-existent tab()
- Fix time API: use now_iso() instead of non-existent iso8601()
- Replace amity.export() with plugin methods (not in API)
- Add json/form fields to HttpRequest schema
- Add text field to HttpResponse schema
- Import Amity Classic Light/Dark teal color schemes
- Add login page gradient background
Add plugin packaging to build.rs so plugins are automatically rebuilt
when source files change. This ensures the plugin archives stay in sync
with the source code, fixing issues where old CSS/templates would persist
after changes.

- Add build dependencies: tar, flate2, ed25519-dalek, base64, rand
- Package plugins from plugins/ directories into .tar.gz archives
- Sign archives using key from ~/.config/amity/keys/*.private
- Only rebuild when source files are newer than the archive
Add signature 🤝 branding to navigation and login pages to match main
branch style. Update color palette from teal to blue theme.
Refactor all plugin API modules to use *ApiMethods traits as the single
source of truth for both runtime registration and type generation:

- Add macros.rs with traits for all API modules (Log, Storage, Kv, Http,
  Template, Nav, Notify, Events, Schedule, Time, Users, Capability, Mls,
  Share, WebSocket, HttpSig, Protocol, Imports)
- Refactor each v1 module to implement its trait via helper struct pattern
- Update build.rs to parse traits and generate types/amity.lua
- Add ImportsApiMethods for amity.export/import (direct table methods)
- Add plugin lifecycle documentation (init, enable, disable, unload)

This three-way binding ensures:
- Compiler enforces implementation completeness (missing method = error)
- types/amity.lua always matches actual implementation
- LSP documentation reflects runtime behavior

The only static content in types/amity.lua is the plugin lifecycle, as
these are methods plugins must implement, not amity API methods.
- Fix amity.http.register -> amity.http.route
- Fix amity.time.now_iso -> amity.time.iso8601
- Remove local types/amity.lua copy
- Add .luarc.json pointing to repo root types folder for LSP support
- Create js-api plugin with amity.js providing http, ws, notify, dialog,
  events, storage, user, plugin, and autocomplete modules
- Add emoji picker and @mention autocomplete (ported from main branch)
- Add /api/emojis endpoint serving bundled Unicode emoji data
- Compile TypeScript to minified JS during cargo build using swc
- Update amity-plugin CLI:
  - Add `build` command for TypeScript compilation and archive creation
  - Add TypeScript support to `init` (plugin.ts, tsconfig.json, types/)
  - Update `update-types` to refresh both Lua and TypeScript definitions
  - Auto-discover signing keys in ~/.config/amity/keys/ with multi-key prompt
Plugin System:
- Fix runtime panic in imports.rs by converting block_on to async functions
- Add Lua module system with amity.require() for loading modules from lib/
- Add plugin dependency validation with semver constraint support
- Add emoji filter for navigation icons using shortcode-to-unicode mapping

Theming Plugin (CSS):
- Add toolbar, breadcrumbs, and sortable table header components
- Add context menu, dropzone, and progress bar styles
- Add spinner and skeleton loader animations
- Add table checkbox styling with row selection

JS API Plugin:
- Add amity.upload with progress callback support
- Add amity.contextMenu for showing/hiding context menus
- Add amity.shortcuts for keyboard shortcut registration
- Add amity.clipboard for copy/read operations
- Fix build.rs to include ModuleApiMethods in type generation, so
  amity.require and amity.list now appear in types/amity.lua
- Add prominent warnings in macros.rs about mandatory trait system usage
- Add warning in v1/mod.rs pointing developers to macros.rs

The trait system ensures types/amity.lua stays in sync with the actual
implementation. Bypassing it breaks plugin developer autocomplete/docs.
- Fix plugin hot reload by unloading from registry before update/delete
- Add grant/revoke routes for server capabilities
- Update admin plugins page with capability management UI
- Show capability descriptions and danger levels
- Parse manifest capabilities and match against database grants
- Add StorageAdminApiMethods trait with create/update/delete/test/get/list_backends
- Implement storage admin API in storage.rs with admin permission checks
- Export StorageType from storage module
- Add new capabilities: storage:admin, kv:system, navigation:admin_menu
- Add user:read as alias for user:info capability
- Improve plugin template rendering with global Tera registration
- Add default navigation section redirect for authenticated users
- Fix plugin routes to use allocated storage backend
The custom tostring function was returning 'unknown' for Error values,
making it impossible to debug errors from async functions. Now properly
converts Error to string using its Display implementation.
- Plugins now load disabled by default, require capability grants to enable
- Admin UI shows pending capabilities with clickable badge to open grant dialog
- Added "Grant All" button for bulk capability granting
- Fixed capability matching to use unique JSON per capability type
- Added user capability consent flow with redirect to consent page
- Users can view and revoke plugin permissions from profile page
- Fixed plugin route dispatcher to extract body, query, headers, and JSON
- Plugin context now updated per-request with current user's admin status
- Added plugin_consent.html template for user capability consent
- Add scope_path_async to resolve user:N to actual usernames for storage paths
- Extract wrapping key from session for transparent encryption in plugin routes
- Auto-create storage root directories for filesystem backends
- Make mkdir create parent directories recursively
- Return options["root"] as root_path for fs backends in list/get APIs
- Add list_with_metadata support for file listings with full metadata
- Filter out current directory from list_with_metadata results
- Add multipart/form-data parsing to plugin routes
- Pass form fields (with filename, content_type, data) to Lua handlers
OpenDAL requires trailing slash to list directory contents, otherwise
it returns only the directory entry itself. Added trailing slash in
the list function and improved logging for debugging storage operations.
- Add trailing slash to directory paths for OpenDAL listing
- Fix response body handling to use raw bytes (mlua::String)
- Support custom headers in plugin responses
- Skip Nil values when checking for JSON response field
- Add multipart form parsing for file uploads
- Add extensive debug logging for storage operations
- Add delete_recursive to StorageBackend for deleting non-empty directories
- Update Lua storage.delete to use recursive delete
- Allow plugins to specify nav_section and nav_tab in template context
- Fix file size display by calling stat() for each file in listings
Cross-plugin function calls:
- Rewrite import/export to route calls through source plugin's sandbox
- Serialize/deserialize arguments between Lua states
- Add PluginsMap shared reference for cross-plugin access

Tera template string filters:
- str_first: Get first character
- str_last: Get last character
- char_at(index): Character at index
- substring(start, end): Substring extraction

Plugin lifecycle:
- Add post_init hook called after ALL plugins are loaded
- Safe to use cross-plugin imports in post_init
- Document in types/amity.lua for LSP support
Previously post_init only ran at server startup via call_post_init_all().
Now it also runs when a plugin is enabled later via enable_plugin(),
ensuring cross-plugin initialization works regardless of when a plugin
is enabled.
HTTP Signing Key Management:
- Support multiple signing keys with fingerprints and algorithms
- Add key lifecycle: generate_key, list_keys, get_key, set_primary,
  revoke_key, delete_key
- Auto-generate RSA key on first access if none exists
- Export keys as JWKS for .well-known/jwks.json
- Support JWA algorithms: RS256, RS384, RS512, EdDSA

Server URL API (amity.server):
- get_url() returns configured external URL for generating links
- get_info() returns full server info (url, bind_address, port, instance_id)
- Configurable via AMITY_EXTERNAL_URL or [server] external_url in config
- Add optional_dependencies field to Manifest for non-required plugins
- Add optional capabilities section for individually grantable permissions
- Block plugin enable if required dependencies are not enabled
- Require all-or-nothing grants for required capabilities (use Grant All)
- Allow individual grant/revoke only for optional capabilities
- Show dependency status (installed/enabled) in plugin info dialog
- Update admin UI to distinguish required vs optional capabilities
- Add amity.http.parse_form() to manually parse URL-encoded form data
- Automatically set req.form when Content-Type is application/x-www-form-urlencoded
- Handle array fields (name[]) by collecting values into Lua arrays
- Proper URL decoding including %XX hex sequences and + as space
- Add unit tests for url_decode and parse_form_data
- Plugin installation wizard: inline modal with AJAX upload, capability
  listing, and quick-setup endpoint combining grant-all + enable
- Dynamic log level control: persisted to database with runtime handle,
  UI in admin logs page
- Error pages: proper error.html template instead of raw Jinja2 output
  when plugins fail to render
- CSS components: empty-state, spinner, loading-state, skeleton loading
  placeholders, alert color variables, button loading states, improved
  mobile navigation
- JS utils module (amity.utils): uuid, base64, path utilities, debounce,
  throttle, formatBytes
- Lua utils module (amity.utils): uuid, base64, path utilities, mime_type
Includes Dockerfile using debian:trixie-slim with official deb packages,
docker-compose.yml with separate volumes for database and file storage,
and a make docker target for building the image.
- Fix scheduled callbacks to use call_async() instead of call(), enabling
  async operations (HTTP, storage, etc.) in scheduled jobs
- Replace fragile string-based capability error detection with proper
  error type downcasting using CapabilityPendingApprovalError
- Walk error source chain to find capability errors wrapped by mlua
Adds amity.utils.url_encode(str) for percent-encoding strings per RFC 3986.
Useful for constructing URLs with query parameters in federation plugins
(WebFinger, ActivityPub, OAuth redirects, etc.).
- Add Public Server URL setting to admin UI for federation configuration
- Make amity.server.get_url() read from database setting with config fallback
- Enables ActivityPub and other federation plugins to use correct keyId URLs
- Calculate ETag from content hash for all static file endpoints
- Handle If-None-Match header to return 304 Not Modified when appropriate
- Change Cache-Control from immutable to must-revalidate with 1 hour max-age
- Applies to plugin static files, theme files, and custom theme CSS
micke merged commit 55b9493bd7 into main 2026-01-12 09:49:02 +01:00
micke deleted branch amity-ng 2026-01-12 09:49:02 +01:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: micke/amity#1
No description provided.
Delete branch "amity-ng"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?