Commit graph

10766 commits

Author SHA1 Message Date
dawnsystem
52f08daa00 Update BITACORA_MAESTRA.md to correct duplicate timestamps and log recent project review session. Enhance AI scanner confidence thresholds in ai_scanner.py, improve model loading safety in model_cache.py, and refine security checks in security.py. Update numpy dependency in pyproject.toml. Remove unused styles and clean up component code in the UI. Implement proper cleanup in Angular components to prevent memory leaks. 2025-11-15 23:59:08 +01:00
dawnsystem
1a572b6db6
Merge pull request #51 from dawnsystem/copilot/add-ai-settings-page
Add AI configuration settings page with comprehensive controls
2025-11-15 18:10:06 +01:00
dawnsystem
7f615d578e
Merge branch 'dev' into copilot/add-ai-settings-page 2025-11-15 18:05:36 +01:00
copilot-swe-agent[bot]
d91e4a2051 feat(settings): add AI configuration settings page
- Add AI settings keys to ui-settings.ts with proper defaults
- Create AiSettingsComponent with full functionality
- Add AI tab to main settings component
- Implement toggles for AI scanner, ML features, and advanced OCR
- Add sliders for auto-apply and suggest thresholds
- Add ML model selector dropdown
- Add test button for AI sample document
- Add AI performance statistics display
- Integrate AI settings into main settings form and save logic
- Add comprehensive tests for AI settings component

Co-authored-by: dawnsystem <42047891+dawnsystem@users.noreply.github.com>
2025-11-15 16:15:38 +00:00
dawnsystem
1bec62ccb9
Merge pull request #50 from dawnsystem/copilot/add-ai-status-indicator
[WIP] Add AI status indicator to UI navbar
2025-11-15 17:03:29 +01:00
copilot-swe-agent[bot]
09979d0156 feat: implement AI Status Indicator in navbar
- Created AIStatus data model with TypeScript interfaces
- Implemented AIStatusService with polling mechanism
- Created AIStatusIndicatorComponent with icon, tooltip, and animations
- Added component to navbar between notifications and user menu
- Includes unit tests for service and component
- Shows AI active/inactive state with visual indicators
- Displays statistics: documents scanned, suggestions applied
- Shows pending deletion requests with badge
- Processing animation when AI is active
- Link to AI configuration settings
- Non-intrusive design following existing patterns

Co-authored-by: dawnsystem <42047891+dawnsystem@users.noreply.github.com>
2025-11-15 15:55:59 +00:00
dawnsystem
77f1593d0a
Merge pull request #53 from dawnsystem/copilot/add-deletion-requests-dashboard
Add UI for deletion requests management
2025-11-15 16:42:56 +01:00
dawnsystem
b975db6f39
Merge branch 'dev' into copilot/add-deletion-requests-dashboard 2025-11-15 16:42:36 +01:00
copilot-swe-agent[bot]
37faf607df docs: Update BITACORA_MAESTRA.md with deletion UI implementation
Co-authored-by: dawnsystem <42047891+dawnsystem@users.noreply.github.com>
2025-11-15 15:36:24 +00:00
copilot-swe-agent[bot]
5edfbfc028 feat: Complete deletion requests management UI implementation
- Backend API:
  - Added DeletionRequestSerializer and DeletionRequestActionSerializer
  - Added DeletionRequestViewSet with approve/reject/pending_count actions
  - Added /api/deletion_requests/ route

- Frontend:
  - Created deletion-request data model
  - Created deletion-request.service.ts with full CRUD operations
  - Created DeletionRequestsComponent with status filtering (pending/approved/rejected/completed)
  - Created DeletionRequestDetailComponent with impact analysis display
  - Added /deletion-requests route with permissions guard
  - Implemented approve/reject modals with confirmation
  - Added status badges and pending request counter

- All code linted and built successfully

Co-authored-by: dawnsystem <42047891+dawnsystem@users.noreply.github.com>
2025-11-15 15:33:45 +00:00
dawnsystem
0dc1061f50
Merge pull request #48 from dawnsystem/copilot/add-ai-suggestions-panel
Add AI suggestions panel to document detail page
2025-11-15 16:14:34 +01:00
copilot-swe-agent[bot]
0fd8706337 feat: Complete AI Suggestions Panel integration with document detail component
- Added AI Suggestions Panel component with TypeScript, HTML, SCSS, and tests
- Integrated panel into document detail component
- Added conversion logic from DocumentSuggestions to AISuggestion format
- Implemented apply/reject handlers for suggestions
- Added @angular/animations package and configured animations
- Added missing Bootstrap icons (magic, clock, chevron-down/up, etc.)
- Added visual confidence indicators (high/medium/low with colors)
- Implemented responsive design for mobile and desktop
- Added animations for apply/reject actions
- Component shows suggestions grouped by type (tags, correspondent, document type, storage path, date)
- All builds and lints pass successfully

Co-authored-by: dawnsystem <42047891+dawnsystem@users.noreply.github.com>
2025-11-14 18:08:47 +00:00
dawnsystem
8cf5f1bd1e
Merge pull request #45 from dawnsystem/copilot/implement-ml-model-cache
Implement efficient caching for ML models
2025-11-14 17:31:28 +01:00
dawnsystem
5b6c54d077
Merge branch 'dev' into copilot/implement-ml-model-cache 2025-11-14 17:31:16 +01:00
copilot-swe-agent[bot]
4f654e4e56 docs: Update BITACORA_MAESTRA with ML cache implementation details
Co-authored-by: dawnsystem <42047891+dawnsystem@users.noreply.github.com>
2025-11-14 16:10:20 +00:00
copilot-swe-agent[bot]
f1161ce5fb feat(ml): Complete ML model caching implementation with settings and startup integration
Co-authored-by: dawnsystem <42047891+dawnsystem@users.noreply.github.com>
2025-11-14 16:08:41 +00:00
dawnsystem
dfe4fe6655
Merge pull request #44 from dawnsystem/copilot/add-ai-scanner-command
Add management command to scan existing documents with AI
2025-11-14 17:04:04 +01:00
copilot-swe-agent[bot]
3ab970c3e7 style: Add trailing commas per ruff linting rules
Co-authored-by: dawnsystem <42047891+dawnsystem@users.noreply.github.com>
2025-11-14 15:54:26 +00:00
copilot-swe-agent[bot]
879a65ed9f docs: Add documentation for scan_documents_ai command
Added comprehensive documentation in administration.md including:
- Command syntax and all available options
- Usage examples for different scenarios
- Explanation of confidence levels
- Performance considerations for large document sets

Co-authored-by: dawnsystem <42047891+dawnsystem@users.noreply.github.com>
2025-11-14 15:53:06 +00:00
copilot-swe-agent[bot]
920998f6b5 fix: Fix linting issues in scan_documents_ai command
- Fixed blank line whitespace issues
- Changed logger.error with exc_info=True to logger.exception
- Prefixed unused unpacked variables with underscore
- Applied ruff formatting to both command and test files

Co-authored-by: dawnsystem <42047891+dawnsystem@users.noreply.github.com>
2025-11-14 15:51:14 +00:00
dawnsystem
451851ca3d
Merge pull request #70 from dawnsystem/revert-54-copilot/add-webhook-system-ai-events
Revert "Implement webhook system for AI events notifications"
2025-11-14 16:40:43 +01:00
dawnsystem
04ced421b8
Revert "Implement webhook system for AI events notifications" 2025-11-14 16:40:31 +01:00
dawnsystem
f00424f7c4
Merge pull request #54 from dawnsystem/copilot/add-webhook-system-ai-events
Implement webhook system for AI events notifications
2025-11-14 16:37:04 +01:00
dawnsystem
21db1cc6d6
Merge branch 'dev' into copilot/add-webhook-system-ai-events 2025-11-14 16:30:15 +01:00
copilot-swe-agent[bot]
5ae18e03b5 Merge branch 'dev' into copilot/add-webhook-system-ai-events
Resolved merge conflicts in:
- src/documents/ai_deletion_manager.py: Kept webhook integration alongside dev changes
- src/documents/ai_scanner.py: Kept webhook integration and applied_fields tracking
- src/documents/models.py: Integrated AISuggestionFeedback model with webhook imports

All conflicts resolved maintaining both webhook functionality and new AI suggestions features from dev branch.

Co-authored-by: dawnsystem <42047891+dawnsystem@users.noreply.github.com>
2025-11-14 15:26:54 +00:00
copilot-swe-agent[bot]
ebc906b713 chore(webhooks): Remove unused imports (time, timedelta)
Co-authored-by: dawnsystem <42047891+dawnsystem@users.noreply.github.com>
2025-11-14 15:17:05 +00:00
copilot-swe-agent[bot]
7b8c45c44f feat(webhooks): Complete webhook system implementation with admin and docs
- Add AIWebhookEvent and AIWebhookConfig models to Django admin
- Import webhook models in documents/models.py for Django recognition
- Create comprehensive AI_WEBHOOKS_DOCUMENTATION.md with:
  - Setup and configuration guide
  - Event payload examples
  - Security and HMAC signature verification
  - Integration examples (Slack, Discord, custom endpoints)
  - Troubleshooting guide
  - API reference
- Admin interface for webhook management and monitoring
- All webhook events are tracked and logged

Co-authored-by: dawnsystem <42047891+dawnsystem@users.noreply.github.com>
2025-11-14 15:15:17 +00:00
dawnsystem
f76a546bea
Merge pull request #43 from dawnsystem/copilot/add-ai-suggestions-endpoints
Add API endpoints for AI suggestions with tracking and statistics
2025-11-13 19:08:36 +01:00
copilot-swe-agent[bot]
e9d64e6aac refactor: Apply code review improvements to AI suggestions API
- Remove unused imports (Document from serializers, override_settings from tests)
- Add explanatory comments to all empty except clauses
- Create SUGGESTION_TYPE_CHOICES constant and SuggestionSerializerMixin for DRY
- Implement type-specific validation (value_id for ID types, value_text for text types)
- Remove redundant Document.DoesNotExist handler (already handled by DRF)
- Optimize ai_suggestion_stats query (21 queries → 3 queries using aggregation)
- Update documentation to clarify unsupported custom_field/workflow in apply endpoint

Addresses all 14 code review comments from Copilot PR reviewer.

Co-authored-by: dawnsystem <42047891+dawnsystem@users.noreply.github.com>
2025-11-13 08:22:57 +00:00
copilot-swe-agent[bot]
075f58734a security: Fix stack trace exposure vulnerabilities in AI suggestions API
- Remove internal error details from API responses
- Log full error details server-side only
- Provide user-friendly error messages
- Fixed 5 CodeQL security alerts (py/stack-trace-exposure)

Co-authored-by: dawnsystem <42047891+dawnsystem@users.noreply.github.com>
2025-11-13 06:12:47 +00:00
copilot-swe-agent[bot]
c7a129cbea docs: Add comprehensive API documentation for AI Suggestions endpoints
Co-authored-by: dawnsystem <42047891+dawnsystem@users.noreply.github.com>
2025-11-13 06:09:30 +00:00
copilot-swe-agent[bot]
9542c32fb8 feat(ai): Add AI Suggestions API endpoints with tracking and statistics
- Created serializers for AI suggestions (AISuggestionsSerializer, ApplySuggestionSerializer, etc.)
- Added AISuggestionFeedback model for tracking applied/rejected suggestions
- Implemented 4 new API endpoints:
  - GET /api/documents/{id}/ai-suggestions/ - Get AI suggestions for a document
  - POST /api/documents/{id}/apply-suggestion/ - Apply a suggestion and record feedback
  - POST /api/documents/{id}/reject-suggestion/ - Reject a suggestion and record feedback
  - GET /api/documents/ai-suggestion-stats/ - Get accuracy statistics
- Created database migration for AISuggestionFeedback model
- Added comprehensive test suite (28 test cases) covering all endpoints
- Supports tracking for tags, correspondents, document types, storage paths, titles
- Calculates accuracy rates and confidence scores for AI improvement

Co-authored-by: dawnsystem <42047891+dawnsystem@users.noreply.github.com>
2025-11-13 06:07:11 +00:00
dawnsystem
9cb2064460
Merge pull request #52 from dawnsystem/copilot/add-deletion-request-endpoints
Add API endpoints for deletion request approval workflow
2025-11-13 07:01:00 +01:00
copilot-swe-agent[bot]
b8d554631c fix: Remove duplicate deletion approval code and fix formatting
- Removed duplicate DeletionApprovalView from views.py
- Removed duplicate DeletionApprovalSerializer from serialisers.py
- Removed duplicate URL route /api/ai/deletions/approve/
- Fixed missing blank line in serialisers.py before AISuggestionsRequestSerializer
- Kept DeletionRequestViewSet as the correct implementation

Co-authored-by: dawnsystem <42047891+dawnsystem@users.noreply.github.com>
2025-11-13 05:59:16 +00:00
dawnsystem
d9a4edf9c5
Merge branch 'dev' into copilot/add-deletion-request-endpoints 2025-11-13 06:51:58 +01:00
copilot-swe-agent[bot]
dd3536d3ff docs: Update BITACORA_MAESTRA with API endpoints implementation details
Co-authored-by: dawnsystem <42047891+dawnsystem@users.noreply.github.com>
2025-11-13 05:46:32 +00:00
dawnsystem
86060730e5
Merge pull request #69 from dawnsystem/copilot/optimize-database-indexes
feat(db): Add performance indexes for DeletionRequest queries
2025-11-13 06:40:12 +01:00
dawnsystem
00f8f2c5d8
Merge pull request #66 from dawnsystem/copilot/create-django-migration-deletion-request
Create Django migration for DeletionRequest model
2025-11-13 02:04:07 +01:00
copilot-swe-agent[bot]
3ae23cddfa docs: add migration documentation for DeletionRequest
Created comprehensive documentation for migration 1076 including:
- Overview and migration details
- Complete field descriptions
- How to apply and rollback
- Testing instructions
- Performance and security considerations

Co-authored-by: dawnsystem <42047891+dawnsystem@users.noreply.github.com>
2025-11-13 00:52:30 +00:00
copilot-swe-agent[bot]
ad2df8c7ff feat(migrations): add Django migration for DeletionRequest model
Created migration 1076_add_deletion_request.py to add the DeletionRequest model
to the database schema. This model tracks AI-initiated deletion requests that
require explicit user approval before any documents are deleted.

Migration includes:
- All model fields (created_at, updated_at, requested_by_ai, ai_reason, status, etc.)
- ManyToMany relationship with Document model
- Foreign keys to User model (user and reviewed_by)
- Custom indexes for common query patterns (status+user, created_at)
- Proper ordering and metadata

Fixes #10

Co-authored-by: dawnsystem <42047891+dawnsystem@users.noreply.github.com>
2025-11-13 00:51:41 +00:00
dawnsystem
487f6ff75d
Merge pull request #41 from dawnsystem/copilot/add-integration-tests-ai-scanner
Add integration tests for AI Scanner consumer pipeline
2025-11-13 01:40:56 +01:00
dawnsystem
9c1cd2638d
Merge pull request #42 from dawnsystem/copilot/add-granular-ai-permissions
Add granular permissions for AI features
2025-11-13 01:38:53 +01:00
copilot-swe-agent[bot]
7658a5571b Address code review feedback: fix imports, docstrings, and add comments
- Move all imports to module level (removed from inside methods)
- Add missing `status` import from rest_framework
- Fix docstring formatting to comply with PEP 257
- Add explanatory comments to empty except clauses
- Improve error message for document not found
- Add warning comment about thread-safety in config update
- Add TODO comments for storing approval/rejection reasons
- Remove unused StoragePath import from tests
- Remove duplicate permission imports at end of file

Co-authored-by: dawnsystem <42047891+dawnsystem@users.noreply.github.com>
2025-11-12 17:24:43 +00:00
copilot-swe-agent[bot]
7bb516a6a5 refactor: Address code review feedback for AI Scanner integration tests
- Add PAPERLESS_ENABLE_AI_SCANNER check in consumer._run_ai_scanner() method
- Move AIScanResult import to top of test file to follow DRY principle
- Create helper methods: create_empty_scan_result_mock() and get_test_file_with_name()
- Reduce code duplication by using helper methods across all tests
- Improve performance test threshold from 10s to 1s for mocked operations
- Add TODO comment for proper performance testing with real ML models
- Remove duplicate 'from documents.ai_scanner import AIScanResult' imports from 7 test methods

Co-authored-by: dawnsystem <42047891+dawnsystem@users.noreply.github.com>
2025-11-12 16:23:11 +00:00
copilot-swe-agent[bot]
cd5c7afdcd Changes before error encountered
Co-authored-by: dawnsystem <42047891+dawnsystem@users.noreply.github.com>
2025-11-12 15:42:23 +00:00
copilot-swe-agent[bot]
b16394fdbb feat(db): Add performance indexes for DeletionRequest model
- Add composite index (user, status, created_at) for listing queries
- Add index for reviewed_at field for filtering reviewed requests
- Add index for completed_at field for filtering completed requests
- Create migration 1076_add_deletionrequest_performance_indexes
- Expected improvement: list queries <100ms, filter queries <50ms

Addresses: [AI Scanner] Índices de Performance para DeletionRequest
Epic: Migraciones de Base de Datos

Co-authored-by: dawnsystem <42047891+dawnsystem@users.noreply.github.com>
2025-11-12 15:42:00 +00:00
copilot-swe-agent[bot]
cc9e66c11c Changes before error encountered
Co-authored-by: dawnsystem <42047891+dawnsystem@users.noreply.github.com>
2025-11-12 15:39:22 +00:00
copilot-swe-agent[bot]
d159277707 Changes before error encountered
Co-authored-by: dawnsystem <42047891+dawnsystem@users.noreply.github.com>
2025-11-12 15:38:37 +00:00
copilot-swe-agent[bot]
894f7e231d Changes before error encountered
Co-authored-by: dawnsystem <42047891+dawnsystem@users.noreply.github.com>
2025-11-12 15:38:07 +00:00
copilot-swe-agent[bot]
5695d41903 Changes before error encountered
Co-authored-by: dawnsystem <42047891+dawnsystem@users.noreply.github.com>
2025-11-12 15:37:44 +00:00