Activity

Summary

Add a secure Introspection API that allows the dashboard to access Laravel application logs, execute whitelisted artisan commands, and view container resource stats. This enables features that Coolify's API cannot provide.

---

Background & Research

What Coolify's API Does NOT Provide

1. Laravel Application Logs

The Coolify API endpoint GET /applications/{uuid}/logs returns container stdout/stderr (nginx access logs, PHP-FPM output), NOT Laravel's storage/logs/laravel.log. Example of what it returns:

``` 127.0.0.1 - [15/Jan/2026:09:01:26 +0000] "GET /up HTTP/1.1" 200 ```

This is useless for debugging Laravel application issues.

2. Artisan Command Execution

The execute command endpoint `POST /applications/{uuid}/execute` was commented out in the Coolify codebase:

"The `/execute` endpoint needs to be rethought. So it will stay in commented status until stable."

Related issue: #4312

There is no API method to run artisan commands remotely.

3. Container-Specific Resource Stats

Coolify's Sentinel feature provides server-wide metrics (total server CPU/memory), not per-application stats. It's designed for server monitoring, not single-app introspection.

The Solution: Self-Contained Introspection

The Laravel application can introspect itself without external dependencies:

Verified working - tested on production container:

``` === Memory === Used: 229 MB Limit: unlimited

=== CPU === usage_usec 31831176 user_usec 22285073 system_usec 9546103

=== Disk === 21G used / 75G total (29%) ```

---

Proposed Architecture

Route Registration

Register routes under a configurable prefix (default: /coolify/introspect). Routes should:

• Only be registered when introspection is enabled in config
• Use a dedicated middleware stack for authentication
• Be excluded from standard web middleware (CSRF, etc.)

Endpoints

Authentication Model

This must be locked down tight. Exposing artisan execution is dangerous if not properly secured.

Layer 1: Feature Toggle

• Disabled by default
• Requires explicit opt-in via config and environment variable

Layer 2: Dedicated Token

• Separate from Coolify API token
• Random 64-character string
• Stored in environment, not committed to repo

Layer 3: HMAC Request Signing

• Every request must include:
  • X-Coolify-Timestamp: Unix timestamp
  • X-Coolify-Signature: HMAC-SHA256 of timestamp + method + path + body
• Reject if timestamp > 60 seconds old (prevents replay attacks)
• Reject if signature doesn't match

Layer 4: Command Whitelist

• Only explicitly allowed artisan commands can be executed
• Default whitelist should be conservative (read-only commands)
• User can extend in config

Layer 5: Rate Limiting

• Aggressive rate limits, especially on /artisan
• Suggested: 60/min for reads, 10/min for artisan

Layer 6: Audit Logging

• Every introspection call logged to dedicated file
• Include: timestamp, endpoint, IP, command (if artisan), result

Configuration

```php // config/coolify.php 'introspection' => [ 'enabled' => env('COOLIFY_INTROSPECTION_ENABLED', false), 'token' => env('COOLIFY_INTROSPECTION_TOKEN'), 'prefix' => env('COOLIFY_INTROSPECTION_PREFIX', 'coolify/introspect'),

'allowed_commands' => [
    // Read-only / safe commands
    'about',
    'env',
    'route:list',
    'migrate:status',
    'queue:monitor',
    'schedule:list',
    'horizon:status',
    
    // Cache operations
    'cache:clear',
    'config:cache',
    'config:clear',
    'route:cache',
    'route:clear',
    'view:cache',
    'view:clear',
    'event:cache',
    'event:clear',
    
    // Queue operations
    'queue:retry',
    'queue:flush',
    'queue:restart',
    'horizon:pause',
    'horizon:continue',
    'horizon:terminate',
],

'rate_limits' => [
    'stats' => 60,      // per minute
    'logs' => 60,
    'health' => 60,
    'artisan' => 10,
],

'logs' => [
    'path' => storage_path('logs'),
    'allowed_extensions' => ['log'],
    'max_file_size' => 10 * 1024 * 1024, // 10MB - truncate if larger
    'default_lines' => 100,
    'max_lines' => 1000,
],

], ```

---

Implementation Details

Stats Collection

CPU Usage Calculation

The cgroup `cpu.stat` file provides cumulative microseconds. To get percentage:

1. Read `usage_usec` at time T1
2. Wait interval (or store previous reading)
3. Read `usage_usec` at time T2
4. Calculate: `(T2 - T1) / (interval_usec * num_cpus) * 100`

For a simpler approach, just return the raw values and let the frontend calculate delta between polls.

Memory Usage

• `/sys/fs/cgroup/memory.current` - bytes currently used
• `/sys/fs/cgroup/memory.max` - limit (may be "max" meaning unlimited)
• Calculate percentage if limit is set

Disk Usage

• `disk_free_space('/')` - bytes free on root
• `disk_total_space('/')` - total bytes
• Focus on root partition; optionally check `storage_path()` specifically

Log Reading

Security Considerations

• Validate filename contains no path traversal (`..`, absolute paths)
• Only allow files within configured log directory
• Only allow configured extensions (prevent reading `.env`, etc.)
• Truncate extremely large files
• Consider stripping sensitive data patterns (API keys, passwords)

Pagination

• Read from end of file (newest first)
• Return line count, file size, and whether truncated
• Support `offset` and `limit` parameters

Log Entry Parsing

Laravel logs have a standard format. Consider parsing into structured data:

• Timestamp
• Level (ERROR, WARNING, INFO, DEBUG)
• Message
• Stack trace (if present)

Artisan Execution

Security Considerations

• NEVER allow arbitrary command strings
• Whitelist approach: command must be in allowed list
• Validate arguments against allowed patterns where applicable
• Some commands accept dangerous arguments (e.g., `--force`)
• Consider argument whitelisting for sensitive commands

Execution

• Use `Artisan::call()` not shell execution
• Capture output via `Artisan::output()`
• Set reasonable timeout
• Return exit code, output, and execution time

Dangerous Commands to NEVER Allow

• `tinker` - arbitrary PHP execution
• `db:*` - database manipulation
• `migrate` (without `--status`) - schema changes
• `key:generate` - breaks encryption
• `storage:link` - filesystem changes
• `vendor:publish` - filesystem changes
• Any command accepting `--path` or file arguments

Health Checks

Aggregate health information from Laravel internals:

Queue Health

• Jobs in queue (by queue name)
• Failed jobs count
• Horizon status (if installed)

Cache Health

• Cache driver status
• Connection test (get/set)

Database Health

• Connection status
• Connection count (if available)
• Slow query count (if logging enabled)

Storage Health

• Disk space on storage partition
• Log directory size

---

Dashboard Integration

The dashboard should call these endpoints to power:

1. Logs Panel - Real-time log viewer with level filtering and search
2. Console Panel - Dropdown of allowed commands with execute button
3. Stats Widget - Live container CPU/memory/disk gauges
4. Health Widget - Queue depth, cache status, DB status indicators

Polling Strategy

• Stats: Poll every 5-10 seconds
• Health: Poll every 30 seconds
• Logs: Poll every 5 seconds OR use SSE/WebSocket for real-time

---

Files to Create/Modify

New Files

• `src/Http/Controllers/IntrospectionController.php` - Main controller
• `src/Http/Middleware/AuthenticateIntrospection.php` - HMAC auth middleware
• `src/Services/ContainerStats.php` - Cgroup reading logic
• `src/Services/LogReader.php` - Log file reading with security
• `src/Services/ArtisanRunner.php` - Command execution with whitelist

Modified Files

• `config/coolify.php` - Add introspection config section
• `routes/web.php` - Register introspection routes (conditionally)
• `src/CoolifyServiceProvider.php` - Register middleware, conditionally load routes

---

Security Checklist

Before shipping, verify:

• Introspection disabled by default
• Token required and validated
• HMAC signature validated with timing-safe comparison
• Timestamp expiry prevents replay attacks
• Artisan commands strictly whitelisted
• Log file access restricted to configured directory
• Path traversal impossible in log file names
• Rate limiting enforced
• All access logged to audit file
• No sensitive data leaked in responses
• Error messages don't reveal system details

---

References

• Coolify execute endpoint disabled: https://github.com/coollabsio/coolify/issues/4544
• Coolify Sentinel (server-wide, not per-app): https://coolify.io/docs/knowledge-base/server/sentinel
• cgroups v2 documentation: https://docs.kernel.org/admin-guide/cgroup-v2.html
• Laravel Artisan programmatic usage: https://laravel.com/docs/artisan#programmatically-executing-commands

---

Questions to Resolve

1. Should we support Server-Sent Events (SSE) for real-time log streaming, or is polling sufficient?
2. Should the introspection routes be on a separate port/domain for additional isolation?
3. Should we add IP whitelisting as an optional additional security layer?
4. How should we handle log files larger than the configured max size - truncate from start or end?