Merge pull request #73 from any4ai/dev

chore(deps): update Next.js and related dependencies to version 15.3.6 in pnpm-lock.yaml and package.json for improved stability and performance

Merge pull request #72 from any4ai/dev

fix(Base): update minimatch import to named import for consistency and clarity in module usage

✨ feat(crawl): introduce `scrape_paths` for selective content extraction, enhancing cost efficiency by allowing users to specify which URLs to scrape while still discovering links from included paths. Update documentation and API specifications accordingly.

Merge pull request #71 from any4ai/dev

✨ feat(preNav): enhance pre-navigation data capture with improved logging, error handling, and timeout management, ensuring better visibility and reliability in data processing.

✨ feat(Job): add Unicode sanitization method to prevent PostgreSQL JSONB errors and update job result insertion to use sanitized data.

✨ feat(template): enhance caching mechanism by allowing cache TTL configuration from environment variables, with support for disabling cache, improving flexibility in template handling.

✨ feat (template): implement comprehensive template handling with access control, processing logic for scrape, crawl, and search operations, and enhanced variable mapping and validation, improving overall template functionality and error management.

✨ feat: add support for enum type in template variable validation, enhancing error handling for invalid enum values and updating template configuration to include allowed values and options.

✨ feat: enhance template handling by adding URL transformation support, improving query transformation logging, and updating documentation for new options in scrape and search schemas.

Merge pull request #70 from any4ai/dev

chore(deps): refresh pnpm-lock.yaml to match workspace (remove axios from search)

✨ feat: enhance search functionality by adding support for SearXNG and AC Engine, updating search options, and improving error handling in search execution.