Run and Customize Local LLMs with Cortex

Cortex (https://cortex.so/docs) is an open-source framework designed to serve as standalone API server or as the "brain" for robots, encompassing capabilities in vision, speech, language, tabular data, and action processing.

Notably, it offers the flexibility to operate locally so independently of cloud services, making it suitable for various light deployment environments.

Key Features:

• User-Friendly Command-Line Interface (CLI): Inspired by Ollama, Cortex provides a straightforward CLI, simplifying interaction and management.

• Comprehensive C++ Implementation: The framework is fully implemented in C++ (https://github.com/janhq/cortex.cpp), allowing it to be packaged into both desktop and mobile applications, enhancing portability and performance.

• Versatile Model Integration: Users can pull models from multiple sources, including Hugging Face (gguf format) and Cortex's Built-in Model Library, ensuring flexibility in model selection.

  cortex pull [model_id] cortex pull mistral

Note: it's recommended to have more than 8 GB of RAM available to run 3B models, 16 GB for 7B models, and 32 GB for 14B models.

• Deployment Flexibility: It can function as a standalone API server or be integrated into applications such as Jan.ai, providing adaptability in various use cases.

  cortex start cortex stop

If GPU hardware is available, Cortex leverages GPU acceleration by default, enhancing performance for demanding tasks.

Conclusion:

In summary, Cortex offers a robust and flexible solution for integrating advanced cognitive capabilities into robotic systems, with an emphasis on openness, performance, and adaptability.