Large Action Models Guide: AI That Gets Things Done

Generative artificial intelligence has stirred up quite a storm. This breakthrough technology has become the focus of intense scrutiny, application, and further research, accounting for a fair share of both groundbreaking innovations and downright laughable gimmicks. Just a few months ago, courtesy of the Rabbit R1, the world was introduced to the concept of LAM — a large action model — a previously unused term, but too promising not to be examined more closely.

But as we’ve seen, not all tech wonders are what they’re cracked up to be. Is there a real value hidden in large action models? Instinctools’ very own AI gurus have done the groundwork for you.

What is a large action model?

Large action models (LAMs) are a type of AI designed to translate human intent into action (potentially) autonomously. LAMs aspire to be platform-agnostic, general-purpose, action-oriented agents capable of performing tasks across any website or service. 

A LAM adds an advanced twist to eminent large language models. Unlike LLMs, large action models move beyond natural language understanding and generation by adding another core element into the equation — action. Amplified by advanced multi-step logical reasoning, LAMs can execute complex, interconnected actions, balancing both textual and external, interactive contexts.

Techwise, large action models build on neural models like LLMs, but the neuro-symbolic programming core of LAMs also integrates the strengths of symbolic artificial intelligence, a technology known for empowering intelligent systems with human-like reasoning. An open-source large action model can also pair logic programming with computer vision and language models to enhance reasoning and planning.

As for now, LAM-based solutions, such as the aforementioned Rabbit large action model, still require pretty thorough prompt engineering to perform an action as intended. This means that the promise of LAMs is still to be fulfilled but already represents a core milestone that can transform the way we approach and interact with AI.

— Pavel Klapatsiuk, AI Lead Engineer, *instinctools

Let’s peel back the layers: key capabilities of LAMs

While LLMs are limited to text processing, LAM’s domain of expertise is much wider thanks to the combination of the conventional AI interpretability and adaptive capabilities of cutting-edge machine learning.

Ability to handle complex decision-making tasks solo

In LAM solutions, the combination of reinforcement learning from human feedback (RHFL) and neuro-symbolic AI parlays into more effective planning and reasoning, enabling LAMs to execute tasks of an abstract nature. 

Akin to us, humans, LAMs can factor in different variables, weigh options, and determine the best course of action.

For example, in customer services, a LAM can process a return or handle complex customer queries. Besides, large action models can analyze past interactions and tap into contextual understanding to automate complex tasks more effectively.

More importantly, LAM solutions can automate nuanced tasks more quickly, easily, and with fewer resources. With conventional AI systems, companies require extensive coding efforts to break down a use case into a set of rules and steps and then integrate it into the existing systems. LAMs can potentially make this process less of a tall order by using natural language to encode workflows.

Integration with third-party systems and IoT devices

Large action models are also poised to interact with third-party systems, including databases, various applications, and IoT devices, to analyze massive datasets, perform actions on your behalf, remotely control devices, and execute other tasks previously confined to human hands. For example, a LAM system can access third-party apps to make reservations, process financial transactions, get stock market information, and more.

Let’s see how this rising superpower plays out in the Rabbit large action model. Once a user logs into apps on rabbithole (secure cloud hub), the LAM can navigate the apps on the user’s behalf and handle digital errands. However, as Rabbit is essentially a stand-alone device, it doesn’t interact with the apps on your phone. Instead, it has custom versions of specific apps in the cloud. 

Real-time task execution and adaptation

Thanks to the procedural memory inherent in them, large action models can pick up new skills through repetitive training and perform automated functions faster and with precision and accuracy. In some sense, this capability is similar to human cognition when babies learn to perform actions. However, LAM’s procedural memory is shackled by its underlying architecture and training data. 

Along with procedural memory, LAMs also incorporate the ability to keep a mental note of users’ requirements and preferences. In simple terms, built-in personalized memory allows a LAM-based system to memorize the user’s preferred commute route or frequently scheduled meetings. 

For example, the company behind Rabbit is planning on launching the teach mode — a capability that allows users to show the system how to do specific, non-trivial tasks on niche apps and workflows. 

Doesn’t that ring a bell? Well, it should, as it’s essentially AI agents, rebranded. 

Unlike ChatGPT or Gemini, LAMs are trained on demonstrations and actions to predict what action to take based on a request. While this action-focused nature makes LAMs a whole new breed compared to LLMs, the difference between AI agents and LAMs is not so distinct.

— Pavel Klapatsiuk, AI Lead Engineer, *instinctools

What can LAMs do that gen AI agents can’t?

Names, especially when well-chosen, can dramatically influence the visibility, understanding, and adoption of products and ideas. Remember when the concept of the cloud spearheaded by Amazon and Google changed the game? The thing is that “the cloud” existed before the term was coined and was known as the less catchy “remote storage” and “distributed computing”. But once the concept changed the PR manager, it became much easier to understand and more appealing to both businesses and consumers.

The same goes for large action models. 

If you look closely at their functions and capabilities, you’ll see that they follow in the footsteps of gen AI agents and multi-agents. The latter perform the same functions but just happen to have a less flashy signboard.

Don’t just take our word for it — take Microsoft’s. Here’s what tricks AI assistants have up their sleeve, according to the tech giant:

• Thanks to the integration of LLMs, AI agents can plan and sequence actions to achieve specific goals. 
• Leverage various tools, including code execution, search, and computation capabilities through function calling to improve the effectiveness of task execution. 
• Perceive the environment through sensors such as cameras and microphones, analyzing the visual, auditory, and other sensory input to process environment data. 
• Along with memorizing behaviors, AI assistants can also remember past interactions associated with tool usage and perception to inform future actions and continuously improve over time. 

Overall, LAM systems can be thought of as a more advanced subset of AI agents that are specifically cut out for action and interaction with the real world — as opposed to simple reflex agents.

How LAMs move from words to action in six steps

Similar to an AI robotics system, LAMs go by the hierarchical approach to action representation and execution. To perform tasks, large action models decompose complex actions into smaller, more manageable sub-actions. The latter can then be reused in different contexts, supercharging the flexibility and planning capability of LAMs.

Processing multimodal input

Large action models are activated by user input, which serves as the starting point for their operations. LAMs can process a variety of input, including text, images, and potentially user interactions. The ability to analyze multimodal data promotes a more natural and intuitive user experience while also broadening the scope of tasks LAMs can perform.

For example, Rabbit R1 integrates a Perplexity-based answer engine to analyze text input without any knowledge cutoff. 

Decoding human intention

Once user input enters the LAM’s bloodstream, the system infers the meaning behind it, using a combination of advanced techs, such as symbolic AI and neural networks. Large action models analyze the whole spectrum of cues, such as language, past behavior, external context, and other signals to determine the underlying human intentions behind the input. 

Interpreting user interface

To execute complex tasks and effectively interact with interfaces, large action models need to analyze what they see on screen. That’s why a LAM gets a thorough understanding of buttons, fields, and images in application interfaces to accurately identify the purpose and functionality of UI elements within a given application. After that, the system can seamlessly interact with the appropriate element based on what it has learned.

Decomposing the task and performing action sequencing

Once assigned to action oriented tasks, a large action model first breaks them down into steps, creating a hierarchical structure. Symbolic reasoning allows the system to model actions and determine an optimal sequence of actions that will get the model from point A to point B. 

Based on the analysis of the input and the identified tasks, the LAM generates precise prompts — augmented by data on prior experiences and codified domain knowledge — that guide the subsequent actions and allow the system to draw upon.

Executing an action

On its final leg, a LAM can execute actions either independently or by connecting to external systems and tools such as web automation frameworks. Large action models can use APIs to communicate with third-party systems — for example, they can access a weather API to analyze the current weather conditions. But most importantly,  some LAMs can also send commands to devices, while others can interact with web applications by simulating user actions, such as clicking buttons, filling out forms, and navigating between pages.

Analyzing the results and learning from feedback

Comparable to other AI-based solutions, large action models AI are lifelong learners, always evolving and responding to feedback. Thanks to reinforcement learning, LAMs can create an iterative learning loop that improves by simulating actions, evaluating their outcomes, and adjusting future behavior accordingly.

Also, large action models allow for human oversight that helps drift the model in the right direction and improve their performance over time by injecting feedback into LAMs.

The inner mechanics of LAMs take after those of AI agent systems. However, in agent systems, there is more of a hierarchical structure, where subagents have specific roles, and a manager subagent assigns and coordinates tasks, whereas LAMs typically handle decomposition and planning within a more unified framework.

— Pavel Klapatsiuk, AI Lead Engineer, *instinctools

LAM use cases: beyond the hype and straight to the actual value

While it might seem like yet another shifting of goalposts, replacing ‘language’ with ‘action’ is one of those cases where one word changes everything — including the area of application. Large action models pick up where LLMs left off, representing an evolution from generative to actionable AI technology — a tech boon lots of industries have been ripe for.

Healthcare

The sheer volume of admin tasks, patients’ and admissions make the healthcare industry clamor for automation — a demand that previous-generation AI was able to partially satisfy. 

Large action models can further mend some of the mounting problems faced by healthcare providers, in accordance with applicable regulations, care models, reimbursement approaches, and specific organizational blueprints.

Task execution in EHR processes, documentation, and scheduling is one of those areas where LAM systems can take more clerical tasks off the providers’ shoulders. LAMs can handle dynamic scheduling adjustments based on changing circumstances, factoring in patient preferences, doctor availability, and facility resources.

A LAM-enabled agent can also check on elderly patients outside healthcare facilities, assisting them with minor health issues and booking appointments with healthcare professionals, if necessary.

Large action models can also support clinical decisions by providing personalized treatment plans based on the interplay of different factors, including specific treatment guidelines, patient data, and patient preferences. Unlike conversational AI, LAMs do not require specialized integrations to access healthcare systems such as electronic health records or health information exchanges, processing data in real time and facilitating more efficient decision-making. 

Finance

Over 65% of investors regret their investment decisions. A highly personalized LAM-based support system can prevent those costly investment mistakes by providing tailored investment recommendations based on an investor’s financial situation, risk tolerances, goals, and market data. It can then bring these recommendations into action, i.e. by making trades or transferring funds on behalf of the investor.

For banks and financial institutions, an action-based system bodes well for enhancing customer service. When human agent resources are stretched too thin, LAMs can engage in complex voice interactions to provide immediate support and offer recommendations based on user preferences and prior interactions. 

Loan underwriting is another process that can benefit from the implementation of LAM solutions. To create a credit memo, relationship managers and credit analysts have to sift through 15+ sources on the borrower, loan type, and other factors, — and then, after a few more sweats and back-and-forths, write the document. 

Large action models can relieve managers and analysts of extensive data analysis, enhancing productivity and reducing the time spent on credit-risk memo generation. Leveraging actionable AI, a human user can outline the overall workflow, including specific rules, standards, and conditions, through natural language. The ecosystem of AI agents takes it from there by handling the communication with the borrower, gathering documents, calculating financial ratios, and executing the rest of the leg work.

Supply chain management

The current challenges in supply chain management create a breeding ground for innovation, a task LAMs are up to. As SCM systems usually comprise a whole variety of software, including ERP, WMS, TMS, IoT applications, and others, automation solutions require a whole lot of integrations to access and analyze consolidated real-time data. 

Conversely, actionable AI systems have no problem integrating with industrial control systems and IoT devices. They can execute actions directly, such as collecting data from sensors or triggering maintenance alerts. Here are potential areas for LAM application in supply chains:

• Predictive maintenance — large action models can accumulate data from sensors and other resources to predict equipment failures and send maintenance alerts.
• Quality control — using the combination of computer vision, sensor data, machine learning, and reference data, LAMs can flag quality issues and perform immediate corrective actions.
• Inventory optimization — not only can LAM systems take over complex data analysis tasks, such as recognizing patterns and anomalies in demand data, but they can autonomously respond to changes in demand or supply by adjusting inventory levels, placing orders, and managing transportation. 
• Industrial robots — LAMs can transform human robot interaction, enabling automated systems to understand human intentions and work safely alongside humans.

Along with these real world scenarios, action-focused AI solutions can improve virtually all logistics processes, from route optimization to transportation resource management and vehicle safety systems. For example, actionable AI systems can dynamically adjust routes based on real-time traffic conditions and TMS data. They can then identify the most optimal mode of transportation according to the analyzed data and assign routes to each vehicle based on factors such as vehicle capacity, location, and driver availability.

Literally any enterprise

There is not a single incumbent that wouldn’t benefit from strategic planning capabilities brought into the fold by LAMs. Large action models delve deeper than any other analytics solution, closing the gap between enhanced decision-making and subsequent action.

Let’s have a look at feasible large action model examples that can flip the script in enterprises:

• Customer experience — LAM-enabled chatbots can automate many routine customer service tasks, providing targeted support in real time. By identifying possible equipment failures or customer concerns before they happen, LAMs can automatically initiate tasks like notifying the maintenance crew or placing orders for replacement parts.
• Fraud detection — actionable AI systems can detect fraudulent activity in large datasets of transaction data and automatically implement safeguarding measures in case of emergency.
• Process automation — LAMs can do the heavy lifting of time-consuming tasks, including automated data entry, payment processing, financial analysis, contract management, and document review.
• IT support — action-oriented systems can act as tech co-pilots, solving troubleshooting technical issues and providing necessary user support. 
• Compliance management — large action models can streamline routine compliance tasks, such as generating reports, conducting audits — and even updating records.

The main concern with giving AI power to act still to be addressed

As action-oriented models step ouside the standard, the associated data security and compliance risks go beyond the ones we used to consider as well. Although the integration of neuro-symbolic reasoning grants LAMs more transparency compared to other AI spin-offs, it doesn’t make them immune to errors and biases that can creep into the systems as a result of insufficient prompting, inaccurate data quality, or unforeseen circumstances they were not trained to handle.

So before entitling actionable AI to more tasks make sure you have the essential safeguards in place, including well-defined unified data standards, access to complete, accurate, and up-to-date data, and data security guardrails such as data minimization, anonymization, and encryption. 

As for LAM-specific risk prevention measures, it’s recommended to isolate LAMs from host systems to protect the infrastructure from unintended consequences and provide a controlled environment for LAM testing and experimentation. 

Adversarial testing that simulates real-world attacks on a system and identifies its vulnerabilities, can also shield your company from harmful fallout and make sure the output of actionable AI is free from sensitive data, biases, and inaccuracies.

Another twist to old moves

Although billed as a tech nova, LAMs largely offer capabilities that are already present in AI agents, a more established technology hailed by high-performing companies today. They both can perceive and interact with the environment, reason, adapt behavior over time, and assist in complex decision-making.

So even if this bunny turns out to be a turkey, you can still prepare for the impact of actionable AI and reap its benefits ahead of competitors by integrating AI agents into your workflows.