Learning Science

The need for AI durable skills research

Recent research by Gerlich (2025) found a significant negative correlation between frequent AI tool usage and critical thinking abilities. This was particularly evident in younger participants, who showed higher dependence on AI tools and scored lower on critical thinking assessments compared to older participants. The study attributes this decline to ‘cognitive offloading’, the delegation of thinking tasks to machines, which appears to undermine our capacity for independent analysis.

However, Multiverse recognises that as the world of work evolves, so too will our conceptualisation of intelligence and the skills required for effective AI interaction. Beyond just critical thinking, there exists a whole new set of durable skills that individuals must master to harness AI’s potential.

Our research approach

Our research aimed to investigate the specific durable (soft) and cognitive skills that enable successful AI adoption and integration in the workplace.

We had 3 research questions:

1. What specific durable and cognitive skills are essential for successful and effective AI use in the workplace, and why?
2. How is task performance using AI affected when the relevant durable and cognitive skills are not present?
3. Do durable and cognitive skills for successful AI use vary between experience with AI levels?

We used the following definitions of durable and cognitive skills:

Durable (soft) skills refer to personal attributes and social abilities like communication, adaptability, and ethical awareness that enable effective human interaction and collaboration, representing uniquely human competencies that cannot be algorithmically replaced (Amann & Stachowicz-Stanusch, 2020; Kumar, 2023).

Cognitive skills refer to the mental abilities and processes fundamental to acquiring knowledge and understanding, including analysing, applying, creating, and reasoning, which are essential for learning, decision-making, and critical evaluation of AI outputs (Zhai et al., 2024; Gerlich, 2025).

To ensure the authentic representation of these human skills, we employed a Grounded Theory approach. This is a data led, iterative process that builds theoretical frameworks directly from data, rather than testing pre-existing hypotheses. This allowed us to observe human behaviour in an AI environment, extract and pinpoint core skills from this raw data.

We conducted this observational research using Think Aloud Protocol Analysis (TAP; Ericsson & Simon, 1993), a research method which gathers verbal reports as data. The participants, 20 of Multiverse’s AI users ranging from beginner to expert level, verbalised their thoughts and decisions as they carried out daily tasks using AI. This was paired with follow-up interviews to understand participants’ perceptions of the way they interacted with AI.

Our findings

After collecting our initial data, we conducted thematic analysis which highlighted a set of 13 skills with examples of how each skill optimises AI use in the workplace.

These address research question 1, ‘what specific durable and cognitive skills are essential for successful and effective AI use in the workplace, and why?’, and research question 2, ‘how is task performance using AI affected when the relevant durable and cognitive skills are not present?’

Below, you can see an example skill that was evidenced in our research, ‘Tailoring Communication’. As alluded to above, this example shows how grounded theory research was used to identify specific skills. We analysed the raw data and grouped themes together, undergoing a process of iteration and refinement which eventually led to our final skillset of 13.

1. Tailoring communication: Discerning whether AI output has the desired tone for a particular audience or situation, and refining prompts if it is not.

This skill was observed as participants reviewed AI outputs to ensure a match with their desired tone, to sound like the human user, or to be appropriate for a particular audience. In the TAP analysis, a participant talked about understanding their environment in relation to AI’s outputs:

"The key here is marrying the output of your AI tool to the human world that you live in at work, which is like generally what is the expectation and the culture surrounding what your output should be."

Participant 9
Intermediate/Advanced AI user

Another participant reflected on the key soft and cognitive skills they employed in their AI interactions:

"I like to think about how I would explain this process to a normal person who isn’t a robot. And then that explanation becomes my prompt."

Participant 16
Expert AI user

We also captured evidence addressing research question 2, as participants reflected on the consequences of not tailoring their communication when using AI:

"The consequence would have been additional questions or confusion created by not being very clear and speaking in a voice that was appropriate for the audience that you're working with."

Participant 7
Expert AI user

Whilst this participant candidly explains:

"If I would solely trust and let Chat-GPT guide me in my communications I would truly fail."

Participant 1
Intermediate AI user

A note on critical thinking...

Interestingly, the evidence we captured for cognitive skills when using AI echoes established research demonstrating that when people anticipate future access to information, they exhibit lower rates of information recall but enhanced recall for information location and access methods (Sparrow, Liu & Wegner, 2011). This suggests that memory storage is being relocated rather than diminished, prompting us to reconsider which cognitive abilities are most valuable when working alongside AI systems. Our research supports this phenomenon, suggesting that the challenge lies not in cognitive decline as Gerlich’s research concluded, but in determining which skills to prioritise in an AI-augmented work environment.

Addressing research question 1, the full set of our 13 critical skills for AI adoption is listed below, along with their groupings:

Cognitive skills - Mental abilities used for learning, reasoning, problem-solving, and decision-making.

1. Analytical reasoning: Breaking down complex information for AI to more effectively deliver its instructions; recognising tasks that AI is or is not suitable for.

2. Creativity: Pushing the boundaries of AI use and experimenting with new approaches to drive innovation.

3. Systems thinking: Identifying patterns in AI performance to predict how AI will respond to a task.

Responsible AI use skills - Applying ethical principles to ensure the responsible use of AI, considering its impact on individuals and society.

4. AI ethics: Spotting bias and recognising how it affects AI outcomes; using AI outputs in an ethically sound way to inform business recommendations.

5. Cultural sensitivity: Identifying when AI outputs lack sufficient geographic or cultural awareness.

Self-management skills - Recognising our thoughts, values, feelings, and behaviours, and how they impact our ability to achieve our objectives when using AI.

6. Curiosity: Examining the broader context and requirements of a task to augment AI outputs.

7. Self-regulated learning: Reflecting on the success of a chosen AI approach; partnering with AI to self-assess its outputs.

8. Detail orientation: Fact checking AI for hallucinations and errors; using one’s own domain expertise to ensure accuracy.

9. Adaptability: Iterating and refining one’s approach to interacting with AI based on the quality of outputs.

10. Determination: Patience and willingness to continue trialling new approaches with AI, even during unsuccessful AI interactions.

AI communication skills - Strong interpersonal skills which support the optimisation of AI outputs.

11. Empathetic interaction: Treating AI as an extension of one’s own mind and thoughts; anthropomorphising AI to create more thoughtful, receptive, and intentional dialogue.

12. Tailoring communication: Discerning whether AI output has the desired tone for a particular audience or situation, and refining prompts if it is not.

13. Exchanging feedback: Using AI to proactively seek feedback on work.

Finally, addressing research question 3, our research also revealed that participants at four different AI experience levels exhibited distinct characteristics.

• Basic users tended to focus on task completion with simple prompts and limited evaluation.
• Intermediate users balanced quality and efficiency with growing AI awareness.
• Advanced users optimised AI for strategic tasks, used more complex prompts, and exhibited metacognition i.e. reflected on their own strengths and limitations in the AI interaction.
• Expert users integrated AI into sophisticated workflows, whilst maintaining extensive knowledge of AI’s constraints.
  

Interestingly, we found that female participants consistently underestimated their AI competency in self-assessments, requiring upward adjustments to a higher experience rating based on observed performance - highlighting important implications for how AI confidence is perceived across demographics.

What’s the impact of these findings?

In addition to answering our research questions, we have addressed a critical gap in the literature by conducting bottom up, grounded theory based research. Almost every piece of research or articles written about durable (soft) skills relies on pre-existing definitions of durable and cognitive skills. Our inductive research, on the other hand, observes how these skills naturally emerge and manifest in real workplace contexts - allowing us to discover authentic skill categories which reflect how humans behave in relation to AI.

Multiverse has already recognised the importance of these soft skills and successfully mapped them onto our existing learning programmes. For example, in our AI for Business Value programme, the technical requirement to ‘model business processes using relevant techniques, standards, notation and software tools’, directly connects with the durable skill of ‘Creative Thinking: being confident enough in one’s own AI abilities to push the boundaries of AI use’, demonstrating how durable skills are essential for mastering technical skills.

Additionally, being able to identify these skills allows us to progress towards being able to assess them and measure them, helping employee’s develop deeper and more sustainable AI capabilities beyond more basic AI awareness and technical skills.

And for leaders?

There are several key takeaways for leaders from this research:

Make strategic AI investments: Rather than pursuing blanket AI adoption that can reach billions in expenditure, leaders should evaluate tools based on their specific use cases and longevity, and whether they will unlock your company’s potential or hinder progress. Consider reframing your company’s skill development priorities towards transferrable soft and cognitive skills which in turn enhance any technical competency.

Crucially, focus on investing in learning as much as the tools themselves - creating the time, space and resources for deep and lasting AI adoption is as critical an investment as purchasing the technologies.

Map existing training: If your organisation has existing AI that requires technical training but you aren’t seeing progress in AI adoption, consider mapping that training against our newly identified durable skills. This approach may increase adoption and learning of your already-invested AI technologies. Leaders can also identify where relevant AI durable skills naturally align with technical competencies and integrate them, rather than treating them as separate initiatives.

Normalise cognitive offloading: Help your teams understand that relying on AI for certain tasks isn’t cognitive laziness, but strategic resource allocation that exercises an entirely new set of cognitive capabilities. Leaders can model and encourage when it is appropriate to use AI, while still valuing uniquely human contributions.

References:

• Amann, W., & Stachowicz-Stanusch, A. (2020). Soft skills and their role in employability. Management International Review, 60(4), 485-510.
• BBC. (2025). Government redirects apprenticeship funding towards foundational skills. BBC News.
• Clevry. (2025). Hiring Intelligence Report 2025: Soft skills as top hiring priorities. Clevry Research.
• Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data (Rev. ed.). MIT Press.
• Gerlich, R. N. (2025). AI overreliance and critical thinking decline: Evidence from workplace studies. Journal of Cognitive Technology, 12(3), 45-62.
• Global Skills Agenda. (2025). Core skills for the future workforce: Resilience, creativity, and analytical thinking. World Economic Forum.
• Goldman Sachs Research. (2023). The potentially large effects of artificial intelligence on economic growth. Goldman Sachs Economics Research.
• Kumar, S. (2023). Soft skills in the age of AI: Redefining human value in automated workplaces. Human Resource Management Review, 33(2), 178-195.
• Leça, B. P., & de Souza Santos, M. (2025). Cognitive skills development in AI-enhanced learning environments. Educational Technology Research, 41(1), 23-38.
• Nadeem, A. (2024). The irreplaceable human: Soft skills as competitive advantage in AI integration. Business Strategy Review, 35(4), 112-128.
• Sparrow, B., Liu, J., & Wegner, D. M. (2011). Google effects on memory: Cognitive consequences of having information at our fingertips. Science, 333(6043), 776-778.
• University of Oxford. (2025). AI skills wage premium study: Technical competencies in the modern workplace. Oxford Internet Institute.
• World Economic Forum. (2025). Future of Jobs Report 2025: Skills disruption and workforce transformation. WEF Publications.
• Zhai, X., Chu, X., Chai, C. S., Jong, M. S. Y., Istenic, A., Spector, M., ... & Li, Y. (2024). A review of artificial intelligence (AI) in education from 2000 to 2020. Educational Technology Research and Development, 72(1), 1-45.

At Multiverse, we focus on four key competencies that drive learner success and business impact: our coaches are industry experts, data-driven, connectors, and guides. Together, these competencies form our Compass Framework, which we use to hire, train, and evaluate our coaches.

The four competencies of the Multiverse Compass Framework

According to apprentices and apprentice managers, their coach is the most influential part of the learning experience. Here’s what makes them exceptional:

• Industry experts: Multiverse coaches have the expertise to deliver effective, enjoyable, and relevant learning experiences. Most join following years of working in industry, adding insight and experience to their teaching. With an average of 60 hours of technical upskilling per year, our coaches stay updated on the latest tools, techniques, and trends.
• Data-driven: Our coaches meticulously plan sessions by integrating data to understand learner progress and client needs. This preparation ensures purposeful, personalised interactions. We help coaches collect, monitor, and act on this data with proprietary tools like Coach World and weekly coach enablement workshops.
• Connector: Multiverse coaches foster personal and professional growth by building trust and creating safe environments for taking risks and asking questions. Our Coach Academy equips coaches with these essential skills, which they refine through ongoing observations, workshops, and feedback.
• Guide: Our coaches help learners identify and complete impactful projects by asking relevant questions, encouraging idea sharing, and providing constructive, actionable feedback, usually within 24 hours. Customer Success Managers collaborate with each client to create joint success plans, ensuring our coaches understand their strategies and priorities. We provide training and toolkits to help them put this into action. At Multiverse, coach roles are specialized, so learners are always talking to a subject matter expert with the coaching skills to support their needs.

These competencies work in unison. Industry expertise forms the foundation; our ability to transfer knowledge and skills attracts many clients to Multiverse, but it’s just the beginning.

Data-driven preparation makes sessions engaging and relevant, while connection allows coaches to help learners thrive – whether they are practising new skills, finding opportunities to apply them, or overcoming challenges.

Finally, guidance ensures that individual learners’ actions align with clients’ strategic objectives.

How apprentices lead cost transformation

In 2023, a new customer approached Multiverse with financial challenges. They were launching a major cost-saving initiative and needed support.

They placed 50 apprentices in data-focused programmes, with each apprentice tasked to bring their new skills back to their teams. This goal was outlined in the Joint Action Plan created by the customer and the account executive, serving as a key reference for the coach throughout the apprenticeships.

Each month, apprentices honed their data skills – analysis, visualisation, and automation – during workshops and individual sessions. In group coaching, the coach aligned apprentices on their organisation’s strategic challenge, and used it to frame the content of the sessions.

For instance, during the data quality unit, the coach prompted apprentices to discuss the data challenges they faced within their organisation and connected these challenges to tools and techniques learned in the program.

When poor data quality emerged as a significant issue, the coach guided the conversation toward problem-solving:

• How might apprentices apply their data cleaning skills?
• What approaches would be most effective?
• Which additional skills, like engaging relevant stakeholders, could ensure their projects receive the necessary support?

Throughout their apprenticeship, the coach provided tailored support for each apprentice, from tutoring sessions to stretch content to assistance with workplace challenges.

By any measure, the apprenticeships were transformative. Cost assessments that took weeks became instant. A new spend management system, developed by one apprentice, was implemented by every category lead in the organisation. And over £50 million in savings were identified.

In summary

This is just one of thousands of examples from our employer partners across the US and UK. We take pride in being a strategic partner for transformations in areas like AI and data, and it's our coaches, and our coaching model, that drive exceptional outcomes like these every day.

Bridging the Skills Gap: The Need for Effective Learning

We are facing a future where a majority of the workforce faces skill gaps, yet not everyone has the opportunity to bridge them. According to the World Economic Forum (WEF; 2023), by 2027, 60% of the workforce will have an urgent need to reskill. Critically, however, the WEF reports only half of these workers have the training opportunities they need today. At Multiverse, we believe in education as a lever for business transformation, as opposed to being an added benefit to the workplace. As demand for new skills, particularly around big data and AI continues to surge, the notion of a single training session being sufficient is outdated. Instead, continuous educational initiatives that align with real-world demands are crucial. This arms employees with new skills that can directly improve job performance and facilitate, en masse, an adaptable and agile workforce. Consequently, understanding what makes learning effective to support these skills gaps and organisational needs is increasingly important.

A Dynamic System: The Zone of Optimal Learning Effectiveness

While complex, learning can be understood through its key components—cognitive, emotional, and environmental influences. In the workplace, this complexity is further intensified by the need to align diverse stakeholder priorities, ranging from individual career development to organisational performance metrics. It involves not only technical skills but also critical thinking, problem-solving, and the need to transfer learning to new situations—abilities that are not easily measured by conventional educational assessments. Workplace learning is therefore characterised by a complex ecosystem containing diverse stakeholders, interconnected elements, and dynamic interactions (Wang & Wang, 2018). At Multiverse, we have created the ZOLE framework—the Zone of Optimal Learning Effectiveness—to deliver a comprehensive and dynamic approach to building an effective learning experience that aligns educational initiatives with organisational goals and individual skills gaps. Recognising the complexities of workplace learning, we emphasise the critical need for alignment among three dynamic, dependent and interactive systems: the learner’s educational environment, the learner, and their workplace environment.

How ZOLE works in practice

Creating ZOLE involves a dynamic interplay among multiple factors:

1. Multiverse (the educational environment): This refers to the curriculum, methodologies, resources, social context, and delivery methods used in Multiverse’s programs. Our environment is designed to be engaging, challenging, relevant, and practical, ensuring that learners can connect theoretical knowledge with practical application seamlessly. At Multiverse, we also have an emphasis on guided learning, where we provide a dynamic ecosystem of support using a blend of on-demand AI and human-centered support systems so that no learner is ever alone in their journey.
2. The learner: This encompasses the individual attributes of learners, including their readiness to learn, cognitive abilities, motivation, prior knowledge, and specific skills gaps. We use a model of Learning Agility to understand the full spectrum of an individual's learning capabilities, focusing on their cognitive strategies, motivational drives, and actionable behaviours. This approach allows us to create personalised learning pathways that not only cater to each learner’s unique needs and skills gaps, but also optimise their ability to apply and benefit from their learning in the workplace to address organisational goals.
3. The workplace environment: The workplace environment includes the culture, support systems, and opportunities within a learner’s organisation that facilitate continuous learning and the application of new knowledge, skills and behaviours. A supportive workplace culture that encourages learning and provides opportunities to apply these new skills is crucial for achieving learning effectiveness. Managers, in particular, play a pivotal role in creating opportunities for learners to apply new knowledge in both low- and high-stakes situations, while also offering essential and constructive feedback to support the learning process. At Multiverse, we also tailor our learning pathways to align with specific organisational goals and learner skills gaps, ensuring that our educational initiatives facilitate the seamless application of knowledge, skills and behaviours to support this learning process. This also sustains a feedback loop that continuously refines and enhances both learner engagement, performance and overall effectiveness within the workplace environment.

ZOLE encapsulates the dynamic nature of workplace learning—it's not a one-size-fits-all scenario. Instead, it’s a tailored approach that caters to the specific skills, needs and contexts of both the learner and the organisation. This concept illustrates that learning effectiveness is not a static achievement, but a constantly evolving state that occurs when three key factors—the learner, the educational environment, and the workplace environment—are aligned. By maximising this alignment, ZOLE ensures that educational experiences are impactful and create tangible, long-term organisational benefits.

ZOLE in action

At Multiverse we have 1,000+ employer partners across the UK and the US, where 93% of learners remain with their employer post their learning experience. The driver for these results is ZOLE. Imagine you are an Online Merchandiser working in the e-commerce team for a consumer products business. Your main responsibilities are to oversee online product presentation to optimise the shopping experience and drive sales. Both you and your employer see a big opportunity for the use of AI in your role and so you have enrolled on Multiverse’s AI for Business Value program. The programme begins by equipping you with foundational AI knowledge, its potential for business transformation, and methodologies for identifying real-world applications. You're immediately encouraged to apply this new understanding to your current role, identifying impactful AI solutions for your business. This involves collaboration with departmental stakeholders, ensuring that the ideas you develop are both practical and validated through feedback, embodying a seamless integration between educational learning and workplace application. Your Multiverse coach, an industry expert, also considers your specific needs. This personalised approach includes setting relevant objectives and goals, as well as celebrating your achievements, and fostering a feedback loop that aims to enhance your confidence and success within your organisation. Throughout the rest of the program, you'll further analyse business needs, implement AI solutions, and lead change, taking others through the change process and demonstrating its impact. Unlike traditional learning environments that may rely on hypothetical scenarios, Multiverse's approach is deeply rooted in the realities of your workplace and is driven by the Zone of Optimal Learning Effectiveness (ZOLE). This ensures the learning is not only relevant but also impactful, providing you with opportunities to apply AI innovations that significantly benefit your business and consumer experience. Ultimately, this enhances your career prospects and trajectory, exemplifying what effective workplace learning should achieve.

Final Thoughts

In today’s rapidly evolving world, the importance of learning effectiveness cannot be overstated. Understanding and optimising the processes that make learning impactful is crucial for individual and organisational success. By focusing on ZOLE, we enable a dynamic and adaptable learning ecosystem. This holistic approach ensures that our learning initiatives create impact.

For more information, please read:

1. Di Battista, A., Grayling, S., Hasselaar, E., Leopold, T., Li, R., Rayner, M., & Zahidi, S. (2023, May). Future of jobs report 2023. In World Economic Forum, Geneva, Switzerland. https://www. weforum. org/reports/the-future-of-jobs-report-2023
2. Wang, M., & Wang, M. (2018). Understanding Workplace e-Learning as a Complex Dynamic System. E-Learning in the Workplace: A Performance-Oriented Approach Beyond Technology, 75-84

This might sound simple, but this internal monitoring process is an incredibly powerful skill that we all possess: metacognition.

What is metacognition?

In essence, metacognition is thinking about your own thinking. It involves observing your thought patterns, tracking your attention, identifying areas where your knowledge may be lacking, and using these insights to bolster how you learn and retain new information. It relies on three main elements; metacognitive knowledge, skills, and experiences. Metacognitive knowledge entails understanding how you think, while skills are about knowing how to regulate your learning. Experiences, on the other hand, involve thinking about and adjusting your approach to learning in the moment. Let's take the example of learning a new language again. If you ever feel that you're not taking in the content from a book, your metacognitive skills and experiences are kicking in, nudging you to make a change.

“Metacognition is at the root of all learning” - James Zull (2011)"

Why does metacognition matter? Well, it arms you with the power to steer your own learning. Having a deeper understanding of our cognitive processes enables us to adjust our learning strategies to get better results. Let's face it, if you find that absorbing information from a book isn't working for you, wouldn't it be better to explore other methods such as interactive practice?

There is a wealth of literature to show that engaging in metacognitive practices can enhance learning outcomes (Akyol & Garrison, 2011; Anthonysamy, 2021; Stanton, Sebesta & Dunlosky, 2021). Learners who set clear learning goals, track progress, and reflect on their learning experiences can improve their acquisition of new knowledge and skills (Efklides, 2011). In addition, metacognition plays a crucial role in fostering a deeper understanding and mastery of new concepts as well as critical thinking (Shea & Frith, 2019; Wozniak, 2015). At Multiverse, we believe that encouraging our learners to reflect on how they think and learn allows them to take control of their own learning journey and build the skills needed to tackle complex challenges. This proactive approach to developing metacognitive abilities empowers our learners to solve problems and equips them with the cognitive tools needed to navigate the workplace.

The Multiverse Perspective

Our guided learning techniques nurture the metacognitive abilities of our learners, fostering a positive impact on their educational experiences, workplace performance, and everyday life skills.

Three examples of metacognitive practices used at Multiverse are:

• A proprietary metacognition assessment that is used to better identify where additional support is required for our learners
• Our expert tutors and AI-powered coach use the Socratic method to encourage learners to think critically and self-reflect, thus helping to enhance their analytical skills
• Regular progress reviews and self-reflection among learners, through peer-to-peer discussions and active reflection activities, facilitating continuous improvements in their learning experiences

To us, metacognition is a fundamental aspect of our pedagogical approach, arming learners with the necessary cognitive tools for success in a rapidly changing society.

Want to learn more about metacognition? We recommend reading:

1. Anthonysamy, L. (2021). The use of metacognitive strategies for undisrupted online learning: Preparing university students in the age of pandemic. Education and information technologies, 26(6), 6881-6899.
2. Akyol, Z., & Garrison, D. R. (2011). Assessing metacognition in an online community of inquiry. The Internet and higher education, 14(3), 183-190.
3. Wozniak, K. (2015). Supporting Adult Learners' Metacognitive Development with a Sociotechnical System (Doctoral dissertation, DePaul University)
4. Efklides, A. (2011). Interactions of metacognition with motivation and affect in self-regulated learning: The MASRL model. Educational psychologist, 46(1), 6-25
5. Stanton, J. D., Sebesta, A. J., & Dunlosky, J. (2021). Fostering metacognition to support student learning and performance. CBE—Life Sciences Education, 20(2).
6. Shea, N., & Frith, C. D. (2019). The global workspace needs metacognition. Trends in cognitive sciences, 23(7), 560-571
7. Zull, J. E. (2011). From brain to mind: Using neuroscience to guide change in education. Routledge

Learner feedback is traditionally viewed as a passive transmission of information from a teacher to a learner. However, the modern learning landscape emphasises a more engaging and responsive process centered around the learner, where the exchange of ideas is just as crucial as the information itself (Griffiths, Murdock-Perriera,& Eberhardt, 2023). The potential role of AI and particularly, ChatGPT, cannot be overemphasized in the development of this modern landscape. At Multiverse, we believe that these technologies could revolutionise work-based learning environments by offering effective feedback and positioning the learner as active participants in their feedback process.

A Comparative Study: Traditional Coaching Methods vs AI-feedback

In a recent study (Teasley, 2023), we explored the potential of ChatGPT for delivering meaningful feedback. We compared AI-driven feedback with traditional coaching methods, taking into account the learner’s acceptance and reactions to AI-assisted feedback. We also explored the interaction between both the AI and the learner (Neurerer, et. al, 2018). Although previous research has explored assisted feedback (Maier & Klotz, 2022), this work is at the forefront of using AI to deliver feedback in work-integrated learning environments.

Initial hypotheses were that apprentices would prefer coach feedback over AI-generated feedback. Surprisingly, we found that 70% of apprentices showed a preference for receiving both AI and coach feedback. ChatGPT offered a greater amount of feedback that encouraged self-regulation and autonomy in learners compared to human coaches, showcasing the reliability of AI in providing feedback. Furthermore, we found no significant difference in feedback effectiveness between ChatGPT and coaches, with ChatGPT's feedback slightly favored. Virtual rapport assessments indicated a moderately positive perception of ChatGPT's feedback for its human-like qualities and coherence. Qualitative feedback showed a preference for combining AI's specific and objective feedback with the personal touch and context understanding of human coaches.

Our study suggests that while AI can offer specific and objective feedback, the nuanced understanding and personal engagement provided by human feedback is irreplaceable, advocating for a complementary use of both AI and human feedback in educational contexts. At Multiverse, our exceptional coaches are at the heart of our learning experience. They offer personalized engagement through direct human interaction, which is enhanced by the use of AI technology. Our learners also have access to real-time AI feedback, whenever they need it, through our new on-demand AI tutor. This provides our learners with the tools and resources to reap the benefits from both human and AI feedback.

In summary, by utilizing the cognitive apprenticeship model and AI-enablement (Amankwatia, 2023) we can offer real-time coaching, adaptively scaffold support based on learner performance, and encourage reflective practice through dialogue, enhancing understanding and skill acquisition in a collaborative learning environment.

Study Details

Our study was a mixed methods design which used data from thirteen apprentices enrolled in a technology consulting degree programme. Naturally occurring coach feedback was compared with ChatGPT-generated feedback. This feedback was generated and coded against an Agentic Feedback taxonomy. Surveys measuring apprentice perceptions of feedback, acceptance, motivation, and virtual rapport were developed from the Feedback in Learning Scale (FLS; Jellicoe & Forsythe, 2019).

Survey data was compared and differences were tested for significance and effect sizes. Qualitative data was analysed for key themes and reported. Inter-coder reliability was calculated for feedback coding trials (overall agreement, 79.8%).

Overall, the study demonstrated that ChatGPT's feedback on digital apprenticeship assignments matched the agentic quality of coach feedback and suggests the potential for AI tools to enhance feedback in work-integrated learning by complementing human inputs with timely, specific, and effective feedback (Teasley, 2023).

References

• Amankwatia, T. (2023). Using AI with Cognitive Apprenticeship Theory, Upscaling and Retooling. The Evolllution. Retrieved March 22, 2024, from https://evolllution.com/technology/tech-tools-and-resources/using-ai-and-cognitive-apprenticeships-to-upskill-and-retool-adult-learners
• Griffiths, C. M., Murdock-Perriera, L., & Eberhardt, J. L. (2023). “Can you tell me more about this?”: Agentic written feedback, teacher expectations, and student learning. Contemporary Educational Psychology, 73, 102145.
• Hattie, J., & Timperley, H. (2007). The power of feedback. Review of educational research, 77(1), 81-112.
• Jellicoe, M., & Forsythe, A. (2019, August). The development and validation of the Feedback in Learning Scale (FLS). In Frontiers in Education (Vol. 4, p. 84). Frontiers Media SA.
• Maier, U., & Klotz, C. (2022). Personalized feedback in digital learning environments: Classification framework and literature review. Computers and Education: Artificial Intelligence, 3, 100080.
• Neururer, M., Schlögl, S., Brinkschulte, L., & Groth, A. (2018). Perceptions on authenticity in chat bots. Multimodal Technologies and Interaction, 2(3), 60.
• Teasley, W.P. (2023). Evaluating the suitability of ChatGPT to deliver effective feedback in work-integrated learning environments. (Unpublished master's thesis). University of Oxford, Oxford. http://dx.doi.org/10.13140/RG.2.2.26629.77287

Rest assured, as humans, our learning capabilities extend beyond those of pigeons. Behaviourists assume that the best learning is when teachers or instructors take control over the learning process, actively reinforcing learners in order to get desirable learning outcomes. Although we can learn a lot from this approach, the “carrot on a stick” notion of reward learning is not holistic. What about the past experiences learners have? What agency do they have to shape their own experience?

Zoom into modern day, education has shifted from an era of teachers being a “Sage on the stage” to a “guide on the side”. This shift has been powered by the increasing popularity of Constructivism. Constructivists believe that learning is an active process where individuals construct new knowledge based on their prior experiences and interactions with the environment. This approach emphasises the importance of hands-on experiences, social interactions, and reflection in the learning process. Constructivist theory suggests that learners build their understanding through exploration, problem-solving, and collaboration rather than passively receiving information.

When we look at learning through the lens of constructivism, we want to maximise opportunities where learners build their own understanding, and a truly powerful way to do this is through social interaction.

“Social learning can be defined as joining with others to make sense of and create new ideas.” - (Bingham and Conner, 2015)

How does social learning work at Multiverse?

Social learning can happen casually when you have a conversation with someone, or it can happen during a structured group learning exercise. Where and when people learn socially can vary. At Multiverse, we use four research-backed strategies to drive collaborative learning in our programmes. These four engines are presence, inclusion, accountability, and reliance.

Presence

Presence helps learners feel like they are part of a community and are able to interact with each other while focusing on a similar goal. The feeling of presence in our programs is often facilitated through two key strategies:

• Cohort-based Learning: Learners are grouped together in cohorts, allowing them to form bonds, support each other, and collaborate on various learning activities. Cohorts provide a structured environment for learners to engage with peers who share similar interests and goals.
• Group Coaching: Through group coaching sessions, learners receive guidance and support from experienced coaches. These sessions not only provide valuable insights and knowledge but also encourage interaction and collaboration among learners within the group.

Inclusion

Inclusion is another important aspect of our collaborative learning approach, aimed at fostering a healthy culture and promoting psychological safety among learners.

• Multiverse community: Multiverse apprentices get lifelong membership to a global community of professionals who are taking their careers to the next level. This community experience plays a vital role in providing a supportive and inclusive environment where learners feel valued, respected, and encouraged to share their ideas and perspectives without fear of judgement. With a mix of career support, development opportunities and exclusive events, apprentices can find forums and groups of like minded individuals who share similar interests and passions.

Accountability

Accountability is essential for ensuring that learners take ownership of their learning journey and are responsible for achieving specific outcomes. To instil a sense of accountability, we often incorporate the following mechanisms:

• Progress Reviews: We invite coaches, apprentices and their managers to take part in regular reviews of apprentice progress to help track their development and identify areas for improvement. These reviews provide valuable feedback and guidance to learners, helping them stay on track and motivated to reach their goals.
• Project Feedback: As Applied Learning is core to our apprenticeship model, at Multiverse we use a mix of coach, peer and AI feedback to provide timely feedback to apprentices who are applying new skills in their role and driving business impact as a result. Our feedback mechanisms foster a culture of continuous learning and help learners to gain new perspectives to improve their skills.

Reliance

Reliance is a key strategy used to reinforce the importance of teamwork and collaboration among learners. By encouraging learners to rely on each other for support and success, we promote a culture of interconnectedness and mutual trust. How reliance is embedded can vary by programme but two key examples are:

• Hackathons - Participating in hackathons challenges learners to work together on innovative projects, collaborate on problem-solving, and showcase their collective skills and abilities. This helps apprentices build their confidence by simulating work-based projects that they may have to tackle in their current or future roles.
• Learning Pods - Learning pods are small groups of learners who work together on specific projects or learning activities. These pods encourage learners to share knowledge, support each other, and collaborate towards achieving common goals, enhancing the overall learning experience.

Want to learn more about social learning methodologies? We recommend reading:

1. Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18(1), 32–42.
2. Garrison, D. R., Anderson, T., & Archer, W. (2000). Critical inquiry in a text-based environment: Computer conferencing in higher education. The Internet and Higher Education, 2(2-3), 87-105. https://coi.athabascau.ca/coi-model/
3. Hyslop-Margison, E. J., & Strobel, J. (2007). Constructivism and education: Misunderstandings and pedagogical implications. The Teacher Educator, 43, 72-86.
4. Kagan, S. (2001). Kagan Structures and Learning Together: What is the Difference? Kagan Online Magazine, Summer 2001. Kagan Publishing. https://www.KaganOnline.com
5. Vygotsky, L. S., & Cole, M. (1978). Mind in society: Development of higher psychological processes. Harvard University Press.
6. Bingham, T., and Conner, M. (2015). The new social learning: connect. collaborate. work., 2nd Edition. ATD Press.
   

We drive learning through MAGE – Measured, Applied, Guided, and Equitable learning. These principles guide our curriculum development and learning experience design. In other words, these pillars are the standards we hold ourselves to when deciding what learning we offer and how we deliver it.

“We unlock economic opportunity and potential for individuals and organizations by closing the skills gaps of today and tomorrow through measured, applied, guided and equitable learning”

Measured learning

Measured Learning means we collect the right data at the right time in order to demonstrate that learning is occurring as anticipated, as well as capturing whether it is having the desired impact. This calls for continuous effort, but at Multiverse, we’re committed to achieving it.

We build upon established learning measurement frameworks like Learning-Transfer Evaluation Model (Thalheimer, W., 2018) to create a map of data we collect throughout the learning journey. Not only do we measure whether knowledge, skills and behaviors can be evidenced; but we measure a litany of other data points that allow us to monitor learning performance and triage support as necessary. In many cases, we also support our learners in highlighting and celebrating the real-world transfer of this learning into workplace performance gains.

"One of our apprentices was able to demonstrate a saving of 20 hours per week through the automation of a previously manual financial process. At the heart of this was an automated dashboard taught on one of our data programs."

Applied Learning

Applied Learning means that our learning happens in a real-world context, learning what you need to know when you need to learn it. As Josh Bersin (2018) states, the more we learn in the flow of work (opens new window) the more impact we can have. We facilitate an active learning environment by encouraging apprentices to apply what they learn within the context of their own role. This makes it a transformative experience that facilitates lasting changes in mindset, perspective, attitudes as well as knowledge and skills. We support this in a myriad of ways including:

1. We focus on knowledge, skills and behaviors that are in demand; from first principles, our portfolio is assessed against market requirements
2. We mandate applied learning projects throughout every learning journey to ensure immediate real-world application
3. We collaborate with managers to ensure learning is applied in line with job requirements

Guided learning

Guided Learning means that learners are continually supported on their learning journey through a unique blend of AI-powered, on-demand and human-centered coaching. Our aim is to ease learners into that “goldilock’s zone” of challenge (Vygotsky, 1978; Wilson et al., 2019); ensuring they can do more with the guidance of knowledgeable experts and other forms of scaffolded support.

The benefits of a guided learning process have long been documented (e.g. Bloom, 1984). However, many learning providers have felt the tension between providing guided experiences and scaling their delivery through remote, productised experiences. At Multiverse, we believe this is a false choice, and aim to do both through our blend of AI and human approaches to coaching. We also believe there are a range of knowledgeable experts within a learning journey, encouraging apprentices to not only learn from their coaches but also their peers and extended Multiverse community.

Equitable learning

Equitable Learning means learning is accessible to everyone and can be used as a means to open up career possibilities. It means that everyone's unique qualities are valued and represented in our learning experience. As such, we’re continuously aiming to assess what makes each learner unique (e.g. how they think, how they learn, what motivates them) such that support and guidance can be tailored to each individual.

Within our Learning Science team, we have a blend of experts in workplace psychology and learning assessment to continue to put each individual learner at the center of each experience. As emphasized by Multiverse’s mission “Providing equitable access to economic opportunity for everyone”, equity and inclusion are at the heart of what we do.

By incorporating the MAGE framework into how we think about, build and deliver learning we can ensure we deliver learning effectively. In particular, this framework has been developed with on-the-job, professional learning in mind; as we at Multiverse help solve your business-critical problems and prepare you for the future of work.

References

• Bloom, B. S. (1984). The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring. Educational researcher, 13(6), 4-16.
• Josh Bersin (2018, July 8) “A New Paradigm For Corporate Training: Learning In The Flow of Work”. Josh Bersin. https://joshbersin.com/2018/06/a-new-paradigm-for-corporate-training-learning-in-the-flow-of-work/
• Knowles, M. S., Holton III, E. F., & Swanson, R. A. (2014). The adult learner: The definitive classic in adult education and human resource development. Routledge.
• Thalheimer, W. (2018). The learning-transfer evaluation model: Sending messages to enable learning effectiveness.
• Vygotsky, L. S., & Cole, M. (1978). Mind in society: Development of higher psychological processes. Harvard university press
• Wilson, R. C., Shenhav, A., Straccia, M., & Cohen, J. D. (2019). The eighty five percent rule for optimal learning. Nature communications, 10(1), 4646.
• World Economic Forum. (2024). Reskilling Revolution. World Economic Forum. Retrieved 20/03/2024, from https://initiatives.weforum.org/reskilling-revolution/home

Learning Agility is not only important regarding an individual's potential to learn (Eichinger & Lombardo., 2004), but also their effectiveness at work (DeRue et al., 2012). At Multiverse, we double down on the concept of applied learning; learning that translates into improved performance and therefore career success. This concept is therefore critical for us to measure. But what exactly do we mean by Learning Agility?

Learning Agility - What it really means

Learning Agility has historically been used more often in leadership frameworks than learning frameworks. We see this as a missed opportunity. Learning Agility can help us delve into understanding the intricate patterns of behaviour, the methodologies individuals use to learn, and the specific factors that significantly influence their learning processes. This provides a clear and detailed view of their strengths and potential areas of growth, enabling the development of more personalised strategies for their learning and development. In simpler terminology, Learning Agility not only aids in identifying potential but also assists in mapping out the most effective and personalized pathways for individual learning and growth.

How we define Learning Agility at Multiverse

Our model of Learning Agility is inspired by and builds on both the literature and our own empirical research findings. In simple terms, it splits learning agility into three crucial areas: the Head (thinking capabilities), the Heart (motivation), and the Hands (action).

• The Head: How do learners process information, use strategies to speed up learning, and apply knowledge independently? For example, our metacognition assessment helps to understand how well individuals can pick up and utilise newly learned skills in their day-to-day work.
• The Heart: What is it about learning that an individual finds most motivating? It involves a deep-dive analysis of seven unique motivation factors. It’s all about understanding how different motivations directly affect how someone interacts with the learning journey, and how they affect learning outcomes.
• The Hands: What are the behavioural traits that can either help or hinder the learning process? We delve into specifics, looking at things like ownership and openness, to reveal the natural tendencies of each learner.

Applying this in practice

All this additional insight about each learner is only valuable if it is actionable. It’s imperative that this data is used in a manner that helps optimize applied learning outcomes.

Here are a few ways in which we are using these insights at Multiverse:

• Design learning plans that work with individual thinking skills, motivational profiles, and natural tendencies.
• Spot potential learning hurdles early on, allowing for early intervention, increased retention and engagement, leading to better learning outcomes.
• Make the best use of resources by matching learners to the most suitable level of support.

In Summary

Our model of Learning Agility is a major step forward in understanding and applying individual differences research within a professional development context. By zeroing in on the individual elements of the Head, Heart, and Hands, training providers can deliver unprecedented levels of personalisation and effectiveness in their training programs. Plus, it leads to a more inclusive and equitable learning environment. It's all about giving every professional a fair shake at reaching their full potential.

References

• DeRue, D. S., Ashford, S. J., & Myers, C. G. (2012). Learning agility: In search of conceptual clarity and theoretical grounding. Industrial and Organizational Psychology, 5(3), 258-279.
• Eichinger, R. W., & Lombardo, M. M. (2004). Learning agility as a prime indicator of potential. People and Strategy, 27(4), 12.