Tag Archives: Assessment

Four Ways to Think About Workplace Learning

15 Jun

I’ve been talking lately to people who are charged with supporting learning in their business, as part of a new (to me) genre of professionals sometimes called Directors of Learning, or Chief Learning Officers. I’ve asked myself how I would go about designing support for learning in an organization, if I were in their shoes. Here are the results. I would think about four things: kinds of learning, location of learning, the activity continuum, and the zone of learning.

1. Kinds of Learning

Technical Learning

Of kinds of learning, I think of three: technical, adaptive, and systematic. Technical learning constitutes all those fixed skills, knowledges, procedures, and competencies in which what you are learning is relatively fixed and known. You have to learn a discrete set of information or to do things in the right way. It could be about facts, steps, or cultural norms. A particular equation. A software program. A way of participating in a discussion. The knowledge particular to your profession. It’s content you master. Stuff to add to the “container of you.”

Adaptive Learning

Adaptive learning, drawing on Ronald Heifetz, is different. It’s learning in which you yourself grow or change. The “container of you” gets bigger, better able to handle complex, ambiguous phenomena. In adaptive learning you discover and rewrite your assumptions about the world. Adaptive learning focuses more on the “you” part of you, or what’s there underlying the technical knowledge. Adaptive learning is more difficult, messy, and personal, and makes you acknowledge and address all sorts of anxieties, tensions, worries, and self-limitations. I would guess most workplaces are thinking about technical learning. But I would be impressed if many go beyond technical to embrace adaptive learning.


Just to show you how important adaptive learning is, I note that innovation is usually adaptive learning. That is, we’re not hardwired to innovate, and it’s not a technical skill (though it has some technical components); to be able to innovate, we have to change the way we see the world to allow it to be an ok, and not deathly-scary task: we have to learn to be ok with taking risks, failing, ideating like crazy, restraining evaluation, etc.

Systematic Learning

Systematic learning is when you attempt to understand things at a systems-level: as complex and recurring processes, like ecosystems or steam engines. On one level, just trying to see what is going on in your workplace as combination of systems–financial systems; systems of time and investment of time; systems of feelings, emotions, trust, or morale; decision-making systems; environmental systems–is already using systematic learning. The even more interesting application, though, is to the learning processes themselves. This is where you understand individual and collective learning (and their interaction with performance) as interlocking ecosystems, each with natural laws, growth, change, inputs, outputs and so on. This is probably the ultimate goal of any learning officer in a company: to have the learning systems of the company be as visible and as well attended to as well as are, say, finances.


Under systematic learning, I want to mention learning assessments, or the ways to know what is being learned. Without them you can’t “see” how you’re doing, so that you can make adjustments. If you can’t see and make adjustments, you don’t have a system that you can manage or understand. (It’s there but you have not found it yet).

2. Location of Learning

Location has two values or poles: Engaged and Disengaged.

Disengaged and Engaged Learning

Disengaged learning is learning outside of the context in which it is meant to be applied. Engaged is learning in the context. For example, you can read Ted Williams’ book about hitting a baseball, you can get someone to throw you a ball so that you can take a few swings, and you can play in a real game. Those are three steps along the path from disengaged learning to engaged learning.

The traditional view of learning is that it is disengaged. And much is. But engaged learning can be some of the most effective. If you cast your mind back over your life of learning, and you dig out one or two examples of where you feel you learned the most or the best, the chances are they will favor the engaged end of the spectrum. Engaged learning can take a lot of forms: apprenticeships, internships, mentoring, debriefs, just-in-time learning, difficult conversations, “gamified” work.

Doing and Learning

You may think that engaged learning just sounds like work, or like “doing.” Well it is. It’s a false dichotomy to think that learning isn’t doing. You have to do to learn. Yet there are also ways you can “do” that aren’t requiring much learning, where you’re essentially repeating things you mastered long ago. In my opinion, as an individual and an organization, you want your “doing” to have as much learning in it as you can. If you’re not learning a lot in the doing, that particular doing is probably ripe for automation.

3. Activity Continuum

Your learning will fall somewhere on a particular continuum I call the activity continuum. At the left end of the continuum is a kind of traditional, reductive, linear, conduit way of thinking about learning. On this end of the continuum we see learning as a kind of passive thing: we are transferring fixed discrete “things” to the learner. At the other end it’s active: the learner is understood to be doing or reflecting or making-meaning or becoming in some new, meaningful way. The poles can be seen to cohere across technical, adaptive, and systematic learning.

In the realm of technical learning, at the left pole, you will have a kind of simple, old-school lecture, or a handout. At the right pole you will have much more engagement, and probably less traditional content, more meta-cognitive thought going on, and more game-like or real-world structure. You can read a book about how to play Worlds of Warcraft or you can work through the tutorial. The tutorial falls towards the right pole. Similarly, in adaptive learning, you can learn about yourself, or you can actually work on your own behavior change. My favorite adaptive learning method, Immunity to Change, famously includes both poles. Also, systems can be understood to fall towards the passive or the active side of the continuum. There can be systems that are reductive and conduit-like, like the basic use of a learning management system (as a document repository). And systems that are dynamically changing, like something in a virtuous cycle of improvement, such as bamboo. The bamboo plant gets sun and nutrients and grows; the larger plant gets more sun and nutrients and grows more.  Eventually it’s a forest, an ecosystem.

I recommend inching towards the right pole wherever you can, but a few instances of left-pole thinking are ok. The problem is that our default is overwhelmingly left-pole, and that is an issue. (Why do we favor passive learning? Probably because it’s easier for the teacher . . . but I digress).

4. Zone of Learning

Thinking about the “zone” of learning is inspired by Vygotsky’s idea that there is a space, like a sweet spot, where you will learn best–a particular growth edge that, if you find it, will be simultaneously most compelling for you, encourage your best learning, and give you the most positive feedback, resulting in you wanting to learn even more.

This idea argues for designing learning that is tailored as much as possible to where the learner is. It requires we be able to assess where people are, and be able to adjust the learning content and experience to fit their needs. Of course the challenge in the workplace is that we will be unlikely to have the kind of master teachers, curricular experts, learning designers, and psychologists who can really make this kind of thoughtful analysis and then design learning activities to it. There are new sorts of automated assessments that can begin to help us, so some hope exists that we’ll be able to do a little bit along these lines soon. And, in any event, it is still worth asking, for any learner, what is the appropriate “zone” for their learning? If we tend towards one-size-fits-all (which is often the case), does this work for everyone? Are there basic things we can do to start to accommodate differences? What are the differences?

Another way to come at this challenge is to trust the learner. Oftentimes a self-aware learner knows best what they should be studying next. I give a personal example: I am an intermediate trombone player. I am actually more interested at this point in hearing people just a step or two ahead of me play, than I am in hearing virtuoso performers, as much as I like the latter. Why? Because the top edge of my learning zone at the moment is advanced intermediate, not virtuosity. I will learn better, grow more, and have more positive feedback with the more relevant goal.

Pieces of an Ecology of Workplace Learning

9 Apr

Lately I’ve been saying that you should cultivate learning in your organization as you might manage an ecological resource, like a forest, or any other complex system of high priority (like your computer network or your budget). As if learning were a “cognitive enterprise infrastructure” or worked like a kind of water cycle. But how would you do that, and what would it be like, and how would it be different than what you do when you think of your workplace as a kind of machine that consistently produces material stuff? I am not totally sure, but here I take a guess at nine possible pieces of an ecology of workplace learning.

  1. Cultivate Development, Rather Than Manage Performance. The point is not to manage people’s performances, but rather to get them to develop as much as they can, on the assumption that more highly evolved people do better things. The annual performance review that tracks behaviors against rather limited metrics and has a kind of binary output (wrong or OK) here evolves into something more like a coaching relationship in an experiential context: growth is the focus, not proscription. You look for activities that are motivating to the individual, that are a bit out of their comfort zone, yes, and you expect to support them in iterative cycles of trying things out, reflecting, adapting, and trying them out again. And you might add a variety of unheard-of supports and activities to help people think and reflect and be aware of themselves in a variety of dimensions, drawing on things from personality styles assessments to mentoring relationships to counseling-like activities, such as item 2, below. The trick is that these things, that we kinda do now in a knee-jerk way, away from work, would be more like the work.
  2. Support Cognitive Development. According to the work of Bob Kegan and Lisa Lahey (whom I talk about a lot), we evolve through a series of increasingly sophisticated ways of seeing ourselves and the world around us. That is, we can so develop, if given the right environment. And with this increased epistemological sophistication comes a better ability to deal with and thrive in complex environments. One such complex environment is the increasingly global, flat, multi-cultural, resource-starved, post-ideological, environmentally-challenged, a-traditional, scary world of today. To help people be effective in this kind of world requires activities that help us know differently; Kegan and Lahey’s Immunity to Change coaching process is one such structure. Having done it myself, I am amazed by its ability to make you reinvent the way you think about yourself and the world in which you engage; I fairly salivate to try it with a team of colleagues in a workplace. The downside? It’s an intensive, six-month process of bi-weekly meetings, invoking much deep personal questioning; that’s a huge investment. But in an ecology of workplace learning, invest in people is what you do: no rain means no rivers means no seas means no evaporation means no rain, etc.
  3. Assess Development in New Ways. In Higher Education we try to assess student learning, and it’s a challenge. But we don’t even try to assess faculty and staff learning; and the generic workplace doesn’t generally assess staff learning, either. But we should.  To promote development over production, we have see where this development is happening, individually and in teams. Of course it gets tricky: it’s easy to see your behaviors, but how do you see what’s going on behind the behaviors? Fortunately there are new kinds of tools that have potential in this regard: things like the Developmental Testing Service’s reflective judgment assessments, such as their test of managerial decision-making, which asks you to analyze complex, asymmetrical workplace problems, assesses you according to a complex scale of cognitive development rooted in Kurt Fischer’s work, and gives you (you yourself, the test taker!) rich feedback about your strengths and ways in which you can improve, data which feeds right back into the coaching relationship I mention above.
  4. Represent the Learning Ecosystem. If you’re going to try to manage an ecosystem, you need some kind of a representation of it. As the water cycle has its famous circular chart with arrows and the budget has its classic representations in profit/loss statements and balance sheets, so does the learning system have something. I don’t know what it will look like, exactly; but I imagine it will be something like the famous Kellogg Logic Model, which the well-known foundation suggests you use to understand your various high-stakes interventions, and which helps you see programmatic inputs, outputs, assessments, changes. With a key difference: the effect of your ecology isn’t an output external to you, it’s an evolution of your ecology. So a learning logic model would show as its characteristic feature a looping back upon its constantly changing self.
  5. Analyze How We Work; Analyze Our Culture. Part of learning is seeing yourself learn. That may be the single biggest difference between a learning organization and a producing organization: the learning organization sees itself and not just the things it makes. We will need to learn to pay attention not just to the products of our culture but to our culture, not just to the deliverable of the project, but to the way we work together on the project. For that a lot of tools exist already, like various kinds of post-activity group reflection encouraged in psychologically safe spaces, that let anyone share their experiences along the way. But new tools will help: the same sort of analytics thinking that has been transforming everything around us can help transform how we work together: social and network analyses to show us how we engage, corpus-linguistics analyses on the big data of our communications and cultural artifacts; these will help us, too, to see the patterns that make up our togetherness.
  6. Assign Staff to Cultivate Learning. Of course you can’t really have a garden without a gardener. And all the network analysis and group reflection exercises you might want to use won’t be that helpful unless it’s somebody’s job to watch learning in the organization at a meta level: to gather relevant data, assess its meaning, and help the group understand where it’s going. The teacher, if you will, of the organization. This would be a new thing: we’re used to thinking of Chief Information Officers, Chief Information Security Officers, Chief Executive Officers: this would be a Chief Learning Officer. Although of course it needs to be more than one person. And of course everyone has to be involved. But still the CLO might help organize it all. How much of your people resources should you put into learning, CLO and everything else thrown in? I propose 20% as a start. But I suspect it should be more, maybe up to 50%. Maybe 63%.
  7. Find New Ways to Gather and Share Ideas. Which Means Liking Them. One of the most important things in your organization are the ideas in people’s minds. The business world is just beginning to learn that to be relentlessly innovative, they have to gather and tend ideas in new ways, because ideas are the seed of innovation, be these ideas from their staff, their customers, their partners, their competitors. (See my last post for more on this). Part of this idea-tending requires a real cultural change–towards the acceptance and collective cultivation of ideas–and away from the general distrust of all things new that naturally grows up in an organization designed to perform consistently. Let me say that again: we will have to learn to like each other’s ideas.  And treat them, as it were, like a community resource, like, as it were, children. Because without them growing and maturing, we’ll fail. Businesses are starting to do this by building open, inclusive, idea-participation systems called Ideation Engines or Idea Stock Markets that aim to make the ideas in the group transparent and collectively developed. But I suspect you can go a long way without a particularly unique tool (a shared spreadsheet might work as well).
  8. Create Loops and Groups. In my perhaps over-simplified way of thinking, learning comes down to loops (in that feedback and reflection are crucial) and groups (in that learning is social; and in that your co-learners are as important for your learning as your own mind). So I think much of the key work of the Chief Learning Officer and her team will boil down to finding or building, and supporting, new sorts of groups in which people are desirous of learning together, and in adding “loops” to existing processes, to work reflection into the fabric of the organization.
  9. Do Some Old-School Ethnography. I am continually amazed by the complexity and mystery of people and of organizations. And by the fact that all you need to do to begin seeing and unravelling (or ravelling) the mystery is to observe people and ask questions (of course taking notes and writing down the answers). This is the way anthropologists settled on coming to know things as complex and mysterious as entire alien (to them) cultures. Libraries and IT departments have recently begun seeing that ethnography helps them understand the mysterious complexities of cultures alien to them, too (their customers). And it will work for you. On a certain level you can see an ethnographically-inclined research project as a kind of mirror to the people (if its results are shared with the people it studies), a loop at a high level, that both honors people and lets them see what’s going on. I think a lot about the emphasis in the popular Reggio Emilia model on the artful documentation of what the learners are doing; an ethnographic approach to your own organization is like that.

About the GRE

9 Jan

I took the GRE in November last.  It had been 20 plus years since I endured a high stakes, standardized test. My reflections.

To begin with, the GRE is stressful. Going in the testing room is something like entering the Green Zone in Baghdad. You’re signed in and out, you have your picture taken, you’re scanned for electronic devices, and you’re monitored during the exam. You repeat the process to go to the restroom. Also you sit in a grey cubicle in front of a computer for 4+ hours. No food. No music. Stress hormones floating in the air.

Even if the environment itself weren’t stressful, the GRE would be stressful for social reasons. Why? Because you have to do well on it to go to graduate school, get an advanced degree, get to do the thing you’ve pinned your hopes and dreams on, retroactively make your parents’ lives purposeful, etc. Images of your mom, neighbors, people you went to church or shul with loom up in front of you in the grey booth, threatening to return with disappointed expressions if you fail, etc.

Registering for the test is also rather Kafkaesque–the various medieval agreements, legalistic jargon, complex instructions, multiple and repeated threats to cancel your test and not refund your money! if you do something wrong–all of this is perhaps even worse for your peace of mind than the test-taking environment.  One of the things in particular that you are required to sign forces you print the text of a long-winded license agreement by hand.  Imagine doing that with iTunes.

Compared to the above, the test itself is relatively benign. There are multiple thirty-minute sections testing three things: quantitative reasoning (or math), analytical writing (where you make an argument about someone else’s argument), and verbal reasoning, which is a kind of mix of logic and language. If you say this here, what should you say over there, etc. But for me the GRE sections or the particular questions aren’t really the problem.

One problem is just the stress: do people effectively demonstrate rich and varied learning in high stakes environments? Probably not. Should they have to? No, in my opinion. Another problem is that the GRE doesn’t really test anything people do. When, for example, do you ever have to write an essay in thirty minutes? Without an opportunity to revise? Or without a spell checker? And when, if you’re not a mathematician, do you solve quadratic equations for the heck of it? An assessment that has no connection to what you do in life, or even what you would do in graduate school (trust me, you get more than a half hour to write your essay in graduate school), isn’t helping you, although it might be helping someone else.

Maybe the worst problem I have with the GRE is on a conceptual basis. It’s with the idea that you can and should test people in isolation as fixed entities. That the test assumes I have, say, a kind of inherent knowledge essence that is discrete and knowable separated from the people I work with or the things I am doing. That I am a container more or less filled of the stuff teachers put into me. You see where I’m going: in a sense the GRE takes up the teacher-centric education idea that says knowledge is conveyed in little packages from a positive knowledge pole (the teacher) to the negative pole (me) and that we can measure the amount transfered like we can measure rainfall by looking at a graduated cylinder in the yard.

For one thing, the limits of thinking of ourselves as “fixed” are made clear by Carol Dweck’s compelling research. In short: people who see themselves as “fixed,” or “good,” say, give up when things get tough; people who don’t care as much what they are, but instead focus on getting better and being resilient and trying again succeed in the hard spots. The GRE and the way it’s given seems to reinforce the fixed mindset. Either you’re good (and can deposit an essay in a half hour) or you’re not (and can’t); no room for someone to, say, draft, reflect, tweak, get feedback, adjust, do some more research, etc.

I’m much more interested in people as developing and growing beings and want to see them learning over time–in actual contexts. So for me a better assessment would more or less look like a record of our growth throughout school. Be it the kind of complex, multi-domain cognitive development assessments the Developmental Testing Service (that’s DTS as opposed to ETS) offers, be it a great e-portfolio, be it a discursive report on the student, their work, and their environment (my preference).

Of course this doesn’t exist as far as I know. What to do in the meantime? Well one thing we could do is drop the requirement of the GRE and reinvest the money invested in taking it into beefing up the teams of thoughtful people who are tasked with reading applications for admission.

The Vygotsky Challenge

23 Nov

I was reading an article on Lev Vygotsky, the influential Soviet psychologist, and I was struck (again) by his emphasis on the social context of learning, and by what that implies for the way we organize ourselves in education.

In contrast to the individual orientation that permeates just about all organized learning, Vygotsky stresses the importance of focusing on the supporting structure, the social context, the scaffolding around the student.

According to Wertsch and Tulviste’s “L.S. Vygotsky and Contemporary Developmental Psychology” (in An Introduction to Vygotsky, 2nd Ed, Routledge, 2005), “mental functioning in the individual can be understood only by examining the social and cultural processes from which it derives” (60). Vygotsky’s Zone of Proximal Development, for example, which famously describes the area between what the person has learned and can learn (with help from teachers and adults and friends and culture) is not so much about improving individual learning, but rather about improving the social and cultural context in which that learning happens. In other words–you can improve the individual’s learning by focusing on the larger group (63).

This is about as revolutionary a thought as can be imagined for schools.

Generally speaking, institutions of education are designed with the goal of improving students as individuals. Everything we do is organized around individual students, from enrollment, to advising, to assessment, to grades. If we think about culture at all, it’s also student focused; how faculty interact with students. How students interact with students–in the classroom, in the dorm, in student clubs.

Nowhere do we really think to a similar degree about the larger culture of the institution. How faculty and staff and students as a collective whole, say, talk to each other, help each other, learn together, share ideas. Whether we trust each other. Whether we all get to have input, say, to identify institutional problems together. Whether we solve our problems together, etc.. In other words, we think a lot about assessing individual student learning, but we don’t really think about assessing the context around that learning.

Just as a simple little example, consider the amount of energy–resources, planning, assessment, time, space, books, chairs, etc.–that goes into just one regular college course. Then think of the corresponding amount of investment we make in the development of a given faculty or staff member. A faculty or staff member might get a few workshops and a retreat or conference in a given year, but there is no commensurate institutional investment in planning and guiding and supporting and assessing such learning. A student gets a teacher, a curriculum, an advisor, expectations, a dean, a dorm life supervisor, and on and on. Faculty and staff might get part of a manager or a chair and someone to review an activity report; but they of course get so much less support in their own development that I feel silly making the comparison. And that’s just thinking of faculty and staff themselves as individuals, and not taking into account their social context, which gets even less attention still, and which, after all, is the real point.

If we take Vygotsky to heart, we should be thinking precisely about how our faculty and students and staff–all of us–work together, share, think, learn, develop–as a community.  Observing and assessing and understanding and improving the bigger culture should be a priority, and doing it well should translate into vast gains for students. For a better culture will make a bigger zone of proximal development.  (I should note that many have developed Vygotsky’s idea here further, but it has not significantly penetrated into the DNA of our organizations. Yet.)

So that’s what I call the Vygotsky challenge: as we’re thinking about redesigning our educational institutions to better help students learn in this the rambunctious digital age, we should also think about how we assess and improve the culture around them, which means focusing on the “other” people hovering around the school’s halls, and on how we all talk to and treat each other. Such a focus will be a wonderful boon for those mysterious non-student people (who will feel that it’s finally OK for them to learn and develop, too) and, ultimately, help the students. Maybe as well as or better than anything else we might do from within the traditional, individually-focused paradigm?

We might one day even go so far as to no longer distinguish between students and staff and faculty, who are, after all, just learners at different points on the continuum, but I’m perhaps getting carried away.

Pedagogical Space and Political Space

18 Jul

I am reflecting on the difference between pedagogical and political space. I tend to prefer the former; I don’t think you can escape the latter. My thoughts below.

Pedagogical space is where people organize themselves to learn. It has what they call “psychological safety.” You can be vulnerable and wrong. In fact, that is the point–to be wrong, to reflect on it, and to adjust. You do messy things like encounter challenging new information and try to make sense of it, or surface your assumptions (things you believed but didn’t know you believed) and adjust them. All with a group of people doing the same thing. There is multi-directional communication, but it isn’t out of control: everyone gets a chance to (and has to) talk; no one person dominates. Reflection and feedback are the coins of the realm. Not just “I love it or I hate it” feedback, but careful, generative, constructive feedback that tells you what you understand well and where you can think a little bit more. File assessment under the feedback category: in pedagogical space, assessments are going all the time–informal, formal, and in-between–of your learning, of the group’s learning, of the teaching, of the interaction, of the value of the information the class is working on, on the goals of the course. Oh, and you buy in and have say, but you also have responsibilities. You have the chance to CHOOSE to be in the space, and to choose the particular way you’ll go about your learning (from a discreet list, likely), but you also have to help uphold the norms. Pedagogical conversations incline towards wondering–sharing data with proposed interpretations. Finally, pedagogical space leaves more capacity in its wake (because people learn, are more developed, have relationships, know better how to learn).

Political space is different. The goal isn’t to learn, it’s to influence or control. Communication is generally unidirectional–from the person who wants to do something to the people who are suposed to let her; the communication is minimal (just enough to get approval) there is little chance for feedback. When feedback happens, it’s stilted, shrunken, a critique (with an ulterior purpose) or a sycophantic agreement. There is less choice–you’re there because it’s your job, say, or it’s the place you use to tell people what to do. Vulnerability is to be avoided (because people will use that to divert your plans), and the last thing you want to do is reflect on your own assumptions. You also desire not to be wrong, and not to admit it if you are. Your gaze is outward, towards the external world you intend to change. There is no real assessment of the team or its conversation or its purpose–if there is assessment, it’s usually of the product of the work of the team. Political conversations incline towards the dogmatic–positions or interpretations without evidence. And, finally, political space leaves the same or less organizational capacity in its wake.

Having said all this you probably can’t have all one or the other, and any group or conversation probably only leans one way or the other. Neither is completely bad or good. Political space, for instance, is what you want when you don’t need to learn, but just need to act. It’s efficient. On the other hand, if you want your team to do some new work it’s never conceived of before, or grow in some important way, or takle a challenging new problem, political space isn’t going to work on its own. Political space does seem to be the societal default, but it doesn’t need to be; by contrast, it seems the great leaders use pedagogical space to great advantage to convey their message. One thing that does seem clear: the people involved need to know what space they’re in. A part of the team following the rules of one when the rest of the group is in the other would be awkward. So maybe the ultimate goal is to let the team pick the one you want based on what you are trying to do, and be open about it.