Here’s a useful assessment rubric created by the University of Wisconsin – Stout that can be applied to ePortfolios. I would consider adding links within the criteria to exemplars of best practice but I think it provides a solid basis for evaluating student work.
https://www2.uwstout.edu/content/profdev/rubrics/eportfoliorubric.html
It feels like I’ve been banging on about this subject forever now but with assessments now finalised, it seems like a good time for a final wrap up.
In broad terms, I was a little disappointed with my students. It might have been a bad time of year to run this subject, with its demanding workload, but the majority of them seem to have only put in the absolute barest effort needed to pass. Assessment instructions which I thought were pretty clear weren’t followed and most of the reflections lacked any great insight. I had to ask many of them to rework and resubmit their assessments just to meet the minimum requirements.
What this does make me ask is whether this is the fault of my students or me.
As I haven’t taught formal classes for more than a decade, there are a lot of things that I haven’t had to deal with in teaching an ongoing subject with rigorous formal assessment. I also have a tendency at times to over-complicate things because it just seems like it makes them better. This probably also extends to my communication with my students and my expectations of them.
Fortunately, I am still keen to try this again.
Even during the marking process, as I had to walk away from the computer and swear at the walls, I was constantly reshaping the course structure, the assessments and the class activities in my mind to help avoid some of the issues that were arising. The fact that a handful of the “good” students were able to understand and follow my instructions suggests that I’m on the right track at least and am not entirely to blame but the fact that more than a few got things quite wrong does tell me that there is more work to be done.
I need to make it clearer that when students are creating draft learning resources, they actually need to be resources – things, objects – rather than broad and loose activity plans for a class. I need to explain clearly that the final learning resources should be the same as the draft learning resources but improved based on testing and feedback. To be honest, these things seemed so self evident to me that I couldn’t conceive of anyone not getting it but there we are.
I tried to put into practice a number of ideas that I’ve encountered in the education design community about getting students more involved in designing parts of their own assessments but this really just confused more people than it helped. (Which was a shame as I do believe that it is a valid and valuable approach)
I tried to give my learners freedom to follow their particular learning needs and interests but for the most part this ended up just giving them the opportunity to follow the path of least resistance and allowed for some fairly lazy work. I also should’ve factored into my thinking that the first week of a class is often going to be plagued by technical (logins not working) and administrative hassles and try to make allowances for this in not expecting too much work to be achieved in the first week. (That said, we have a strong need to demonstrate engagement in class activities to receive funding for students that later drop out and I was certainly able to prove that)
I think next time around there will need to be a little less freedom, a bit more structure and lot more clarity and simplicity.
On the whole I am happy that I have managed to get these teachers doing things they haven’t done before and I think they have developed useful skills and knowledge. I’d just like to do more.
Life got quite busy in the last few weeks, so screenface had to go on the back-burner for a little while. I think it’s worth taking a look at what happened with the DDLR & DDeLR (Design & Develop Learning/eLearning Resources) subjects and what I might do with them next time.
What happened?
The majority of teachers taking the DDLR subjects have a reasonable expectation that this is a class where they will be able to develop some rich skills in using our eLearning platform to make new things for their students.
The units and elements of competency however are heavily focused on a design and development process for learning resources. The assessments for the teachers (who are the students in this case) hinge on providing evidence that they have considered the characteristics of their cohorts and mapped out a plan for whatever resource they are building. (This should include documenting necessary materials, sources of support and risk planning possible contingencies that may arise). They then need to create the resource, test it with peers or students and make refinements to it before final implementation.
All in all, sensible practice and (I assume) something that most teachers already do as a matter of course in their teaching practice. (Whether or not they formally name the steps in the process is another matter)
What the units and elements of competency don’t particularly care about is what the teachers learn about in terms of usability, readability, general design principles and, of course, the use of a range of new technological tools to get it all done. (Which is what they are most interested in addressing)
So we already have tensions built into the subject in the conflict between what the teachers want and need and what they have to demonstrate and be assessed on.
While we started with a full house of 14 people on the first day, numbers quickly dwindled to a dedicated core of 6 – 7. (A number of factors came to play here including personal issues for a couple of the cohort and running this subject at the very end of semester, when these teachers are themselves inundated with their own grading and teaching responsibilities)
For those that remained, we were able to provide what I hope was an engaging range of activities and training in design principles for usability, copyright and the use of our eLearning platform. (I was well supported by a member of my team – Jo – who also kindly filled in for me when I was away).
Assessment items have been slow to come in – possibly due to the onerous nature of evidence requirements for the subjects. Learners are required to provide 4 draft learning resources (with accompanying design documents and student group profiles) of which 2 are then tested and refined into final learning resources.
I tried to streamline this process in the first week by having the class work on a draft learning resource in the first week – a checklist that might be used to test the quality of their other learning resources. There has been a fair amount of confusion about this and I need to consider whether it is worth trying again and also how I go about explaining the concept.
The idea was to get the class thinking about important qualities in their learning resources and also to get some more buy-in in their own assessments, by effectively designing part of their grading tool. (This is not a graded subject but my intention was that by having them use their learning resource checker on their other resources, they would be more mindful of issues relating to pedagogy, content and technology.
What have I learned?
I need to lower my expectations of what can be achieved in the first lesson. We were beset with technical and enrollment questions that disrupted my carefully planned series of tasks and activities.
I had also put too much faith in the technical skills of the cohort and their ability to effectively use our LMS. I tried to do too many clever things – setting up conditional release on activities so that the learners could only access certain activities or resources after completing others.
I didn’t provide sufficient information about how the class might submit assessment items which were from their own development courses in our LMS. The assessments were set up as an assignment dropbox to receive files. I ended up telling people to create a word document with some screenshots and a link to the resources that they had created but this should have been explicitly stated in the assessment instructions.
I am happy that I was able to be flexible enough with the course to ask the learners what tools they were most interested in learning about and reshaping the course to accommodate this. A core principle of adult learning is that adults need to see the value in what they are being taught and this was an easy way to achieve this.
I’ve been able to speak to the previous teacher of this subject and she also struggled with a number of these issues – hopefully input from a wider group of colleagues might offer some solutions.
I’ve been a bit caught up preparing for this course and consequently this post has been sitting in the draft section for a while now. I ran the first class last Friday (17/10) and it seems like a good idea to share some reflections.
I’m going to leave the pre-class post up as an interesting contrast.
(Before running the class)
As I continue to work on the Design and Develop Learning Resources and Design and Develop eLearning Resources subject (can anyone explain why an eLearning resource should not just be folded into an expanded definition of Learning resource?), I am now at the point where I need to work out what we will do each week.
Previous work on this has led to the development – well, adaptation really – of an assessment structure that should hopefully work well. I’m trying to incorporate as much assessment into in-class activities as possible and also get the learners to take ownership of some of their assessment by having them design the assessment criteria (while still ensuring that all the necessary assessment items are addressed). This also lets us get a flying start on the process of learning about designing and developing resources by working together on one in class. I’m thinking that using a TPACK (Technology, Pedagogy and Content Knowledge) framework to evaluate learning resources seems like a solid base at this point.
The course as it has been delivered previously seems like a very rich opportunity for our teachers to learn about using our LMS (Moodle – called eLearn here) but the more I look at the elements of competency, the more I have to wonder how relevant some of the material really is. Refocusing the course on designing and developing learning resources will have to be a priority. Topics on designing assessments and forum activities and using our learning object repository are undoubtably valuable but not relevant in this specific instance.
(After the class)
One of the things about having a more theoretical approach to teaching is that it can be very easy to get excited about trying a load of new things and using a lot of ed tech (Moodle to be precise) without really thinking through the limitations of the class.
I spend a lot of time researching approaches to teaching with technology and providing 1-to-1 support for teachers at their desks. I also run semi-regular workshops for small groups of teachers about using specific tools. What I haven’t done is taught a full subject in a proper class setting over a number of weeks – well not in the last ten years anyway.
The first week is always going to be a little bumpy – learners turning up to class who haven’t enrolled yet (or properly) and thus have no access to our eLearning platform. The other thing I sometimes forget – but really shouldn’t – is that few teachers have the same level of skill, enthusiasm or experience in using our LMS as I do. So designing the lesson for Week 1 as primarily a series of sequential activities in Moodle in the first week is probably not the ideal approach. Actually, there’s no probably about that.
Furthermore, getting learners to use new online tools that seem perfectly straightforward (Padlet) can and will take much longer than anticipated.
On top of this, I decided that it would be fun to try to gamify the course. Not hugely but using a 12 sided die to randomise the process of calling on learners to answer questions and making use of the activity restriction function in Moodle (you can’t see one activity until you complete the previous one) really does complicate an already messy session unnecessarily.
Something else that I’d decided (based on sound pedagogical principles) was that getting the students to create a resource that can be used to identify criteria in their assessment would be a useful way to engage them with the content and get them to think more meaningfully about what is important in designing and developing learning resources. On reflection, I guess creating a resource that can be used to measure the quality of other created resources gets a little meta and might be overly complicated. I should’ve also considered that these teachers would be far more interested in developing workable resources for their own students and not for themselves and their classmates.
All in all, I think I tried to do too much, too cleverly and expected far more of the students than I should’ve. I should’ve made more allowances for lower levels of e-learning and digital literacy and factored in the necessary messiness of getting everyone started.
So now I need to simplify and streamline this course. Several of the activities were successful and we did have a reasonably meaningful and deep discussion about what is important to consider in the process of designing learning resources, so I don’t consider the class to be a total wash. We also were able to identify specific learning resources that the students are interested in learning about – several of which (marking rubrics) were nowhere on my list of things to cover in this course.
So it’s back to it, I guess.
Now the focus of this project on Designing the Design & Develop Learning Resources course is on pinning down the assessments. J’s assessments for DDLR 3&4 seem strong but I just want to see whether it’s possible to streamline them slightly – largely to allow learners to knock over the analysis (and design) components quickly. (Given that they should presumably have a decent idea what their students are already like and already design resources with this in mind)
After a couple of hours of looking over this, I’m wondering whether it mightn’t have been better to try to write up my own assessment ideas first and then look at J’s for additional inspiration. It’s quite difficult to look past the solid work that has already been done. I’m still mindful of the fact that the amount of documenting and reporting seems a little high and am trying to find ways to reduce this while still ensuring that the learner addresses all of the elements of competency.
One of the bigger hurdles I face with this combined subject is that the elements of the units of competency are similar but not the same. For the analysis and design sections, they match up fairly well, with only mild changes in phrasing but the development, implementation and evaluation components start to differ more significantly. Broadly speaking, both of these units of competency appear to be targeted more at freelance education designers than practicing teachers – the emphasis on talking to the client and checking designs with the client (when the teacher would clearly be their own client) requires some potentially unnecessary busy work for the teacher wanting to be deemed competent here.
I’ve tried to address the differences between the elements of competency by clustering them with loosely matching ones from the other unit of competency in this fairly scrappy looking document. I’ve also highlighted phrases that look more like deliverable items.
This made it much easier to look over the existing assessment documents and resources to firstly check that all of the elements were addressed and secondly to feel confident that I am sufficiently across what is required in this subject.
Broadly speaking, the existing assessment items cover these elements of competency pretty well, I only needed to add a few extra questions to the design document template to address some aspects that it might be possible for learners to overlook.
These questions are:
So as it stands, I think I’ll be largely sticking to the existing assessment plan with only a few minor changes. (Largely because my predecessor knows her stuff, which has been tremendously helpful). I am still keen to find ways to address as much of this assessment as possible in class activities – being mindful of the fact that learners may not make every class and there needs to be a certain amount of flexibility.
Overall though – and clearly the dates will need to be changed, this is what the assessments look like.
The next step is to update the subject guide and add my amendments to the existing documents. I do also need to devise a marking guide for the learning resources themselves – there is something appealing in the idea of having the learners create this as one of their draft resources as the unit of competency document does stretch to define learning resources as including assessment resources too. This seems like a great opportunity to get the learners thinking more critically about what makes a good learning resource.
Today is all about pinning down the most appropriate types of assessments for this subject. Yesterday I think I got a little caught up in reviewing the principles of good assessment – which was valuable but it might also be better applied to reviewing and refining the ideas that I come up with.
For what it’s worth, these are the notes that I jotted down yesterday that I want to bear in mind with these assessments. DDLR Assessment ideas
Looking over the four versions of this subject that my colleague J has run in the last 2 years has been particularly enlightening – even if I’m not entirely clear on some of the directions taken. The course design changed quite substantially between the second and third iterations – from a heavily class-based activity and assessment focus to more of a project based structure. (For convenience I’ll refer to the subjects as DDLR 1, 2, 3 and 4)
DDLR 1 and 2 provide an incredibly rich resource for learning to use eLearn (our Moodle installation) and each week is heavily structured and scaffolded to guide learners through the process of developing their online courses. The various elements of the units of competency are tightly mapped to corresponding activities and assessment tasks – moreso in DDLR 2. (Image from the DDLR subject guide)
I have to wonder however whether the course provides too much extra information – given the relatively narrow focus on designing and developing learning resources. Getting teachers (the learner cohort for this subject) to learn about creating quizzes and assignments in Moodle is certainly valuable but are these truly learning resources? This may well be one of the points where my approach to this subject diverges.
The shift in approach in DDLR 3 and DDLR 4 is dramatic. (As far as a diploma level course about designing learning resources might be considered dramatic, at least.) The assessments link far more closely to the units of competency and all save the first one are due at the end of the subject. They are far more formally structured – template based analysis of the target audience/learners, design documents, prototypes and finished learning resources, as well as a reflective journal.
It does concern me slightly that this subject has a markedly lower rate of assessment submission/completion that the two preceding ones. That said, this subject is often taken by teachers more interested in the content than in completing the units of competency and that may just have been the nature of this particular cohort.
This new assessment approach also seems far more manageable from a teaching/admin perspective than the previous ones, which required constant grading and checking.
My feeling is that this is a more sustainable approach but I will still look for ways to streamline the amount of work that is required to be submitted.
The next step was to map the various elements of competency to assessment items. The elements for both units of competency are written differently enough to need to be considered separately (unfortunately) but they both still broadly sit within the ADDIE (Analyse, Design, Develop, Implement, Evaluate) framework. ADDIE seems like a useful way to structure both the course and the assessments so I have mapped the elements to this. I have also highlighted particular elements that are more indicative of outputs that might be assessed. Working through the analysis process will be quite dry (and could potentially come across as slightly patronising) so finding an engaging approach to this will be important.
(I’m also quite keen to bring digital badges into this process somehow, though that’s a lower priority at the moment)
Finally, I had a few ideas come to me as I worked through this process today that I might just add without further comment.
DDLR / DDeLR ideas
Get the class to design and develop a (print based? ) checklist / questionnaire resource that might be used to address DDLR 1 and DDeLR 1 UoCs. Get someone else in the class to use it to complete their Analysis phase.
Can I provide a range of options for the forms the assessment/resource pieces might take?
Try to develop a comprehensive checklist that teachers can use on the resources that they produce to raise the quality overall of resources at CIT. (Again, this could be a student led tool – the benefit of this is that it makes them think much more about what a good resource requires – does this meet any UoCs??)
Convert the print based Analysis document into a web resource – book tool or checklist maybe? Also possibly fix the print based one first – from a deliberately badly designed faulty version. (Lets me cover some readability / usability concepts early)
How much of this subject is leading the learners by the hand? How much is about teaching them how to use eLearn tools?
Could one of the learning resources be about developing something that teaches people how to use a particular eLearn tool???
Need to identify what kinds of resources teachers can make. Good brainstorm activity in week 1.
Think about the difference between creating a learning resource and finding one and adding it to your course. (Still important but tied to the UoC?)
If I give teachers the option to use previously developed resources (authenticity issues??), they should still provide some kind of explanatory document AND/OR edit the resource and discuss what changes they made and why.
Need to consider the relative strengths and weaknesses of the various types of tools.
In-class feedback of learning resources to better support the evaluation and implementation based elements of competency.
One activity (possible assessment) could be for learners to gather information needed to do an analysis from a partner in the group. (and vice versa) Might lead to a more critical examination of what information is being sought. Learner might even provide suggestions for design/development?
Sample resources?
In three weeks time (Friday 17/10) I’ll start teaching the Design and Develop Learning Resources (DDLR) and Design and Develop eLearning Resources (DDeLR) subjects for the Diploma of Vocational Education and Training. (Dip VET)
It’s been ten years since I last taught a formal subject. (But I’ve run a bucket-load of workshops and provided a lot of 1-1 training and support in that time)
I thought it might be a useful process to document my process as I continue to design and develop this course over the next three weeks. Obviously I’ve already spent a fair amount of time looking over the units of competency (linked above) – the holy documents within VET that define exactly what a learner needs to be able to demonstrate at the end of the course. These also outline the types of evidence that can be used to demonstrate competency and provide additional information about suggestions about interpreting the elements that make up the units of competency.
(As a side note, I know a number of people in Higher Education – the university sector – that shudder when they hear competency mentioned but it has been interesting to note how frequently it does seem to be coming up in discussions of the future of adult learning lately)
I’ve also spent a decent amount of time looking over the courses designed and delivered by my colleagues and dug down into the approaches that they have taken – as well as having long chats with them and other people on my team. (The logical thing to do would be to just tweak and re-deliver their old course but where’s the fun in that?)
There is also a practical consideration in refreshing my own course design and development skills. I have even toyed with the idea of trying to gamify the entire course but that seems unnecessarily over-ambitious. Maybe next year, when I have a better sense of how this subject runs in a conventional form.
So I started by blocking out exactly what it is that I need to accomplish in these three weeks. (Clearly other things will arise that will take priority but it is currently holidays so I have two weeks – boss free also – to get stuck into things like this with hopefully minimal disturbance) The final week is left free for feedback, editing and contingencies.
Assessment seems the logical place to start as I know what I need to cover, I just need to make sure that the learners provide enough evidence that they know it as well.
As with all good lists, the first item is to make the list. Nice to get a quick win on the board. From there it seemed prudent to revisit the subject I recently took as a student about assessment for some inspiration and that has given me some handy tools and processes that I might have eventually arrived at myself but not nearly as quickly. Once I’ve designed the assessments I think I’ll come back to this to make sure I haven’t missed anything vital.
Ensuring that the assessments are targeted at the appropriate Australian Qualifications Framework (AQF) level – 5 in this instance – and that every element of both the units of competency are addressed are the key factors here.
Taking another look at the assessments that the previous teacher of this subject – Jo – designed comes next. She’s run this subject four times now and so has had a good opportunity to refine her assessment tools. Being part of the Education Design and Technology team, we all maintain high quality online courses and Jo has invited me to make use of anything I find in her courses. (Thanks Jo – with four courses and subject guides to pore over, I might be some time)
Slides from my presentation at the ANU CAP innovation showcase about using game elements in teaching
Maybe you should get them playing on their phones in class then.
I ran a small session this morning with some of our teachers from Accounting and Law about Kahoot – a great free online quiz game.
Learners simply visit http://kahoot.it on their smart phone/tablet/laptop/computer and enter the PIN associated with your quiz game. (Which you are showing through the projector)
They then choose a nickname to use.
Questions appear as your can see in the image above. There is a timer on the side and once everyone has answered (or the timer runs out) the answer is revealed
Points are giving for getting the answer right and also for the speed of answering. At the end there is a final leaderboard and you can download a spreadsheet of results.
This can be a fun and quick way of seeing which areas of content your students have understood and which they might need more support with.
Setting up a Kahoot quiz is also very straight-forward – everyone in the session had a playable quiz game up and running within ten minutes from scratch.
Just go to http://getkahoot.com to set up a free account and get started.
(Yes, looking back, this reads like an ad but I have nothing to do with Kahoot, I just think it’s cool)
On the weekend I wrapped up studying Negotiation and Conflict Resolution, a free 4 week/module online course offered through Open2Study
Open2Study is the MOOC* offering of Open Universities Australia, a more conventional provider of free online education and training.
Structurally, Open2Study courses use a fairly standard framework – four modules composed of ten short videos (4-10 mins) of a presenter talking to camera. After each video is a single multi-choice question and after the final video is a ten multi-choice question assessment quiz covering the content from the module. Discussion forums are also provided but not supported/moderated by the person delivering the course as far as I can see and links to additional reading/resources are also made available.
In terms of rich educational design, there’s something left to be desired in this model but at the same time, it’s a free resource drawing on the skills and knowledge of some talented people and it does provide a decent introduction to the topic.
Negotiation and Conflict Resolution was developed and presented by Dr Andrew Ley of the Macquarie University Graduate School of Management. The first two modules focus on negotiation, the third touches on conflict resolution (with a slightly heavy leaning toward understanding the origins of conflict at the expense of practical solutions in conflict resolution) and the final module focuses on communication skills.
Dr Ley is an engaging and clearly knowledgeable presenter and the videos make decent (and sparing) use of basic text and the occasional image to illustrate key points. The videos are delivered via YouTube and make effective use of the Interactive Transcript functionality. This enables learners to quickly scan the content of the video and click the text to skip to particular sections of the video.
This is an effective way of dealing with the difficulties in scanning the information presented in videos.
The discussion forums were scantly used – Dr Ley did suggest that learners discuss particular questions in the forums at several points in the later videos but it was left to the learners to start these posts and subsequently this tended not to happen. A more effective strategy for using the forums would’ve seen either Dr Ley or an Open2Study moderator facilitating discussion here by raising an initial question and providing responses to interesting student comments (while not dominating the thread). Wrapping up the discussion with a summary of the thread and/or further issues to consider would also be good practice.
A PDF with references for further reading is provided for each module, however it seems like an oversight not to provide these as URLs where possible.
Overall I’m glad to have the opportunity to access this level of training for free and at my leisure and if you are interested in gaining a richer understanding of some of the key principles and strategies of Negotiation and Conflict Resolution, it is well worth signing up for this course. The next intake starts on 15/09/14 at https://www.open2study.com/courses/negotiation-and-conflict-resolution
From an educational design perspective, it’s adequate but there is room for improvement – but hey, it’s free, university level education.
Next up I’ll be checking out the Open2Study User Experience for the Web course, so here’s hoping some of those lessons flow through to this blog.
*I say MOOC because this has come to be the default usage of the term but my personal feeling is that true MOOCs should have a far richer learning model than content/quiz/content/quiz
You must be logged in to post a comment.