Part 1
List of Tables....................................................................................................................... 4
Introduction......................................................................................................................... 5
Industrial Experience
Projects........................................................................................... 5
WIER................................................................................................................................... 5
Purpose of the
Evaluation................................................................................................... 6
Research Questions............................................................................................................ 6
Research questions
(students)........................................................................................... 6
Research questions
(supervisors)...................................................................................... 7
Evaluation Methodology..................................................................................................... 8
Rationale.......................................................................................................................... 8
Evaluation Framework..................................................................................................... 8
Participants...................................................................................................................... 9
Data Collection Tools....................................................................................................... 9
Evaluation Method......................................................................................................... 10
Data Collection............................................................................................................... 11
Survey and Poll
Descriptions.......................................................................................... 13
Summary of Results........................................................................................................... 14
Limitations of Research.................................................................................................... 15
Implications and
Suggestions............................................................................................ 16
Conclusion and Further
work........................................................................................... 16
References........................................................................................................................ 17
Appendix 1........................................................................................................................ 18
Survey 1 Part A
(students): Weeks 4 – 7 Semester 1....................................................... 18
Survey 1 Part A:
Students’ comments on usability of WIER............................................. 19
Appendix 2........................................................................................................................ 21
Survey 1, Parts A and B
(students): Weeks 4 – 12 Semester 1......................................... 21
Survey 1 (Part B):
Students’ comments on usability of WIER.......................................... 22
Appendix 3........................................................................................................................ 23
Poll 2 (students):
Weeks 9 - 12....................................................................................... 23
Appendix 4........................................................................................................................ 24
Survey 2 (supervisors):
Week 9 Semester 1 – Week 5 Semester 2................................... 24
Survey 2: Supervisor’s
comments on usability and effectiveness of WIER...................... 26
Appendix 5........................................................................................................................ 27
Survey 3 (students):
Weeks 3 – 7 Semester 2.................................................................. 27
Survey 3: Students’
comments on usability of WIER........................................................ 28
Appendix 6........................................................................................................................ 30
Survey 4 (students):
Weeks 3 – 7 Semester 2.................................................................. 30
Students’ comments
about the IE project subject and their experience of group work... 36
Table 1 Schedule for On-line Polls and Surveys.................................................................. 12
Table 2. Students' Ratings of Usability of WIER................................................................... 18
Table 3. Students' Ratings of the Usability of WIER
in Semester 1....................................... 21
Table 4. Students' Ratings of Usefulness of WIER
Resources............................................... 23
Table 5. Supervisors Nominations of Project Phases
for which WIER is the Most and Least Useful for Students 25
Table 6. Supervisors' Ratings of the Effectiveness of WIER for Project
Management.......... 26
Table 7. Comparison of Students' Ratings of Usability of WIER in Semester 1
and 2........... 27
Table 8. Students' Ratings of the Usefulness WIER Resources............................................. 31
Table 9. Students' Ratings of the WIER Site......................................................................... 31
Table 10 Variables producing a significant impact as
follows............................................. 32
Table 11. Students' Ratings of the IE Project Organisation................................................. 32
Table 12. Students' Ratings of the Seminars......................................................................... 32
Table 13. Students' Ratings of IE Project Workloads........................................................... 33
Table 14. Students' Ratings of IE Project Assessment........................................................... 34
Table 15. Students' Ratings of the Experience of Group Work............................................. 34
Table 16. Students' Ratings of their IE Project Client.......................................................... 35
Table 17. Students' Ratings of their IE Project Supervisor................................................... 35
Table 18. Students' Ratings of the IE Project Coordinators................................................. 35
Table 19. Students' Ratings of the IE Project Subject.......................................................... 36
Of critical importance in the provision of Web-based learning environments is establishing their usability for the target audience, and effectiveness in terms of meeting the educational objectives. However, this important task in the development of educational technology is often poorly done or neglected. A recent survey of 104 educational projects that made significant use of computer technology found the evaluation of learning technology is limited and rarely determines the validity of the innovation in terms of its educational context (Bain, 1999).
An integrated Web-based learning environment, designed for students working on Industrial Experience Projects, has been developed by staff in the Faculty of Information Technology. The website, which is known as WIER (Web Industrial Experience Resources Web), was developed during 2000 and first used by students in 2001. An evaluation of student and staff usage of and reactions to the WIER site was performed from March through November 2001 by teaching and research staff in the School of Computer Science and Software Engineering.
The purpose of this report, which is presented in two parts, is to describe the design of the process that was used to evaluate WIER and to present the results of that evaluation. Part 1 of the report, presented in this document, describes the evaluation process and presents results of data gathered from surveys. Part 2 of the report presents results of the analysis of log files.
Students in their final year of the Bachelor of Computing degree at Monash University undertake an industrial experience (IE) project in which they design, develop, and deliver a small computer system for a client. The project runs for two consecutive semesters and comprises one quarter of the normal workload for the final year of this degree program. Students are organised into groups of four or five for their IE project work. This provides them with real-life experience in project co-ordination and management. Each project group is allocated a supervisor who is usually a staff member, however sometimes supervision is done by postgraduate students or people working in industry. The supervisors provide guidance and advice as needed. They monitor the progress of their group and are responsible for a major part of their assessment. The coordination of the IE project groups and supervisors is managed by two IE project group coordinators. A fuller description of the IE project subject can be found at (Hagan, Tucker, & Ceddia, 1999)
The WIER Web site was developed to provide students with an integrated learning environment to use during their industrial experience (IE) project work. The site provides general information, a calendar of events, facilities for project management including time tracking facility and document management, and various forms of communication via news groups and a discussion forum. The site also provides access to a repository of resources including standards documents, document templates and samples of past projects. The WIER site is also used by IE project group supervisors to monitor the progress of their students and by the IE project coordinators who manage the IE project subject.
The WIER Web site was designed by staff in the School of Computer Science and Software Engineering in collaboration with staff from the School of Network Computing and the Gippsland School of Computing. The site was developed during 2000 with funding obtained from a Technology Innovation Fund (TIF) grant from the Faculty of Information Technology. The WIER Web site is located at: http://csse.monash.edu.au/WIER/. Access to most facilities on this site is password restricted (Ceddia & Tucker, 2001). More details about the WIER site can be found in (Ceddia, Tucker, Clemence, & Cambrell, 2001)
The evaluation of the WIER site aimed
to determine how the students and supervisors used the site, determine their
reactions to the site in terms of its usability and usefulness, and establish
its effectiveness as an integrated learning environment to assist in IE project
work.
The
purpose of the evaluation was to provide feedback to:
·
the designers
and developers of the WIER site
· the coordinators of the IE Project subject
It was intended that the evaluation would highlight problems which could be dealt with immediately, and provide information and suggestions for future enhancements or modifications. Furthermore, the students’ usage and reactions to the site would provide information for educators planning instruction in the use of the site, or development of resources or facilities for the site.
For the evaluation of WIER a series of research questions were designed to provide information about:
· site usage
· usability (general, and for specific resources and facilities)
· usefulness (general, and for specific resources and facilities)
· effectiveness of the project management tools provided by the site
Separate research questions were addressed for students and the supervisors as detailed in the following sections.
1. How do the students use the WIER site?
a) Frequency of use
b) Time of day accessed
c) Time spent at site
d) Times spent on particular tasks
e) Resources accessed
f) Tasks attempted or completed
g) Activities attempted or completed within a task
2. How do the students rate the usability of the WIER site?
a) Interface
i. Navigation (navigation aids e.g. menu)
ii. Orientation (user control, orientation aids e.g. page labels)
iii. Ease of finding information
iv. Ease of use (e.g. scrolling)
v. Consistency
vi. Readability – use of graphics, colours, fonts etc
vii. Page layout
viii. Response time
b) Specific resources (ease of downloads, ease of use etc)
i. Time tracker
ii. File manager
iii. Document templates
iv. Sample documents
v. Discussion groups
vi. Individual group forum
vii. General news system
viii. Group news system
ix. Past projects
x. Calendar of events
3. How do the students rate the usefulness of the WIER site?
a) General aspects of site
i. Relevance of information
ii. Accuracy of information
iii. Currency of information
b) Specific resources
i. Time tracker
ii. File manager
iii. Document templates
iv. Sample documents
v. Discussion groups
vi. Individual group forum
vii. General news system
viii. Group news system
ix. Past projects
x. Calendar of events
4. How useful is the WIER site for project management?
a) Task scheduling
b) Time management
c) Communication with other group members
d) Preparing documentation
5. What are students’ reactions to the site?
e) Degree of learner control (freedom/dictate)
f) Motivation to use
g) Ease of use (anxiety/boredom)
1. How do the supervisors use the WIER site?
a) Frequency of use
b) Resources accessed
2. How do the supervisors rate the usability of the WIER site?
a) Navigation
b) Orientation (user control)
c) Ease of finding information
d) Ease of use (e.g. scrolling)
e) Consistency
f) Readability – use of graphics, colours, fonts etc
g) Page layout
h) Response time
3. How do the supervisors rate the usefulness of the WIER site?
a) For group work
b) For project management
c) For specific project phases
Assessing the effectiveness of a Web-based learning environment is a complicated task. The difficulty of evaluation of such complex environments is exacerbated by a scarcity of practical guidance as to how this should be done (May, 2001). Alexander (Alexander, 1999) stresses that evaluation of a learning technology should be performed at all phases of the development process (design, development, implementation and institutionalisation). There is some discussion in the literature about appropriate data collection and methods that should be employed at each phase. A framework outlined by Bain (Bain, 1999) proposes different evaluation processes that are appropriate for each phase.
The evaluation process detailed in this document describes the evaluation of WIER at the implementation phase of the development of the site. According to the evaluation framework proposed by Bain (Bain, 1999), the foci of evaluation at this stage is summative and aims to establish the educational appropriateness of the product and whether the learning outcomes are as intended. This is determined from peer review, formative monitoring of student involvement and experiences with the environment, and student learning outcomes. In order to do this it is necessary to collect data from a number of sources, generally using different methods. This forms what may be termed an eclectic mixed-methods pragmatic paradigm as described by (Phillips, Bain, McNaught, Rice, & Tripp, 2000) (Phillips et al., 2000). This allows for a “mixed approach to data production and analysis, with both qualitative and quantitative information obtained in the evaluation process” ((Phillips et al., 2000).
The WIER site is a multifaceted site, providing many features and integrating facilities that can be used by multiple types of users: students, supervisors, clients and coordinators. To perform a rigorous evaluation of the different aspects of the WIER site identified in the previous section, data was collected from a number of sources. The evaluation used a combination of data gathering from log files, survey methods and student results. The log files provided quantitative data on usage of the site and the surveys provided a combination of quantitative and qualitative data about reactions and experiences with the site.
The WIER site was evaluated in terms of its functionality, usability and influence on project management, by gathering reactions of the principal users of the site, namely, students and supervisors. The evaluation aimed to determine problems or difficulties with the site that could be addressed immediately or in longer term revisions, and also determine the effectiveness of the site for its intended purpose of assisting with project management. This involved a combination of formative and summative evaluation processes (May, 2001). The formative evaluation was conducted when the students were using the site for their IE project work; the summative evaluation was conducted when they had completed their IE project.
The choice of methods available to use for the evaluation was limited. It was not possible to use an experimental method with a control group as the students were using the site as an integral part of their IE project work. The subject coordinators wanted to use a method that was the least intrusive and disruptive to the students as possible. Furthermore, the evaluation was to be carried out by academic staff with limited time and resources. Considering all these issues, it was decided that surveys administered online and data collected on log files would be the most efficient way of collecting data. Performing the evaluation while the students used WEIR in a real context gave a high level on of authenticity to the evaluation (Oliver, 2000).
In 2001 there were 175 students enrolled in the IE project
subject, organised into 35 project groups. There were 30 supervisors, some
supervising several groups. In addition there were two co-coordinators who
managed all the project groups. This
was the first group of students, supervisors and coordinators to use the site.
The evaluation of WIER Web site was done using data collected from students and supervisors via on-line surveys and polls, paper-based surveys, and logfiles produced from students’ and supervisors’ interactions with the site.
Student interactions with the WIER site were monitored and recorded in a database. The data collected on student interactions provided general information about their usage of the WIER site. The frequency of access to the site, pages accessed, and time spent in the site gave an overall pattern of usage. Further examination of the data provided insights into particular students’ usage.
The reactions of students and supervisors to the WEIR site and their opinions of the usability and usefulness of the site were gathered from on-line questionnaires. These were in the form of polls that consisted of questions requiring single responses, and surveys that contained single-response questions and questions requiring open-ended responses. An administrative facility included in WIER allows for the dynamic creation of the surveys and polls with the specification of a date activation range and intended participants
The original intention was to collect all data via on-line surveys. The surveys were scheduled to be administered at different stages throughout the year from the WIER site. However, due to the low response rates for these surveys it was decided to use a paper-based questionnaire for the last survey at the end of the year.
The on-line surveys and polls were published on the WIER site over the course of the year, during the dates specified in the schedule in Table 1. The students were notified about each survey at their weekly seminars; supervisors were notified by email. The surveys were accessed from the WIER site when the user logged on. The survey was not shown one it had been responded to or it was out of date. Each survey and poll were presented only once to each student or supervisor. Participation was voluntary, however an incentive scheme was set up to encourage students to respond. Students who responded to a poll or survey collected karma points. Karma points were also earned for using the WIER site. The names of the students with the highest number of karma points were published on the WIER site continuously throughout the year. However, in spite of this incentive scheme and continual reminders, both on-line and face-to-face, the response rates were low for all the on-line surveys.
The evaluation will be described in terms of the separate phases of formative and summative evaluation.
The student and supervisor interactions with the WIER site were collected from Week 4 in Semester 1 to the end of Semester 2. . This aspect of the data collection formed a naturalistic enquiry. The paradigm of naturalistic observation describes the situation where data is collected in a natural setting unnoticed by the learner. The details were stored in an Access database.
The data stored in the logfiles provided information on site usage. This was used in the formative evaluation stage to provide basic details of students’ and supervisors’ usage of the site. Knowledge of the frequency and extent of student use of the site helped the IE project coordinators plan their education program and make adjustments to the survey schedule.
A more detailed analysis of the student usage of the site was done as part of the summative evaluation. The details of the interactions collected were analysed in Part 2 of this evaluation and are reported in a separate document.
Two on-line surveys were used to determine student and supervisors ratings of the usability of WIER. These surveys were used in the formative part of the evaluation of WIER. The first survey, presented to students and supervisor early in Semester 1, gathered initial reactions to the site. We felt it was essential to determine, as early as possible, any difficulties students were experiencing with the site and, if possible, make modifications to the site or provide extra instruction in its use. The second survey, presented to students in mid Semester 2, gathered their opinions of the site towards the end of their IE project. As only two supervisors completed the first survey it was decided not to include them in the second survey.
The same questionnaires were used for each survey (with slight modification) to enable comparisons to be made and establish whether their views of the usability of the site changed with familiarity and more experience of use. The analysis of the comparison with the first survey will be used in longer-term redesign or enhancements to the site.
A poll and a paper-based survey were used to determine student ratings of the usefulness of specific facilities and resources provided on WIER.
The poll was made available late in Semester 1. It was considered that by this stage the students had had the opportunity to use most features of WIER. The poll provided a quick mechanism for the students to give a rating for each resource as part of the formative evaluation.
A paper-based survey was administered at the end of Semester 2. This provided a summative evaluation of the usefulness of the resources and a comparison with the poll in Semester 1 will be used in longer-term redesign or enhancements to the site.
As part of the summative evaluation the usefulness and effectiveness of the site for project management was gauged from the students and supervisors. Two different surveys were used to determine this.
The supervisors were surveyed at the middle of the year with an on-line survey and the students were surveyed at the end of the year with a paper-based survey.
The data collection schedule was designed to collect data for each aspect of the evaluation. This is shown in Table 1.
Table 1
Schedule for On-line Polls and Surveys
Date
|
Survey or Poll
|
Evaluation phase
|
Students
|
Supervisor
|
Semester 1, 2001 |
||||
|
Week 4 onwards (20/3->) |
Poll 1 |
Formative & summative |
Intended use |
|
|
Weeks 4 – 12 (23/3 - 20/4) Part A (5/5 - 26/5) Part B |
Survey 1* |
Formative |
Usability |
Usability |
|
Weeks 9 – 12 (1/5 – 25/5) |
Poll 2 |
Formative |
Usefulness of specific
resources |
|
|
Week 9 (Semester 1) – (4/5 – 14/8) |
Survey 2 |
Formative |
|
Usefulness for project
management |
Semester 2, 2001 |
||||
|
Weeks 3 – 7 (Semester 2) (30/7-31/8) |
Survey 3 |
Formative |
Usability (Research question 2) |
|
|
By final submission of project Paper based survey |
Survey 4 |
Summative |
Usefulness of specific
resources Project management (research question 4) |
|
* This survey was published twice due to a low response rate the first time it was published. However, each student was only able to respond to the survey once.
This poll asked the students to state what they intended to do while accessing the WIER site. This enabled their intentions to be matched with their actual activities, which were gained from the interactions captured on a logfile.
The poll was available to students each time they logged on to the WIER site. The poll remained on the WIER site all year. A total of 204 students responded over the course of the year and responses were gained from ??? different students.
This survey was presented twice during Semester 1 over a period of nine weeks. These are referred to as Part A and Part B.
The survey aimed to determine the students’ opinions of the usability of the WIER site. The students were required to rate various aspects of usability on 5 point Likert scales. In Part A provision was made for them to provide open-ended comments for each question and in Part B provision was made at the end of the survey for general comments.
The descriptions and analysis of the results for Part A of this survey are presented in Appendix 1 and the complete results for Part A and Part B are presented in Appendix 2.
This survey in the form of a poll aimed to determine the students’ opinions of the usefulness of different facilities and resources provided on the WIER site. The students were required to rate the usefullness of various resources on 5 point Likert scales.
The descriptions and analysis of the results of this poll are presented in Appendix 3.
This survey aimed to determine the supervisor’s opinions of the usefulness of the WIER site for their students, and its effectiveness as a tool for project management. The questions required supervisors to select from fixed responses or rate various aspects of the site’s usefulness for project management on 5 point Likert scales. The last question provided an opportunity for open-ended comments.
The descriptions and analysis of the results for this survey are presented Appendix 4.
This survey was similar to Survey 1. The survey aimed to determine the students’opinions of the usability of the WIER site. The students were required to rate various aspects of usability on 5 point Likert scales. . The last question provided an opportunity for open-ended comments...
The descriptions and analysis of the results for this survey are presented in Appendix 5.
This survey was similar to Survey 1. The survey aimed to determine the students’opinions of the usability of the WIER site. The students were required to rate various aspects of usability on 5 point Likert scales. The last question provided an opportunity for open-ended comments...
The descriptions and analysis of the results for this survey are presented in Appendix 6.
The following is a summary of results of the student and supervisor surveys.
In the first survey the students indicated that they were generally happy with most aspects of usability of WIER. They found the orientation within the site easy, and were satisfied with the consistency, readability, and organisation of the screen elements. However, some found it difficult to locate information and navigate the site and felt the response time was slow.
More specific information about the usability of WIER was gained from the open-ended survey responses. Most comments were made about the Timetracker facility, some students found it inflexible and difficult to use. Other comments were made about the slow response time of the site, the amount of scrolling necessary on some pages and the number of logins.
In response to these surveys the code WIER code was reviewed and several parts rewritten to try and improve the response time.
In the second survey the students indicated that the aspect of usability they had had most difficulty with was response time. Despite efforts to improve response time after the first survey the students rated this aspect of WIER significantly lower than the in the first survey. They tended to rate most other aspects of the WIER site lower, however these differences were not statistically significant and with the different sample sizes for the surveys it is difficult to interpret these results further.
The open-ended comments were consistent with comments made in the first survey. Students were unhappy with the amount of scrolling, slow response time and login mechanism. They had some suggestions for the improvement of the Timetracker.
The usability evaluation highlighted the need to provide more instruction earlier in the IE project on how to use the WIER site and also provide more explanation of the rationale for some of the functionality.
In the formative stage of the evaluation the students indicated that the most useful resources were the document templates and sample documents, and the least useful were the individual group forum, discussion groups and Timetracker. Their ratings are consistent with the amount of usage we would expect of these resources at this stage of their IE project work. One exception however is the Timetracker. All students should have been using this regularly to record their times, so it is interesting that they did not find it useful.
In response to these surveys the IE project coordinators gave further demonstrations on WIER at the IE project seminars to raise students awareness of and give further information about some of the facilities that the student were not using (e.g. Discussion Forum) or not appreciating (e.g. Timetracker).
By the time of the summative evaluation the students had had the opportunity to use all aspects of WEIR over the full lifecycle of their IE project work. These survey results present a different picture to the earlier survey. The students indicated that the task tracker, document templates, sample documents and calendar of events were the most useful facilities. These are all related to project management. They did not find the communication facilities provided were very useful. The least useful facility was the past projects, however there were not many projects available on the site at the time of the surveys. This may become a more useful resource when more projects are placed here.
The effectiveness of WIER for project management was determined as part of the summative evaluation.
The supervisors indicated that they felt WIER was effective for project management. They generally indicated that they felt their groups were managing better than previous groups they had supervised. A significant difference they noted was that their students were less dependent on them for information than previous groups.
The students also indicated that they felt the site was useful for project management. A regression model showed that the time graph, file manager, subject information and risk lists produced a significant impact on the usefulness of WIER for project management.
There were several problems experienced during this evaluation of WIER.
During the course of the evaluation it was necessary to make quite a few changes to the original evaluation design. The survey schedule was planned and questionnaires outlined at the beginning of the year, before the students commenced their project work, however this was modified quite extensively during the course of the year. There were some refinements made to the original sets of questions, however the main changes involved rescheduling and lengthening the times that the surveys were made available to the students and supervisors in an attempt to improve the response rates.
The problem of poor survey response rates became apparent with the first survey, which was republished after it was discovered that only 12.5% of the students had participated. The experience with the first survey indicated that the schedule was overly ambitious and the number and size of the surveys were reduced. .The low response rates continued throughout the year and various attempts to encourage participation had little effect. It was decided therefore to change the format of the final survey to a paper-based survey. By this stage it was felt that a paper questionnaire administered to students at their presentations would elicit a better response rate than a survey which relied on students accessing the WIER site in their own time.
The low response rates to the on-line surveys also caused concerns about the representativeness of the sample. It would be reasonable to expect that the students who responded to the survey could be the more frequent WIER users and not typical of the whole group.
We were interested in the views of the supervisors, however their low response rate meant that very little usable feedback was gained from this source.
Another problem encountered with the evaluation was caused by the complexity of the site. To comprehensively evaluate all the features available on WIER, the surveys would have been prohibitively large. Therefore, the approach taken was to rate aspects of usability, usefulness and effectiveness of the whole site rather than specific features. The general responses meant there was a chance of hidden variability as no quantitative measures of individual facilities were obtained. The only information obtained about individual facilities was from the open-ended comments and not all students responded to these.
These experiences highlighted the difficulty of collecting information on-line and the risk of relying on one source of data collection. The low response rates to the on-line surveys showed a reluctance of many students to respond to this form of survey, indicating that the low sample obtained was unlikely to be representative of the whole group. This impacted on the validity of the results obtained. Also, although the survey questionnaires were extensive, the data collected did not provide all the information required to fully evaluate all aspects of the WIER site. For example, the usability of individual resources was not evaluated.
Students also showed a reluctance to provide detailed responses to the on-line surveys. There were limited responses to the open-ended questions on these surveys. To obtain more detailed responses, other forms of data collection such as interviews should be used
The response rates and responses to the open-ended questions were increased using paper-based surveys. However, a problem using paper-based surveys is that there is a time delay between students’ use of the site and their survey responses. More immediate forms of data collection such as observation, video, and think-aloud, would potentially provide more accurate and detailed information of students’ reactions to the site.
The decision to use on-line surveys to gather information for this evaluation was a pragmatic one, however in hindsight a combination of surveys and other forms of data collection would provide a more accurate and extensive evaluation
This document reports on an evaluation of WIER a Web-based learning environment. The importance of this work is emphasised by a search of the literature, which showed that evaluation of learning technology is limited and rarely determines the validity of the innovation in terms of its educational context.
The evaluation of WIER was performed in the late development and implementation stages of the website construction. The responses from the students, while not negative, has not been as positive as had been hoped by the IE project coordinators and the developers of the site. However it must be taken into account that this was the first running of the subject and the IE coordinators were inexperienced in its use. The information gained from the evaluation has enabled the IE project coordinators to present WIER to the students with more understanding of the difficulties they may face in using the site. It is intended that an evaluation will be performed again in 2002. This evaluation will place more emphasis on paper-based surveys, and other forms of gathering data will be explored.
S. Alexander. (1999). An evaluation of innovative projects involving communication and information technology in higher education. Higher Education Research & Development, 18(2), 173-183.
J. D. Bain. (1999). Introduction. Higher Education Research & Development, Special Issue: Learning centred evaluation of innovation in higher education, 18(2).
J. Ceddia, & S. Tucker. (2001). Industry Projects. Available: http://www.csse.monash.edu.au/~wier/ [2001, 12 June 2001].
J. Ceddia, S. Tucker, C. Clemence, & A. Cambrell. (2001). WIER - Implementing artifact reuse in an educational environment with real products. Paper presented at the 31st Annual Frontiers in Education Conference, Reno, Nevada.
D. Hagan, S. Tucker, & J. Ceddia. (1999). Industrial experience products: a balance of process and product. Computer Science Education, 9(3), 106-113.
A. J. May. (2001). User-centred evaluation of complex IT&T within an operational environment. Journal of Research and Practice in Information Technology, 33(1), 83-97.
M. Oliver. (2000). An introduction to the evaluation of learning technology. Educational Technology & Society, 3(4).
R. Phillips, J. Bain, C. McNaught, M. Rice, & D. Tripp. (2000, 7 December 2001). Handbook for Learning-centred Evaluation of Computer-facilitated Learning Projects in Higher Education. Phillips, Rob. Available: http://cleo.murdoch.edu.au/projects/custd99/handbook/handbook.htm [2002, 21/1/2002].
This survey was available for three weeks. Only 22 students out of a possible 175 responded to the survey giving a response rate of 12.6%. It is interesting to note that the usage log showed that during the period of the survey most students had accessed WIER.
The survey asked to rate the usability of the WIER site. It consisted of eight questions each requiring the students to rate an aspect of usability on a 5 point Likert scale. The results are shown in Table 2. The students were also given the opportunity to provide open-ended comments on each aspect of usability. These are shown on page 19.
The highest usability rating was for the consistency of the site and there was high agreement amongst the students about this. There were only three comments from the students about the site consistency and these were all positive.
The lowest usability rating was for the response time of the site, however there was the highest variation in the students’ opinions about this. Their open-ended comments indicated that the response time varied and was very slow at times. However one student had found no problems, even when accessing WIER from home with a slow connection.
Table 2. Students' Ratings of Usability of WIER
Question
|
N
|
Mean
|
SD
|
|
1.
How easy is it navigate the WIER site? |
22 |
2.7 |
0.9 |
|
2.
How easy is it know where you are within the WIER site at
any time? |
22 |
2.4 |
1.0 |
|
3.
How easy is it to locate information on the WIER site? |
22 |
2.6 |
0.9 |
|
4.
How easy is it to view information on the WIER site e.g.is
there too much scrolling? |
22 |
2.7 |
1.0 |
|
5.
How would you rate the consistency of the page layout i.e.
does each page have the same look &
feel? |
22 |
1.7 |
0.8 |
|
6.
How would you rate the general layout of the pages in terms
of the organisation of the screen
elements e.g. menus, text areas? |
22 |
2.4 |
1.1 |
|
7.
How would you rate the readability of WIER pages in terms of
choice of colours, fonts, graphics, etc? |
21 |
2.4 |
0.8 |
|
8.
How would you rate the response time of the WIER site? |
22 |
2.9 |
1.3 |
The following comments were provided by students on each aspect of usability of WIER.
How easy is it navigate the WIER site?
· why do you have logins everywhere? I mean, you get one in the main page and one when you click on students without login in b4.
· all together a pretty good website. could do with some editing for the risk list and time/task breakdown... people do make typing mistakes! =)
· it is very tedious and takes up a lot of time to get around..and I have to be very certain of what to do without being able to try it out. coz certain things can’t be deleted. i dun find it that user friendly at all
· Poor time logging event. If even a small error has been created, it is impossible to change that. Atleast provide the flexibility to correct spelling mistakes.
· quite good
· Doon’t find anything challenging about it.
· A bit difficult at first, but you get used to it
· many important links are hidden. Eg. Why is the different between the [Students] and the [Students > Homepage] link?? Is the same. Another example; after I add my task, I have a click back or home and click task again to view the task table again.
How easy is it know where you are within the
WIER site at any time?
· okay only
· Some areas are more clear than others.
· I never know where I am. Eg. Currently, I know I am answering a survey, but I don’t know where my location. the title only show "Survey", but where I am?
· Easy =) no need to go anywhere else, all you need is mainly in one page
How easy is it to locate information on the WIER site?
· At first glance i had no idea where things were. but after the seminars, well everything is easy to find =)
· Good
· Everything can be accessed from the menu on the left.
· I was lost the when I first use this website. I was lost two the second time. A good website will never make one lost -- even if he/she is a first time user. A site map should do a good job here.
How easy is it to view information on the WIER site e.g. is there too much scrolling?
· Give you that one. LOTS of scrolling Also for the time tracker, is so SLOW to load back the page after you entered one time. So i think is better that the login time page appears and loads on a new window and not on the current
· too much scrolling; when in task tracker, if you add time to a task, it closes all tasks on your return to the task tracker page.
· can’t really pinpoint this one, but reading through the discussion page can get rather annoying with all those boxes... maybe have like a threat instead for all the replies?
· If there was less scrolling then there would be too much button clicking. It is at a happy medium.
· Keep the task/time tracker expanded even after you have logged a time, if you have chosen expanded to begin with
· Eg. The forum [Site bug]; the scrolling is quite long. Also, the task table, the viewing is really bad. It doesn’t have any good sorting system thus it make everything look every messy. Anyway, the font size is good. I like it :)
How would you rate the consistency of the page layout i.e. does each page have the same look & feel?
· Kewl
· every page is exactly the same.
· The consistency is good :)
How would you rate the general layout of the
pages in terms of the organisation of the screen elements e.g. menus, text
areas?
· Time Tracker page is a pain to read when you have time allocated
· unable to edit the time log especially when the time spend and date has been wrongly typed
· I like the layout. Nothing special.
· The use of DHTML or JavaScript(Didn’t really check the source code) menu navigation is a bad idea in my opinion. It can never work properly with Netscape. The organisation is okay but there are plenty of room for improvement. Eg. Where is the file manager? -- it’s located under Students > Homepage; the title doesn’t make any sense. Project Administration or Shared Project Resources can be a better name. As for the text areas, I think making it larger will be good. For instant, the post reply in forum. The subject text box viewable area is very small - only 20 chars.
How would you rate the readability of WIER
pages in terms of choice of colours, fonts, graphics, etc?
· Everything is good except for the time tracker. HEHE i know you can’t fix it but then you asked my opinion =)
· Blue on white. Can’t get much easier.
· Some colours make the links hard to read. Eg. The left navigation itself is not easy to read. The work is in dark blue and the background the black. Previously, the font colour in posting a reply in forum is really bad. It was fix and it’s good looking now :)
How would you rate the response time of the
WIER site?
· This site is really slow, specially the day before things like the progress summary is due
· sometimes very slow. Maybe you need a new server?
· sometimes its really slow and takes ages..
· I have never had any problem connecting even on my slug connection at home.
· Depends on when and where you are I guess, but my last experience using the time tracker, it took forever to load up everytime you wanted to enter a new time
· The support is good :)
This survey was available initially for three weeks in Semester 1 (Part A). Only 22 students of a possible 175 responded, although the usage log shows that during the period of the survey most students had accessed WIER. The survey was then published for a second time (Part B), and a further 16 students responded. This raised the overall response rate to 22%.
The survey asked the students to rate the usability of the WIER site. It consisted of eight questions each requiring the students to rate an aspect of usability on a 5 point Likert scale. The results are shown in Table 3. The students were also given the opportunity to provide open-ended comments. For Part A students were able to comment on each aspect of usability. These are shown on page 19. For Part B general comments were requested. These are shown on page 22.
The mean values for the usability ratings were all below 3, the neutral mid-point value of the 5 point scale, indicating a tendency towards finding each aspect of the site excellent and the usability easy. The means values were compared to the mid-point value using one sample t-tests. These showed statistical differences for four aspects of usability, indicating that the students tended to find the orientation within the site easy, and were satisfied with the consistency, readability, and organisation of the screen elements.
The most negative responses were for navigation, locating and viewing information and the site response time however these were not statistically significantly different from the neutral mid-point of the scale.
Table 3. Students' Ratings of the Usability of WIER in Semester 1
Question
|
N
|
Mean
|
SD
|
t
|
|
1.
How easy is it navigate the WIER site? |
38 |
2.9 |
1.0 |
-0.78 |
|
2.
How easy is it know where you are within the WIER site at
any time? |
38 |
2.6 |
1.0 |
-2.43 * |
|
3.
How easy is it to locate information on the WIER site? |
38 |
2.9 |
1.0 |
-0.41 |
|
4.
How easy is it to view information on the WIER site e.g.is
there too much scrolling? |
38 |
2.9 |
1.2 |
-0.58 |
|
5.
How would you rate the consistency of the page layout i.e.
does each page have the same look &
feel? |
38 |
2.0 |
0.9 |
-7.03 * |
|
6.
How would you rate the general layout of the pages in terms
of the organisation of the screen
elements e.g. menus, text areas? |
37 |
2.6 |
1.0 |
-2.67 * |
|
7.
How would you rate the readability of WIER pages in terms of
choice of colours, fonts, graphics, etc? |
37 |
2.3 |
0.9 |
-5.06 * |
|
8.
How would you rate the response time of the WIER site? |
38 |
2.9 |
1.3 |
-0.51 |
|
* p < 0.05 |
||||
The following comments were provided by students on the usability of WIER:
· It would be good if the students and client has a chat room or a forum, where they could post up documents to discuss.
· i click back, and then it says can\'t load page.
· then i click on students, and the page loads (but this is probably a thing with php).
· response is better in uni than at home
· i think the details of each groups project outline n contact details should be accessible by everyone... this would actually help in terms of resource and knowledge sharing... this method would prove very effective as this would actually improve the quality of projects produced...
This poll was available for four weeks. Seventy-seven students of a possible 175 responded to the poll. This gives an overall response rate of 44%.
For this poll the students were asked to rate the usefulness of ten resources provided on the WIER site. The students rated each resource on a 5 point Likert scale , where 1 = useful, 5 = useless). The results are shown in Table 4.
The most useful resources were the document templates and sample documents, the least useful were the individual group forum, discussion groups and time tracker.
The mean values for the usefulness ratings were compared to the value 3, the neutral mid-point value of the 5 point scale, using one sample t-tests. These showed statistical differences for five resources, indicating that the students tended to find the file manager, document templates, sample documents, general news system and calendar of events useful.
Table 4.
Students' Ratings of Usefulness of WIER
Resources
Resource
|
N
|
Mean
|
SD
|
t
|
|
1. Time Tracker |
77 |
2.9 |
1.5 |
-0.86 |
|
2. File Manager |
77 |
2.7 |
1.4 |
-2.12 * |
|
3. Document Templates |
76 |
2.5 |
1.5 |
-3.01 * |
|
4. Sample Documents |
76 |
2.4 |
1.5 |
-3.37 * |
|
5. Discussion Groups |
75 |
2.9 |
1.1 |
-0.51 |
|
6. Individual Group Forum |
76 |
3.0 |
1.2 |
-0.20 |
|
7. General News System |
76 |
2.7 |
1.2 |
-2.17 * |
|
8. Group News System |
76 |
2.8 |
1.1 |
-1.69 |
|
9. Past Projects |
76 |
2.8 |
1.4 |
-1.28 |
|
10. Calendar of Events |
77 |
2.6 |
1.4 |
-2.43 * |
|
* p < 0.05 |
||||
This survey was available for fourteen weeks. However in that time only 12 supervisors out of a possible 30 responded, giving an overall response rate of 40%.
The survey asked the supervisors to rate the usefulness of the WIER site for their students and its effectiveness as a tool for project management. The survey consisted of twelve questions. Seven of the questions required supervisors to evaluate the effectiveness of the WIER site for project management. The results are shown in Table 5. The last question provided an opportunity for open-ended comments. These are shown on page 26.
1. How many IE groups have you previously supervised?
Eleven of the supervisors had experience with IE group supervision. Most (7) had previously supervised more than five groups and four had supervised from three to five groups; only one had supervised less than three groups.
2. At what project phase(s) is your group currently at?
The supervisors were asked to specify what phase or phases of the IE project their groups were working on. This question allowed multiple responses because at this stage it was possible for groups to be working on more than one phase of their project. In considering the responses to this question it must be remembered that due to the long time frame this survey was available (15 weeks) this data does not give a snapshot of the projects at any one stage.
Half of the supervisors indicated that their groups were working on more than one phase of their IE project, and typically one of these was coding. Eight of the 12 project groups were coding and half were testing. From the earlier survey responses, a couple of groups were working on analysis and a couple on design; from the later survey responses, one group was at the installation phase. No group was at the training phase.
3. & 4. During what project phases has WIER been most useful and least useful to your students?
The question asking supervisors to nominate which phases of the project work WIER had been most useful and least useful allowed multiple responses. The results are shown in Table 5. The most frequent responses were for the analysis and design phases, however when interpreting these results it must be remembered that this survey was held during the middle the year when most groups had not started the installation or training phases and very few had done any testing.
Table 5.
Supervisors Nominations of Project Phases for which WIER is the Most
and Least Useful for Students
|
Project Phase |
Number of projects |
|
|
Most Useful |
Least Useful |
|
|
Analysis |
6 |
0 |
|
Design |
5 |
1 |
|
Coding |
1 |
2 |
|
Testing |
1 |
1 |
|
Installation |
0 |
3 |
|
Installation |
1 |
1 |
Questions
5 - 11
The mean values for the effectiveness of the WIER site for project management are shown in Table 6. Five of the seven ratings were below 3, the neutral mid-point value of the 5 point scale, indicating a tendency towards finding the site effective for project management. The means values were compared to the mid-point value using one sample t-tests. These showed a statistical difference for only one aspect, indicating that the supervisors felt that their students were less dependent on them for information than previous groups they had supervised.
Table 6.
Supervisors' Ratings of the Effectiveness of WIER for Project Management
Question
|
N
|
Mean
|
SD
|
t
|
|
5. Compared
to your past IE groups, is this group more/less dependent on your for
information and guidance? |
10 |
2.4 |
0.7 |
-2.71 * |
|
6. Compared
to your past IE groups, how well has this developed the required project
documents? |
11 |
2.4 |
1.3 |
-1.64 |
|
7. Compared
to your past IE groups, how well is this group keeping to the project
timeline? |
11 |
3.1 |
1.2 |
0.25 |
|
8. Compared
to your past IE groups, how well does this group work together? |
10 |
2.5 |
1.6 |
-1.00 |
|
9. Compared
to your past IE groups, how well does this group know and understand the
overall requirements of the subject (deadlines, deliverables, procedures
etc.)? |
11 |
2.5 |
1.4 |
-1.32 |
|
10. Compared
to your past IE groups, how do you feel this group is performing at this
stage of the project? |
11 |
3.0 |
1.4 |
0.00 |
|
11. How useful
is the WIER site to you in your role as an IE group supervisor? |
11 |
2.6 |
1.0 |
-1.17 |
|
* p < 0.05 |
||||
The following comments were provided by supervisors on the usability and effectiveness of WIER.
· I'm afraid that I've not really made much use of WIER. I'll try to take more advantage of it next time. Part of the problem for me is that I don't like pages that force me to have Javascript turned on, and the font rendering of some of these pages is basically illegible for me (Netscape 4.75 under Linux)
· I receive regular emails re progress, meetings, User contact and use WEIR only infrequently.
· I have struck some load and response problems, and I know that messages to me via WEIR take measurably longer than via the normal email system.
· I feel this Survey System should be Introduced to all Monash Subjects [For Lecturers as well as Tutors and Students] Good way for Improvement.
· Students have complained about cluminess of GUI
· i think 'dont know'
· This group is less capable than past groups.
This survey was available for three weeks. A total of 74 students out of a possible 175 responded to the survey giving a response rate of 42.3%. This was almost twice the response rate of the earlier surveys.
The survey was similar to the Survey 1. For this survey the students were asked to rate the usability of the WIER site. The survey consisted of eight questions each requiring the students to rate an aspect of usability on a 5 point Likert scale. The results are shown in Table 7. The students were also given the opportunity to provide open-ended comments at the end of the survey. These are shown on page 28.
The highest rating was for the consistency of the site and there was good agreement amongst the students about this.
It is interesting to compare the results of this survey with Survey 1 as shown in Appendix 2 (p. 21). It could be expected that with more experience using WIER, students would find it easier to navigate and orient themselves in the site, however their survey responses did not indicate this. In Survey 3 the rating for the question asking how easy it was to view information on the site rate above the mid-point of the range indicating that they found some difficulty finding information and more difficult than they indicated in Survey 1. This difference was almost significant.
The students were not happy with the response time on the WIER site in Survey 1, and they were less happy with this in Survey 3. This difference was statistically significant.
Table 7. Comparison of Students' Ratings of
Usability of WIER in Semester 1 and 2
Question
|
Mean
|
SD
|
Mean
|
SD
|
t
|
|
1.
How easy is it navigate the WIER site? |
2.9 |
1.0 |
2.8 |
1.0 |
-0.39(72) |
|
2.
How easy is it know where you are within the WIER site at
any time? |
2.6 |
1.0 |
2.8 |
1.3 |
0.55(72) |
|
3.
How easy is it to locate information on the WIER site? |
2.9 |
1.0 |
3.1 |
1.0 |
0.96(72) |
|
4.
How easy is it to view information on the WIER site e.g.is
there too much scrolling? |
2.9 |
1.2 |
3.4 |
1.0 |
1.96(72) |
|
5.
How would you rate the consistency of the page layout i.e.
does each page have the same look &
feel? |
2.0 |
0.9 |
2.1 |
1.0 |
0.40(72) |
|
6.
How would you rate the general layout of the pages in terms
of the organisation of the screen
elements e.g. menus, text areas? |
2.6 |
1.0 |
2.7 |
1.0 |
0.67(71) |
|
7.
How would you rate the readability of WIER pages in terms of
choice of colours, fonts, graphics, etc? |
2.3 |
0.9 |
2.2 |
0.8 |
-0.38(71) |
|
8.
How would you rate the response time of the WIER site? |
2.9 |
1.3 |
3.5 |
1.1 |
2.28(72) * |
|
* p < 0.05 |
|||||
The following comments were provided by students on the usability of WIER. Most of the comments were suggestions for modifications or additions to the functionality. However there were a few students who were not happy with the response time and the login mechanism.
· why do you have logins everywhere? I mean, you get one in the main page and one when you click on students without login in b4.
· Why can't the page open up the students homepage directly?
· Wier is been a benefical assest to the project and group in regards to its various functionality and ease of use. (Task time tracker, news postings, File Manager etc)
· would be good if u could add task all from one screen and it updates from there; currently from one screen to another - waits too long and may get lost as to what ure doing in the first place
· The time tracker always takes you back to the main page after times have been logged. It really should offer a link back to the particular category that the time was logged underno
· a more flexible Task Time Tracker, File Manager, etc. which incorporates modification and deletion of entries
· the new site stuffed up already on this server
· Please get rid of this WIER site... its a head ache.. By any chance has the developer ever heard or seen a button called "DELETE" coz its missing here, and if you make a mistake no way to correct it. like making a car without a reverse gear.
· In term of scrolling, there're bit of scrolling. The thing to note that it's a bit inflexible to navigate, for example if I'm in the Task Tracker, to go other functionality (provided for students, e.g. Discussion Forum, File Manager, etc.), I have to the Students Homepage first
· Main problem was response time - lengthy whiteouts before page appears, sometimes extremely difficult/impossible to access the site - and I have ADSL
· Response time sometimes not very good. You need to wait a while till the pages get displayed.
· Response Time - sometimes extremely slow
· Why can't the page open up the students homepage directly?
· Just too slow, better than before yes, but still too slow.
This survey was a paper-based survey administered to students at the final presentations of their IE project work. All students who attended, which was most of the group of 175, were given a questionnaire. One hundred and fifty seven valid surveys were returned giving a response rate of 89.7%.
The survey contained questions about the IE project organisation, WIER site and resources, IE project organisation, seminars, workloads, assessment, clients, supervisors, coordinators.
All questions in the survey, except for four text response questions, required the students to give a response on a 5 point Likert scale. In each case a one sample t-test was used to compare the mean of the responses with the value 3, the neutral mid-point of the scale.
The results are presented under topic headings.
The students were asked to rate the usefulness of ten resources provided on the WIER site. The students rated each resource on a 5 point Likert scale, where 1 = high, 5 = low). The results are shown in Table 8.
The mean values for most of the usefulness ratings of resources were above 3, the neutral mid-point value of the 5 point scale, indicating a tendency towards finding these resources useful. The means values were compared to the mid-point value using one sample t-tests. These showed statistical differences for six of these resources, indicating that the students tended to find the file manager, task tracker, time graph, document templates, subject information and news items useful. However, the mean ratings for the discussion groups and resources were significantly lower than the mid-point value of the scale indicating that the students did not find these useful.
It is interesting to compare the results of this survey with Poll 2 as shown in Appendix 3 (p. 23). There were significant differences in students’ opinions of most resources from semester 1 to semester 2. Four resources (time tracker, document templates, sample documents and calendar of events) became the most useful resources by Semester 2. However the discussion groups were significantly less useful.
Other aspects of WIER that the students rated are shown in Table 9. All of these were rated significantly greater than the mid-point value of the scale indicating that they found the calendar of events clear, it was easy to find information and perform tasks on WIER.
The following comments on the usefulness of WIER were provided in the general comments section of the survey.
· Although WIER was helpful, it was too slow to be practical
· Time task tracker not flexible
Table 8.
Students' Ratings of the Usefulness WIER Resources
Resource
|
N
|
Mean
|
SD
|
t
|
|
Discussion Groups |
123 |
3.3 |
1.2 |
3.34 * |
|
File Manager |
142 |
2.6 |
1.2 |
-3.67 * |
|
Task tracker |
151 |
2.1 |
1.1 |
-9.59 * |
|
Time graph |
152 |
1.9 |
1.0 |
-14.23 * |
|
Documents and templates |
149 |
2.1 |
1.0 |
-11.23 * |
|
Subject information e.g. roles, final submission |
148 |
2.1 |
1.0 |
-10.30 * |
|
Past projects |
140 |
2.9 |
1.2 |
-0.58 |
|
Resources e.g. URLs, books |
135 |
3.2 |
1.2 |
2.06 * |
|
News items |
133 |
2.6 |
1.0 |
-4.01 * |
|
Risks lists |
148 |
2.9 |
1.2 |
-0.92 |
|
* p < 0.05 |
||||
Table 9. Students' Ratings of the WIER Site
Question
|
N
|
Mean
|
SD
|
t
|
|
How clear did you find the calendar of events? |
155 |
2.3 |
0.9 |
-9.01 * |
|
How would you now rate the WIER site, in terms of the navigation, and how easy is it to find information, and perform tasks? |
153 |
2.7 |
0.9 |
-3.98 * |
|
How would you now rate the WIER site, in terms of the support it provides for project management? |
151 |
2.5 |
1.0 |
-5.74 * |
|
* p < 0.05 |
||||
The relationship of the students’ ratings of the usefulness of WIER resources on their valuation of its support for project management were investigated using a stepwise regression[1].
Seven of the ten usefulness variables, which produced coefficient values greater than 0.3 when correlation against the support for project management, were used in the regression. These variables were regressed against the support for project management, producing a model with an R2 of 0.44, significant at F = 24.28 (4,122) for p < 0.05. Four variables produced a significant impact as shown in Table 10.
Table 10
Variables producing a significant impact
as follows
|
Resource |
Standardised Beta |
t |
Significance |
|
Time graph |
0.285 |
3.876 |
0.00 |
|
File manager |
0.320 |
4.585 |
0.00 |
|
Subject information |
0.267 |
3.877 |
0.00 |
|
Risk lists |
0.192 |
2.626 |
0.01 |
The students were asked to rate aspects of the IE project subject. The students rated each aspect on a 5 point Likert scale (where 1 = high, 5 = low, unless otherwise indicated). The results are shown in Table 11 through Table 15. For all these aspects the students rated the IE project significantly higher than the mid-point of the scale.
Table 11. Students' Ratings of the IE Project
Organisation
|
|
N
|
Mean
|
SD
|
t
|
|
How satisfied were you with the way the projects were allocated? |
156 |
2.5 |
1.0 |
-5.83 * |
|
How satisfied were you with the way the projects were formed? |
156 |
2.1 |
1.0 |
-10.67 * |
|
* p < 0.05 |
||||
Table 12. Students' Ratings of the Seminars
|
|
N
|
Mean
|
SD
|
t
|
|
How would you rate the usefulness of the seminars in providing information for the subject? |
151 |
2.7 |
0.9 |
-4.40 * |
|
* p < 0.05 |
||||
The following are suggestions by the students about the seminar program provided:
What seminar topics would you leave out?
· About past experiences in a project
· Conflict handling
· Discussion group
· File Manager
· Guest Lecturer
· How to navigate a WIER website (2)
· Roles and deliverables
· Team management
· Testing
· The video (2)
·
List any seminar topics that should be included?
· Documentation standards and formats, what should be included, how to produce documents (instead of giving template on WIER (2)
· If any problem occurs in the project, how students should deal with it.
· Entrepreneurship
· Experience from previous projects
· Interpersonal and Communication skills
· Marking guidelines
· More about testing different Systems
· More about group management (3)
· More about project management (3)
· Presentation skills and details on assessment
· Technical information regarding technologies used in projects
· Time management
· Timelines of the project is not clear. Functional specification etc...
· Using WIER site in greater detail and practicality
Table 13. Students' Ratings of IE Project Workloads
|
|
N
|
Mean
|
SD
|
t
|
|
How reasonable were the deadlines for deliverables? |
155 |
2.5 |
0.9 |
-6.85 * |
|
How well does the workload match a 12-point subject? |
155 |
2.5 |
1.1 |
-5.43 * |
|
* p < 0.05 |
||||
For these questions the students rated each aspect on a 5 point Likert scale (where 1 = disagree, 5 = agree).
Table 14.
Students' Ratings of IE Project Assessment
|
Do
you feel that the different assessment methods used in this subject e.g.
presentations, walkthrough, peer assessment, supervisor interview, supervisor
assessment of the project? |
N
|
Mean
|
SD
|
t
|
|
Provide a fair/valid assessment of your individual work in the project? |
156 |
2.6 |
1.2 |
-7.74 * |
|
Provide a fair/valid assessment of the group’s work in the project? |
156 |
2.4 |
1.0 |
-9.83 * |
|
* p < 0.05 |
||||
Table 15. Students' Ratings of the Experience of
Group Work
|
|
N
|
Mean
|
SD
|
t
|
|
Given the project you had, how helpful was it working in a group, as opposed to doing the project yourself? |
153 |
1.8 |
0.9 |
-17.03 * |
|
How beneficial was the experience of working in a group? |
152 |
1.7 |
0.9 |
-19.05 * |
|
* p < 0.05 |
||||
The students were asked to rate the IE project staff. The students rated each aspect on a 5 point Likert scale (where 1 = low, 5 = high). The results are shown in Table 16 though Table 18. For all these aspects the students rated the IE project significantly higher than the mid-point of the scale.
Table 16. Students' Ratings of their IE Project
Client
How
would you rate your client in the following areas?
|
N
|
Mean
|
SD
|
t
|
|
Giving feedback on your work |
157 |
2.4 |
1.1 |
-6.28 * |
|
Providing information on project requirements |
156 |
2.5 |
1.2 |
-5.87 * |
|
Being available to you on a regular basis |
156 |
2.6 |
1.2 |
-4.07 * |
|
* p < 0.05 |
||||
Table 17. Students' Ratings of their IE Project Supervisor
How
would you rate your supervisor in the following areas?
|
N
|
Mean
|
SD
|
t
|
|
Providing feedback, advice and reviewing the work of the group |
155 |
2.1 |
1.2 |
-9.85 * |
|
Resolving conflicts within the group |
152 |
2.2 |
1.2 |
-7.76 * |
|
Keeping track of your individual tasks and programs |
155 |
2.2 |
1.2 |
-7.88 * |
|
* p < 0.05 |
||||
Table 18. Students' Ratings of the IE Project
Coordinators
How
would you rate your coordinators in the following areas?
|
N
|
Mean
|
SD
|
t
|
|
Providing feedback, advice and reviewing the work of the group |
156 |
2.2 |
0.9 |
-10.38 * |
|
Resolving conflicts within the group |
150 |
2.5 |
1.0 |
-5.56 * |
|
Provide timely information relating to the organisation of the subject |
156 |
2.2 |
1.0 |
-9.68 * |
|
"{* p < 0.05 |
||||
Table 19. Students' Ratings of the IE Project Subject
|
|
N
|
Mean
|
SD
|
t
|
|
How did you rate the IE Project Subject in terms of helping you to get the type of job you are after? |
150 |
2.2 |
1.0 |
-9.84 * |
The relationships between the students’ ratings of the clients, supervisors, coordinators and group work on their rating of how useful the IE Project subject was for getting a job was determined using correlations.
The students’ valuations of how useful the IE Project subject was for getting a job showed the strongest relationship with their ratings of the feedback (r = 0.376, p < 0.01) and information (r = 0.345, p < 0.01) from the client and feedback from the supervisor (r = 0.328, p < 0.01).
These relationship were further investigated using regression [2]. This produced a model with an ANOVA significant at F = 15.96 and three significant predictors. However the R2 value of 0.25 indicates the model should be rejected.
The students were asked to nominate the best aspects of their IE project subject and their, experience of group work. Fifty five percent of students provided comments, about a quarter of these provided multiple comments. The student comments were coded into seven categories:
· Working in a group
· Working on a real world project
· Learning new skills
· Project management
· Interaction with the client
· Putting existing skills into practice
· General and other coments
In considering the number of responses from these categories it should be remembered that the students were requested to comment on the group work specifically, so the high number of responses to this category should be seen in this light. These categories are described below
Forty five students (41.3% of those who responded) commented that the best aspect of the IE project was the group work and the opportunities it afforded to work in a team, sharing ideas and learning from others. A few mentioned conflicts within their group, but saw the challenges of having to work together to resolve differences and reach agreement on decisions as positive experiences, giving them a realistic picture of what they would have to deal with in the real world. On a more personal level, a couple of students that the project had given them the opportunity to meet new people and form new friendships. Typical comments included:
· Being able to work in a group of people of different attitudes, we argue, we fight but eventually we come to an agreement - what life is all about.
· Dealt with real people, real attitudes, real human behaviour.
· Meeting different types of individuals and working with them - made new friends
· Have a feel of how it will be in a work force to work in a group.
· I think that this subject was great as it gave us the experience of working on a 'real' world problem.
· Provides a very good indication of the work environment that will exist in most I.T. projects
One student mentioned a specific learning opportunity:
· Being able to learn and get new ideas from external programmers working on subsystems not part of scope but integral to our system
Twenty students (18.3% of those who responded) commented that they appreciated the opportunity their project had given them to learn new skills. Most mentioned they had learnt technologies that were previously unfamiliar to them e.g. PHP, ASP, VB. A couple of students felt they had developed communication skills. Some comments
· It really stressed you up(?!) and force you to learn a lot of new technologies that had not been used before!
· Learning communication with others, within the group, with lecturers, clients
· Gives us a great opportunity to actually work with real clients and learn a great deal thru it
Seven students (6.4% of those who responded) commented that their project work had given them an opportunity to put existing skills into practice, bringing together many things they had learnt in their course into one project.
Ten students (9.2% of those who responded) made general comments about their experiences. Some comments.
· The entire project was a completely new learning experience for me. I have never had a project done of this scope with so many parties involved (group, supervisor, client etc)
· A very good subject, which is good b/c of the challenge it brings in all aspects of computing
Some students commented on more personal aspects of:
· (Opportunity to) develop a sense of self-awareness.
· The satisfaction of making a difference to someone, or an organisation with I.T.
· The smile on everyone's faces when we got it done
The students were asked to nominate the best aspects of their IE project subject and their, experience of group work. Sixty nine percent of students provided comments, about a quarter of these provided multiple comments. The student comments were coded into seven categories:
· Workload
· Group members not performing
· Group organisation
· Project allocation and definition
· Problems with client, supervisors, coordinators
· Group conflict
· Other coments
In considering the number of responses from these categories it should be remembered that the students were requested to comment on the group work specifically so the high number of responses to this category should be seen in this light. These categories are described below
Twenty students (23.0% of those who responded) were unhappy with what they felt was the high workload of the IE project subject. Most indicated that their project work impacted on their other subjects. A few mentioned the difficulty of trying to meet project deadlines. Typical comments were:
· Quite a stressful experience. Good for experience but bad strain on other subjects.
· Time-consuming in a way that interrupted the regular studies.
Sixteen students (18.4% of those who responded) indicated that group members not doing their fair share of the work was a problem. It is interesting that only a couple commented on poor quality of work by other group members. Two students stated that group members had cheated on project time recording Some comments were quite strongly stated on these issues.
· Terrible group member - group's reputation ruined!!!!!!! Dealt with real people, attitudes and behaviour
· Most work done by certain group members. Others went on holidays.
· People not doing work and adding fake time to WEIR
Fifteen students (17.2% of those who responded) indicated that they had experienced difficulties with organisation and management of their group. The most common problem was trying to organise meeting times with group members and others. It was interesting that one student believed that this would not be an issue in the real world. Others problems were with the task allocation and lack of knowledge or skills among the group members. Typical comments were:
· Hard to organise times when everyone might be available i.e. group members, clients, supervisors...A lot of late nights!!
· Spreading the workload between group members
Project allocation
and definition
Sixteen students (16% of those who responded) indicated dissatisfaction with the project allocation. Most commented on the unevenness of the projects’ scope or level of difficulty. One student felt it was unfair that some groups had the opportunity to work on new technologies whilst others didn’t. Two students were unhappy with their projects indicating that they felt they were not “real” IE projects. A couple of students stated that they would like to chose their own projects and a couple of others indicated they would like to have the opportunity to choose their own group members. This last comment was a little strange because, in fact, the students are given the opportunity to form their own groups.
· It is not fair for those groups which get the tough topic - there is a big difference b/t the tough and easy one - UNFAIR!
· The project scope was confusing and its hard to achieve our best due to this.
· Unfair that some groups get a very complex project with using new technology but some groups get a system which has a small scope and very simple compared to the others.
· No opportunity to select group members from the same courses which eliminates timetabling issues (easier for meetings etc..)
Ten students (11.5% of those who responded) indicated that they had experienced problems with their client. The couple of students who provided details indicated that their client had shown a lack of knowledge, given misleading information or had been slow in response to questions or requests for information. Three students (3.4% of those who responded) indicated they had had problems with their supervisor, however no specific details were given. One student indicated uncertainty about the fairness of the coordinators.
· and the client unsure of the requirements - made it really difficult
· Our supervisor sucks!!! He's too old for this!!!!
· Having coordinators who may be biased?
Nine students (9% of those who responded) indicated that conflicts within their group were a problem. However their comments were not specific about the causes.
Ten students (11.5% of those who responded) provided comments on a range of other issues. Most of these related to specific difficulties faced with their project such as having to learn new skills or lack of support or resources. A couple of students were unhappy with the assessment process. Some comments.
· Not having enough people/support to get help on technical issues
· Having to learn new programming language which is not taught formally
· Peer group assessment I am totally against. I don't believe students are qualified to grade a student in a professional manner, as it become a popularity contest.