APL93 Conference Evaluation Survey

by Marc Griffiths, . (marcg@utcc.utoronto.ca)

Introduction

This report is an edited summary analysis of the evaluation survey conducted at the APL93 Conference. The conference was sponsored by the Association for Computing Machinery’s Special Interest Group on APL (ACM SIGAPL) and held in Toronto, Canada from 15 to 19 August 1993. Eighty-one questionnaires were returned and analysed. This represents about 25% of an estimated 320 delegates who attended the conference. While this is a good response rate, the sample size is too small for meaningful cross-correlations.

In measuring the success or otherwise of the conference as a whole, bear in mind that two reports appeared in July 1993 which cited an average drop in attendance at conferences in Canada of 20% to 25% compared to a year earlier.

Summary of Analysis

1. Almost three-quarters of the respondents constituted repeat business. Of the newcomers, word of mouth was most frequently cited as the method through which they first learned of the conference. More than one respondent noted a desire for more outsiders to be brought in to the conference, both as attendees and speakers.
2. Over one-fifth of the respondents praised the technical programme, but an almost equal number were dissatisfied with it. Most of the latter want more papers, panels, workshops and tutorials dealing with (commercial) application design and development, and computing topics such as object-oriented programming and graphical user interfaces.
3. Many comments indicated the importance of providing adequate facilities for social and professional interaction outside the planned social events. These manifested themselves as requests for space and time for informal interaction among the delegates between sessions through the day, and for gatherings in the evenings. In addition, several respondents asked for space and equipment to allow attendees to experiment with software which had been discussed in a presentation.
4. The vendor exhibits and forums were among those parts of the conference which elicited the most positive reactions from the largest number of respondents.
5. Cancelled sessions were repeatedly mentioned as a source of annoyance. Some respondents had come to APL93 specifically to attend these tutorials, papers, etc. The feelings of frustration and disappointment were exacerbated further by the failure of the conference organizers to notify delegates of the cancellations in time for an alternative session to be selected. Indeed, in the case of the APL Help Facilities panel, no one knew that the moderator had failed to appear until the audience had assembled in the room.
6. Respondents remarked on shortcomings in the conduct of speakers and panel moderators. These included poor preparation, lax timekeeping and failure to appear.
7. A significant number of respondents encountered difficulties in finding rooms on the conference site. Floor plans of the appropriate buildings and improved signs in corridors and stairwells as well as outside each of the rooms would have remedied this problem.
8. The complaints about some of the events (vendor forums, exhibitors’ reception and the City Hall reception) were directed at the amount and type of food. There always appeared to be too little of all kinds, including both vegetarian and non-vegetarian. The two lunch-time vendor forums presented their own problems, as the food was intended for the forum attendees only, but other conference delegates did not hesitate to help themselves to occasionally generous portions. Other respondents suggested that coffee and pastries or muffins should be available for half an hour prior to the start of each day’s presentations; that lunch be provided; and that the conference fee include lunch.
9. Some respondents complained about various aspects of the pricing – too high, not in the currency of the host country, and too few things included as part of the registration fee.

Method

A copy of the conference evaluation questionnaire was included in each registration kit. Additional copies were available at the information desk for delegates who wanted to fill in a questionnaire but had forgotten to bring it with them. There was no control mechanism to prevent the return of more than one questionnaire by an individual, but there is no evidence that anyone submitted multiple copies.

The responses to questions 1 to 5 and question 7 were coded and input to a spreadsheet program and simple analyses were carried out. Responses which did not comply with the instructions on the questionnaire were deemed invalid and excluded from the analysis. For example, questions 2, 5 and 7 asked the respondent to select one from several choices. Where the respondent selected more than one, none of the choices were included in the analysis.

Some questions were open-ended. Analysis of the replies to these was limited to drawing up a list of categories in order to group the replies, and to classifying the responses according to the categories. The categories themselves were inferred from the replies; another person looking at the responses might well have used different categories or allocated the replies differently. Question 6 (Briefly describe how you use APL.) was too general to yield meaningful results beyond the fact that a broad spectrum of people attended the conference. If the questionnaire is used again, this question should be reworded so as to avoid the problems noted above. Bear in mind that (i) not all conference attendees are users, and (ii) it may be desirable to distinguish between APL and J. The large number of comments supplied in response to questions 8 (What did you like most about the conference?) and 9 (What did you like least about the conference?) necessitated the omission of much of the detail.

For questions 10 (How would you improve this conference?) and 11 (What suggestions do you have for future conferences?) fewer comments were made for each section, so many of these have been provided verbatim within the detailed section of this report. There is little or no detailed discussion of the results of questions 8 to 11.

Principal Findings

1. Almost 60% of the respondents have access to the Internet. [Q1]
2. Over two-thirds (68.4%) of the respondents first heard about APL93 through attendance at APL92 or by direct mail. [Q2]
3. Almost three-quarters (74.1%) of the respondents had attended a previous ACM SIGAPL sponsored conference. [Q3]
4. Over two-thirds (68.8%) of the attendees came to the conference primarily for one of three reasons: (i) to learn about new applications of APL or J, or their application in a specific discipline (36%); (ii) to meet friends or colleagues (20.3%); (iii) to learn about the APL or J languages (12.5%). [Q5]
5. The most popular parts of the programme were those associated with the vendors – the vendor exhibits, the exhibitors’ reception, and the vendor forums. The least popular were the birds-of-a-feather (BOF) sessions, workshops and panels. [Q7]
6. The Casa Loma banquet, the tutorials and the vendor exhibits were the three most favourably rated parts of the programme. The exhibitors’ reception, the Art Gallery and the panels fared most poorly in the ratings. [Q7]
7. Respondents offered 93 comments on what they liked most at the conference. These centred on the people and the atmosphere, the technical programme and comments pertaining to the friendliness of the people and the atmosphere dominated the replies (30.1%). The Programme Committee had felt that the quality of the papers was very high, and most of the positive comments about the programme supported this (22.6%). Three respondents cited Timo Laurmaa’s paper, “Undocumented Features of APL.” Several attendees praised the tutorials very highly, and one was pleased by the “emphasis and interest in J.” The vendor exhibits were also praised (11.8%). [Q8]
8. Paper presentations, plenary sessions, and tutorials all featured as important aspects of the conference in terms of popularity, although the overall assessment of the papers was less than good (2.14). The reasons for this may be accounted for by comments in Q9 to Q11 below, particularly the recurring requests for more papers and tutorials about commercial applications and broader computing topics such as software engineering, graphical user interfaces, etc.
9. Respondents voiced 73 negative comments about APL93. The criticisms were aimed largely at aspects of the organization (timetabling, signs and site) (38.4%), the technical programme (21.9%) and the perceived pre-occupation of the organizers with making money (15.1%). [Q9]
10. With respect to improving the APL93 conference, almost one-quarter (23.4%) of the 64 comments suggested changes to the programme content and scheduling, with another fifth (20.3%) directed at improvements to certain areas of information and communication. A further 10.9% offered ideas regarding reducing costs or simplifying pricing. [Q10]
11. Of 34 proposed future conference locations, the largest number (12) were situated in continental North America, but four of these sites have hosted the conference in 1991 and 1993. Seven sites on the Pacific Rim were put forward, and five in Eastern Europe/Russia. [Q11]
12. Of 26 proposed themes for future APL conferences, over one-third (9) suggested commercial or packaged APL applications, followed by seven for the APL or related (J, K) language, and six for a computing theme such as software engineering, graphical user interfaces, and object-oriented programming. Suggestions for papers, workshops and tutorials fell into the same categories in roughly the same proportions. [Q11]