Estructura del sistema

Participants were first invited to examine screenshots and screencasts of Discoverer, and its key features were also briefly summarized at the beginning of the interviews. Then, the participants offered their views about Discoverer, comments about its limitations and suggestions for its improvement, as well as considerations about its possible uptake.

Relevance of the problem addressed

Coherently with the feedback from educators in the previous studies and the literature (UNESCO, 2017), participants considered the problem of discoverability as a very relevant challenge in the OER ecosystem:

OERs discoverability is definitely a relevant challenge [PartB]

PartA, having a background in search engines, initially claimed that discoverability was the main challenge, quickly acknowledging, however, that quality too was a fundamental issue, and concluding that discoverability was one of a few main challenges:

Discoverability is the main challenge: finding resources… and good quality resources… maybe it is one of the two or three main challenges [PartA]

Quality of the resources is indeed another major challenge, recurring under different themes, and will be addressed under “Limitations and suggestions for improvements”.

Overall assessment of the proposed strategy and strengths

Participants, independently from their different background, were positive about the prototype. For example:

The prototype is excellent! This is a wonderful work, it is really good. I would definitely recommend it to colleagues. [PartD]

First and foremost, I really like what you have done. [PartE] I love your approach and I think that it is a brilliant approach. [PartA]

An interviewee did not comment on it:

I don’t know it enough to comment on it. [PartB]

but he added a generic positive remark:

Any tool improving the discoverability of OERs is going to be useful. [PartB]

This is actually debatable, because there is the risk that additional tools in general, such as one more search platform, might just worsen the situation in the OER ecosystem, by increasing its fragmentation. PartC pointed out, indeed, that this proliferation of platforms makes it more difficult to propose alternatives to Google:

[Other challenges are] awareness in a highly fragmented context, habits: people are used to Google, it takes effort to change habits [PartC]

In this regard, PartD praised the strategy to develop Discoverer as a reusable component on top of existing search engines, exactly with the objective to avoid this fragmentation:

You don't want to build another one [search platform], [a reusable component to reduce fragmentation is] very good […] that is excellent! [PartD]

Along the same lines, PartA pointed out:

At some point you have too many people stirring the pot and at some time you need to be an executive chef who is sort of unifying things rather than dividing things [PartA]"

This challenge of fragmentation will be further discussed more in general under the broader theme “OERs challenges”.

Participants expressed appreciation for specific features of the prototype, such as its exploratory orientation:

Brilliant approach, really, really attractive, I love it, nice exploratory approach. [PartA]

its domain-oriented similarity metric:

Good idea, it is an interesting concept of similarity. [PartC]

or its simple interface:

I think it would be easy for them to use this tool. The interface looks easy. [PartD]

They appreciated its QBE approach, that is the possibility to issue a query starting from any sample resource:

And I really love that idea, that you have, and looking at a resource just some random web page somewhere on the internet, and I can click that button and it is going to

feedback into this loop. [PartD]

as well as from any snippet in a standard Google SERP:

They are going to use a web browser, they are going to use Google: having either filters or plugins that are right on top of the search queries from Google, is going to be key.

[PartD]

They also praised the importance of addressing the work context of educators:

Integration into existing workflows

While the strategy to explicitly target educators’ tasks was appreciated, the interviewees suggested other contexts where the prototype could be equally useful. In addition to educators, they suggested that it could benefit stakeholders such as learners or course developers.

Coherently with the previous study, an interviewee suggested that students could conveniently use Discoverer to find remediation material:

In addition to teachers, it could benefit learners, especially those struggling with their studies: they could use it to find additional complementary learning material. [PartC]

Another interviewee, a flipped classroom evangelist, considered that Discoverer could be conveniently used by learners in that context:

I put the discovery of resources more in the hands of my [flipped classrooms] students, […] putting that in the students hands, that would be very useful, that is wonderful.

[PartD]

Finally it was suggested that Discoverer could be useful for course material developers:

It could benefit groups of course material developers too. [PartC]

I could use it to […] help members in my courses development team finding material to adapt and integrate in new courses. [PartC]

Limitations and suggestions for improvements

Participants mentioned explicitly only two “limitations” of Discoverer, preferring to address other limitations implicitly, by offering “suggestions for improvements”. Hence the two separate codes anticipated in the preliminary coding map were merged.

The first limitation explicitly mentioned, at micro-level, was the lack of explanation capabilities:

Not providing explanations about its results. [PartA]

In reality, explanations were not included on purpose. Indeed, explanations would necessarily expose the formal learning objectives to the users, while all but one educators previously consulted, preferred them to be invisible. This suggestion could be partially explained because the interviewee, like the only educator who suggested to make the formal learning objectives explicit, was comfortable with their use. Yet the interviewee, reacting to my justification for the lack of explanation capabilities, argued that:

Some of the users, once they become more experienced and wish to be even more efficient, will likely require this. [PartA]

This is indeed a helpful observation: the preferences and behaviour of some of the educators, might well change in response to the use of the prototype over a long period (Kules and Shneiderman,

2008). This effect could be exposed by a longitudinal evaluation of the prototype – which is a good suggestion for a further research activity.

Providing explanations was considered very important for this interviewee, who reinforced the idea formulating it also as a suggestion for improvements. In this case, possibly because of the previous discussion, he added a detail:

Provide concise but effective explanations on demand. [PartA]

Letting users control this functionality looks like a good compromise, because users with more familiarity with formal learning objectives might turn this feature on, while it could remain invisible to other users.

The second limitation explicitly indicated, was the dependence of the prototype on a specific metadata standard:

The weakness of the process is its dependence on LRMI [PartA]

The interviewee argued that, on the contrary, his approach (he developed a search engine), even if inelegant, was independent of any particular metadata standard:

We of course took the opposite approach, we just said we are going to assume no commonality, we are going to... our approach was not elegant in any way, but recognizing that there is not any elegance right now in the environment so therefore the

approach probably isn't going to be very elegant, is going to be very brute force. For each repository there is going to be a different metadata structure there, and we were just

going to have to deal with that, and we were willing to deal with that. So, two diverging approaches that reflect where we are. [PartA]

Discoverer, in reality, is partially independent from LRMI, because it retrieves the educational alignments (learning objectives) exploiting the so called “data services” of the Learning Registry. These data services can extract the learning objectives from metadata expressed with LRMI, as well as with other standards. This comment, however, shows that a commonly accepted metadata standard, despite LRMI being a recent development backed by Google, is still an open challenge. This will be discussed under the following theme, directly related to metadata, in Section 9.3.2.

The participants in this activity, given their long involvement with open material, were definitely more aware of the importance of openness for OERs, compared to the educators involved in the previous studies. PartC, experienced in leading teams of course developers exploiting open material, argued about the need to provide easy to understand information about the level of openness of the resources:

Indicate, in simple terms, the level of openness of the resources identified. [PartC] Support people in quickly understanding the licensing constraints. [PartC]

A similar concern for licencing specifications to be expressed in simple terms, was indicated by PartB too:

We are thinking about possible new definitions of OER, trying to keep them simple enough and legally exhaustive. [PartB]

This concern was echoed by an educator involved in the evaluation of RepExp (Section 7.3.3), who suggested to highlight the OERs in the results, in order to discriminate them from other educational resources. The opportunity to provide simple information to clarify licencing constraints is also frequently mentioned in the literature (Anderson and Leachman, 2019). Yet Discoverer simply displays the metadata available, that may or may not include information about the level of openness. Hence the prototype could be improved, for example, by mapping these metadata to a more immediately evident visual representation of the level of openness (reusable, remixable, etc.).

Another suggestion concerned the need to provide information about the quality of the resources:

Integrate quality information. [PartA]

This concern was reinforced by another comment, previously reported, where quality was indicated as a relevant issue:

Discoverability is the main challenge: finding resources… and good quality resources… [PartA]

While the prototype, at the moment, does not display any information about the quality of the resources, it is acknowledged that quality is critical to the success of OERs repositories (Clements, 2016) and the ecosystem in general (UNESCO, 2017), and is definitely worth of further research.

Other suggestions concerned more specific aspects. These include adding filters:

Integrate the possibility to filter on text complexity. [PartC]

This can be implemented, provided that the related metadata are available in the search platform which Discoverer integrates. In case these were not available, it is still possible to compute some readability indexes, such as the Gunning Fog index or the Automated Readability Index, as approximated indicators.

Another suggestion concerned the use of different similarity metrics:

This possibility, indeed, was foreseen and discussed since the early phases of this research.

A bilingual interviewee, working in a non-English speaking country, and fully conscious of the language barrier, suggested to integrate a translation service:

If we can help with translations, and even using auto-translations of some of these resources into the native language of the searcher, that may be helpful as well. [PartD]

Machine translation is, of course, a research field in its own. However, even an approximate translation, that can be implemented using existing services, could be acceptable for certain use cases:

You are not necessarily going to use those resources you found, you don't need to use a complete port to help your craft, but it can give you an inspiration. [PartD]

Indeed, this is a common use case for OERs: de los Arcos et al. (2016) report that 80.5% of the educators they interviewed, used OERs to get inspiration and new ideas. Language issues, being a broad challenge for OERs, will be also discussed under the third theme.

PartD, having User Experience expertise, recommended to make the prototype easy to install:

Make sure it is easy to install. Package it up so that it is really easy to use, definitely the highest priority to make it really easy. [PartD]

Indeed, the current version of Discoverer, being a prototype, is not particularly easy to install. This problem was highlighted already by a few test-users involved in previous evaluations, but could be conveniently fixed in an operational version.

Finally, it was suggested to use the search history to improve the relevance of the queries:

Are the search history to change the queries at all into your database backend? [PartD]

Indeed, the efficiency of the queries could be improved, taking into account previous searches or other information (e.g. the user profile). This information could be employed to disambiguate terms and prioritize the results most relevant for the users, for instance.