Over the past few weeks, the Wikimedia community has engaged in a discussion of the Wikimedia Foundation’s plans for search and discovery on the Wikimedia projects. More recently, there has been confusion in the press and among community members about the Foundation’s plans and intention. Although we’ve participated in those discussions in other places and other ways, we want to clarify what we are, and are not doing, at the Foundation.
Since Wikipedia was founded in 2001, the amount of freely available knowledge on Wikimedia projects has grown exponentially. Today, Wikipedia and its sister projects have more than 35 million articles across nearly 300 languages, nearly 30 million images, and terabytes more across the other Wikimedia sites and projects. As the quantity, quality, and diversity of this content has grown, so too has the challenge of helping people find the most relevant information.
At the Wikimedia Foundation, we see a clear need to improve search and discovery on the Wikimedia projects. Improvements will help our users access and share in the knowledge they seek, and in doing so, bring us closer to reaching our mission and helping open knowledge stay accessible and relevant.
Search engineering is not new to the Wikimedia projects. We originally relied on a “homegrown” search engine, written by Wikimedia volunteers and based off Apache Lucene, later replaced with Elasticsearch and the CirrusSearch extension integrated with MediaWiki sites.
These improvements were significant, and we felt encouraged to continue improving in other areas such as relevance, user experience, multi-language support, multi-projects search, and incorporating new data sources for our projects. The Wikimedia community has generally been supportive throughout, indicating through surveys, support tickets, and direct feedback that improvements should continue. And the Wikimedia Foundation has communicated about our plans and improvements in many places, including on the Discovery team’s portal and on the Wikimedia blog.
We created the Discovery team in April 2015 to focus resources on improving this core part of the Wikimedia experience and making knowledge even more accessible, connected, diverse and discoverable in an open and transparent way. To support these efforts, we received a grant from the Knight Foundation for initial research.
These grant terms—and our plans—are straightforward. We intend to research how Wikimedia users seek, find, and engage with content. This essential information will allow us to make critical improvements to discovery on the Wikimedia projects. And in keeping with our values, we will make our findings public, in order for the world to better understand the way we all engage with free, open knowledge.
What are we not doing? We’re not building a global crawler search engine. We’re not building another, separate Wikimedia project. We’re committed to our mission of helping the world access and interact with free knowledge.
Despite headlines, we are not trying to compete with other platforms, including Google. As a non-profit we are noncommercial and support open knowledge. Our focus is on the knowledge contributed on the Wikimedia projects. We strive to make all Wikimedia content available under an open license, which can then be used by anyone for any purpose, including commercial and non-commercial use. This means everyone, including commercial entities, can and do use results of our work.
As we thought about the future of discovery and open knowledge, we continued to explore ideas. We have responsibility towards the open web, the future of free content, and our allies in the open culture community. We may pursue some of these ideas, such as including other sources of open knowledge. Other ideas we have explored and modified or rejected. This is a normal, if sometimes confusing, part of the process of thinking creatively and widely about the shared future of open knowledge.
What we do know is that open dialogue with the Wikimedia community is critical to finding meaningful solutions that match our movement’s unique needs.
Community feedback was planned as part of the Knowledge Engine grant, and is essential to identifying the opportunities for improvement in our existing search capacity. It has been vital to helping the Discovery team develop their product roadmap and projects. It is a major component of the deliverables in our grant-funded research. We intend for this to become even more apparent in the coming months, as we begin further surveys and research.
You can learn more about this work on the Discovery team’s page on MediaWiki.org. We look forward to sharing these findings, and working together to identify the best path forward. Through information sharing and conversation, we will arrive at the best past forward.
It is true that our path to this point has not always been smooth, especially through the ideation phase. By sharing our publicly available product goals, planning, status updates and the Knowledge Engine grant agreement, we are making an effort to improve our collaboration. A public dashboard for this project has also been publicly available. We invite you to continue to engage with us: we will be releasing the results of recently concluded Wikimedia Foundation strategy consultation, in which more than 500 people have participated. In 2015 we made real progress in our technical work, and we intend to do more in 2016. We welcome your comments and thoughts, and your continued guidance as we progress.