The Researcher's Guide to the Data Deluge: Querying a Scientific Database in Just a Few Seconds

There is a clear need for interactive exploration of extremely large databases, especially in the area of scientific data management where ingestion of multiple Terabytes on a daily basis is foreseen. Unfor- tunately, current data management technology is not well-suited for such overwhelming demands.There is a clear need for interactive exploration of extremely large databases, especially in the area of scientific data management where ingestion of multiple Terabytes on a daily basis is foreseen. Unfor- tunately, current data management technology is not well-suited for such overwhelming demands.

In light of these challenges, we should rethink some of the strict requirements database systems adopted in the past. We envision that next generation database systems should interpret queries by their intent, rather than as a contract carved in stone for complete and correct answers. The result set should aid the user in un- derstanding the database's content and provide guidance to con- tinue the data exploration journey. A scientist can stepwise explore deeper and deeper into the database, and stop when the result con- tent and quality reaches his satisfaction point. At the same time, response times should be close to instant such that they allow a scientist to interact with the system and explore the data in a con- textualized way.

Several research directions are carved out to realize this vision. They range from engineering a novel database kernel where speed rather than completeness is the first class citizen, up to refusing to process a costly query in the first place, but providing advice on how to reformulate it instead, or even providing alternatives the system believes might be relevant for the exploration patterns ob- served.