BigTable

BigTable

BigTable is a compressed, high performance, and proprietary database system built on Google File System (GFS), Chubby Lock Service, and a few other Google programs; it is currently not distributed or used outside of Google, although Google offers access to it as part of their Google App Engine. It began in 2004"First an overview. BigTable has been in development since early 2004 and has been in active use for about eight months (about February 2005)." [http://andrewhitchcock.org/?post=214 Google's BigTable] ] and is now used by a number of Google applications, such as MapReduce, which is often used for generating and modifying data stored in BigTable ["Bigtable can be used with MapReduce, a framework for running large-scale parallel computations developed at Google. We have written a set of wrappers that allow a Bigtable to be used both as an input source and as an output target for MapReduce job". pg 3 of "Bigtable: A Distributed Storage System for Structured Data", 2006] , Google Reader,"Reader is using Google's BigTable in order to create a haven for what is likely to be a massive trove of items." [http://googlereader.blogspot.com/2005/10/google-reader-two-weeks.html Official Google Reader] blog.] Google Maps,"There are currently around 100 cells for services such as Print, Search History, Maps, and Orkut." [http://andrewhitchcock.org/?post=214 Google's BigTable] ] Google Book Search, "My Search History", Google Earth, Blogger.com, Google Code hosting, Orkut, and YouTube ["Their new solution for thumbnails is to use Google’s BigTable, which provides high performance for a large number of rows, fault tolerance, caching, etc. This is a nice (and rare?) example of actual synergy in an acquisition." [http://kylecordes.com/2007/07/12/youtube-scalability/ YouTube Scalability Talk] ] . Google's reasons for developing its own database include scalability, and better control of performance characteristics. ["We have described Bigtable, a distributed system for storing structured data at Google....Our users like the performance and high availability provided by the Bigtable implementation, and that they can scale the capacity of their clusters by simply adding more machines to the system as their resource demands change over time...Finally, we have found that there are significant advantages to building our own storage solution at Google. We have gotten a substantial amount of flexibility from designing our own data model for Bigtable." from the Conclusion of "Bigtable: A Distributed Storage System for Structured Data", 2006]

BigTable is a fast and extremely large-scale DBMS. However, it departs from the typical convention of a fixed number of columns, instead described by the authors as "a sparse, distributed multi-dimensional sorted map", sharing characteristics of both row-oriented and column-oriented databases. BigTable is designed to scale into the petabyte range across hundreds or thousands of machines, and to make it easy to add more machines to the system and automatically start taking advantage of those resources without any reconfiguration".* [http://radar.oreilly.com/archives/2006/05/database_war_stories_7_google.html "Database War Stories #7: Google File System and BigTable"] ]

Each table has multiple dimensions (one of which is a field for time, allowing versioning). Tables are optimized for GFS by being split into multiple "tablets" - segments of the table as split along a row chosen such that the tablet will be ~200 megabytes in size. When sizes threaten to grow beyond a specified limit, the tablets are compressed using the secret algorithms BMDiff [http://blog.outer-court.com/archive/2005-10-23-n61.html Google's Bigtable] ] and Zippy, which are described as less space-optimal than LZW but more efficient in terms of computing time. The locations in the GFS of tablets are recorded as database entries in multiple special tablets, which are called "META1" tablets. META1 tablets are found by querying the single "META0" tablet, which typically has a machine to itself since it is often queried by clients as to the location of the "META1" tablet which itself has the answer to the question of where the actual data is located. Like GFS' master server, the META0 is not generally a bottleneck since the processor time and bandwidth necessary to discover and transmit META1 locations is minimal and clients aggressively cache locations to minimize queries.

Other Implementations

The Hadoop project has made some progress toward a working implementation of BigTable. They call this project HBase.

"Just as Bigtable leverages the distributed data storage provided by the Google File System, HBase will provide Bigtable-like capabilities on top of Hadoop." [ [http://wiki.apache.org/hadoop/Hbase#background HBase - Hadoop Wiki, Background section] ]

Another open source project, [http://www.hypertable.org Hypertable] , launched on February 4, 2008 with 0.9 alpha release.

ee also

*MapReduce
*Column-oriented DBMS

References

External links

* [http://labs.google.com/papers/bigtable.html Bigtable: A Distributed Storage System for Structured Data] -(official paper; [http://labs.google.com/papers/bigtable-osdi06.pdf PDF] )
* [http://www.cs.washington.edu/htbin-post/mvis/mvis?ID=437 BigTable: A Distributed Structured Storage System] ( [http://video.google.com/videoplay?docid=7278544055668715642&q=bigtable video] )
** [http://www.uwtv.org/programs/displayevent.asp?rid=2787 more video]
** [http://andrewhitchcock.org/?post=214 Google's BigTable] -(notes on the official presentation)
* [http://www.baselinemag.com/article2/0,1540,1985047,00.asp "How Google Works"]
* [http://glinden.blogspot.com/2005/09/googles-bigtable.html "Google's BigTable"] -(from the blog "Geeking with Greg")
* [http://glinden.blogspot.com/2006/05/c-store-and-google-bigtable.html C-Store and Google BigTable]
* [http://www.niallkennedy.com/blog/archives/2006/11/google-mondrian.html Mondrian uses BigTable] - by Guido van Rossum
* [http://wiki.apache.org/lucene-hadoop/Hbase Bigtable-like structured storage for Hadoop HDFS] - (from the Lucene-hadoop wiki)


Wikimedia Foundation. 2010.

Игры ⚽ Нужно сделать НИР?

Look at other dictionaries:

  • BigTable — est un système de gestion de base de données compressées, haute performance, propriétaire, développé et exploité par Google[1]. C est une base de données orientée colonnes, dont se sont inspirés plusieurs projets libres, comme HBase, Cassandra ou …   Wikipédia en Français

  • BigTable — Эта статья содержит незавершённый перевод с иностранного языка. Вы можете помочь проекту, переведя её до конца. BigTable  проприетарная высокопроизводительная база данных, построе …   Википедия

  • Google BigTable — BigTable ist ein proprietäres Hochleistungs Datenbanksystem des Unternehmens Google. Es baut unter anderem auf dem Google File System (GFS) und Chubby Lock Service auf. Es wird zurzeit beispielsweise beim Platform as a Service Dienst (PaaS)… …   Deutsch Wikipedia

  • HBase — is an open source, column oriented, distributed database modeled after Google s BigTable and written in Java. It is developed as part of Apache Software Foundation s Hadoop project and runs on top of HDFS (Hadoop Distributed File System),… …   Wikipedia

  • Comparison of structured storage software — Not to be confused with COM Structured Storage. Structured storage is computer storage for structured data, often in the form of a distributed database.[1] Computer software formally known as structured storage systems include Apache… …   Wikipedia

  • Distributed data store — A distributed data store is a blurred concept and means either a distributed database where users store their information on a number of nodes, or a network in which a user stores their information on a number of peer network nodes . Contents 1… …   Wikipedia

  • NoSQL — This article is about the class of database management systems. For the specific relational database management software, see NoSQL (RDBMS). Structured storage redirects here. It is not to be confused with COM Structured Storage. In computing,… …   Wikipedia

  • MapReduce — is a software framework introduced by Google in 2004 to support distributed computing on large data sets on clusters of computers.[1] Parts of the framework are patented in some countries.[2] The framework is inspired by the map and reduce… …   Wikipedia

  • Google App Engine — Google App Engine …   Википедия

  • NoSQL — (steht für englisch Not only SQL) ist eine Bewegung, eine neue Art von Datenbanken voranzutreiben. Es handelt sich dabei um Datenbanken, die einen nicht relationalen Ansatz verfolgen und damit mit der langen Geschichte von relationalen… …   Deutsch Wikipedia

Share the article and excerpts

Direct link
Do a right-click on the link above
and select “Copy Link”