Beta
×

Welcome to the Slashdot Beta site -- learn more here. Use the link in the footer or click here to return to the Classic version of Slashdot.

Thank you!

Before you choose to head back to the Classic look of the site, we'd appreciate it if you share your thoughts on the Beta; your feedback is what drives our ongoing development.

Beta is different and we value you taking the time to try it out. Please take a look at the changes we've made in Beta and  learn more about it. Thanks for reading, and for making the site better!

Google's BigQuery Vs. Hadoop: a Matchup

samzenpus posted about a year and a half ago | from the in-this-corner dept.

Google 37

Nerval's Lobster writes "Ready to 'Analyze terabytes of data with just a click of a button?' That's the claim Google makes with its BigQuery platform. But is BigQuery really an analytics superstar? It was unveiled in Beta back in 2010, but recently gained some improvements such as the ability to do large joins. In the following piece, Jeff Cogswell compares BigQuery to some other analytics and OLAP tools, and hopefully that'll give some additional context to anyone who's thinking of using BigQuery or a similar platform for data. His conclusion? In the end, BigQuery is just another database. It can handle massive amounts of data, but so can Hadoop. It's not free, but neither is Hadoop once you factor in the cost of the hardware, support, and the paychecks of the people running it. The public version of BigQuery probably isn't even used by Google, which likely has something bigger and better that we'll see in five years or so."

Sorry! There are no comments related to the filter you selected.

Hadoop is much better and stable (1, Interesting)

stinkyworm (2897591) | about a year and a half ago | (#43454133)

With Google's tendency to randomly quit working on products and techs I would never use them. That is why Hadoop is much better option.

Re:Hadoop is much better and stable (0)

Anonymous Coward | about a year and a half ago | (#43454209)

Hah. Not saying this BigQuery thing is any better, but Hadoop could greatly improve their reputation by not emulating PHP in the passing-your-own-unit-tests realm.

Re:Hadoop is much better and stable (0)

Anonymous Coward | about a year and a half ago | (#43455079)

yeah, Google has a giant dart board where they randomly decide what to stop working on next. I hear google.com will be shut down next month too! Oh wait, maybe Reader was quit for one of the reasons they ALREADY STATED? Butt hurt much?

I bet you spell Microsoft with a $ too, and can explain to me why this is also Sonys fault, cuz rootkit.

*looks at UID*
oh wait.. nevermind....

Re:Hadoop is much better and stable (0)

Anonymous Coward | about a year and a half ago | (#43455337)

Looks at UID... coward

Re:Hadoop is much better and stable (0)

Anonymous Coward | about a year and a half ago | (#43456313)

No, their decision isn't completely random. Your dart board analogy is not correct. There is a random component that they've talked about in the past, but if the metrics that they've made public are correct then the random part is less than 50% of the reason for their decision. Out of the 39 products they've killed so far, there was over three years on average between the announcement and the decision to no longer allow public access to the product. For a big data project that typically has a life span of less than two years, depending on Google's somewhat random and frequent killing of products, it is not a major risk. Only eight of their products screwed over their customers by revoking access in a period of less than two years.

Again, you're wrong about the dart board part, or were you trying to make a funny analogy about Google's giving up on Dart? It is so dead that they haven't published a roadmap.

Re:Hadoop is much better and stable (1)

Dahamma (304068) | about a year and a half ago | (#43456651)

Big whoosh on that one! His entire post was obviously sarcasm, and was pointing out the GP post was a typical anti-Google article troll...

Re:Hadoop is much better and stable (0)

Anonymous Coward | about a year and a half ago | (#43456247)

Google decides to stop allowing their customers to use products after an average of 1,459 days:

http://www.guardian.co.uk/technology/2013/mar/22/google-keep-services-closed [guardian.co.uk]

This means that in less than four years on average we can expect Google to no longer allow us access to our data or do any processing on that data. If you're working with worthless, short-term data then using Google makes sense. If you need a product for more than four years, then history has proven Google to be unreliable.

Re:Hadoop is much better and stable (3, Insightful)

PylonHead (61401) | about a year and a half ago | (#43456405)

You understand that that number is flawed, right? He only figures in the average lives of products that Google has killed. It's kind of like looking at all the people who died of heart attacks, finding out they lived to an average of 48 years old, and then telling the general population that, on average, they're going to die of a heart attack when they're 48 years old.

But please, jump on the anti-google circle jerk. It seems to be the thing to do at the moment.

Re:Hadoop is much better and stable (1)

markhb (11721) | about a year and a half ago | (#43463329)

I just turned 48, you insensitive clod!

Pathetic summary (4, Insightful)

geek (5680) | about a year and a half ago | (#43454189)

"The public version of BigQuery probably isn't even used by Google, which likely has something bigger and better that we'll see in five years or so"

With in-depth analysis like that who needs the full article.........

Re:Pathetic summary (2)

Animats (122034) | about a year and a half ago | (#43454501)

With in-depth analysis like that who needs the full article.........

Right. This is just a troll for a short blog posting. There are no benchmarks or examples at all. This looks like "sponsored content".

click of a button? (1)

schlachter (862210) | about a year and a half ago | (#43454271)

That's so 2008. Wake me up when I can process terabytes of data with the sound of my voice, the wave of my hand, or the wave pattern of my brain. ;P

Re:click of a button? (1)

gandhi_2 (1108023) | about a year and a half ago | (#43454633)

meh.

the triggering of my cron job.

or you can (0, Flamebait)

nimbius (983462) | about a year and a half ago | (#43454305)

avoid the protracted outages, painful licensing, free access by federal authorities and data mining by a private multinational and just do this instead http://hypertable.org/ [hypertable.org]

wrong link (0)

Anonymous Coward | about a year and a half ago | (#43454307)

wrong link in article

Google is too ubiquitous (-1, Troll)

Anonymous Coward | about a year and a half ago | (#43454317)

Google really is too ubiquitous. There really does need to be limits on how far reaching one company can become.

Take, for example, Google's expansion into being an ISP. There is not a hint of benevolence there in wanting people to have a fast connection. Google wants more eyeballs for its ads. Full stop. You are not the customer with companies like Google -- you're the product. I wouldn't choose Google as my ISP if they were free. I have never trusted Google or anyone else with my data. They are too big and people love it. I can hear the shouts of capitalistic joy even now...

Re:Google is too ubiquitous (2)

Dancindan84 (1056246) | about a year and a half ago | (#43454525)

What, you think the big telecoms are more benevolent? With the big telecoms they honestly don't care how good their service is most of the time. They'll bill you either way, and your other option(s) are either non-existent or more of the same with a different name.

If Google's model is making me the product, at least they have an investment in keeping their service up. I can't view their ads if I'm offline. Plus competition is going to drive relative costs down and service up. If they can monetize the fact that I did a search trying to figure out what this, "Harlem Shake" thing is on April 3rd, they're welcome to. Anything important I do is encrypted.

Re:Google is too ubiquitous (-1, Offtopic)

Anonymous Coward | about a year and a half ago | (#43454615)

I agree RE other ISPS.

You avoided my original premise: Google is too ubiquitous. They have become too powerful. Apple and Google are the new Microsoft, as much as other may want to disprove this. We are seeing a clear path towards closed-in/walled-in fiefdoms and people having to choose a "lord" to align themselves with. Why does the Internet always have to be about "monetization"? I'd like to see open, standards-compliant offerings that are truly "free" as in freedom and very low cost, offerings that are purposely disruptive to the Apple/Google duopoly. Wouldn't you? I'm hoping Firefox OS proves to be one of these. Let's hope as a non-profit, they can remain sane and not driven towards the money pit. Good tech doesn't have to be about the money. Some of the best tech is both free as in freedom and free as in beer. Look at OpenBSD, for example. Not much better in terms of a secure server environment. Same with Firefox (for now).

Re:Google is too ubiquitous (0)

Anonymous Coward | about a year and a half ago | (#43455259)

So instead of a dialog, this post got a -1. Great. Slashdot really has drank the Kool-Aid.

What are you even talking about? (2)

sidragon.net (1238654) | about a year and a half ago | (#43456341)

[I]nstead of a dialog, this post got a -1.

You're talking about politics and conspiracy theories in an article about big data. Yes, that is off topic.

Why does the Internet always have to be about "monetization"? I'd like to see open, standards-compliant offerings that are truly "free" as in freedom and very low cost...

You're living in a dreamland. Like it or not, electricity, hardware, and wires cost money.

I'm hoping Firefox OS proves to be one of these. Let's hope as a non-profit...

FYI, Mozilla Foundation is funded, in large part, by Google.

Look at OpenBSD, for example. Not much better in terms of a secure server environment.

And it has scant adoption. Meanwhile, the rest of us are charging ahead and getting stuff done with steadily advancing tools rather than messing around with arcane operating systems that have 10-year-old feature sets.

Re:What are you even talking about? (0)

Anonymous Coward | about a year and a half ago | (#43457247)

I'll step in here...

OpenBSD does not have 10-year-old feature sets. OpenBSD has pioneered about every advance in security known to modern operating systems.

test (0)

Anonymous Coward | about a year and a half ago | (#43454445)

test to see how this works.

Re:test (0)

Anonymous Coward | about a year and a half ago | (#43457019)

You tested idiot-positive! Congrats!

Splunk (1)

Tuki (613364) | about a year and a half ago | (#43454451)

IMHO, Splunk crushes all of these solutions. While it is not "free", it has an incredibly short time to value. The first time I stood it up in production, it took less than two hours - most of our time was spent checking our work. Now, I can quickly analyze large volumes of data, and only have to manage a single software component. I love it!

Re:Splunk (0)

Anonymous Coward | about a year and a half ago | (#43454571)

You are totally clueless.

Re:Splunk (1)

mr_don't (311416) | about a year and a half ago | (#43454769)

What are some of the cost/performance metrics of Splunk when data gets large (common for game developers).

How does Splunk do on data sizes in the 500 Gig range? And how much does it cost?

Re:Splunk (1)

Tuki (613364) | about a year and a half ago | (#43460969)

There are big tech companies processing over 100 TBs a day. There are also game developers like Zynga that use Splunk for "big data" analytics. As far as cost, I am not sure what to say... contact your local sales guy ;). We process 500 GB a day, but incrementally upgraded our license. I'd have to dig up the costs of each...

Google? Really? (-1, Troll)

RocketScientist (15198) | about a year and a half ago | (#43454697)

I mean, if you want to have an ad agency host your databases, you've got lots of other options:

R&R Partners
Mccan Ericson
BBDO
J Walter Thompson
Omnicom Group
Young & Rubicam
DDB
Olgivy and Mather
Saatchi & Saatchi
Leo Burnett

Personally, I think I'll try to find a company where cloud computing is their core business, so they don't just write off the service a few months down the road as not-profitable and leave me hanging.

Why the hell would want to have your mission critical systems hosted by an ad agency?

Re:Google? Really? (0)

Anonymous Coward | about a year and a half ago | (#43454721)

You, sir, should be commended for having common sense. Sadly, too many others do not and will worship at the alter of the ad company because they are cool.

Re:Google? Really? (1)

Anonymous Coward | about a year and a half ago | (#43456581)

Bad analogy fail. Google doesn't create the ads, they serve them. Calling them an ad agency would be like calling TV networks ad agencies.

Not that it matters. The fact is Google runs the largest "cloud" computing network in the world. Of course, that doesn't necessarily make them the best platform for other businesses. But given Hadoop is based on Google's MapReduce and GFS designs, they clearly have expertise in the field, and to pretend otherwise is a complete and utter troll.

Article contains plenty of misleading comments (5, Insightful)

mr_don't (311416) | about a year and a half ago | (#43454745)

First of all, this article isn't a comparison or matchup - it's just a speculative post by someone who has done very little research and obviously lacks domain knowledge in the space. There is no mention of use cases, data sizes, performance, costs.

Hadoop is an open-source framework for distributed data processing, specifically an implementation of the MapReduce framework. BigQuery is a hosted service that allows you to run queries over massive datasets via an API. There are tools built on top of Hadoop that allow for fast querying over large datasets (Impala), and there are even tools that are not Hadoop based that provide this as well (Spark + Shark). However, actually using these tools is a whole different game - the author makes so mention of how many nodes/VM are required to compare the query performance of BigQuery.

Then there's data sizes. The author makes a strange claim that BigQuery "queries don’t run instantly; one of the samples took 3.3 seconds to grind through 3.49 Gigabytes of data. But that’s clearly fine for quick lookups." Huhn? What tool(s) are you comparing against? BigQuery allows users to run full table aggregate ad-hoc queries over really really big datasets (i.e. terabytes). In public talks, Google has demonstrated that it is possible to run regular expression match queries, with sums and aggregations, over several terabytes of data in under a minute. In order to do this with a MapReduce-based system, what needs to be done - perhaps use something like Hive, or write a custom MapReduce function - and what is the performance in this case? For the same use case, what is the cost of using some of the "OLAP" tools that the author describes? Would love to see some benchmarks.

Re: "In the end, BigQuery is just another database."

Huhn? BigQuery is not a database at all - it doesn't support CRUD operations on data - rather it is an append-only analytics tool. And conversely, databases, relational or not, aren't really the right tools for full table scan ad-hoc queries over many terabytes, which is what BigQuery is designed to do. BigQuery is a developer's product, and one that can be integrated with existing web apps via RESTful API. Hadoop has it's own development role and story (and tools like Cascading are really great) but it's not designed as the backend for interaction via a RESTful API out of the box - it takes a bit more work to provide Hadoop as a service for developers to integrate with an application.

Re: "The public version of BigQuery probably isn't even used by Google, which likely has something bigger and better that we'll see in five years or so."

BigQuery is based on Google's internal Dremel, which is used everyday by Google. There is a very public research paper describing Dremel (much the same as how Google described MapReduce years ago). Read about what is available in Dremel versus what is available in BigQuery: http://research.google.com/pubs/pub36632.html [google.com]

Re:Article contains plenty of misleading comments (1)

gajop (1285284) | about a year and a half ago | (#43455151)

This comment contains more information than the article.
Thanks :)

Re:Article contains plenty of misleading comments (1)

BlueItalian (1016784) | about a year and a half ago | (#43464007)

Thanks for this, I was going to post pretty much the same thing. In the same vein of Dremel, my company recently open sourced a similar tool called Druid. If you wan to play with it, it's available on github at https://github.com/metamx/druid [github.com]

Courtesy Devops_Borat (1)

Drunkulus (920976) | about a year and a half ago | (#43455731)

Mining of Big Data is problem solve in 2013 with zgrep.

Re:Courtesy Devops_Borat (0)

Anonymous Coward | about a year and a half ago | (#43477045)

twat.

And when you have Petabytes (0)

Anonymous Coward | about a year and a half ago | (#43456915)

You use vertica

Re: And when you have Petabytes (0)

Anonymous Coward | about a year and a half ago | (#43456939)

http://willsllc.github.com/blog/how-we-use-vertica-at-gsn/

Check for New Comments
Slashdot Login

Need an Account?

Forgot your password?