On BigQuery, streamed data doesn’t show up and running standard SQL queries with legacy SQL is hard. Also, many wrongly think BigQuery doesn’t compress data.
Here are the top three WTFs that I regularly hear from new BigQuery users.
BigQuery has an awesome streaming API. When new users start streaming data into BigQuery, the first thing they do is eyeball the table using the preview to check if the data have arrived safe and sound (and understandably so) . The problem is that there is a delay in streamed data showing up in the table preview. In addition to that, over the years, I’ ve come to learn that the table preview (and the table details for that matter) is buggy, and unfortunately, can’ t be trusted. Mwaaah!
This one can be absolute noodle-scratcher for new users. I’ ve even witnessed some folk have total meltdowns and hurl their machines across the office (at me) in utter frustration. BigQuery has two SQL dialects: standard and legacy.
The confusion arises from the fact that by default BigQuery runs queries using legacy SQL. As a result, I commonly see new users start writing their queries using standard dialect (as they should do) , but they fail to realize that they need to explicitly tell BigQuery they are using standard SQL. As a consequence, their queries won’ t run, and they get all sorts of gnarly errors spat back at them. Ouch!
Just like everything else, BigQuery isn’ t perfect. But it’s pretty darn close if you ask me. Yes, it’s got some minor annoyances — but show me a tool that doesn’ t, and I’ ll buy you a pint (if you live nearby and like Guinness) . The way it scales and its zero-ops model are what set it apart from all the rest. When I can effortlessly smash through 100 billion rows (10TB) in 45 seconds, then I’ ll happily live with a few monitors been thrown in my direction by new users after they’ ve spent a day trying to figure out why their streamed data isn’ t showing up. Ah, sheer bliss.
