While writing SQL in bigquery I was thinking about how this task comes with a cost. First, it takes a lot to do all of this tech. Second, it takes a lot of patience to get bigquery to work. We are trying to re-write tableau calculations in SQL because we have found that the prototyping stage is done and we need to deal with a specific set of lag time.
I started learning SQL while I was in college at University of North Texas, while learning SQL I was also working 3 fast food jobs, and that’s the axe I needed to sharpen to be able to say I”m writing SQL for money, and using my personal LLC to bill and invoice clients. It’s cool to have ten years under your belt since college. It’s also a cool time for finding jobs that require a decade of experience. That would be an interesting role. It’s what I offer now publicly to any company or government agency. Getting to this point takes time.
I like Bigquery for the fact that it’s not that bad after you get past the UX of getting here. I could write a blog about that one day but this isn’t a high priority because no one is asking for a bigquery tutorial. Maybe this will be a good future thing to discuss.
Bigquery allows for non-technical people to do things that are rather deep in the SQL realm of things and you can do this by simply being patient and googling a bit more than you might desire. A few more stack overflows than you may desire will be needed to grow. But when is that any different from other technologies. Just know people are not incentivized to keep those posts up-to-date.
Tasks are things to be completed and often if you’re trying to do a task more than once it needs to be automated.
(I made a task scheduler that’s free.)
Parent child relationships are not always present. Often we are left to come up with these solutions in life by best guessing. Then there’s the time where you grab someone 3x better than you to challenge the entire solution.
A literal cost. Also, you start to look at your skills for gaps, and even worse you consider yourself wrong most of the time you’re doing everything.
Most don’t speak of the cloud in a negative light. Not the cloud data adopters.
The cloud is an expensive place to run applications and only big companies can utilize these tools.
To be able to afford a cloud compute means you’re doing good financially.
Cloud computing[1] is the on-demand availability of computer system resources, especially data storage (cloud storage) and computing power, without direct active management by the user.[2] Large clouds often have functions distributed over multiple locations, each location being a data center. Cloud computing relies on sharing of resources to achieve coherence and economies of scale, typically using a “pay-as-you-go” model which can help in reducing capital expenses but may also lead to unexpected operating expenses for unaware users.[3]
Cloud computing is cool. Also, cloud storage is expensive too.
Cloud storage is a model of computer data storage in which the digital data is stored in logical pools, said to be on “the cloud“. The physical storage spans multiple servers (sometimes in multiple locations), and the physical environment is typically owned and managed by a hosting company. These cloud storage providers are responsible for keeping the data available and accessible, and the physical environment secured, protected, and running. People and organizations buy or lease storage capacity from the providers to store user, organization, or application data.
Cloud storage services may be accessed through a colocated cloud computing service, a web service application programming interface (API) or by applications that use the API, such as cloud desktop storage, a cloud storage gateway or Web-based content management systems.
As you start writing SQL and build sophisticated transactions, you start to learn to write good and bad SQL.
In big data, the mistakes cost money. There’s more thought in big data. You don’t want to query a lot of data unnecessarily.
I enjoy work. Work is fun. Data is a fun puzzle.
Starting my tableau consulting services company helped me put my resume in front of people and often I get a chance to help with big data solutions.
The past few days has been focused on optimizing the SQL we wrote last year. The more you get into this big data world, the more you think before you run your query.
Now that I have 10 years of experience of data under my belt it’s fun to look at big data projects for “how we can write this better.”
When data is small and not 30 to 200gigs each run, life is easier, and when it’s huge… you have to pause and think. Otherwise you’re literally blasting your cash down the data pipeline.
Well if this blog sucks, you can jump over to my twitter where I just posted all the #bitcoincrash meme’s and videos created by the adopters.
Trackbacks/Pingbacks