Can you choose different public data sources quickly, efficiently, and use that data for data science too? Finding correlations in public data sources used to be a lot of work to wrangle.
What I said above, for most, would require multiple people.
Data architect – helps you architect your data
Database admin – helps you access your data
Data scientist – helps you with predictive analytics
Developer – migrates data science models into code
BI Developer – helps you build a dashboard
We always intend to find correlations in public data sources.
Finding correlations in public data sources is a lot like finding a pattern on your wallpaper.
It takes consideration, time, planning, counting, memorizing, anyways…
But now we can automate most of this stuff 😉
Our prediction is that ethnicity, deaths, births, birth fertility, homelessness, and dozens of more measures will offer insights into average survey scores.
The more our measures correlate, the better we can predict our future.
Remember before reading the findings that statistics are opinionated. Correlation does not mean causation.
Noticing trends in our data sources.
Here’s a scatter plot displaying HCA survey average scores per state and Total Homelessness in 2014. The points on the graph are individual states. You will notice a negative trend sloping down as the survey scores increase, the amount of homeless people decreases too.
Scoring correlations in our data sources.
If we score the correlation between the two values, across every state, it would enable us to determine what metrics follow a similar pattern.
With scores, we can understand what correlates the most, or does not correlate to the hospital survey scores.
There are hundreds of hospitals per state. Here’s the top ten bar chart, showing how many hospitals per state are being averaged.
Mastering Multiple Public Data Sources
If you want to bring multiple data sources together, you need to be very good at SQL, data manipulation, data architecture, and a lot of spare time in a spreadsheet if this sounds like new skills. Finding correlations is another beast.
Finding a data scientist in your office may be tricky if they are busy or not available, which generates a deep dive in learning R, Python, Statistics, or Excel martial arts.
Luckily, we KNIME our data, which means we can do all the complex SQL, data manipulation, data architecture, and pipe it through a data science model, in the same window.
What would have taken my group multiple weeks of constant work, only took us 1.5 hours!
Looking for a public data source?
Know that picking a public data source has become readily available in the past few years, and being able to quickly identify powerful data sources has become a distinguishable skill to master.
If you are interested in picking up this skill, read our solution below.
I stumbled on this request in a team project last week!
“Everyone, pick multiple data sources…”
This can sound like an easy request…
Where do you get your data?
What data is accurate?
What source of data can we trust?
What decisions are we attempting to make?
We want to find data sources that will add value to our audience, their questions will have answers, and their next steps will be clear.
In this blog, you will learn how…
You will learn how our team quickly accomplished our project using one simple whiteboarding methodology.
You’re probably thinking, ‘Yes, of course, I can pick a data source, I’m passionate about x, y, z.’
The more you grow in the data industry, the more you realize every new data source comes with new hurdles or barriers to entry.
Is the data clean? Is the data usable? Do we have access to the data?
You begin applying padding around your work, based on previous experiences, and avoid painful experiences.
If you choose to read this entirely, you will be armed with a solution to any brainstorming meeting, and you will learn where we found 3 different data sources online.
That’s right, I’m talking about your future in data.
Eventually, you’re going to hit that ten-year threshold. A decade of database-related solutions.
When you hit ten years, you start to become scared of ‘lots of disparate data sources,’ and learn to avoid these workloads.
When your meeting is going great… Get ready for that meeting to slow down.
Someone is around the corner to suggest another table of data…
The data source will put the brakes on your progress. A new data source that has never been cleaned, prepped, approved… Major ‘nope’ in my line of work…
You were asked, “What data source do you want to use?”
Feel free to play along, mentally put yourself in a group setting, and give yourself a few minutes to pick multiple data sources to present in front of a class of 40 professional peers.
You will be building decisions out of these data sources. Graphs, pie charts, forecasting, insights, etc.
Lastly, work as a team to generate a data solution surrounding these data sources. While you’re at it, don’t get too involved in figuring out what answers you will gain from your data source.
We were tasked to generate a project to kick-start our success stories. In this process, it seemed really challenging to decide on what to choose.
It was interesting because even though we were all eager to get started… We struggled to find the right granularity.
You have access to making documents aka superpowers!
Data lakes, big data, data warehouses, … Each row of data offers a granularity.
Are we looking for an address? Zip code? State? Country?
Stating different parent and child relationships are all great and good. But until these words hit a piece of paper, Google Slides, or a whiteboard… We are left remembering what was said and making decisions based on our memory.
A zip code would be a more granular view of your data than State…
But without documenting our decisions, how do we collectively come together to suggest a path forward?
Also, why am I talking about documents?
Well – documentation is really helpful and it’s tangible information you can point at and say, “LOOK”… and in return, you will likely receive feedback.
Finding multiple data sources in any single industry can feel like a daunting task.
When you’re actively finding multiple public data sources – there are a lot of options.
Noticed we just said “public data sources”…
Yes, as opposed to a private data source! If you’re new in the data industry, let me dive in.
Public data source – safe to download, safe to share, available for everyone.
Private data source – not safe to download, not safe to share, not available for everyone.
Here are a few up and coming data source providers. These are public data sets and available to everyone.
Here’s my ‘buyer beware.’ Or maybe we should say ‘extracter beware.’
Anyone can add a data source, anyone can change the data source, and Wiki’s are public – anyone can make an edit. That’s why teachers recommend we avoid using them in scholarly reports because some random person likely added that information on the Wiki.
How did our team pick a data source?
Originally, we struggled to find a data source because we were not documenting our ideas.
Our ideas were lost as soon as we said “anything” out loud.
We would say a great idea, agree it was a great idea, but failed to generate next steps.
What made a difference was documenting our brainstorming session on the dry erase board.
Having everything on the whiteboard allowed everyone to see what was being said.
Not sure how to get started whiteboarding?
Whiteboarding is usually as simple as making a word cloud and circling good items and drawing a line through bad items.
The table below demonstrates Pearson Correlation value and ap-value, to determine its statistical relevance. The smaller the p-value, the more significant our data becomes.
We are using Pearson Correlation to compare the HCAHPS hospital survey data against different ways to categorize the count of homeless people per state. Sheltered Homeless Individuals and Sheltered Unaccompanied Youth are the most statistically relevant.
Using HCAHPS Hospital Survey Data from Medicare.gov was the beginning of a group project. And a quick hypothesis that if your hospital gets bad survey scores, it’s likely because there are other outliers that correlate.
HCAHPS Survey data gives us quick insights into many different values but to do a quick analysis, we want to stick with an average of the values, and the data offers an average, per hospital.
Thousands of people take a survey or submit a survey.
The average of those values shown in the HCAHPS Hospital Survey Data.
A score from 1 to 5.
1 being the worst and 5 being the best score.
Hopefully, your hospital has a decent score!
Dave Chappelle laughing about hospital survey data
There are hundreds of hospitals in the HCAHPS Survey Data.
When looking at the website above, it’s important to understand how many hospitals per state, to understand the sample size at a state level.
The bar chart below is a distinct count of hospitals per state.
HCAHPS Hospital Survey Data – the number of hospitals per state. In Tableau Desktop by Tyler Garrett.
HCAHPS Hospital Survey Data has many Correlations
HCAHPS Hospital Survey Data has a lot of negative correlations, from a Pearson correlation perspective. In my studies of this data, it was easy to see several correlating factors that supported my initial hypothesis.
If you have bad survey scores,…
It’s likely you also have a high population of homeless people, high death rates, low fertility rates, and also Census 2010 households ethnicity aggregations will more than likely tell a story too.
Finding a Correlation is not Finding Causation.
Just because your state loses the test – remember its an average per state. Correlation is not causation.
Yes, this data can tell you the worst hospitals in your state. It’s important to note; correlation is not causation and it’s important to talk about these potentially predictive outliers.
Follow the link above and start analyzing other data related to the medical field or read more about the metadata below.
Usually, I would leave that data solution here but today I’m a little past that and will be blogging more about the correlations, and try to find statistical relevance in these measured values.
More about HCAHPS Hospital Survey Data
HCAHPS Hospital Survey data has a little bit of metadata too.
As of May 19, 2018, HCAHPS Hospital Survey Data has been over 20,000 downloads and over 166,000 views.
The row count is a staggering 266,000 rows by 23 columns. Learn more about this dataset here. It’s a list of hospital ratings for the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS). HCAHPS is a national, standardized survey of hospital patients about their experiences during a recent inpatient hospital stay.
GUS FRUH – None shall pass, None shall pass at Gus Fruh in Austin Texas!!!
My son grabbed a very warlock like staff. #lotr
He enjoyed playing in the water at Gus Fruh.
Runoff water from the most recent storm in Austin, Texas.
By the way… when it rains in Austin Texas, we get many nice attractions. Like Gus Fruh!
But be safe around the water when it’s scary, people have come up missing and found in the water.
So, be careful and don’t do stupid stuff in rapids after a heavy rain.
Gus Fruh was originally shown to me as a climbing wall, bouldering cave, and a bit of hiking.
For rock climbing at Gus Fruh, be on the lookout for a few good reads;
RockClimbing.com – “Just Past the Fern Bar Wall, some Sport some Trad, routes listed from left to right.”
RockClimbing.com really goes all out when talking about Gus Fruh.
If you want to learn more about places on the Greenbelt, check out this page here.
MountainProject.com – Despite the fact this area contains some excellent walls and routes, it is not quite as popular as other areas such as Seismic or New Wall. This area is well shaded.
Because most of the clips start higher than most rock climbing places around Austin, Tx.
It’s not as easy to get to as other walls and new wall offers a lot of routes natives prefer.
My wife recently started going to the greenbelt and exploring. She found a great little swimming hole, the locals call it Gus Fruh!
This magic stick my son Andoni found also has some sharp edges, so I’m going around blocking face strikes.
You’re welcome future andoni face..
I had to make sure you didn’t hit these people too, they were all about the, “none shall pass at Gus Fruh!”
Gus Fruh was perfect for us or Rock climbers.
Gus Fruh is perfect for the family or rock climbers.
You really enjoy being outdoors more than indoors, when you’re feeling stressed or blue, be sure to take a walk outside, go to the park, and relax. You enjoy being outdoors way more than indoors, do what you can to enjoy the environment.
If you’re considering going to Gus Fruh, head there early before the hungover crowd arrives, and leave when they start appearing!