Some Realities of Big Data
- The critical, business aspect of analytics must be built in-house. This work cannot (must not) be outsourced.
- Analytics-driven mindset is a deep, organization-wide, culture change. This will take years to develop.
- Prediction is the easy part. Actionable insights are of real value.
- You can (and must aim to) analyze the whole population. Sample is old school and (almost) obsolete.
- Missing data is ok. Machine learning algorithms can work with gaps.
- Machine learning algorithms will be incorrect in the beginning. They will improve in accuracy as they get trained with more data.
- You will never have all the data (you think) you need.
- You will never be able to analyze all the data at your disposal.
- Not all predictions will be repeatable, scaleable.
- There is more value to 3 different data-sets of say, 100,000 records each, than one big set of 10 million records.