As everyone on earth spends more and more time online, data volumes have exploded. Every click and swipe is a new data point. The average person spends nearly 6.5 hours online each day, which creates 15.9 terabytes of data daily. But the volume of business data puts that to shame. 41% of organizations are managing 500 petabytes of data at any given time; nearly 30,000x of the average person.
In 2006, British mathematician Clive Humby coined the phrase, “data is the new oil.” Global data volume has grown over 500x since then! But if data is the new oil, where are the pipelines? As efficient as you might think the world has gotten at data collection and transfer, you’d be surprised.
Extract, transform, and load (ETL) is a methodology for “combining data from multiple sources into a large, central repository.” While there are large established ETL platforms, some would joke that Microsoft Excel is the most common ETL tool in the world.
People can use Excel to manipulate comma-separated value (CSV) files; a text-file format that uses “commas to separate values, and newlines to separate records.” Most people think of CSVs as the crappier spreadsheet they accidentally used to download their bank statements or birthday contacts.
But in reality, most Fortune 500 companies are using hundreds of millions of spreadsheets each year to send data back and forth. Today, that’s the cutting edge in terms of pipelines for the new oil. Unsurprisingly, they’re quite brittle.
OneSchema is attempting to build a better pipeline. Whether companies are processing CSVs every now and then, or consistently as part of their business, OneSchema has built embeddable capabilities around every use case. The biggest problem that OneSchema is solving is the massive fringe cases in any given data. Whether it's dates, numbers, addresses, phone numbers, and on and on. For any particular data point, there are 20+ ways to store it.
OneSchema believes there is a place for a universal data normalization engine. Capture the edge cases across millions of spreadsheets and find the unifying language to cleanly manage the entire collection. From there, AI is perfectly suited to step in and optimize the manual processes that massive enterprises are spending tens of millions of dollars and thousands of working hours to manage. That’s the dream.
To learn more about OneSchema and how the company is building the pipelines for the new oil, check out our new memo on the company.
