Many organizations’ primary goal when working with Aptitive, especially now, is to improve their data and analytics capabilities so they can get a clear historical perspective and be better able to forecast what the future holds. Chief among their data and analytics concerns? Speed and efficiency. So it’s no surprise these focuses are at the core of my five modern data and analytics predictions for the next year and beyond.
1. Faster and More Efficient Data Engineering
Enabling faster and more efficient data engineering is the most important thing happening in the data ecosystem right now. One pivotal technology I’m particularly excited about is Apache Arrow.
With Arrow, you can use the tools you want and the data platforms you want and work with them all together, instead of independently. Essentially, it allows more efficient data transportation by avoiding unnecessary disk writes and performing in-memory operations. This speeds up the performance of applications that process large data sets without any retooling required from your development team. As more and more technologies adopt Arrow, it’s going to be an “under-the-hood” game-changer.
2. Near-Real-time Access to Data
This is what everyone wants right now, and the first step toward real-time data access will be to phase out daily batch processing. Snowpipe is one service Aptitive’s teams are using to facilitate micro-ingestion of data. It enables you to load data in micro-batches, making data available in minutes.
Our teams may also consider orchestrating real-time views of data and batch views in parallel, combining those when running the ETL. This allows for ongoing near-real-time access to data while maintaining overall processing efficiency and a big-picture view of your data.
3. Keep-It-Simple Data Architecture
Our goal at Aptitive is to quickly get our clients to business insights. The key to providing lasting access to these business insights is ensuring our clients are able to support the data architecture that comes out of our projects. So our clients don’t need to seek out talent with the knowledge and skills to provide ongoing support – a difficult ask – we’re focusing on making our architectures simpler, easier to maintain, and easier to extend.
4. Faster Time-to-Insights and Machine Learning
This goes back to the above theme: keeping data architecture simple to increase time-to-insights. While ultimately a great back-end data architecture is important, you don’t need to wait on perfection – focus on getting those business wins and implementing predictive machine learning use cases.
Some clients may start with no reporting capabilities but want to get to machine learning as fast as possible. Aptitive works to get these clients the data they need without compromising the back end. In other words, we’re starting with quick insights and then iteratively building out the architecture from there.
5. Widespread and Embedded Analytics
Many organizations are familiar with having to navigate to separate applications to update data and to view the dashboards used for reporting on that same data. That’s where embedded reporting in your business applications, or center of excellence reporting portals, can play a big role.
Think about a budgeting or financial planning application. Customers need to be able to update their data and also get reporting out of the tool, rather than accessing multiple platforms. This could be accomplished by building a large, centralized modern data warehouse. Adopting this model more broadly is likely the biggest upcoming evolution in analytics.
I expect each of our projects this year to touch on more than one of these five modern data and analytics predictions. More than that, these will be the building blocks of data and analytics developments in the years to come.
- 3 Data Integration Best Practices Every Successful Business Adopts
- Advanced Data Insights in 2020
- Building a Data Analytics Roadmap? Answer These 3 Questions
Fred Bliss is the CTO at Aptitive. He brings over 15 years of experience solving complex business problems through data solutions including cloud integration, data warehouse modeling, ETL, and front-end reporting implementations.