Industrial organisations Involved
Companies in the transport sector ; Companies in the Logistics and Supply Chain Management sector.
Technical/scientific Challenge
The considerable technical challenge is to accurately identify, extract, transfer and integrate reliable and fast changing data from traffic events and alerts to be followed by sophisticated and exhaustive analysis (descriptive and predictive). On one hand, the large volumes of data being transferred, the enormous diversity of traffic sensors, events, and alerts to be encompassed, and the proliferation of the transmission protocols is a very difficult technical and scientific task. On the other hand, consistent assurance and maintenance of collected data quality while selecting the right tools and algorithms for data analytics and presentation, is not a trivial task.
Solution
To solve those challenges, a referential IoT architecture is proposed, consisting of the following components: data extraction and cleaning module NiFi, data distribution module Kafka, data storage module Hadoop HDFS, data analytics modules Hive, Impala and Hue, and data presentation module PowerBI. Building such an architecture enables flexibility and speed in data loading and cleaning up from a multitude of traffic information platforms, different payload formats (CSV, JSON, XML, etc.) and transmission protocols (HTTPS, FTPS, SFTP, etc.), reliable ingestion, dispatch, consumption and storage of the extracted traffic events, powerful data analysis engines for both streaming and batch processing, and, finally, convenient visual presentation of the results ready to be used by the business stakeholders.
Business impact
The big data, generated in the Transport and Logistics & SCM fields, is not adequately collected, organized and used for the optimization of business processes, activities and services.
The companies, operating in those sectors, could benefit from the real-time analytics and predictive instruments, for:
- Optimizing transportation routes
- Decreasing transportation and logistics expenses
- Optimizing staff management
- Warnings and alerts for traffic incidents, natural disasters and extraordinary events
- Real-time management of autonomous vehicles
Benefits
- Flexible and reliable, multi-protocol and multi-network data extraction
- Distributed high availability file system and event dispatching system
- Powerful data analytics including streaming and batch processing
- Cost effective Open-Source components