Big Data Platform MVP Development for a Global Consulting Company
About Our Client
The Client is an international company that provides business and technology consulting services for large-scale construction projects in oil & gas, chemical, power, mining, pharmaceutical, and other sectors.
Challenge
The Client wanted to develop from scratch a complex platform to streamline the delivery of consulting services to their customers. In a long-term perspective, the platform was expected to be enriched with machine learning (ML) capabilities to provide the Client’s customers with recommendations on their construction projects. The Client lacked competencies to develop the solution in-house and was looking for a reliable vendor to deliver the MVP.
Solution
ScienceSoft provided the Client with dedicated specialists to build the front end and the back end of the solution. Our team worked with the Client’s project manager based on the Agile methodology with one or two-week sprints.
In close collaboration with the Client, ScienceSoft’s experts have prioritized the solution’s functionality, based on the software requirements specification. Among the priority features were the secure storage of voluminous customer data, customer data archiving, and advanced data processing capabilities. When designing the solution architecture, the team took into account the scalability requirements and used Docker containers with Docker Swarm to achieve that.
In the course of 10 months, ScienceSoft delivered a complex MVP based on Delta Lake with a mechanism for multi-layered data storage. The platform enabled quick processing of heterogeneous data on the Client’s projects from multiple sources and the possibility to track the record of the data added, modified and deleted. The Client could extract any of the existing project versions in JSON, CSV, Excel, PDF formats, amend, and compare it with other versions, which facilitated the work on long-lasting construction projects. All alternations made in the versions could be automatically highlighted with different colors, making it convenient for the users to analyze the changes. Besides, ScienceSoft’s experts set a notification system to inform users on successful completion of certain operations, e.g., the compression of a large file.
Results
The Client was completely satisfied with the MVP that enabled collaboration on voluminous project data and facilitated the delivery of consulting services to the Client’s customers. The delivered solution is currently being enriched with machine learning capabilities.
Technologies and Tools
JavaScript, Scala, Python, React, Node.js, Nest.js, GraphQL, Apache Spark, Apache Livy, Hadoop, Delta.io, Knox.
