Groundbreaking parallelisation technique advances computational efficiency

MIT researchers have pioneered an innovative system that promises to accelerate the execution of computer programs while maintaining their accuracy.

MIT researchers have pioneered an innovative system that promises to accelerate the execution of computer programs while maintaining their accuracy. This groundbreaking technique hinges on the concept of parallelisation, allowing certain types of computer programs, specifically shell scripts, to run at significantly faster speeds. The revolutionary aspect of this system lies in its ability to ensure the accuracy of results even as programs execute at dramatically enhanced speeds. This breakthrough opens new horizons in fields such as web scraping, data analysis, and natural language processing, among others.

The power of parallelisation and universal usage

At the heart of this technique is the Unix shell, an omnipresent programming environment created half a century ago and still widely utilised. The system created by MIT researchers parallelises programs running in the Unix shell. This means that it dissects program components into smaller parts that can simultaneously operate on multiple computer processors, leading to a notable reduction in the original runtime of tasks.

Nikos Vasilakis, a research scientist at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), emphasises the impact of this development. He notes, “The potential users of such programs are innumerable – data scientists, biologists, engineers, economists, and more. They now have the ability to automatically expedite their programs without the worry of inaccuracies in the results.”

Effortless acceleration and guaranteed accuracy

The system simplifies the process for programmers who develop tools for various users, ensuring no special adjustments to their program commands are required to enable this automatic, error-free parallelisation. Vasilakis, who leads a committee of researchers working on this system, affirms the ease and reliability of this automatic parallelisation.

Vasilakis is the senior author of the group’s latest research paper, presented at the USENIX Symposium on Operating Systems Design and Implementation. The co-authors hail from diverse backgrounds, including MIT, the University of Pennsylvania, Warsaw Staszic High School, Aarno Labs, Google, and the Stevens Institute of Technology.

The unix shell and the challenge of parallelisation

The researchers’ system, dubbed as PaSh, centres on scripts that run in the Unix shell. Scripts, sequences of commands that instruct a computer to perform a calculation, have always presented a complex problem when it comes to correct and automatic parallelisation – a problem that researchers have strived to resolve for decades.

While the Unix shell allows for multilanguage scripts, its flexible and dynamic structure makes it challenging to parallelise these scripts using traditional methods. For instance, some parts of the program are dependent on others, determining the sequence in which components must run. Incorrect sequencing results in program failure.

The PaSh system resolves this issue through a “just-in-time” parallelisation approach, effectively speeding up more components than conventional methods that attempt parallelisation in advance. This also ensures the accelerated program remains accurate.

The dual promise of acceleration and accuracy

In testing, PaSh proved its effectiveness by running hundreds of scripts without breaking a single one. The system showcased its power by running programs six times faster on average compared to unparallelized scripts, with a maximum speed-up of nearly 34 times. It also boosted the speeds of scripts that other approaches failed to parallelise.

PaSh’s performance has two-fold significance. On the one hand, it represents a fully correct transformation, and on the other, it provides an indirect benefit. Its design allows other researchers and users in industry to build on top of this work. Vasilakis expresses his excitement at the prospect of receiving user feedback and seeing how they enhance the system.

Whether it is in the world of web scraping, data analysis, or any other field that involves large-scale computations, the introduction of PaSh and the concept of error-free parallelisation mark the dawn of a new era. This is a time where computers can deliver faster results without the fear of errors – a future of computing that is both exciting and promising.

The research was supported by the Defense Advanced Research Projects Agency and the National Science Foundation.

Vey Law

Vey Law is a reporter at Breakthrough.

Latest from Blog