Distributed Masonry is proud to announce Onyx 0.10.0, a major step forward for Onyx’s stream processing engine. Version 0.10 brings enormous performance improvements, support for in-order message processing, and enhancements for writing stateful streaming applications. In this post, we’ll take a look at the recent activity on the project, summarize what’s been delivered, and look forward to what’s ahead. New streaming engine Onyx 0.10 rewrites the messaging and fault tolerance layer using the best in...
Read more...I’m excited to announce the official release of the Java API for Onyx, closing Onyx’s oldest GitHub issue: 2 years, 9 months, 4 days after its creation. onyx-java is a comprehensive API that opens up access to virtually every Onyx feature from Java - without writing a single line of Clojure. onyx-java is a significant advancement as it marks the first language other than Clojure to have end-to-end support. The Java API is particularly interesting...
Read more...Distributed Masonry is pleased to offer the first public look at our newest product: Pyroclast. Pyroclast is a fully-hosted cloud PaaS that extends the Onyx Platform’s core mission: to make large scale distributed data processing more economical for businesses. Working with high volumes of data typically involves curating a highly skilled, high salaried team of engineers, a significant investment in tooling, and a lengthy delivery schedule between the project start date and return on investment....
Read more...I’m pleased to announce the release of a new library for Onyx: onyx-local-rt. onyx-local-rt is an alternate runtime for Onyx that is pure and deterministic. This runtime, as the name suggests, differs from the distributed runtime in that it only runs locally. onyx-local-rt has no dependencies aside from Onyx itself, supports nearly the full functional API, and is capable of running in ClojureScript. Backing up for a moment, we can break Onyx down into two...
Read more...2016 has been an exciting time for Onyx. Over the last year, Onyx has been widely adopted by companies with high volumes of data and strict latency requirements. Distributed Masonry, the entity that commercially supports Onyx, has had the privilege of seeing the platform used to solve a wide variety of problems. After more than a year of providing consulting services and training, Distributed Masonry has taken the next step and has begun build a...
Read more...I’m pleased to announce the release of Onyx 0.9.7. This release features a major design upgrade to improve cluster-wide scalability. We’ve also shipped a number of highly requested usability fixes that will make developing with Onyx much more pleasant. See the full changelog for all patches to 0.9.7. We’re frequently asked about the maturity of Onyx, so in this post, I’m going to discuss the design changes that we made - specifically in the context...
Read more...Onyx’s strength has always been it’s data-driven nature thanks to its data based API. This API can be thought of as a low-level language for your distributed computation. A data structure based API forces you to understand and decide all-the-things upfront. A common approach to conquer this problem is to include a sort of meta-language in the data structure. In all but the simplest cases, this leads to recreating a new programming language in your...
Read more...I’m excited to reveal one of the most exciting usability enhancements yet to the Onyx Platform. Onyx version 0.9.6 contains a revamped module for performing static analysis on job submission. In this post, we’ll look at how Onyx uses static analysis, what the benefits are, and how we’ve significantly advanced upon our previous work. Data Driven APIs: A Double Edged Sword Onyx’s dominant quality is that it uses a purely data driven information model as...
Read more...Deprecation notice: We’ve discontinued support for this product. We recommend directly using tools like Kubernetes for Onyx deployment. We’re pleased to announce the release of our newest creation - Engraver. Engraver is a tool for managing and deploying Onyx cluster infrastructure. We’ve developed Engraver to address operational concerns, one of the most painful aspects of dealing with large distributed applications in any ecosystem. Engraver is our recommended approach for building production-grade clusters, though its usage...
Read more...Strapping in for a Jepsening Onyx is a high performance, distributed, fault tolerant, scalable data processing platform. Onyx programs are described in immutable data structures allowing jobs to cross language and machine boundaries at runtime. Testing Distributed systems are incredibly powerful for dealing with massive amounts of load and providing high availability. Ensuring that your system behaves correctly under stress, however, is a notoriously difficult problem. All of this power is useless if you can’t...
Read more...Doubling Down on Onyx Distributed systems are hard. Building Onyx - the high performance, fault tolerant, scalable distributed data processing platform - has been no exception. From day one when I wrote the first line of code, I realized that I was investing in a long term engineering effort targeted at commerical adoption. Since then, I’ve been lucky enough to have dozens of wonderful developers contribute code, documentation, tutorials, and other materials. While I always...
Read more...I’m happy to announce the release of Onyx 0.8.4. This release includes our next-generation scheduling component, largely backed by an independent library named BtrPlace. Today we’ll be introducing a new feature called Task Colocation, an advanced scheduling technique that uses information about the physical layout of your cluster to optimize the allocation of work units. First, let’s talk about why something like this is warranted. A Distributed System Much of the challenge in designing a...
Read more...I’m tremendously proud to announce the official release of Onyx 0.8.0. In the time since the last release, we’ve been working hard on solving an inherent complexity in stream processing - state management. Today, we’re unveiling a new suite of features to alleviate much of the difficulty in crafting stateful streaming programs. (full changelog here). Onyx is a scalable, distributed, fault tolerant, high performance data processing platform written in Clojure. It’s able to transparently handle...
Read more...