By Keith Steele and Jim White, Chair and Co- Chair of the EdgeX Technical Steering Committee
Over the past few years, it became clear that many IoT deployments would face challenges related to latency, network bandwidth, reliability and security, which could not be addressed in cloud-only models and with at least 40% IoT data stored, processed, analyzed…at or near the edge.
The Edge has since become the focus for major new developments, with some pretty big technical challenges to solve.
The Edge is a pretty heterogeneous world with a variety of platforms, a diverse collection of OS and OS variants, multiple hardware options and a ‘Thing’ protocol soup. Industrial: BACnet, Modbus, OPC-UA etc; Wireless: BLE, Z-Wave, Zigbee etc; Messaging: MQTT, AMQP, DDS etc; a variety of cloud platforms, Azure IoT Hub, AWS IoT Platform, Google IoT Core, IBM Watson IoT Platform. Add to that your favorite selection of Applications, edge analytics/intelligence, security, system management etc. Where do you start?
The EdgeX Foundry community was formed to meet this technology challenge with the ambition of creating the pervasive open Edge platform for the Internet of Things that would drive innovation, global IOT market adoption, velocity and scale.
We aimed to build and promote EdgeX as the common open platform unifying edge computing enabling and encouraging a rapidly growing global ecosystem of IoT solutions providers to create a rich set of multi-vendor interoperable plug-and-play components. These components would be ‘EdgeX certifiable’ to ensure interoperability and compatibility. We would also collaborate with relevant open source projects, standards groups, and industry alliances to ensure consistency and interoperability across the IoT. We were ambitious!
After 4 code releases, we feel ready to declare the product V1 and ready for ‘live project deployment. EdgeX Foundry’s Edinburgh (Version 1.0) is quite the milestone in any software product, but what does it mean for EdgeX?
First, it means the EdgeX development community now believes the EdgeX functionality – especially that as represented by the service APIs – has reached a level of stability and maturity that users of EdgeX can embed EdgeX in their products and solutions without worrying that the underlying framework changes overnight. It also means that the EdgeX development community has signed up to carefully monitor backward compatibility going forward.
The community will provide warnings to its user base when a new major version of the platform is forthcoming that will be non-backward compatible (as denoted by semantic version 2.0), and checking that any upgrade to a minor release is backward compatible and thereby allowing users to upgrade with minimal or no issues.
We Have Arrived
Unofficially, to those that have worked to design, architect, develop, test, produce, and market EdgeX for the past two years in the Linux Foundation, it means “Arrival”. We have arrived – not at our destination, but at a point to which we expect and anticipate an explosion of EdgeX use in IoT/edge deployments.
We have already seen evidence of this in the months leading up to the formal release. EdgeX is now being used as the base platform of a multi-billion-dollar global system integrator while at the same time being used by a local ice-cream manufacturer in Sri Lanka.
We are now over 100 unique contributors to the EdgeX project and between this past April and May alone EdgeX developer downloads from the project GitHub jumped 80% from 2500 to 4500 per month! We have also seen over a 100% increase in the amount of traffic to the EdgeX website.
The Edinburgh release, along with being version 1.0, includes a lot of new features and improvements underneath the covers.
Since project inception, two EdgeX services provided the means to register for and distribute data to north side clients like HTTP endpoints, MQTT topics and cloud providers (Azure IoT Hub, Amazon IoT Core, etc.). The Export Client service allowed anything (internal or external to EdgeX) to register as a consumer of the sensor data collected by EdgeX.
The Export Distribution service filtered, transformed and formatted the sensor data for each registered consumer. Unfortunately, as we discovered over time, these services were not as flexible as we would have liked and they did not scale well as the number clients increased.
The EdgeX community came together to architect/design a new form of north side service – called application services – which are principally designed similar to function-as-a-service engines scene in the market today.
With the Edinburgh release, the new application services are available to use as replacements services to the export services for getting EdgeX data to consuming client systems. These services are created one-per-client and so can be as small or as big as they need to be without impact to the overall system. It will, when completely functional equivalents, allow us to retire the export services in a future release.
The Edinburgh release features new and improved SDKs. EdgeX now features two device service SDKs – one in Go and the other in C – for creating south side connectors that help collect data from sensors and drive actuation in devices in protocol dependent fashion.
Functionally equivalent, the two SDKs allow the user community to pick the language most convenient and comfortable to connect their “things” based on use case and hardware requirements. In addition to the south side SDKs, there is also an SDK to create the aforementioned application services – the new north side EdgeX services. This helps ease the creation of new north side connectors that deliver data to on-prem, cloud, or enterprise applications.
The application functions SDK allows users to create services from a collection of functions. Each function acts on the event data coming from a sensor and allows the function to take a single step in preparing the sensor event data for north side systems before passing it to the next function. Built in functions include those that filter and format the data, but the SDK allows users to easily create and stitch together any functions they can conceive to prepare and get data to all sorts of applications and system.
Binary Data Support
EdgeX has always dealt with discrete integer, float, string and boolean data coming from sensors. Things like temperature and vibration readings, status indicators, etc. obtained from sensors was easily captured, stored, used locally to actuate and/or shipped to north side systems. With the Edinburgh release, EdgeX can now ingest and use binary data. A picture, short clip of video/audio, or any data that can be represented in Concise Binary Object Representation (CBOR – see http://cbor.io/) can be sent into and through EdgeX services. CBOR is a binary data serialization format loosely based on JSON. This enables EdgeX to participate in a wider variety of IoT/edge use cases.
Like any application, EdgeX uses a lot of infrastructure from databases to messaging systems. In this release, there was a lot of work under the covers to add abstractions around infrastructure like the database, configuration service and messaging pipes. This abstraction allows for easier replacement of infrastructure used by EdgeX in the future or by the user community. For example, if you don’t like the fact that EdgeX uses Zero MQ between Core Data and north side services – replace it with an MQTT message broker and MQTT messaging.
The improvements in abstraction and decoupling from infrastructure was particularly highlighted in this release by the project’s addition of a complete set of Redis using services. Thanks to the Redis Labs partnership, and the abstraction work, users can now choose to use Redis or Mongo with core, metadata and other database using services provided directly by the EdgeX project.
Testing and Cleanup
Much of the work completed for the Edinburgh release won’t be seen or used by consuming projects. It is work to thoroughly test EdgeX.
A new performance framework was created to help measure how well EdgeX behaves and to ensure it can achieve its use cases within acceptable time limits. Many additional unit and black box tests were added. New infrastructure was also added to measure how much of the system is being tested (something that the community hopes to improve over the coming releases). Additional under-the-cover improvements in flexibility, understandability, and performance were made in areas of configuration bootstrapping, scheduling, device profiles, API gateway and the security secret store. Finally, EdgeX was outfitted with the ability to trace data from ingestion at the sensor to export at north side services to help support better debugging and performance metrics for this and future releases.
Documentation and Help
In advance of the 1.0 milestone, the community also worked hard to improve assistance provided to the user community. We switched from RocketChat to the better known/used Slack (edgexfoundry.slack.com) for project related chat communications.
These communication channels are used for internal as well as user community support and help. Today we address hundreds of questions weekly with a community of nearly 400 (and growing) members across a variety channel topics.
We also made significant additions and updates to our documentation (docs.edgexfoundry.org). The addition of several getting started and example guides we hope will help to onboard new users and developers more quickly.
During this release cycle, EdgeX became a member of LF Edge (lfedge.org). LF Edge is an umbrella organization that aims to establish an open, interoperable framework for edge computing independent of hardware, silicon, cloud, or operating system. In the LF Edge, EdgeX Foundry became the first (and to date only) project to make it to Stage 3. Stage 3 is for projects that have reached their growth goals and are now on a self-sustaining cycle of development, maintenance, and long-term support.
As always, we cannot thank enough the dozens of people in our member companies, individual volunteers, and the Linux Foundation that create EdgeX Foundry and make it the best open source IoT platform available today.
Version 1.0 is quite the milestone and it is on the backs of our community that it has happened. Thank you.
EdgeX speeds time to market, reduces the cost of IoT Edge infrastructure functionality, which together with the power of an open ecosystem of complementary providers maximizes the potential IoT projects to be a success. This is the power or the EdgeX ecosystem – creating opportunity for commercial value-add tied to the open APIs that facilitate interoperability at scale.
This is the power of EdgeX, bringing together an ecosystem of both open source and commercial value-add. Further, the open, vendor-neutral nature of EdgeX is critical to helping the world realize the true potential of IoT.
This milestone clearly important bit it is “perhaps, the end of the beginning” as Winston Churchill once said. The future of EdgeX is bright with a growing ecosystem of users, partners and developers with releases, deployments, and efforts to making the creation of IoT/edge solutions even easier in the future. Check back with us in October as we release “Fuji” – our anticipated fifth community release.
Thanks for reading, we hope you’ll join us and we certainly hope you’ll try our code … here.