All Posts By

jwhite

An Enhanced Delhi Code with More Bells and Whistles

By | Blog

A few weeks back, the EdgeX Foundry community released Delhi.  This release (the third public major release of EdgeX in a little more than a year) included many new features and I outlined them in my last blog post . Today, the project announced the availability of an enhanced Delhi release, with a smaller collection of new and updated capabilities built on top of Delhi.  

The Delhi code release offers so many new features, I’m not going to list them all. Instead, I’d like to focus on what’s new with this enhanced Delhi release.  In particular, the enhanced version of Delhi begins to allow for freedom of choice with regard to databases in EdgeX. With this release, several of the services (core data, metadata and export client specifically) have been engineered to use either MongoDB (the long used default persistence for EdgeX) or Redis. This improvement to the EdgeX platform is significant for several reasons:

  • It highlights the ability for organizations to select and more easily use the data store that best fits their use case and system needs.  Platform support, performance characteristics, licensure issues, in-memory options, etc. are all architectural considerations when looking at persistence in your IoT platform.  
  • It is the first step in providing proper abstraction and loose coupling around the persistence layer.  Eventually, this work which we hope will be completed for the Edinburgh release (April 2019) will allow architects more freedom to customize, extend, and replace this layer based on their persistence needs.
  • EdgeX is all about providing interoperability, flexibility and facilitating choice at the edge – choice in sensor connectivity, analytics, cloud connectivity, deployment, etc. This new feature again showcases EdgeX’s flexibility – flexibility in persistence realm.  Future releases of EdgeX, using patterns established with this database abstractions, are looking at offering even more flexibility and interoperability in areas like messaging, security, communications, system management, etc.

The EdgeX community (which includes members of the Redis Labs team) worked throughout the Delhi release to simultaneously refactored several of the EdgeX microservices to offer Redis as embedded data services.  Specifically, this means we:

  • Incorporated the EdgeX services with the tools needed to connect to databases such as Redis and MongoDB
  • Leveraged Redis’ multi-model capability and data structures to serialize EdgeX data models for persistence, and index them for queries
  • Decoupled the EdgeX models from a single persistence mechanism
  • Solve identity issues, such as identifying sensor readings, in a database-independent way
  • Added Redis to the EdgeX deployment/orchestration facilities
  • Provided Redis initialization and bootstrapping scripts in support of EdgeX

Again, all of this work is important first steps toward more unilateral independence and choice with regard to persistence in EdgeX in future releases.

In addition to the work to provide alternate database connectors in several key EdgeX microservices, the enhanced Delhi code will also include the following:

  • New device service connectors, created from the new SDKs made available for Modbus and MQTT.  These were device services created with the new Go and C Devcie Service SDKs that were made available with the Delhi release.  Device connectors provide the “thing” or sensor/device connectivity in EdgeX.
  • A simple example device service simulator that developers can use to learn the EdgeX device service framework and speed up their development efforts.
  • Additional and improved documentation that includes all the new features from the Delhi release.
  • The EdgeX Foundry snap published in the the Snap Store (https://snapcraft.io/edgexfoundry) for the first time.

It should be mentioned that with the new Device Service SDKs, we are seeing a real escalation in EdgeX “thing” connectivity.  As I write this post, several additional Device Services have been created beyond what is offered in the “dot” release. So stay tuned to the EdgeX community outlets for more in this area coming soon.

Big shout out to the technical community for helping us achieve another technical milestone To learn more about the Redis connection, please click on this blog.

If you have questions or comments, visit the EdgeX Foundry Slack Channel and share your thoughts in the #community channel.

EdgeX Foundry Releases Delhi and Plans for Edinburgh

By | Blog

The technical details on the current and next release of EdgeX Foundry

Jim White, Dell Technologies IoT Platform Development Team Lead & EdgeX TSC Vice Chair

This week, EdgeX Foundry announces the release of Delhi, our third major release in 12 months.  With each release that this growing community kicks out – I get a little nostalgic.  EdgeX started its life in my kitchen.  Like a lot of open source software efforts – it started humbly.  In the case of EdgeX, it started on my laptop, propped open on my kitchen island one summer weekend about 4 years ago with an adult beverage or two.

Today, Edgex is being developed around the globe by a lot of tremendous software engineering talent.  I was proud to start EdgeX, but I am even more proud to be part of an exciting group of international engineers building the most flexible, interoperable, open source IoT platform on the planet. This new release, Delhi, contains a new service and several key features that have our community and potential customers excited about the future of EdgeX Foundry.

Delhi Release

System Management

Most importantly, this release contains the first EdgeX system management capability.  A new service – the System Management Agent (SMA) – serves as the coordinator for control plane information (status, configuration and metrics around EdgeX services) and control actions on EdgeX services (start, stop, and restart).  Cloud or third party systems can call on the API provided by the SMA to trigger the actions or to get the control plane data they need. The SMA can serve as a one-stop shop for managing an instance of EdgeX.  Each EdgeX micro service has a corresponding management API that the SMA calls on to help control that service (example – stop the service) or pull back its latest configuration or metrics.  The SMA and management API provided by each service will be expanded in future releases of EdgeX and will one day offer control plane data and actions via alternate protocols (like LWM2M or SNMP).

Device Service SDKs

The original platform that started on my kitchen island was a Java platform.  Last year, the community embarked on an endeavor to replace the bulky and slow Java services and tools with Go and C services and tools.  Device Services – the EdgeX services that connect “things” to the platform are still in Java but not for long.  With the Delhi release, two new Device Service SDKs have been created.  The Go and C SDKs are allowing the community to create smaller, lighter, faster device/sensor connecting services that will allow EdgeX to operate in very limited resource compute environments – the thin edge in IoT solutions.

Improved Service Resiliency and Decoupling

In the past, the EdgeX microservice dependencies required the services be brought up with wait times between the startup of each service to allow the system time to bring things up in order.  The services now are able to automatically check and detect when a dependent is in place and to become fully functional only after a dependent is up.  This greatly reduces the start of all of EdgeX from minutes to seconds – around 5-10 seconds on average.  The services now detect a dependent has gone down and will keep retrying that service until it comes back up.  This gives EdgeX more resiliency than it had in the past.

EdgeX User Interfaces

EdgeX was built to facilitate machine to machine communications.  As such, it did not have a user interface.  With the Delhi release, user interfaces were created – largely to help visually showcase EdgeX functionality and to help developers – these UI also serve to show how EdgeX can be driven and operated.

Improved Core and Supporting Services

Many of the core and supporting services in EdgeX were improved and more unit testing was added to allow the community to keep an eye on the quality of the overall system and compatibility of future features of EdgeX.  The Scheduling Service was also transitioned from Java to Go.

More Secure

Security services were initiated in our last release (California).  The Delhi release includes the next wave of security features such as access control to grant access to appropriate services, and improved security service bootstrapping.

Better Tested

As mentioned, more unit tests were added to the EdgeX services.  Additionally, automated blackbox tests are in place for each service to make sure the public API of each service performs as documented.

Ecosystem Growth

In addition to all this hard work, our EdgeX Foundry ecosystem continued to expand during this release cycle.  Among the new companies to join EdgeX Foundry are Basking Automation, Data Ahead, Intel, Redis Labs, and ZEDEDA.  This community also launched the availability of a developer kit and community demonstrator (for showcasing EdgeX during conferences and IoT events).  Find out more about these here.  Stay tuned to the EdgeX news and announcement pages as some big organizations are preparing to announce their joining early in the New Year.

Edinburgh

OK – no time to rest on our laurels.  What’s next EdgeX?  I am glad you asked!  Indeed, the EdgeX technical steering committee (TSC) along with several community members met in Edinburgh Scotland to plan and scope our next release – code named Edinburgh – which is scheduled for April 2019 (I assure you the release name and TSC meeting place alignment happened a little serendipitously).

As Keith Steele, Chair of the EdgeX Foundry Technical Steering Committee, reported in his blog post, the results of the meeting were “superb” – as the Scots like to say.

The Edinburgh release has been scoped to include a rather significant amount of new features as well as making improvements on the current code base (“cleaning up technical debt” as we like to say).  Given the increase in community membership and number of contributors to the project, we feel this scope is within reach.  Here are some of the major efforts the membership has planned for the next 5-6 months.

Binary Data Support

The Edinburgh release of EdgeX will support the ingestion, use, and export of binary data (using CBOR format).  To date, EdgeX has supported ingesting, using and exporting integer, float, string, and boolean types of discrete data collected from sensors.  Many of the edge use cases involve video images, audio data, and other data that is binary data serialized.

Automated Performance and Security Testing

EdgeX has come a long way over the past few releases in increasing and improving its quality through testing.  In the Edinburgh release, performance and security testing will be automated.

Cornucopia of new Device Services

The new Go and C Device Service SDKs created with the Delhi release allow the community to create smaller, faster, less-resource consuming replacements for the Java Device Services that now serve to connect “things” to EdgeX.  This includes Modbus, BACNet, BLE, MQTT and SNMP Device Services.  As there has been a bit of a pent up demand to get new Device Services for EdgeX due to the development of the SDKs, we believe this release will contain even more thing connectors than we could have imagined a year ago.

The paint on the new SDKs isn’t even dry and we already have the community showing off some early prototypes of new Device Services today and the numbers are impressive.  In one small example, our old Java Modbus Device Service was 141MB in size and consumed 184MB of memory.  The Go version produced with the new SDK is 16MB in size and uses just 8MB of memory.

Application Services

The current EdgeX Export Services, while functional, create scalability issues. The current EdgeX capability relies on the Export Services to know and understand all possible endpoint distribution mechanisms and transformation capability necessary to support all clients – even if the use case requires only a single, simple client (like sending the raw sensor event data to an MQTT pipe in JSON). As the number and type of EdgeX north side clients (cloud, enterprise and on-prem IoT servers) expands, this service will be too big and complex to support the north side needs.

The Edinburgh release will feature a new type of “exporting” service – the Application Service.  The initial and simple Application Services in the Edinburgh release will be designed and implemented to support smaller, more tailored exportation needs. These services will contain just the functionality (filtering, transformation, enrichment, etc.) needed for a single endpoint. To address multiple endpoints, multiple Application Services will be created (versus having one massive export service as EdgeX has today). In a way, the Application Services will begin to look more like the south side Device Services – that is functionality dedicated to a particular client protocol and data need. Long term over the course of a number of releases, Application Services will replace EdgeX Export Services as the north side distribution facility.

Improved User On-boarding & Support Plan

EdgeX Foundry has been a developer driven effort.  As the platform is and will continue to be used in real world IoT solutions, it becomes imperative that the community improve the quality and volume of documentation, tutorials, examples, videos, etc. to help user communities go from day 0 to production as quickly as possible.  It is also imperative that EdgeX institute a long term support plan/strategy to give the user community assurances about what they can expect from the open source effort they put into their products.  The TSC Chair (Keith Steele) and TSC Vice-Chair (me) are dedicating our resolve to address these two concerns as part of the Edinburgh release cycle.

Certification Program

A certification program will be outlined in concert with the Edinburgh release. A certification program will enable third parties creating EdgeX services to verify their services as alternative or enhancing capability to those provided by the EdgeX open source effort.  This will allow 3rd parties to add value (proprietary or open source) to EdgeX that customers can rely on to meet the EdgeX APIs and work without additional code change (enabling a plug-and-play ecosystem).  Various levels of certification are being considered, from micro service replacement certification (validating alternate or commercial implementations of EdgeX micro services satisfy API requirements along with performance metrics and quality checks) to full EdgeX deployments (for commercial versions of EdgeX).  Additional certification processes may be developed around particular cross cutting features such as security.

And More…

  • Use of Vault namespaces for storing secrets for the micro services.
  • Initial EdgeX secrets (needed to start Vault/Kong) will be encrypted on the file system using a secure storage abstraction layer – allowing other implementations to store these in hardware stores (based on hardware root of trust systems)
  • Refactor EdgeX database-using services (Core Data, Metadata, Export Client, Logging, Notifications, and Scheduling) to be more loosely coupled to the persistence mechanism (currently MongoDB). This will better facilitate the use of alternative persistent stores and technology in future implementations and even allow the project to select alternate or additional reference implementation databases in future releases.
  • A smaller, lighter, faster rules engine service to replace the last of the EdgeX Java micro services.
  • System Management will offer service health/status checks and additional metrics
  • Updating the versioning and dependency management system for all Go micro services (replacing a deprecated technology)
  • Upgrading all 3rd party open source tools included with EdgeX (Consul, Kong, Vault, etc.) to use the latest releases

For a full list of the Edinburgh release scope and roadmap, see the project Wiki.

It’s a large scope, but the community and team of developers is growing.  Contact our Developer Advocate, Michael Hall, if you and your team would like to be a part of our effort – it’s an exciting time to be part of an exciting and industry impacting project.  I have every confidence the EdgeX Edinburgh release will make some news in April 2019… as long as I can keep the team away from all the Scottish whisky we brought home from our trip!

Welcome California!

By | Blog

Written by Jim White, Vice Chair of the Technical Steering Committee and Distinguished Engineer and Project Lead of the IoT Platform Development Team within Dell Technologies IoT Solutions Division

The second major release of EdgeX Foundry is now available!

While EdgeX is only a year old, our community is demonstrating its staying power with the second major release in its first year.  The California release, which follows Barcelona, shows the commitment and dedication of many who see the importance and potential of developing a flexible, open source, IoT software platform for the edge that provides connectivity and interoperability while still allowing value add.

So, what is new with the California release?  A lot! But before we get into the details, I want to highlight that the biggest focus of this release was to introduce a few key security capabilities and to make EdgeX smaller and faster.

Security

EdgeX began its existence without security and organizations wanting to leverage the platform had to add their own security capability. Today, EdgeX incorporates some of the first security elements.  These initial elements, while useful on their own, are essential building blocks to additional security features in the future.

The first security elements include a reverse proxy that helps protect the REST API communications and a secrets store.  With the EdgeX reverse proxy in place – as provided by incorporating an open source product called Kong – any external client of an EdgeX micro service must first authenticate themselves before successfully calling on an EdgeX API.

The secure storage facility was provided by incorporating the open source Vault (Hashicorp) product, and it allows items such as username/password credentials, certificates, secure tokens, etc. to be persisted and protected within EdgeX.  These types of “secrets” will allow EdgeX to, for example, encrypt data, make HTTPS calls to the enterprise, or connect EdgeX to a cloud provider in a secure manner.

Performance and Scalability

The EdgeX Foundry Technical Steering Committee decided early last year in the project’s formation that we would release twice a year – once in April and once in October.  You probably noticed that it’s not April.

Last year, we decided that EdgeX needed to be smaller and faster to better function effectively at “the edge”, which the largely-Java code from the seed donation was going to make difficult. To do this, we needed to rebuild the EdgeX microservices in Go Lang – and do so by our spring 2018 release.  This was not a small endeavor and it was made at a time when the EdgeX Foundry developer community was just coming on board.  We knew it would take a bit more time, but we were committed to this, and added two more months to this release cycle.

The extra time was well worth it!  With the California release, we’ve dramatically lowered the footprint, startup time, memory and CPU usage. Take a look at the statistics below, which compares services from our first community release last October (Barcelona) to our current release (California).

We still have work to do, but it’s now possible to run all of EdgeX on something like a Raspberry Pi 3.

Additional Features

In addition to the initial security capabilities and reducing the size and latency of the platform, this release includes other work – some visible to the user while some features are more hidden but improve the overall quality of EdgeX.

  • Several additions were made to the export services to provide additional “northbound” connectivity, to include connectors for XMPP, ThingsBoard IoT, and Brightics IoT
  • We improved the documentation and now have documentation stored with the code in Github – allowing it to be maintained and updated more like code by the community
  • Arm 64 is now fully supported.  In fact we worked with the Linux Foundation to add external environments and tools to create native Arm 64 artifacts.
  • We added blackbox tests for all the micro services.  These are now kicked off as part of our build and continuous integration processes.
  • Other improvements were made to our continuous integration – to help streamline developer contributions

We invite you to try out the California release today (Docker Compose file here)!  

On to Delhi

Our next release, named Delhi, will come out in October 2018.  Due to the extended release cycle for California, the Delhi release cycle is going to be short. The significant features planned for Delhi include:

  • Initial manageability services and capability
  • Device Service SDKs (Go/C) and at least one example device service
  • The next wave of security features to include access control lists to grant access to appropriate services and improved security service bootstrapping
  • Better/more unit testing and added performance testing
  • Adding the last of the refactored and improved Go Lang microservices
  • Outlining options and a potential implementation plan for alternate or additional database support
  • An EdgeX UI suitable for demos and smaller installations

Come join us!

We would like to thank the talented men and women who are working very hard to turn the vision announced when the EdgeX project launched in April 2017 into the product we see emerge and improve with each release.  In the past six months, we have seen the number of unique authors contributing to the project code base double to more than 50. We hope you’ll consider joining our growing development community to build on this momentum by contributing to the Delhi release as well as using EdgeX in your edge/IoT solutions!

If you have questions or comments, visit the EdgeX Rocket.Chat and share your thoughts in the #community channel.