Tuesday, November 02, 2010

ECF 3.4 Remote Services

ECF 3.4 was recently released. This release (along with Helios/3.3 and upcoming releases) heavily emphasized the implementation of OSGi 4.2's Remote Services specification. Our community is pushing us to continue this emphasis, and so we will.

Here are some reasons to use ECF's OSGi 4.2 Remote Services implementation:

  1. Standards Compliant: It is fully compliant with the Remote Services standard. No lock-in...now and forever

  2. Asynchronous Remote Services: Unlike other implementations of this standard, right now it provides support for Asynchronous Remote Services

  3. Multi-Transport: Right now it supports multiple network discovery protocols (e.g. Zookeeper, DNS-SD, SLP, Zeroconf, static xml-file), and multiple distribution transports (e.g. r-OSGi, ECF generic, XMPP, JMS, Http/REST-based protocols, JavaGroups)

  4. Extensibility through Modularity: The open discovery and remote services APIs allow new discovery and distribution implementations to be substituted at will...proprietary or open...without requiring any additional work to support the OSGi standard

  5. Enterprise support: We are completing (for ECF 3.5) our implementation of the Remote Services Admin specification. The progress on this can be easily and publicly tracked...contributions, test/testing, and early uses are welcomed and encouraged.

  6. Open Community: ECF is not just open source, but also has a completely open, diverse, growing, active...and most importantly...a contributing community

  7. Open Process: We've moved to GIT, to make community support and contributions easier

  8. Multi-Framework: ECF remote services now runs on Felix (and probably other OSGi frameworks as well)

  9. Robustness through Community Usage

  10. Low-license fee: $0 :)

Monday, November 01, 2010

Innovation and Openness

There is a Sunday NY Times article about what promises to be an interesting book:

Innovation: It Isn't a Matter of Left or Right

Johnson apparently makes the claim that 'collaborative, non-proprietary, open networks' are of high importance for technology innovation. This strikes me as true, and explains my intuition that open source projects like Eclipse and ECF are well-positioned to create value through innovation.

Thursday, September 02, 2010

Asynchronous Remote Services - choices, choices

In ECF's Helios release, we released an implementation of the OSGi remote services standard specification (chapter 13 in compendium).

In addition to the full spec implementation...which is based upon synchronous remote service proxies...we added support for asynchronous remote services. This provides non-blocking access to remote OSGi services. This gives remote service consumers choices...allowing them to invoke remote services synchronously (i.e. by making a blocking method call on the proxy), and/or asynchronously (with a guarantee that the calling thread will not block).

I think that one nice thing about this approach is that the service host implementer has to do exactly nothing to make these consumer choices available. The implementation of the service host is exactly the same.

There are two styles of asynchronous access supported: an asynchronous callback (like GWT), and a future result, from the Actor model of computation. These two styles of of asynchronous access...along with the specified synchronous proxy...provides remote services consumers with some useful choices for creating reliable distributed systems and applications.

Sunday, May 02, 2010

Making Sense of Complexity

The NY Times Sunday Opinion section has an article today: Making Sense of Complexity.

The ideas are (mostly) presented in reference to complexity in social systems...but as someone interested in (reducing) complexity in software systems, as well as the psychology of complex system design and development...I also found the thoughts interesting from a software architecture and design point of view.

Tuesday, April 27, 2010

Asynchronous Remote Services - The future or the callback

In previous postings I described how ECF is now making it very easy for OSGi service developers to expose asynchronous/non-blocking remote method calls to clients.

In short, all that's now required is to create an asynchronous version of the service's OSGi service interface. See this documentation for example and source. Just declaring this asynchronous interface is all that's needed. At proxy discovery time, ECF's implementation of OSGi remote services will provide the implementation of this asynchronous interface.

Future or Callback

There are various approaches to doing asynchronous remote method invocation, and two common ones are callbacks and futures. For example, GWT uses callbacks, while Amazon EC2 uses futures for exposing asynchronous access to their APIs (like SNS, SQS, etc). ECF's asynchronous remote services supports both of these approaches (futures and callbacks). The asynchronous service interface declaration can, for a given synchronous method declaration, use either a callback, or a future, or both.

For example, let's say we have the following synchronous service interface method:

String foo(String bar);

The async declaration for this method using a callback would look like this:

void fooAsync(String bar, IAsyncCallback);

The async declaration for thie method using a future would look like this:

IFuture fooAsync(String bar);

And that's it. The remote service client can then use either/both of these fooAsync methods (if they are declared, of course), simply by casting the proxy to the async service interface type and calling the appropriate fooAsync method with the necessary params.

In this way, the remote service designer can determine what asynchronous style the client will have available...by declaring fooAsync using callback, future, both, or neither.

Friday, April 16, 2010

Asynchronous Remote Services - part 2

In a previous posting, I described how ECF has introduced a simplified approach for allowing OSGi remote services to be accessed asynchronously.

In contrast to my recent postings, that have been getting rather long, I'll just redirect you to a wiki page describing how to use asynchronous services...and leave it at that. Happy Friday.

Wednesday, April 14, 2010

OSGi Remote Services and ECF - Asynchronous services

In a previous posting, I discussed/presented some of the support for asynchronous access to OSGi remote services that currently exists in ECF's implementation.

In a blog posting earlier this week, Peter Kriens discussed some of the efforts going on in the EEG on adding asynchronous support for remote (and even local) services. One of his comments in that blog posting was that ECF's asynchronous support could be considered awkward, because of the complexity/unfamiliarity of using the API.

I've been intending to add easier/more natural mechanisms for asynchronous remote access than what we already have, and what's going on in the EEG and Peter's blog was great incentive to complete some more of that work. The existing mechanisms are somewhat awkward, but they also make a very strong/flexible foundation...and so it's possible to build new mechanisms on the existing mechanisms.

Normal/Synchronous Proxies

In our 'hello' remote services example, we have this service interface:

public interface IHello {
public void hello(String from);
}

Consumers of this remote service receive a proxy that implements the IHello interface, and then clients can synchronously invoke the hello method to make a remote call:

proxy.hello{"slewis");

Since in java method calls are blocking, the thread that calls the hello method will block if (e.g.) the network is slow, the service host is slow (or blocks). It would be nice if we had a way (on the consumer/client) to call the hello method and guarantee that it will not block...while still somehow getting the result (if any)...when the remote call is successful...or getting information about the failure if things fail/go wrong (e.g. because of network failure).

Asynchronous Proxies

We've just added support for asynchronous proxies in ECF're remote services implementation. What this means is that if an interface is declared like this (and in the same package as the IHello interface):

public interface IHelloAsync extends IAsyncRemoteServiceProxy {

public void helloAsync(String from, IAsyncCallback callback);

public IFuture helloAsync(String from);

}

the ECF remote service distribution system will automatically create a proxy that implements the IHelloAsync interface on the consumer/client.

If the helloAsync(String,IAsyncCallback) method is called by the consumer:

proxy.helloAsync("slewis",new IAsyncCallback() {
void onSuccess(Object result) {
System.out.println("we got result="+result);
}
void onFailure(Throwable exception) {
System.out.println("oh no!");
exception.printStackTrace();
}
});

the consumer thread will not block, and success/result or failure will be asynchronously communicated to the caller via proxy calling the appropriate method on IAsyncCallback.

In addition to using the callback, futures (IFuture) are also supported. All that must be done to allow the consumer to use a future result is to declare a helloAsync method that returns an IFuture:

public IFuture helloAsync(String from);

The only thing required to get this to happen on the consumer/client is to declare the *Async interface (IHelloAsync). Then, at proxy creation time on the remote service consumer, if this *Async interface exists, it will be implemented by the proxy, and usable by the client.

Note that the *Async interface declaration is the only thing that's needed to get this to work with any service interface. The service host implementation doesn't need to actually implement the *Async interface, and the ECF remote services distribution will create a proxy that implements the *Async interface automatically. Further, like other things ECF, this is all done in a transport-independent way, so all the existing providers (JMS, XMPP, ECF generic, JavaGroups, Skype, REST, SOAP, etc., etc.) support this addition immediately with no further work.

Google Web Toolkit uses a very similar approach to support asynchronous remote procedure call. In addition to callbacks, however, ECF's asynchronous proxy also has support for futures. This allows the consumer/client to choose the desired invocation style: synchronous, asynchronous-callback, or asynchronous-futures.

Saturday, April 10, 2010

OSGi Remote Services from ECF - Distribution

In a previous posting, I discussed the use of the ECF discovery API as part of our implementation of the OSGi 4.2 remote services specification.

The second major part of ECF's implementation of OSGi 4.2 remote services is distribution.

What is Distribution?

Distribution is what happens to actually invoke a remote service and optionally return some result. Here's a brief summary of the essential functions of distribution:

Remote Service Consumer

[Prior to caller actually using service]
1. Create a proxy for the remote service
[When caller actually uses remote service]
2. Marshal/Serial any arguments for the remote call
3. Put call request (method and serialized parameter) on the wire using some protocol

Remote Service Host

1. Take request off the wire (using same protocol)
2. Un-marshal method and arguments
3. Lookup corresponding service/method
4. Invoke appropriate service with given arguments
5. Marshal return value
6. Put result on the wire using some protocol

Remote Service Consumer

4. Take response off the wire (using same protocol)
5. Un-marshal result
6. Return result to caller

One way to think of it is that distribution is responsible for making what looks like a local method call to a local OSGi service actually be a remote call.

Two of the critical functions of distribution...for both Consumer and Host are

1) Marshaling/Serialization...of arguments and return values
2) Use some protocol to communicate request/response over network

As with discovery, the ECF project has created an abstract API for distribution, which is called the ECF remote services API. Like other ECF APIs, this is a transport-independent API, which exposes a programmatic way to accomplish the functions of distribution (as described above), but does not imply/require any particular implementation of marshaling/serialization, nor imply/require any particular network protocol.

ECF has providers that define specific implementations of marshaling and network protocol. For example, we have a REST-API, that supports the creation of specific REST providers. This REST API includes JSON and/or xml-based serialization, and uses HTTP as the protocol. We also have a similar SOAP API for SOAP-based services.

We also have a number of other providers that are complete and available...e.g. ones based upon XMPP, JMS, ECF generic, Skype's app protocol, JavaGroups/multicast. Further, since all of these providers are open source, if desired they can be extended or copied to implement custom providers based upon whatever serialization and wire protocol (e.g. an existing system) is desired...with our without the ECF team's involvement.

Note the ECF implementation of the OSGi 4.2 remote services specification is guaranteed to work with any of these providers...no matter who writes it. This because our implementation of the OSGi 4.2 remote services spec simply uses any all implementations of the ECF remote service API (no matter what the serialization and/or networking protocol).

The flexibility here is extremely useful when selecting serialization formats and/or network protocols, because there are/will always be so many serialization formats and/or network protocols to choose from...their appropriateness will always depend upon the use case...as well as the need for integration with existing systems. For example...e.g. json over http, custom xml over http, object serialization over tcp, xml over jms, soap over http, etc, etc...which makes sense depends upon the use case and things like networking/interoperability requirements.

Since this distribution function is separated out into a distinct, abstract, module (i.e. the ECF remote services API), it makes it possible to mix and match existing protocols and new protocols...both closed and open...with existing serialization formats or new serialization formats...crossed with whatever discovery protocol is appropriate and/or desired.

Tuesday, April 06, 2010

OSGi Remote Services from ECF - Discovery

Released in Feb, ECF 3.2 has full support for the OSGi 4.2 remote services specification.

As with any general technology, there are potentially many use cases for remoting OSGi services, and any given implementation won't support all those use cases. It's therefore very important that any technology be extensible to support use cases that were not envisioned originally.

OSGi Remote Services: A tale of discovery and distribution

In providing access to a remote service there are at least two network-created issues that must be addressed for any remoting technology to work. In this post I'll discuss discovery, and in subsequent posts talk about distribution.

Modularity for Network Discovery

When a new service is made available...via a server, or a peer, or a device, etc...any consumers/clients of that service must somehow be made aware of that service, and given sufficient information to be able to access that service. A very common example of being made 'aware' of a web service is receiving (via email, or a web page, or twitter, or whatever) the URL for that service...e.g. the twitter user status service URL is http://twitter.com/statuses/user_timeline.json.

With OSGi remote services the notion of a URL is generalized to an endpoint. As with all OSGi services, service properties provide metadata about the remote service (including but not limited to the endpoint)...and this metadata is sufficient for a consumer to actually access/use the service.

There are potentially many ways to discover a remote service. There are network discovery protocols (e.g. zeroconf/bonjour, Service Location Protocol (SLP), Apache Zookeeper), as well as static xml or other formatted files, custom http-based service registries, etc., etc.

To deal with the required flexibility, ECF has an abstract discovery API (org.eclipse.ecf.discovery). This is a network-protocol-independent API for discovering things over the network. I use 'things' because the discovery API isn't only for discovering remote OSGi services, and it can also be used to discover devices, other applications (an example of this is that since Apple's iTunes uses zeroconf to publish itself, it's possible to interoperate with iTunes and/or other iMac and iPhone applications from within an OSGi runtime).

The providers/protocols that we ship with ECF now are zeroconf/bonjour, SLP, and we have a pending contribution for Apache Zookeeper. We also currently have support for static xml-file-based discovery of remote services and are working on support for use of DNS-SD for wide-area dns-based discovery.

The ECF discovery API effectively separates network discovery into a distinct module, and allows the reuse of existing network protocol implementations, OR substitution of one's own approach to discovery to meet custom use cases (such as discovering remote services only behind a firewall, etc).

This modularization enables reuse, since all the other parts of ECF's remote services implementation (e.g. the distribution...i.e. remote method marshalling/unmarshalling, etc) can be reused without modification. This is so because ECF's OSGi remote services implementation simply uses any/all discovery API providers at runtime to publish the remote service. This makes any new discovery API provider automatically and immediately compliant with the OSGi remote services specification.

The reuse and extensibility is a positive side effect of the modularity provided inherently by OSGi, along with the separation of concerns built into ECF's implementation of OSGi remote services. In a future posting(s) I'll discuss the distribution module of ECF's remote services implementation...referred to as the ECF remote services API.

Reference:

OSGi 4.2 Remote Services
ecf-dev mailing list

Tuesday, March 23, 2010

OSGi Enterprise and ECF Remote Services

Earlier today the OSGi 4.2 Enterprise spec was announced by David Bosschaert.

As part this announcement David listed some implementations of the OSGi 4.2 Remote Services specification, but for some strange reason he neglected to include the EclipseRT implementation from from ECF project.

So, just to be clear, EclipseRT/ECF 3.2 also already has support for OSGi 4.2 Remote Services. See here for details, links to docs, examples, and public support forums...as well as descriptions of other features in this implementation.

OSGi/EclipseRT in Amazon Cloud - p2

In a previous post, I announced the availability of a public Amazon Image (AMI) for the Amazon EC2 service that includes several EclipseRT 3.6 technologies...including Jetty 7.0.1 and Equinox 3.6M5.

p2 is also included in the image, and this allows install/update in a running OSGi web application server. Also included in the AMI is Jetty + Equinox 3.6M5 that does not include p2.

Tuesday, March 16, 2010

OSGi/EclipseRT in Amazon Cloud

I've created a public Amazon Image (AMI) from a recent build (3.6 stream) of the EclipseRT. The parts of EclipseRT included in this image were Jetty, Equinox, p2 provisioning, and a very simple Hello World servlet application (with source).

Here is documentation about how to get the image, start an EC2 instance, and run your own server.

Here is an instance of this Hello World servlet application running on my instance.

Friday, February 19, 2010

ECF 3.2 Now Available

ECF 3.2 is now available here. The emphasis/theme for this release is support for Service-Oriented Architecture (SOA), specifically through support of the OSGi 4.2 Remote Services standard.

Highlights
Congratulations and thanks are due to the ECF community

Tuesday, February 16, 2010

OSGi Remote Services and Sync vs. Async

ECF 3.2 contains an implementation of the new OSGi 4.2 remote services standard. This release is coming out later this week (Feb 19).

One thing that developers may discover when building and testing distributed applications is that synchronous remote procedure call (RPC) can have surprising behaviors. In my view, this is because we reflexively understand that normal/local/in memory method call is synchronous and fast...i.e. that the calling thread blocks until the method is complete (and optionally a result is returned), OR the method fails/throws an exception in languages that have structured exception handling.

So at best the remote call's I/O behavior will lead to large performance variability (i.e. the remote call will be orders of magnitude slower...and variable based upon network performance), and at worst the caller thread could hang/block indefinitely. This violates our expectations about method invocation.

To address this problem, frequently asynchronous remote method call and/or non-blocking messaging is used...so that the caller can be guaranteed that the calling thread will not block. Note that depending upon the application requirements and expectations, it may be fine that synchronous/blocking RPC is used. OTOH, it may be very important that remote services not block...for user experience, and or overall system performance expectations. It depends upon the use case...and I don't believe there is any one, 'right' answer for all situations.

ECF's implementation of the OSGi 4.2 remote services spec has support for asynchronous remote method call. This support is exposed via our IRemoteService contract, which is made available to remote service consumers (for example code, see tutorial here). This contract exposes two mechanisms for making asynchronous remote method calls

The IRemoteService reference associated with a remote service proxy is accessible via any/all ECF remote services. If not needed, however, it's invisible and so doesn't impose any complexity burden.

Currently, the OSGi 4.2 remote services spec does not articulate any methods for asynchronously accessing a remote service, but my understanding is that this is an area for future standardization.

Thursday, February 04, 2010

Goodness through OSGi Standards

ECF recently announced full support for OSGi 4.2's remote services standard with our upcoming 3.2 release.

Today, I learned that a community member has successfully used Spring dm, along with ECF's remote services implementation to do declaratively-specified remote services. They have agreed to contribute the example to ECF, and so expect to see it as part of ECF soon.

People have also used ECF remote services with OSGi declarative services.

And, of course, one can use remote services programmatically as well.

Among other things, this allows a wide variety of existing tooling to be used to construct, use, and debug remote services...all made possible by having an open standard for distributing an OSGi service.

Tuesday, January 12, 2010

Motivation 3.0

A question that I ask myself periodically is this: Why work on open source projects? In my case, why work on ECF? What is my motivation to do so?

There's an interesting book by Daniel Pink that examines human motivation called Drive: The Surprising Truth About What Motivates Us. There's also a nice YouTube video by the author from last year's TED conference.

In reading the first part of the book, I've resonated with his assertion that intrinsic motivation is sometimes stronger than extrinsic motivation...particularly when creativity (aka innovation) is involved. And I suspect I am not alone in this...at least among people who are passionate about innovation, community, software technology, and open systems.

I believe understanding motivations is important...because lots of assumptions about how innovation comes about...i.e. who does it and why...are actually dependent upon underlying assumptions about motivation.

Sunday, January 10, 2010

SOAP, REST, and ECF remote services

In addition to supporting the OSGi 4.2 remote services specification, we on the ECF team have also been working on support for accessing REST-style services, as well as those that use the Simple Object Access Protocol (SOAP).

The ability to support all these styles of service-oriented architecture is fundamentally enabled by ECF's provider architecture, but since mentioned that in a recent posting, in this post I'm going to touch on something useful that's enabled by this provider architecture.

Most remote services (whether web services, OSGi remote services, REST-based services, etc) have two basic roles

  1. The service host...aka the 'server'

  2. The service consumer...aka the 'client'


Typically, the service host is first 'registered'...to make it available for remote access, and after that the service consumer then accesses/uses the remote service (e.g. makes remote method calls on a proxy, issues http requests for resources, sends/receives messages, or has some other way of actually accessing/calling the remote service).

Note that the service host need not be exposed by a 'server' (although it frequently is)...in some systems clients can register/expose services as well as servers. And it's probably obvious that service consumers don't have to be 'clients' either...i.e. they can be servers that are communicating with other servers. This is why I use the role names 'host' and 'consumer' rather than 'server' and 'client' when referring to remote services.

OSGi remote services use the OSGi service registry...for both the host's registration of services, and the consumer's lookup and access to a remote service. Whatever the transport used to implement the distribution, typically both the host's registration and the consumer's lookup are done via the OSGi service registry. This typical use case implies, however, that both the host framework and the consumer framework have access to the OSGi service registry (since both registration and lookup are via the service registry).

But what if you would like to consume a service that doesn't use OSGi (and therefore doesn't have a service registry)? With ECF's remote services API, along with our recently added REST and SOAP support, we've enabled this use case (non-OSGi service host) while still providing the benefits of using OSGi remote services for the consumer.

How, you say? First of all, ECF's transport independent architecture allows clients to talk whatever protocol they require to communicate with a remote process...so, for example, the XMPP provider is able to communicate with any remote system that uses the (standardized) XMPP protocol. This does *not* have to be an OSGi-based system.

Second, ECF now has a very small, remote service client API...specifically to allow consumers to interact with non-OSGi services, while still using the OSGi service registry (if they wish) on the client. Since both the ECF remote services API, as well as this new remote service client API are also transport independent, and have both explicit support for REST-style and SOAP-based transports, providers for specific REST-based protocols and/or SOAP-based protocols are easy to create.

As an example, I recently created a SOAP/Web services client (for an existing web service), using Apache Axis (to convert wsdl to java), the WTP tooling (for automating the generation of the java code from wsdl), and this new remote services client API. To the non-OSGi service host, this client looks/behaves like all other clients. The same idea...for an existing REST-based API (Twitter in this case), is shown by this demo.

Unlike other clients, however, these clients can use OSGi to maximum value: i.e. to structure the client in a modular way, to handle the dynamic requirements of a remote/networked/unreliable service, or even (re) expose the proxy as a another remote service...that other consumers can access. This can be used for building load balancing of web services, or to aggregate sets of services...as well as other purposes.

Note there is no additional tooling required to build such a client...since your favorite tools for creating the SOAP-based and/or REST-based clients may be used, alongside APIs and tooling for interacting with the OSGi service registry...e.g. Eclipse PDE, OSGi declarative services and/or others.

Sunday, January 03, 2010

OSGi Remote Services from ECF

The ECF project has just finished our initial implementation of the OSGi 4.2 Remote Service specification (chapter 13 in compendium section).

I want to highlight a few distinctive features of ECF's implementation. I'll be doing other/more blog posts to go into details about some of these.

Transport Independence

ECF's implementation currently works with the following distribution transports: JMS (ActiveMQ), r-OSGi, JavaGroups, XMPP, Skype, ECF generic, and with the following discovery protocols: SLP, zeroconf, static xml file-based discovery. The ECF implementation is immediately usable with any...or all...of these providers...even within a single application if desired. Further, ECF's remote services and discovery APIs allow other distribution and/or discovery systems to be plugged in underneath our implementation...meaning that other distribution systems (e.g. open source distribution systems such as Riena, Apache CXF, and/or proprietary distribution systems) can reuse/leverage our implementation...and avoid the work otherwise necessary to implement the OSGi specification themselves. ECF's implementation can also run alongside other remote services implementations without conflict.

Asynchronous and Synchronous

Distributed applications and services frequently need to use asynchronous/non-blocking remote invocation patterns...in addition to synchronous, proxy-based remote method call. Right now, ECF has built-in support for both asynchronous invocation (e.g. asynchronous listeners, futures, one-ways), as well as proxy/synchronous invocation...giving the services and application programmer a transport-independent choice of which to use. Currently, the OSGi 4.2 spec does not yet specify asynchronous invocation patterns for remote services, but ECF's implementation does have it.

Lightweight

The entire ECF implementation of the spec...along with ECF's remote services API...is < 150k of code. Further, the OSGi execution environment requirements are minimal (CDC 1.1/Foundation 1.1). Combined with a small provider (like r-OSGi), and this allows even small devices to both expose/host and consume standardized remote services.

Standard, Open Source, Open Team, Open Process

The ECF implementation is fully compliant with the OSGi 4.2 remote services specification...for any and all current and future ECF remote services providers. Further, as mentioned above, other providers can now get this compliance for free. Of course, ECF's implementation is open source, but it is also produced by a vendor-neutral project team, and fully open community-driven process. In addition, we have support for a distributed version of the OSGi EventAdmin service, for doing publish-and-subscribe-based applications using a standard event bus API/service.

Extensible

ECF's implementation is deeply extensible, allowing control or customization of every aspect of the distribution of an OSGi service (e.g. custom discovery, custom marshalling/serialization, custom transport/wire protocol, etc) for those that need it. This is enabled by the open APIs that ECF exposes, and allows a wide variety of deployment requirements and application-level use cases to be easily supported.

Summary

It's All About Modules. ECF's modular structure is indeed enabled by OSGi's modularity, but we have also applied modular design to every level of APIs and implementations. For example, lightweightness, extensibility, and transport independence all are enabled by our modular designs...to separate API from implementation, and to separate distinct subsystems (e.g. discovery and remote services APIs).

We are preparing a release that includes this code...so to immediately download/use ECF's implementation, or engage the community...for requesting enhancements, reporting bugs, contributing (e.g. examples, providers, customizations), helping with testing, or getting support please consider joining the ecf dev mailing list or see the ecf dev resources page.