Faceţi căutări pe acest blog

Se încarcă...

vineri, 17 octombrie 2014

Modular REST applications with Karaf features for OSGi Jax-RS Connector

The purpose of this article is to let you know how easy  it is to develop modular REST (JAX-RS) applications on OSGi, particularly Apache Karaf.

For some time I'm working on improving the way I deliver applications. My focus is on quality, ease of understanding and speed of delivery. My framework of choice for some time is the OSGi platform (mostly used on top of Karaf, but I'm getting bolder - going for bare-bone, native containers). 

Regarding web applications I admit I don't like sessions and I am strongly inclined to develop stateless applications. Since I like standards and the benefits they provide, my choice for a web framework has narrowed down to JAX-RS for which there are a few implementations. 

I came across a project called osgi-jax-rs-connector who's aim is to simplify web application  development using JAX-RS on OSGi. The way it works is you write your JAX-RS annotated resources and you publish them in the registry. Once there, the JAX-RS Publisher from osgi-jax-rs-connector will find them, take notice of the annotation and publish them.  That's it.

In the project README on github, you will find links to articles detailing the whole process.

All i did was to  add a features file for Apache Karaf so you can try it out easily. I've made a pull request with my code to make it part of the original code base and hopefully it will soon. 

I'll reproduce the steps below. You start by building the project and installing the features in Apache Karaf:

    feature:repo-add mvn:com.eclipsesource.jaxrs/features/0.0.1-SNAPSHOT/xml/features
    feature:install scr http
    feature:install jax-rs-connector jax-rs-provider-moxy
    install mvn:com.eclipsesource.jaxrs/jax-rs-sample/0.0.1-SNAPSHOT
After this just go to: http://localhost:8181/sercices/greeting 

You can check the whole project on my github account in the mean time: step by step .

There are other solution out there for publishing JAX-RS resources using OSGi HttpService. Another interesting approach is Neil Bartlett's JAX-RS OSGi extender . The main advantage (in my opinion) of using the approach taken by Connector is the fact that you publish objects instead of the extender building them for you. This means that I am free to choose the way I build my object and I also have the opportunity to inject dependencies in it before I publish it  - hello CDI. I can build my objects using CDI via pax-cdi or with declarative services (as you can see in my sample code) and I am free to inject stuff in it before I expose it for registration with HttpService. That is a pretty powerful thing. I hope to show you how this is done soon. 

   
 

marți, 19 iunie 2012

java mbox parsing with Apache James Mime4j

Today I just made a small push to Apache James Mim4j trunk. The change-set adds mbox parsing capabilities to mime4j. It's a one class Iterator that you can use to split an mbox file into individual messages and after this parse them with mime4j.

I also added an simple example here.

This is how you can use it:

for (CharBufferWrapper message : MboxIterator.fromFile(mbox).charset(ENCODER.charset()).build()) {
System.out.println(messageSummary(message.asInputStream(ENCODER.charset())));
count++;
}


joi, 17 mai 2012

JETM performance monitoring for Apache James

The story

Today I decided to improve how people can monitor Apache James email server. James already has some monitoring available via: Each of the above monitoring options is useful and necessary as it's used to monitor different aspects of how James runs. With regard to performance, only JMX monitoring can provide metrics that allow a user to get a feel of the server state.
With JMX enabled you can just launch jconsole to access the attributes and methods that James exposes: number of active connections active per component, limits, etc. You have complete monitoring access to the following services:
  • IMAP
  • POP3
  • SMTP
  • LMTP
  • RemoteManager
  • DNSService
  • Queues

The nice thing about JMX is that it's integrated with the JVM (no external libraries also) and can provide very precise information on almost every aspect of your application like memory, execution of methods, number of method calls, etc. You can see what James exposes on the Java Management Extension page. JMX is good so why another solution to monitor James? Well, in order for JMX monitoring to work you have to write some code and once you get it in there it's not easy to turn off. Since James is component based, disabling JMX is very easy: you just have to comment out some spring beans declarations. So the second part is not an issue, but the first one is.
Monitoring all James components means you have to write code for each component. This will make the code-base larger and harder to maintain. Here's where Java™ Execution Time Measurement Library or JETM shines. It's a general library that you can use in your application to do monitoring. It's small, compact, adds little overhead and provides all the basic stuff that you need, including an web interface to view your results.
The nicest part about JETM is that it has spring integration. Because James uses spring extensively (the IOC part) integrating the two was a matter of a few hours (took my time reading).

How it works

The JETM library has integration with spring framework and uses spring AOP. It creates a Proxy to intercept calls to the beans that you wish to monitor.

The result

and detailed:

The code

You won't find much here, just some configurations. You can see the full diff here.
In a few words: add the library to your project pom.xml file:
        <dependencies>
<dependency>
<groupId>fm.void.jetm</groupId>
<artifactId>jetm</artifactId>
<version>1.2.3</version>
</dependency>
<dependency>
<groupId>fm.void.jetm</groupId>
<artifactId>jetm-optional</artifactId>
<version>1.2.3</version>
</dependency>
</dependencies>
Next declare the monitoring beans in your project (jetm-monitoring.conf in our case):
    <bean id="etmMonitor" class="etm.core.monitor.NestedMonitor"
init-method="start" destroy-method="stop" />
<bean id="etmHttpConsole" class="etm.contrib.console.HttpConsoleServer"
init-method="start" destroy-method="stop" autowire="constructor"/>

<bean id="etmMethodCallInterceptor"
class="etm.contrib.aop.aopalliance.EtmMethodCallInterceptor"
autowire="constructor"/>

<bean id="etmAutoProxy"
class="org.springframework.aop.framework.autoproxy.BeanNameAutoProxyCreator">
<property name="interceptorNames">
<list>
<value>etmMethodCallInterceptor</value>
</list>
</property>
Here EtmMonitor is responsible for collecting and aggregating measurements points. The measurements points are method calls intercepted by EtmMethodCallInterceptor. The final step is to add the list of beans from your project that you wish to monitor (also in jetm-monitoring.conf). Here I added most of the beans exported by James.

    <bean id="etmAutoProxy"
class="org.springframework.aop.framework.autoproxy.BeanNameAutoProxyCreator">
<property name="interceptorNames">
<list>
<value>etmMethodCallInterceptor</value>
</list>
</property>
<!-- add the beans that you wish to monitor to the list bellow -->
<property name="beanNames">
<list>
<value>usersrepository</value>
<value>recipientrewritetable</value>
<value>domainlist</value>
<value>mailrepositorystore</value>
<!--
The folowing beans require CGLIB2 to be on the classpath -->
<!--
<value>mailqueuefactory</value>
<value>blobTransferPolicy</value>
<value>jmsConnectionFactory</value>
-->
<value>jmsTransactionManager</value>
<value>mailprocessor</value>
<value>mailetcontext</value>
<value>mailspooler</value>
<value>mailetloader</value>
<value>matcherloader</value>
<value>filesystem</value>
<value>dnsservice</value>
<value>fetchmail</value>
<value>smtpserver</value>
<value>pop3server</value>
<value>lmtpserver</value>
<value>imapserver</value>
<value>imapDecoder</value>
<value>imapEncoder</value>
<value>locker</value>
<value>datasource</value>
</list>
</property>
</bean>
Spring will do the rest (we import the jetm-monitoring.conf in spring-server.xml to bring it in context).

Final notes

You can do much more than what I've showed. You can export the metrics JETM collects via JMX and console, not just HTTP console. Please read the online docs for more details.
You can get the source code as example by checking out James App . After discussing this on James mailing list a number of other promising monitoring libraries turned up: metrics from Codehale and Servo from Netflix. They both deserve a look.

duminică, 4 martie 2012

Trying out (pending) James 3.0-beta4 HBase mail store - take 1

Today I decided to try the newest (pending) release of James : 3.0-beta4.
I have access to a VM cluster of machines running HBase and I wanted to see how things work, maybe do a small stress test.

Downloaded the binary release from the staging repo, unpacked and began configuring the James to enable the hbase mailstore implementation.

Changed a line in mailbox.conf to enable hbase mailstore provider, changed
import resource="classpath:META-INF/org/apache/james/spring-mailbox.xml"
from spring-server.xml to
import resource="classpath:META-INF/org/apache/james/spring-mailbox-hbase.xml"
and uncommented the mailbox declaration from spring-mailbox-hbase.xml to make the Hbase mail store implementation available to the spring context. After this I copied hbase-site.xml (and hdfs-site.xml) inside the James conf directory.

And with that last step James is configured to send emails to the Hbase cluster. All things done, I started James with bin/run and then.... Exception.

| james.mailprocessor | Unable to init mailet LocalDelivery: org.apache.mailet.MailetException: Could not load mailet (LocalDelivery);
nested exception is:
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.apache.james.transport.mailets.LocalDelivery': Injection of resource dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hbase-mailboxmanager' defined in class path resource [META-INF/org/apache/james/spring-mailbox-hbase.xml]: Cannot resolve reference to bean 'hbase-sessionMapperFactory' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hbase-sessionMapperFactory' defined in class path resource [META-INF/org/apache/james/spring-mailbox-hbase.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [org.apache.james.mailbox.hbase.HBaseMailboxSessionMapperFactory]: Constructor threw exception; nested exception is java.lang.RuntimeException: java.lang.IllegalArgumentException: Not a host:port pair: �%32649-- removed hostnames-- net,60000,1329515254741
.....


Searched the cause of exception and found out that it's because of a version mismatch. Unfortunately the cluster is running hbase 0.92 and the client is running 0.90. The client serialization is incompatible with the one used by the server.

Aside for the fact that the exception was fatal there is an upside to this: James is trying to connect to the the cluster.

Just found out that HBase 0.92 is on Maven Central, but without tests so this will make upgrading impossible because of the unit tests which require it.

The tests jar will be available in 0.92.1 (as stated on hbase-dev). If that tekes too long I will build and test it with a private release of Hbase 0.92.

Hope things move a bit faster with 0.92.1 with respect to artifacts being available on Maven Central.

miercuri, 8 februarie 2012

NIO Iterator over messages in mbox file

I've started working on a small project called mbox-iterator that I wish to integrate with mime4j later, when it's more usable.

The idea is to provide an Iterator over all the messages in a mbox file and provide access to the raw data. You can then use mime4j to parse the message and do whatever. It's very good for use cases when you don't need the data to be processed and would like to do your own processing or just need access to the raw data.

The project uses java NIO Memory Mapped files to map the file into memory. This will use the OS to manage the memory and file loading for you you get a ByteBuffer that you can use to accces the date. We are not using this ByteBuffer directly, because it's bytes and we need character sets so we need to decode the bytes. We get use a Charset to get a CharsetDecoder for the encoding we need. CharsetDecoder returns a CharBuffer instance and because it implements CharSequence we can use a regex to determine boundaries between messages.

I was hoping that CharBuffer would share the memory with the MappedByteBuffer that we get initially but it seems that this is not the case. We have some memory copying here because the CharBuffer we get is an instance of java.nio.HeapCharBuffer. I was thinking we could have zero Java heap memory and use just the O.S. pages for holding the mbox data but as it turns out, the process of translating bytes to chars needs some memory to keep the chars.

It would have been nice to use asCharBuffer() and provide zero-copy access, but darn, maybe with a future Java version.

Let's return to our cattle:

When we find such a boundary, we return a slice()ed CharBuffer to that message. We also set position and limit so that we get just the message. This has the advantage of using the same memory as the CharBuffer ww do matching on and avoids unnecessary memory copy operations.

I have tested it on a small mbox (135kb) and performs ok. I'm planning more tests with a 2gb mbox. Using NIO memory mapped files we can map very large files and use the OS cache and buffer memory so we can avoid GC activity and unnecessary memory copy operations.

The following areas need improvements before the project is usable:

- testing for different types of mbox files
- better regex to match different mbox formats (mboxcl, mboxrd and the likes)
- better regex for matching From_ lines (now we miss some From_ lines that have MAILER_DAEMON instead of email address)
- support for mboxes larger than 2GB - use more ByteBuffers to map portions of the files, watch out for mails that spill over boundaries.


I'm pretty new to NIO so if you have suggestions of how to do this better I'm open to suggestions and pull requests.

You will fins a simple example that splits one mbox into individual messages in the project sources.

Happy hacking.


References and links:

[1] http://www.kdgregory.com/index.php?page=java.byteBuffer
[2] http://en.wikipedia.org/wiki/Mbox
[3] http://qmail.org/man/man5/mbox.html
[4] http://james.apache.org/mime4j/index.html
[5] https://github.com/ieugen/mbox-iterator

marți, 27 septembrie 2011

New job

A few days ago I joined Big Data Solutions Team at 1and1. They have nice offices in Bucharest, but most of all I like the team and the projects.

Besides the fact that I will continue to work with HBase and Hadoop I will also expand in using Lucene, Solr and Mahout. All Apache technologies :).

I plan to use the experience I gain to integrate more functionality in Apache James, first on my list is mail search over HBase.

duminică, 4 septembrie 2011

James HBase mailbox is in: Give it a spin

Today I managed to finish integrating the HBase mailbox implemenation for James. I committed all project code based that I developed during Google Summer of code, including the integration tests.

I also added a spring context definition so now it's very easy to use. Just fire up James and supply it (put on the class path) with the xml configuration file for your HBase cluster. Users and domains can also be stored in HBase or in another supported repository (like a database).

This flexibility is achieved because the way James is built. All the important parts are spring components that can be interchanged easily. Also, if you did not already know: James is moving towards OSGI.

So what are you waiting for, go try out the new James goodies on your cluster and send us some feed-back if you have time.

Just remember you have to get the latest trunk version for this and do some compilation. Here you can get detailed instructions on how to do that.

If you are not in a hurry you can wait until we make a new release or check the nightly builds.