MY(a)ware

Selenium the joy and horror - Cached page data, fails verification

2009-11-04T15:41:00.000+02:00

Up until Selenium IDE was created, manual testing was such a PITA. Now we have Selenium, bus as all new technologies and tools, there are a few quirks there...

If your tests are run very fast, and to save time you need them to be very fast, you will have issues with data verification.

Firefox sometimes displays cached version of a page, even though all the possible parameters and headers have been set not to cache the page.
I found that doing two open commands in a row helps this problem most of the time.
If, that does not help, then adding pause 500 in between the open commands helps.

Definitely turn off all panels of Firebug!

Do not forget to set "Pragma: no-cache", "Expires: [right now]", "Last-modified: [two days ago]", "Cache-control" and other headers.

The definitive guide of connecting Apache via LDAP SSL to ActiveDirectory + Subversion

2009-06-11T00:59:00.005+03:00

Being taken though hell I of configuring the damn authentication mechanism.
I needed to integrate Apache with ActiveDirectory via LDAP SSL. The manual SUCKS! It does not say anything useful, except for the syntax of the directives.
This article was the most definitive of configuring the integration. But lacks the information about how to make the SSL work correctly.

Here's the kicker, it is as simple as the following several elements(this is in the top of /etc/apache2/sites-available/default):
# Make sure ve don't care about the server's certificate, because we don't
LDAPVerifyServerCert off
LDAPTrustedMode SSL
# The server's client cert information: the cert and the matching private key
LDAPTrustedGlobalCert CERT_BASE64 /etc/apache2/sites-available/cert1.pem
LDAPTrustedGlobalCert KEY_BASE64 /etc/apache2/sites-available/key1.pem

These go into the ROOT. Do not try to put them in the Location, nor Directory, nor VirtualHost.

And the main change to enable the SSL transport:
AuthLDAPURL "ldaps://adserver.example:636/DC=adserver,DC=example?sAMAccountName?sub?(objectClass=*)" SSL

These are the steps:

1. Create the client Key and Certificate

The cert1.pem and key1.pem are created like described here:

openssl genrsa 1024 > key1.pem
openssl req -new -x509 -nodes -sha1 -days 365 -key key1.pem > cert1.pem

For an additional configuration reduction bonus you can have it in one single file:
cat cert1.pem key1.pem > pcert.pem
LDAPTrustedGlobalCert CERT_BASE64 /etc/apache2/sites-available/pcert.pem

2. Enable the correct modules on Apache HTTP Server 2.2.

On my Ubuntu system the module enabling is done like this:

sudo a2enmod alias

sudo a2enmod auth_basic

sudo a2enmod authnz_ldap

sudo a2enmod authz_default

sudo a2enmod authz_user

sudo a2enmod dav

sudo a2enmod dav_svn

sudo a2enmod ldap

Or uncomment these elements in the httpd.conf:
LoadModule actions_module modules/mod_actions.so
LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule authn_default_module modules/mod_authn_default.so
LoadModule authnz_ldap_module modules/mod_authnz_ldap.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule ldap_module modules/mod_ldap.so

And the modules that are prescribed by the Subversion.

3. The access restrictions and Subversion

The short info for configuring the access restrictions.
This will allow people in the users in SVN Writers to commit and SVN Readers will be able to checkout and connect.

<Location>
# This enables Subversion
DAV svn
# Location of the Subversion repository
SVNPath /home/ldaptest/

# How we are going to get authenticated
AuthBasicProvider ldap
AuthType Basic
AuthzLDAPAuthoritative on
AuthName "My Subversion server"
#The URL of the ActiveDirectory server
AuthLDAPURL "ldaps://adserver.example:636/DC=adserver,DC=example?sAMAccountName?sub?(objectClass=*)" SSL
# Credentials for the Apache HTTP to connect to the A/D to issue queries
AuthLDAPBindDN "subversion@adserver.example"
AuthLDAPBindPassword 555

# Limit all write operations to users within SVN Writers group
<LimitExcept>
require ldap-group CN=SVN Writers,CN=Users,DC=adserver,DC=example
</LimitExcept>
# Limit logon and reading to only users in SVN Readers group
require ldap-group CN=SVN Readers,CN=Users,DC=adserver,DC=example
</Location>

4. Add the following to the default site on Apache HTTP

The file is /etc/apache2/sites-available/default
Right at the TOP.

# Make sure ve don't care about the server's certificate, because we don't
LDAPVerifyServerCert off
LDAPTrustedMode SSL
# The server's client cert information: the cert and the matching private key
LDAPTrustedGlobalCert CERT_BASE64 /etc/apache2/sites-available/cert1.pem
LDAPTrustedGlobalCert KEY_BASE64 /etc/apache2/sites-available/key1.pem

Be careful how you truncate...

2009-06-02T11:16:00.006+03:00

Here's one example how truncation has made the entry look like it's a conspiracy theorists' website:
The text below reads:"...See the video at the end for the Test Results! NOT...posted by Kipay".

This one is from Instructibles

5 signs of a hijacked client

2009-05-24T22:30:00.006+03:00

The context rant

While doing my main line of work, I never shy away from doing some consultancy work for clients. And I mean no programming, I mean business operation consultancy. Though I am a application development consultant and application developer, I do have enough experience to identify when a company has issues with overhead in work. I mean premature optimization is what a lot of programmers are all about and sometimes the business side benefits from that very well.

For the topic at hand. I had this good friend nagging me for a long amount of time for a consult. He is a successful businessman and no one in their right mind would call his business a failure. It's a nice Anonymous LTD that manages to turn over a few million euros per year with 12 people. So I signed up. I do not consider charging my friends for any non technical work. Since I follow the ideology of having a lot of friends at the expense of quick income.

The "motherload"

So I started rather quickly on analyzing Anonymous LTD. The analysis part was quite fun and easy. I liked talking to people and they gladly shared all of the positives and negatives of their work. Some even had suggestions :) And that is where I started to suspect that something was wrong.

The employees are de facto subordinate non management staff.
The sales people were "afraid"(read subordinate) of not the management, but to the accounting. Yeah, that is when I had the first WTF moment.
There is a department that everyone tries to please, and not the management.
The sales and other employees felt that they were somehow more dependent on accounting rather then the strategical management. Remember, accounting is a support operation in non accounting related companies.
The management displays uncertainty and consults some department on business matters.(Legal and purely technical are different stories)
Really, when I went on to ask why the hell was there no CEO in the company, my friend, the director called up head of accounting to ask!!!!
You present a solution that improves managerial oversight and operations visibility and it's shot down by management.
I proposed a new reporting tool/function, that was integrated into the stock tracking and order system. The director and top sales manager said: "Wow! Great! We really want it!". And two days later, they cut out the functionality, because it had not enough financial data(the stock tracking and order system, was not suppose to have any financial info).
The management's position changes whenever the company gets new, non management level, people on board.
I already had the contract secured and signed. The company that was suppose to perform the work was selected and already analyzing requirements. Great! I thought. Then a new person is hired into the sales team. The contract is cancelled, because of hudge requirements changes.

The company had an issue with their stock to quote to order to delivery operation. It is unreliable and overburdened by out of date data. They still don't have any automation of warehouse stock information. Why? Because the new head sales-person wants to see a dashboard with all the client's information orders, payment statuses, credit and in real time and the accounting do not want to let that information out of their hands.

That is what I call a hijacked company:
A company that is taken over by a nonessential department and weak, collaborating, management.
Be very, very careful, it's a trap of uncertainty.

Aftermath

I managed to push trough a CEO for the company, because the managing director was and is my friend and a little too soft. And last time I checked, he still keeps accounting at bay and the director is non managing.

Why is software not that scalable?

2009-03-22T23:38:00.010+02:00

(FYI: This is a 3 AM rant, so don't take it too hard, if anyone reads this at all)

Most software out there relies so much on shared data, that it does not easily scale.

The problem with that is, that somehow developers don't understand that most of the time the data may be allowed to be stale(though still relevant) - see how Amazon Dynamo's clients handle it. Or that most data is accessed only by one process at a time.
In fact, when looking at human interaction systems we see that most of the data is stale when people perform a certain task.

Here are some point that I describe below:

Allow data to be stale, there is very little reason of making sure everything is always up to date all the time, spread the data as much as possible
Check out the data access patterns, most data is access
Understand that need for data does not involve one process/thread being hung up on one resource
If you need stuff to be up-to-date in real time, share the responsibility - aggregating data is faster and easier

On the freshness of data.

Sample from most of the governmental institutions: laws and regulations have a time of activation that is shared in advance. A lot of decisions are made at a low level without checking the central(at a national level) legal repository every time. That is when data is stale, but still very much relevant.

Sample from most businesses: business rules are much more dynamic than at the governmental institution, but still we see that most of them change very little. Sure you can change it at once, but you would never expect any of you own businesses employees to act on the changed rules, until the employee gets the message about a change in the rules.

Sample from every society that has laws that change or may be changed: when a law is passed, you don't know that is has changes some other law or made something illegal or legal instantly. It has a date of coming into force. Up until then every person will know most important laws that are in effect and when the new law comes into effect.

Then there is the access to data.

Sample from banking: chances are that there will be no more than ONE "process" using an account data at a time. In fact, I bet that, the absolute majority of operations are performed by one process accessing ONE account. The reality is, that the only part that the bank needs to be absolutely sure about is the "credit" operation(Say bye to "credit->debit|undo transaction"). The debit operation can be reasonably fast in asynchronous mode and can send notification of a failure.

Sample from real life: if you don't find your personal mug in the cupboard you don't just stand at the cupboard until someone returns it! You go and look around. The mug is the data here and the person is the process. Standing and waiting for it ti be returned is equivalent of locking a resource.
As this sample shows, locking and waiting is not always the best option. The reality is, that it's not the process that has taken the resource that locks, it's the waiting resource that locks and waits.

My proposed strategy, is acquire ownership of the data and leave a note stating who knows it's whereabouts.

So the morale is don't lock, but take. Don't wait, but look around.

Now. I can't really comment on systems where real time data is essential. But in medicine real time means means the difference between life and death. But those systems are embarrassingly parallelizable.

Remember, our world works on Eventually Consistent Model, not the, ironically fatally named, ACID one.

Just look around, the world is already massively parallel, why create new ideas when mother nature has provided us with a lot of the answers.

WOW! Moment

2009-03-07T12:59:00.001+02:00

Rarely do I go "Wow!". But this is something incredible!
Check out Aurora Concept

Make your Hibernate data objects smart?

2008-12-25T14:43:00.003+02:00

In a race to learn Hibernate and create products for my former job, I have found a simple way to make my Hibernate data objects smarter.
We all know th standard way to access the Hibernate session using the SessionHolder*.

This is a basic abstract class(shown below) that will add the .add(), .update() and .delete() methods to your data objects.

Just extend this class by your Hibernate DTO like this:
public class Address extends BaseData

And then just after populating the new Address object, you write address.add().

public abstract class BaseData implements Serializable {
 
    public String add(){
 
        try{
 
            SessionHolder.currentSession().getSess().save(this);
 
            SessionHolder.currentSession().getSess().flush();
 
        }catch(Exception ex){
 
            SessionHolder.endSession();
 
            ex.printStackTrace();
 
            Messenger.addFatalMessage(ex.getMessage(),ex.getMessage());
 
            return StandardResults.FAIL;
 
        }
 
        return StandartResults.SUCCESS;
 
    }
 
    public String update(){
 
        try {
 
            SessionHolder.currentSession().getSess().update(this);
 
            SessionHolder.currentSession().getSess().flush();
 
            return StandartResults.SUCCESS;
 
        } catch (Exception ex) {
 
            SessionHolder.endSession();
 
            ex.printStackTrace();
 
            Messenger.addFatalMessage(ex.getMessage(),ex.getMessage());
 
            return StandardResults.FAIL;
 
        }
 
    }
 
    public String delete(){
 
        try {
 
            SessionHolder.currentSession().getSess().delete(this);
 
            SessionHolder.currentSession().getSess().flush();
 
            return StandartResults.SUCCESS;
 
        } catch (HibernateException ex) {
 
            SessionHolder.endSession();
 
            ex.printStackTrace();
 
            Messenger.addFatalMessage(ex.getMessage(),ex.getMessage());
 
            return StandardResults.FAIL;
 
        }
 
    }
   
 
}

This particular class was designed to be used in a JSF web application. Hence, the return types for methods are String's and there is a reference to StandardResults and Messenger class.
Messenger class is a generic class to present the user with a message.
StandardResults is what it's name stands for.

Please note, this may be used in JSF tables and lists only in places where the data is changed rarely or not concurrently.
But placing this element in a column in a h:dataTable will work as intended:

Also on your edit form page, you may use it without any potential problems, like this:

And on your new entry form page similarly:

In the end this technique saves a lot of time, by removing the need to create add, update and remove mehtods in your JSF managed-beans for those simple and boring CRUD operations.

* - For those who don't: see here or in simple terms this thing holds the session for the current thread and context - in the example, the call SessionHolder.currentSession() is translated to HibernateUtil.getSessionFactory().getCurrentSession().

How about a nice future for DBMS query languages?

2008-12-15T01:21:00.004+02:00

Now thinking of SQL, it is a perfectly good language, but it is old. It is being pushed now to something that was not really intended for it.

We start needing to have more complex structures returned from our databases. See emergence of Amazon's SimpleDB, CouchDB and such.

What about a database where we talk to it in something more familiar? Even JavaScript seems nice... Well, the nice thing is that it's OOP model is basically tuples, the thing that most RDBMS'es understand perfectly well...
The problem at hand is the relation, but that is for another "pondering night".

Imagine this SQL query:

SELECT * FROM table_a WHERE id > 5 and name = 'Test'

In JavaScript form:

table_a.filter( {id:gt(5),name:"Test"} ).all()
(gt(5) here is a function returning a function that compares input so that it's greater than 5)

Readable? Not too much. Flexible? Definitely. Can it be improved? Absolutely!

There are more readable languages there... Like Python or Ruby.

How imperfect can you be to be good enough?

2008-12-12T16:08:00.003+02:00

How imperfect can you be to be good enough?

This is so true for software development and much more than just software development. There is no 100% reliability, no 100% testability.

Nothing is absolute, and absolute is nothing.

Quack my class: A lil' bit more dinamicity in Java

2008-12-10T23:50:00.002+02:00

Java is criticized for being too rigid due to static typing. But what do we know about type dynamicity in Java:

Reflection and bean introspection
In theory you can write statically untraceable code using only refection API, that would constitute dynamicity.
Java 1.4 Proxy API
Need to have a stub for an interface? This is the tool for you. Supply an interface and InvocationHandler and you will get an Object that implements the Interface and will forward the method calls to your InvocationHandler.
ClassLoading is on demand
You may load classes from different locations on demand and invoke them thus adding new functionality on the fly. This leads to ability to generate code, compile it and load without intervention of a developer.
Java 5 introduced the instrumentation API
Java 5 gave us something new. From this version we can redefine classes via a standard interface – java agents.
The upcoming invokedynamic JVM isntruction
invokedynamic is less about Java as a language, but more to introduce more languages to JVM, specially the dynamic ones.

On the scene we have OO “dynamic” languages that follow something called duck typing.
Duck typing - “if it walks like a duck and quacks like a duck, therefore it must be a duck”, the languages that have that scheme only determine if the variable will accept the method call or have a particular property/instance variable with corresponding name. But they will “complain” if the call was made incorrectly.

In essence if you provide any object that has a method walk, to code that expects object of type Human, it will execute without any problems.

Duck typing in Java

True duck typing in Java is not really possible. But, there is always a “but”.
Say there are 2 classes that are identical in their method signature, duck typing would allow those to be “interchangeable”. In Java you would need both of them to implement same interface to do it “correctly”.

We can also do some hacking and create an interface and use that. As for the classes we could create an InvocationHandler and a Proxy, witch would forward the method calls to the classes.[(2) in the list above].

The code would be like the following:

HashMap anything = new HashMap();

DuckInvocationHandler dih = new DuckInvocationHandler(anything);

anything.put("test", "Congratulations, it WORKS!");

MyMap duck = (MyMap)Proxy.newProxyInstance(Thread.currentThread().getContextClassLoader(), new Class[]{MyMap.class}, dih);

System.out.println("Out: "+ duck.get("test"));

The interface to represent what the result we want:

public interface MyMap {

Object get(Object i);

}

The InvocationHandler implementation:

java.lang.reflect.InvocationHandler has only one method to override.

@Override

public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {

Method mth = nonDuckClass.getMethod(method.getName(), method.getParameterTypes());

return mth.invoke(nonDuck, args);

}

This approach:

Is standards based
Will work only with interfaces

There is another way. A popular library for class manipulation is CGLib. It features the ability to Proxy a class, by creating a subclass of it at runtime.

How will the code look? CGLib also has an InvocationHandler, and it also has only one method invoke. So just replace java.lang.reflect.InvocationHandler with net.sf.cglib.proxy.InvocationHandler, and the conversion process is done for the handler.

The proxy creation process is a bit longer, but does have it's own advantages. The code:

// Use caching Enhancer object

Enhancer e = new Enhancer();

e.setUseCache(true);

e.setCallback(duckInvocationHandlerInstance);

e.setClassLoader(o.getClass().getClassLoader());

e.setInterceptDuringConstruction(false);

if (asType.isInterface())

e.setInterfaces(new Class[] { asType });

else

e.setSuperclass(asType);

Object output = e.create();

This approach:

Caching : As you can see on line 3 we set the Enhancer to use cache.
Works with both classes and interfaces

What are we missing? Type safety.
For this to work correctly before you make the class quack as you wish, you need to check if the type and return type classes are compatible. For this I have created a small library, ready to use, but still a bit limited.

The limitation is, that return and parameters values are not proxied. Therefore parameters have to of the same type and return type has to be either same type or subtype.

The result

We may or may not want anything to do with duck typing. The reasons against is that Java was built to be strict. It violates the core Java philosophy – more checks during compile time. Although, this way is much better then totally blind duck typing, since you have to specify the interface.

Where might we use it?

The main reason for using it, is to be able to practically alias some APIs and possibly use different versions of one API in the same code.

Dynamic Dependency Injection

2008-12-02T17:11:00.007+02:00

Dynamic Dependency Injection, Context Aware Injection or "on the fly" configuration.

Something that should be real easy, though is not thought of in Spring nor in Guice.

What is DDI?

The challenges that DI meets are well defined. And are already met. This is the next thing, in removing more glue code from you application code.
The target of DDI is to to remove as much as possible of non value bearing code. Value bearing code is the pure logic. Like Seam removes the need to get the data from the JSF forms and places them in the @In annotated field. And you are left only with the need to do actual value bearing logic.
Sample from here:


  public String register()
  {
    List existing = em.createQuery("select username from User where username=#{user.username}")
      .getResultList();

    if (existing.size()==0)
    {
      em.persist(user);
      log.info("Registered new user #{user.username}");
      return "/registered.xhtml";
    }
    else
    {
      FacesMessages.instance().add("User #{user.username} already exists");
      return null;
    }
  }

In the sample above your job as a developer is to do checks and proceed with the right branch of operation.
That is 100% value bearing code, since it does exactly what it's intention is:

Check the data store for duplicate values
Save the data if not exists
Report error if exists

The @In annotated field is populated based on the context in witch the EJB was run. In Spring and Guice, it is awkward when doing the same. Mostly the beans are configured once and relations are more dynamic.

Now DDI is another step forward. I say: Why remove only the setting and basic configuration? Let us remove the data fetching code and "script it".

Spring DI code would be:


  @Autowired
  private WarehouseQuantityDAO dao;

  @Autowired
  private OrderItemTO oi;

  public boolean checkAvailableQuantity(){
    int availableQuantity = dao.getQuantity(oi.getProductCode());
    return availableQuantity > oi.getOrderedQuantity();
  }

While DDI code would be something similar:


  @Autowired
  @Bound(name="JPAEntity_named_query",bind="orderItem.productCode")
  private int availableQuantity;

  @Autowired
  private OrderItemTO orderItem;

  public boolean checkAvailableQuantity(){
    return availableQuantity > oi.getOrderedQuantity();
  }

Though the end code seems similar, we did get rid of the DAO class.
The @Bound annotation takes the value of orderItem.productCode property and sets the parameter in the JPA named query. The JPA query returns an Integer, so we just set the value just prior to the calling of the method.

In this way we can remove a lot of code that deals with data storage, when the essence of the logic is not data store.

Meaning that if we change the register() method code from Seam sample, it might be not as obvious that the whole idea of the code is to check the data store. Though it would make the code shorter:

@Autowired
@Bound(name="JPAEntity_named_query",bind="user.username")
private List existing;

public String register()
{
  if (existing.size()==0)
  {
      em.persist(user);
      log.info("Registered new user #{user.username}");
      return "/registered.xhtml";
  }
  else
  {
    FacesMessages.instance().add("User #{user.username} already exists");
    return null;
  }
}

But the logic is the same.
This could be done via AOP in Spring and maybe in Guice.

Referenced material:
http://www.springframework.org/
http://code.google.com/p/google-guice/
http://seamframework.org/

Requirements Weight Matrix

2008-12-01T18:22:00.000+02:00

Histories

     The weight matrix came into being after a business analysis project at one medium sized company. During their business analysis we discovered that most of their unproductive time is spent on looking up the information on the products in stock.
     The salesmen communicated with clients mostly by phone, after initial cooperation contract was signed. So informing the clients weather their order could be fulfilled the same day or in a month was not an easy task.

What I ended up with

     So while doing simple feature mapping I ended up with the following matrix:

This is only a sample

	Management	Sales	Accounting	Weight
(Features/Group Modifiers)	1.5	1	0.9	Total
Show items in warehouses	O	ET	R	3.58
Send notification emails	ED	ED	ED	2.55
Remove item reservation	R	O	RC	2.18
Reserve items in warehouses	N	EC	R	1.95
Pending shipment packages	O	N	O	1.8
Order edit	N	E	N	1.5
Order entry	N	EC	N	1.5
Salesman report	R	N	RC	1.43
Warehouse item movement report	N	N	EC	1.35
Order cancel	N	EC	N	1
Shipment package info	N	N	R	0.45

Modifiers
C - critical or core business(job role)	1.5
T - client critical	2
D - downtime possible.	0.75

Marks
E - everyday use	1
O - occasional use(once per week)	0.75
R - rare use(once per month and rarer)	0.5
N - never used or unknown functionality	0

     The matrix has to be filled out by each group separately. Anonymously and personally.
     To aggregate, you need to get the median point from each group, but it should not be too far from the mean. (Substitute marks and modifiers with the actual values)

     The weight matrix is constructed of the most basic functionality against user groups and marked with importance to each user group.
     Each feature is a functionality that is expressed by up to 4 words(each noun acronym is one word) , so XYZ process start is a valid function descriptor.

     A functionality does not imply a UI button or action. Better, yet, UI functionality should be separated from actual functionality(reports are pure UI, no matter how much processing is done in the backend, except when reports are auto-generated)

     Each user group marks each functionality with marks and modifiers.

Marks:

E - everyday use
O - occasional use(at least once per week or 2 times per month)
R - rare use(once per month)
N - never used or unknown functionality or not relevant to job role

Modifiers:

C - critical or core business
will loose time/revenue as a result of failure, possibility of loosing clients
T - client critical
will loose clients as a result of failure or equatable to loosing clients
D - downtime possible
has little or no everyday impact on operations

     Each of these marks have their weight, each modifier is a multiplier and each users group has it's own multiplier.
     In the sample group multipliers Management - 1.25, Sales - 1 and Accounting - 0.9. And mark and modifier values are listed below.
     It is not impossible for all to have the same multiplier.

   There are basic rules to the values though:

Highest value is ET - Everyday - Client Critical
EC - Everyday Core Business is the second highest value and greater or equal to OT Occasional Client Critical
EC > E and EC >= OT
OC - Occasional and Critical to business should have greater value than Everyday use
OC > E and OC <>
E - Everyday should have the nominal value like 1 and should be greater of equal to RT - Rare Client Critical
E >= RT
... and so on, the common equation is(please feel free to modify it):
N <>

     To have the most important functions, all the responses are converted to numbers, all the multipliers are applied and each functionality's values are summed up.
     The result is that you have real representation of the value of the functionalities across whole business and with points of interest of each group of users/stake holders/beneficiaries.

The end result

     From the business side the example demonstrates that in fact for that company to see the warehouse stock in real time was the most important feature and indeed that was the fact.
     As you can see that the order related activities are not at the top, this is because the company being analyzed was a wholesale company, where the amount of orders was easily manageable and had a distinct paper trail.

     This matrix made it simple in prioritizing features to be implemented, witch lead to a well accepted project delivery. And the project did not get tangled up in the management's view of how things were actually working.*
     It was a win-win situation, because the sales people got their information in real time and the management got a good view from above. All in all, increased productivity never hurt anyone.

This maybe new or not so new idea, so I will not rush to claim that I invented it.

* - Note, none of the team were stupid, and there are a lot of acceptable controls over the business for the management. And the system ended up quite flexible and usable.

NetBeans 6.5 vs Eclipse 3.4 Ganymede on Java Coding

2008-11-02T16:19:00.004+02:00

Netbeans IDE and Eclipse IDE
I like and use both of these IDE's for different reasons. Netbeans is perfect for non Java development and mixed language development. Eclipse is the best option for Java coding. Why? Here is my reasoning.

Netbeans:

Great technology integration
the level of integration of assisting technologies as well as frameworks is much better than Eclipse.
Added ability to support a lot of other technologies
Ruby with Ruby on Rails, PHP, JavaScript and Groovy are supported and integrated
Out of the box usable

Eclipse:

Huge community
gives access to various plugins, though their quality depends on the developer
Incredible flexibility
Basically everything may be tweaked and configured

Now why is Netbeans falling behind on Java coding? (IMHO)
Some will say that incremental compilation is the point of weakness, but I really see it a lesser issue in productivity.
The code assist is the main problem. And some defaults are enforced.

Code complete: Netbeans has been improved incredibly. Eclipse has groups of code completion. Each code completion group has it's own popup. Each popup is opened(switched to) by pressing Crtl+Space repeatedly.
Code complete: Eclipse has not only classes and code generation elements in the code completion popup, but the code templates. Such as typing main and pressing Ctrl+Space in Eclipse will give you an option to create a main method. Netbeans way is write psvm and hit code expansion button(CEB, mine is TAB).
Code complete: Intrusive template expansion in Netbeans. Just write in pst and hit CEB in Netbeans and, even though you just wanted a variable name for your PreparedStatement, you will get "printStackTrace();" in place of your "pst". And nothing warns you into what the template will be expanded...
Code expansion button = CEB, is a horrible thing if it's mapped to SPACE!
Custom Run configurations are a good idea in Eclipse. Though I still like the button to run the Application in Netbeans.
Netbeans enforces some defaults. The one that I hate the most is that everything in "Test Packages" has to be a jUnit test. Why? Tests may be simple classes with main methods! But you don't have an option to run it using something else other than jUnit.

Basically those are few simple reasons why I don't use Netbeans for Java coding.

Java ClassLoading

2008-04-16T04:48:00.002+03:00

The essential part of the java world, but why is it so rigid and cumbersome?

Every time we start Java the class loader is the essential part that drives the startup process.
Every single piece of code that you have ever written for java is handled by some ClassLoader.
Because it's not that easy. When a developer can develop an application that effectively uses ClassLoaders, I consider that person to have advanced level of knowledge of Java.

ClassLoader is essential in defining a Class. Basically a class lives attached to a class loader. That is both a good and a bad thing.

The main positive thing is that we can have multiple versions of the same class without conflicts
The main negative thing(in current state) is that we need to remove the whole ClassLoader(with it's children) to reload our changes.

The basic fact is that these 2 points are not mutually exclusive.
But this way is mature. It's been with us for a very long time. We are actually used to it in it's current state.

That is where tools like ZeroTurnaround's JavaRebel comes in our view. The idea is nice, it's executed well, and it definitely shows that there is a need for it.

They use the new Instrumentation API.
Instrumentation API is great, but it does have a little problem: no schema changes are allowed.
That means that most of your changes will need the old process.

Is it time we had a new method in ClassLoader that would unload a class(ClassLoader.undefineClass)? Or reload it, at least?
I mean, the dynamic languages had that forever. PHP is a language that has a lot of inexperienced developers, and they still rarely manage to shoot themselves with that "feature". Does everyone consider everyone in java land an idiot that has to be constricted in every possible way?

Java(and Java EE) deployment lifecycle is criticized a lot. It's long. Very long. And the compile time is not the main factor here, it's the deployment itself. With the current need to trash the whole ClassLoader hierarchy we are definitely taking too much time and memory consumption(resource usage) does suffer.
Every single (re-)deploy goes through the same procedure on the lowest level:

Application specific operations
Remove any references to the ClassLoader(s) of the application
Create a new ClassLoader
Link the ClassLoader to the deployed code
Application specific operations

And this is an example where I just added a new logging method and a few statements to a class! As you look at it you might think what a sad state we are in!
And if you have a thread there, guess you will have to restart the JVM to get rid of the old code from the memory.

Now, I am NOT an expert in JVM internals, but how hard is it to change the piece of data in memory? Java classes are just data in the memory, like everything else.

So what do you think? Is it time we have an ability to change the bytecode more dynamically?
To have the classes reloaded through ClassLoader? Or, at least, the Instrumentation API would have the restriction on schema changes removed?

What are your opinions?

Do you have a backout profession?

2008-04-09T06:33:00.009+03:00

We are living in a world where needs for professionals often rise and fall.
Since the US is in a rough patch right now, we are seeing a decline of developer positions. Some are due to outsourcing to India. Others just because there is less Financing and projects are stopped.
Europe is much the same, just the condition is generally a bit better. But hey, US is a major trading partner for us. And since I see little of American made goods on the shelves(and a lot of EU goods on American shelves), I expect that EU is mainly exporting to US a ton of products.
So we in EU WILL be starting to feel the problems over the Atlantic some time in the near future.

Software developer is a really volatile profession.

We rarely need to be at the client's side.
We may be successfully outsourced to another continent,
maybe the moon would be a possible destination if there would be enough population :)
Most developers are closed off people, at least during the crunch hours. That leads to indifference of the management towards employees they don't "see".

I mean if the cleaning person would be sick for a week you would notice it. Even the office administrator, office manager are more visible professions than a software developer.

So basically professions which services can be acquired via telecommunication technologies are really volatile by now. And those professions are usually the ones that are not really "visible".

So when you are planning your future DO you have a backout profession?

I do. I was thinking of it for quite some time. For some reason it was obvois you cannot rely on a single profession where you are dependant on single technology. Specially where it, in theory, may be able to automate itself in the future.(AI making AI, Robots making Robots)
My filosophy had always been: Do not put all eggs in a single basket.

My profession of nessecity* is a cook. A backout profession as I think of it.
(Don't get me wrong, I LOVE cooking)

Why?
Simple.
There are some really basic professions, that cannot be outsourced:

a cook
a healer
a barber
etc...

The simple stuff will always be here.
These professions are the ones that satisfy our basic life needs. So I would never expect them tho disappear any time in the future, unless the huamn species are wiped out or the earth blows up.

And in the end I will still do coding and development for fun. No matter where I end up.
It's not about the money or stability, it's about doing what I enjoy.
And by a coincidence my second thing that I enjoy is one of the most basic professions of all - cooking.

(I do filosphy when I'm sleepy... Can't help myself...)

* - both sides in this word: others need for; my need for.

GOOD vs BAD? Hugh....

2007-09-24T16:13:00.000+03:00

I have 2 morals to life that cover everything:

Life is all about HARD choices between BAD and equally BAD
"Ignore it and it will go away" DOES NOT WORK!

Like it or not there is NO right answer to anything that impacts our life. The choices that you have to make are never GOOD VS BAD, its rather what you think is good VS what you think is bad or even worse what is GOOD for you VS what is GOOD for someone else.

Trying to ignore problems makes them pile up and they will crush you eventually.

Perspecitves by expereince

2007-09-06T15:09:00.000+03:00

With all our experiences we tend to have a biased view. That is how smart people operate. To get to wise level we need to understand that time changes everything.

Same goes for programming...
When you first need to evaluate frameworks, methods and other stuff. You must not jump to conclusions due to last bad experience. I was burnt a lot of times due to selecting the framework that was "cool" last time I used it, and was nowhere as good for a new project. Now I can say that I will not throw out a method or framework just because it's was not as good as I expected last project.
At one time I was angry with RoR for not supporting PostgreSQL, that led to dumping it on next project as a candidate, at the end we dumped all our work and rewrote it in RoR and still made the deadline, just because I "saw the light" and tried it once more. And unfortunately I did the same mistake on next project, but it was the other way around.... RoR was selected just because it was fast... We got in a lot of trouble(mostly due to ACID-icity) and had to switch to Java EE and .NET combo...(I will not give any technical specifics of both projects)
After those 2 project I now have a mind that is more open to possibilities, thus giving me the edge over people that see only one solution to any problem...

So the moral is biblical: Don't judge and you will not be judged....

When approaching any problem you have to let the "old bad" stuff be on the same plank as new and cool.

Testing paralell code(the way YOU want)

2007-08-17T15:03:00.000+03:00

In my quest for coding nirvana I came across the need to test parallel code correctly.

Current testing toolkits are rather scarce on this front. There is no thread management included with them.
Once I needed that all code would be tested under 2 conditions: threaded and serial.
I had to create a new revision with a lot of code changes. The way I wanted it to be would be through an implicit thread manager, that can be told to force the code to be serial.

So far I thought up a simple technique of enhancing Thread class by adding callbacks to constructors and start method. This technique actually works. All you need is to define a transformer by the means of Java 6.0 instrumentation functionality. And using ASM from Objectweb you can add hooks to Thread object, otherwise it is not allowed to redefine java.lang.* classes.

And I expect that someone will sometime find a way of adding more manageability to Threads during testing.
Therefore we could catch thread related bugs like race conditions etc...
Unfortunately wait method on Object is native, so it would be hard to control waiting aspect of testing...

The price and value of software

2007-07-20T12:31:00.000+03:00

I am a person that likes to compare alternatives.
Software space is not excluded from the comparison space.

I have some strict moral rules on illegaly using software. If you're using some software that generates revenue or lowers costs, you MUST pay for it. I agree that MS Office is a good product and I can pay 800 euros(with Windows) to use it when needed, but enabling software like Windows, brining NO value by itself, gives me 0 reasons to buy it. Windows wins out ONLY on games, nothing else would keep me with Windows.

Entertainment software(games and such) are different species, but I follow the same rules.
There I can say that if I buy that particular title will it keep men entertained for the same time as I would have spent the same money on an alternative? Say cinema? New titles go up to 60 euros and more. FPS are the biggest problems, I can complete HL2 in 6 hours non interruptible time. Now once you played, it's no longer that interesting and with a single ticket to a cinema costing < 5 euros with movies 1.5 to 2 hours I think spending the same time sitting in a cinema costs LESS while providing almost the same entertainment(personal view).

So basically I would argue that like some prices are calculated based on how much you would spend on alternatives minus some small percentage, software prices should do the same.
Software for business purposes should be priced not higher that the possible time savings. Custom software is another issue.
Entertainment software prices should definitely be recalculated to honor the possible alternatives. Therefore im my country MOST of games are illegal copies. I personally play only WoW, the only game that I can say brings more entertainment than alternatives.

Application developement process NIRVANA

2007-04-17T05:40:00.000+03:00

What would you like coding to evolve to? I did a little thought on that and came up with some simple ideas.

Going in the footsteps of Spring framework, we want LESS plumbing code. Java, XML or other.*

I asked myself what I wanted to ease my "typing pain"?
For each piece of code that covers some logic, I want to say(ok, ok type in) I need this to operate. And I mean not like Spring does it, I mean like "I need the third letter from 17th page of a book that the client has selected to do my job". I don't want verbose XML's or Java helper classes for that. And I want it to be generic as much as possible, sure some things will still have to be coded and I do not mind doing it for too long.

Defining process elements should be a simple operation. I like that it's already simple @ the top, but down here I am still in hell. Simple flows should be SIMPLE! (Map a to b and save to data store - should take no longer than 1 hour) Calling save method on data store should be something for optimization or out of the ordinary... Creation may be a different story...

Otherwise Spring has it handled good, since auto-wiring components is a thing that I need to still be lazy...

At the end I thought "Damn I'm LAZY".

But then I thought of threads! Why J2EE containers "restrict" us using them? Why JMS became the solution to do parallel processing?
Could we make so, that we declare that this logic is run in parallel and has a barrier? And I mean that that thread would be practically invisible, have the same auth token and maybe even run in the same global TX?

The next thing is about saving state. I want be able to save state in such a way that the stack and execution points are saved. Therefore when restoring state the code could run from a certain point onwards, having the same local var's as if it had just come out of a long wait call. And I will not go to the extreme here, and be thankful for a partial implementation for this functionality... Some language constructs with some "save state here, and restore from here"...
But this should directly linked with transactional memory. Ans since I am not for global transactional memory, I would just love transactional blocks.

* - I am a person that sees extremes as answers. Fortunately I am smart enough to restrain those "feelings", but I'll be the most extreme here.

JDBC 4.0, really?

2007-02-03T00:09:00.000+02:00

Why is half the stuff missing from JDBC 4.0 that was there?
I waited for killer things like the Queries and DataSets (iBatis goes BOOM!).
Although I would prefer that creating multi column data type would be there, instead of Queries and DataSets.
It would be nice to have something like:

@JdbcDataType
public class SomeData{
String a;
@Alias("c")
String b:
// Setters and getter and other stuff ......
}

PreparedStatement pst = connection.prepareStatement("SELECT a,c FROM SOME_COMPLEX_VIEW_OR_FUNCTION WHERE a= ?a AND b = ?c ");
pst.setObject(instanceOfSomeData);
ResultSet rs = pst.executeQuery();
while(rs.next()){
SomeData data = rs.getObject(SomeData.class);// And of course we need overloaded method
} // That takes a factory instance for objects

Simple, easy.

IBM releases Java SDK 6 Early access, finally

2007-02-03T00:02:00.000+02:00

Glad to hear that IBM has got a bit more progressive on their software front.

Maybe we will get a certified for Java EE 5 WebSphere application server?

Surely many great products have emerged from IBM, but somehow for last few years IBM has been a bit slow in adopting technologies. Good to see the wind changing. Hopefully they still will be able to keep the quality of products and quality standards HIGH on "stable" releases.
Well in my personal opinion DB2 ROCKS! and if WAS would be just half of what DB2 is than LOOKOUT Oracle and Bea!

SOA and RL(Real Life)

2006-11-11T02:58:00.000+02:00

WS-* is probably the most popular SOA base.
But if you want real improvements on the base level focus on decoupled components!
Decoupled is the thing that will keep your code clean and easy.

Of course this is not real life where SOA = WS. SOAP(Simple Object Access Protocol) please remove S out of SOAP or at least change it to something like OAP(Object Access Protocol). Have you tried to understand all the specs for WS? XML is easy and readable, XML Schemas introduce complexity(but NP), but WS has so many elements that I get lost in them. WS is slow, WS is complex, WS is not easy.

Writing decoupled components implies such techniques as coding against interfaces, breaking down components into logically separate modules(not just refactoring classes) and other decoupling methods.
When you look at 3 tier architectures, MVC and IoC(Dependency injection) you will see that they are decoupled into logical(not everywhere) parts.
When not to go berserk and decouple everything from everything example: Accounting logic should not be decoupled from Tax application logic(most of the times), file storage logic is logically with file attribute logic.
What you see that is not core part(and in "core" I mean that logic that is bound to the process that invokes it) is usually should be moved out of the code and expose it as a "service"(no reference to SOA or WS).

The best thing that works for me is just imagining thing in reality.
How would you use accounting services?

You would go to an accountant and request to calculate your taxes. You do not care what the accountant does as long as you get the correct result.
When you have cash and want to deposit it you go to the bank, wait in line and give your money, id and deposit form and you wait for response that your transaction is complete(In this scenario money = data, bank teller = DA component, safe = data store).
Your secretary is writing a letter after you.(your words = data, secretary = DA component, pen and paper = data store, and you may even get corruption)

In 1 - you don't have to learn all the tax laws = easier life, 2 - you don't have to store money at home = less chances to be robbed, 3 - you don't need to write and check spelling(I was not the best speller in school maybe you were?)

JUST TO NOTE: COMMUNITY EVOLVED STANDARDS ARE GENERALLY BETTER THAN A STANDARD CREATED BY A STANDARD BODY!

DA - data access

Hide the developers! The "business" is coming

2006-11-01T11:26:00.000+02:00

What a the developer on the ground needs the most is simplicity. Any developer should be like a technical guy. The architect/project lead comes and says do that and that and do not worry about anything else. That is why there are PMs and architects - they are the "buffer" between all the crazy requests end client makes, even on the latter stages of the project. I recently was in a project that wanted to introduce new requirements and concept change 2 weeks after the development phase had ended and all functionality was finalized.

If you are a developer that is talking to "the client" your productivity as a developer falls tremendously. Be careful what you promise and NEVER EVER say "yes".
If you ever tried to look at how the specification is being created you'll see that for every requirement PMs battle with "the client". I wouldn't call it a requirement session but rather requirement battlefield.

Be careful though as a developer you have to protect yourself against bad PMs and architects. Never let PM or architect say "this is easy" when you know that this is not easy with the technology that was selected. When youre alone from you team in a meeting with "the client" always say "we will have to check it and we'll get back to you on it" because even "ok" will seem to the client that you are ready to accept his terms where you may be not.

So remember if you're a developer concentrate on the technical side and leave "the client" to PM.

"business" - is usually the end client of the project.

(Most of this stuff is my personal mental note)

Testing everything

2006-10-30T13:26:00.000+02:00

I always need to test.
Never do trust that anything will work. I always get this urge to test. I have absolutely no faith in theory. Practice makes perfect ... not theory.

Developing applications without the ability to test if the code is working is too much for me, it's like painting blindfolded having both your hands in casts.
Note to the tech staff and IT security staff: If you do not give developers full access to every resource that will be used(not actually the real thing but at least an accessible copy) than you will 100% get security breach because the code could not be tested under real conditions.

The testing is a very head on process that allows the developer to test most of the code for most visible bugs. If we cannot test that means we are back in the pre-PC era. Look at what we develop and do not forget that testing is one of the most important things in the current development methods. I for one write code fast and not always do it right, because it would take me an awful lot of time to write clean and bug free code without testing like in assembler or machine code for microcontrollers.