When a filter must take into account the values of pixels other than the ones being filtered, it's no longer possible to use the simple RGBFilter class. Instead you have to work directly with java.awt.image.ImageFilter. You must override the two setPixels() methods to implement your filter.
public void setPixels(int x, int y, int w, int h, ColorModel model,
byte pixels[], int offset, int scan)
public void setPixels(int x, int y, int w, int h, ColorModel model,
int pixels[], int offset, int scan)