Q: How to make the JMX custom authentication work? I am using the password and access file based authentication on JMX. When building my JMXConnectorServer, i use the property names and it works fine. 
Map<String, String> env = new HashMap<String, String>();
env.put(ApplicationProperties.JMX_PWD_FILE_PROP, pwdFile);
env.put(ApplicationProperties.JMX_ACCESS_FILE_PROP, accFile);
connectorServer = JMXConnectorServerFactory.newJMXConnectorServer(jmxServiceURL, env, mBeanServer);

However, now i want to use a custom authenticator and i implemented my own LoginModule to have a encrypted password in the password file. Thus the ideas is to have an encrypted password and plain text user name in the password file. 
public class ABCDJMXLoginModule implements LoginModule {

    private CallbackHandler callbackHandler;
    private Subject subject;
    private String u_username;
    private String u_password;
    private JMXPrincipal user;
    private Properties userCredentials;
    private String passwordFile;
    private String f_username;
    private String f_password;

    private static final Logger logger = LoggerFactory.getLogger(ABCDJMXLoginModule.class);

    public boolean abort() throws LoginException {
        // TODO Auto-generated method stub
        return false;
    }

    public boolean commit() throws LoginException {
        // TODO Auto-generated method stub
        return true;
    }

    public void initialize(Subject subject, CallbackHandler callbackHandler, Map<String, ?> sharedState,
            Map<String, ?> options) {
        this.subject = subject;
        this.callbackHandler = callbackHandler;
    }

    public boolean login() throws LoginException {
        try {
            attemptLogin();
            loadPasswordFile();
        } catch (Exception e) {
            logger.info("Exception, e");
        }
        if (u_username == null || u_password == null) {
            throw new LoginException("Either no username or no password specified");
        }
        logger.info("Password from user and file : " + u_password + " :: " + f_password);
        if (u_password.equals(f_password)) {
            return true;
        }
        return false;

    }

    public boolean logout() throws LoginException {
        // TODO Auto-generated method stub
        return true;
    }

    private void attemptLogin() throws LoginException {
        Callback[] callbacks = new Callback[2];
        callbacks[0] = new NameCallback("u_username");
        callbacks[1] = new PasswordCallback("u_password", false);
        try {
            callbackHandler.handle(callbacks);
        } catch (IOException e) {
            logger.error("IOException", e);
        } catch (UnsupportedCallbackException e) {
            logger.error("UnsupportedCallbackException", e);
        }
        u_username = ((NameCallback) callbacks[0]).getName();
        user = new JMXPrincipal(u_username);
        char[] tmpPassword = ((PasswordCallback) callbacks[1]).getPassword();
        u_password = tmpPassword.toString();
        logger.info("UserName : " + u_username);
        logger.info("Password : " + u_password);
        System.arraycopy(tmpPassword, 0, u_password, 0, tmpPassword.length);
        ((PasswordCallback) callbacks[1]).clearPassword();
    }

    private void loadPasswordFile() throws IOException {
        FileInputStream fis = null;
        passwordFile = "c:\\abcd.jmx.enc.password.file";

        try {
            fis = new FileInputStream(passwordFile);
        } catch (SecurityException e) {
            logger.error("Security Exception", e);
        }
        BufferedInputStream bis = new BufferedInputStream(fis);
        userCredentials = new Properties();
        userCredentials.load(bis);
        bis.close();
        f_username = u_username;
        f_password = (String) userCredentials.get(f_username);
        logger.info("UserName before Decrypt : " + f_username);
        logger.info("Password from file before Decrypt : " + f_password);
        // decrypt the password from file and later compare it with user password from JConsole
        if (f_password != null) f_password = Cryptography.decrypt(f_password);
        logger.info("Password from file after Decrypt : " + f_password);
    }

}

When i use the following code and try to connect via JConsole nothing happens.
Map<String, String> env = new HashMap<String, String>();
env.put(ApplicationProperties.JMX_PWD_FILE_PROP, pwdFile);
env.put(ApplicationProperties.JMX_ACCESS_FILE_PROP, accFile);
env.put("jmx.remote.x.login.config", "com.splwg.ejb.service.management.ABCDJMXLoginModule");
connectorServer = JMXConnectorServerFactory.newJMXConnectorServer(jmxServiceURL, env, mBeanServer);

Any ideas on why this happens ? For sure, i am also not coming into the ABCDJMXLoginModule class - I have some print statements there and none of them get printed. Any sort of ideas and solutions are appreciated. I tried with the property "com.sun.management.jmxremote.login.config" too. I was expecting that mentioning the property in the environment and passing it to the JMXCOnnectorServer would do all the trick. 
Am i missing something ? 

A: env.put("jmx.remote.x.login.config", "com.splwg.ejb.service.management.ABCDJMXLoginModule");

jmx.remote.x.login.config property is meant to set a context name, not your LoginModule class name.
Instead or putting your LoginModule class name, you need to create your jaas module config file and refer to it via a system property, for example:
-Djava.security.auth.login.config=path/to/your/login/config/file.cfg
Sample config:
MyConfig {
  com.splwg.ejb.service.management.ABCDJMXLoginModule REQUIRED
    configKey1="config Value 1"
    configKey2="config Value 2"
}

In your java code:
env.put("jmx.remote.x.login.config", "MyConfig");

I would also remove
env.put(ApplicationProperties.JMX_PWD_FILE_PROP, pwdFile);

as I think it might cause that a standard jmx authentication mechanism is used instead of JAAS (but I am not sure about it).
With the above changes you should have jmx with jaas based authentication (with your custom login module) and authorization using accFile.

A: Just following this useful post to implement something similar. I had to add some semi colons to the MyConfig syntax
MyConfig {
    com.splwg.ejb.service.management.ABCDJMXLoginModule REQUIRED
    configKey1="config Value 1"
    configKey2="config Value 2";
};

Also following code from the original ABCDJMXLoginModule class:
u_password = tmpPassword.toString();

should be:
u_password = new String(tmpPassword);

