This tutorial explains the way how vanilla/CPP AI works, how to modify it, how to make custom AI for specific creature and how to workaround/fix known issues. What does CPP has to do with this? Community Patch greatly enhances vanilla AI, adds numerous fixes and new features. It's still same AI but works much better overally.
NWN is using so called talent engine for around 80% of the AI routines. Two engine hardcoded functions, GetCreatureTalentBest and GetCreatureTalentRandom will search all creature abilities and returns a talent that can be used with ActionUseTalentOnObject or ActionUseTalentAtLocation. Talents are categorized into 22 categories, as can be seen in categories.2da (see image #2). You can change each spell/feat AI category in feat.2da via CATEGORY column and spells.2da via Category column. Note however, that many feats have no category on purpose due to several issues/bugs so it's not recommended to mess with feats category. As for spells, you need to know that AI is able to cast spell on any target no matter the spell settings in 2da doesn't allow it, and that AI has no safety mechanisms to check whether such spell is used properly. So you need to be careful when changing spell AI category otherwise creatures might cast beneficial spells on PC.
The rest of the AI routines are rule-based checks, "if I can't see enemy use true seeing", "if I have invisibility and enemy won't see me, turn invisible", etc. These routines are supplement to the talent system to make AI smarter and more intelligent.
This is very basic explanation how AI works. Now to be more specific. First, each creature has several events which triggers AI scripts. You can modify these scripts in creature properties - Scripts tab (see image #3). NWN has basically only 2 AI sets, nw_c2_default* for any creature and nw_ch_ac* for henchmans, summons, familiars and animal companions. Some HotU creatures will have x2_def_* script set however this AI script set has no additional features except the ability to modify how many attacks per round will creature have, except this all it does is that it will execute nw_c2_default* scripts. I'm not a fan of x2_def_* script set, I think it's better to just use nw_c2_default* for better performance.
Anyway, each of these scripts might contain additional AI behavior such as what to do, if there are doors blocking creature movement (nw_c2_defaulte.nss) or when to switch attack target (nw_c2_default6.nss - OnDamaged event). This is the top layer of AI and if you want to modify this behavior you need to modify these scripts directly. These scripts then calls DetermineCombatRound() function (found in nw_i0_generic.nss include library) which will call the internal AI routines I was talking about above.
There are several steps that AI performs each time it's called.
call to chooseTactics() which will try to select talents from other categories.
class - each call, function will pick one of the three classes to start with (note: havily bugged - CPP fixes this), each class has different settings.
There are three action variables, Magic, Compassion and Offense. Each class has different base values, and Magic is increased by the character Hit Dice. To force creature to cast spells you will want Magic to be as high as possible, to prevent creature from casting spells you will want Magic to be equal 0. Compassion controlls how much is creature a "team player" and Offense controlls creature aggresivity, creatures with Offense <= 5 will try to flee from fight (basically used on commoners).
After this, AI will try to perform magic talents first (if Magic setting is high enough (50+) and if no such talents are found or Magic isn't high enough, it will try combat talents such as dragon breaths, sneaking, Ki damage and melee attack.
There is one last important thing and thats "talent filter" (bkTalentFilter in x0_inc_generic) which will do some intelligence/sanity check whether the spell/feat has sense to use (inflict spells used on undead target, mind affecting used on immune target, etc.) and will cancel the talent usage if the talent has no sense to use. This function is great place to fix many AI issues, however the function is not used correctly on many places (see this forum thread for details). CPP fixes this however and implements many new checks.
Don't forget, that to change anything in internal part of the AI, you need to recompile all scripts thats are calling DetermineCombatRound function to apply the changes! Or 70_ai_generic/70_ai_henchman if you are using CPP 1.72 beta where is AI externalized and can be modified without this.
Obviously, one can create his own creature set which will implement AI from scrach, or just change the external AI behavior. Some examples of this: Low AI set, Hit&Run AI.
To make your own AI is time intensive and requires to first understand how vanilla AI (or some of the custom AIs from vault - Jasperre's, TonyK) work.
But there is better way and thats the HotU custom AI system. HotU implemented an internal AI override script which can be set with string variable "X2_SPECIAL_COMBAT_AI_SCRIPT" on creature. There are 4 vanilla custom AI scripts, x2_ai_demo, x2_ai_shadow, x2_ai_behold and x2_ai_mflayer which you can use as a reference when working on your own custom script. This script can either completely rewrite the internal AI or just do something extra and then call back vanilla AI.
Personally I found this to be best as a special AI for specific creatures such as bosses or just creatures, where the vanilla AI doesn't perform as I want it to.
This way, you know what abilities your creatures has so you don't need to use talent system at all and you can code whole AI via rule-based checks and some randomization.
Note: I don't recomend to look on x2_ai_shadow, much, unless you use Community Patch where is this AI rewritten, as this AI code has several issues with action cancels and vanilla shadow often stucks and do nothing.
Magic 100, Compassion 50, Offense 50 - such creature will use all available buffs and protections before going offensive.
Now, I will try to give help with most commont AI issues.
My spellcasters doesn't cast spells.
First of all, make sure your creature doesn't wear armor/shield (if bard, sorcerer, wizard). Has appropriate ability score, appropriate number of class levels if the spells are setup on Spells list and not Special abilities and lastly if creature has Combat Casting feat.
Combat Casting is crucial, without this feat, spellcasters tends to go into melee combat if PC get too close to them. To make them cast even in this situation, you need to give them combat casting. Also, if you spawn creature too close to PC it will have same effect which can influence creatures like water elemental etc. who then won't use their drown ability. Either give them combat casting or override their Magic score to high value.
Also the problem might be in too high level - the talent system fails to work on creatures with caster levels above 40 or total lvl above 80. +-, basically it's bugged on creatures like this so try to lower the HD and increase HPs manually instead. If the reason for this high level is spell dmg, rather try to use spell caster level override feature in Community Patch.
If none of this helps, try to make your creature as a copy of another caster creature such as orc shaman and add spells/raise level manually. Creature wizard might add feats from 1.69 which will get creature stuck if you don't have CPP installed. By making copy you will avoid this.
My creature doesn't want to use throwing weapons.
This is a bug in vanilla AI, where the AI thinks there is no ammo in inventory, CPP fixes this.
My creature doesn't cast specific spell.
Clairaudience and clairvoyance, darkness, identify, knock, light, polymorph self, shapechange, silence, legend lore, find traps, continual flame, one with the land, camoflage, bloodfrenzy, inflict spells, amplify, greater sanctuary.
Some of these spells are problematic (darkness), some are actually used outside talent system in specific circumstances (henchman ai can cast knock and find traps), some are simply forgotten. Community patch added category to some of them but not all from balance reasons. Many builders doesn't care their creature is not using spells they gave her and keep it that way which then might cause balance issues when these creatures starts using strong talents they had no access to before.
Greater Restoration - is used as cure spell, it will never be used for healing in vanilla AI, unless you change category to 5. Even if you do so, the cure talent code will still be able to use GR since it checks for spells manually it actually avoids talent system.
Mass Heal, Circle of healing - has TALENT_CATEGORY_BENEFICIAL_HEALING_AREAEFFECT however this category is not looked for in the internal AI.
Community Patch doesn't change this for the balance reasons. If you want your creature to use these spells you can change AI category to 5.
Sorcerers and their different spellcasting.
Sorcerers are bit different as unlike wizards, they don't have fixed number of spells but rather known spells and the number of spells they can cast is calculated from sorcerer level and charisma score. What is problem is that vanilla AI tends to cast only few spells no matter creature knowns much more of them. The problem is that the GetCreatureTalentBest tends, for same category, always return the same spell. From this reason sorcerers often casts very stupidly spamming one/two spells over and over.
This can be however changed by builder, using integer variable "X2_SPELL_RANDOM" = 1 on creature will force to use GetCreatureTalentRandom instead of Best, thus resulting in big variety of spells, although it can often use spell of the lower level than best possible.
Note: Community Patch automatically enabled X2_SPELL_RANDOM for sorcerers and bards. So this is not needed, however if you wouldn't be satisfied with the randomized casting of your sorcerer with CPP installed, you can set the "X2_SPELL_RANDOM" to -1 which will restore the vanilla behavior.
There are several issues with Epic spells. It's partially spell engine issue and partially AI issue.
First, note, that almost every feat has the spell that is triggered when this feat is used. And you can set creature this spell in Special Ability tab with an advantage of setting caster level and even number of uses - want creature with 99 Greater Ruins? Just give her Greater Ruin via Special Ability tab and set 99 uses.
Second, feats are generally heavily bugged. Especially feats that has spell that needs to be used at location doesn't work at all due to the bug in ActionUseTalentAtLocation and will cause creature to stuck. This is why are summoning feats disabled. Also, the caster level calculation doesn't work properly and returns 0. This is why the Epic warding and Epic mage armor spells lasts only 1 round (with CPP) or do nothing entirely (without CPP).
So, to fix this remove the epic spell feats and add them as special ability spells. OR/AND use community patch caster level override feature which will allow you to bypass the maximum of 15caster lvl in toolset.