>>Heidi: ...for joining us.
I know folks will probably still be entering.
I'm Heidi Melz of James Bell Associates.
This is the first of two webinars that we're
presenting this month on the use of administrative
data in the evaluation of child welfare programs.
Today's presentation is on the mining of administrative
data to identify and understand at risk populations.
Just some housekeeping first.
All the phone lines are muted and we're going
to keep them muted during the presentation,
but we'd love to hear from you if you have
questions as we go along.
To submit a question, if you'll look on your
events center page in the upper right, there
is a chat icon.
If you click on that, it'll pop up the chat
feature on the right side of the page there.
You can type your question in that text box,
and be sure if you would to select presenter
where it says send to and then I'll receive
it.
I'll keep an eye on that box as we go along
and I'll bring questions up to our speaker
during the presentation.
We are recording the webinar and the recording
will be available on the James Bell Associates
website shortly.
So I'd like to go ahead and introduce you
to our speaker, Dr. Dana Weiner.
Dr. Weiner is a policy fellow at the Chapin
Hall Center for Children at University of
Chicago.
She provides analytic consultation and policy
guidance to child welfare jurisdictions around
the country, and at home in Illinois she served
for six years as senior policy advisor to
the director of the CSF there, and as the
evaluation liaison for the Permanency Innovations
Initiative, or as it's known, the PII Project.
And it's through PII that I met Dana.
The Administration for Children and Families
launched the PII project in 2010.
It was a sixth project demonstration to test
strategies to improve permanency among children
in foster care who have the most serious barriers
to permanency.
Illinois was one of those six sites, and you'll
be hearing today about how they used administrative
data to help them identify their target population
for the project.
So with that, turn it over to you, Dana.
>>Dana: Okay, great.
Thanks, Heidi.
So as Heidi said, much of what I'll be speaking
about is drawn from the experience with the
Illinois Permanency Innovations Initiative,
but I'll give some other examples as well.
So today what I'm hoping we'll do is talk
about some of the key elements for productive
data mining.
We'll talk through what I think are the key
steps in the approach to data mining to shed
light on the risk factors.
And then we'll talk about examples.
So I'll basically walk you through the steps
using the Illinois PII example, but I'll also
probably refer to the work that we're doing
elsewhere in Illinois as well as in New York
City.
And then we'll talk a little bit about some
of the considerations and risks that are entailed
in using administrative data and how we might
face guard against those.
Just to refer also to the OPRE framework for
research and evaluation, I would characterize
this type of research data mining for target
population refinement, I would characterize
that as an exploratory descriptive study.
So if you're familiar with that framework,
it talks about different types of research,
and I was thinking about, "Well, where does
this fit in?"
And I think even as an exploratory descriptive
study, although we can talk about that, you
know, as the conversation progresses,
Okay, so starting out just with some key elements
for what I think are important to productive
data mining.
And we'll talk about how these are laid out
in Illinois PII.
First, a data analytic work group that includes
broad representation, so data technicians,
data analysts who can do statistical analysis,
researchers who work with the data and practitioners
and other people who are familiar with how
the data are actually generated.
We'll talk about how we formed that group
and how we made use some of that group.
The second thing is access to stakeholders
for assistance with variable definition and
interpretation.
So what that means is, so we're using administrative
data, it's not data that we collected with
the survey, it's data that we were lucky enough
to inherit, but we don't always have the deepest
understanding of what it means or what people
mean when they put a certain rating in a certain
field or when they use or don't use a particular
variable.
So it's really important to have access to
the people who are generating the data, and
those are oftentimes people on the front line.
They may be internal quality assurance folks
at a child welfare agency, but having access
to those people to be able to say, ''You know,
this field, I'm not sure if it means what
I think it means,'' is really important.
And I put variable definition on there also.
As we get into talking about the same steps
that I've kind of left that out of this discussion,
but in the case of Illinois PII, as Heidi
said, we were looking to understand those
kids at risk of long-term foster care.
That is kids who we thought were going to
get stuck in foster care and not exit to permanency.
But even that outcome getting, stuck in foster
care, required a lot of discussion in conversation
to figure out what was the right way to operationalize
that variable.
And I think it's important to have others
weigh in on that also.
Ultimately, I'll talk to you about what our
definition was, but I think it was helpful
to have a lot of different representation
in the conversation.
And the third one may seem obvious.
You can't do this data mining work if you
don't have access to data and information.
And that means both that you need the data
sharing agreements in place to be able to
access the data and there has to be pathways
for transferring the data and linking it possibly.
But it also means, and this is the fourth
bullet, that you have enough data to do the
kind of analysis that you wanna do.
So data that are limited in scope or in duration
or in the population that they pertain to
will limit the validity and the generalizability
of the findings.
So, you know, we oftentimes have in the rest
of my job, we have conversations with jurisdictions
that want to engage in predictive analytics
or other analyses, but we first have to go
through a process of figuring out who actually
have the data you need to do that.
So particularly when you're trying to predict
things and understand risk, we'll get further
into this, you need enough data over enough
years to have observed at the outcome you're
concerned about in many, many cases so that
you can not only determine what is correlated
with variability in that outcome, but so that
you can then test those ideas on a test sample.
We can get into more detail about this, but
you really need a lot of data and it has to
be comprehensive.
Okay.
So, in my initial version of this deck, I
was going to show you these one at a time,
but I have no animation today.
So you're seeing them all at once and I'll
just walk you through what I think are the
steps for this data mining process.
And remember, I have not included the variable
definition piece of this, although we can
talk about that.
Okay, so the first step in the process, at
least as it went in Illinois, and I think
that was a good methodical, lengthy process
that produced some comprehensive and meaningful
results.
The first step was to look at everything we
already know.
And so, I've put that here as literature review,
and it did involve going to the literature,
but it also involved assembling this group
of researchers.
In Illinois, we're very fortunate to have
many university partners who've spent many
years using the data to help the child welfare
system understand their challenges.
So it involved bringing all of those people
to the table to talk about and consolidate
everything that we already knew about this
population, these kids who were at greatest
risk for getting stuck in foster care.
That was the first step.
The second step that we went through was then
to begin to look at those variables that seemed
like they were going to be important, or even
any of the variables.
In some cases we took...like we used the CANS,
the child and adolescent needs and strengths,
in Illinois, and we took the entire CANS,
and we looked at the correlation of every
item with the outcome we were interested in,
which in this case was being in foster care
for three years after entering.
We looked at each of those variables separately,
so I'm calling that bivariate analysis, and
I'll show you pictures of it in a few slides.
And we also just looked to describe the population
we were interested in, in general terms.
And those activities under...looking back
at the literature and everything we have learned
and doing these univariate and bivariate analyses
were really meant to generate hypotheses before
we moved into the process of more multi-variate
analysis to try to test those hypotheses.
So the next step was to do that.
And again, I'm gonna give you...I'm gonna
do the detailed Illinois walkthrough, but
I'm kind of giving you the overview of the
steps.
The next step was to engage in this multi-variate
modeling to try to understand, "Well, what
are the predictors that, together in varying
amounts, predict risk of being in foster care
for more than three years?"
And, you know, we were able to do that to
varying degrees.
One of the things that became really clear
to us was that, you know, there are a number
of approaches to doing this predictive modeling.
Continuous linear modeling, we collaborated
with some folks in New York City who used
machine learning techniques, but it all depends
on having data, enough data on the relevant
thing.
So for instance, in Illinois, in one of our
modeling exercises, we've found that we could
come up with pretty good models for the risks
in Cook County, which is the county, you know,
surrounding Chicago, but outside of Cook County
where there are many separate court calendar
than many judges who make decisions on a variety
of different...some basis, it was very hard
to come up with one predictive model that
could predict risk across all those areas.
So, you know, it was a learning process for
us to confront some of those limitations.
And there's a lot of discussion in the field,
this is probably a whole other webinar to
talk about modeling and predictive analytics
using administrative data.
We certainly spent a lot of time thinking
about that with our partners in New York and
Illinois and looking at different approaches
to doing that.
But the idea with that multi-variate modeling
is that you're testing some of your hypotheses
that you've generated about what are the risks,
because these models yield then a set of predictors
that then presumably could be acted upon to
reduce risk, and that was the idea in Permanency
Innovations Initiative.
Heidi, I just wanna pause for a second and
see if you have any burning questions that
have come in because I know I talk really
fast and I don't wanna lose anyone.
>>Heidi: No, none yet.
>>Dana: Okay, then I'll keep going.
Okay, so then once we had a predictive model,
we shifted back to descriptive statistics,
meaning that once we use our predictive model
to say these kids, it's the kids that entered
after age nine and have had trauma symptoms
or whatever the constellation of factors are,
once we defined the set, then we wanted to
go back and learn everything we could about
that population in part so that we could identify
the interventions that would ameliorate the
risk.
Because the models may tell us what the predictors
are, what are the things that correlate with
the negative outcome, but they don't tell
us why those things are predictors and they
certainly don't tell us how to reduce the
risk.
So to do that, we engaged in a number of other
activities.
We, of course, did descriptive analysis, and
that was just to understand, well, what is
the average age and family constellation and
what are the permanency goals and all those
kinds of things to get a better look at what
that population was struggling with.
We also did something called latent class
analysis, which I'll talk about more specifically
in a little while, which was an attempt to
understand any heterogeneity.
Because every time we come up with a target
population, we kind of make this assumption,
okay, because they're similar in their level
of risk is one group, but a latent class analysis
can help reveal whether there maybe are three
or four or five underlying subgroups in our
target population.
So the other activities that we did to try
to understand the mechanisms by which the
factors produced risks, where we did case
record review to select cases that looked
from our data like they would be at high risk
to try to understand what were the case dynamics
there.
And we did focus groups and interviews with
caseworkers and other people in the field
to try to understand why are these factors,
how is it that these factors are generating
risks.
Now the conversation that took place, you
know, because in the PII project we had a
lot of technical assistance.
And the conversation that took place over
and over again with those technical assistance
was, ''Well, we see that you have the target
population and the risk factors, but what
are the barriers to permanency?''
Now we were in the midst of a shift to a trauma
informed system, and trauma was really the
thing that we wanted to focus on and we did
observe the trauma is a risk factor, but we
really took for granted that everybody would
just understand why trauma was a problem for
permanency.
And we really did have to articulate that
for people.
It was not clear, it we certainly was not
clear enough to point us toward a particular
intervention.
So this process of doing qualitative research
and trying to understand the heterogeneity
was really so that we could get enough of
an understanding of the barriers to permanency,
the mechanism of risk, so that we could select
an appropriate intervention.
And then once we did all of that, we went
back to the administrative data to try to
understand, okay, so we've selected our intervention,
now we need to know how many therapists we
need and where should we put them, how many
kids are likely to come through and meet the
eligibility criteria, how do we translate
the research findings about risks into a set
of criteria that you could apply to an individual
case and say, ''Yes, you should be in this
intervention.
You are at risk.''
And then how do we make accurate estimates
about the capacity that we need?
So I'll just pause there, because that's a
lot of information I just gave you.
Any questions about that?
I guess I'm posing that to Heidi.
Any questions, Heidi?
>>Heidi: Yep, there's one question.
Do we use all predictors from bivariate to
multi-variate or only the significantly associated
ones?
>>Dana: Interesting question.
Now that speaks to a larger philosophical
kind of issue.
Depending on the approach that you use to
predictive modeling, you either...straight
up predictive analytics, like the kind that
the machine learners do, which comes from
more of a corporate environment, takes all
of the data, every single thing you have and
throws it in the pot and looks to see what
rises to the surface.
That means anything, like, is the kid right
handed or do they have blue eyes?
I mean, that's ridiculous.
Obviously you're not gonna use that, but I
mean things that might not even be actionable.
And the idea is that anything that adds to
the precision of the model is fair game.
What we engaged in and what we continue to
engage in, I think would probably be more
accurately termed explanatory analytics because
it is not a theoretical.
It is actually driven by our understanding
of case dynamics, our knowledge that we've
already accumulated in the literature, and
the things that we've already seen empirically
matter.
Now that said, thinking back to bivariate
analyses, what we ended up with there was
a set of odds ratios.
So for everything we said, ''Okay, well, if
you have this year symptom, it makes you 30%
more likely to be in long-term foster care,
and if you this symptom, it makes you 300%
more likely.''
There was a lot of variability there and we
did actually...we ultimately distilled it
down to some dimension.
So we took all of the externalizing symptoms,
like the acting out symptoms, and we said
yes/no on that dimension.
We took all of the internalizing symptoms
and we said yes/no on that dimension.
You know, we tried to distill it a little
bit more in the PII exercise, but I think
there was...I guess my answer is, it depends
what approach you're using it.
There's a valid argument to do it either way.
Are there questions, Heidi?
>>Heidi: Nope, not at the moment.
>>Dana: Okay.
So one other caution, I would say, you know,
I have case record reviews on here and that
was important for illuminating how it happens
that these cases that had the risk factors
ended up in long-term foster care.
But I would just caution, and maybe you all
know this, but those are good for exploring
mechanisms and good for generating new hypotheses
to test but not good for testing hypotheses.
So, you know, I've definitely seen examples
in the field where, in wanting to establish
a pattern, people go to case reviews and I
would caution against that because, you know,
it's difficult obviously from a small sample,
but I think it can be tremendously valuable
for generating hypotheses and for getting
ideas about the mechanisms.
Okay, we move on to the next slide.
Okay.
So, you know, and I was thinking about what
are the key elements, what was in the back
of my mind was, well, this is to protect against
the risk of this.
And so, I figured I would just make that explicit.
So the reason we want to include all of those
internal stakeholders and front line practitioners
is so we don't misunderstand a variable that's
in the administrative data and think it means
something that isn't what it means because
that is a risk.
And I think sometimes we run the administrative
data and we think we have this really interesting
finding and then we'd go back to the internal
stakeholders and they say, ''Oh no, people
just use that variable because they know they're
gonna get this service if they use it, and
so it's routinely not used correctly in the
field.
And so, that really means something else.''
So it's really important to be able to have
access to those people.
The other potential risk is that you can make
incorrect assumptions about the mechanisms
by which risk leads to outcome.
And so, you know, I mean, one example in the
data mining that we did with our partners,
you know, there were a lot of partner researchers
in this.
One of the findings was that no termination
of parental rights by two years, these kids
are at greater risk of long-term foster care.
And you could think about why that might be,
but that's where kind of not only the stakeholder
conversations, but qualitative research case
record reviews and focus groups can really
come in handy in understanding those mechanisms.
The third would be incorrect interpretation
of the findings, thinking things mean something
that is different from what they mean.
And, you know, a lot of these examples are
a similar kind of thing where as the researcher
looks a certain way, for instance, in New
York City, in our predictive analytics exercise,
we were trying to understand the predictors
of families coming into frequent contact with
preventive child welfare services.
And one of the predictors that came up was
more severe allegations at the outset, at
the investigation, seemed to predict fewer
contacts with preventive services.
And at first that seemed counterintuitive,
but when we actually talk to people, well,
I'm worth of your allegation may result in
a child removal, which then would create less
opportunity for the family to come into contact
with preventive services.
But without having this ongoing dialogue,
it's really hard to make sure that you're
accurately interpreting the findings.
Now I don't have an answer to the inadequate
data problem.
I mean, there's a risk that if you only have
data, you know, we've been in places where
they said, ''Well, we only fill out this tool
on the cases that we think need this service.''
Well, right there, then you have a systemic
bias in who you have the data on.
So it's going to influence what you find and
it won't be generalizable to the rest of the
population.
I just paused there as I change the slide.
Heidi, feel free to interrupt me if you have
questions.
>>Heidi: Yeah, I will.
Nothing here, though.
>>Dana: Okay, so I'm gonna move into talking
about the Illinois PII example because I think
it makes the point about a lot of these steps
and considerations.
So as you've heard in Illinois, in PII, we
were interested in identifying the population
at greatest risk of long-term foster care,
and then in describing that population so
that we could identify the barriers to permanency.
We also then analyze the heterogeneity and
the target population to identify characteristics
in subgroups that might be particularly amenable
to intervention.
So this was really...preparing for this webinar
was really a walk down memory lane.
I went through like six years worth of documents
and some things that brought back all kinds
of memories.
But I just wanted to show you because I mentioned
that we had this data analytic work group,
and I could have done it put points and bullets,
but I have to take a snapshot of one of our
agendas from one of those data analytic work
group meetings because I just wanted you to
see how we engaged all of these other researchers
at the table.
In some ways, there were a lot of people who
had their hands on different parts of the
elephant.
So there were people who have studied psychotropic
medication, utilization, and people who had
studied subsidized guardianship, and everybody
kind of had a different view on, well, what
are the problems that may lead to long-term
foster care?
So we got everyone around the table, we defined
our goal, which I just described to you, we
talked about the roles of everyone at the
table, and then we talked about what we were
gonna do.
This was a group that was gonna look at alternative
hypotheses and make sure that we left no stone
unturned in understanding the barriers to
long-term...to permanency, and this is gonna
be a group like a sounding board that we were
gonna use to help interpret the findings because
these were people who had had their hands
in all different parts of the Illinois data.
And then we went through this process of summarizing
for the group what do we already know.
And this was from one of our meetings.
We talked about, okay, we know there's a relationship
between trauma, strengths, risk and placement
stability.
We went over a document that summarized the
research because we actually wrote up a summary
of all of the previous research.
We talked about the bi-variate analysis that
we did, which I'll show you in a minute.
And then we talked about the model building
and person-centered analysis.
So it might seem silly that I put that in,
but I just want to see a legitimate proof
that we engage this group of researchers in
that way.
Okay, so the next two slides are pictures
of some of the bivariate analyses we did,
although when I looked at this and I thought,
"Maybe that's actually trivariate."
This was not a modeling exercise, but it was
an attempt to understand how trauma and age
related to permanency but not in a modeling
way, just in look at the difference in rates.
So I'll just quickly walk you through this.
On the left-hand side, you see, we broke up
the population into three age groups, and
this was just kids entered during one fiscal
year and this was their age of entry.
So there was the young kids, the school aged
kids, and the adolescents.
Then we looked at the percent that had actionable
trauma, and it was highest among the adolescence,
as you can see.
And then we looked among the kids that exited
to permanency, how did the rate of trauma
exposure and actionable trauma problems differ
between the group that went to permanent homes
and the group that didn't go to permanent
homes?
And what you can see when you look at this
is the differential.
The biggest impact of trauma on permanency
seem to be among these 9 to 12-year olds.
The difference there was the most dramatic.
It was 48% of the kids who exited to permanency
did not have actionable trauma and 32% did
have actionable trauma.
So I'm sorry, I'm sorry.
The kids who had no actionable trauma, 48%
of them exited to permanency and the kids
who had actionable trauma, 32% of them exited
to permanency.
And so, the relative rate of permanence was
0.67, it was the lowest among that group of
9 to 12-year olds.
So that was...while we were building these
hypotheses, this was one of the things that
was really driving us, was this understanding
that the kids in this age group who had trauma
exposure, it seems to have the biggest effect
on their ability to reach permanency.
So then, you know, anytime you have a lot
of researchers at the table, it takes a lot
of convincing.
So we showed these graphics a million different
ways to try to get...buy in from everyone
to see that it was these kids entering between
9 and 12 years old that may be at the greatest
risk for getting stuck.
There were lots of arguments made about other
age groups, but this is one of the graphics
that we thought really made that point using
the administrative data.
What you're looking at here is those kids
at the second anniversary of entering care,
so remember we were saying age of entry in
the last slide, 9 to 12, so this would be
like the 11 to 14-year olds 2 years later.
At the two-year mark, permanency within the
next two years.
So the red bars are kids who exited to adoption
within two years after their second anniversary
and the blue part of the bar are kids who
exited to reunification within two years after
their second anniversary of entering care.
And you can kind of see where it begins to
drop off, around 12 at the second anniversary.
So this provided more support for this idea,
that if we were gonna intervene at the two-year
mark, which is what we ultimately chose to
do, that we should be focusing on these kids
who were 9 to 12 when they entered or who
are ultimately 11 to14 by the second anniversary.
Again, I just said a lot of potentially confusing
stuff.
Any questions about any of that?
>>Heidi: No more questions at the moment.
>>Dana: Okay.
So then we moved into multi-variate modeling
and we ended up with fixed types of predictors
of the risk of long-term foster care.
One was age, which stayed significant in all
of our models over nine at entry, one was
that parental rights issue, if there had been
no termination by two years, one was the Cook
County kids were much more likely to stay
longer and there are lots of hypotheses out
there for why that is, one was placement type.
So if the youth had ever been placed in a
congregate care facility, that stands for
institution group home, that IGH code, they
had greater likelihood of being in foster
care long-term.
If they had placement instability, and we'll
talk about how we define that, and if they
had mental health or trauma symptoms or risky
behaviors.
So, yeah.
>>Heidi: Yeah, I just stepped in, there was
a question about how actionable trauma was
measured.
Can you just...
>>Dana: Yeah.
Actionable trauma, actually everything in
that list bullet was measured using the CANS,
and there was a great concern among people.
So I mentioned we had this multi-disciplinary
group.
There was a great concern among clinical folks
in our group that there were plenty of other
symptoms that were not labeled trauma symptoms,
but that were often secondary to trauma and
indistinguishable to untrained practitioners
from trauma symptoms.
So things like asset regulation, attention
problems, internalizing problems like anxiety
and depression, even risk behaviors.
What we did is we ultimately came up with
a list of 43 CANS items, and in our system
in Illinois, the CANS was used very pervasively
at this time.
So it was used in every initial case as they
came in and then it was used periodically
by various programs.
So we have pretty good CANS data on these
cases.
And so, we use this list of 43 CANS items
and those were the items ultimately that we
used for eligibility for this, but the CANS
was the source.
And the CANS has trauma experiences and then
trauma symptoms, and here this bullet pertains
to the symptoms because there's lots of trauma
exposure among youth in foster care that aren't
always accompanied by symptoms.
Other questions, Heidi?
>>Heidi: Nope, not at the moment.
>>Dana: Okay.
So ultimately have that list.
We chose to focus on these three in defining
the target populations.
And there were a variety of reasons for that,
some of them will talk about them, some had
to do with sample size, some had to considerations
around the intervention.
But I'm gonna get into more detail about how
we picked these three.
But ultimately these were the three: age over
nine at entry placement instability, and mental
health trauma or risks symptoms.
The reason age is in yellow and the other
two are in green is because obviously age
is not actionable, but placement in stability
and the mental health and trauma symptoms
are actionable.
And those were things that we were aiming
to address with our intervention.
>>Heidi: And then, Dana, can you just repeat
what IGH is for those who visited?
>>Dana: Oh, yeah, sorry,.
That is institutional group home.
That just means congregate care, ever having
been placed in congregate care.
And we see that's the pattern that's pretty
pervasive.
We see that all over the place, that spending
time in congregate care lengthens the overall
length of time until permanency.
And even one could argue the likelihood of
achieving permanency.
Okay.
So in a perfect world, we would take our data
analytic results and we would translate that
into our target population and that would
be that.
But we don't live in a perfect world and,
you know, certainly the world we're working
in is messy in there.
A lot of other moving parts, and you can never
like kind of press pause on your system and
say, ''Okay, we're going to do the pilot now
so everything else just take a rest while
we do this.''
So you have a lot to consider.
We had to consider the fact that there were
two other federal projects going on and the
project officers were all concerned that we
might step on each other's toes, meaning that
we might inadvertently include kids from one
of these other projects in our sample or they
might include kids from ours and theirs, and
then we would not be able to distinguish when
evaluating these pilots whether the effect
was due to the intervention that was the focus
of the evaluation or whether it was because
there was some additive effect of being involved
in both.So it's really important to the federal
project officers and to us that we work around
each other neatly.
So we had this Kin Connections Program that
was intervening at entry to foster care for
kids ages 6 to 13 only in Cook County.
And then we had this Adult Connections Project
where they were intervening with these exiting
who are 17 and older.
So what we did was we got together, myself
as the evaluation liaison on PII and then
Sonia Leathers who is working on the Kid Connection
Project, and Scott Leon who is working on
the Adult Connections Project.
And we started meeting regularly to try to
figure out, ''Okay, so this is the PII sample,
how many kids is it likely to...how many kids
are at risk of being involved in both?"
And we just kept working on the eligibility
criteria until we really made it so that it
was extremely unlikely that any youth would
be asked to participate in both of those studies.
We also had to consider the sample size.
And in a few slides I'll talk to you about
our power analysis.
But because of sample size considerations,
we couldn't restrict to just a certain type
of living arrangement.
So we couldn't just say, ''We're only gonna
work with relative foster parents or non-real
to foster parents.''
We had to consider multiple different types
of placements, multiple regions, and we didn't
restrict based on the parental rights status,
all of those were sample size considerations.
And then from an implementation standpoint,
once we selected our interventions, we excluded
the larger congregate care settings because
they had established treatment regimens that
were often, we hope, using evidence-based
treatment approaches and we didn't think that
we could implement another approach.
We thought it would do more harm than good
to implement another approach in addition
to what they were doing.
So all of those considerations in mind led
to this target population definitions.
So the first bullet explains that thinking
about the age risk factor and the federal
project overlap together, we settled on including
youth ages 11 to 16 at the second anniversary
of entering foster care.
Considering the symptom risks, the placement
stability risk, and the sample size considerations
led us to settle on including youth with either
1 placement change and/or 1 symptom up from
the list of 43 symptoms at the second anniversary
of entry.
Now, I've just summed up in this slide like
months and months of work and many, many documents
in like two bullets.
Each of these, the second bullet in particular,
we generated multiple analyses to try to understand
well, what if we do, and versus or, and what
if we require one move versus two versus three
because we were just trying to keep recalibrating
using the administrative data, looking, you
know, retrospectively to make sure that we
would have enough kids in our sample based
on the rule that we made.
Everyone still with me?
>>Heidi: One more question.
Did you have to use stratified random sampling
method, and this is a related to multi-variate
analyses.
>>Dana: It sounds like it's about the evaluation
itself.
We used random assignment.
We were aiming to have every eligible youth
assigned to either the intervention or the
control group.
We did not use stratified random sampling
because we were really trying to meet...we
were aiming for, I think, 410, 420 kids in
both intervention and control, and therefore
every youth that was eligible was randomized
to either intervention or control.
So we didn't use a stratified sampling design.
If the question is pertaining to the actual
analyses, when we get...
>>Heidi: Sorry, there were two different questions.
Yep.
>>Dana: Okay.
Wait, is the question still about the random...?
>>Heidi: A separate question about the method
of the multivariate analysis.
>>Dana: Oh, I have to go...
So Andy Zinn was the analyst who has then
at Chapin Hall.
I was at Northwestern.
I have to go back and see what he used.
We've used a number of different like software
packages and techniques to do this.
People have used the HLM and other ways, but
I would have to go back and look and see exactly
what Andy did on to build the models, which
I can do.
Like I said, I've thousands of documents.
Okay, so the next thing that we did...
Heidi, should I keep going?
>>Heidi: Yeah, go ahead.
>>Dana: Okay, all right.
The next thing that we did was we wanted to
understand heterogeneity.
So we did this latent class analysis and that
was to identify subgroups.
Now the reason we did this was because we
were focused on a trauma intervention based
on what we learned from the predictive analytics.
The trauma interventions, as you may know,
some of them are focused really on a discreet
single trauma like TFCBT and a trauma narrative
around something that happened.
Some of them were focused on helping kids
recover from complex trauma, which we defined
as more than one type from this list, physical,
sexual, emotional abuse, neglect, or witnessing
family violence.
And some were more kind of broadly focused.
So we wanted to understand, well, of these
kids in this sample that we've now identified,
do we actually have meaningful subgroups?
And so, we measured symptoms in four clusters,
trauma, behavioral, emotional, and internalizing,
and we ran a latent class analysis using a
sample of a little over 1000 kids and we found
3 clusters, the quarter of the kids in the
sam...
Yeah.
>>Heidi: So there's another question about
this term actionable trauma and whether it's
a technical definition or whether it's more
general term being used to refer to a collection
of traits.
>>Dana: Oh, good question.
Okay, so that's kind of CANS language, so
actionable trauma or actionable anything,
the way that CANS rated is in terms of the
action that needs to be taken.
So zero means you don't need to do anything,
a one means watchful, waiting kind of monitor
this, but it's not an actionable problem yet,
but two means intervention is needed, and
the three names immediate and intensive intervention
is needed.
When we're using this for research purposes
or analytic purposes, we tend to draw the
line between a one and a two because a zero
and a one, you don't need to do anything right
now, and a two and a three, you need to do
something.
So that's what we mean when we say actionable.
It means there was something that happened
that actually requires an intervention, kind
of the threshold.
The CANS is not a diagnostic tool, but it's
a threshold for action.
Good question.
>>Heidi: Thank you.
>>Dana: Okay.
So anyway, we ended up with this three-cluster
solution.
About a quarter of the kids met the profile
for complex trauma, 95% of them met that criterion
and they had high rates of symptoms in all
the groups.
The majority of the kids in the sample had
less symptom complexity, only less than half
of them met the complex trauma criteria and
they have lower rates of symptoms and less
co-morbidity.
But they all had, you know, high rates of
exposure to some kind of trauma.
But then the third cluster of 15% really looked
like a behaviorally disordered group that
had higher rate of detention involvement.
They all had behavioral dysregulation issues,
high rates of affect dysregulation.
They were mostly male.
So it gave us a sense of like, ''Okay, we
have these three groups, we should pick an
intervention that's going to apply to at least
the majority of them.''
And so, then we had some numbers to apply
to the population and we made some more decisions.
So we thought, well, if we're doing a complex
trauma intervention, we might have 60% of
a sample meet the criteria.
If we do a targeted trauma intervention, all
the youth with symptoms and trauma experiences
other than neglect would be appropriate.
And so, this got us closer to estimates.
I'm realizing that I don't have a ton of time
left, so I'm gonna cruise through some of
these.
Heidi, do you wanna tell me what time do you
want me to stop for general questions?
>>Heidi: No, I think you can just keep going.
You've certainly got another 15 minutes and
I think you're doing fine.
>>Dana: Okay, great.
Okay.
So then I mentioned we also did a power analysis.
In order to do the power analysis, we couldn't
do that until after we selected our intervention.
The point of the power analysis, as you probably
all know, is to determine the sample size
necessary to detect the true difference, if
it exists, between the intervention and the
control group with sufficient power.
And it depends on the effect size that is
expected for the intervention.
So you need to know, I mean in our case we
were choosing an intervention that has some
evidence based, so we had to go to those developers
and say, ''What should we expect?''
Now they really couldn't tell us definitively
what we should expect in terms of an effect
size on permanency outcomes, but they could
tell us, ''Well, when we tested this on effect
regulation, we tend to see, you know, a 0.5
effect size.''
When we talk to them and extrapolated what
we might find on permanency outcomes, we estimated
that we should try to detect a pretty small
effect of 0.3.
And we wanted power set at 0.8 which is, you
know, standard good power.
And that led us to say that we needed about
a 190 per group.
Ultimately, we settled on a target of 420
kids that will be randomized to intervention
and control.
Okay.
So then I kinda think this gets into the fun
part.
I mean all of this was fun.
But then once we know our intervention and
we know what our target population is and
we've identified what they need, then we go
back to the administrative data to try to
help implement this thing to say, ''Okay,
where are those kids and how many of them
are there?
How many therapists do we need and where should
we put them?''
Answering those questions required more analytics,
but also some geospatial analysis to look
at where on a map where the kids who met the
eligibility criteria, which, of course, we
could do because it came from the historical
data.
So we looked at the variation in distribution
by region, by permanency goal, by placement
type and agency, and to start to build an
understanding of what the implementation would
take, we used two cohorts.
We used the historical cohort that would be
big enough that we could, you know, deduce
from the proportions there what the population
was likely to look like, and then we took
what we call the startup sample.
Meaning if we started this study today and
we took kids who met criteria over the last
four months, how many would there be in different
region?
So I'll show you just a little...some examples
of what that looked like.
Here you can see the historical sample has
647 kids.
The current sample which was like a four-month
slice, has a 101 kids.
And this was just a look at what kinds of
places they were placed in because we wanted
to know, are we gonna be dealing mostly with
relatives, how many kids are in group homes,
if we include those, and we learned that the
decision to include group homes was not that
consequential because it was really only like
3.4% of kids that met criteria would be in
group homes.
Remember we had already excluded larger congregate
care facilities.
This was a look at the permanency goals for
the startup sample, that four-month sample
because we wanted to know we were only going
to include biological parents in the intervention
if they were still working on reunification.
So we wanted to know how many bio parents
are we likely then to have this, let us know
that.
You can see how like we're still massaging
the administrative data to give us a sense
now of a much different thing, which is what
are we likely to need for capacity for the
intervention.
Okay, so this might be too small for you to
read the region names, but you don't really
need to read them.
The point here is that each colored slice
of the beautiful rainbowy bars is a different
region, sub-region in Illinois.
And this was an effort to say if we slice
it up by quarter, because again, we're staffing
this up with therapists that we're training
in an EDP, we need to think about their caseload
sizes.
So in each region, how many are we likely
to see in each half year that are gonna come
eligible?
So this gave us a sense of, okay, so in Peoria
we're likely to have roughly 40 kids coming
through every 6 months, with 6 months was
roughly the length of the intervention.
So we probably need two therapists.
There may be three therapists there, whereas
in a smaller one, whereas in Rockford, we
may only need one therapist, but this is really
helpful to the group that was planning the
implementation.
We'll just pause there.
There's any other questions about that?
>>Heidi: One question came up that relates
to a more general question about measuring
mental health symptoms and well-being using
administrative data when some doesn't use
a standard measure like the CANS.
>>Dana: Oh, that is such a good question,
and it's something we're dealing with in multiple
places.
You know, so most systems have a like tools
that are used as part of investigation that
identify LEVs, things like parental substance
abuse or developmental disabilities among
kids.
I mean in some places we've had to dig really
deep to try to find anything that would give
more nuance understanding of the groups.
I mean, whoever asked that question is not
alone.
Many systems don't have a universal assessment.
It gives enough information particularly about
trauma.
But where we can't find it, we've looked and
engaged stakeholders and thinking about where
would it be captured?
You know, in some cases the answer is in linking
data with other systems that do capture the
information, although sometimes linking data
can be far more challenging than just sticking
with the data of one system.
So I don't have a great answer except to say
that we have used some of those other things.
I mean we try not to use like allegation or
anything as a proxy for that, but we have
used kind of like the safety and risk factors
present in investigation because some of those
do signify clinical needs.
Anything else, Heidi?
>>Heidi: Nope, that's it for now.
>>Dana: Okay.
So I'll just say a couple words about these
other examples that we're working on where
we're using administrative data.
Wait, hold on.
Did I skip one?
No, I did not.
Okay.
So in New York City we're working on predictive
models to identify risk factors for frequent
involvement.
I mentioned that for trying to understand
which families that come into contact with
preventive services are likely to have four,
five, six hotline calls?
And could we...if we could recognize them
by constellation of factors, present at the
first time we seen them, because we may offer
more intensive services or case consultation,
we're also using that to try to understand
which providers are serving more challenging
populations of kids, both applications of
administrative data to improve kind of the
precision and efficiency and effectiveness
of practice.
And certainly in Illinois, there are many
other examples of these, of predictive models.
The most recent one is probably our work to
identify which kids are at risk for high end
congregate care placements.
And that was really a full, like another practice
like this one that I've described, where it
started with...it started with analyses, and
then it led to a set of eligibility criteria
for a pilot of a therapeutic foster care as
a diversion from congregate care.
But we used all of these strategies I'm talking
about, both the risk factors translated into
eligibility criteria as well as the estimate
by region.
All of these strategies we used to inform
that pilot.
And it actually...we helped the department
when they were writing the RFP for the therapeutic
foster care program, we helped them use the
research that had been done to inform what
they should ask for in the RFP.
That was really kind of a good example of
a self-contained analysis.
Oh, I just got a question directed to me.
Wait, it's fading.
How do I look look at it?
Oh, no.
Heidi, did you see that question?
It just came up in the corner of my screen
from, I believe, Jeremy.
>>Heidi: No, I did not get it.
>>Dana: Yeah, I see it.
I've got it, it's okay.
I found it.
It says, "You mentioned the qualitative data
after multi-variate analyses.
In these examples, how did you integrate the
qualitative data with the administrative records?"
Okay, so in the New York example, in both
of these cases, we had tremendous stakeholder
involvement.
So where we sat down with these folks.
And actually in New York I came to the table
after they already done the work to operationalize
the outcome of frequent involvement, which
involved some qualitative research as part
of that process.
But there also has been along the way identifying
particular cases to make sure.
So I wanna emphasize one other thing that
I said, which is that when we do these predictive
analytics, we divide the data we have into
the development sample and the test sample.
So we're developing the model using only about
half to two-thirds in the data and then we
take those models and we apply them to the
other portion of the data to see how good
they are at distinguishing cases that experienced
the outcome versus don't experiences.
And that's really key to understanding, you
know, it's like we're evaluating our own analytic
work to understand whether the models are
actually doing a good job at predicting.
But part of that process can also be to pull
actual cases which has happened over the course
of that New York work to say, ''Okay, we're
gonna actually now post some of the cases
that are confirmed to be at risk and look
at them to see how did that play out.''
In Illinois, there has been a very long standing
and kind of far reaching effort to understand
the barriers to exiting congregate care.
So I think the qualitative analysis was probably
less directly related to predicting ultimate
placement in congregate care and more trying
to understand the barriers to discharge.
There was a case review of about, well, I
can't remember, I feel like they did a review
of something like 150 cases that had been
in congregate care longer than was clinically
needed or beyond some particular threshold
established by the department and they literally
reviewed each of those cases to try to understand
what were the barriers.
And that information also went into informing
the Therapeutic Foster Care RFP because they
wanted to know like who are the kids who are
getting stuck in congregate care so that we
know what the programs need to offer.
Like who will they be serving?
Jeremy, hopefully that answers your question.
So nice that I got a direct question, like,
so lonely when you can't hear anyone posing
questions.
So thanks, Jeremy.
Any other questions right now?
>>Heidi: Nope, not on this end.
>>Dana: Okay, we are almost done.
I'll just go to my last slide.
So as always, researchers always have all
these caveats and things you should, you know,
we'll worry about this and worry about this
before you release any of this.
So I couldn't leave you without giving you
some caveats.
So certainly the process I'm describing is
meant to be iterative and collaborative, and
I'm sure you've gotten that from what I've
been saying that, you know, involvement of
stakeholders and staff at all levels in addition
to other researchers is really important to
making meaning, but you also have to tolerate
that it's an iterative process so that it's
not meant to be a one and done like here's
your predictive analytics results.
It really has to be a back and forth of, "Oh,
why did we find this about this allegation?
Oh, maybe we should also incorporate this
other variable or look for an interaction
effect."
And that's happened many times in these processes
where we go back and we treat the data slightly
differently as a result of what we learned.
The second thing is that, and this pertains
to the larger debate in the field about the
legitimacy of predictive analytics, is that
it's really important to be transparent and
all of this.
In every case I've described, the models that
we've built, we've shared with everyone.
We said, ''This is what we put in it, this
what we got out of it."
When we refine it or change something, we
put that out there.
When we decide how we're going to use it,
it was really clear with people about how.
Because I think a lot of the objections to
predictive analytics that it's kind of a black
box.
People don't understand what's going into
coming up with those decisions.
And in this case, in PII, we're talking about
eligibility decisions, may not be such a big
deal, but when these models are being used
for other things, there are concerns about
transparency.
Along those lines, we think it's really important
to consider where you wanna plug in this powerful
analytic output because you don't wanna make
some decisions based on quantitative information,
and this speaks to the last point, the predictive
algorithm outputs are not meant to supplant
clinical judgment, which can take into account
many factors that may not be measured, as
one of you mentioned, you may not be measuring
strengths or protective factors or other things
that a caseworker or an administrator could
take into account when looking at a case.
So really this approach is meant to take a
lot of information and distill it as a guide
to making decisions or planning intervention
but not meant to be used instead of people
thinking about what should be done.
So that's kind of the conclusion of my prepared
comments with three minutes to spare.
I don't know if there are any remaining questions.
>>Heidi: There was one that came up about
defining frequent involvement was the definition.
>>Dana: Oh, yeah.
So in New York City, they had the frequently
encountered families work group and it met
I think for about a year before settling on
some definitions.
And the definitions, I have to go back to
them, but they were things like more than
one investigation within six months or...there
were like the six different outcomes that
we were interested in predictiving.
And they were kind of...they varied, you know,
one had to do with intergenerational involvement.
So frequent involvement, meaning we saw this
case when this kid was a child and then we
saw the child be apparent in a case, and that
was also considered one definition of frequent
involvement.
So whatever the outcome was of the six, we
then had to operationalize it in the data,
come back to the group and say, ''Okay, this
is how we're defining this.
We're gonna use, we're gonna look back in
the last six months.
In any case that had more than one investigation
is gonna count as a hit, like that they experienced
this."
And then we had the group, you know, debate
it with us, some were really clear cut and
others were a little more difficult, like
the intergenerational one was a little harder.
>>Heidi: If there are any other questions
at this point, please go ahead and send them
along in the chat box.
I don't see any others.
All right, thank you, Dana, very much, really
appreciate it.
And thank you, everyone, for your time.
I really appreciate you being here.
>>Dana: My pleasure.
Thanks for having me.
>>Heidi: Take care y'all.
