Before we get started, we are
required to inform all attendees that this session is being recorded.
Microphones and webcams have been disabled
to protect your personally identifiable information.
Communicate and for today's session will be via the chat box.
And with that, I'm going to turn it over to Jan Kucklick to get us started.
Jan. Hi. Thanks. Thanks, Shannan.
Good morning.
Good afternoon, everyone, and welcome to this webinar.
This is, as you all know, a webinar is part of a series of webinars
that we've been doing for our CRRC applicants
who were invited back to submit full proposals,
recordings from all of the webinars we have done, including
this one are on the CRRC
website under the Technical Assistance tab.
So if you need to go back and re-watch any of the prior ones
or view any of the prior ones, if you weren't able to attend,
they're all there now.
And another reminder that our frequently asked questions on
the website is constantly being updated.
So always check that for any questions you may have and answers.
And then another note that the fellowship webinar, which is one of the webinars
that we did last week,
that we will be having a separate FAQ section of the website
devoted to questions and answers for the fellowship,
and that should be on the website next week.
So today on this webinar, we're going to hear a little bit
more about monitoring and measuring performance.
As we've done in the past,
we're going to hold all the questions until the end,
but don't hold your questions in your head.
Go ahead and put it in the chat.
That's where we'll be taking all the questions.
And so as a question occurs to you
go ahead and put it in the chat and then at the end of the webinar,
we'll come back to those questions and get some answers for you.
So with that, I'd like to introduce our speaker today.
Our speaker is Michael Migliori and he is with NOAA's Office
for Coastal Management. So I'll turn it over to Michael.
Hello, everyone.
My name is Michael Migliori and I am
the NOAA Office for Coastal Management Lead for performance
measurement and evaluation.
And I'm happy to be here today to talk to you a little bit
about the Climate Resilience Regional Challenge
and how performance measures and evaluation fit into this program.
A little bit about NOAA's overall strategy
for evaluation of this program.
How evaluation and performance monitoring fits
into the notice of funding opportunity
and the criteria for
scoring of the applications.
And then also a little bit about the performance measures
for the program itself and how we're going to go
about organizing them and collecting those measures.
So let's get into talking about
the importance of performance monitoring,
monitoring and measuring performance.
NOAA and the Office for Coastal Management.
We have a robust history of performance
measurement and evaluation for our various programs,
and this new program under the Inflation
Reduction Act is no exception to that.
Really, when we
think about performance measurement and monitoring,
a lot of it has to do with understanding
exactly what we're doing and being accountable
to the American people and the funding and the trust that has been put
within our program and our recipients to implement
what we say we're going to do.
It also is a communication tool
so we can communicate our successes and challenges
and describe the important impact
and communicate
about the issues that we are addressing through the program.
There's also the opportunity to use performance measurement
to improve projects.
So in monitoring a project over the course
of the award period
and collecting data and monitoring, we can find areas of improvement
and readjust to help us meet those goals
and objectives that we've put forward as part of our project.
And also
performance measurement and monitoring can be used for program improvement.
So looking across all the various projects
that are happening under this program
and collecting that information and using that to
inform future programs
that we might have that are similar to this.
And finally, performance measurement
and monitoring we use for evaluation.
And so, you know, NOA
is developing a robust evaluation plan that includes
internal and external evaluation for our investments here.
And so the Climate Resilience Regional Challenge
will fit into that broader portfolio of evaluating our efforts under IRA.
Before we get into a little bit of the detail
on the performance measures specific to
and evaluation specific to
the program here, I just wanted to reiterate that,
performance measurement is only one part of the reporting
that would be required under the grants for this competition.
The notice of federal funding opportunity has details
on what those reporting requirements are.
But just wanted to reiterate that there are requirements
for semiannual reports that would include performance measures, comprehensive
report, financial reports, reporting on property and equipment inventory.
And, you know, just if you have questions on the
broader reporting requirements,
I refer you to the notice of federal funding opportunity.
But we will be talking specifically about the performance measures here today.
So I want to start here with
directing you to the
notice of funding opportunity.
And we include
information on evaluation
as part of the funding opportunity
for the Climate Resilience Regional Challenge.
We talk about monitoring
effectiveness of the proposed activities
and that applies
to both track one and track two projects.
And then for track two, we also talk in the funding opportunity.
about long term effectiveness and implementation
and looking at collecting data on outcomes and impacts. So
the NOFO talks
about this in a little bit of detail
and talks about it in two different ways
or talks about it in different ways for the different tracks.
For both tracks,
the funding can be used for evaluation
activities, but only during the period of the award.
So when you're planning your application and you're thinking about
including evaluation or performance measurement activities,
only evaluation of performance
measurement that occurs during the period of award are eligible
to be funded through this competition.
And really, when we think about the track one and track two,
track two has a monitoring component that maybe track one
does not have as an activity, but these bullets here are taken directly from
the funding opportunity
as it relates to where evaluation fits in.
So evaluation
is included as a criteria for the scoring for the applications.
It is included in the technical and scientific merit criterion
for both track one and track two.
I would refer you to the funding opportunity to look at the points
that are required under this or that are available under this criteria.
But I want to point out that evaluation is
one part of this criterion.
So there are other things beyond evaluation that are included here.
And for track one, the
collaborative building and strategy development.
The question that is asked for
the review is does the proposal include activities for evaluating
and reporting performance of overall effort
and specific adaptation
actions?
So that is something that the reviewers will be looking for
in the technical and scientific merit
portion of the application.
And for track
two implementation of resilience and adaptation strategies.
Again, it's in the technical and scientific merit portion of the review
and it's one part of that criterion
the reviewers will be considering.
Does the proposal include activities for evaluating and reporting performance
of overall efforts and specific adaptation actions, including anticipated social
and economic benefits include plans for long term monitoring.
So just so we can go back and forth
between the two and see the difference.
The track two has this included portion
of longer term monitoring,
anticipated social and economic benefits,
and going back to track one,
those portions of evaluation and monitoring
are not included and not expected.
Or not part of the
review criteria.
So I've just spoken about the specific
element of that scientific
merit criteria as it relates to evaluation.
But evaluation elements
can also appear in
other parts of your application and thus
other parts of the proposal scoring.
So when you're thinking about what
type of evaluation or monitoring
that you want to include in your proposal,
you would want to think about adequate capacity
to conduct performance monitoring, reporting and evaluation.
Do you have the staff that has that expertise?
Do you need to provide training for that?
Are you going to fund
for external contracts, for monitoring and evaluation?
Are there partnerships that you are going to leverage
or that you could leverage for evaluation and how the timing
of your milestones and evaluation fit into your project plan?
So when we talk about
evaluation,
The evaluation element can appear in more
than that scientific technical element.
So the following slides
are going to be looking at project level performance
measures for the Climate Resilience Regional Challenge.
Detailed guidance on the
collection frequency, definition of terms,
data collection mechanisms.
Those things are being finalized,
the actual terminology and phrasing is being finalized
and that guidance will be provided to successful applicants
during the grant negotiation stage or shortly thereafter.
So just want to share with you
the types of measures that are going to be part of the projects.
They are organized according to the four priority issues,
four program priority issues.
Some of them will apply to track one, some of them will apply to track two
and some of them will apply to both tracks.
We are expecting that each project
will be selecting a set of project level measures
that are relevant to their project and their goals and objectives.
The projects can create
other performance measures that are relevant to their projects
and goals and outcomes that do not appear in this list or the following slides.
And we want to make sure that we are measuring things
that are relevant and appropriate to the projects
and getting to the outcomes under the four different priority areas.
And ultimately the selected projects will be developing
targets for their measures in consultation with the Office for Coastal Management.
So the expectation is that each project
will have a set of performance measures
that are relevant to the project and its goals and objectives
and that targets will be developed for those measures.
And we will be, and the projects will be reporting on them
through the course of their awards.
So just to walk through the measures
that are organized by the four program priorities.
Program priority one is the risk reduction.
So acquisition, the number of acres acquired
habitat restoration, number of acre restored
natural infrastructure, linear feet, square feet,
acre feet of improvements, depending
on the type of natural infrastructure
planning, number of multi-jurisdictional adaptation plans and strategies
and policies, number of laws, regulation policies, standards to reduce
risk or enhance resilience and number of communities that benefit from those.
Some of these are broad and not defined yet and so the forthcoming
guidance will lay out the reporting
requirements, define these elements
and clarify outstanding issues
so that there is uniformity in how we collect the measures.
Program priority two
regional coordination and collaboration, collecting information on the number of
coastal counties benefiting from challenge supported activities
and number and type of entities receiving challenge funding.
Program priority three, enduring capacity.
Number of good jobs supported, number
of training and capacity building activities completed.
Number of individuals trained and engaged in capacity
building number and type of data
sets, information products and tools developed to inform resilience, planning
and implementation And then program priority four, equity and inclusion.
Number of Justice 40 Communities participating number of tribes, tribal
organizations or Indigenous organizations serving as lead entities.
Number of tribes, tribal organizations, or other organizations
serving Indigenous communities that are collaborators or partners.
Total funding awarded to tribes, tribal organizations,
or other organizations serving Indigenous communities.
Number of tribes, tribal organizations,
or other organizations serving Indigenous communities participating.
Number of disadvantaged communities serving as lead entities
and amount of funding provided.
Number of disadvantaged communities
serving as collaborators or partners, and the amount of funding provided,
and number of disadvantaged communities participating.
So again, that's a long list of measures.
And I just want to reiterate that it's not expected that
every measure would apply to every project
and that you are empowered
to develop additional measures that meet your needs
and are relevant to your projects and the goals and objectives,
and that projects would be selecting
a subset of these measures and developing targets
and working with NOAA's Office for Coastal Management
to develop those targets
and incorporate them into the projects
during the negotiation stage
of Grant making.
And this slide just
kind of summarizes what I just said
in that the process
for performance measurement
for the Climate Resilience Regional Challenge
involves
identifying the project level measures
that are relevant to your projects,
developing targets
and working with NOAA, Office for Coastal Management on that,
and then the reporting process
ongoing throughout the course of the award.
So in addition to this webinar
and a user guide,
there are resources
that are accessible that can help you
with developing
measures related to resilience.
So this first one is resiliencemetrics.org,
and that is a project that was supported by, funded
through the National Estuarine Research Reserve System Science Collaborative
on Developing Resilience Metrics.
And then we also have a tool on the Digital coast,
Coastal Community Resilience Indicators and rating systems.
And so those are resources that are available to you
to help you think about approaches to performance measurement
and developing
performance measures for your projects.
And Jan, I think.
Do you want to talk about the technical assistance?
Yeah, Well, so a few things.
First of all, Michael, one question for you.
We also have the job aid that is available as well.
So Shannan, yup thank you.
Shannan just posted that in the chat.
So do you want to speak to this real quickly?
Michael?
Sure.
So the job aid
that was just shared is a few pages
and it walks through some of the considerations
for developing
performance measurement and evaluation.
Kind of asks some questions
to prompt you to think about different aspects
about performance measurement and your projects
and is a good place to start
when you're thinking about developing measures.
Or considering which measures of the existing measures
that we shared are relevant to your projects.
Good.
Thanks, Michael. Yep.
So folks, we encourage you to take a look at that, if that will help.
And so we've got a couple
a couple of questions have come in.
Before we do that, actually,
thanks for those questions.
We've got just a couple quick polls that we want to ask,
if you wouldn't mind going to the Triangle Circle Square
and click on the two polls, Shannan, just launched both of them.
One is just checking in on if you're considering requesting
some technical assistance for evaluation and performance measures
between now and when application proposals are due.
That will give us a sense of
potential questions that might be coming in after this webinar.
So let us know about that.
And then also the question that we always like to ask about
if this webinar was a good use of your time.
So appreciate you looking at and taking those two polls. And
then what I'd like to do is
go to some of these questions that have been coming in.
Michael So the first question I think actually I can answer,
which is just asking if these resources will be on the Web page soon.
And the answer is yes.
Michael had quite a bit of information in those slides.
I think the checklist that we just shared will help a lot, but we will definitely
be getting those resources in as well.
The next question is around qualitative evaluation.
Is this something that NOAA is interested in?
Is that something you can answer Michael?
I will answer that, yes.
It's something that NOAA is interested in and I
hinted at it at the beginning of the presentation
in that NOAA has a broader strategy
for evaluation still under development for
all the efforts under IRA,
as well as the bipartisan infrastructure law
and qualitative evaluation
is included in that.
It does not appear in
this presentation in detail here
focused mostly on performance measurement, but we also are interested
in qualitative evaluation, if that's part of the evaluation plan
that the proposals would like to include.
So evaluation
broadly is included as one of those criteria.
And so that would include qualitative
evaluation, if that's something you would like to include.
All right, Thanks, Michael.
Okay.
So another question regarding reporting.
Does NOAA want detailed scientific monitoring reports for each action?
And if so, is this only when NOAA funding is going towards that monitoring?
And if not, are annual project progress tracking
reports okay with some information on monitoring results?
Sure. So it's a great question.
I think it would be dependent upon the activity
and I think the specific approach that would be decided,
we'd cleared
that up in our negotiation if the award is selected for funding.
So I think an applicant should propose
what they think would be most effective.
That would clearly satisfy the evaluation criteria
and then if NOAA wanted to augment
or modify an approach, we would
work on that cooperatively in the negotiation phase.
And there is a data management
or monitoring plan that's required, I believe, as part of the application.
Is that correct?
Would that
help inform how that might be approached?
Yeah, I think so. Michael.
All right. Thanks, Adam and Michael.
All right.
Let's see. Next question.
Should we provide estimated targets for each measure within our CRRC proposal,
or should we just identify anticipated measures
and wait for the negotiation period to develop each target?
I'll go ahead and answer that one and
you can develop
targets as part of your proposal,
but if you don't,
you will be working with NOAA to develop those targets during negotiation.
And if you
include targets as part of your proposal,
NOAA will be negotiating and working with you to finalize those targets.
So there's no requirement that you create targets.
But if you are able to and would like to,
that can be included as part of the proposal.
thank you.
Let's see, for proposals with multiple elements
that fit within multiple program priorities.
Should we identify measures by element or for the overall project?
Michael, I'm happy to take that one.
Yeah, I think both.
So if there are activities that
for example, you're conducting
a bunch of trainings, right, we would want to understand
the participants that are attending those trainings.
And then if you're conducting other types of activities where you're
also engaging with constituents, we would want to know them too.
And those may roll up into like a total number of constituents engaged.
So I think I think it would be both.
And again,
we would certainly work with our recipients during the negotiation
phase to make sure that everybody was on the same page in terms of
what would be, you know,
moderated and reported for NOAA’s purposes, but also ensure
that the evaluation needs met other,
the applicant recipient's needs as well.
Thanks, Adam.
And someone noted it's important to avoid double
counting, correct?
Yes. Yeah, obviously.
Yeah.
I think it would be important to avoid double counting.
Great. Thank you.
All right.
Next question has a couple parts to it,
so I'll read the whole thing, Michael, and then I can reread pieces of it.
Is there a standard methodology applicants should use to calculate job creation?
What is the timeframe for job creation?
We have some actions where positions will be created
for six weeks and others where jobs are created for three years.
Which time frame should we use?
And then should job numbers
only be for FTEs?
Okay, I'll start with that.
I'll start with answering that one. In
that is a measure
that is being collected across
NOAA for the different programs under IRA.
I think it might even be something that's being collected across
all of the Department of Commerce
and so NOAA’s Office for Coastal Management is awaiting
instructions and further details about how that's going to be counted.
And captured.
That may not be a satisfactory answer to get to some of those details.
And those details are very important about
the extent,
you know, for FTE versus temporary.
And we just don't have that captured.
We don't we don't have those directions just yet.
But I would encourage you,
if this is an important part of
your project proposal, to
consider developing
some measures related to that.
And again, we could work
during the negotiations to fit that
within the parameters that NOAA and commerce are developing.
Okay, great.
Thank you.
Another question Will there be an opportunity
to discuss protections for sensitive indigenous knowledge
and data during the negotiation process with NOAA?
Yeah, absolutely.
That definitely would be part of our discussion.
We really value the importance of data sovereignty
and we would certainly be able to address
that in a in a negotiation about a data management plan.
All right.
Right now, there are no, oh wait here we are.
We have a couple more.
Can the data management plans be high level documents
per resilience action that simply outlines
how the monitoring data will be saved and disseminated?
I'll take that one too.
I think, for specific elements of a data management plan.
I think the most important thing to remember is that
it is a requirement.
And I'll speak to more of this later that
there will be,
every application will have things
that need to be adjusted or corrected.
And there will be things that are specific
to specific applications dependent upon the activities.
So I think
we'd have to understand more specifics about this particular situation.
So you could send an email to resiliencechallenge@noaa.gov,
and we could get into more details there,
but just know that we don't expect
all of the forms to be perfect when they're submitted.
What's most important is that the reviewers understand your approach
and then if there are technical issues and NOAA identifies,
we would work with you to adjust those
during that negotiation phase.
Thanks, Adam.
And so another data management plan question.
Can you clarify and I think you may have
already said this, but let's just double check,
Can you clarify
whether the data management plan is required at the time of application
or must be developed if selected for award
during the negotiation period?
I'm 99%
sure I want to go to the NOFO and check, but I'm 99%
sure that the data management plan is required as part of the application,
and then finalizing that would be a part of our negotiation.
And I will
check and lean on one of my colleagues Jan, and we'll get back to everybody
in just a second. Okay.
So we will double check that triple check that and answer to a question
that was just posted.
Yes. These questions and the responses
will be incorporated into our FAQ section of the Web page.
So we will be putting them all in there as well.
All right. Another question.
If we are working with different communities, can the evaluation criteria
and protocol vary by site
to allow for community self-determination?
There will
also be an overarching holistic evaluation across the sites.
So the question is can the evaluation criteria and protocol
vary by site to allow for community self-determination?
I'm seeing Adam nod.
I would say absolutely, and
especially if that is
what's needed.
So and in regards to the previous question, Jan,
we did confirm that a data management plan is required as part of the application.
All right.
I think so.
Yeah, is required, but that can be modified during negotiation as needed.
All right. Thank you.
And thanks for answering the question about
yes to the evaluation criteria and protocol can vary by site.
Another question In your experience, what have been
the most successful metrics to highlight?
Example, are there any successes or failure stories that come to mind?
I don't know that we've ever run a competition with performance metrics
quite like this before.
Many of our grant
programs, we have defined performance measures
that are required and here we are
given the scope of the projects and the various types of projects.
We are providing a suite of measures
to choose from or to develop your own.
I would say, though, that
knowing that there are
four priority areas
that are part of this program
thinking about addressing
all the priorities would be important.
That's something that I would say that would be important
is to look to see that it's comprehensive
of all the aspects of the program.
I can't think of any pitfalls right now.
All right.
I'm going to
give folks another opportunity if you have any other additional questions
for Michael or Adam on the performance
measures and monitoring, go ahead and throw them in.
Why you may be typing or thinking about this.
I'm actually going to turn it over to Adam.
He had a few additional more general competition
things that he wanted to share with you all.
So. Adam.
Yeah, Thanks, Jan.
So I want to thank everybody for sending in questions
through the ResilienceChallenge@noaa.gov Email address.
We anticipate getting more questions as the deadline approaches.
We're doing our best to respond soon as possible.
Our target is three days.
So please, we appreciate your patience
and we just really encourage you to make sure that you've
reviewed all the resources
that we've provided on the website prior to sending those questions in.
So I also want to reinforce the comment I made earlier.
You know, there's no such thing as a perfect proposal.
There's never been an application that's been submitted where our grants
management division doesn't identify
parts of forms that need to be corrected
or where more clarification is needed.
So that will come no matter what.
So I really encourage to spend your time
making sure that your application is clear and that reviewers are going
to really understand what you're trying to accomplish, the value and the impact,
because
those technical issues are going to be there in every application
and they'll be an opportunity to address them.
So I'm just going to go through a few questions
that we've gotten quite a number of and
hopefully these will prevent you from having to
ask the same.
So we have gotten questions about the SF-424 forms
and which only has room for four years.
But if you're working on a five year budget, how do you show that fifth year?
So you can use another SF-424A form for the fifth year,
so you can just add another page so that there's
a total of five years.
That SF-424A,
only one is needed for each application
for the lead applicant.
So there are some links, as I mentioned on the program website
related to our budget template and our budget guidance.
So for people with questions about the SF-424A
refer to those links to our budget guidance.
In terms of
links, people asked if they can include links in the application.
So our reviewers are not required to review the links.
You can keep them, you can include them, but if you have critical information
that you want to ensure the reviewer reviews, it should be included
either in the narrative itself or as part of the supplemental materials.
So just wanted to reinforce that.
Got some questions about the environmental compliance questionnaire.
The environmental compliance questionnaire is only required
if the award is selected for funding.
It is not required to be submitted as part of the application.
So just wanted to clarify that as well.
We got some questions about letters of intent
and specifically where they should be sent.
So we want to be clear that if you mail letters
of intent to NOAA, the reviewers will not see them.
Hey. Hey, Adam, I'm sorry to interrupt you.
You said letters of intent. Did you mean letters of support?
I'm sorry. Yes, thank you. Letters of support.
Thank you, Jan.
So, letters of support must be included as part of the application.
If they're going to be seen by reviewers,
NOAA is not able to take up or modify the application.
So we can't take letters that we get from the mail
and then include them in an application.
So if you want the letter of support to be seen by reviewers,
it has to come in with your application.
If for any
reason you wanted to submit one directly to NOAA,
you could mail it to us
and we're going to put the address in the chat.
If you're looking for an individual
to address that letter of support to, you could just list,
you can address it to the NOAA selection official.
Let's see,
we've gotten some questions about future funding opportunities.
Will we run this competition again?
So we have not been appropriated additional funding for this program.
So we are not currently planning to run another competition.
With that said, it states within the notice of funding opportunity
that if additional funds were appropriated that we would not have to run
another competition, that we could find applications
that were received through this competition in future years.
We also got a couple of questions
about whether it was allowable to have an evaluation done in-house.
Yes, it is allowable to do in-house evaluation.
We do expect that internal
or project funded staff would be conducting the monitoring
and reporting for those NOAA performance measures.
But if the scope of the evaluation would require
some unbiased perspective, then
an external evaluation may make more sense, but
it's not a requirement.
And then lastly, we've been getting a couple of questions
about whether project directors need to also
have an account in the ERA system.
And yes, they do.
The applicant organization and the project director must be registered.
So I'm hopeful that those are some of the repeat questions we've got.
So I just wanted to
communicate this
here and hopefully they are helpful for you.
Thanks, Jan.
Thanks so much. Adam.
And we've had one more question come in
regarding this webinar.
So for either Adam or Michael,
is there an example of a data management plan that we can share?
We will look into that and if we are able to find one,
we can put it on our website.
It's one of those resources.
So I can't promise that we have one, but
we will do our best to find one.
All right.
Thanks, Adam.
All right.
I have not seen any other questions come in.
So as a reminder, as Adam mentioned, if you do have additional questions
around this topic or anything else around the competition,
check out the website, review that FAQ list.
And if you don't see your question there, then go ahead and send it
to the ResilienceChallenge@noaa.gov email address.
And also just another reminder that there's still a little bit of time
left to request that one on one technical assistance that we are offering
and the link is to complete to request that assistance.
The link to that form is on the website.
So that's just another reminder about that.
And with that,
we have one more question, and I don't know, Adam,
if you want to try to answer this one, and it may be on the FAQ.
If there's a government shutdown, would that affect any of the deadlines?
We do not anticipate it would affect our submission
deadline at this point, but we are we are planning.
But we do not plan to extend the deadline at this point.
All right.
Thank you.
All right.
So with that, I'd like to thank you all for joining us today.
This is the last webinar in our series.
And again, just a reminder to look at the website,
the FAQ list as well as the past webinars.
And thank you all so much. Have a great rest of your day.